Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Runner-files-blobs · L-261015-0db4de (#1438)
Louis Choquele51a98dd4567
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a40fb40e9..e096a6ec3 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,11 @@- **The method file store**: a step reads a file of its method where reading is allowed, at the moment it uses it, through `MethodFileStoreProtocol`, whose `read_tree_listing(tree_digest=..., read_scope=...)` and `read_blob(sha256=..., read_scope=...)` return a tree's listing and a file's bytes under the run's read scope, each read checked against the tree's digest (`pipelex.core.method_trees.method_files`). The local store, `LOCAL_METHOD_FILES`, serves the trees this process's libraries hold, and a plugin whose activities run apart from the process that loaded the method claims `HubSlot.METHOD_FILE_STORE` with `PluginRegistrar.claim_method_file_store`; a file the store does not hold for the run fails its step with the new `MethodFileUnavailableError`. An orchestrator running a job out of process copies every tree the job's crate names into the store its workers read before its `execute` or `start` returns, since the runner may drop the files as soon as it does, and `read_stored_trees(tree_digests=..., read_scope=...)` reads them for that copy, each as a store keeps it, its listing under its digest and its files under their hashes. The local store finds a file by its hash in one lookup and reads a file on disk off the event loop.- **Relative URLs in HTML templates read the method's own files**: a relative URL in a `PipeDocGen` HTML template, inline or a template file, such as an image's `src`, a stylesheet's `href`, a font, or a `url()` inside a stylesheet the template links, names a file of the template's method tree, resolved by the MTHDS standard's rules against the template file, or against the bundle file of an inline template, and read through the method file store under the run's read scope wherever the document prints, so a logo, a stylesheet and a font travel with their method. Only images, stylesheets and fonts are read, by suffix, so a URL a run's input supplies cannot make a document carry the method's other files. A URL the template writes as it is is checked when the method loads, and a URL built from the inputs that climbs above the tree's root, names a folder, a missing file, a file the tree's membership rule keeps out or a file that is no asset fails the print with the new `TemplateUrlRefusedError`, an input error naming the URL and the template; a URL with a scheme, such as `https://`, `data:` or `pipelex-storage://`, is read as before.- **`RenderJob.base_url` and the method-file arm of `RenderResources.load`**: an HTML job whose template belongs to a method tree carries the URL a document engine joins the document's relative URLs to, under `METHOD_FILE_URL_PREFIX` (`https://method-files.pipelex.invalid/`, which never resolves), and `RenderResources.load` reads a URL joined under it, a stylesheet's own URLs included, from the document's method tree, with no media type; `DocumentComposition.url_base` carries the template's place in its tree to the print stage. They join plugin API version 7, and an engine that never reads `base_url` keeps refusing a relative URL.+- **`StorageProviderAbstract.load_size`**: a storage provider reads an object's size without its bytes, from a file's metadata, an S3 `HEAD` or a GCS object's metadata, and a provider that does not override `_load_size` measures the object it loads, so an external storage plugin keeps working.+- **The API server's blob routes, `POST /v1/blobs/missing` and `PUT /v1/blobs/{sha256}`**: the server keeps a blob store in its configured storage, under `<scope>/blobs/<sha256>`, declares it with `blob_store: true` on `GET /v1/version`, and serves the MTHDS protocol's blob routes, so a sender asks which of its method's files the runner lacks, uploads each one to the signed target the answer names, and then sends the method by hash; the scope is the request's `read_scope`, else the caller's id, the targets start with `PUBLIC_BASE_URL` when it is set, else with the URL the request arrived at, are signed with `BLOB_UPLOAD_SIGNING_KEY` (a key per process when unset) and expire after `BLOB_UPLOAD_URL_TTL_SECONDS`, and once the server has read a tree in full it records its listing under `<scope>/trees/<digest>`, so asking with that digest answers in one lookup for blobs of that tree. A blob counts as held only at the size the sender states, read from the storage's metadata, so bytes held short are asked for again and written over by their upload. `BlobMismatch`, `BlobUploadNotAuthorized` and `BlobStoreCorrupted` are new `error_type`s.+- **`files` takes every entry form, beside `dependencies`**: on `/v1/execute`, `/v1/start` and `/v1/validate`, each value of `files` is the file's text, `{"base64": "…"}` or `{"sha256": "…", "size": n}` naming bytes the blob store holds, so a method's templates and other binary files travel with it, and `dependencies` carries the method packages it calls by address, each a map of its own keyed by its address; `/v1/validate` also takes `read_scope`. A hash the server does not hold is a `422` `MissingBlobs` listing the hashes in `missing_blobs`, a hash it holds at another size than the entry states a `422` `BlobMismatch` listing each with its held size in `mismatched_blobs`, checked before any byte is read, and a tree over `MAX_METHOD_TREE_FILES`, `MAX_METHOD_TREE_FILE_KIB` or `MAX_METHOD_TREE_TOTAL_KIB`, the files `resources` joins to it included, or a method with more than `MAX_METHOD_DEPENDENCIES` dependency trees or more than `MAX_METHOD_REQUEST_TOTAL_KIB` across its trees, a `422` `MethodTreeTooLarge`.+- **`method_provenance.tree_digest`**: a run by `method_ref` reports the digest of the method tree it loaded beside the commit it was read from (`MethodProvenance.tree_digest`).+- **Validating a method directory**: `validate_bundle`, `validate_bundles_in_process` and `BundleValidatorProtocol.validate_bundles` take `method_dir`, a method's tree validated whole as the caller's, with `methods_dirs` holding the packages shipped with it, every `.py` file of the tree compiled for its syntax and refused by its tree path and line, and the graph drawn from the tree's entry pipe; `write_method_tree` and `WRITTEN_METHOD_TREES` (`pipelex.libraries.written_trees`) write a tree out and make every listing of its directory, while it is held, the tree as written.### Changed@@ -29,16 +34,25 @@- **One method tree's Python per process (Breaking)**: a run holds the Python of the tree it imports from until its library is torn down, and a step of another tree that would load its Python meanwhile is refused with `TreePythonHeldError` naming both trees by a bundle each holds and the start of its digest, never by a path of the host, so a run whose own steps and a dependency package's steps both call Python is refused locally; once no run holds a tree, its root leaves `sys.path` and the next tree evicts every module imported from a file of it, whatever imported it. A top-level module of a tree named like a standard-library module or like a module importable outside the tree is refused at validation, `structures` excepted, and on a sandbox-hosted deployment, whose sandbox holds its own packages, only a standard-library name is, and while a tree is held, importing a file its membership rule keeps out, such as a `.gitignore`d module, is refused naming the rule, and no bytecode is written into the tree.- **The function registry is retired and `@pipe_func()` takes no argument (Breaking)**: `FuncRegistry`, `func_registry`, `FuncRegistryUtils`, `FuncRegistryError`, `RuntimeHub.get_func_registry` and the `name=` argument of `@pipe_func` are removed, no function is discovered or imported while a library loads, and `is_marked_pipe_func(func=...)` tells a function carrying the marker.- **Kernel func ops take the function (Breaking)**: `pipelex.kernel.func_ops.call_registered_function` becomes `call_function(function=..., memory=...)`, and `run_func` takes the `function` itself in place of `function_name`.-- **Plugin API version 7 (Breaking)**: `PLUGIN_API_VERSION` moves to 7, so a plugin declaring `targets_api = 6` is refused at discovery with `PluginApiVersionMismatchError`: a bundle validator's `validate_bundles` receives `submitted_files` in place of `bundle_resources`, a PipeFunc executor's transported request names its tree by `tree_digest` and carries no `files`, the transported entrypoint takes the directory the call's files were unpacked into, an orchestrator running a job out of process copies the job's trees before it returns, and `claim_method_file_store` joins the registrar's menu.-- **`LibraryCrate` names its method trees by digest (Breaking)**: `python_sources`, `bundle_resources` and `host_bundle_files` give way to `tree_bundles`, which names each pipe's bundle by its tree's digest and its tree path (`TreeBundleRef`), so a crate carries no file's bytes, no listing and no path of the host that assembled it, a pipe built from it resolves a file its bundle names lexically and reads nothing, a Temporal workflow loading it included, and its fingerprint ignores the trees. Bundles loaded from content strings sit at the root of the tree of the files submitted with them, which the files of a library directory whose own tree holds no bundle, such as the directory a server writes a request's Python into, join; two directories submitting different files at one path refuse the load with `BundleResourcePathError`, naming the path in the tree alone and joining none of their files, and a structure class is refused in every tree the load lists, a package or repository nested in a library directory included. A load that validates a crate builds each pipe as a member of its tree when a library of this process holds the tree under the digest the crate names, so a `PipeDocGen` HTML template file is checked as it was where the crate was assembled, and refuses such a step with the new `PipeDocGenTemplateFileUncheckedError` when no library holds its tree, rather than passing its template unchecked; a load skipping validation still reads no file. `LibraryCrate.concepts_only()` gives the crate's concepts and domains without its pipes.+- **Plugin API version 7 (Breaking)**: `PLUGIN_API_VERSION` moves to 7, so a plugin declaring `targets_api = 6` is refused at discovery with `PluginApiVersionMismatchError`: a bundle validator's `validate_bundles` receives `submitted_files` in place of `bundle_resources` and takes `method_dir` and `methods_dirs`, its `mthds_contents` becoming optional, a PipeFunc executor's transported request names its tree by `tree_digest` and carries no `files`, the transported entrypoint takes the directory the call's files were unpacked into, an orchestrator running a job out of process copies the job's trees before it returns, and `claim_method_file_store` joins the registrar's menu.+- **`LibraryCrate` names its method trees by digest (Breaking)**: `python_sources`, `bundle_resources` and `host_bundle_files` give way to `tree_bundles`, which names each pipe's bundle by its tree's digest and its tree path (`TreeBundleRef`), so a crate carries no file's bytes, no listing and no path of the host that assembled it, a pipe built from it resolves a file its bundle names lexically and reads nothing, a Temporal workflow loading it included, and its fingerprint ignores the trees. Bundles loaded from content strings sit at the root of the tree of the files submitted with them, which the files of a library directory whose own tree holds no bundle join; two directories submitting different files at one path refuse the load with `BundleResourcePathError`, naming the path in the tree alone and joining none of their files, and a structure class is refused in every tree the load lists, a package or repository nested in a library directory included. A load that validates a crate builds each pipe as a member of its tree when a library of this process holds the tree under the digest the crate names, so a `PipeDocGen` HTML template file is checked as it was where the crate was assembled, and refuses such a step with the new `PipeDocGenTemplateFileUncheckedError` when no library holds its tree, rather than passing its template unchecked; a load skipping validation still reads no file. `LibraryCrate.concepts_only()` gives the crate's concepts and domains without its pipes.- **A `PipeFunc` call names its method tree (Breaking)**: `PipeFuncExecutionRequest.tree_digest` names the tree of the called step, and the process running the call reads its files, every one but the `.mthds` files and a generated `structures` module, through the method file store under the run's read scope (`read_pipe_func_call_files`). The direct executor's transported run writes them into a temporary directory, generates `structures` unless the method shipped one, imports the function by its import path under the process's hold on that directory and unloads its modules before removing it; the transported entrypoint takes as a required third argument the directory a sandbox client unpacked the files into, since a sandbox reaches no method file store, and runs `DirectPipeFuncExecutor.run_pipe_func_transported_at` on it. Only the called step is built in the tree written for the call, which holds none of the tree's bundles: `load_from_crate` and `rehydrate_library_and_memory` take `written_pipe_trees`, keyed by the step's pipe ref, and every other step, a `PipeDocGen` of the same tree included, is built as the crate names it, its template file named by the crate's digest. The transported library's id is public, `TRANSPORTED_LIBRARY_ID`.- **A `PipeDocGen` template file is read where the document prints (Breaking)**: `DocumentComposition.template_file`, a `ComposedTemplateFile` naming the file by its tree's digest and tree path (`MethodFileRef`), replaces `template_path` and `template_resource`, and the print stage reads it through the method file store under the run's read scope, renders an `.html` one with the step's data, carried as a `TemplateRenderContext` (the stuffs the step's inputs name, the concepts and domains of the run's crate with none of its pipes, the crate's fingerprint, and the run's runtime parameters), through the template file stage the interpreter installs at boot, a dry print included, in the run's own library where this process holds it and otherwise in a library loaded from the concepts, which the process keeps for later prints (`RenderLibraryCache`), and hands an office one's bytes to its engine, whose template check moves from the step's dry run into the dry print. An `.html` template file whose bytes changed after its tree was listed is refused at load, never read from the disk unchecked. `make_render_job` is synchronous again and takes the rendered `html` or the template's bytes; an inline `template` is still rendered where the step runs.- **Files submitted with content strings make one method tree (Breaking)**: `PipelexMTHDSProtocol`, `validate_bundle`, `validate_bundles_in_process`, `acquire_library`, `pipeline_run_setup`, `resolve_crate_from_contents` and `BundleValidatorProtocol.validate_bundles` take `submitted_files`, a mapping from the path a step names to the file's bytes, in place of `bundle_resources`, and `LibraryManagerAbstract.add_submitted_files` replaces `add_bundle_resources`. The run names their tree by its digest and carries none of the files, so nothing is stored under the run's `assets/bundle/` before it leaves the process, the 512 KiB bound on `.html` template files is lifted, and files that cannot make one tree, such as two paths equal once case-folded, raise `BundleResourcePathError`. A step names a submitted file by any spelling of its path, so `templates//invoice.docx` reads the file submitted as `templates/invoice.docx`.- **A relative URL in an HTML template names a file of the method (Breaking)**: a literal relative URL of an HTML template, or of a stylesheet it links, that names no image, stylesheet or font of the method's tree now fails the method's load, `pipelex validate` and dry runs included, where it loaded and failed only at print. A relative path such as `images/logo.png` that an engine printing HTML hands `RenderResources.load` as written now resolves in the template's method tree, and is refused with `TemplateUrlRefusedError` when the template belongs to none, where an unscoped run read it from its working directory; a root-absolute local path, and every URL of a document printed from the auto-layout or an office template, read as before.+- **A method sent to the API server is a method tree (Breaking)**: `files` and `bundle_b64` are read as one map under the MTHDS standard's tree rule, so a key holding a dot file or directory (`.mthds/` included), an empty, `.` or `..` segment, a backslash or a colon, a key given twice, or two keys naming one file is a `422` `InvalidMethodFiles` carrying the standard's `tree_refusal` (`code` and `names`) in place of `InvalidBundle`, and a package the method calls travels in `dependencies` beside `files`, while a `bundle_b64` archive still carries it under `.mthds/methods/<name>/`. The whole tree is written out and loaded as the caller's library directory, so a template beside its bundle needs no `resources` entry and counts against the tree's limits rather than those of `resources`, an image, stylesheet or font only a template's relative URL names travels with the method, a file sent in `resources` joins the tree with no step needing to name it, a bundle that does not parse is a `422` naming it by its tree path, and with no `pipe_code` the run starts at the tree's entry pipe by the protocol's rule.+- **`method_ref` is read from its commit and run as a tree (Breaking)**: a package fetched by address is read from the git objects of the commit it resolved to, never from the clone's working tree, and runs or validates as a tree sent in `files` does, its template files and Python included, the packages it ships under `.mthds/methods/<name>/` becoming dependency trees that `/v1/validate` now loads too; its entry pipe is its manifest's `main_pipe`, so a manifest declaring none answers `default_pipe_ref: null`.+- **`/v1/validate` validates a method tree whole (Breaking)**: a method sent as `files` or fetched by `method_ref` is validated as it would run, every `.py` file of it compiled for its syntax, a file that does not parse or is not UTF-8 making the verdict `is_valid: false` with an item at its tree path and line, and its `default_pipe_ref` and graph follow the tree's entry pipe, both `null` for a tree with none; `mthds_sources` accompanies `mthds_contents` alone.### Fixed+- **The local storage provider publishes a write whole**: `LocalStorageProvider` writes to a temporary file beside the key, flushes it, then renames it onto the key, so a full disk or a killed process never leaves a file cut short under a key, and two writers of one key each land whole.+- **A bundle file that is not UTF-8 text is a located diagnostic**: a `.mthds` file read from disk that does not decode is a parser error naming its source and the line and column of its first undecodable byte, so a method tree sent to the API server holding one is the caller's `422` on a run and an invalid verdict on `/v1/validate`, never a server error.+- **Python a method tree holds that cannot parse is its diagnostic on every interpreter**: a `.py` file holding a NUL byte, which Python 3.11 refuses with a `ValueError` where later versions raise a `SyntaxError`, or one nested too deeply to parse, is an invalid verdict located at its tree path, and a `PipeFunc` whose module is such a file is refused as one that does not parse.- **A `template_file` no sent file could be keyed by is a blueprint item at its step**: a bundle loaded from content strings, as the API's routes and `validate_bundle` load one, whose `PipeDocGen` step's `template_file` holds a `..` segment or a NUL character, or names no file, is refused when it parses, whatever files the request sends, as one `blueprint_validation` item with `field_path` `pipe.<code>` whose message names the field and not the path, instead of a load refusal saying the file was not supplied. A bundle read from disk still resolves a `..` in its method tree.+- **A `method_ref` package's template files reach its steps**: a `PipeDocGen` step of a package run or validated by `method_ref` reads the template file the package holds beside its bundle, where the run before had no way to read it.+- **A request's own method that does not load is the caller's `422`**: a method sent to the API server whose bundle does not parse, or whose tree cannot load, is refused as the caller's fault, naming each file by its tree path, instead of a `500` naming the server's temporary directory.+- **A TOML syntax error names the bundle's source**: a bundle read from a file under a source the caller gives names that source in its TOML syntax error, never the file's path on the host.### Removeddiff --git a/api/CLAUDE.md b/api/CLAUDE.mdindex a06cd8abd..4f6f4d032 100644--- a/api/CLAUDE.md+++ b/api/CLAUDE.md@@ -27,8 +27,10 @@ pipelex_api/routes/health.py # GET /health (no auth)version.py # GET /v1/version (no auth — MTHDS Protocol handshake)+ blob_upload.py # PUT /v1/blobs/{sha256} (no bearer auth — the signed upload target authorizes it)pipelex/pipeline.py # POST /v1/execute, /v1/start (MTHDS Protocol run routes)+ blobs.py # POST /v1/blobs/missing (MTHDS Protocol blob route; the signed upload targets)validate.py # POST /v1/validateresolve.py # POST /v1/resolve (closure → normalized crate)pipe_io.py # POST /v1/pipe-io (closure → one pipe's, or every pipe's, I/O artifacts; no dry run)@@ -46,7 +48,7 @@ scripts/hatch_build.py # The metadata hook that writes the dependencies and the lockstep pin```-This server is the reference implementation of the [MTHDS Protocol](https://mthds.ai): `POST /execute`, `POST /start`, `POST /validate`, `GET /models`, `GET /version` under the `/v1` base path, tagged `x-mthds-protocol: true` in the committed OpenAPI artifact (`docs/api-server/openapi/pipelex-api.openapi.yaml` at the repository root, regenerated via `make openapi-export`, drift-checked via `make openapi-check`). Contract nesting: MTHDS Protocol ⊂ Pipelex API ⊂ Pipelex hosted API.+This server is the reference implementation of the [MTHDS Protocol](https://mthds.ai): `POST /execute`, `POST /start`, `POST /validate`, `GET /models`, `GET /version`, and the optional blob routes `POST /blobs/missing` and `PUT /blobs/{sha256}`, under the `/v1` base path, tagged `x-mthds-protocol: true` in the committed OpenAPI artifact (`docs/api-server/openapi/pipelex-api.openapi.yaml` at the repository root, regenerated via `make openapi-export`, drift-checked via `make openapi-check`). Contract nesting: MTHDS Protocol ⊂ Pipelex API ⊂ Pipelex hosted API.**Only the protocol's own operations carry `x-mthds-protocol`** — the flag is how a conformance suite or a third-party runner extracts the portable subset of the artifact, so tagging a Pipelex route would misrepresent the standard. Everything else this server serves is a Pipelex API extension: `/resolve` + `/codegen`, `/pipe-io`, `/models/check`, and `/lint` + `/format`. Watch `/resolve` and `/codegen` in particular: they *look* protocol-shaped (they speak the `/validate` verdict discipline, and the crate `/resolve` emits is genuinely standard-owned — the MTHDS Library Crate Format, so its wire fields stay brand-neutral), but the routes are ours and the standard specifies no type projection at all. `tests/unit/test_openapi_contract.py` pins the tagged set exactly, in both directions.diff --git a/api/pipelex_api/blob_store.py b/api/pipelex_api/blob_store.pynew file mode 100644index 000000000..2267fd67c--- /dev/null+++ b/api/pipelex_api/blob_store.py@@ -0,0 +1,190 @@+"""The runner's blob store: bytes keyed by their SHA-256, and the listings of the trees it holds in full.++A sender asks which of its hashes the runner lacks (`POST /v1/blobs/missing`), uploads only those+(`PUT /v1/blobs/{sha256}`), then sends its method as a map of file entries naming them by hash. This runner keeps those+bytes in its configured storage provider, local or cloud, so a self-hosted deployment gets the shape the hosted plane+has (`wip/extra-files/design.md`, D5 and D6), under one prefix per caller:++ <scope>/blobs/<sha256> a file's bytes, written once and never changed+ <scope>/trees/<hex> a tree's listing, under the hexadecimal part of its digest++The scope is the run's trusted read scope, the one a multi-tenant host sends for the organization it serves, or else the+caller's own id (`single-tenant` on a deployment that identifies no caller): a hash entry resolves only in the caller's+own scope, never across tenants, so asking whether a hash is held tells a caller nothing of another's files.++A listing is written only after every file it names is held, by this runner once it has read a request's tree in+full, and never by an upload: `PUT /v1/blobs/{sha256}` writes under `blobs/` alone, so an uploaded file that happens to+be a well-formed listing proves nothing. A listing present means the whole tree is present, which lets+`POST /v1/blobs/missing` answer a tree's digest with one lookup.++Every object under `blobs/<sha256>` holds, whole, the bytes whose SHA-256 is that hash: the key is the hash this runner+computes from the bytes it writes, the runner's own `PUT /v1/blobs/{sha256}` writes only bytes it checked against the+path's hash, and every built-in storage provider publishes a write whole or not at all (the local one through a+temporary file renamed onto the key). So an object's stored size, read from its metadata without reading its bytes, is+the size of that hash's bytes, and nothing deletes or rewrites a blob. A listing, written once each of its files was+held at its size, therefore stays true.++From this the blob store answers without reading what it does not need:++- `POST /v1/blobs/missing` lists a hash this runner does not hold at the size the sender states, an object held at+ another size included, which only a storage that does not publish writes whole could leave short, and which the+ upload then writes over.+- A run looks up each hash's stored size first. A hash held at another size than an entry states is a 422+ `BlobMismatch`, never `MissingBlobs`, since uploading those bytes again would change nothing: the entry is wrong.+ Only a blob held at the size its entries state is read, so the bytes read are bounded by the sizes the request+ states, which its census already held to the tree's limits, and each is checked against its hash once read. Bytes+ that do not hash to their key mean the storage itself is damaged, which no caller can repair: a 500+ `BlobStoreCorrupted`.++Lookups and reads run a bounded number at a time, so one request naming many blobs never opens them all at once.+"""++from collections.abc import Iterable, Mapping+from functools import partial++from mthds.method_tree.digest import DIGEST_PREFIX, sha256_hex, tree_listing+from mthds.method_tree.file_entry import BlobStore, InMemoryBlobStore+from mthds.method_tree.models import MethodTree+from pipelex.core.method_trees.exceptions import MethodFileUnavailableError+from pipelex.core.method_trees.method_files import TreeListing+from pipelex.runtime_hub import get_storage_provider+from pipelex.tools.misc.async_utils import gather_bounded+from pipelex.tools.storage.exceptions import StorageFileNotFoundError+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract++from pipelex_api.error_types import ErrorType+from pipelex_api.errors import raise_internal_server_error++BLOBS_PREFIX = "blobs"+TREES_PREFIX = "trees"+_BLOB_CONTENT_TYPE = "application/octet-stream"+_LISTING_CONTENT_TYPE = "text/plain; charset=utf-8"+# How many lookups of a blob's size, which read metadata alone, and how many reads or writes of a blob's bytes, one+# request runs at a time.+MAX_CONCURRENT_BLOB_LOOKUPS = 16+MAX_CONCURRENT_BLOB_TRANSFERS = 4+++class RunnerBlobStore:+ """The blobs and tree listings of one caller's scope, over the deployment's storage provider."""++ def __init__(self, *, scope: str, storage: StorageProviderAbstract) -> None:+ self._scope = scope+ self._storage = storage++ @classmethod+ def make_for_scope(cls, *, scope: str) -> "RunnerBlobStore":+ """The blob store of `scope`, over the storage provider the runtime booted with."""+ return cls(scope=scope, storage=get_storage_provider())++ @property+ def scope(self) -> str:+ return self._scope++ def _blob_key(self, *, sha256: str) -> str:+ return f"{self._scope}/{BLOBS_PREFIX}/{sha256}"++ def _tree_key(self, *, tree_digest: str) -> str:+ return f"{self._scope}/{TREES_PREFIX}/{tree_digest.removeprefix(DIGEST_PREFIX)}"++ async def _holds_key(self, *, key: str) -> bool:+ try:+ await self._storage.load_size(f"{PIPELEX_STORAGE_SCHEME}{key}")+ except StorageFileNotFoundError:+ return False+ return True++ async def stored_size(self, *, sha256: str) -> int | None:+ """The size of the object stored under the hash `sha256`, read from its metadata, or None when there is none."""+ try:+ return await self._storage.load_size(f"{PIPELEX_STORAGE_SCHEME}{self._blob_key(sha256=sha256)}")+ except StorageFileNotFoundError:+ return None++ async def holds_blob(self, *, sha256: str, size: int) -> bool:+ """Whether the bytes whose SHA-256 is `sha256` are held at `size`, an object of another size being no blob."""+ return await self.stored_size(sha256=sha256) == size++ async def stored_sizes(self, *, sha256s: Iterable[str]) -> dict[str, int | None]:+ """Each hash's stored size, None for a hash not held, looked up a bounded number at a time."""+ hashes = list(sha256s)+ sizes = await gather_bounded([partial(self.stored_size, sha256=sha256) for sha256 in hashes], max_concurrency=MAX_CONCURRENT_BLOB_LOOKUPS)+ return dict(zip(hashes, sizes, strict=True))++ async def _read_verified(self, *, sha256: str, size: int) -> bytes:+ """The stored bytes of `sha256`, held at `size`, checked against their hash.++ Raises (as a problem document): a 500 `BlobStoreCorrupted` for bytes that are not `size` bytes hashing to `sha256`.+ """+ data = await self._storage.load(f"{PIPELEX_STORAGE_SCHEME}{self._blob_key(sha256=sha256)}")+ if len(data) != size or sha256_hex(data) != sha256:+ msg = (+ f"The bytes this runner's blob store holds under {sha256} do not hash to it: its storage is damaged, "+ "and its operator must remove that object for the blob to be uploaded again."+ )+ raise_internal_server_error(message=msg, error_type=ErrorType.BLOB_STORE_CORRUPTED)+ return data++ async def read_held(self, *, sizes: Mapping[str, int]) -> BlobStore:+ """Read these held blobs, each at its stored size, as the synchronous blob store a map of file entries is read with.++ Args:+ sizes: Each hash to read, with the size it is held at, which the caller bounded before asking.+ """+ blobs = await gather_bounded(+ [partial(self._read_verified, sha256=sha256, size=size) for sha256, size in sizes.items()],+ max_concurrency=MAX_CONCURRENT_BLOB_TRANSFERS,+ )+ return InMemoryBlobStore(blobs)++ async def store_blob(self, *, data: bytes) -> str:+ """Hold `data` under its SHA-256, writing it unless it is held at its size already, and return the hash.++ An object under the hash at another size, which only a storage that does not publish writes whole could leave,+ is written over.+ """+ sha256 = sha256_hex(data)+ if not await self.holds_blob(sha256=sha256, size=len(data)):+ await self._storage.store(data, key=self._blob_key(sha256=sha256), content_type=_BLOB_CONTENT_TYPE)+ return sha256++ async def holds_tree(self, *, tree_digest: str) -> bool:+ """Whether the listing of the tree `tree_digest` names is held, which means every file of the tree is."""+ return await self._holds_key(key=self._tree_key(tree_digest=tree_digest))++ async def tree_hashes(self, *, tree_digest: str) -> set[str] | None:+ """The hashes of the files of the tree `tree_digest` names, when its listing is held, else None.++ The listing is read and checked against the digest, so a sender naming a tree it does not send learns nothing+ held that its own blobs are not part of.+ """+ try:+ listing = await self._storage.load(f"{PIPELEX_STORAGE_SCHEME}{self._tree_key(tree_digest=tree_digest)}")+ except StorageFileNotFoundError:+ return None+ try:+ tree_listing_read = TreeListing.parse(tree_digest=tree_digest, listing=listing)+ except MethodFileUnavailableError:+ # Not the listing the digest names: the tree is not held.+ return None+ return set(tree_listing_read.files.values())++ async def record_tree(self, *, tree: MethodTree, contents: Mapping[str, bytes]) -> None:+ """Hold every file of a tree read in full, then its listing, so a later request can send it by hash.++ The listing is written last, once every file it names is held, so a listing present always means its whole+ tree is, and a tree whose listing is held already is left as it is, without a lookup per file.++ Args:+ tree: The tree, as a map of file entries made it.+ contents: Each file's bytes by tree path.+ """+ if await self.holds_tree(tree_digest=tree.digest):+ return+ await gather_bounded([partial(self.store_blob, data=data) for data in contents.values()], max_concurrency=MAX_CONCURRENT_BLOB_TRANSFERS)+ await self._storage.store(tree_listing(tree.files), key=self._tree_key(tree_digest=tree.digest), content_type=_LISTING_CONTENT_TYPE)++ async def missing_blobs(self, *, sizes: Mapping[str, int]) -> list[str]:+ """The hashes of `sizes` not held at their size, in the mapping's order."""+ stored = await self.stored_sizes(sha256s=sizes)+ return [sha256 for sha256, size in sizes.items() if stored[sha256] != size]diff --git a/api/pipelex_api/blob_upload.py b/api/pipelex_api/blob_upload.pynew file mode 100644index 000000000..0c9b828f1--- /dev/null+++ b/api/pipelex_api/blob_upload.py@@ -0,0 +1,102 @@+"""The signed upload targets `POST /v1/blobs/missing` names, and their check on `PUT /v1/blobs/{sha256}`.++A sender uploads a blob to the target the runner named, carrying exactly the headers it named, so the target itself+must say that the upload is allowed, as a presigned URL does on an object store. This runner signs each target it+names: its URL is its own `PUT /v1/blobs/{sha256}` with a query naming the blob store's scope, an expiry and an+HMAC-SHA256 signature over the scope, the hash and the expiry. The upload route checks the signature and the expiry,+writes under the scope the signature binds and nowhere else, and answers a missing, forged or expired signature with a+403 `BlobUploadNotAuthorized`. The upload route therefore needs no bearer token: only a caller who passed the+deployment's authentication on `POST /v1/blobs/missing` holds a target, for one hash, in its own scope, for a while.++The key is `BLOB_UPLOAD_SIGNING_KEY` when the deployment sets it, which it must when several instances serve one+deployment, since a target named by one instance is checked by whichever instance receives the upload; otherwise each+process draws a random key of its own at its first signature.++A target's URL starts with `PUBLIC_BASE_URL` when the deployment sets it, the address its clients reach the API at,+such as `https://runner.example.com`: behind a proxy that terminates TLS or serves the API under a path prefix, the URL+the runner received a request at is not the one a client can upload to. Unset, the target starts with the URL of the+request that asked for it.+"""++import hashlib+import hmac+import secrets+import time+from functools import cache+from urllib.parse import urlencode, urlsplit++from fastapi import Request+from pipelex.system.environment import get_optional_env++from pipelex_api.error_types import ErrorType+from pipelex_api.errors import raise_internal_server_error+from pipelex_api.limits import BLOB_UPLOAD_URL_TTL_SECONDS++PUBLIC_BASE_URL_ENV = "PUBLIC_BASE_URL"+_UPLOAD_ROUTE_NAME = "put_blob"++SCOPE_PARAM = "scope"+EXPIRES_PARAM = "expires"+SIGNATURE_PARAM = "signature"+++@cache+def _signing_key() -> bytes:+ configured = get_optional_env("BLOB_UPLOAD_SIGNING_KEY")+ if configured:+ return configured.encode("utf-8")+ return secrets.token_bytes(32)+++def _signature(*, scope: str, sha256: str, expires: int) -> str:+ message = f"{scope}\n{sha256}\n{expires}".encode()+ return hmac.new(_signing_key(), message, hashlib.sha256).hexdigest()+++def signed_upload_query(*, scope: str, sha256: str) -> str:+ """The query string of an upload target for one blob in one scope, valid for `BLOB_UPLOAD_URL_TTL_SECONDS`."""+ expires = int(time.time()) + BLOB_UPLOAD_URL_TTL_SECONDS+ return urlencode({SCOPE_PARAM: scope, EXPIRES_PARAM: expires, SIGNATURE_PARAM: _signature(scope=scope, sha256=sha256, expires=expires)})+++def _public_base_url() -> str | None:+ """`PUBLIC_BASE_URL` without its trailing slash, or None when unset.++ Raises (as a problem document): a 500 `ServerMisconfigured` for a value that is not an absolute http or https URL+ without a query or a fragment.+ """+ configured = get_optional_env(PUBLIC_BASE_URL_ENV)+ if not configured:+ return None+ parts = urlsplit(configured)+ if parts.scheme not in {"http", "https"} or not parts.netloc or parts.query or parts.fragment:+ msg = f"Server configuration error: {PUBLIC_BASE_URL_ENV} must be an absolute http or https URL, such as https://runner.example.com."+ raise_internal_server_error(message=msg, error_type=ErrorType.SERVER_MISCONFIGURED)+ return configured.rstrip("/")+++def upload_target_url(*, request: Request, scope: str, sha256: str) -> str:+ """The signed URL a sender uploads one blob to: this runner's `PUT /v1/blobs/{sha256}`, at its public address."""+ query = signed_upload_query(scope=scope, sha256=sha256)+ base_url = _public_base_url()+ if base_url is None:+ return f"{request.url_for(_UPLOAD_ROUTE_NAME, sha256=sha256)}?{query}"+ return f"{base_url}{request.app.url_path_for(_UPLOAD_ROUTE_NAME, sha256=sha256)}?{query}"+++def is_upload_authorized(*, scope: str | None, sha256: str, expires: str | None, signature: str | None) -> bool:+ """Whether an upload carries a signature this runner made for this hash and scope, still within its expiry.++ Every part of the query is the caller's text, read as such: an expiry that is not a decimal integer, or a signature+ holding characters no signature this runner makes holds, is no valid target, never a failure of the check.+ """+ if scope is None or expires is None or signature is None:+ return False+ if not expires.isascii() or not expires.isdecimal():+ return False+ expires_at = int(expires)+ if expires_at < int(time.time()):+ return False+ # Compared as bytes: `compare_digest` refuses text holding a character outside ASCII, which a forged signature may.+ expected = _signature(scope=scope, sha256=sha256, expires=expires_at).encode("ascii")+ return hmac.compare_digest(expected, signature.encode("utf-8", "surrogatepass"))diff --git a/api/pipelex_api/bundle.py b/api/pipelex_api/bundle.pyindex 4ca721045..d4a58a155 100644--- a/api/pipelex_api/bundle.py+++ b/api/pipelex_api/bundle.py@@ -1,104 +1,33 @@-"""Materialize a caller-supplied method bundle into a temporary library directory.+"""Decode a `bundle_b64` archive into its entries, under the guards a zip archive needs.-A run request may carry the whole method — the `.mthds` bundle plus its PipeFunc-Python (`pipe_func.py`) and a `requirements.txt` — instead of only-the inline `mthds_contents` text. Two transport forms are accepted, exactly one-per request:+A run request may carry its method as `bundle_b64`, a base64-encoded zip archive of the method's folder, beside the+map of file entries `files` that replaces it. This module only opens the archive: it bounds the encoded string before+decoding it, refuses an archive with too many members, and reads each member through a bounded stream so a zip bomb+whose header under-reports its size cannot decompress past the ceiling. The entries it returns keep the names the+archive gives them; `pipelex_api.method_tree_request` reads them as a map of file entries, so the archive's names are+judged by the one rule of the method tree, as the keys of `files` are: a name that is not a tree path, or that names a+file the membership rule keeps out of every tree, is refused, and so is one name given twice.- - `bundle_b64`: a base64-encoded zip archive of the bundle directory.- - `files`: a `{relative_path: text_content}` map (the zip's contents, unzipped).--The two are equivalent: `files` ≡ the zip's entries. This module decodes either-form, enforces the ingest guards (both transport forms are refused together; a-hard file-count and total-size ceiling; per-entry path-safety against absolute-paths and `..` traversal; a zip-bomb guard that bounds actual decompression),-writes the surviving files into a fresh temp directory, and hands that directory-back so the runner can load it via `library_dirs`.--A bundle may also ship the method packages it calls by address, each under-`.mthds/methods/<name>/` at its root with its `METHODS.toml` there, the layout the-standard names the local method cache. `partition_bundle_entries` sorts the entries-into the bundle's own `.mthds` files, its other files, and those packages, refusing-every other `.mthds` path with a `422 InvalidBundle` naming the entry, and two-packages declaring the same address with one naming both directories, and-`materialized_methods_dir` writes the packages into a temp directory of their own,-`<tmp>/<name>/…`, which the runner hands the engine as `methods_dirs`. In a sandbox-hosted-deployment the load path reads every `.py` as source text, never importing it:-it refuses a bundle whose Python declares a structure class-(`MethodStructuresRefusedError`, a 403) and captures the rest onto the crate for-the sandbox. The caller is responsible for the hosted-mode gate.--Nothing here imports or executes the bundle's Python — it only writes bytes to-disk. The caller cleans the directory up via the `materialized_bundle` context-manager's guaranteed teardown.+Nothing here writes to disk, imports or executes the archive's Python."""-from __future__ import annotations-import base64import binascii-import shutil-import tempfileimport zipfile-from contextlib import contextmanagerfrom io import BytesIO-from pathlib import Path, PurePosixPath-from typing import TYPE_CHECKING, NamedTuple, NoReturn-from mthds.package.discovery import MANIFEST_FILENAME-from mthds.package.exceptions import ManifestError-from mthds.package.manifest.parser import parse_methods_tomlfrom pipelex import logfrom pipelex.tools.log.error_fields import error_fields-from pydantic import ConfigDict-from pydantic.dataclasses import dataclassfrom pipelex_api.error_types import ErrorTypefrom pipelex_api.errors import raise_bad_request, raise_payload_too_large, raise_validation_errorfrom pipelex_api.limits import MAX_BUNDLE_FILES, MAX_BUNDLE_TOTAL_BYTES-if TYPE_CHECKING:- from collections.abc import Generator---@dataclass(frozen=True, config=ConfigDict(arbitrary_types_allowed=True))-class MaterializedBundle:- """A bundle written to disk: the directory to load and the relpaths written."""-- directory: Path- relpaths: tuple[str, ...]-- @property- def has_python_sources(self) -> bool:- """True when the bundle ships any `.py` — the trigger for the hosted-mode gate."""- return any(relpath.endswith(".py") for relpath in self.relpaths)---def _safe_relpath(name: str) -> PurePosixPath:- """Validate one bundle entry name and return it as a normalized relative POSIX path.-- Rejects anything that could escape the destination directory: absolute paths,- Windows drive/backslash forms (a bare drive prefix like `C:foo` has no slash or- backslash yet is drive-relative on Windows, so `:` is rejected outright), and any- `..` component. Directory-only entries (trailing slash) return an empty path and- are filtered by the caller.- """- if not name or name in {".", "./"}:- return PurePosixPath()- if "\\" in name:- msg = f"Bundle entry {name!r} uses backslashes; use forward-slash relative paths only"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- if ":" in name:- msg = f"Bundle entry {name!r} contains ':' (a Windows drive/stream form); use plain relative paths only"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- pure = PurePosixPath(name)- if pure.is_absolute():- msg = f"Bundle entry {name!r} is an absolute path; only relative paths are allowed"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- if any(part == ".." for part in pure.parts):- msg = f"Bundle entry {name!r} escapes the bundle root via '..'; not allowed"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- return pure+# Base64 inflates by 4/3; a zip is compressed, so a bundle whose DECOMPRESSED content is within+# the ceiling encodes to well under this. Bounding the base64 string BEFORE decoding stops a+# ~100 MiB request-body (the only other bound) from being expanded into ~75 MiB of heap just to+# be rejected later by the decompressed-size guard. Slack (+4) covers padding.+_MAX_BUNDLE_B64_CHARS = MAX_BUNDLE_TOTAL_BYTES * 4 // 3 + 4def _guard_count(count: int) -> None:@@ -113,21 +42,16 @@ def _guard_running_total(total_bytes: int) -> None:raise_payload_too_large(message=f"Bundle exceeds the {MAX_BUNDLE_TOTAL_BYTES // 1024} KiB decompressed-size limit")-# Base64 inflates by 4/3; a zip is compressed, so a bundle whose DECOMPRESSED content is within-# the ceiling encodes to well under this. Bounding the base64 string BEFORE decoding stops a-# ~100 MiB request-body (the only other bound) from being expanded into ~75 MiB of heap just to-# be rejected later by the decompressed-size guard. Slack (+4) covers padding.-_MAX_BUNDLE_B64_CHARS = MAX_BUNDLE_TOTAL_BYTES * 4 // 3 + 4-+def decode_bundle_archive(*, bundle_b64: str) -> list[tuple[str, bytes]]:+ """Decode a base64 zip and return its file members as `(name, bytes)`, in archive order, directories skipped.-def _entries_from_zip(bundle_b64: str) -> list[tuple[PurePosixPath, bytes]]:- """Decode a base64 zip and return its (safe relpath, bytes) file entries.+ Zip-bomb guard: each member is read through a bounded stream so a lying uncompressed-size header cannot force+ unbounded decompression — the running total is checked against `MAX_BUNDLE_TOTAL_BYTES` as bytes are pulled. A+ cheap length check on the still-encoded string runs FIRST, so an oversized payload is refused before it is buffered+ into memory as decoded bytes. A name the archive gives twice is returned twice, for the tree's reader to refuse.- Zip-bomb guard: each member is read through a bounded stream so a lying- uncompressed-size header cannot force unbounded decompression — the running- total is checked against `MAX_BUNDLE_TOTAL_BYTES` as bytes are pulled. A cheap- length check on the still-encoded string runs FIRST, so an oversized payload is- refused before it is buffered into memory as decoded bytes.+ Raises (as problem documents): an oversized archive (413 `PayloadTooLarge`), a value that is not base64 (400+ `InvalidBase64`), and one that is not a zip archive or holds no file (422 `InvalidBundle`)."""if len(bundle_b64) > _MAX_BUNDLE_B64_CHARS:raise_payload_too_large(message=f"bundle_b64 exceeds the {MAX_BUNDLE_TOTAL_BYTES // 1024} KiB compressed-size limit")@@ -145,16 +69,13 @@ def _entries_from_zip(bundle_b64: str) -> list[tuple[PurePosixPath, bytes]]:log.debug("A bundle is not a valid zip archive", fields=error_fields(exc=zip_error))raise_validation_error(message="bundle_b64 is not a valid zip archive", error_type=ErrorType.INVALID_BUNDLE)- entries: list[tuple[PurePosixPath, bytes]] = []+ entries: list[tuple[str, bytes]] = []total_bytes = 0with archive:members = [info for info in archive.infolist() if not info.is_dir()]_guard_count(len(members))budget = MAX_BUNDLE_TOTAL_BYTESfor info in members:- relpath = _safe_relpath(info.filename)- if not relpath.parts:- continue# Bounded read: pull at most (remaining budget + 1) bytes so a zip bomb whose header# under-reports its size still cannot decompress past the ceiling.with archive.open(info) as member:@@ -162,275 +83,5 @@ def _entries_from_zip(bundle_b64: str) -> list[tuple[PurePosixPath, bytes]]:total_bytes += len(data)_guard_running_total(total_bytes)budget = MAX_BUNDLE_TOTAL_BYTES - total_bytes- entries.append((relpath, data))- return entries---def _entries_from_files(files: dict[str, str]) -> list[tuple[PurePosixPath, bytes]]:- r"""Validate a {relpath: text} map and return its (safe relpath, bytes) entries.-- JSON can carry a lone surrogate as an escape (`"\ud800"`), which has no UTF-8 form, so an entry whose name or- content holds one is refused with a `422` naming it rather than failing later, when it is encoded or written.- """- _guard_count(len(files))- entries: list[tuple[PurePosixPath, bytes]] = []- total_bytes = 0- for name, content in files.items():- try:- name.encode("utf-8")- except UnicodeEncodeError:- msg = f"Bundle entry {name!r} has a name holding a lone surrogate, which is not valid Unicode text"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- relpath = _safe_relpath(name)- if not relpath.parts:- msg = f"Bundle entry {name!r} has no filename"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- try:- data = content.encode("utf-8")- except UnicodeEncodeError:- msg = f"Bundle entry {name!r} holds a lone surrogate, which is not valid Unicode text"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- total_bytes += len(data)- _guard_running_total(total_bytes)- entries.append((relpath, data))+ entries.append((info.filename, data))return entries---def _refuse_colliding_paths(entries: list[tuple[PurePosixPath, bytes]]) -> None:- """Refuse two entries of one path, and an entry that is also the directory of another.-- Entry names are normalized (`a/./b` and `a//b` are `a/b`) and a zip may hold one member name twice, so two entries- can name one file: a check reading the first copy, as the shipped packages' manifest check does, would pass content- that differs from the last copy, which is the one written to disk and read by the engine. A file that another entry- is placed under could not be written at all.- """- paths: set[PurePosixPath] = set()- for relpath, _ in entries:- if relpath in paths:- msg = f"Bundle has more than one entry for the path '{relpath.as_posix()}'; each file may appear only once"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- paths.add(relpath)- for relpath, _ in entries:- for parent in relpath.parents:- if parent in paths:- msg = f"Bundle entry '{parent.as_posix()}' is a file, and also the directory of the entry '{relpath.as_posix()}'"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)---class ParsedBundle(NamedTuple):- """A decoded-and-validated bundle held in memory, NOT yet written to disk.-- Splitting parse from materialize lets the caller apply the sandbox-hosted gate- (which keys on `has_python_sources`) BEFORE any disk write — so a bundle destined- for a 403 never touches the filesystem. Entries are `(safe relpath, bytes)`.- """-- entries: tuple[tuple[PurePosixPath, bytes], ...]-- @property- def has_python_sources(self) -> bool:- """True when the bundle ships any `.py` — the trigger for the hosted-mode gate."""- return any(str(relpath).endswith(".py") for relpath, _ in self.entries)---def parse_bundle(*, bundle_b64: str | None, files: dict[str, str] | None) -> ParsedBundle:- """Decode + guard a bundle into an in-memory `ParsedBundle` (no disk writes).-- Exactly one of `bundle_b64` / `files` must be supplied; supplying both is a- caller mistake (they are the same content in two forms) and is refused. All- ingest guards (base64, size, count, path-safety, zip-bomb, colliding paths) run- here, so the caller can inspect `has_python_sources` and reject BEFORE- materializing to disk.- """- if bundle_b64 is not None and files is not None:- msg = "Provide either bundle_b64 or files, not both"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- if bundle_b64 is not None:- entries = _entries_from_zip(bundle_b64)- elif files is not None:- entries = _entries_from_files(files)- else:- msg = "No bundle supplied (bundle_b64 and files are both absent)"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- _refuse_colliding_paths(entries)- return ParsedBundle(entries=tuple(entries))---# Where a bundle ships the method packages it calls: `.mthds/methods/<name>/`, at the bundle's root and nowhere else.-_MTHDS_DIR_NAME = ".mthds"-_METHODS_DIR_NAME = "methods"-_VENDORING_LAYOUT = (- "A method package shipped with the bundle goes under '.mthds/methods/<name>/' at the bundle's root, "- "with its manifest at '.mthds/methods/<name>/METHODS.toml'."-)---class VendoredMethodPackage(NamedTuple):- """A method package a bundle ships under `.mthds/methods/<name>/`, held in memory."""-- name: str- """The package's directory name under `.mthds/methods/`."""-- full_address: str- """The address its manifest declares, `address/name`, as a reference names it and a refusal names the package."""-- entries: tuple[tuple[PurePosixPath, bytes], ...]- """Every file of the package, by its path relative to the package directory."""---class BundlePartition(NamedTuple):- """A bundle's entries sorted into its own `.mthds` files, its other files, and the packages it ships."""-- mthds_entries: tuple[tuple[PurePosixPath, bytes], ...]- library_entries: tuple[tuple[PurePosixPath, bytes], ...]- vendored_packages: tuple[VendoredMethodPackage, ...]---def _refuse_entry(*, owner: str, relpath: PurePosixPath, reason: str) -> NoReturn:- msg = f"{owner} entry '{relpath.as_posix()}' {reason}. {_VENDORING_LAYOUT}"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)---def _vendored_package(*, owner: str, name: str, entries: list[tuple[PurePosixPath, bytes]]) -> VendoredMethodPackage:- """Check one shipped package's directory and read the address its manifest declares."""- package_label = f"'.mthds/methods/{name}/'"- if name.startswith("."):- msg = f"{owner} ships {package_label}, a hidden directory, which is never searched for a package. {_VENDORING_LAYOUT}"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- manifest_bytes = next((data for relpath, data in entries if relpath == PurePosixPath(MANIFEST_FILENAME)), None)- if manifest_bytes is None:- msg = f"{owner} ships {package_label} without a {MANIFEST_FILENAME}, so no reference could find it. {_VENDORING_LAYOUT}"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- manifest_path = f".mthds/methods/{name}/{MANIFEST_FILENAME}"- try:- manifest = parse_methods_toml(manifest_bytes.decode("utf-8"))- except UnicodeDecodeError:- msg = f"{owner} entry '{manifest_path}' is not valid UTF-8. {_VENDORING_LAYOUT}"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- except ManifestError as exc:- msg = f"{owner} entry '{manifest_path}' is not a valid method package manifest: {exc.message} {_VENDORING_LAYOUT}"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- # The name a reference matches, as the engine's discovery gives it: the manifest's, else the directory's.- package_name = manifest.name if manifest.name is not None else name- return VendoredMethodPackage(name=name, full_address=f"{manifest.address}/{package_name}", entries=tuple(entries))---def _refuse_shared_identities(*, owner: str, packages: tuple[VendoredMethodPackage, ...]) -> None:- """Refuse shipped packages a reference could not tell apart, rather than let the directory order pick one.-- A reference is matched against a package's full address without regard to case, as the engine's lookup compares it,- so two packages whose full addresses differ only in case are one identity too.- """- by_identity: dict[str, list[VendoredMethodPackage]] = {}- for package in packages:- by_identity.setdefault(package.full_address.casefold(), []).append(package)- for sharing in by_identity.values():- if len(sharing) < 2:- continue- directories = ", ".join(f"'.mthds/methods/{package.name}/'" for package in sharing)- msg = (- f"{owner} ships several method packages declaring the address '{sharing[0].full_address}': {directories}. "- "A reference matches a package by its address without regard to case, so it could not tell them apart: "- "ship one package per address."- )- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)---def partition_bundle_entries(entries: tuple[tuple[PurePosixPath, bytes], ...], *, owner: str = "Bundle") -> BundlePartition:- """Sort a bundle's entries into its own `.mthds` files, its other files, and the method packages it ships.-- An entry under `.mthds/methods/<name>/`, whatever its extension, belongs to the shipped package `<name>`, which- leaves the bundle's own content and its library directory. Every other `.mthds` path component is refused with a- `422 InvalidBundle` naming the entry rather than dropped, since nothing could ever find what it holds: a file- elsewhere under `.mthds/`, a `.mthds/methods/` store below the root, a file directly under `.mthds/methods/`, and a- `.mthds/` inside a shipped package, whose own dependencies are not loaded. A shipped package must be a directory- whose name does not start with a dot and whose `METHODS.toml` parses, and the address that manifest declares is read- here, to name the package in a refusal. Two shipped packages declaring the same full address, compared without regard- to case as a reference is matched, are refused too, naming the address and both directories, since nothing but their- directory order would decide which one answers. Counting the bundle's own `.mthds` files is the caller's: a bundle- whose only `.mthds` files are shipped ones has no content of its own.-- Args:- entries: The bundle's `(safe relpath, bytes)` entries.- owner: How a refusal names the bundle, for a method package fetched by address as for a request's bundle.-- Returns:- The three groups, each in the bundle's entry order.- """- mthds_entries: list[tuple[PurePosixPath, bytes]] = []- library_entries: list[tuple[PurePosixPath, bytes]] = []- vendored_entries: dict[str, list[tuple[PurePosixPath, bytes]]] = {}- for relpath, data in entries:- parts = relpath.parts- if parts[0] == _MTHDS_DIR_NAME:- if len(parts) < 4 or parts[1] != _METHODS_DIR_NAME:- _refuse_entry(owner=owner, relpath=relpath, reason="is under '.mthds/' but not inside a shipped method package")- package_relpath = PurePosixPath(*parts[3:])- if _MTHDS_DIR_NAME in package_relpath.parts:- _refuse_entry(- owner=owner,- relpath=relpath,- reason="is a '.mthds/' directory inside a shipped method package, whose own dependencies are not loaded",- )- vendored_entries.setdefault(parts[2], []).append((package_relpath, data))- elif _MTHDS_DIR_NAME in parts:- _refuse_entry(owner=owner, relpath=relpath, reason="is under a '.mthds/' directory below the bundle's root, which is never searched")- elif relpath.suffix == ".mthds":- mthds_entries.append((relpath, data))- else:- library_entries.append((relpath, data))- vendored_packages = tuple(- _vendored_package(owner=owner, name=name, entries=package_entries) for name, package_entries in vendored_entries.items()- )- _refuse_shared_identities(owner=owner, packages=vendored_packages)- return BundlePartition(mthds_entries=tuple(mthds_entries), library_entries=tuple(library_entries), vendored_packages=vendored_packages)---@contextmanager-def materialize_parsed(parsed: ParsedBundle, *, prefix: str = "pipelex-bundle-") -> Generator[MaterializedBundle, None, None]:- """Write an already-parsed bundle into a fresh temp directory, cleaned up on exit.-- The yielded `MaterializedBundle.directory` is safe to pass as a `library_dirs`- entry; it is removed when the context exits, on both the happy and error path.- """- directory = Path(tempfile.mkdtemp(prefix=prefix))- try:- root = directory.resolve()- relpaths: list[str] = []- for relpath, data in parsed.entries:- target = (directory / relpath).resolve()- # Defense-in-depth: even after per-part validation, confirm the resolved target stays- # under the temp root before writing (guards against symlink/edge normalization surprises).- if root != target and root not in target.parents:- msg = f"Bundle entry {str(relpath)!r} resolves outside the bundle root"- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- target.parent.mkdir(parents=True, exist_ok=True)- target.write_bytes(data)- relpaths.append(relpath.as_posix())- yield MaterializedBundle(directory=directory, relpaths=tuple(relpaths))- finally:- shutil.rmtree(directory, ignore_errors=True)---@contextmanager-def materialized_methods_dir(packages: tuple[VendoredMethodPackage, ...]) -> Generator[Path, None, None]:- """Write shipped method packages into a temp directory of their own, as `<tmp>/<name>/…`, cleaned up on exit.-- The directory is laid out like `.mthds/methods/`, one directory per package, which is what the engine's- `methods_dirs` take, and its path holds no `.mthds/methods` a walk-up from a bundle could take for a store.- """- entries = tuple((PurePosixPath(package.name) / relpath, data) for package in packages for relpath, data in package.entries)- with materialize_parsed(ParsedBundle(entries=entries), prefix="pipelex-methods-") as materialized:- yield materialized.directory---@contextmanager-def materialized_bundle(*, bundle_b64: str | None, files: dict[str, str] | None) -> Generator[MaterializedBundle, None, None]:- """Parse AND materialize a bundle in one step (convenience for callers that don't gate).-- Equivalent to `parse_bundle(...)` followed by `materialize_parsed(...)`; a caller- that must apply the sandbox-hosted gate before disk writes should use the two- steps directly instead.- """- with materialize_parsed(parse_bundle(bundle_b64=bundle_b64, files=files)) as bundle:- yield bundlediff --git a/api/pipelex_api/bundle_resources.py b/api/pipelex_api/bundle_resources.pyindex 2c1c92e6e..00282f24c 100644--- a/api/pipelex_api/bundle_resources.py+++ b/api/pipelex_api/bundle_resources.py@@ -1,37 +1,30 @@-"""Decode the files a request submits with its bundle that its steps read, such as a template file.+"""Decode the files a request sends in `resources`, the files its bundle's steps read, such as a template file.-They come in two ways. `resources` is a JSON object mapping each file's bundle-relative path, the exact path a step-names (a `PipeDocGen` step's `template_file`), to the file's bytes in standard base64, beside `mthds_contents` or a-method bundle. A method bundle (`bundle_b64` or `files`) may also ship the file beside its `.mthds` file, which-`files_its_steps_read` picks out. Every bundle of the request is read as sitting in one directory, so one path holds-one file. The decoded files go to the runtime as its `submitted_files`, keyed by canonical path, which refuses a file-none of the bundle's steps names once the bundle is parsed, and holds the rest as the files of the tree the bundle-sits at the root of: the run's crate names that tree by its digest and carries none of their bytes.+`resources` is a JSON object mapping each file's bundle-relative path, the exact path a step names (a `PipeDocGen`+step's `template_file`), to the file's bytes in standard base64. Beside `mthds_contents`, every bundle of the request is+read as sitting in one directory, so one path holds one file, and the decoded files go to the runtime as its+`submitted_files`, keyed by canonical path, which refuses a file none of the bundle's steps names once the bundle is+parsed, and holds the rest as the files of the tree the bundle sits at the root of. Beside a method tree (`files` or+`bundle_b64`), the decoded files join the tree at their paths from its root (`pipelex_api.method_tree_request`), where+the tree's rule judges them as it judges the tree's own files.Refusals, in the order they are checked:- more `resources` than `MAX_RESOURCES_PER_REQUEST` → 413 `PayloadTooLarge`;- a path that cannot name a file inside the bundle (absolute, a `..` segment, a backslash, a colon, empty, or asecond spelling of a path already submitted) → 422 `InvalidResourcePath`, before any value is decoded;-- more files than `MAX_RESOURCES_PER_REQUEST`, counting those the method bundle ships → 413 `PayloadTooLarge`;- a file whose encoded length already exceeds the per-file ceiling → 413 `PayloadTooLarge`, before decoding;- a value that is not valid base64 → 400 `InvalidBase64`, as `bundle_b64` is;-- two different files under one path, sent in `resources` and shipped in the bundle, or shipped beside two of its- `.mthds` files → 422 `InvalidResourcePath`;-- a decoded or shipped file over `MAX_RESOURCE_FILE_BYTES`, or a running total over `MAX_RESOURCES_TOTAL_BYTES` → 413.+- a decoded file over `MAX_RESOURCE_FILE_BYTES`, or a running total over `MAX_RESOURCES_TOTAL_BYTES` → 413."""import base64import binascii-from collections.abc import Iterable, Mapping-from pathlib import PurePosixPathfrom typing import NoReturnfrom pipelex import log-from pipelex.core.bundle_resources.bundle_resource import canonical_resource_path, canonical_resource_path_or_none+from pipelex.core.bundle_resources.bundle_resource import canonical_resource_pathfrom pipelex.core.bundle_resources.exceptions import BundleResourcePathError-from pipelex.mthds_parsing.exceptions import MthdsParserError-from pipelex.mthds_parsing.parser import MthdsParserfrom pipelex.tools.log.error_fields import error_fieldsfrom pipelex_api.error_types import ErrorType@@ -43,65 +36,24 @@ from pipelex_api.limits import MAX_RESOURCE_FILE_BYTES, MAX_RESOURCES_PER_REQUES_MAX_RESOURCE_B64_CHARS = MAX_RESOURCE_FILE_BYTES * 4 // 3 + 4-def _refuse_two_files_under_one_path(*, path: str, where: str) -> NoReturn:+def refuse_two_files_under_one_path(*, path: str, where: str) -> NoReturn:+ """Refuse two different files sent under one path, naming the path, with a 422 `InvalidResourcePath`."""msg = (- f"Two different files are named '{path}', {where}. Every bundle of a request reads its files as if it sat in one "- "directory, so a path names one file: send one of them, or rename one and the step that reads it."+ f"Two different files are named '{path}', {where}. A path names one file of a method: send one of them, or rename "+ "one and the step that reads it.")raise_validation_error(message=msg, error_type=ErrorType.INVALID_RESOURCE_PATH)-def files_its_steps_read(- *, mthds_entries: Iterable[tuple[PurePosixPath, str]], other_entries: Iterable[tuple[PurePosixPath, bytes]]-) -> dict[str, bytes]:- """The files a method bundle ships beside its `.mthds` files that their steps read, keyed by the path a step names.-- Each `.mthds` file is parsed for the files its steps name, such as a `PipeDocGen` step's `template_file`, which is- looked up relative to that file's own directory in the bundle. The bundle's `.mthds` files load as content strings,- so these files travel with them as the `resources` a request sends do, wherever the run goes. A file a step names- that the bundle does not ship is left out, for `resources` to supply or the load to refuse saying so, and a `.mthds`- file that does not parse is skipped, for the load to give its verdict on.-- Raises (as a 422 `InvalidResourcePath`): two steps naming one path, beside two `.mthds` files that ship different files.- """- shipped = dict(other_entries)- files: dict[str, bytes] = {}- for mthds_relpath, mthds_content in mthds_entries:- try:- blueprint = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=mthds_content)- except MthdsParserError as exc:- log.debug("A method bundle's file did not parse, so the files its steps read are left to its load", fields=error_fields(exc=exc))- continue- for pipe_blueprint in (blueprint.pipe or {}).values():- for named_path in pipe_blueprint.named_file_references:- canonical_path = canonical_resource_path_or_none(path=named_path)- if canonical_path is None:- continue- data = shipped.get(mthds_relpath.parent / canonical_path)- if data is None:- continue- known = files.get(canonical_path)- if known is not None and known != data:- _refuse_two_files_under_one_path(path=canonical_path, where="shipped beside two of the bundle's .mthds files")- files[canonical_path] = data- return files---def decode_bundle_resources(*, resources: dict[str, str] | None, bundle_files: Mapping[str, bytes] | None = None) -> dict[str, bytes] | None:- """The files the bundle's steps read, decoded and keyed by canonical path, or None when there are none.-- `resources` are the ones the request sends, in base64. `bundle_files` are the ones its method bundle ships, as- `files_its_steps_read` picks them out. Both are held to the same limits, together, and a file in both must be the same.- """- if not resources and not bundle_files:+def decode_bundle_resources(*, resources: dict[str, str] | None) -> dict[str, bytes] | None:+ """The files `resources` sends, decoded and keyed by canonical path, or None when there are none."""+ if not resources:return None- sent = resources or {}- shipped = bundle_files or {}- if len(sent) > MAX_RESOURCES_PER_REQUEST:- raise_payload_too_large(message=f"resources exceeds the {MAX_RESOURCES_PER_REQUEST}-file limit (got {len(sent)})")+ if len(resources) > MAX_RESOURCES_PER_REQUEST:+ raise_payload_too_large(message=f"resources exceeds the {MAX_RESOURCES_PER_REQUEST}-file limit (got {len(resources)})")# Every path first, so a second spelling of one file is refused as such before any value is decoded.sent_by_path: dict[str, tuple[str, str]] = {}- for path, encoded in sent.items():+ for path, encoded in resources.items():try:canonical_path = canonical_resource_path(path=path)except BundleResourcePathError as exc:@@ -110,12 +62,6 @@ def decode_bundle_resources(*, resources: dict[str, str] | None, bundle_files: Mmsg = f"Two files submitted with the bundle name the same file '{canonical_path}'."raise_validation_error(message=msg, error_type=ErrorType.INVALID_RESOURCE_PATH)sent_by_path[canonical_path] = (path, encoded)- file_count = len(sent_by_path.keys() | shipped.keys())- if file_count > MAX_RESOURCES_PER_REQUEST:- msg = (- f"The files the bundle's steps read, in resources and in the bundle, exceed the {MAX_RESOURCES_PER_REQUEST}-file limit (got {file_count})"- )- raise_payload_too_large(message=msg)files: dict[str, bytes] = {}total_bytes = 0for canonical_path, (path, encoded) in sent_by_path.items():@@ -127,13 +73,6 @@ def decode_bundle_resources(*, resources: dict[str, str] | None, bundle_files: M# DEBUG: the refusal below is a caller's mistake, which the error handler already logs; this keeps the cause.log.debug("A submitted file's base64 could not be decoded", fields=error_fields(exc=decode_error))raise_bad_request(message=f"The file '{path}' in resources is not valid base64", error_type=ErrorType.INVALID_BASE64)- shipped_data = shipped.get(canonical_path)- if shipped_data is not None and shipped_data != data:- _refuse_two_files_under_one_path(path=canonical_path, where="one sent in resources and one shipped in the bundle")- files[canonical_path] = data- for canonical_path, data in shipped.items():- files.setdefault(canonical_path, data)- for canonical_path, data in files.items():if len(data) > MAX_RESOURCE_FILE_BYTES:raise_payload_too_large(message=f"The file '{canonical_path}' a step reads exceeds the {MAX_RESOURCE_FILE_BYTES // 1024} KiB per-file limit"@@ -141,4 +80,5 @@ def decode_bundle_resources(*, resources: dict[str, str] | None, bundle_files: Mtotal_bytes += len(data)if total_bytes > MAX_RESOURCES_TOTAL_BYTES:raise_payload_too_large(message=f"The files the bundle's steps read exceed the {MAX_RESOURCES_TOTAL_BYTES // 1024} KiB total-size limit")+ files[canonical_path] = datareturn filesdiff --git a/api/pipelex_api/error_types.py b/api/pipelex_api/error_types.pyindex 27048d71c..7149dc4e6 100644--- a/api/pipelex_api/error_types.py+++ b/api/pipelex_api/error_types.py@@ -58,9 +58,34 @@ class ErrorType(StrEnum):INVALID_MODEL_REFERENCE = "InvalidModelReference"INVALID_BASE64 = "InvalidBase64"PAYLOAD_TOO_LARGE = "PayloadTooLarge"- # A run request carried a malformed method bundle (`bundle_b64` / `files`): a corrupt zip, an- # unsafe entry name (absolute path or `..` traversal), or both transport forms supplied at once.+ # A run request carried a malformed method bundle (`bundle_b64` / `files`): a corrupt zip, a+ # bundle with no `.mthds` file, or both transport forms supplied at once.INVALID_BUNDLE = "InvalidBundle"+ # A request's map of file entries (`files`, a tree of `dependencies`, a `bundle_b64` archive read as+ # one) is not a method tree the MTHDS standard admits: a key given twice, a key that is not a tree+ # path or names a file the membership rule keeps out of every tree, keys that collide, an empty map,+ # or a malformed entry. A 422 whose `tree_refusal` member carries the standard's refusal code and+ # the paths it names.+ INVALID_METHOD_FILES = "InvalidMethodFiles"+ # A request names, by hash, bytes this runner's blob store does not hold at the stated size. A 422+ # whose `missing_blobs` member lists every such hash at once, so a sender learns in one answer what to+ # upload (`POST /v1/blobs/missing`, then `PUT /v1/blobs/{sha256}`).+ MISSING_BLOBS = "MissingBlobs"+ # A method tree, or a blob, passes one of this runner's limits on a tree's file count, its total+ # size or one file's size (`MAX_METHOD_TREE_FILES`, `MAX_METHOD_TREE_TOTAL_KIB`,+ # `MAX_METHOD_TREE_FILE_KIB`). A 422 naming the limit, refused before any blob is read or uploaded.+ METHOD_TREE_TOO_LARGE = "MethodTreeTooLarge"+ # `PUT /v1/blobs/{sha256}` was sent bytes whose SHA-256 is not the one its path names,+ # `POST /v1/blobs/missing` named one hash twice with different sizes, or a request's hash entry states a size other+ # than the one this runner holds that hash's bytes at, which uploading them again would not change. A 422 whose+ # `mismatched_blobs` member, on a request's entries, lists each such hash with the size held.+ BLOB_MISMATCH = "BlobMismatch"+ # The bytes this runner's blob store holds under a hash do not hash to it: the storage is damaged. A 500 no caller+ # can repair; the operator removes the object, and the blob is then uploaded again.+ BLOB_STORE_CORRUPTED = "BlobStoreCorrupted"+ # `PUT /v1/blobs/{sha256}` was called without the signed query `POST /v1/blobs/missing` names in its upload+ # target, with one whose signature does not hold, or past its expiry. A 403: the upload is not stored.+ BLOB_UPLOAD_NOT_AUTHORIZED = "BlobUploadNotAuthorized"# A request's `resources` map keys a submitted file by a path that cannot name a file inside the# bundle: absolute, with a `..` segment, a backslash or a colon, empty, or a second spelling of a# path already submitted. A 422: the key is the path a step names, relative to the bundle.diff --git a/api/pipelex_api/errors.py b/api/pipelex_api/errors.pyindex e50238bdc..b1bf7fb9d 100644--- a/api/pipelex_api/errors.py+++ b/api/pipelex_api/errors.py@@ -15,6 +15,7 @@ are the single translation point for everything the API does not authoritself."""+from collections.abc import Mappingfrom typing import Any, NoReturnfrom pipelex.base_exceptions import ErrorDomain@@ -49,6 +50,7 @@ def _raise_api_error(status: int,error_domain: ErrorDomain,headers: dict[str, str] | None = None,+ members: Mapping[str, Any] | None = None,) -> NoReturn:"""Build the RFC 7807 document and raise `ApiError`.@@ -67,6 +69,9 @@ def _raise_api_error(request_id=None,error_domain=error_domain,)+ if members:+ # Extension members of RFC 7807 (section 3.2), such as `missing_blobs`: data a client acts on, beside the prose.+ document.update(members)raise ApiError(status_code=status, document=document, headers=headers)@@ -75,6 +80,15 @@ def raise_validation_error(message: str, error_type: ErrorType = ErrorType.VALID_raise_api_error(error_type=error_type, message=message, status=422, error_domain=ErrorDomain.INPUT)+def raise_validation_problem(*, message: str, error_type: ErrorType, members: Mapping[str, Any]) -> NoReturn:+ """Raise a 422 RFC 7807 problem response for invalid caller input, carrying extension members a client acts on.++ The members join the document beside `detail`, such as the `missing_blobs` a sender uploads before it sends the+ request again, or the `tree_refusal` naming the standard's refusal code and the paths it names.+ """+ _raise_api_error(error_type=error_type, message=message, status=422, error_domain=ErrorDomain.INPUT, members=members)++def raise_bad_request(message: str, error_type: ErrorType = ErrorType.BAD_REQUEST) -> NoReturn:"""Raise a 400 RFC 7807 problem response for a malformed request."""_raise_api_error(error_type=error_type, message=message, status=400, error_domain=ErrorDomain.INPUT)diff --git a/api/pipelex_api/limits.py b/api/pipelex_api/limits.pyindex dd287e60e..9acf0e0f7 100644--- a/api/pipelex_api/limits.py+++ b/api/pipelex_api/limits.py@@ -23,11 +23,20 @@ MAX_METHOD_REF_LEN = 512 # `method_ref` selector strings; a fixed schema bound,MAX_JSON_NESTING_DEPTH = 128DEFAULT_MAX_CALLBACK_URLS = 5DEFAULT_MAX_CALLBACK_URL_LEN = 2048-DEFAULT_MAX_BUNDLE_FILES = 128 # entries in a materialized method bundle (.mthds + .py + requirements.txt)+DEFAULT_MAX_BUNDLE_FILES = 128 # members of a `bundle_b64` archive, its shipped packages' files includedDEFAULT_MAX_BUNDLE_TOTAL_KIB = 8 * 1024 # 8 MiB decompressed across the whole bundle (zip-bomb guard)DEFAULT_MAX_RESOURCES_PER_REQUEST = 32 # files submitted with a bundle that its steps read (`resources`)DEFAULT_MAX_RESOURCE_FILE_KIB = 5 * 1024 # 5 MiB decoded per submitted file, such as an office templateDEFAULT_MAX_RESOURCES_TOTAL_KIB = 10 * 1024 # 10 MiB decoded across a request's submitted files+# A method tree a request sends (`files`, each tree of `dependencies`, a `bundle_b64` archive) or fetches by `method_ref`,+# whatever form its files travel in: inline, Base64 or by hash. The request body bounds only the inline entries.+DEFAULT_MAX_METHOD_TREE_FILES = 1000+DEFAULT_MAX_METHOD_TREE_TOTAL_KIB = 64 * 1024 # 64 MiB across the files of one tree+DEFAULT_MAX_METHOD_TREE_FILE_KIB = 32 * 1024 # 32 MiB for one file of a tree, and for one blob uploaded+# Every tree one request carries together, the method's own and its dependency trees, beside each tree's own limits.+DEFAULT_MAX_METHOD_DEPENDENCIES = 32 # dependency trees beside the method's own+DEFAULT_MAX_METHOD_REQUEST_TOTAL_KIB = 128 * 1024 # 128 MiB across the method's tree and its dependency trees+DEFAULT_BLOB_UPLOAD_URL_TTL_SECONDS = 3600 # how long an upload target `POST /v1/blobs/missing` names stays validDEFAULT_MAX_METHOD_CACHE_CLONES = 64 # cached method-package clones (one per resolved commit SHA)DEFAULT_MAX_METHOD_CACHE_TOTAL_KIB = 512 * 1024 # 512 MiB across all cached clonesDEFAULT_MAX_METHOD_CACHE_AGE_HOURS = 24 # a cached clone unused for this long is evicted@@ -50,6 +59,15 @@ MAX_RESOURCES_PER_REQUEST = get_positive_int_env(env_var="MAX_RESOURCES_PER_REQUMAX_RESOURCE_FILE_BYTES = get_positive_int_env(env_var="MAX_RESOURCE_FILE_KIB", default=DEFAULT_MAX_RESOURCE_FILE_KIB) * 1024MAX_RESOURCES_TOTAL_BYTES = get_positive_int_env(env_var="MAX_RESOURCES_TOTAL_KIB", default=DEFAULT_MAX_RESOURCES_TOTAL_KIB) * 1024+MAX_METHOD_TREE_FILES = get_positive_int_env(env_var="MAX_METHOD_TREE_FILES", default=DEFAULT_MAX_METHOD_TREE_FILES)+MAX_METHOD_TREE_TOTAL_BYTES = get_positive_int_env(env_var="MAX_METHOD_TREE_TOTAL_KIB", default=DEFAULT_MAX_METHOD_TREE_TOTAL_KIB) * 1024+MAX_METHOD_TREE_FILE_BYTES = get_positive_int_env(env_var="MAX_METHOD_TREE_FILE_KIB", default=DEFAULT_MAX_METHOD_TREE_FILE_KIB) * 1024++MAX_METHOD_DEPENDENCIES = get_positive_int_env(env_var="MAX_METHOD_DEPENDENCIES", default=DEFAULT_MAX_METHOD_DEPENDENCIES)+MAX_METHOD_REQUEST_TOTAL_BYTES = get_positive_int_env(env_var="MAX_METHOD_REQUEST_TOTAL_KIB", default=DEFAULT_MAX_METHOD_REQUEST_TOTAL_KIB) * 1024++BLOB_UPLOAD_URL_TTL_SECONDS = get_positive_int_env(env_var="BLOB_UPLOAD_URL_TTL_SECONDS", default=DEFAULT_BLOB_UPLOAD_URL_TTL_SECONDS)+MAX_METHOD_CACHE_CLONES = get_positive_int_env(env_var="MAX_METHOD_CACHE_CLONES", default=DEFAULT_MAX_METHOD_CACHE_CLONES)MAX_METHOD_CACHE_TOTAL_BYTES = get_positive_int_env(env_var="MAX_METHOD_CACHE_TOTAL_KIB", default=DEFAULT_MAX_METHOD_CACHE_TOTAL_KIB) * 1024MAX_METHOD_CACHE_AGE_SECONDS = get_positive_int_env(env_var="MAX_METHOD_CACHE_AGE_HOURS", default=DEFAULT_MAX_METHOD_CACHE_AGE_HOURS) * 3600diff --git a/api/pipelex_api/main.py b/api/pipelex_api/main.pyindex 962529c9f..c08d4de45 100644--- a/api/pipelex_api/main.py+++ b/api/pipelex_api/main.py@@ -31,6 +31,7 @@ from pipelex_api.exception_handlers import register_exception_handlersfrom pipelex_api.middleware import RequestIdMiddleware, request_body_size_middlewarefrom pipelex_api.openapi_schema import PipelexFastAPIfrom pipelex_api.routes import router as api_router+from pipelex_api.routes.blob_upload import router as blob_upload_routerfrom pipelex_api.routes.health import router as health_routerfrom pipelex_api.routes.version import router as version_routerfrom pipelex_api.security import get_auth_dependency@@ -187,6 +188,11 @@ fastapi_app.include_router(health_router)# detection before they have credentials).fastapi_app.include_router(version_router, prefix="/v1")+# `PUT /v1/blobs/{sha256}` is authorized by the signed query of the upload target `POST /v1/blobs/missing` names+# (`pipelex_api.blob_upload`), so a sender uploads with exactly the headers that target names: it mounts without the+# auth dependency, and only a caller who passed it on `POST /v1/blobs/missing` holds a target.+fastapi_app.include_router(blob_upload_router, prefix="/v1")+# Register all other routes WITH authentication, the dependency selected by the AUTH_MODE env var. An# unrecognized AUTH_MODE raises `InvalidAuthModeError` here and the app fails to boot, the same fail-fast# posture as ERROR_DISCLOSURE above: a mistyped mode must never serve these routes unauthenticated.diff --git a/api/pipelex_api/method_cache.py b/api/pipelex_api/method_cache.pyindex 39e21fe01..beb6af907 100644--- a/api/pipelex_api/method_cache.py+++ b/api/pipelex_api/method_cache.py@@ -17,9 +17,13 @@ deployment-mode change (or a ceiling change) applies immediately.Concurrency: clones land in a per-request staging directory and are installed with an atomic`rename`, so two concurrent requests for the same SHA cannot corrupt each other — the loser-of the rename race discards its staging copy and uses the winner's. The resolution seam-(`pipelex_api.method_source`) copies everything it needs out of the cached clone before the run-starts, so eviction can never pull a directory out from under a running pipeline.+of the rename race discards its staging copy and uses the winner's. Fetches run concurrently,+each on a thread of its own, so a request reads its package from the clone while another+request's miss may be evicting: `pinned_package` pins the clone for as long as its caller reads+it, and eviction skips a pinned clone. The resolution seam (`pipelex_api.method_source`) reads+everything it needs out of the clone under that pin, before the run starts, so eviction never+pulls a directory out from under a read or a running pipeline. The pins are this process's: a+deployment running several worker processes gives each its own `METHOD_CACHE_DIR`."""import os@@ -29,6 +33,8 @@ import tempfileimport threadingimport timeimport uuid+from collections.abc import Generator+from contextlib import contextmanagerfrom functools import cachefrom pathlib import Path@@ -138,9 +144,39 @@ class MethodCloneCache:def __init__(self, *, root_dir: Path) -> None:self._root = root_dirself._lock = threading.Lock()+ # How many callers read each clone, by its directory's name; eviction skips a clone named here.+ self._pins: dict[str, int] = {}- def get_or_fetch(self, *, ref: MethodRef, clone_url: str | None = None) -> FetchedMethodPackage:- """Return the package a reference points at, cloning only when its SHA is not cached.+ @contextmanager+ def pinned_package(self, *, ref: MethodRef, clone_url: str | None = None) -> Generator[FetchedMethodPackage, None, None]:+ """The package a reference points at, its clone kept from eviction until the caller leaves the block.++ Args:+ ref: The parsed method reference.+ clone_url: Override for the derived clone URL (tests, non-default remotes).++ Raises:+ The failures of `_fetch_and_pin`.+ """+ package = self._fetch_and_pin(ref=ref, clone_url=clone_url)+ try:+ yield package+ finally:+ with self._lock:+ self._unpin(clone_name=package.clone_dir.name)++ def _pin(self, *, clone_name: str) -> None:+ self._pins[clone_name] = self._pins.get(clone_name, 0) + 1++ def _unpin(self, *, clone_name: str) -> None:+ remaining = self._pins.get(clone_name, 0) - 1+ if remaining > 0:+ self._pins[clone_name] = remaining+ else:+ self._pins.pop(clone_name, None)++ def _fetch_and_pin(self, *, ref: MethodRef, clone_url: str | None) -> FetchedMethodPackage:+ """Return the package a reference points at, its clone pinned, cloning only when its SHA is not cached.Always re-runs package location and the bounds check against the (possibly cached)clone — only the clone itself is cached, never a verdict about it.@@ -164,7 +200,9 @@ class MethodCloneCache:with self._lock:if clone_dir.is_dir():os.utime(clone_dir) # LRU bookkeeping: mtime is the eviction order- return self._package_from_clone(ref=ref, clone_dir=clone_dir, commit_sha=commit_sha)+ package = self._package_from_clone(ref=ref, clone_dir=clone_dir, commit_sha=commit_sha)+ self._pin(clone_name=clone_dir.name)+ return package# Miss: clone into a per-request staging dir (outside the lock — clones are slow),# then install atomically. A tag repointed between ls-remote and clone resolves to the@@ -183,8 +221,13 @@ class MethodCloneCache:# A concurrent request installed the same SHA between the check and the# rename; its copy is identical (same commit), so use it.log.debug(f"A concurrent request installed method clone {fetched.commit_sha} first, and its copy is used")- self._evict()- return self._package_from_clone(ref=ref, clone_dir=target_dir, commit_sha=fetched.commit_sha)+ os.utime(target_dir) # the clone just installed, or used, is the newest entry+ try:+ package = self._package_from_clone(ref=ref, clone_dir=target_dir, commit_sha=fetched.commit_sha)+ self._pin(clone_name=target_dir.name)+ finally:+ self._evict()+ return packagefinally:shutil.rmtree(staging_dir, ignore_errors=True)@@ -204,7 +247,8 @@ class MethodCloneCache:def _evict(self) -> None:"""Enforce the age, count, and byte bounds — oldest-first, keeping the newest entry.- Called with the lock held, right after an install. Best-effort by design: a clone+ Called with the lock held, right after an install. A clone a caller is reading, pinned, is+ never evicted, though it still counts toward the bounds. Best-effort by design: a clonedirectory another process already removed is simply skipped."""entries = sorted(@@ -217,13 +261,16 @@ class MethodCloneCache:now = time.time()survivors: list[Path] = []for entry in entries:- if entry != newest and now - entry.stat().st_mtime > MAX_METHOD_CACHE_AGE_SECONDS:+ if entry != newest and entry.name not in self._pins and now - entry.stat().st_mtime > MAX_METHOD_CACHE_AGE_SECONDS:shutil.rmtree(entry, ignore_errors=True)else:survivors.append(entry)+ clone_count = len(survivors)total_bytes = sum(_directory_size_bytes(entry) for entry in survivors)- while len(survivors) > 1 and (len(survivors) > MAX_METHOD_CACHE_CLONES or total_bytes > MAX_METHOD_CACHE_TOTAL_BYTES):- oldest = survivors.pop(0)+ evictable = [entry for entry in survivors if entry != newest and entry.name not in self._pins]+ while evictable and (clone_count > MAX_METHOD_CACHE_CLONES or total_bytes > MAX_METHOD_CACHE_TOTAL_BYTES):+ oldest = evictable.pop(0)+ clone_count -= 1total_bytes -= _directory_size_bytes(oldest)shutil.rmtree(oldest, ignore_errors=True)diff --git a/api/pipelex_api/method_source.py b/api/pipelex_api/method_source.pyindex 9420d0bec..9266bc725 100644--- a/api/pipelex_api/method_source.py+++ b/api/pipelex_api/method_source.py@@ -1,248 +1,365 @@-"""Resolve a `method_ref` into run/validate/tooling inputs: fetch → locate → refuse → materialize.+"""Resolve a `method_ref` into a method tree: fetch, read from the commit's git objects, list, refuse.The wire field `method_ref` carries a globally resolvable address —`github.com/<owner>/<repo>[/<selector>][@<tag>]` — and THIS runner is its resolver (thelayered extension policy's Rule 3: an address needs no catalog, so it is a layer-2 concept;the hosted-only `method_id` never reaches this server). The grammar, the fetch-at-tag, the-manifest-identity package location, the bounds, and the structures check all live in-`pipelex.methods`; this module composes them behind the SHA-keyed clone cache-(`pipelex_api.method_cache`) and shapes the result for each route family:--- The run routes and `/validate` use :func:`fetched_method_source`: the package's `.mthds`- files travel as `mthds_contents` (paired with their real relative paths as- `mthds_sources`, so diagnostics carry true per-file labels), and only the non-`.mthds`- files are materialized into a temporary `library_dirs` entry — exactly the split the- method-bundle transport uses, so a fetched package runs the same proven path. The- packages it ships under its own `.mthds/methods/<name>/` are partitioned out of both, as- a bundle's are, into a temporary `methods_dirs` entry: the run routes hand it to the- engine, and `/validate` does not, resolving such a dependency through the installed store- or fetch-on-miss.-- The tooling routes (`/resolve`, `/codegen`, `/pipe-io`) use- :func:`fetch_method_mthds_files`: only the package's own `.mthds` files, as `files[]` items,- paired with the manifest's `main_pipe` so the per-pipe route defaults its selector exactly- as a run does. No Python ever loads there, so the execution-locus gate does not apply.--The security gate (packaging invariant 7 — execution locus decides): `.mthds` content is-data, always acceptable. On a deployment that is NOT sandbox-hosted, a fetched package-carrying ANY `.py` is refused with the same 403 the bundle transport uses-(`CustomCodeRequiresSandbox`) — running it would import customer code in-process. On a-sandbox-hosted deployment, PipeFunc `.py` is acceptable (captured as text, executed in the-network-blocked sandbox) but a package declaring `StructuredContent` subclasses is refused-loudly (`MethodStructuresRefusedError` → 403): structure classes would have to be imported into-the runner's own process, and the rule-naming error teaches authors to express types as MTHDS-concepts. The library load refuses the same classes again, for a bundle as for a package.--Everything a run needs is copied OUT of the cached clone before this module yields — the-`.mthds` text into memory, the rest into a per-request temp directory — so cache eviction-can never race a running pipeline.+manifest-identity package location and the bounds all live in `pipelex.methods`; this module+composes them behind the SHA-keyed clone cache (`pipelex_api.method_cache`), so a package is+cached by its commit and a second request for the same commit fetches nothing.++The package is a method tree (`mthds/docs/spec/method-tree.md`): the files of the package's directory in the resolved+commit, read from the commit's git objects as stored, with no checkout filter, line-ending conversion or+`export-ignore`, so its digest is the same on every machine, and filtered by the membership rule read with that+commit's own `.gitignore` files and the package's own `exclude` patterns. A submodule contributes no file, and a+symbolic link committed in the package is refused unless the rule excludes it. The tree's digest is recorded in the+run's provenance (`method_provenance.tree_digest`), beside the commit.++The packages a package ships under its own `.mthds/methods/<name>/`, the layout of the local method cache, are out of+its tree (a dot directory), and each becomes a dependency tree of its own, keyed by the address its manifest declares,+as a dependency a request sends in `dependencies` is.++The routes use it two ways:++- The run routes and `/validate` use :func:`fetched_method`: the tree and its dependency trees, read in full, which the+ routes write out and load as a request's `files` are (`pipelex_api.method_tree_request`), so a fetched package and a+ sent method run the same path, the files a step reads, such as a `PipeDocGen` template, included.+- The tooling routes (`/resolve`, `/codegen`, `/pipe-io`) use :func:`fetch_method_mthds_files`: the tree's `.mthds`+ files, as `files[]` items labelled with their tree paths, paired with the manifest's `main_pipe` so the per-pipe route+ defaults its selector as a run does. It reads only what the membership rule needs to tell which `.mthds` files are+ the tree's, never the shipped packages, so it neither stages nor hashes the rest of the package. No Python ever loads+ there, so the execution-locus gate does not apply.++The security gate (packaging invariant 7 — execution locus decides) is the request tree's+(`pipelex_api.method_tree_request.ensure_python_allowed`): on a deployment that is not sandbox-hosted, a package+carrying any `.py` is refused with a 403 `CustomCodeRequiresSandbox`; on a sandbox-hosted one, PipeFunc `.py` is+acceptable but a package declaring `StructuredContent` subclasses is refused (`MethodStructuresRefusedError` → 403).++Everything a run needs is read OUT of the cached clone before this module returns — the tree's bytes into memory —+while the clone is pinned against eviction (`MethodCloneCache.pinned_package`), so cache eviction never races a read or+a running pipeline."""-from collections.abc import Generator-from contextlib import ExitStack, contextmanager-from pathlib import Path, PurePosixPath-from typing import NamedTuple, NoReturn+import os+import shutil+import subprocess+import tempfile+from collections.abc import Callable, Generator+from contextlib import contextmanager+from pathlib import Path+from typing import NamedTuple-from mthds.package.discovery import MANIFEST_FILENAME+from mthds.method_tree.discovery import BUNDLE_SUFFIX, GITIGNORE_NAME, MANIFEST_NAMEfrom pipelex import log-from pipelex.config import is_pipe_func_sandbox_hosted+from pipelex.libraries.method_trees import list_package_method_tree+from pipelex.methods.exceptions import MethodFetchErrorfrom pipelex.methods.fetching import FetchedMethodPackage, MethodProvenancefrom pipelex.methods.method_ref import parse_method_ref-from pipelex.methods.structures_check import ensure_no_structured_content_python-from pydantic import BaseModel, ConfigDict, Field-from pipelex_api.bundle import BundlePartition, ParsedBundle, materialize_parsed, materialized_methods_dir, partition_bundle_entriesfrom pipelex_api.error_types import ErrorType-from pipelex_api.errors import raise_forbidden, raise_validation_error+from pipelex_api.errors import raise_validation_errorfrom pipelex_api.limits import MAX_MTHDS_FILE_BYTESfrom pipelex_api.method_cache import get_method_clone_cache+from pipelex_api.method_tree_request import (+ RequestMethod,+ RequestTree,+ add_dependency,+ dependency_address_of,+ ensure_method_within_limits,+ ensure_tree_within_limits,+ is_package_dir_name,+)from pipelex_api.schemas.models import MthdsFileItem-_SKIPPED_DIR_NAMES = {".git", "__pycache__"}+GIT_TIMEOUT_SECONDS = 60+# The git modes of a tree entry this reader treats apart from a file: an executable file, a symbolic link, and a submodule's commit.+_GIT_MODE_EXECUTABLE = "100755"+_GIT_MODE_SYMLINK = "120000"+_GIT_MODE_SUBMODULE = "160000"+# Where a package ships the method packages it calls, the layout of the local method cache.+_VENDORED_METHODS_DIR = Path(".mthds") / "methods"+# The files the membership rule reads to judge an entry beside the entry's own name: the patterns of a `.gitignore` file+# and of the manifest, a manifest marking a nested package, and a `pyvenv.cfg` marking a virtual environment.+_MEMBERSHIP_RULE_NAMES = frozenset({GITIGNORE_NAME, MANIFEST_NAME, "pyvenv.cfg"})-class FetchedMethodSource(BaseModel):- """A fetched package shaped for the run/validate path (see the module docstring)."""+class FetchedMethod(NamedTuple):+ """A fetched package as a method: its tree with the dependency trees it ships, and what the run records."""- model_config = ConfigDict(frozen=True)-- mthds_contents: list[str]- mthds_sources: list[str]- library_dirs: list[str] | None = None- methods_dirs: list[Path] | None = Field(- default=None, description="The methods directory holding the packages the package ships under its own `.mthds/methods/`."- )- main_pipe: str | None = Field(default=None, description="The manifest's declared entry pipe; a request `pipe_code` overrides it.")+ request_method: RequestMethod+ package: FetchedMethodPackageprovenance: MethodProvenance+ @property+ def label(self) -> str:+ """How messages name the package: its address, quoted."""+ return f"'{self.package.full_address}'"+class FetchedMthdsFiles(NamedTuple):"""A fetched package shaped for the tooling path: its `.mthds` files plus the manifest's entry pipe."""files: list[MthdsFileItem]- """The package's `.mthds` files as `files[]` items, each labelled with its real relative path."""+ """The package's `.mthds` files as `files[]` items, each labelled with its tree path."""main_pipe: str | None"""The manifest's declared `main_pipe` (a bare pipe code), or None when the manifest declares none."""-def _fetch_package(method_ref: str) -> FetchedMethodPackage:- """Parse the reference and fetch its package through the SHA-keyed clone cache.+class _GitTreeEntry(NamedTuple):+ mode: str+ object_sha: str+ path: str- Every failure mode is a distinct pipelex `MethodRefError` subclass, rendered by the- global handler as RFC 7807 `problem+json` with the class name as `error_type` and the- status the API maps for it (`pipelex_api.exception_handlers._ERROR_TYPE_STATUS_OVERRIDES`).++def _run_git(*, clone_dir: Path, args: list[str], stdin: bytes | None, package_address: str) -> bytes:+ """Run one git command on a cached clone, its output as bytes.++ Raises:+ MethodFetchError: git is not installed, the command fails, or it times out."""- ref = parse_method_ref(method_ref)- package = get_method_clone_cache().get_or_fetch(ref=ref)- # The tag and the commit say which version of the package a hosted run executed; the tag is `None` when the- # reference named none and the default branch was fetched.- log.info(- "A method reference was resolved to a package",- fields={- "method_ref": ref.ref_str,- "package_address": package.provenance.address,- "fetched_tag": package.provenance.tag,- "commit_sha": package.provenance.commit_sha,- },- )- return package+ try:+ result = subprocess.run( # noqa: S603 — fixed argv, no shell; every argument is ours or an object id git printed+ ["git", "-C", str(clone_dir), *args], # noqa: S607+ input=stdin,+ capture_output=True,+ check=True,+ timeout=GIT_TIMEOUT_SECONDS,+ )+ except FileNotFoundError as exc:+ msg = "git is not installed or not found on PATH"+ raise MethodFetchError(msg) from exc+ except subprocess.CalledProcessError as exc:+ msg = f"Could not read method package '{package_address}' from its commit: {exc.stderr.decode('utf-8', errors='replace').strip()}"+ raise MethodFetchError(msg) from exc+ except subprocess.TimeoutExpired as exc:+ msg = f"Timed out reading method package '{package_address}' from its commit"+ raise MethodFetchError(msg) from exc+ return result.stdout++def _package_tree_entries(*, package: FetchedMethodPackage) -> list[_GitTreeEntry]:+ """Every entry of the package's directory in the fetched commit, by its path relative to the package's directory.-def _package_files(package: FetchedMethodPackage) -> list[Path]:- """The package's files, deterministically ordered, with VCS/tooling residue skipped."""- files: list[Path] = []- for file_path in sorted(package.package_dir.rglob("*")):- relative_parts = file_path.relative_to(package.package_dir).parts- if any(part in _SKIPPED_DIR_NAMES for part in relative_parts):+ Raises:+ MethodFetchError: git cannot list the commit's tree, or a path in it is not UTF-8 text.+ """+ package_rel = package.package_dir.relative_to(package.clone_dir).as_posix()+ # The clone the cache keys by the resolved commit is checked out at that commit, so its HEAD is the commit.+ args = ["ls-tree", "-r", "-z", "--full-tree", "HEAD"]+ prefix = ""+ if package_rel != ".":+ args += ["--", f":(literal){package_rel}"]+ prefix = f"{package_rel}/"+ output = _run_git(clone_dir=package.clone_dir, args=args, stdin=None, package_address=package.full_address)+ entries: list[_GitTreeEntry] = []+ for record in output.split(b"\0"):+ if not record:+ continue+ header, _, raw_path = record.partition(b"\t")+ mode, _, rest = header.decode("ascii").partition(" ")+ _, _, object_sha = rest.partition(" ")+ try:+ path = raw_path.decode("utf-8")+ except UnicodeDecodeError as exc:+ msg = f"Method package '{package.full_address}' holds a file whose name is not UTF-8 text, which no method tree can hold."+ raise MethodFetchError(msg) from exc+ if not path.startswith(prefix):continue- if file_path.is_file():- files.append(file_path)- return files+ entries.append(_GitTreeEntry(mode=mode, object_sha=object_sha, path=path.removeprefix(prefix)))+ return entries-def _decode_mthds_text(data: bytes, *, relative: str, package_address: str) -> str:- try:- content = data.decode("utf-8")- except UnicodeDecodeError:- raise_validation_error(message=f"File '{relative}' in method package '{package_address}' is not valid UTF-8.")- if len(data) > MAX_MTHDS_FILE_BYTES:- msg = f"File '{relative}' in method package '{package_address}' exceeds the {MAX_MTHDS_FILE_BYTES // 1024} KiB per-file limit."- raise_validation_error(message=msg)- return content+def _read_git_objects(*, package: FetchedMethodPackage, object_shas: list[str]) -> dict[str, bytes]:+ """The contents of these blobs of the clone, read in one `git cat-file --batch`, by object id.+ Raises:+ MethodFetchError: git cannot read an object, or its answer is not the one asked for.+ """+ if not object_shas:+ return {}+ unique_shas = sorted(set(object_shas))+ output = _run_git(+ clone_dir=package.clone_dir,+ args=["cat-file", "--batch"],+ stdin="".join(f"{object_sha}\n" for object_sha in unique_shas).encode("ascii"),+ package_address=package.full_address,+ )+ contents: dict[str, bytes] = {}+ cursor = 0+ for object_sha in unique_shas:+ line_end = output.index(b"\n", cursor)+ header = output[cursor:line_end].decode("ascii").split(" ")+ if len(header) != 3 or header[0] != object_sha:+ msg = f"Could not read the object {object_sha} of method package '{package.full_address}' from its commit."+ raise MethodFetchError(msg)+ size = int(header[2])+ start = line_end + 1+ contents[object_sha] = output[start : start + size]+ cursor = start + size + 1+ return contents+++def _stage_package(*, package: FetchedMethodPackage, staging_dir: Path, keeps: Callable[[str], bool] | None = None) -> None:+ """Write the package's directory as its commit stores it into `staging_dir`, from the clone's git objects.++ A symbolic link is written as one, for the tree's lister to refuse unless the membership rule excludes it, and a+ submodule contributes no file.++ Args:+ package: The fetched package.+ staging_dir: The directory to write it into.+ keeps: Which paths, relative to the package's directory, to write; every path when None.+ """+ entries = [entry for entry in _package_tree_entries(package=package) if keeps is None or keeps(entry.path)]+ blobs = _read_git_objects(package=package, object_shas=[entry.object_sha for entry in entries if entry.mode != _GIT_MODE_SUBMODULE])+ for entry in entries:+ if entry.mode == _GIT_MODE_SUBMODULE:+ continue+ target = staging_dir.joinpath(*entry.path.split("/"))+ target.parent.mkdir(parents=True, exist_ok=True)+ data = blobs[entry.object_sha]+ if entry.mode == _GIT_MODE_SYMLINK:+ target.symlink_to(os.fsdecode(data))+ continue+ target.write_bytes(data)+ if entry.mode == _GIT_MODE_EXECUTABLE:+ target.chmod(0o755)-def _partition_package(package: FetchedMethodPackage, *, files: list[Path]) -> BundlePartition:- """Sort a fetched package's files as a bundle's are: its own `.mthds`, its other files, and the packages it ships.- The refusals are a bundle's too: a `.mthds` path no reference could find is a `422` naming the file.+def _read_listed_tree(*, root: Path, package_address: str, label: str) -> RequestTree:+ """List a staged package directory as a method tree and read every file of it into memory.++ Raises:+ MethodTreeLoadError: The standard refused the tree, it passed a limit, or it could not be read."""- entries = tuple((PurePosixPath(file_path.relative_to(package.package_dir).as_posix()), file_path.read_bytes()) for file_path in files)- return partition_bundle_entries(entries, owner=f"Method package '{package.full_address}'")+ tree = list_package_method_tree(root=root, package_address=package_address)+ contents = {tree_path: tree.file_location(tree_path).read_bytes() for tree_path in tree.tree_paths}+ request_tree = RequestTree.make_from_contents(contents=contents)+ ensure_tree_within_limits(tree=request_tree, label=label)+ return request_tree-def _refuse_no_own_mthds(package: FetchedMethodPackage, *, partition: BundlePartition) -> NoReturn:- msg = f"Method package '{package.full_address}' contains no .mthds file."- if partition.vendored_packages:- msg = (- f"Method package '{package.full_address}' contains no .mthds file of its own: "- "the files under '.mthds/methods/' are the packages it ships."+def _read_package_method(*, package: FetchedMethodPackage) -> RequestMethod:+ """The package's tree and the dependency trees it ships, read from the commit's git objects."""+ staging_dir = Path(tempfile.mkdtemp(prefix="pipelex-package-"))+ try:+ package_root = staging_dir / "package"+ package_root.mkdir()+ _stage_package(package=package, staging_dir=package_root)+ label = f"Method package '{package.full_address}'"+ tree = _read_listed_tree(root=package_root, package_address=package.full_address, label=label)+ dependencies: dict[str, RequestTree] = {}+ vendored_root = package_root / _VENDORED_METHODS_DIR+ if vendored_root.is_dir():+ for vendored_dir in sorted(vendored_root.iterdir()):+ # A shipped package is a directory with a name a package's directory can take, as a `bundle_b64`+ # archive's are (`read_archive_method`); anything else under `.mthds/methods/` is no package.+ if not vendored_dir.is_dir() or vendored_dir.is_symlink() or not is_package_dir_name(vendored_dir.name):+ continue+ vendored_label = f"The package '.mthds/methods/{vendored_dir.name}/' of method package '{package.full_address}'"+ dependency_tree = _read_listed_tree(root=vendored_dir, package_address=vendored_label, label=vendored_label)+ address = dependency_address_of(tree=dependency_tree, fallback_name=vendored_dir.name, label=vendored_label)+ add_dependency(dependencies=dependencies, address=address, tree=dependency_tree, label=vendored_label)+ request_method = RequestMethod(tree=tree, dependencies=dependencies)+ ensure_method_within_limits(request_method=request_method, label=label)+ return request_method+ finally:+ shutil.rmtree(staging_dir, ignore_errors=True)+++@contextmanager+def _fetched_package(method_ref: str) -> Generator[FetchedMethodPackage, None, None]:+ """Parse the reference and fetch its package through the SHA-keyed clone cache, its clone pinned in the block.++ Every failure mode is a distinct pipelex `MethodRefError` subclass, rendered by the+ global handler as RFC 7807 `problem+json` with the class name as `error_type` and the+ status the API maps for it (`pipelex_api.exception_handlers._ERROR_TYPE_STATUS_OVERRIDES`).+ """+ ref = parse_method_ref(method_ref)+ with get_method_clone_cache().pinned_package(ref=ref) as package:+ # The tag and the commit say which version of the package a hosted run executed; the tag is `None` when the+ # reference named none and the default branch was fetched.+ log.info(+ "A method reference was resolved to a package",+ fields={+ "method_ref": ref.ref_str,+ "package_address": package.provenance.address,+ "fetched_tag": package.provenance.tag,+ "commit_sha": package.provenance.commit_sha,+ },)- raise_validation_error(message=msg)+ yield package-def _apply_execution_locus_gate(package: FetchedMethodPackage, *, python_relpaths: list[str]) -> None:- """Refuse Python that would execute where it must not (see the module docstring)."""- if not python_relpaths:- return- if not is_pipe_func_sandbox_hosted():- msg = f"Method package '{package.full_address}' ships custom Python (.py); running it requires a sandbox-hosted deployment."- raise_forbidden(message=msg, error_type=ErrorType.CUSTOM_CODE_REQUIRES_SANDBOX)- ensure_no_structured_content_python(package_dir=package.package_dir, package_address=package.full_address)+def _keeps_for_bundle_listing(path: str) -> bool:+ """Whether a package's file is one the tooling read writes: a `.mthds` file, or one the membership rule reads."""+ name = path.rsplit("/", 1)[-1]+ return path.endswith(BUNDLE_SUFFIX) or name in _MEMBERSHIP_RULE_NAMES-@contextmanager-def fetched_method_source(method_ref: str) -> Generator[FetchedMethodSource, None, None]:- """Fetch a `method_ref`'s package and shape it for the run/validate path.+def _read_package_bundles(*, package: FetchedMethodPackage) -> dict[str, bytes]:+ """The `.mthds` files of the package's tree by tree path, read without staging or hashing the rest of the package.- Yields the package's `.mthds` files as `(mthds_contents, mthds_sources)` pairs, its- non-`.mthds` files (PipeFunc `.py`, `requirements.txt`, …) materialized into a temporary- `library_dirs` entry, and the packages it ships under its own `.mthds/methods/<name>/`- materialized into a temporary `methods_dirs` entry, each cleaned up on exit. The- execution-locus gate runs before anything touches disk.+ Only the `.mthds` files and the files the membership rule reads are written out, so the lister keeps exactly the+ `.mthds` files the whole tree would keep. The packages the package ships stay out, being under a dot directory.+ """+ staging_dir = Path(tempfile.mkdtemp(prefix="pipelex-package-bundles-"))+ try:+ _stage_package(package=package, staging_dir=staging_dir, keeps=_keeps_for_bundle_listing)+ tree = list_package_method_tree(root=staging_dir, package_address=package.full_address)+ return {tree_path: tree.file_location(tree_path).read_bytes() for tree_path in tree.tree_paths if tree_path.endswith(BUNDLE_SUFFIX)}+ finally:+ shutil.rmtree(staging_dir, ignore_errors=True)+++def fetched_method(method_ref: str) -> FetchedMethod:+ """Fetch a `method_ref`'s package and read it as a method tree, with the dependency trees it ships.Args:method_ref: The raw `method_ref` string from the request.Raises:- MethodRefError subclasses: parse, fetch, location, bounds, and structures failures —- each rendered as `problem+json` by the global handler.- ApiError: the 403 custom-code gate on a non-sandbox deployment, and 422s for a- package whose `.mthds` content this server cannot accept.+ MethodRefError subclasses: parse, fetch, location and bounds failures, each rendered as `problem+json` by the+ global handler.+ MethodTreeLoadError: The standard refused the package's tree (a 422 naming the package by its address).+ ApiError: A tree past this runner's limits, or a shipped package without a manifest (422s)."""- package = _fetch_package(method_ref)- files = _package_files(package)- python_relpaths = [file_path.relative_to(package.package_dir).as_posix() for file_path in files if file_path.suffix == ".py"]- # The whole package, the packages it ships included, so a structure class anywhere in it is refused before anything is copied.- _apply_execution_locus_gate(package, python_relpaths=python_relpaths)-- partition = _partition_package(package, files=files)- mthds_contents: list[str] = []- mthds_sources: list[str] = []- for relpath, data in partition.mthds_entries:- relative = relpath.as_posix()- mthds_contents.append(_decode_mthds_text(data, relative=relative, package_address=package.full_address))- mthds_sources.append(relative)- if not mthds_contents:- _refuse_no_own_mthds(package, partition=partition)- # The manifest is already consumed (identity + `main_pipe`); materializing it into the library dir would hand the- # local loader a package boundary it must not see.- other_entries = tuple((relpath, data) for relpath, data in partition.library_entries if relpath.name != MANIFEST_FILENAME)-- with ExitStack() as stack:- library_dirs: list[str] | None = None- if other_entries:- library_bundle = stack.enter_context(materialize_parsed(ParsedBundle(entries=other_entries)))- library_dirs = [str(library_bundle.directory)]- methods_dirs: list[Path] | None = None- if partition.vendored_packages:- methods_dirs = [stack.enter_context(materialized_methods_dir(partition.vendored_packages))]- yield FetchedMethodSource(- mthds_contents=mthds_contents,- mthds_sources=mthds_sources,- library_dirs=library_dirs,- methods_dirs=methods_dirs,- main_pipe=package.manifest.main_pipe,- provenance=package.provenance,- )+ with _fetched_package(method_ref) as package:+ request_method = _read_package_method(package=package)+ provenance = package.provenance.model_copy(update={"tree_digest": request_method.tree.tree.digest})+ return FetchedMethod(request_method=request_method, package=package, provenance=provenance)def fetch_method_mthds_files(method_ref: str) -> FetchedMthdsFiles:- """Fetch a `method_ref`'s package and return its `.mthds` files as `files[]` items, plus its `main_pipe`.-- The tooling-route shape: each item pairs the file's content with its real relative path- as `source`, so crate provenance and diagnostics carry true per-file labels. Only the- package's own `.mthds` data travels, never that of a package it ships under its- `.mthds/methods/`, since these routes resolve no address-based dependency. The package's- Python (if any) never loads on these routes, so the execution-locus gate does not apply- here. The manifest's `main_pipe` rides beside the- files so a per-pipe projection can default its selector the way a run does — the manifest- is the package author's declaration of the entry pipe, and dropping it here would make- `main_pipe` buy them nothing on the tooling routes.+ """Fetch a `method_ref`'s package and return its tree's `.mthds` files as `files[]` items, plus its `main_pipe`.++ The tooling-route shape: each item pairs the file's content with its tree path as `source`, so crate provenance and+ diagnostics carry true per-file labels. Only the package's own `.mthds` files travel, never those of a package it+ ships under its `.mthds/methods/`, since these routes resolve no address-based dependency. The manifest's `main_pipe`+ rides beside the files so a per-pipe projection can default its selector the way a run does.Args:method_ref: The raw `method_ref` string from the request.Raises:MethodRefError subclasses: parse, fetch, location, and bounds failures.+ MethodTreeLoadError: The standard refused the package's tree.ApiError: 422 for a package with no `.mthds` file or one this server cannot accept."""- package = _fetch_package(method_ref)- partition = _partition_package(package, files=_package_files(package))+ with _fetched_package(method_ref) as package:+ bundles = _read_package_bundles(package=package)+ label = f"'{package.full_address}'"items: list[MthdsFileItem] = []- for relpath, data in partition.mthds_entries:- relative = relpath.as_posix()- content = _decode_mthds_text(data, relative=relative, package_address=package.full_address)- items.append(MthdsFileItem(content=content, source=relative))+ for tree_path, data in bundles.items():+ try:+ content = data.decode("utf-8")+ except UnicodeDecodeError:+ raise_validation_error(message=f"File '{tree_path}' in method package {label} is not valid UTF-8.")+ if len(data) > MAX_MTHDS_FILE_BYTES:+ msg = f"File '{tree_path}' in method package {label} exceeds the {MAX_MTHDS_FILE_BYTES // 1024} KiB per-file limit."+ raise_validation_error(message=msg)+ items.append(MthdsFileItem(content=content, source=tree_path))if not items:- _refuse_no_own_mthds(package, partition=partition)+ raise_validation_error(message=f"Method package {label} contains no {BUNDLE_SUFFIX} file.", error_type=ErrorType.INVALID_BUNDLE)return FetchedMthdsFiles(files=items, main_pipe=package.manifest.main_pipe)diff --git a/api/pipelex_api/method_tree_request.py b/api/pipelex_api/method_tree_request.pynew file mode 100644index 000000000..27798512f--- /dev/null+++ b/api/pipelex_api/method_tree_request.py@@ -0,0 +1,727 @@+"""A method a request carries as a tree: reading its map of file entries, and writing it out to load it.++A run or validate request names its method by a map of file entries, `files`: each key a tree path, each value the+file's text, its bytes in Base64 (`{"base64": …}`) or the hash of bytes this runner already holds (`{"sha256": …,+"size": …}`), as the MTHDS standard writes a tree that travels (`mthds/docs/spec/method-tree.md`, "The File Entry").+The map is the tree: the runner reads it with the standard's own reader, from the JSON text the request carried, since a+key given twice and a `size` written `13.0` are only visible there, and never applies the membership rule to it again.+A dependency the runner cannot fetch travels beside it in `dependencies`, a tree of its own keyed by its address. A+`bundle_b64` archive is read the same way, its entries made a map, so every form a method travels in is judged by the+one rule of the tree: a key that is not a tree path, or that names a file the membership rule keeps out of every tree,+is refused whatever form carried it.++What this module does, in order:++1. Every limit of this runner on a tree (`MAX_METHOD_TREE_FILES`, `MAX_METHOD_TREE_TOTAL_KIB`,+ `MAX_METHOD_TREE_FILE_KIB`) is checked on the sizes the entries state, before any blob is read: a 422+ `MethodTreeTooLarge` naming the limit.+2. The bytes of every hash entry are read from the runner's blob store, under the caller's scope, at once.+3. The map is read (`read_file_map`): a refusal of the standard is a 422 `InvalidMethodFiles` whose `tree_refusal`+ member carries the refusal's code and the paths it names, and hash entries the store does not hold at their stated+ size are one 422 `MissingBlobs` whose `missing_blobs` member lists every such hash.+4. The files `resources` sends join the tree (`with_resources`), the same bytes under a path the tree holds or a new+ file at a tree path; other bytes under a path the tree holds are a 422 `InvalidResourcePath` naming it.+5. Python is honored only on a sandbox-hosted deployment: a method holding a `.py` file is refused with a 403+ `CustomCodeRequiresSandbox` before anything is written, and on a sandbox-hosted deployment a tree declaring a Python+ structure class is refused (`MethodStructuresRefusedError`, a 403).+6. Each tree read in full is recorded in the blob store, its files then its listing, so the next request can send it+ by hash, and `POST /v1/blobs/missing` answers its digest with one lookup.+7. The trees are written out into a temporary directory and held as written (`pipelex.libraries.written_trees`), so+ the load reads them back as written and names their files by their tree paths, then removed when the request ends.+"""++import base64+import json+import shutil+import tempfile+from collections.abc import Generator, Mapping+from contextlib import ExitStack, contextmanager+from pathlib import Path+from typing import Any, NamedTuple, NoReturn, cast++from mthds.method_tree.digest import is_sha256_hex, tree_digest, tree_files_from_contents+from mthds.method_tree.discovery import BUNDLE_SUFFIX, MANIFEST_NAME+from mthds.method_tree.exceptions import MethodTreeRefusedError, TreeSourceError+from mthds.method_tree.file_entry import BASE64_KEY, SHA256_KEY, SIZE_KEY, BlobStore, InMemoryBlobStore, excluded_name_violation, read_file_map+from mthds.method_tree.models import MethodTree+from mthds.method_tree.tree_path import path_collisions, tree_path_violation+from mthds.method_tree.vocabulary import TreeRefusalCode+from mthds.package.exceptions import ManifestError+from mthds.package.manifest.parser import parse_methods_toml+from mthds.package.manifest.schema import MethodsManifest+from pipelex.cli.installed_methods import PROVENANCE_FILENAME+from pipelex.config import is_pipe_func_sandbox_hosted+from pipelex.libraries.written_trees import WRITTEN_METHOD_TREES, write_method_tree+from pipelex.methods.structures_check import ensure_no_structured_content_in_library_sources, ensure_no_structured_content_python+from pipelex.mthds_parsing.parser import MthdsParser+from pipelex.pipeline.exceptions import MethodEntryError+from pipelex.pipeline.method_entry import EntryCandidate, MethodEntry, select_entry+from pipelex.pipeline.validate_bundle_translation import translate_to_validate_bundle_error++from pipelex_api.blob_store import RunnerBlobStore+from pipelex_api.bundle import decode_bundle_archive+from pipelex_api.bundle_resources import refuse_two_files_under_one_path+from pipelex_api.error_types import ErrorType+from pipelex_api.errors import raise_forbidden, raise_validation_error, raise_validation_problem+from pipelex_api.limits import (+ MAX_METHOD_DEPENDENCIES,+ MAX_METHOD_REQUEST_TOTAL_BYTES,+ MAX_METHOD_TREE_FILE_BYTES,+ MAX_METHOD_TREE_FILES,+ MAX_METHOD_TREE_TOTAL_BYTES,+)++FILES_FIELD = "files"+DEPENDENCIES_FIELD = "dependencies"+PYTHON_SUFFIX = ".py"+FILES_LABEL = "The `files` map"+ARCHIVE_LABEL = "The `bundle_b64` archive"+_METHOD_LABEL = "The method sent in `files`"+_WITH_RESOURCES_LABEL = "The method's tree with the files sent in `resources`"+# How a request's method is named in the messages of the entry rule: "The method sent in `files` holds no …".+REQUEST_METHOD_LABEL = "sent in the request"+# Where a `bundle_b64` archive ships the method packages it calls, the layout of the local method cache.+VENDORED_METHODS_PREFIX = ".mthds/methods/"+# A file the membership rule judges a directory by, which stands in for any file inside a directory being judged.+_ANY_FILE_NAME = "file"+++class RequestTree(NamedTuple):+ """A method tree read in full: the tree, and each file's bytes by its tree path."""++ tree: MethodTree+ contents: dict[str, bytes]++ @property+ def has_python(self) -> bool:+ return any(tree_path.endswith(PYTHON_SUFFIX) for tree_path in self.contents)++ @property+ def bundle_paths(self) -> list[str]:+ return [tree_path for tree_path in self.tree.tree_paths if tree_path.endswith(BUNDLE_SUFFIX)]++ @classmethod+ def make_from_contents(cls, *, contents: dict[str, bytes]) -> "RequestTree":+ """The tree these files make, each at its tree path, already judged as a tree's files."""+ files = tree_files_from_contents(contents)+ return cls(tree=MethodTree(files=files, digest=tree_digest(files)), contents=contents)+++class RequestMethod(NamedTuple):+ """A method as trees: its own, and the dependency trees it carries, keyed by their addresses."""++ tree: RequestTree+ dependencies: dict[str, RequestTree]++ @property+ def has_python(self) -> bool:+ return self.tree.has_python or any(dependency.has_python for dependency in self.dependencies.values())+++class TreeDocuments(NamedTuple):+ """The `files` and `dependencies` members of a request body, each map as the JSON text a receiver reads it from."""++ files: str | None+ dependencies: dict[str, str] | None+++class _JsonObject(NamedTuple):+ """A JSON object as the pairs it was written with, in document order, a key given twice included."""++ pairs: list[tuple[str, Any]]+++class _JsonConstant(NamedTuple):+ """`NaN`, `Infinity` or `-Infinity`, which `json.loads` accepts and JSON does not hold, kept as it was written.++ The body's other members are the rest of the server's to judge, so a constant is refused here only inside a map of+ file entries, where it reaches the standard's reader as it was written, and that reader refuses it.+ """++ text: str+++def _parse_keeping_pairs(text: str) -> Any:+ return json.loads(text, object_pairs_hook=_JsonObject, parse_constant=_JsonConstant)+++def _serialized(value: Any) -> str:+ """The JSON text of a value parsed keeping its pairs, written back as it was parsed.++ A key given twice stays twice, and a number keeps its form: an integer stays an integer, and a number written with a+ fraction or an exponent comes back as a float, which the standard's reader refuses as a `size`. A string is written+ with JSON's escapes, so a lone surrogate reaches the reader as the escape it was. The body's nesting was bounded+ before it was parsed (`pipelex_api.json_body`), which bounds this recursion.+ """+ if isinstance(value, _JsonObject):+ members = ",".join(f"{json.dumps(key)}:{_serialized(member)}" for key, member in value.pairs)+ return f"{{{members}}}"+ if isinstance(value, _JsonConstant):+ return value.text+ if isinstance(value, list):+ return f"[{','.join(_serialized(item) for item in cast('list[Any]', value))}]"+ return json.dumps(value)+++def tree_documents_of(*, body: bytes) -> TreeDocuments:+ """The `files` and `dependencies` members of a request body, each map as JSON text, or None where a member is absent.++ The body was already parsed and checked as JSON (`pipelex_api.json_body`); it is parsed again here keeping every+ object's pairs, since a receiver reads a map from its text, so a route calls this only for a body that sends a+ tree. It is decoded as `json.loads` decodes bytes, UTF-8, UTF-16 or UTF-32, as the route's own parser accepted it.+ A member given twice in the body is read as JSON reads it, the last one counting, and a `null` member is absent.++ Raises (as problem documents): a 422 `InvalidJSON` for a body that does not decode or parse, and a 422+ `InvalidMethodFiles` for `dependencies` that is not an object, or that gives one address twice.+ """+ try:+ parsed = _parse_keeping_pairs(body.decode(json.detect_encoding(body), "surrogatepass"))+ except (UnicodeDecodeError, ValueError) as exc:+ # The route's parser read these bytes already, so this is a second line of defence: `JSONDecodeError` and+ # an integer longer than Python converts are both a `ValueError`.+ raise_validation_error(message=f"Request body is not valid JSON: {exc!s}", error_type=ErrorType.INVALID_JSON)+ if not isinstance(parsed, _JsonObject):+ return TreeDocuments(files=None, dependencies=None)+ members = dict(parsed.pairs)+ files_member = members.get(FILES_FIELD)+ files_document = _serialized(files_member) if files_member is not None else None+ dependencies_member = members.get(DEPENDENCIES_FIELD)+ if dependencies_member is None:+ return TreeDocuments(files=files_document, dependencies=None)+ if not isinstance(dependencies_member, _JsonObject):+ msg = "`dependencies` must be a JSON object mapping each dependency's address to its map of file entries."+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_METHOD_FILES)+ dependency_documents: dict[str, str] = {}+ for address, dependency_member in dependencies_member.pairs:+ if address in dependency_documents:+ raise_validation_problem(+ message=f"`dependencies` gives the address '{wire_name(address)}' twice.",+ error_type=ErrorType.INVALID_METHOD_FILES,+ members={"tree_refusal": {"code": TreeRefusalCode.DUPLICATE_KEY, "names": [wire_name(address)]}},+ )+ dependency_documents[address] = _serialized(dependency_member)+ return TreeDocuments(files=files_document, dependencies=dependency_documents)+++class _MapCensus(NamedTuple):+ """What a map's entries state before any is read: how many files, how many bytes, and which hashes at which sizes."""++ file_count: int+ stated_bytes: int+ largest_path: str | None+ largest_bytes: int+ stated_sizes: dict[str, set[int]]+++def _stated_size(entry: Any) -> tuple[int, str | None]:+ """The size an entry states, and the hash it names when it is a hash entry; zero for an entry the reader will refuse."""+ if isinstance(entry, str):+ return len(entry.encode("utf-8", errors="surrogatepass")), None+ if not isinstance(entry, _JsonObject):+ return 0, None+ fields = dict(entry.pairs)+ encoded = fields.get(BASE64_KEY)+ if isinstance(encoded, str) and len(fields) == 1:+ # The decoded length of well-formed Base64, its padding taken off; the reader refuses any other value.+ return len(encoded) * 3 // 4 - encoded[-2:].count("="), None+ sha256 = fields.get(SHA256_KEY)+ size = fields.get(SIZE_KEY)+ if isinstance(sha256, str) and is_sha256_hex(sha256) and isinstance(size, int) and not isinstance(size, bool) and size >= 0:+ return size, sha256+ return 0, None+++def _census(*, document: str) -> _MapCensus | None:+ """The census of a map of file entries, or None when the document holds no JSON object, for the reader to refuse."""+ try:+ parsed = _parse_keeping_pairs(document)+ except (ValueError, RecursionError):+ return None+ if not isinstance(parsed, _JsonObject):+ return None+ stated_bytes = 0+ largest_path: str | None = None+ largest_bytes = 0+ stated_sizes: dict[str, set[int]] = {}+ for tree_path, entry in parsed.pairs:+ size, sha256 = _stated_size(entry)+ stated_bytes += size+ if size > largest_bytes:+ largest_path, largest_bytes = tree_path, size+ if sha256 is not None:+ stated_sizes.setdefault(sha256, set()).add(size)+ return _MapCensus(+ file_count=len(parsed.pairs),+ stated_bytes=stated_bytes,+ largest_path=largest_path,+ largest_bytes=largest_bytes,+ stated_sizes=stated_sizes,+ )+++def wire_name(name: str) -> str:+ """A name a refusal reports, as JSON can carry it: a code point UTF-8 cannot encode, a lone surrogate, as its escape."""+ return name.encode("utf-8", errors="backslashreplace").decode("utf-8")+++def ensure_within_tree_limits(*, file_count: int, total_bytes: int, largest_path: str | None, largest_bytes: int, label: str) -> None:+ """Refuse a tree past one of this runner's limits, naming the limit, with a 422 `MethodTreeTooLarge`."""+ if file_count > MAX_METHOD_TREE_FILES:+ msg = f"{label} holds {file_count} files, past this runner's limit of {MAX_METHOD_TREE_FILES} files in a method tree."+ raise_validation_error(message=msg, error_type=ErrorType.METHOD_TREE_TOO_LARGE)+ if largest_path is not None and largest_bytes > MAX_METHOD_TREE_FILE_BYTES:+ msg = (+ f"{label} holds the file '{wire_name(largest_path)}' of {largest_bytes} bytes, past this runner's limit of "+ f"{MAX_METHOD_TREE_FILE_BYTES // 1024} KiB for one file of a method tree."+ )+ raise_validation_error(message=msg, error_type=ErrorType.METHOD_TREE_TOO_LARGE)+ if total_bytes > MAX_METHOD_TREE_TOTAL_BYTES:+ msg = f"{label} holds {total_bytes} bytes, past this runner's limit of {MAX_METHOD_TREE_TOTAL_BYTES // 1024} KiB for a method tree."+ raise_validation_error(message=msg, error_type=ErrorType.METHOD_TREE_TOO_LARGE)+++def ensure_tree_within_limits(*, tree: RequestTree, label: str) -> None:+ """Refuse a tree read in full that passes one of this runner's limits, with a 422 `MethodTreeTooLarge`."""+ largest_path: str | None = None+ largest_bytes = 0+ for tree_path, data in tree.contents.items():+ if len(data) > largest_bytes:+ largest_path, largest_bytes = tree_path, len(data)+ ensure_within_tree_limits(+ file_count=len(tree.contents),+ total_bytes=sum(len(data) for data in tree.contents.values()),+ largest_path=largest_path,+ largest_bytes=largest_bytes,+ label=label,+ )+++def ensure_trees_within_request_limits(*, dependency_count: int, total_bytes: int, label: str) -> None:+ """Refuse a method whose trees together pass this runner's limits, with a 422 `MethodTreeTooLarge`.++ Each tree is held to the tree limits on its own; these bound what one request carries in all, its method's tree and+ every dependency tree beside it, so the memory and the disk a request takes do not grow with its dependencies.+ """+ if dependency_count > MAX_METHOD_DEPENDENCIES:+ msg = f"{label} carries {dependency_count} dependencies, past this runner's limit of {MAX_METHOD_DEPENDENCIES} dependency trees."+ raise_validation_error(message=msg, error_type=ErrorType.METHOD_TREE_TOO_LARGE)+ if total_bytes > MAX_METHOD_REQUEST_TOTAL_BYTES:+ msg = (+ f"{label} holds {total_bytes} bytes across its tree and its dependency trees, past this runner's limit of "+ f"{MAX_METHOD_REQUEST_TOTAL_BYTES // 1024} KiB for a method and its dependencies."+ )+ raise_validation_error(message=msg, error_type=ErrorType.METHOD_TREE_TOO_LARGE)+++def ensure_method_within_limits(*, request_method: "RequestMethod", label: str) -> None:+ """Refuse a method read in full whose trees together pass this runner's limits, with a 422 `MethodTreeTooLarge`."""+ trees = [request_method.tree, *request_method.dependencies.values()]+ ensure_trees_within_request_limits(+ dependency_count=len(request_method.dependencies),+ total_bytes=sum(len(data) for tree in trees for data in tree.contents.values()),+ label=label,+ )+++def _refuse_map(*, refusal_error: MethodTreeRefusedError, label: str) -> NoReturn:+ """Answer the standard's refusal of a map: the hashes to upload, or the code and the paths it names."""+ refusal = refusal_error.refusal+ match refusal.code:+ case TreeRefusalCode.MISSING_BLOB:+ msg = (+ f"{label} names, by hash, bytes this runner does not hold at their stated size: {', '.join(refusal.names)}. "+ "Upload them (POST /v1/blobs/missing, then PUT /v1/blobs/{sha256}) and send the request again."+ )+ raise_validation_problem(message=msg, error_type=ErrorType.MISSING_BLOBS, members={"missing_blobs": list(refusal.names)})+ case (+ TreeRefusalCode.SYMBOLIC_LINK+ | TreeRefusalCode.SPECIAL_FILE+ | TreeRefusalCode.INVALID_TREE_PATH+ | TreeRefusalCode.NORMALIZATION_COLLISION+ | TreeRefusalCode.PATH_COLLISION+ | TreeRefusalCode.EMPTY_TREE+ | TreeRefusalCode.LONE_FILE_EXCLUDED+ | TreeRefusalCode.GITIGNORE_NOT_UTF8+ | TreeRefusalCode.DUPLICATE_KEY+ | TreeRefusalCode.EXCLUDED_NAME+ | TreeRefusalCode.MALFORMED_ENTRY+ | TreeRefusalCode.REFERENCE_INVALID+ | TreeRefusalCode.REFERENCE_LEAVES_TREE+ | TreeRefusalCode.REFERENCE_NOT_A_TREE_PATH+ | TreeRefusalCode.REFERENCE_NOT_IN_TREE+ | TreeRefusalCode.REFERENCE_IN_ONE_FILE_TREE+ | TreeRefusalCode.FUNCTION_NAME_WITHOUT_MODULE+ | TreeRefusalCode.FUNCTION_NAME_INVALID_SEGMENT+ | TreeRefusalCode.MODULE_NOT_IN_TREE+ | TreeRefusalCode.MODULE_AMBIGUOUS+ ):+ raise_validation_problem(+ message=f"{label} is not a method tree. {refusal_error.message}",+ error_type=ErrorType.INVALID_METHOD_FILES,+ members={"tree_refusal": {"code": refusal.code, "names": [wire_name(name) for name in refusal.names]}},+ )+++async def _held_blobs(*, blob_store: RunnerBlobStore, stated_sizes: dict[str, set[int]], label: str) -> BlobStore:+ """The bytes a map's hash entries name that the blob store holds, looked up by size before any is read.++ A hash held at another size than one of its entries states is refused, since uploading its bytes again would not+ change the size the store holds them at. Every other held hash is held at the one size its entries state, which the+ census bounded, so reading them reads no more than the census allowed. A hash not held is left out, for the reader+ to report among the blobs this runner does not hold.++ Raises (as a problem document): a 422 `BlobMismatch` whose `mismatched_blobs` member lists each hash held at another+ size, with that size.+ """+ stored = await blob_store.stored_sizes(sha256s=sorted(stated_sizes))+ mismatched = {sha256: size for sha256, size in stored.items() if size is not None and stated_sizes[sha256] != {size}}+ if mismatched:+ described = ", ".join(f"{sha256} (held at {size} bytes)" for sha256, size in mismatched.items())+ msg = (+ f"{label} states a size for a hash other than the size of the bytes this runner holds under it: {described}. "+ "A hash names one set of bytes, so state the size of those bytes."+ )+ raise_validation_problem(+ message=msg,+ error_type=ErrorType.BLOB_MISMATCH,+ members={"mismatched_blobs": [{"sha256": sha256, "held_size": size} for sha256, size in mismatched.items()]},+ )+ return await blob_store.read_held(sizes={sha256: size for sha256, size in stored.items() if size is not None})+++async def read_request_tree(*, document: str, blob_store: RunnerBlobStore | None, label: str, census: _MapCensus | None = None) -> RequestTree:+ """Read one map of file entries a request carried, its hash entries from the runner's blob store.++ Args:+ document: The map's JSON text, as the request carried it.+ blob_store: The blob store of the caller's scope, or None for a map that names no bytes by hash.+ label: How a refusal names the map, such as "The `files` map".+ census: The map's census when the caller took it already, else taken here.++ Raises (as problem documents): `MethodTreeTooLarge`, `InvalidMethodFiles` and `MissingBlobs`, each a 422.+ """+ if census is None:+ census = _census(document=document)+ held: BlobStore = InMemoryBlobStore()+ if census is not None:+ ensure_within_tree_limits(+ file_count=census.file_count,+ total_bytes=census.stated_bytes,+ largest_path=census.largest_path,+ largest_bytes=census.largest_bytes,+ label=label,+ )+ if census.stated_sizes and blob_store is not None:+ held = await _held_blobs(blob_store=blob_store, stated_sizes=census.stated_sizes, label=label)+ try:+ reading = read_file_map(document, blob_store=held)+ except MethodTreeRefusedError as exc:+ _refuse_map(refusal_error=exc, label=label)+ except TreeSourceError as exc:+ raise_validation_problem(message=f"{label} is not a method tree. {exc.message}", error_type=ErrorType.INVALID_METHOD_FILES, members={})+ request_tree = RequestTree(tree=reading.tree, contents=reading.contents)+ ensure_tree_within_limits(tree=request_tree, label=label)+ return request_tree+++def _base64_map_document(*, entries: list[tuple[str, bytes]]) -> str:+ """A map of Base64 entries written from `(key, bytes)` pairs, a key given twice kept twice for the reader to refuse."""+ pairs = [(key, _JsonObject(pairs=[(BASE64_KEY, base64.b64encode(data).decode("ascii"))])) for key, data in entries]+ return _serialized(_JsonObject(pairs=pairs))+++def manifest_of(*, tree: RequestTree, label: str) -> MethodsManifest | None:+ """The `METHODS.toml` at a tree's root, parsed, or None when the tree holds none there.++ Raises:+ MethodEntryError: The manifest is not UTF-8 text, or not a valid manifest (a 422 in the input domain).+ """+ manifest_bytes = tree.contents.get(MANIFEST_NAME)+ if manifest_bytes is None:+ return None+ try:+ return parse_methods_toml(manifest_bytes.decode("utf-8"))+ except UnicodeDecodeError as exc:+ msg = f"The {MANIFEST_NAME} of {label} is not UTF-8 text."+ raise MethodEntryError(msg) from exc+ except ManifestError as exc:+ msg = f"The {MANIFEST_NAME} of {label} is not a valid manifest: {exc.message}"+ raise MethodEntryError(msg) from exc+++def is_package_dir_name(name: str) -> bool:+ """Whether `name` can name a package's directory: one tree path segment the membership rule keeps.++ That refuses an empty name, `.` and `..`, a name holding a separator or a character no tree path holds, and a name+ the rule keeps out of every tree, such as a dot directory or `node_modules`. Both readers of a method's shipped+ packages, a `bundle_b64` archive's and a fetched package's, hold each directory under `.mthds/methods/` to it, as+ the name a manifest declaring none gives its package.+ """+ if not name or "/" in name or tree_path_violation(name) is not None:+ return False+ return not excluded_name_violation(f"{name}/{_ANY_FILE_NAME}")+++def dependency_address_of(*, tree: RequestTree, fallback_name: str, label: str) -> str:+ """The address a dependency tree's manifest declares, `address/name`, as a reference names the package.++ A manifest declaring no name takes `fallback_name`, the name of the directory the tree is written as, which is the+ name the engine's discovery gives such a package; it must then be a name a package's directory can take.++ Raises (as a 422 `InvalidMethodFiles`): a tree with no valid `METHODS.toml` at its root, or one whose manifest+ declares no name when `fallback_name` cannot name a package's directory.+ """+ try:+ manifest = manifest_of(tree=tree, label=label)+ except MethodEntryError as exc:+ raise_validation_error(message=exc.message, error_type=ErrorType.INVALID_METHOD_FILES)+ if manifest is None:+ msg = f"{label} holds no {MANIFEST_NAME} at its root, so no reference could find it as a package."+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_METHOD_FILES)+ if manifest.name is not None:+ return f"{manifest.address}/{manifest.name}"+ if not is_package_dir_name(fallback_name):+ msg = (+ f"{label} declares no name in its {MANIFEST_NAME}, so it takes the last segment of its address, "+ f"'{wire_name(fallback_name)}', which cannot name a package's directory: declare the package's name in its {MANIFEST_NAME}."+ )+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_METHOD_FILES)+ return f"{manifest.address}/{fallback_name}"+++def dependency_dir_name(*, address: str) -> str:+ """The directory a dependency tree is written as: the last segment of its address, the name a reference finds it by."""+ return address.rsplit("/", 1)[-1]+++def add_dependency(*, dependencies: dict[str, RequestTree], address: str, tree: RequestTree, label: str) -> None:+ """Add a dependency tree under its address, refusing an address a reference could not tell from one already there.++ A reference matches a package by its address without regard to case, so two addresses differing only in case are+ one identity too.+ """+ for known in dependencies:+ if known.casefold() == address.casefold():+ msg = (+ f"{label} declares the address '{wire_name(address)}', which another dependency of the request declares too: a reference "+ "matches a package by its address without regard to case, so it could not tell them apart."+ )+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_METHOD_FILES)+ dependencies[address] = tree+++async def read_request_method(*, documents: TreeDocuments, blob_store: RunnerBlobStore) -> RequestMethod | None:+ """The method a request sends in `files`, with its `dependencies`, read in full, or None when it sends no `files`.++ Each dependency tree's manifest must declare the address it is keyed by.++ Raises (as problem documents): the refusals of `read_request_tree`, and a 422 `InvalidMethodFiles` for+ `dependencies` without `files` or a dependency keyed by another address than its manifest's.+ """+ if documents.files is None:+ if documents.dependencies:+ msg = "`dependencies` accompany the method's own `files`: send the method's tree in `files`, and its dependencies beside it."+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_METHOD_FILES)+ return None+ dependency_documents = documents.dependencies or {}+ # Every map's census first, so the request's trees together are held to their limits before any blob is read.+ files_census = _census(document=documents.files)+ dependency_censuses = {address: _census(document=dependency_document) for address, dependency_document in dependency_documents.items()}+ ensure_trees_within_request_limits(+ dependency_count=len(dependency_documents),+ total_bytes=sum(census.stated_bytes for census in [files_census, *dependency_censuses.values()] if census is not None),+ label=_METHOD_LABEL,+ )+ tree = await read_request_tree(document=documents.files, blob_store=blob_store, label=FILES_LABEL, census=files_census)+ dependencies: dict[str, RequestTree] = {}+ for address, dependency_document in dependency_documents.items():+ label = f"The dependency '{wire_name(address)}'"+ dependency_tree = await read_request_tree(+ document=dependency_document, blob_store=blob_store, label=label, census=dependency_censuses[address]+ )+ declared = dependency_address_of(tree=dependency_tree, fallback_name=dependency_dir_name(address=address), label=label)+ if declared != address:+ msg = f"{label} is keyed by an address its {MANIFEST_NAME} does not declare: it declares '{declared}'."+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_METHOD_FILES)+ add_dependency(dependencies=dependencies, address=address, tree=dependency_tree, label=label)+ request_method = RequestMethod(tree=tree, dependencies=dependencies)+ ensure_method_within_limits(request_method=request_method, label=_METHOD_LABEL)+ return request_method+++async def read_archive_method(*, bundle_b64: str) -> RequestMethod:+ """The method a `bundle_b64` archive carries, read as a map of its entries, with the packages it ships.++ An entry under `.mthds/methods/<name>/` belongs to the method package `<name>` the archive ships, the layout of the+ local method cache, which becomes a dependency tree keyed by the address its manifest declares; the cache's+ provenance sidecar at the package's root is left out, as no file of the package. Every other entry is+ a file of the method's tree at the path the archive names it by, so any other entry under a dot directory, `.mthds/`+ included, is refused by the tree's rule.++ Raises (as problem documents): the archive's own refusals (`decode_bundle_archive`), the refusals of+ `read_request_tree`, and a 422 `InvalidMethodFiles` for a shipped package with no manifest, or two declaring one address.+ """+ archive_entries = decode_bundle_archive(bundle_b64=bundle_b64)+ own_entries: list[tuple[str, bytes]] = []+ vendored_entries: dict[str, list[tuple[str, bytes]]] = {}+ for name, data in archive_entries:+ package_name, _, package_path = name.removeprefix(VENDORED_METHODS_PREFIX).partition("/")+ # A package directory the membership rule would keep out of the store, a dot directory, is no package: its+ # entries stay the method's own, which the tree's rule refuses.+ if name.startswith(VENDORED_METHODS_PREFIX) and package_path and is_package_dir_name(package_name):+ # The store's provenance sidecar records how the package was installed, and is no file of its tree.+ if package_path != PROVENANCE_FILENAME:+ vendored_entries.setdefault(package_name, []).append((package_path, data))+ else:+ own_entries.append((name, data))+ tree = await read_request_tree(document=_base64_map_document(entries=own_entries), blob_store=None, label=ARCHIVE_LABEL)+ dependencies: dict[str, RequestTree] = {}+ for package_name, package_entries in vendored_entries.items():+ label = f"The package '{VENDORED_METHODS_PREFIX}{package_name}/' of {ARCHIVE_LABEL[0].lower()}{ARCHIVE_LABEL[1:]}"+ dependency_tree = await read_request_tree(document=_base64_map_document(entries=package_entries), blob_store=None, label=label)+ address = dependency_address_of(tree=dependency_tree, fallback_name=package_name, label=label)+ add_dependency(dependencies=dependencies, address=address, tree=dependency_tree, label=label)+ request_method = RequestMethod(tree=tree, dependencies=dependencies)+ ensure_method_within_limits(request_method=request_method, label=ARCHIVE_LABEL)+ return request_method+++def with_resources(*, request_method: RequestMethod, resources: Mapping[str, bytes] | None) -> RequestMethod:+ """The method with the files `resources` sends joined to its tree, each at its path from the tree's root.++ A file the tree already holds with the same bytes changes nothing; other bytes under a path the tree holds, or a path+ that cannot name a file of a tree, are refused naming the path.++ Raises (as problem documents): a 422 `InvalidResourcePath` for two different files under one path or a path no tree+ could hold, and a 422 `MethodTreeTooLarge` for a tree the files take past this runner's limits.+ """+ if not resources:+ return request_method+ contents = dict(request_method.tree.contents)+ for path, data in resources.items():+ held = contents.get(path)+ if held is not None:+ if held != data:+ refuse_two_files_under_one_path(path=path, where="one in `resources` and one in the method's files")+ continue+ violation = tree_path_violation(path)+ if violation is not None or excluded_name_violation(path):+ reason = violation or "the membership rule keeps it out of every method tree"+ msg = f"The file '{wire_name(path)}' sent in `resources` cannot be a file of the method's tree: {reason}."+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_RESOURCE_PATH)+ contents[path] = data+ collisions = path_collisions(contents)+ if collisions:+ msg = f"The files sent in `resources` collide with the method's files: {', '.join(repr(path) for path in collisions)}."+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_RESOURCE_PATH)+ merged = RequestMethod(tree=RequestTree.make_from_contents(contents=contents), dependencies=request_method.dependencies)+ ensure_tree_within_limits(tree=merged.tree, label=_WITH_RESOURCES_LABEL)+ ensure_method_within_limits(request_method=merged, label=_WITH_RESOURCES_LABEL)+ return merged+++def _python_sources(*, tree: RequestTree) -> dict[str, str]:+ """A tree's Python sources by tree path, skipping a file that is not UTF-8, which no interpreter could import."""+ sources: dict[str, str] = {}+ for tree_path, data in tree.contents.items():+ if not tree_path.endswith(PYTHON_SUFFIX):+ continue+ try:+ sources[tree_path] = data.decode("utf-8")+ except UnicodeDecodeError:+ continue+ return sources+++def ensure_python_allowed(*, request_method: RequestMethod, label: str) -> None:+ """Refuse a method shipping Python where it may not run, before anything of it is written.++ On a deployment that is not sandbox-hosted, any `.py` file is refused (403 `CustomCodeRequiresSandbox`): running it+ would import it in-process. On a sandbox-hosted deployment, the method's own tree declaring a Python structure class+ is refused (`MethodStructuresRefusedError`, a 403), naming each file by its tree path; a dependency tree is checked+ once written, by its address (`written_method`).+ """+ if not request_method.has_python:+ return+ if not is_pipe_func_sandbox_hosted():+ msg = f"{label} ships custom Python (.py); running it requires a sandbox-hosted deployment."+ raise_forbidden(message=msg, error_type=ErrorType.CUSTOM_CODE_REQUIRES_SANDBOX)+ ensure_no_structured_content_in_library_sources(sources_by_dir={Path(): _python_sources(tree=request_method.tree)})+++async def record_request_method(*, request_method: RequestMethod, blob_store: RunnerBlobStore) -> None:+ """Hold every tree of a method read in full in the blob store, its files then its listing."""+ await blob_store.record_tree(tree=request_method.tree.tree, contents=request_method.tree.contents)+ for dependency in request_method.dependencies.values():+ await blob_store.record_tree(tree=dependency.tree, contents=dependency.contents)+++def ensure_method_holds_a_bundle(*, request_method: RequestMethod, label: str) -> None:+ """Refuse a method whose own tree holds no `.mthds` bundle, with a 422 `InvalidBundle`."""+ if not request_method.tree.bundle_paths:+ msg = f"{label} holds no .mthds bundle of its own."+ if request_method.dependencies:+ msg += " The packages it carries are dependencies: the method's own bundles go in its own tree."+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)+++def select_tree_entry(*, tree: RequestTree, method_label: str) -> MethodEntry:+ """The pipe a run of a method tree starts at when it names none, by the MTHDS protocol's rule (`select_entry`).++ Each bundle is parsed to read what it declares, its items naming the bundle by its tree path.++ Args:+ tree: The method's tree, read in full.+ method_label: How the entry rule's messages name the method, as in "The method <label> holds no …".++ Raises:+ ValidateBundleError: A bundle of the tree does not parse.+ MethodEntryError: The tree has no single pipe to start at, or a bundle or the manifest cannot be read.+ """+ candidates: list[EntryCandidate] = []+ for tree_path in tree.bundle_paths:+ try:+ content = tree.contents[tree_path].decode("utf-8")+ except UnicodeDecodeError as exc:+ msg = f"The bundle '{tree_path}' of the method {method_label} is not UTF-8 text."+ raise MethodEntryError(msg) from exc+ with translate_to_validate_bundle_error():+ blueprint = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=content, mthds_source=tree_path)+ candidates.append(EntryCandidate(tree_path=tree_path, blueprint=blueprint))+ manifest = manifest_of(tree=tree, label=f"the method {method_label}")+ return select_entry(candidates=candidates, manifest=manifest, method_label=method_label)+++class WrittenMethod(NamedTuple):+ """A method written out and held as written, for the load: its tree's directory, and its dependencies' directories."""++ method_dir: Path+ methods_dirs: list[Path] | None+ tree: MethodTree+++@contextmanager+def written_method(*, request_method: RequestMethod) -> Generator[WrittenMethod, None, None]:+ """Write a method's trees into a temporary directory, hold each as written, and remove them all on the way out.++ The method's own tree is written as one directory, the library directory of the load; each dependency tree as+ `<name>/` in a methods directory of its own, the layout the engine's `methods_dirs` take, so two dependencies sharing+ a name never meet. On a sandbox-hosted deployment, a dependency tree declaring a Python structure class is refused,+ naming it by its address, as a fetched package is (`MethodStructuresRefusedError`, a 403).+ """+ with ExitStack() as stack:+ temp_dir = Path(tempfile.mkdtemp(prefix="pipelex-method-"))+ stack.callback(shutil.rmtree, temp_dir, ignore_errors=True)+ method_dir = temp_dir / "tree"+ written = write_method_tree(contents=request_method.tree.contents, destination=method_dir, expected_digest=request_method.tree.tree.digest)+ stack.enter_context(WRITTEN_METHOD_TREES.holding(tree=written))+ methods_dirs: list[Path] = []+ for index_dependency, (address, dependency) in enumerate(request_method.dependencies.items()):+ methods_dir = temp_dir / f"methods-{index_dependency}"+ dependency_dir = methods_dir / dependency_dir_name(address=address)+ dependency_written = write_method_tree(contents=dependency.contents, destination=dependency_dir, expected_digest=dependency.tree.digest)+ stack.enter_context(WRITTEN_METHOD_TREES.holding(tree=dependency_written))+ if dependency.has_python and is_pipe_func_sandbox_hosted():+ ensure_no_structured_content_python(package_dir=dependency_dir, package_address=address)+ methods_dirs.append(methods_dir)+ yield WrittenMethod(method_dir=method_dir, methods_dirs=methods_dirs or None, tree=written)diff --git a/api/pipelex_api/openapi_responses.py b/api/pipelex_api/openapi_responses.pyindex 136534754..9ca2c46cc 100644--- a/api/pipelex_api/openapi_responses.py+++ b/api/pipelex_api/openapi_responses.py@@ -167,6 +167,12 @@ PROBLEM_404_METHOD_PACKAGE: dict[str, Any] = _problem("requested address by manifest identity. The message lists the packages the repository does contain.",)+PROBLEM_403_BLOB_UPLOAD: dict[str, Any] = _problem(+ "`BlobUploadNotAuthorized` — the upload does not carry the signed query an upload target of `POST /v1/blobs/missing` "+ "names, its signature does not hold for this hash and scope, or it has expired. Ask `POST /v1/blobs/missing` again "+ "for a fresh target.",+)+PROBLEM_501_METHOD_REF: dict[str, Any] = _problem("`MethodRefNotSupported` — the request selected its closure by a **registry-form** `method_ref` (not a ""`github.com/...` address), and no server-side method registry resolves those yet. Use an address-form reference "diff --git a/api/pipelex_api/routes/blob_upload.py b/api/pipelex_api/routes/blob_upload.pynew file mode 100644index 000000000..978b23071--- /dev/null+++ b/api/pipelex_api/routes/blob_upload.py@@ -0,0 +1,71 @@+"""`PUT /v1/blobs/{sha256}`: one blob's bytes, uploaded to the target `POST /v1/blobs/missing` named (MTHDS Protocol).++This router mounts OUTSIDE the authenticated composite `/v1` router, like `/v1/version`: the upload is authorized by+the signed query of its target (`pipelex_api.blob_upload`), which only a caller who passed the deployment's+authentication on `POST /v1/blobs/missing` holds, so a sender uploads with exactly the headers the target names. The+bytes are written under the scope the signature binds, whole: a blob is never changed, so uploading bytes the runner+already holds at their size succeeds and changes nothing, and only bytes held short, which a storage that does not+publish writes whole could leave, are written over. Only `blobs/` is ever written here, never a tree's listing under `trees/`,+so an uploaded file that happens to read as a listing proves nothing.+"""++from typing import Annotated++from fastapi import APIRouter, Path, Query, Request, Response+from mthds.method_tree.digest import sha256_hex++from pipelex_api.blob_store import RunnerBlobStore+from pipelex_api.blob_upload import EXPIRES_PARAM, SCOPE_PARAM, SIGNATURE_PARAM, is_upload_authorized+from pipelex_api.error_types import ErrorType+from pipelex_api.errors import raise_forbidden, raise_validation_error+from pipelex_api.limits import MAX_METHOD_TREE_FILE_BYTES+from pipelex_api.openapi_responses import PROBLEM_403_BLOB_UPLOAD, PROBLEM_413, PROBLEM_422, PROBLEM_500++router = APIRouter(tags=["blobs"])++SHA256_PATTERN = r"^[0-9a-f]{64}$"+++ "/blobs/{sha256}",+ status_code=204,+ response_class=Response,+ summary="Upload one blob's bytes to the target POST /blobs/missing named.",+ # Mounted outside the composite `/v1` router, so it inherits none of its shared problem responses: it declares+ # the statuses it can produce itself, and never answers 401.+ responses={403: PROBLEM_403_BLOB_UPLOAD, 413: PROBLEM_413, 422: PROBLEM_422, 500: PROBLEM_500},+ openapi_extra={+ "x-mthds-protocol": True,+ "requestBody": {+ "required": True,+ "content": {"application/octet-stream": {"schema": {"type": "string", "format": "binary"}}},+ },+ },+)+async def put_blob(+ request: Request,+ sha256: Annotated[str, Path(pattern=SHA256_PATTERN, description="The SHA-256 of the body, in lowercase hexadecimal.")],+ scope: Annotated[str | None, Query(alias=SCOPE_PARAM, description="The blob store's scope, as the signed target names it.")] = None,+ expires: Annotated[str | None, Query(alias=EXPIRES_PARAM, description="When the target expires, in seconds since the epoch.")] = None,+ signature: Annotated[str | None, Query(alias=SIGNATURE_PARAM, description="The target's signature.")] = None,+) -> Response:+ """Store one blob's bytes under its SHA-256, as the upload target `POST /v1/blobs/missing` named (MTHDS Protocol).++ The body is the blob's bytes and nothing else. A target whose signature does not hold, or that has expired, is a+ 403 `BlobUploadNotAuthorized`, as is a query that is not one this runner signs, such as an expiry that is not a+ decimal integer; a body past `MAX_METHOD_TREE_FILE_KIB` a 422 `MethodTreeTooLarge`; and a body whose SHA-256 is not+ the path's a 422 `BlobMismatch`. A blob is never changed: uploading bytes the runner already holds answers 204 and+ changes nothing.+ """+ if not is_upload_authorized(scope=scope, sha256=sha256, expires=expires, signature=signature) or scope is None:+ msg = "This upload carries no valid target: ask POST /v1/blobs/missing for one, and upload to the URL it names before it expires."+ raise_forbidden(message=msg, error_type=ErrorType.BLOB_UPLOAD_NOT_AUTHORIZED)+ body = await request.body()+ if len(body) > MAX_METHOD_TREE_FILE_BYTES:+ msg = f"The blob holds {len(body)} bytes, past this runner's limit of {MAX_METHOD_TREE_FILE_BYTES // 1024} KiB for one file of a method tree."+ raise_validation_error(message=msg, error_type=ErrorType.METHOD_TREE_TOO_LARGE)+ if sha256_hex(body) != sha256:+ msg = f"The body's SHA-256 is not {sha256}, the hash the path names: nothing was stored."+ raise_validation_error(message=msg, error_type=ErrorType.BLOB_MISMATCH)+ await RunnerBlobStore.make_for_scope(scope=scope).store_blob(data=body)+ return Response(status_code=204)diff --git a/api/pipelex_api/routes/pipelex/__init__.py b/api/pipelex_api/routes/pipelex/__init__.pyindex 6c0426ba4..55e446ed9 100644--- a/api/pipelex_api/routes/pipelex/__init__.py+++ b/api/pipelex_api/routes/pipelex/__init__.py@@ -1,6 +1,7 @@from fastapi import APIRouterfrom .agent import router as agent_router+from .blobs import router as blobs_routerfrom .codegen import router as codegen_routerfrom .pipe_io import router as pipe_io_routerfrom .pipeline import router as pipeline_router@@ -17,3 +18,4 @@ router.include_router(pipe_io_router)router.include_router(codegen_router)router.include_router(tools_router)router.include_router(agent_router)+router.include_router(blobs_router)diff --git a/api/pipelex_api/routes/pipelex/blobs.py b/api/pipelex_api/routes/pipelex/blobs.pynew file mode 100644index 000000000..4222aaa89--- /dev/null+++ b/api/pipelex_api/routes/pipelex/blobs.py@@ -0,0 +1,143 @@+"""`POST /v1/blobs/missing`: which of a sender's blobs this runner lacks, and where to upload each (MTHDS Protocol).++A sender about to send a method by hash first asks which of its files' bytes this runner holds, then uploads only the+missing ones to the targets the answer names (`pipelex_api.routes.blob_upload`), then sends the request with hash+entries. The blobs live in the runner's blob store (`pipelex_api.blob_store`), under the caller's scope: the request's+`read_scope` query parameter when a multi-tenant host sends one for the organization it serves, else the caller's own+id, the scope a run's hash entries are resolved in.+"""++from typing import Annotated++from fastapi import APIRouter, Query, Request+from pipelex.system.storage_scope import validate_read_scope+from pydantic import BaseModel, ConfigDict, Field, StrictInt++from pipelex_api.blob_store import RunnerBlobStore+from pipelex_api.blob_upload import upload_target_url+from pipelex_api.error_types import ErrorType+from pipelex_api.errors import raise_validation_error+from pipelex_api.json_body import JsonBodyRoute+from pipelex_api.method_tree_request import ensure_within_tree_limits+from pipelex_api.routes.pipelex.pipeline import resolve_blob_scope++router = APIRouter(tags=["blobs"], route_class=JsonBodyRoute)++SHA256_PATTERN = r"^[0-9a-f]{64}$"+_OCTET_STREAM = "application/octet-stream"+_SIZE_DESCRIPTION = (+ "The length of the bytes, a JSON integer written without a fraction or an exponent. A size written 13.0 is refused, "+ "although JSON Schema's integer type admits it."+)+++class BlobRef(BaseModel):+ """A blob, named by its SHA-256 in lowercase hexadecimal and its length in bytes."""++ model_config = ConfigDict(extra="forbid")++ sha256: str = Field(pattern=SHA256_PATTERN, description="The SHA-256 of the bytes, in lowercase hexadecimal.")+ size: StrictInt = Field(ge=0, description=_SIZE_DESCRIPTION)+++class MissingBlobsRequest(BaseModel):+ """The blobs a sender is about to send by hash, and optionally the tree they make up."""++ model_config = ConfigDict(extra="allow")++ blobs: list[BlobRef] = Field(description="The blobs to check, each hash listed once.")+ digest: str | None = Field(+ default=None, pattern=r"^sha256:[0-9a-f]{64}$", description="The tree digest of the method tree the blobs make up (optional)."+ )+++class UploadTarget(BaseModel):+ """Where to upload one blob: send its bytes as the whole body of a request with this method to this URL, with exactly these headers."""++ method: str = Field(description="The HTTP method, such as PUT.")+ url: str = Field(description="Where to send the bytes: this runner's own `PUT /v1/blobs/{sha256}`, with a signed query.")+ headers: dict[str, str] = Field(description="The headers to send, exactly.")+++class MissingBlob(BaseModel):+ """A blob the runner lacks, with where to upload it."""++ sha256: str = Field(pattern=SHA256_PATTERN)+ size: int = Field(ge=0, description=_SIZE_DESCRIPTION)+ upload: UploadTarget+++class MissingBlobsResult(BaseModel):+ """The blobs of the request the runner does not hold."""++ missing: list[MissingBlob]+++def _unique_blobs(*, blobs: list[BlobRef]) -> dict[str, int]:+ """Each hash of the request once, with its size; a hash given twice with different sizes is a 422."""+ sizes: dict[str, int] = {}+ for blob in blobs:+ known = sizes.get(blob.sha256)+ if known is not None and known != blob.size:+ msg = f"The blob {blob.sha256} is listed twice with different sizes, {known} and {blob.size}: one hash names one set of bytes."+ raise_validation_error(message=msg, error_type=ErrorType.BLOB_MISMATCH)+ sizes[blob.sha256] = blob.size+ return sizes+++ "/blobs/missing",+ summary="Which of these blobs the runner lacks, and where to upload each.",+ openapi_extra={"x-mthds-protocol": True},+)+async def find_missing_blobs(+ request: Request,+ request_data: MissingBlobsRequest,+ read_scope: Annotated[+ str | None,+ Query(+ description=(+ "PIPELEX-API EXTENSION — the blob store's scope, as a run's `read_scope`: a multi-tenant host sends the one "+ "of the organization it serves. Omit it and the caller's own id is the scope."+ )+ ),+ ] = None,+) -> MissingBlobsResult:+ """List every blob of the request this runner does not hold, each with a signed upload target (MTHDS Protocol).++ A blob it holds at the stated size is left out, and an empty list means there is nothing to upload; bytes held+ under a hash at another size are listed, for the upload to write over. When the request names the tree the blobs+ make up by its `digest`, this runner has recorded that tree as held in full, which it does only after reading every+ file of a tree a request sent, and every listed blob is a file of that tree, the answer is an empty list at once.+ A hash given twice with different sizes is a 422 `BlobMismatch`, and a blob or a set of blobs past this runner's+ tree limits a 422 `MethodTreeTooLarge`, both refused before anything is uploaded. Each upload target is this runner's own+ `PUT /v1/blobs/{sha256}`, at `PUBLIC_BASE_URL` when the deployment sets it, its query signed for that hash and+ the caller's scope, valid for `BLOB_UPLOAD_URL_TTL_SECONDS`, to send with exactly the headers it names.+ """+ if read_scope is not None:+ try:+ validate_read_scope(value=read_scope)+ except ValueError as exc:+ raise_validation_error(message=str(exc), error_type=ErrorType.INVALID_READ_SCOPE)+ sizes = _unique_blobs(blobs=request_data.blobs)+ largest_sha256 = max(sizes, key=lambda sha256: sizes[sha256], default=None)+ ensure_within_tree_limits(+ file_count=len(sizes),+ total_bytes=sum(sizes.values()),+ largest_path=largest_sha256,+ largest_bytes=sizes[largest_sha256] if largest_sha256 is not None else 0,+ label="The blobs of this request",+ )+ scope = resolve_blob_scope(request, read_scope=read_scope)+ blob_store = RunnerBlobStore.make_for_scope(scope=scope)+ if request_data.digest is not None:+ tree_hashes = await blob_store.tree_hashes(tree_digest=request_data.digest)+ # The digest answers alone only for blobs that are files of the tree it names, which this runner holds in full.+ if tree_hashes is not None and tree_hashes.issuperset(sizes):+ return MissingBlobsResult(missing=[])+ missing: list[MissingBlob] = []+ for sha256 in await blob_store.missing_blobs(sizes=sizes):+ url = upload_target_url(request=request, scope=scope, sha256=sha256)+ upload = UploadTarget(method="PUT", url=url, headers={"content-type": _OCTET_STREAM})+ missing.append(MissingBlob(sha256=sha256, size=sizes[sha256], upload=upload))+ return MissingBlobsResult(missing=missing)diff --git a/api/pipelex_api/routes/pipelex/codegen.py b/api/pipelex_api/routes/pipelex/codegen.pyindex 7a54e0e84..115e92cfc 100644--- a/api/pipelex_api/routes/pipelex/codegen.py+++ b/api/pipelex_api/routes/pipelex/codegen.py@@ -141,15 +141,15 @@ async def codegen_mthds(request_data: CodegenRequest) -> JSONResponse:with CodeGenerationReport(artifact=UsageStatsCodegenArtifact.TYPES, target=usage_stats_codegen_target(target=request_data.target)) as generation_report:- return _generate_codegen_response(request_data=request_data, generation_report=generation_report)+ return await _generate_codegen_response(request_data=request_data, generation_report=generation_report)-def _generate_codegen_response(*, request_data: CodegenRequest, generation_report: CodeGenerationReport) -> JSONResponse:+async def _generate_codegen_response(*, request_data: CodegenRequest, generation_report: CodeGenerationReport) -> JSONResponse:"""The body of `codegen_mthds`, timed by `generation_report`, which an invalid verdict declares failed."""try:# `types` is concept-set-wide: no pipe is selected, so the manifest's `main_pipe` beside the crate is unused.# A future per-pipe kind takes it from the `ResolvedClosure` and hands it to `resolve_requested_pipe`.- crate = resolve_requested_crate(request_data).crate+ crate = (await resolve_requested_crate(request_data)).crateexcept ValidateBundleError as validate_error:generation_report.declare_failed()return invalid_crate_report_response(validate_error.to_error_report())diff --git a/api/pipelex_api/routes/pipelex/crate_ops.py b/api/pipelex_api/routes/pipelex/crate_ops.pyindex 061173d18..b21c6f26b 100644--- a/api/pipelex_api/routes/pipelex/crate_ops.py+++ b/api/pipelex_api/routes/pipelex/crate_ops.py@@ -12,6 +12,7 @@ The invalid arm here is the *crate* verdict: it deliberately omits `/validate`'srides that same static core: it reads the selected pipe's *declared* IO with no dry-run sweep."""+import asynciofrom enum import StrEnumfrom typing import Literal, NamedTuple, NoReturn@@ -83,7 +84,7 @@ class SelectedFiles(NamedTuple):"""The fetched package manifest's `main_pipe` (a bare pipe code). Always None for inline `files[]`, which carry no manifest."""-def selected_files(request_data: MthdsFilesRequest) -> SelectedFiles:+async def selected_files(request_data: MthdsFilesRequest) -> SelectedFiles:"""The files the closure selector names: inline `files[]`, or an address `method_ref`'s package.Every route on the `files[]` envelope reaches it through `resolve_requested_crate`, so they all@@ -105,7 +106,8 @@ def selected_files(request_data: MthdsFilesRequest) -> SelectedFiles:"""if request_data.method_ref is not None:if looks_like_method_ref(request_data.method_ref):- fetched = fetch_method_mthds_files(request_data.method_ref)+ # The fetch runs git and reads the package from disk, so it runs off the event loop, as a run's does.+ fetched = await asyncio.to_thread(fetch_method_mthds_files, request_data.method_ref)return SelectedFiles(files=fetched.files, manifest_main_pipe=fetched.main_pipe)raise_not_implemented("Registry-form method_ref resolution is not available on this server: no method registry is wired. "@@ -129,7 +131,7 @@ class ResolvedClosure(NamedTuple):"""The fetched package manifest's `main_pipe`, carried through from `selected_files` (None for inline `files[]`)."""-def resolve_requested_crate(request_data: MthdsFilesRequest) -> ResolvedClosure:+async def resolve_requested_crate(request_data: MthdsFilesRequest) -> ResolvedClosure:"""Resolve the request's closure selector into a normalized library crate.Inherits the engine core's **loaded-on-success contract**: on success the freshly opened@@ -148,7 +150,7 @@ def resolve_requested_crate(request_data: MthdsFilesRequest) -> ResolvedClosure:ApiError: 501 for a registry-form `method_ref` (see `selected_files`).MethodRefError subclasses: address-form fetch/location failures (see `selected_files`)."""- selection = selected_files(request_data)+ selection = await selected_files(request_data)crate = resolve_crate_from_contents(mthds_contents=[item.content for item in selection.files],mthds_sources=[item.source for item in selection.files],diff --git a/api/pipelex_api/routes/pipelex/pipe_io.py b/api/pipelex_api/routes/pipelex/pipe_io.pyindex db945f1fd..bb0c346b1 100644--- a/api/pipelex_api/routes/pipelex/pipe_io.py+++ b/api/pipelex_api/routes/pipelex/pipe_io.py@@ -141,7 +141,7 @@ async def pipe_io(request_data: PipeIORequest) -> JSONResponse:`application/problem+json` via the global handlers."""try:- resolved = resolve_requested_crate(request_data)+ resolved = await resolve_requested_crate(request_data)except ValidateBundleError as validate_error:return invalid_crate_report_response(validate_error.to_error_report())try:diff --git a/api/pipelex_api/routes/pipelex/pipeline.py b/api/pipelex_api/routes/pipelex/pipeline.pyindex 6725eeb76..215dd0f4c 100644--- a/api/pipelex_api/routes/pipelex/pipeline.py+++ b/api/pipelex_api/routes/pipelex/pipeline.py@@ -3,17 +3,16 @@ from __future__ import annotationsimport asyncioimport hashlibimport hmac-from contextlib import ExitStack, contextmanager-from pathlib import Path, PurePosixPath+from contextlib import asynccontextmanager+from pathlib import Pathfrom typing import TYPE_CHECKING, Annotated, Any, NamedTuple, castfrom fastapi import APIRouter, Depends, Requestfrom fastapi.responses import JSONResponsefrom mthds.protocol.exceptions import PipelineRequestError-from pipelex.config import get_config, is_pipe_func_sandbox_hosted+from pipelex.config import get_configfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.libraries.method_requirements import ensure_served_method_requirements_met-from pipelex.methods.structures_check import ensure_no_structured_content_pythonfrom pipelex.pipe_run.delivery_assignment import DeliveryAssignment, StorageTarget, WebhookTargetfrom pipelex.pipe_run.pipe_run_protocol import PipeRunProtocolfrom pipelex.pipeline.exceptions import ValidateBundleError@@ -31,19 +30,25 @@ from pydantic import ValidationErrorfrom typing_extensions import overridefrom pipelex_api.api_config import get_api_config, resolve_orchestration_mode-from pipelex_api.bundle import (- ParsedBundle,- VendoredMethodPackage,- materialize_parsed,- materialized_methods_dir,- parse_bundle,- partition_bundle_entries,-)-from pipelex_api.bundle_resources import decode_bundle_resources, files_its_steps_read+from pipelex_api.blob_store import RunnerBlobStore+from pipelex_api.bundle_resources import decode_bundle_resourcesfrom pipelex_api.error_types import ErrorType-from pipelex_api.errors import raise_bad_request, raise_forbidden, raise_validation_error+from pipelex_api.errors import raise_bad_request, raise_validation_errorfrom pipelex_api.json_body import decode_json_body-from pipelex_api.method_source import fetched_method_source+from pipelex_api.method_source import fetched_method+from pipelex_api.method_tree_request import (+ REQUEST_METHOD_LABEL,+ RequestMethod,+ ensure_method_holds_a_bundle,+ ensure_python_allowed,+ read_archive_method,+ read_request_method,+ record_request_method,+ select_tree_entry,+ tree_documents_of,+ with_resources,+ written_method,+)from pipelex_api.middleware import request_id_offrom pipelex_api.openapi_responses import (PROBLEM_400_START_REQUIRES_ASYNC,@@ -64,7 +69,7 @@ from pipelex_api.schemas.models import ()if TYPE_CHECKING:- from collections.abc import Generator+ from collections.abc import AsyncGeneratorfrom mthds.protocol.pipe_output import VariableMultiplicityfrom mthds.protocol.pipeline_inputs import PipelineInputs@@ -147,6 +152,16 @@ def _resolve_read_scope(request: Request, *, requested: str | None) -> str | Nonreturn user.user_id if user else None+def resolve_blob_scope(request: Request, *, read_scope: str | None) -> str:+ """The scope of the blob store a request's hash entries resolve in: its read scope, else the caller's own id.++ The same scope a run reads under, so a hash entry resolves only in the caller's own blobs, never across tenants:+ a multi-tenant host sends the organization it serves as the read scope, and a deployment that identifies no caller+ is one tenant, `single-tenant`.+ """+ return read_scope or get_request_user_id(request)++class _RunScopes(NamedTuple):storage_scope: strread_scope: str | None@@ -408,6 +423,8 @@ class ApiRunner(PipelexMTHDSProtocol):execution_config=execution_config,library_id=self.library_id,library_dirs=self.library_dirs,+ # A method the request carries as a tree is the caller's, as the base `execute` threads it too.+ library_dirs_are_callers=self.library_dirs_are_callers,pipe_code=pipe_code,mthds_contents=mthds_contents,bundle_uris=self.bundle_uris,@@ -477,11 +494,10 @@ class ApiRunner(PipelexMTHDSProtocol):async def _ensure_served_method_requirements_met(self) -> None:"""Refuse the request's method when its requirements.txt is not met, before anything loads or is dispatched.- This runner's `library_dirs` are the directories it made for the request, holding the method's non-`.mthds`- files, which the load reads as a host's library directories: there a requirements refusal is reported as the- host's fault, so the caller's own is judged here first, naming no path on this host. A directory that cannot be- listed stays the host's fault, as the load reports it. The listing reads and hashes files, so it runs off the- event loop.+ This runner's `library_dirs` are the directories it wrote the request's method tree into, the caller's own,+ whose load reports a requirements refusal as the caller's too; judging it here first refuses the run before+ any library opens or any run is registered, naming no path on this host. The tree is held as written, so its+ listing is the tree the request sent. The listing reads and hashes files, so it runs off the event loop.Raises:MethodRequirementsFileError: The method's requirements.txt holds a line the standard refuses.@@ -497,11 +513,12 @@ class ApiRunner(PipelexMTHDSProtocol):async def validate_verdict(self,*,- mthds_contents: list[str],+ mthds_contents: list[str] | None,mthds_sources: list[str] | None,allow_signatures: bool,requested_orchestration_mode: str | None,graph_pipe_code: str | None,+ method_dir: Path | None = None,) -> PipelexValidationReport | ErrorReport:"""Validate MTHDS bundles, returning the verdict as a value (the route maps it to a 200).@@ -530,8 +547,12 @@ class ApiRunner(PipelexMTHDSProtocol):`graph_pipe_code` is the pipe the best-effort graph is drawn from, resolved the way a runresolves its entry pipe; `None` keeps the runtime's default, the primary blueprint's- `main_pipe`. The route passes a fetched package manifest's `main_pipe`, so the graph shows- the pipe a selector-less run by that address executes.+ `main_pipe`. The route passes the entry pipe of a method tree, so the graph shows the pipe a+ selector-less run of that method executes.++ `method_dir` is the directory a method tree the request carries was written as, by `files` or+ fetched by `method_ref`: the validation loads it whole, with this runner's `methods_dirs`, in+ place of `mthds_contents`, which is then None, and its refusals are the method's verdict."""# Resolve the effective mode FIRST — a per-request override the deployment policy forbids# is refused (403) here, before any validator dispatch / library load. Mirrors start().@@ -552,7 +573,10 @@ class ApiRunner(PipelexMTHDSProtocol):mthds_contents=mthds_contents,mthds_sources=mthds_sources,allow_signatures=allow_signatures,- library_dirs=library_dirs,+ # A method directory is the whole method: no library directory loads beside it.+ library_dirs=None if method_dir is not None else library_dirs,+ method_dir=method_dir,+ methods_dirs=self.methods_dirs if method_dir is not None else None,# A validation is not a run, but its dry runs and its `pipe_dry_run` event are still# done for the caller this runner was built for — the user and groups a run would state.caller_identity=self.caller_identity,@@ -701,7 +725,16 @@ def _coerce_correlation_field(value: Any) -> str | None:return value[:_MAX_CORRELATION_FIELD_LEN]-async def _parse_request(request: Request) -> tuple[RunRequest, PipelineApiExtras]:+class _ParsedRunRequest(NamedTuple):+ """A run request's body as the routes read it: the typed request, its extras, and its raw bytes."""++ run_request: RunRequest+ extras: PipelineApiExtras+ body: bytes+ """The raw body, which a method sent as `files` is read from, keeping what only the JSON text holds."""+++async def _parse_request(request: Request) -> _ParsedRunRequest:"""Parse and validate the request body.Splits the body into:@@ -734,103 +767,7 @@ async def _parse_request(request: Request) -> tuple[RunRequest, PipelineApiExtra# mistakes, not server faults, so they map to a 422 rather than# escaping to the generic-500 fallback.raise_validation_error(message=str(exc))- return run_request, extras---class _BundleRunSource(NamedTuple):- """What a request's run source resolves to, before the routes add its entry pipe and provenance."""-- mthds_contents: list[str] | None- library_dirs: list[str] | None- methods_dirs: list[Path] | None- bundle_files: dict[str, bytes]- """The files a method bundle ships beside its `.mthds` files that their steps read, keyed by the path a step names."""---def _ensure_vendored_packages_hold_no_structures(*, methods_dir: Path, packages: tuple[VendoredMethodPackage, ...]) -> None:- """Refuse a shipped package whose Python declares a structure class, as a fetched package is on this deployment.-- The library load refuses the bundle's own structure classes, but never reads a shipped package's Python, which is- not in a library directory. Each package is named by the address its manifest declares, and each file by its path- inside the package, never by the temporary directory.- """- for package in packages:- if any(relpath.suffix == ".py" for relpath, _ in package.entries):- ensure_no_structured_content_python(package_dir=methods_dir / package.name, package_address=package.full_address)---@contextmanager-def _bundle_run_source(run_request: RunRequest) -> Generator[_BundleRunSource, None, None]:- """Resolve a run's `(mthds_contents, library_dirs, methods_dirs)` from the request.-- A method bundle KEEPS the proven run path rather than replacing it: its own `.mthds`- text travels as `mthds_contents` — so `main_pipe` resolves exactly as it does for a- plain (non-bundle) run — and its other files (custom PipeFunc `.py`,- `requirements.txt`) are materialized into a temp `library_dirs` entry, where the- load path reads them without importing them. This mirrors "a normal run, plus the Python", instead of- handing the engine a bare directory with no `mthds_contents` (which never resolves- `main_pipe`, since that is only derived from `mthds_contents`).-- The method packages the bundle ships under `.mthds/methods/<name>/` belong to neither:- every file of theirs is written into a temp directory of its own, as `<tmp>/<name>/…`,- which the engine searches first for an address-based reference and never writes, so a- shipped package wins over an installed copy and ends with the request. The other `.mthds`- paths are refused with a `422` naming the entry (`partition_bundle_entries`), and so are two- shipped packages declaring the same address and a bundle whose only `.mthds` files are shipped ones.-- No bundle → yields the request's own `mthds_contents` and no directory (the classic path,- unchanged). Each temp dir is created only when it has files, and cleaned up on exit.-- Security gate (decision 5): a bundle that ships custom Python (`.py`), in a shipped package- or not, is only honored on a sandbox-hosted deployment, where the load path reads the bundle's- own source without importing it and refuses, with a 403 `MethodStructuresRefusedError`, a- bundle whose Python declares a structure class; PipeFunc source is captured for the sandbox.- A shipped package's Python is checked the same way, as a fetched package's is, and never runs.- On a non-hosted deployment, running that code would import it in-process — refused with a 403- rather than executing untrusted code.- """- if run_request.bundle_b64 is None and run_request.files is None:- yield _BundleRunSource(mthds_contents=run_request.mthds_contents, library_dirs=None, methods_dirs=None, bundle_files={})- return- # Parse + guard in memory FIRST, then apply the sandbox-hosted gate BEFORE any disk write —- # a bundle destined for a 403 on a non-hosted deployment never touches the filesystem.- parsed = parse_bundle(bundle_b64=run_request.bundle_b64, files=run_request.files)- if parsed.has_python_sources and not is_pipe_func_sandbox_hosted():- msg = "This bundle ships custom Python (.py); running it requires a sandbox-hosted deployment."- raise_forbidden(message=msg, error_type=ErrorType.CUSTOM_CODE_REQUIRES_SANDBOX)- # Split: the bundle's own `.mthds` text → `mthds_contents` (the proven main_pipe path); its other files- # (`.py`, `requirements.txt`) → a temp `library_dirs` entry the load path source-captures; the packages it- # ships → a temp methods directory.- partition = partition_bundle_entries(parsed.entries)- mthds_entries: list[tuple[PurePosixPath, str]] = []- for relpath, content in partition.mthds_entries:- try:- mthds_entries.append((relpath, content.decode("utf-8")))- except UnicodeDecodeError:- raise_validation_error(message=f"Bundle .mthds file '{relpath}' is not valid UTF-8.", error_type=ErrorType.INVALID_BUNDLE)- mthds_contents = [content for _, content in mthds_entries]- if not mthds_contents:- msg = "Method bundle contains no .mthds file."- if partition.vendored_packages:- msg = (- "Method bundle contains no .mthds file of its own: the files under '.mthds/methods/<name>/' are the method "- "packages it ships, and the bundle's own .mthds files go outside '.mthds/'."- )- raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)- # The `.mthds` files load as content strings, so a template file shipped beside one travels with it as a resource.- bundle_files = files_its_steps_read(mthds_entries=mthds_entries, other_entries=partition.library_entries)- with ExitStack() as stack:- library_dirs: list[str] | None = None- if partition.library_entries:- library_bundle = stack.enter_context(materialize_parsed(ParsedBundle(entries=partition.library_entries)))- library_dirs = [str(library_bundle.directory)]- methods_dirs: list[Path] | None = None- if partition.vendored_packages:- methods_dir = stack.enter_context(materialized_methods_dir(partition.vendored_packages))- if is_pipe_func_sandbox_hosted():- _ensure_vendored_packages_hold_no_structures(methods_dir=methods_dir, packages=partition.vendored_packages)- methods_dirs = [methods_dir]- yield _BundleRunSource(mthds_contents=mthds_contents, library_dirs=library_dirs, methods_dirs=methods_dirs, bundle_files=bundle_files)+ return _ParsedRunRequest(run_request=run_request, extras=extras, body=body)class _ResolvedRunSource(NamedTuple):@@ -838,45 +775,90 @@ class _ResolvedRunSource(NamedTuple):mthds_contents: list[str] | Nonelibrary_dirs: list[str] | None+ library_dirs_are_callers: bool+ """Whether `library_dirs` hold the caller's own method, so a refusal while loading them is the caller's (a 422)."""methods_dirs: list[Path] | Nonepipe_code: str | Nonemethod_provenance: MethodProvenance | None- bundle_files: dict[str, bytes]- """The files a method bundle ships that its steps read, which join the request's `resources`."""---@contextmanager-def _run_source(run_request: RunRequest) -> Generator[_ResolvedRunSource, None, None]:- """Resolve the request's run source: a `method_ref` fetch, a bundle, or the inline contents.-- A `method_ref` resolves through `pipelex_api.method_source.fetched_method_source` (fetch → locate →- execution-locus gate → materialize) into exactly the shape a bundle produces — `.mthds` text- as `mthds_contents`, non-`.mthds` files in a temp `library_dirs` entry, the packages the package- ships under its own `.mthds/methods/` in a temp `methods_dirs` entry — so the engine runs- the same proven path. The entry pipe defaults to the fetched manifest's `main_pipe`; a- request `pipe_code` overrides it. Provenance `(address, tag, commit_sha)` rides back so the- routes can put it on the response. The other two forms delegate to `_bundle_run_source`,- unchanged.+ submitted_files: dict[str, bytes] | None+ """The files `resources` sends beside `mthds_contents`; beside a method tree they join the tree instead."""+++async def _request_method_of(*, run_request: RunRequest, body: bytes, blob_store: RunnerBlobStore) -> RequestMethod | None:+ """The method a run request carries as a tree, `files` or `bundle_b64`, read in full, or None when it carries none.++ The files `resources` sends join the tree. Every tree is recorded in the blob store once the Python gate passed.+ """+ if run_request.bundle_b64 is not None and run_request.files is not None:+ raise_validation_error(message="Provide either bundle_b64 or files, not both", error_type=ErrorType.INVALID_BUNDLE)+ request_method: RequestMethod | None+ if run_request.bundle_b64 is not None:+ request_method = await read_archive_method(bundle_b64=run_request.bundle_b64)+ elif run_request.files is not None or run_request.dependencies is not None:+ # Only a body sending a tree is parsed again, keeping its pairs; an inline run never pays a second parse.+ request_method = await read_request_method(documents=tree_documents_of(body=body), blob_store=blob_store)+ else:+ request_method = None+ if request_method is None:+ return None+ request_method = with_resources(request_method=request_method, resources=decode_bundle_resources(resources=run_request.resources))+ ensure_method_holds_a_bundle(request_method=request_method, label="This method")+ ensure_python_allowed(request_method=request_method, label="This method")+ await record_request_method(request_method=request_method, blob_store=blob_store)+ return request_method+++@asynccontextmanager+async def _run_source(*, run_request: RunRequest, body: bytes, blob_scope: str) -> AsyncGenerator[_ResolvedRunSource, None]:+ """Resolve the request's run source: a method tree (`files`, `bundle_b64` or a `method_ref`), or the inline contents.++ A method tree is written into a temporary directory and held as written, then handed to the runner as its one+ library directory, the caller's own, with the dependency trees it carries as `methods_dirs`: the load reads the+ tree whole, its `.mthds` files as the method's bundles and every other file as one a step may read, such as a+ `PipeDocGen` template or a `PipeFunc` module, and names each by its tree path. A refusal while loading it is the+ caller's invalid method, a 422. With no `pipe_code`, the run starts at the protocol's entry pipe+ (`select_tree_entry`): the `main_pipe` of the manifest at the tree's root, else the one `main_pipe` its bundles+ declare. A `method_ref` is fetched and read from its commit's git objects (`pipelex_api.method_source`), its+ provenance riding back so the routes can put it on the response. The directory is removed when the run returns.++ Inline `mthds_contents` keep their path: the contents load as strings, and the files `resources` sends travel as+ the runtime's `submitted_files`.++ Security gate: a method shipping custom Python (`.py`) is only honored on a sandbox-hosted deployment, refused with+ a 403 before anything is written (`ensure_python_allowed`)."""+ method_provenance: MethodProvenance | None = None+ method_label = REQUEST_METHOD_LABELif run_request.method_ref is not None:- with fetched_method_source(run_request.method_ref) as fetched:- yield _ResolvedRunSource(- mthds_contents=fetched.mthds_contents,- library_dirs=fetched.library_dirs,- methods_dirs=fetched.methods_dirs,- pipe_code=run_request.pipe_code or fetched.main_pipe,- method_provenance=fetched.provenance,- bundle_files={},- )- return- with _bundle_run_source(run_request) as bundle_source:+ fetched = await asyncio.to_thread(fetched_method, run_request.method_ref)+ request_method: RequestMethod | None = fetched.request_method+ ensure_method_holds_a_bundle(request_method=fetched.request_method, label=f"Method package {fetched.label}")+ ensure_python_allowed(request_method=fetched.request_method, label=f"Method package {fetched.label}")+ method_provenance = fetched.provenance+ method_label = fetched.label+ else:+ request_method = await _request_method_of(run_request=run_request, body=body, blob_store=RunnerBlobStore.make_for_scope(scope=blob_scope))+ if request_method is None:yield _ResolvedRunSource(- mthds_contents=bundle_source.mthds_contents,- library_dirs=bundle_source.library_dirs,- methods_dirs=bundle_source.methods_dirs,+ mthds_contents=run_request.mthds_contents,+ library_dirs=None,+ library_dirs_are_callers=False,+ methods_dirs=None,pipe_code=run_request.pipe_code,method_provenance=None,- bundle_files=bundle_source.bundle_files,+ submitted_files=decode_bundle_resources(resources=run_request.resources),+ )+ return+ pipe_code = run_request.pipe_code or select_tree_entry(tree=request_method.tree, method_label=method_label).pipe_ref+ with written_method(request_method=request_method) as written:+ yield _ResolvedRunSource(+ mthds_contents=None,+ library_dirs=[str(written.method_dir)],+ library_dirs_are_callers=True,+ methods_dirs=written.methods_dirs,+ pipe_code=pipe_code,+ method_provenance=method_provenance,+ submitted_files=None,)@@ -919,17 +901,19 @@ async def execute(request: Request) -> JSONResponse:failures propagate untouched: the global `PipelexError` handler in `pipelex_api.exception_handlers`turns them into an RFC 7807 problem response."""- run_request, extras = await _parse_request(request)+ run_request, extras, body = await _parse_request(request)scopes = _resolve_run_scopes(request, extras=extras)- with _run_source(run_request) as source:+ blob_scope = resolve_blob_scope(request, read_scope=scopes.read_scope)+ async with _run_source(run_request=run_request, body=body, blob_scope=blob_scope) as source:runner = ApiRunner(user_id=get_request_user_id(request),storage_scope=scopes.storage_scope,read_scope=scopes.read_scope,extras=extras.analytics_groups,library_dirs=source.library_dirs,+ library_dirs_are_callers=source.library_dirs_are_callers,methods_dirs=source.methods_dirs,- submitted_files=decode_bundle_resources(resources=run_request.resources, bundle_files=source.bundle_files),+ submitted_files=source.submitted_files,)response = await runner.execute(pipe_code=source.pipe_code,@@ -989,7 +973,7 @@ async def execute(request: Request) -> JSONResponse:)async def start(request: Request,- parsed: Annotated[tuple[RunRequest, PipelineApiExtras], Depends(_parse_request)],+ parsed: Annotated[_ParsedRunRequest, Depends(_parse_request)],) -> PipelexApiStartResponse:"""Start a method run and return its pipeline_run_id with a 202 ack (MTHDS Protocol `POST /start`).@@ -1008,22 +992,27 @@ async def start((`StartRequiresAsyncOrchestration`) — use `/execute` — rather than silently blocking and acking.The completion callback (`callback_urls` / storage delivery) fires on the async path."""- run_request, extras = parsed+ run_request, extras, body = parsedscopes = _resolve_run_scopes(request, extras=extras)- # The run source is materialized only for the synchronous setup phase: `start` builds the+ blob_scope = resolve_blob_scope(request, read_scope=scopes.read_scope)+ # The run source is written out only for the synchronous setup phase: `start` builds the# PipeJob before it enqueues, its crate naming each method tree by digest, so the temp dir (a- # bundle's, or a fetched package's) is no longer needed once `start` returns — cleanup on context+ # tree's, sent or fetched) is no longer needed once `start` returns — cleanup on context# exit is safe for the async path. The worker reads a tree's files through its own method file- # store, which the deployment fills, never from this directory.- with _run_source(run_request) as source:+ # store, never from this directory: the orchestrator copies every tree the job's crate names+ # into that store before its `start` returns (`OrchestratorProtocol`), while this process's+ # library still serves them, and no orchestrator keeping a job running after `start` returned+ # may rely on this process's store.+ async with _run_source(run_request=run_request, body=body, blob_scope=blob_scope) as source:runner = ApiRunner(user_id=get_request_user_id(request),storage_scope=scopes.storage_scope,read_scope=scopes.read_scope,extras=extras.analytics_groups,library_dirs=source.library_dirs,+ library_dirs_are_callers=source.library_dirs_are_callers,methods_dirs=source.methods_dirs,- submitted_files=decode_bundle_resources(resources=run_request.resources, bundle_files=source.bundle_files),+ submitted_files=source.submitted_files,)start_result = await runner.start(pipe_code=source.pipe_code,diff --git a/api/pipelex_api/routes/pipelex/resolve.py b/api/pipelex_api/routes/pipelex/resolve.pyindex 22fb7690b..fa99d6ae1 100644--- a/api/pipelex_api/routes/pipelex/resolve.py+++ b/api/pipelex_api/routes/pipelex/resolve.py@@ -71,7 +71,7 @@ async def resolve_mthds(request_data: MthdsFilesRequest) -> JSONResponse:"""try:# Whole-closure route: no pipe is selected here, so the manifest's `main_pipe` beside the crate is unused.- crate = resolve_requested_crate(request_data).crate+ crate = (await resolve_requested_crate(request_data)).crateexcept ValidateBundleError as validate_error:return invalid_crate_report_response(validate_error.to_error_report())try:diff --git a/api/pipelex_api/routes/pipelex/validate.py b/api/pipelex_api/routes/pipelex/validate.pyindex f6b9f1fe6..d9ec01f58 100644--- a/api/pipelex_api/routes/pipelex/validate.py+++ b/api/pipelex_api/routes/pipelex/validate.py@@ -1,24 +1,40 @@+import asynciofrom enum import StrEnumfrom typing import Annotated, Literal, Self, Unionfrom fastapi import APIRouter, Requestfrom fastapi.responses import JSONResponsefrom pipelex.base_exceptions import ErrorReport, ValidationErrorItem-from pipelex.core.qualified_ref import QualifiedReffrom pipelex.pipe_machinery.pipe_factory import PipeFactory+from pipelex.pipeline.exceptions import MethodEntryError, ValidateBundleErrorfrom pipelex.pipeline.input_form import PipeInputFormDescriptor, PipeOutputFormDescriptor+from pipelex.pipeline.method_entry import MethodEntryfrom pipelex.pipeline.validation_render import format_validate_markdown, render_invalid_validation_markdownfrom pipelex.pipeline.validation_report import PipelexValidationReport+from pipelex.system.storage_scope import validate_read_scopefrom pipelex.tools.typing.pydantic_utils import empty_list_factory_of-from pydantic import BaseModel, Field, model_validator+from pydantic import BaseModel, Field, field_validator, model_validator+from pipelex_api.blob_store import RunnerBlobStorefrom pipelex_api.bundle_resources import decode_bundle_resourcesfrom pipelex_api.errors import raise_validation_errorfrom pipelex_api.exception_handlers import problem_response_from_error_reportfrom pipelex_api.json_body import JsonBodyRoute-from pipelex_api.method_source import fetched_method_source+from pipelex_api.method_source import fetched_method+from pipelex_api.method_tree_request import (+ REQUEST_METHOD_LABEL,+ RequestMethod,+ ensure_method_holds_a_bundle,+ ensure_python_allowed,+ read_request_method,+ record_request_method,+ select_tree_entry,+ tree_documents_of,+ with_resources,+ written_method,+)from pipelex_api.openapi_responses import PROBLEM_403_RUN_POLICY, PROBLEM_404_METHOD_PACKAGE-from pipelex_api.routes.pipelex.pipeline import ApiRunner, get_request_user_id+from pipelex_api.routes.pipelex.pipeline import ApiRunner, get_request_user_id, resolve_blob_scopefrom pipelex_api.schemas.models import RESOURCES_FIELD_DESCRIPTION, CallerAnalyticsGroupsMixin, MthdsContentsRequestrouter = APIRouter(tags=["validate"], route_class=JsonBodyRoute)@@ -119,12 +135,28 @@ class ValidateRequest(MthdsContentsRequest, CallerAnalyticsGroupsMixin):),)+ read_scope: str | None = Field(+ default=None,+ description=(+ "PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the scope of the blob store a hash entry of `files` "+ "resolves in, as a run's `read_scope`: a multi-tenant host sends the one of the organization it serves. Omit it "+ "and the caller's own id is the scope."+ ),+ )++ @field_validator("read_scope")+ @classmethod+ def _validate_read_scope(cls, value: str | None) -> str | None:+ if value is None:+ return None+ return validate_read_scope(value=value)+@model_validator(mode="after")def _sources_match_contents(self) -> Self:- # `mthds_sources` labels INLINE contents; a `method_ref`'s sources are the package's own- # file names, so pairing the two is a request-shape bug → 422.- if self.mthds_sources is not None and self.method_ref is not None:- msg = "mthds_sources accompanies inline mthds_contents; a method_ref's sources are the package's own file names"+ # `mthds_sources` labels INLINE contents; a method tree's files are named by their tree paths, so pairing the+ # two is a request-shape bug → 422.+ if self.mthds_sources is not None and self.mthds_contents is None:+ msg = "mthds_sources accompanies inline mthds_contents; the files of a method tree are named by their tree paths"raise ValueError(msg)# `resources` are the files the inline contents' steps read; a `method_ref`'s files are the package's own.if self.resources and self.method_ref is not None:@@ -139,65 +171,39 @@ class ValidateRequest(MthdsContentsRequest, CallerAnalyticsGroupsMixin):return self-def _qualify_manifest_main_pipe(main_pipe: str, *, pipe_refs: list[str]) -> str | None:- """Qualify a manifest's bare `main_pipe` against the pipes the verdict actually saw.-- A `METHODS.toml` `main_pipe` is a bare pipe code, and the runner resolves it exactly as it- resolves a human-supplied selector (`get_optional_entry_pipe`): an already-qualified ref matches- outright, a bare code is matched across the library's own domains, and an ambiguous bare code- resolves to nothing rather than picking a winner. That rule is mirrored here against- `pipe_io_contracts`' keys — the verdict's own complete, qualified pipe set — because the- validate path holds no loaded library to ask (the Temporal-dispatched arm never loaded one in- this process at all).-- Cross-package keys (`alias->domain.code`) are excluded, as they are in the engine: an installed- dependency must never make a host package's own entry pipe ambiguous.+def _effective_default_pipe_ref(report: PipelexValidationReport) -> str | None:+ """The qualified ref of the pipe a selector-less run of inline contents would execute.- Returns:- The qualified ref, or None when the closure declares no such pipe or declares it in several- domains — both cases in which a selector-less run by this address would fail to resolve it.+ The run routes' precedence for inline contents, minus the request selector `/validate` does not have: the+ closure's own declaration, the batch's primary blueprint's `main_pipe` (`select_primary_blueprint`), qualified by+ its domain. The report's `bundle_blueprint` IS that first declaring blueprint (`build_validation_report` selects it+ through the same `select_primary_blueprint`), so qualifying its `main_pipe` reproduces the run's own choice. A+ method tree's default is its entry pipe instead, by the protocol's rule (`select_tree_entry`)."""- own_refs = [ref for ref in pipe_refs if not QualifiedRef.has_cross_package_prefix(ref)]- if main_pipe in own_refs:- return main_pipe- matches = sorted({ref for ref in own_refs if ref.rpartition(".")[2] == main_pipe})- if len(matches) == 1:- return matches[0]- return None---def _effective_default_pipe_ref(report: PipelexValidationReport, *, manifest_main_pipe: str | None) -> str | None:- """The qualified ref of the pipe a selector-less run of THIS request would execute.-- The run routes' precedence, minus the request selector `/validate` does not have- (`pipeline.py`: `pipe_code or fetched.main_pipe`, then the batch's primary blueprint via- `select_primary_blueprint`): a fetched package's manifest `main_pipe` wins, and only when there- is none does the closure's own declaration decide. The per-pipe tooling route applies that same- chain in `crate_ops.resolve_requested_pipe`, so a caller reading this field and a caller omitting- `pipe_ref` on `/pipe-io` are told about the same pipe wherever `/pipe-io` defaults at all — see- the one case below where it refuses to.-- It deliberately states the RUN default, not the per-pipe route's stricter one: a closure whose- domains each declare a `main_pipe` is refused by `/pipe-io` but runs happily — `execute` and- `start` take the first declaring blueprint — so reporting `null` for it would make a consumer- refuse to prepare a method the server would run. The report's `bundle_blueprint` IS that first- declaring blueprint (`build_validation_report` selects it through the same- `select_primary_blueprint`), so qualifying its `main_pipe` reproduces the run's own choice.-- Inline `mthds_contents` carry no manifest, so for them the chain is exactly the closure's- declaration.- """- if manifest_main_pipe is not None:- # A manifest entry that does not resolve yields None rather than falling through to the- # closure's declaration: the run would pass the manifest's code and fail, so naming the- # closure's pipe here would report a pipe no selector-less run of this address executes.- return _qualify_manifest_main_pipe(manifest_main_pipe, pipe_refs=list(report.pipe_io_contracts))blueprint = report.bundle_blueprintif not blueprint.main_pipe:return Nonereturn PipeFactory.make_pipe_ref_with_domain(domain_code=blueprint.domain, pipe_code=blueprint.main_pipe)+def _bundle_texts(*, request_method: RequestMethod) -> list[str]:+ """The texts of a method tree's `.mthds` bundles, in tree path order, as the valid verdict echoes them."""+ tree = request_method.tree+ return [tree.contents[tree_path].decode("utf-8", errors="replace") for tree_path in tree.bundle_paths]+++def _optional_tree_entry(*, request_method: RequestMethod, method_label: str) -> MethodEntry | None:+ """The method's entry pipe, or None when it has none or a bundle does not parse, for the verdict to say why.++ A validation needs no entry pipe: a method without one validates, and its verdict carries no graph and no+ `default_pipe_ref`.+ """+ try:+ return select_tree_entry(tree=request_method.tree, method_label=method_label)+ except (ValidateBundleError, MethodEntryError):+ return None++class ValidReport(PipelexValidationReport):"""The 200 **valid** arm: the canonical `PipelexValidationReport` plus this server's wire-only extras.@@ -216,18 +222,23 @@ class ValidReport(PipelexValidationReport):tool signature wants the output, and neither should pay for the other."""- mthds_contents: list[str] = Field(..., description="The MTHDS contents that were validated (echo of the request)")+ mthds_contents: list[str] = Field(+ ...,+ description=(+ "The MTHDS contents that were validated: the request's `mthds_contents`, or the texts of a method tree's "+ "`.mthds` bundles in tree path order."+ ),+ )message: str = Field(default="MTHDS content validated successfully", description="Status message")default_pipe_ref: str | None = Field(default=None,description=("The qualified `domain.pipe_code` a caller gets by omitting the pipe selector — the pipe a "- "selector-less run of THIS request would execute. On a `method_ref` request it is the fetched "- "package manifest's `main_pipe` (the package author's declared entry pipe, qualified against the "- "closure); otherwise, and when the manifest declares none, it is the closure's primary blueprint's "- "`main_pipe` qualified by its domain. `null` when no entry pipe is determined: no blueprint declares "- "`main_pipe`, or a manifest names a pipe the closure does not declare (or declares ambiguously), in "- "which case a selector-less run by this address would fail to resolve it too."+ "selector-less run of THIS request would execute. For a method tree (`files` or a `method_ref`) it is "+ "the tree's entry pipe by the MTHDS protocol's rule: the `main_pipe` of the `METHODS.toml` at its root, "+ "else the one `main_pipe` its bundles declare. For inline `mthds_contents` it is the closure's primary "+ "blueprint's `main_pipe` qualified by its domain. `null` when no entry pipe is determined, in which case a "+ "selector-less run would be refused too."),)input_form: dict[str, PipeInputFormDescriptor] = Field(@@ -371,37 +382,8 @@ async def validate_mthds(request: Request, request_data: ValidateRequest) -> JSO# validator registry, returning either a validation verdict or a classified fault report.# Only `ErrorReport`s with validation diagnostics are invalid-bundle verdicts (→ 200# InvalidReport); backend/config/runtime reports keep the global problem+json mapping.- if request_data.method_ref is not None:- # Validate-by-address (the addressing design's Phase 2 delta): the runner resolves the- # `method_ref` through the SAME fetch path as a `method_ref` run, and the package's real- # file names feed `mthds_sources` so diagnostics carry true per-file labels. Every- # selector-resolution failure — parse, fetch, no package, ambiguity, the custom-Python- # policy — raises OUT of this block as a non-2xx problem+json: no verdict was produced,- # so it is never rendered as `is_valid: false` (that verdict is about MTHDS content).- with fetched_method_source(request_data.method_ref) as fetched:- validated_contents = fetched.mthds_contents- # The manifest's declared entry pipe outranks the closure's own in the default chain the- # run routes and `/pipe-io` share, so it must reach `default_pipe_ref` — the report itself- # is manifest-blind — and the graph arm, which would otherwise draw the primary blueprint's- # `main_pipe`: a different pipe when the two disagree, and no graph at all when the bundles- # declare none.- # The bare code is passed as is, so the graph arm resolves it exactly as the run does.- manifest_main_pipe = fetched.main_pipe- verdict = await ApiRunner(- library_dirs=fetched.library_dirs,- user_id=user_id,- extras=request_data.analytics_groups,- ).validate_verdict(- mthds_contents=fetched.mthds_contents,- mthds_sources=fetched.mthds_sources,- allow_signatures=request_data.allow_signatures,- requested_orchestration_mode=request_data.orchestration_mode,- graph_pipe_code=manifest_main_pipe,- )- elif request_data.mthds_contents is not None:+ if request_data.mthds_contents is not None:validated_contents = request_data.mthds_contents- # Inline contents carry no manifest, so the closure's own declaration is the whole chain.- manifest_main_pipe = Noneverdict = await ApiRunner(user_id=user_id,extras=request_data.analytics_groups,@@ -413,10 +395,52 @@ async def validate_mthds(request: Request, request_data: ValidateRequest) -> JSOrequested_orchestration_mode=request_data.orchestration_mode,graph_pipe_code=None,)+ default_pipe_ref_known = False+ tree_default_pipe_ref: str | None = Noneelse:- # Unreachable: the envelope's XOR validator already 422'd this shape. Kept for the type- # checker (and as a defensive backstop should the envelope ever loosen).- raise_validation_error(message="provide exactly one of `mthds_contents` or `method_ref`")+ # A method tree: sent as `files`, or fetched by `method_ref` from its commit's git objects through the same path+ # as a `method_ref` run. Every failure to obtain the tree — a map that is not a tree, missing blobs, a fetch, a+ # package location, the custom-Python policy — raises OUT of this block as a non-2xx problem+json: no verdict+ # was produced, so it is never rendered as `is_valid: false` (that verdict is about the method's content).+ request_method: RequestMethod+ if request_data.method_ref is not None:+ fetched = await asyncio.to_thread(fetched_method, request_data.method_ref)+ request_method = fetched.request_method+ method_label = fetched.label+ ensure_method_holds_a_bundle(request_method=request_method, label=f"Method package {fetched.label}")+ ensure_python_allowed(request_method=request_method, label=f"Method package {fetched.label}")+ else:+ blob_store = RunnerBlobStore.make_for_scope(scope=resolve_blob_scope(request, read_scope=request_data.read_scope))+ sent_method = await read_request_method(documents=tree_documents_of(body=await request.body()), blob_store=blob_store)+ if sent_method is None:+ # Unreachable: the envelope's selector validator already 422'd a body with no `files`.+ raise_validation_error(message="provide exactly one of `mthds_contents`, `files` or `method_ref`")+ request_method = with_resources(request_method=sent_method, resources=decode_bundle_resources(resources=request_data.resources))+ method_label = REQUEST_METHOD_LABEL+ ensure_method_holds_a_bundle(request_method=request_method, label="This method")+ ensure_python_allowed(request_method=request_method, label="This method")+ await record_request_method(request_method=request_method, blob_store=blob_store)+ validated_contents = _bundle_texts(request_method=request_method)+ # The entry pipe of the tree, the one a selector-less run executes, is the graph's pipe and the default the+ # verdict reports; a tree with none still validates.+ entry = _optional_tree_entry(request_method=request_method, method_label=method_label)+ tree_default_pipe_ref = entry.pipe_ref if entry is not None else None+ default_pipe_ref_known = True+ with written_method(request_method=request_method) as written:+ verdict = await ApiRunner(+ library_dirs=[str(written.method_dir)],+ library_dirs_are_callers=True,+ methods_dirs=written.methods_dirs,+ user_id=user_id,+ extras=request_data.analytics_groups,+ ).validate_verdict(+ mthds_contents=None,+ mthds_sources=None,+ allow_signatures=request_data.allow_signatures,+ requested_orchestration_mode=request_data.orchestration_mode,+ graph_pipe_code=tree_default_pipe_ref,+ method_dir=written.method_dir,+ )if not isinstance(verdict, PipelexValidationReport):if verdict.validation_errors:return _invalid_report_response(verdict, requested_formats=requested_formats)@@ -430,7 +454,7 @@ async def validate_mthds(request: Request, request_data: ValidateRequest) -> JSO{**dict(report),"mthds_contents": validated_contents,- "default_pipe_ref": _effective_default_pipe_ref(report, manifest_main_pipe=manifest_main_pipe),+ "default_pipe_ref": tree_default_pipe_ref if default_pipe_ref_known else _effective_default_pipe_ref(report),})content = response_data.model_dump(mode="json", serialize_as_any=True, by_alias=True)diff --git a/api/pipelex_api/routes/version.py b/api/pipelex_api/routes/version.pyindex 4143e2782..ea696691e 100644--- a/api/pipelex_api/routes/version.py+++ b/api/pipelex_api/routes/version.py@@ -36,6 +36,9 @@ async def get_version() -> PipelexVersionInfo:The handshake clients use for feature detection: `implementation`identifies this runner, `implementation_version` is this server package'sversion, and `runtime_version` is the underlying pipelex runtime version.+ `blob_store` is true: this runner keeps a blob store over its storage+ provider and serves `POST /v1/blobs/missing` and `PUT /v1/blobs/{sha256}`,+ so a sender may name a file of `files` by its hash."""try:implementation_version = version(IMPLEMENTATION_NAME)@@ -48,4 +51,5 @@ async def get_version() -> PipelexVersionInfo:implementation=IMPLEMENTATION_NAME,implementation_version=implementation_version,runtime_version=runtime_version,+ blob_store=True,)diff --git a/api/pipelex_api/schemas/models.py b/api/pipelex_api/schemas/models.pyindex 2c6786d7c..3fd57e0fe 100644--- a/api/pipelex_api/schemas/models.py+++ b/api/pipelex_api/schemas/models.py@@ -44,17 +44,59 @@ def _ensure_mthds_file_within_bytes_limit(content: str) -> None:raise ValueError(msg)+class Base64FileEntry(BaseModel):+ """A file entry carrying the file's bytes in canonical padded standard Base64."""++ model_config = ConfigDict(extra="forbid")++ base64: str = Field(description="The file's bytes in canonical padded standard Base64.")+++class HashFileEntry(BaseModel):+ """A file entry naming, by their SHA-256 and their size, bytes the receiver already holds in its blob store."""++ model_config = ConfigDict(extra="forbid")++ sha256: str = Field(pattern=r"^[0-9a-f]{64}$", description="The SHA-256 of the file's bytes, in lowercase hexadecimal.")+ size: int = Field(ge=0, description="The file's size in bytes, an integer written with no fraction or exponent.")+++FileEntry = str | Base64FileEntry | HashFileEntry+"""One file of a method tree as it travels: its UTF-8 text, its bytes in Base64, or the hash of bytes the receiver holds."""++FILES_FIELD_DESCRIPTION = (+ "The method as a map of file entries, the MTHDS standard's form of a method tree that travels: each key is a tree "+ "path (relative, forward slashes, in Unicode NFC, no `.` or `..` segment, no segment starting with a dot), and each "+ 'value is the file\'s UTF-8 text, `{"base64": ...}` holding its bytes, or `{"sha256": ..., "size": ...}` '+ "naming bytes this runner already holds (uploaded with `POST /v1/blobs/missing` then `PUT /v1/blobs/{sha256}`). The "+ "map is read from the request's JSON text: a key given twice, a key the membership rule keeps out of every tree "+ "(`.mthds/` included), colliding keys and a malformed entry are a 422 `InvalidMethodFiles` whose `tree_refusal` names "+ "the code and the paths, and hash entries the runner does not hold are one 422 `MissingBlobs` whose `missing_blobs` "+ "lists every such hash. Its `.mthds` files are the method's bundles and every other file is one a step may read, "+ "such as a `PipeDocGen` template or a `PipeFunc` module. With no `pipe_code`, the run starts at the `main_pipe` of the "+ "`METHODS.toml` at the tree's root, or else at the one `main_pipe` its bundles declare. Custom `.py` is only honored "+ "on a sandbox-hosted deployment."+)++DEPENDENCIES_FIELD_DESCRIPTION = (+ "PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method packages the method calls by address that this "+ 'runner cannot fetch, such as one an author has not published, as `{"<address>": <map of file entries>}`: each '+ "value is a tree of its own, read as `files` is, with a `METHODS.toml` at its root declaring the address it is keyed "+ "by. They are searched before the runner's installed copies, for this request alone. Accompanies `files`."+)+RESOURCES_FIELD_DESCRIPTION = ("The files submitted with the bundle that its steps read, such as a PipeDocGen step's `template_file` (an ""`.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file's bundle-relative path, exactly "- "as the step names it, to the file's bytes in standard base64. Every bundle of the request is read as sitting in "- "one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or "- "colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`. "+ "as the step names it, to the file's bytes in standard base64. A path must be relative and written with forward "+ "slashes, with no `..` segment, backslash or colon. Beside inline `mthds_contents`, every bundle of the request is "+ "read as sitting in one directory, and every file must be named by a step, or the request is refused with a 422 "+ "`BundleResourceNotNamedError`. Beside a method tree, `files` or `bundle_b64`, each file joins the tree at its path "+ "from the tree's root, where a file the tree already holds must have the same bytes. "f"By default at most {DEFAULT_MAX_RESOURCES_PER_REQUEST} files, {DEFAULT_MAX_RESOURCE_FILE_KIB} KiB each and "- f"{DEFAULT_MAX_RESOURCES_TOTAL_KIB} KiB in all, decoded, counting the files a method bundle ships for its steps; a "- "deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. A file "- "both sent here and shipped in a method bundle must be the same. The files make one method tree with the bundle at its "- "root, which the run names by its digest: no file's bytes travel with the run, each being read where a step uses it."+ f"{DEFAULT_MAX_RESOURCES_TOTAL_KIB} KiB in all, decoded; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, "+ "`MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. The files make one method tree with the bundles, which the run "+ "names by its digest: no file's bytes travel with the run, each being read where a step uses it.")@@ -86,23 +128,24 @@ class RunRequest(BaseModel):output_name: Name of the output slot to write to.output_multiplicity: Output multiplicity setting.dynamic_output_concept_ref: Override for the dynamic output concept ref.- bundle_b64: PIPELEX-API EXTENSION — base64-encoded zip of the whole method- bundle (`.mthds` + `.py` + `structures/*.py` + `requirements.txt`). Lets a- caller ship custom PipeFunc Python alongside the method; materialized into a- temporary library directory before the run. The method packages the bundle- calls by address may travel with it under `.mthds/methods/<name>/`, each with- its `METHODS.toml`, and are searched before the runner's installed copies.- Mutually exclusive with `files`.- files: PIPELEX-API EXTENSION — the bundle as a `{relative_path: text}` map (the- unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.+ bundle_b64: PIPELEX-API EXTENSION — base64-encoded zip of the method's folder, read+ as a map of its entries, so its names are judged by the method tree's rule as the+ keys of `files` are. The method packages it calls by address may travel with it+ under `.mthds/methods/<name>/`, each with its `METHODS.toml`, and are searched+ before the runner's installed copies. Mutually exclusive with `files`.+ files: The method as a map of file entries, the MTHDS standard's method tree: each+ value its text, its bytes in Base64, or the hash of bytes the runner holds. Read+ from the request's raw JSON text, never from this field, which only documents it.+ dependencies: PIPELEX-API EXTENSION — the method packages the method calls by address+ that the runner cannot fetch, each a map of file entries keyed by its address.method_ref: PIPELEX-API EXTENSION — run a published method by address instead ofinline source. Resolved by THIS runner (git fetch at the tag, package located- by manifest identity); mutually exclusive with `mthds_contents` and with a- method bundle. `pipe_code` may accompany it to override the manifest's- `main_pipe`.+ by manifest identity, its tree read from the commit's git objects); mutually+ exclusive with `mthds_contents` and with a method sent as a tree. `pipe_code`+ may accompany it to override the manifest's `main_pipe`.resources: The files the submitted bundle's steps read, such as a PipeDocGenstep's `template_file`, as `{bundle_relative_path: base64}`. Accompanies inline- `mthds_contents` or a method bundle, never a `method_ref`.+ `mthds_contents` or a method sent as a tree, which they join, never a `method_ref`."""model_config = ConfigDict(extra="allow")@@ -116,21 +159,17 @@ class RunRequest(BaseModel):bundle_b64: str | None = Field(default=None,description=(- "PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — base64-encoded zip of the whole "- "method bundle (`.mthds` + `.py` + `structures/*.py` + `requirements.txt`), materialized into a "- "temporary library directory before the run so custom PipeFunc Python travels with the method. "- "The method packages the bundle calls by address may travel with it under `.mthds/methods/<name>/`, "- "each with its `METHODS.toml`; they are searched before the runner's installed copies, for this run alone. "+ "PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — base64-encoded zip of the method's folder, read as a "+ "map of its entries, so the archive's names are judged by the method tree's rule as the keys of `files` are. "+ "The method packages it calls by address may travel with it under `.mthds/methods/<name>/`, each with its "+ "`METHODS.toml`; they are searched before the runner's installed copies, for this run alone. ""Mutually exclusive with `files`. Custom `.py` is only honored on a sandbox-hosted deployment."),)- files: dict[str, str] | None = Field(- default=None,- description=(- "PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method bundle as a "- "`{relative_path: text}` map (the unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`."- ),- )+ # Documents the wire only: the map is read from the request's raw JSON text (`pipelex_api.method_tree_request`),+ # since a key given twice or a size written `13.0` is visible there alone, so the value is never validated here.+ files: Annotated[dict[str, FileEntry] | None, SkipValidation] = Field(default=None, description=FILES_FIELD_DESCRIPTION)+ dependencies: Annotated[dict[str, dict[str, FileEntry]] | None, SkipValidation] = Field(default=None, description=DEPENDENCIES_FIELD_DESCRIPTION)method_ref: str | None = Field(default=None,min_length=1,@@ -185,6 +224,10 @@ class RunRequest(BaseModel):if not values.get("mthds_contents") and not values.get("mthds_content") and not has_bundle:msg = "resources accompany the bundle whose steps read them: send them with mthds_contents or a method bundle."raise PipelineRequestError(msg)+ # `dependencies` are the packages the method sent in `files` calls, so they accompany that tree.+ if values.get("dependencies") is not None and values.get("files") is None:+ msg = "dependencies accompany the method's own files: send the method's tree in files, and its dependencies beside it."+ raise PipelineRequestError(msg)if values.get("pipe_code") is None and not values.get("mthds_contents") and not has_bundle and not has_method_ref:msg = ("pipe_code and mthds_contents cannot both be empty. Either: both are provided, or if there are no mthds_contents, "@@ -260,6 +303,7 @@ class RunRequest(BaseModel):dynamic_output_concept_ref=request_body.get("dynamic_output_concept_ref"),bundle_b64=request_body.get("bundle_b64"),files=request_body.get("files"),+ dependencies=request_body.get("dependencies"),method_ref=request_body.get("method_ref"),resources=request_body.get("resources"),)@@ -644,7 +688,7 @@ so the static crate routes do not accept it."""class MthdsContentsRequest(BaseModel):- """The `POST /validate` request envelope: `mthds_contents` XOR `method_ref`, plus the sweep's `allow_signatures`.+ """The `POST /validate` request envelope: one of `mthds_contents`, `files` and `method_ref`, plus the sweep's `allow_signatures`.This is the flat-list envelope. `/validate` uses it through this class (its `ValidateRequest`subclass adds a parallel `mthds_sources` for per-file source labels), and the run routes@@ -662,8 +706,17 @@ class MthdsContentsRequest(BaseModel):default=None,min_length=1,max_length=MAX_MTHDS_FILES_PER_REQUEST,- description="MTHDS contents to load (always an array, even for a single file). Exactly one of `mthds_contents` / `method_ref`.",+ description="MTHDS contents to load (always an array, even for a single file). Exactly one of `mthds_contents` / `files` / `method_ref`.",)+ # Documents the wire only: the route reads the map from the request's raw JSON text, never from this field.+ files: Annotated[dict[str, FileEntry] | None, SkipValidation] = Field(+ default=None,+ description=(+ f"{FILES_FIELD_DESCRIPTION} The whole tree is validated, its Python included, by syntax. "+ "Exactly one of `mthds_contents` / `files` / `method_ref`."+ ),+ )+ dependencies: Annotated[dict[str, dict[str, FileEntry]] | None, SkipValidation] = Field(default=None, description=DEPENDENCIES_FIELD_DESCRIPTION)method_ref: str | None = Field(default=None,min_length=1,@@ -671,8 +724,8 @@ class MthdsContentsRequest(BaseModel):description=("PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — validate a published method by reference ""instead of inline contents. Address form: `github.com/<owner>/<repo>[/<selector>][@<tag>]`, resolved by "- "THIS runner through the same fetch path as a `method_ref` run; the package's `.mthds` files feed the "- "validation with their real relative paths as per-file sources. Exactly one of `mthds_contents` / "+ "THIS runner through the same fetch path as a `method_ref` run; the package's tree, read from its commit, is "+ "validated as a `files` tree is. Exactly one of `mthds_contents` / `files` / ""`method_ref`. A resolution failure (fetch, package location, the custom-Python policy) is a non-2xx ""`problem+json` — never an `is_valid: false` verdict, which is reserved for actual MTHDS content."),@@ -690,7 +743,11 @@ class MthdsContentsRequest(BaseModel):def _exactly_one_content_selector(self) -> Self:# Strict XOR (the addressing design's tooling-route rule): a stateless diagnostic call# has no linkage slot, so a second selector could only be ignored — refuse it instead.- if (self.mthds_contents is None) == (self.method_ref is None):- msg = "provide exactly one of `mthds_contents` or `method_ref`"+ selectors = [self.mthds_contents, self.files, self.method_ref]+ if sum(selector is not None for selector in selectors) != 1:+ msg = "provide exactly one of `mthds_contents`, `files` or `method_ref`"+ raise ValueError(msg)+ if self.dependencies is not None and self.files is None:+ msg = "`dependencies` accompany the method's own `files`"raise ValueError(msg)return selfdiff --git a/api/tests/unit/conftest.py b/api/tests/unit/conftest.pyindex d9eaf0b1e..5a42d71e3 100644--- a/api/tests/unit/conftest.py+++ b/api/tests/unit/conftest.py@@ -1,4 +1,6 @@-from collections.abc import Callable+import subprocess+from collections.abc import Callable, Generator+from contextlib import contextmanagerfrom pathlib import Pathimport pytest@@ -8,6 +10,7 @@ from pipelex.methods.package_locator import locate_package_in_clonefrom pipelex.pipelex import Pipelexfrom pipelex.system.runtime import IntegrationModefrom pipelex.test_extras.shared_pytest_plugins import needs_inference_in_pipelex+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProviderfrom pytest import FixtureRequestfrom pytest_mock import MockerFixture@@ -15,6 +18,14 @@ from pipelex_api.api_config import get_api_configfrom tests.unit._constants import STUB_METHOD_COMMIT_SHA, STUB_METHOD_MANIFEST[email protected](autouse=True, name="blob_storage")+def blob_storage_fixture(mocker: MockerFixture) -> InMemoryStorageProvider:+ """Keep each test's blob store in memory: a request's trees are recorded there, and nothing is written to disk."""+ storage = InMemoryStorageProvider()+ mocker.patch("pipelex_api.blob_store.get_storage_provider", return_value=storage)+ return storage++@pytest.fixture(autouse=True)def reset_api_config_fixture(request: FixtureRequest):# Code to run before each test@@ -38,6 +49,13 @@ def reset_api_config_fixture(request: FixtureRequest):pipelex_instance.teardown()+def commit_stub_clone(*, clone_root: Path) -> None:+ """Make a stub clone a git repository holding its files in one commit, as a fetched clone does."""+ git_identity = ["-c", "user.name=stub", "-c", "[email protected]", "-c", "commit.gpgsign=false"]+ for args in (["init", "--quiet"], ["add", "--all"], [*git_identity, "commit", "--quiet", "--allow-empty", "-m", "stub"]):+ subprocess.run(["git", "-C", str(clone_root), *args], check=True, capture_output=True) # noqa: S603, S607++class _StubCloneCache:"""A method clone cache whose network fetch is replaced by a local directory tree.@@ -50,10 +68,11 @@ class _StubCloneCache:self._clone_root = clone_rootself._commit_sha = commit_sha- def get_or_fetch(self, *, ref: MethodRef, clone_url: str | None = None) -> FetchedMethodPackage: # noqa: ARG002+ @contextmanager+ def pinned_package(self, *, ref: MethodRef, clone_url: str | None = None) -> Generator[FetchedMethodPackage, None, None]: # noqa: ARG002located = locate_package_in_clone(clone_root=self._clone_root, requested_address=ref.address)ensure_package_within_bounds(package_dir=located.package_dir, package_address=located.full_address)- return FetchedMethodPackage(+ yield FetchedMethodPackage(ref=ref,full_address=located.full_address,commit_sha=self._commit_sha,@@ -82,6 +101,9 @@ def install_method_package(mocker: MockerFixture, tmp_path: Path) -> Callable[..target = package_dir / relative_pathtarget.parent.mkdir(parents=True, exist_ok=True)target.write_text(content, encoding="utf-8")+ # The runner reads a fetched package from its commit's git objects, so the stub clone is a repository with+ # its files committed, as a real clone is.+ commit_stub_clone(clone_root=clone_root)stub = _StubCloneCache(clone_root=clone_root, commit_sha=commit_sha)mocker.patch("pipelex_api.method_source.get_method_clone_cache", return_value=stub)return package_dirdiff --git a/api/tests/unit/test_blob_routes.py b/api/tests/unit/test_blob_routes.pynew file mode 100644index 000000000..e4811c055--- /dev/null+++ b/api/tests/unit/test_blob_routes.py@@ -0,0 +1,277 @@+"""The blob routes of the MTHDS Protocol: `POST /v1/blobs/missing` and `PUT /v1/blobs/{sha256}`.++A sender asks which of its blobs the runner lacks, uploads each to the signed target the answer names, then sends its+method by hash. These pin the wire: the targets, the write-once upload and its refusals, the scopes that keep one+caller's blobs apart from another's, the tree digest the runner answers in one lookup once it has read that tree in+full, and the `blob_store` flag `/v1/version` declares. The blob store is the in-memory one the conftest installs.+"""++from typing import Any+from urllib.parse import parse_qs, urlsplit++import pytest+from fastapi import FastAPI+from fastapi.testclient import TestClient+from mthds.method_tree.digest import digest_of_contents, sha256_hex, tree_files_from_contents, tree_listing+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider+from pytest_mock import MockerFixture++from pipelex_api.exception_handlers import register_exception_handlers+from pipelex_api.routes import router as api_router+from pipelex_api.routes.blob_upload import router as blob_upload_router+from pipelex_api.routes.version import router as version_router+from tests.unit._constants import VALID_MTHDS++_TEMPLATE = b"<h1>A template</h1>"+_NOTES = b"Some notes"+++def _build_client() -> TestClient:+ app = FastAPI()+ app.include_router(api_router, prefix="/v1")+ app.include_router(version_router, prefix="/v1")+ app.include_router(blob_upload_router, prefix="/v1")+ register_exception_handlers(app)+ return TestClient(app)+++def _blob_ref(data: bytes) -> dict[str, Any]:+ return {"sha256": sha256_hex(data), "size": len(data)}+++def _missing(client: TestClient, *blobs: bytes, digest: str | None = None, read_scope: str | None = None) -> list[dict[str, Any]]:+ body: dict[str, Any] = {"blobs": [_blob_ref(data) for data in blobs]}+ if digest is not None:+ body["digest"] = digest+ params = {"read_scope": read_scope} if read_scope is not None else None+ response = client.post("/v1/blobs/missing", json=body, params=params)+ assert response.status_code == 200, response.text+ missing: list[dict[str, Any]] = response.json()["missing"]+ return missing+++def _upload(client: TestClient, *, target: dict[str, Any], data: bytes) -> Any:+ upload = target["upload"]+ return client.request(upload["method"], upload["url"], content=data, headers=upload["headers"])+++def _assert_problem(response: Any, *, status: int, error_type: str) -> str:+ assert response.status_code == status, response.text+ assert response.headers["content-type"].startswith("application/problem+json")+ problem = response.json()+ assert problem["error_type"] == error_type, response.text+ detail: str = problem["detail"]+ return detail+++class TestBlobRoutes:+ def test_the_version_declares_a_blob_store(self) -> None:+ response = _build_client().get("/v1/version")++ assert response.status_code == 200, response.text+ assert response.json()["blob_store"] is True++ def test_each_blob_the_runner_lacks_comes_with_a_signed_upload_target(self) -> None:+ missing = _missing(_build_client(), _TEMPLATE, _NOTES)++ assert [(blob["sha256"], blob["size"]) for blob in missing] == [(sha256_hex(_TEMPLATE), len(_TEMPLATE)), (sha256_hex(_NOTES), len(_NOTES))]+ for blob in missing:+ upload = blob["upload"]+ assert upload["method"] == "PUT"+ assert upload["headers"] == {"content-type": "application/octet-stream"}+ url_text: str = upload["url"]+ url = urlsplit(url_text)+ assert url.path == f"/v1/blobs/{blob['sha256']}"+ assert set(parse_qs(url.query)) == {"scope", "expires", "signature"}++ def test_an_uploaded_blob_is_held_and_uploading_it_again_changes_nothing(self) -> None:+ client = _build_client()+ (target,) = _missing(client, _TEMPLATE)++ first = _upload(client, target=target, data=_TEMPLATE)+ again = _upload(client, target=target, data=_TEMPLATE)++ assert first.status_code == 204, first.text+ assert again.status_code == 204, again.text+ assert [blob["sha256"] for blob in _missing(client, _TEMPLATE, _NOTES)] == [sha256_hex(_NOTES)]++ def test_a_body_whose_hash_is_not_the_paths_is_refused_and_not_stored(self) -> None:+ client = _build_client()+ (target,) = _missing(client, _TEMPLATE)++ detail = _assert_problem(_upload(client, target=target, data=_NOTES), status=422, error_type="BlobMismatch")++ assert sha256_hex(_TEMPLATE) in detail+ assert [blob["sha256"] for blob in _missing(client, _TEMPLATE, _NOTES)] == [sha256_hex(_TEMPLATE), sha256_hex(_NOTES)]++ @pytest.mark.parametrize("tampering", ["no_query", "forged_signature", "another_scope", "another_hash"])+ def test_an_upload_without_a_valid_target_is_refused(self, tampering: str) -> None:+ client = _build_client()+ (target,) = _missing(client, _TEMPLATE)+ url_text: str = target["upload"]["url"]+ url = urlsplit(url_text)+ query: dict[str, str] = {name: values[0] for name, values in parse_qs(url.query).items()}+ path: str = url.path+ match tampering:+ case "no_query":+ query = {}+ case "forged_signature":+ query["signature"] = "0" * 64+ case "another_scope":+ query["scope"] = "someone-else"+ case "another_hash":+ path = f"/v1/blobs/{sha256_hex(_NOTES)}"+ case _:+ pytest.fail(f"unknown tampering {tampering}")+ data = _NOTES if tampering == "another_hash" else _TEMPLATE++ response = client.put(path, params=query, content=data, headers={"content-type": "application/octet-stream"})++ _assert_problem(response, status=403, error_type="BlobUploadNotAuthorized")+ assert len(_missing(client, _TEMPLATE, _NOTES)) == 2++ def test_an_expired_target_is_refused(self, mocker: MockerFixture) -> None:+ mocker.patch("pipelex_api.blob_upload.BLOB_UPLOAD_URL_TTL_SECONDS", -60)+ client = _build_client()+ (target,) = _missing(client, _TEMPLATE)++ _assert_problem(_upload(client, target=target, data=_TEMPLATE), status=403, error_type="BlobUploadNotAuthorized")++ def test_an_upload_past_the_per_file_limit_is_refused(self, mocker: MockerFixture) -> None:+ client = _build_client()+ (target,) = _missing(client, _TEMPLATE)+ mocker.patch("pipelex_api.routes.blob_upload.MAX_METHOD_TREE_FILE_BYTES", len(_TEMPLATE) - 1)++ _assert_problem(_upload(client, target=target, data=_TEMPLATE), status=422, error_type="MethodTreeTooLarge")++ def test_one_hash_listed_with_two_sizes_is_refused(self) -> None:+ body = {"blobs": [_blob_ref(_TEMPLATE), {"sha256": sha256_hex(_TEMPLATE), "size": len(_TEMPLATE) + 1}]}++ detail = _assert_problem(_build_client().post("/v1/blobs/missing", json=body), status=422, error_type="BlobMismatch")++ assert sha256_hex(_TEMPLATE) in detail++ @pytest.mark.parametrize("size", [12.0, 12.5, "12", True])+ def test_a_size_that_is_not_a_json_integer_is_refused(self, size: Any) -> None:+ body = {"blobs": [{"sha256": sha256_hex(_TEMPLATE), "size": size}]}++ response = _build_client().post("/v1/blobs/missing", json=body)++ assert response.status_code == 422, response.text++ def test_blobs_past_the_tree_limits_are_refused_before_any_upload(self, mocker: MockerFixture) -> None:+ mocker.patch("pipelex_api.method_tree_request.MAX_METHOD_TREE_FILES", 1)++ response = _build_client().post("/v1/blobs/missing", json={"blobs": [_blob_ref(_TEMPLATE), _blob_ref(_NOTES)]})++ _assert_problem(response, status=422, error_type="MethodTreeTooLarge")++ def test_one_scopes_blobs_are_not_held_for_another(self) -> None:+ client = _build_client()+ (target,) = _missing(client, _TEMPLATE, read_scope="org-a")+ assert _upload(client, target=target, data=_TEMPLATE).status_code == 204++ assert _missing(client, _TEMPLATE, read_scope="org-a") == []+ assert [blob["sha256"] for blob in _missing(client, _TEMPLATE, read_scope="org-b")] == [sha256_hex(_TEMPLATE)]+ assert [blob["sha256"] for blob in _missing(client, _TEMPLATE)] == [sha256_hex(_TEMPLATE)]++ def test_a_read_scope_that_is_not_a_scope_is_refused(self) -> None:+ response = _build_client().post("/v1/blobs/missing", json={"blobs": []}, params={"read_scope": "../escape"})++ _assert_problem(response, status=422, error_type="InvalidReadScope")++ def test_a_tree_read_in_full_is_answered_by_its_digest(self) -> None:+ """Once a request sent the tree, the runner holds its listing, and asking with the tree's digest is one lookup."""+ client = _build_client()+ contents = {"main.mthds": VALID_MTHDS.encode("utf-8"), "notes.txt": _NOTES}+ digest = digest_of_contents(contents)+ assert len(_missing(client, *contents.values(), digest=digest)) == 2++ validated = client.post("/v1/validate", json={"files": {"main.mthds": VALID_MTHDS, "notes.txt": _NOTES.decode("utf-8")}})++ assert validated.status_code == 200, validated.text+ assert validated.json()["is_valid"] is True, validated.text+ assert _missing(client, *contents.values(), digest=digest) == []+ assert _missing(client, *contents.values()) == []++ def test_an_uploaded_listing_never_records_its_tree(self) -> None:+ """Anyone can upload bytes that happen to form a listing, so an upload never stands for a tree held in full."""+ client = _build_client()+ contents = {"main.mthds": VALID_MTHDS.encode("utf-8"), "notes.txt": _NOTES}+ listing = tree_listing(tree_files_from_contents(contents))+ (target,) = _missing(client, listing)+ assert _upload(client, target=target, data=listing).status_code == 204++ missing = _missing(client, *contents.values(), digest=digest_of_contents(contents))++ assert [blob["sha256"] for blob in missing] == [sha256_hex(data) for data in contents.values()]++ @pytest.mark.parametrize(+ ("name", "value"),+ [+ pytest.param("signature", "é" * 64, id="signature_outside_ascii"),+ pytest.param("expires", "soon", id="expires_not_a_number"),+ pytest.param("expires", "١٢٣", id="expires_in_other_digits"),+ pytest.param("expires", "-5", id="expires_signed"),+ ],+ )+ def test_a_malformed_target_is_refused_as_no_valid_target(self, name: str, value: str) -> None:+ """Every part of the query is the caller's text: one no target of this runner holds is a 403, never a server error."""+ client = _build_client()+ (target,) = _missing(client, _TEMPLATE)+ url = urlsplit(str(target["upload"]["url"]))+ query: dict[str, str] = {key: values[0] for key, values in parse_qs(url.query).items()}+ query[name] = value++ response = client.put(url.path, params=query, content=_TEMPLATE, headers={"content-type": "application/octet-stream"})++ _assert_problem(response, status=403, error_type="BlobUploadNotAuthorized")++ def test_upload_targets_start_with_the_configured_public_base_url(self, mocker: MockerFixture) -> None:+ """Behind a proxy terminating TLS, the URL a request arrived at is not the one a sender reaches."""+ mocker.patch("pipelex_api.blob_upload.get_optional_env", return_value="https://runner.example.com/api/")+ client = _build_client()++ (target,) = _missing(client, _TEMPLATE)++ url_text: str = target["upload"]["url"]+ assert url_text.startswith(f"https://runner.example.com/api/v1/blobs/{sha256_hex(_TEMPLATE)}?")+ url = urlsplit(url_text)+ query: dict[str, str] = {key: values[0] for key, values in parse_qs(url.query).items()}+ uploaded = client.put(f"/v1/blobs/{sha256_hex(_TEMPLATE)}", params=query, content=_TEMPLATE, headers=target["upload"]["headers"])+ assert uploaded.status_code == 204, uploaded.text++ def test_without_a_public_base_url_targets_start_with_the_request_s_url(self) -> None:+ (target,) = _missing(_build_client(), _TEMPLATE)++ assert str(target["upload"]["url"]).startswith(f"http://testserver/v1/blobs/{sha256_hex(_TEMPLATE)}?")++ def test_a_public_base_url_that_is_not_an_http_url_is_a_configuration_error(self, mocker: MockerFixture) -> None:+ mocker.patch("pipelex_api.blob_upload.get_optional_env", return_value="runner.example.com")++ response = _build_client().post("/v1/blobs/missing", json={"blobs": [_blob_ref(_TEMPLATE)]})++ _assert_problem(response, status=500, error_type="ServerMisconfigured")++ def test_a_blob_held_at_another_size_is_listed_for_its_upload_to_write_over(self, blob_storage: InMemoryStorageProvider) -> None:+ """Bytes held short, as a storage that does not publish writes whole could leave them, are asked for again and repaired."""+ client = _build_client()+ blob_storage.root[f"single-tenant/blobs/{sha256_hex(_TEMPLATE)}"] = _TEMPLATE[:5]++ (target,) = _missing(client, _TEMPLATE)+ uploaded = _upload(client, target=target, data=_TEMPLATE)++ assert uploaded.status_code == 204, uploaded.text+ assert blob_storage.root[f"single-tenant/blobs/{sha256_hex(_TEMPLATE)}"] == _TEMPLATE+ assert _missing(client, _TEMPLATE) == []++ def test_a_held_digest_answers_alone_only_for_blobs_of_its_tree(self) -> None:+ """A digest the runner holds says nothing of blobs outside its tree, which are looked up one by one."""+ client = _build_client()+ contents = {"main.mthds": VALID_MTHDS.encode("utf-8"), "notes.txt": _NOTES}+ validated = client.post("/v1/validate", json={"files": {"main.mthds": VALID_MTHDS, "notes.txt": _NOTES.decode("utf-8")}})+ assert validated.status_code == 200, validated.text++ missing = _missing(client, *contents.values(), _TEMPLATE, digest=digest_of_contents(contents))++ assert [blob["sha256"] for blob in missing] == [sha256_hex(_TEMPLATE)]diff --git a/api/tests/unit/test_blob_store.py b/api/tests/unit/test_blob_store.pynew file mode 100644index 000000000..f23fb55a8--- /dev/null+++ b/api/tests/unit/test_blob_store.py@@ -0,0 +1,183 @@+"""The runner's blob store over a storage provider: what it looks up by size, what it reads, and how many at once.++The storage here counts its calls and the ones in flight, so these pin what the routes cannot show: a blob held at+another size than an entry states is never read, reads and lookups run a bounded number at a time, a short object is+written over by the next upload, and a tree whose listing is held is recorded without a lookup per file.+"""++import asyncio+import json++import pytest+from mthds.method_tree.digest import digest_of_contents, sha256_hex, tree_files_from_contents, tree_listing+from pipelex.tools.storage.exceptions import StorageFileNotFoundError+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract, StoredData+from typing_extensions import override++from pipelex_api.blob_store import MAX_CONCURRENT_BLOB_LOOKUPS, MAX_CONCURRENT_BLOB_TRANSFERS, RunnerBlobStore+from pipelex_api.errors import ApiError+from pipelex_api.method_tree_request import RequestTree, read_request_tree++_SCOPE = "single-tenant"+++class _CountingStorage(StorageProviderAbstract):+ """An in-memory storage counting its reads and size lookups, and the most of each it served at once."""++ def __init__(self) -> None:+ self.objects: dict[str, bytes] = {}+ self.loaded_keys: list[str] = []+ self.sized_keys: list[str] = []+ self.most_loads_at_once = 0+ self.most_lookups_at_once = 0+ self._loads_in_flight = 0+ self._lookups_in_flight = 0++ @override+ async def _load_with_metadata(self, key: str) -> StoredData:+ self._loads_in_flight += 1+ self.most_loads_at_once = max(self.most_loads_at_once, self._loads_in_flight)+ try:+ # Yield, so every read the store started at once is in flight together.+ await asyncio.sleep(0)+ self.loaded_keys.append(key)+ if key not in self.objects:+ msg = f"File not found: '{key}'"+ raise StorageFileNotFoundError(msg)+ return StoredData(data=self.objects[key], mime_type=None)+ finally:+ self._loads_in_flight -= 1++ @override+ async def _load_size(self, key: str) -> int:+ self._lookups_in_flight += 1+ self.most_lookups_at_once = max(self.most_lookups_at_once, self._lookups_in_flight)+ try:+ await asyncio.sleep(0)+ self.sized_keys.append(key)+ if key not in self.objects:+ msg = f"File not found: '{key}'"+ raise StorageFileNotFoundError(msg)+ return len(self.objects[key])+ finally:+ self._lookups_in_flight -= 1++ @override+ async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ self.objects[key] = data++ @override+ async def public_url(self, uri: str) -> str | None:+ return None+++def _blob_key(data: bytes) -> str:+ return f"{_SCOPE}/blobs/{sha256_hex(data)}"+++def _hash_map(entries: dict[str, tuple[bytes, int]]) -> str:+ """A map of file entries naming each file's bytes by hash, at the size given beside them."""+ return json.dumps({tree_path: {"sha256": sha256_hex(data), "size": size} for tree_path, (data, size) in entries.items()})+++def _problem_of(exc_info: pytest.ExceptionInfo[ApiError]) -> dict[str, object]:+ document: dict[str, object] = exc_info.value.document+ return document++[email protected](loop_scope="class")+class TestRunnerBlobStore:+ async def test_a_blob_held_at_another_size_is_refused_before_any_byte_is_read(self) -> None:+ """An entry stating a size other than the held bytes' is a `BlobMismatch`, and the census bounds every read."""+ storage = _CountingStorage()+ blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage)+ large = b"x" * 4096+ await blob_store.store_blob(data=large)++ with pytest.raises(ApiError) as exc_info:+ await read_request_tree(document=_hash_map({"large.bin": (large, 0)}), blob_store=blob_store, label="The `files` map")++ problem = _problem_of(exc_info)+ assert exc_info.value.status_code == 422+ assert problem["error_type"] == "BlobMismatch"+ assert problem["mismatched_blobs"] == [{"sha256": sha256_hex(large), "held_size": len(large)}]+ assert storage.loaded_keys == []++ async def test_a_blob_not_held_is_missing_and_a_held_one_is_read_and_checked(self) -> None:+ storage = _CountingStorage()+ blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage)+ held = b"held bytes"+ absent = b"absent bytes"+ await blob_store.store_blob(data=held)++ with pytest.raises(ApiError) as exc_info:+ await read_request_tree(+ document=_hash_map({"held.txt": (held, len(held)), "absent.txt": (absent, len(absent))}), blob_store=blob_store, label="The map"+ )+ assert _problem_of(exc_info)["missing_blobs"] == [sha256_hex(absent)]++ tree = await read_request_tree(document=_hash_map({"held.txt": (held, len(held))}), blob_store=blob_store, label="The map")+ assert tree == RequestTree.make_from_contents(contents={"held.txt": held})++ async def test_bytes_that_do_not_hash_to_their_key_are_a_server_fault(self) -> None:+ """Damaged storage is no caller's doing, and no upload of theirs would change it: a 500 naming the hash."""+ storage = _CountingStorage()+ blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage)+ data = b"the true bytes"+ storage.objects[_blob_key(data)] = b"the damaged!!!"++ with pytest.raises(ApiError) as exc_info:+ await read_request_tree(document=_hash_map({"file.txt": (data, len(data))}), blob_store=blob_store, label="The map")++ assert exc_info.value.status_code == 500+ assert _problem_of(exc_info)["error_type"] == "BlobStoreCorrupted"++ async def test_lookups_and_reads_run_a_bounded_number_at_a_time(self) -> None:+ storage = _CountingStorage()+ blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage)+ blobs = [f"file number {index_blob}".encode() for index_blob in range(3 * MAX_CONCURRENT_BLOB_LOOKUPS)]+ for data in blobs:+ storage.objects[_blob_key(data)] = data++ stored = await blob_store.stored_sizes(sha256s=[sha256_hex(data) for data in blobs])+ held = await blob_store.read_held(sizes={sha256: size for sha256, size in stored.items() if size is not None})++ assert [held.read_blob(sha256_hex(data)) for data in blobs] == blobs+ assert storage.most_lookups_at_once == MAX_CONCURRENT_BLOB_LOOKUPS+ assert storage.most_loads_at_once == MAX_CONCURRENT_BLOB_TRANSFERS++ async def test_a_short_object_is_not_held_and_the_next_upload_writes_it_over(self) -> None:+ """A storage that does not publish writes whole may leave a blob cut short: it is asked for again, then repaired."""+ storage = _CountingStorage()+ blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage)+ data = b"the whole blob, of which a failed write kept only a part"+ storage.objects[_blob_key(data)] = data[:10]++ assert await blob_store.missing_blobs(sizes={sha256_hex(data): len(data)}) == [sha256_hex(data)]+ await blob_store.store_blob(data=data)++ assert storage.objects[_blob_key(data)] == data+ assert await blob_store.missing_blobs(sizes={sha256_hex(data): len(data)}) == []++ async def test_a_tree_whose_listing_is_held_is_recorded_without_a_lookup_per_file(self) -> None:+ storage = _CountingStorage()+ blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage)+ contents = {f"file_{index_file}.txt": f"content {index_file}".encode() for index_file in range(5)}+ request_tree = RequestTree.make_from_contents(contents=contents)+ await blob_store.record_tree(tree=request_tree.tree, contents=request_tree.contents)+ lookups_after_first = len(storage.sized_keys)++ await blob_store.record_tree(tree=request_tree.tree, contents=request_tree.contents)++ assert len(storage.sized_keys) == lookups_after_first + 1+ assert await blob_store.tree_hashes(tree_digest=digest_of_contents(contents)) == {sha256_hex(data) for data in contents.values()}++ async def test_a_listing_that_is_not_the_digest_s_holds_no_tree(self) -> None:+ storage = _CountingStorage()+ blob_store = RunnerBlobStore(scope=_SCOPE, storage=storage)+ contents = {"a.txt": b"a"}+ digest = digest_of_contents(contents)+ other_listing = tree_listing(tree_files_from_contents({"b.txt": b"b"}))+ storage.objects[f"{_SCOPE}/trees/{digest.removeprefix('sha256:')}"] = other_listing++ assert await blob_store.tree_hashes(tree_digest=digest) is Nonediff --git a/api/tests/unit/test_bundle_archive.py b/api/tests/unit/test_bundle_archive.pynew file mode 100644index 000000000..c01a40a5d--- /dev/null+++ b/api/tests/unit/test_bundle_archive.py@@ -0,0 +1,96 @@+"""Unit tests for `pipelex_api.bundle` — decoding a `bundle_b64` archive under its guards.++Covers the archive's own guards: the encoded-length check before decoding, invalid base64, a+corrupt zip, an empty archive, the member-count ceiling and the zip-bomb bounded decompression.+The names of the entries it returns are judged by the tree's rule where the method is read+(`pipelex_api.method_tree_request`), so this module returns them as the archive gives them.+"""++import base64+import io+import zipfile++import pytest+from pytest_mock import MockerFixture++from pipelex_api.bundle import decode_bundle_archive+from pipelex_api.errors import ApiError++_MAIN_MTHDS = 'domain = "b"\nmain_pipe = "echo"\n'+++def _zip_b64(entries: list[tuple[str, str]]) -> str:+ buffer = io.BytesIO()+ with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive:+ for name, content in entries:+ archive.writestr(name, content)+ return base64.b64encode(buffer.getvalue()).decode("ascii")+++class TestBundleArchive:+ def test_the_members_are_returned_in_archive_order_with_their_bytes(self):+ entries = decode_bundle_archive(bundle_b64=_zip_b64([("main.mthds", _MAIN_MTHDS), ("structures/models.py", "class Foo: ...\n")]))+ assert entries == [("main.mthds", _MAIN_MTHDS.encode("utf-8")), ("structures/models.py", b"class Foo: ...\n")]++ def test_a_directory_member_is_skipped(self):+ buffer = io.BytesIO()+ with zipfile.ZipFile(buffer, "w") as archive:+ archive.writestr("templates/", "")+ archive.writestr("templates/a.txt", "a")+ entries = decode_bundle_archive(bundle_b64=base64.b64encode(buffer.getvalue()).decode("ascii"))+ assert entries == [("templates/a.txt", b"a")]++ @pytest.mark.parametrize("name", ["../evil.py", "/etc/passwd", "a\\b.py"])+ def test_a_name_is_returned_as_the_archive_gives_it_for_the_tree_rule_to_judge(self, name: str):+ entries = decode_bundle_archive(bundle_b64=_zip_b64([(name, "x")]))+ assert entries == [(name, b"x")]++ def test_a_name_given_twice_is_returned_twice(self):+ with pytest.warns(UserWarning, match="Duplicate name"):+ bundle_b64 = _zip_b64([("notes.txt", "a"), ("notes.txt", "b")])+ assert decode_bundle_archive(bundle_b64=bundle_b64) == [("notes.txt", b"a"), ("notes.txt", b"b")]++ def test_an_empty_archive_is_refused(self):+ with pytest.raises(ApiError) as exc:+ decode_bundle_archive(bundle_b64=_zip_b64([]))+ assert exc.value.status_code == 422+ assert exc.value.document["error_type"] == "InvalidBundle"++ def test_file_count_ceiling(self, mocker: MockerFixture):+ mocker.patch("pipelex_api.bundle.MAX_BUNDLE_FILES", 2)+ with pytest.raises(ApiError) as exc:+ decode_bundle_archive(bundle_b64=_zip_b64([(f"file_{index}.py", "x") for index in range(3)]))+ assert exc.value.status_code == 413+ assert exc.value.document["error_type"] == "PayloadTooLarge"++ def test_zip_bomb_bounded_by_total_size(self, mocker: MockerFixture):+ # A highly compressible entry that decompresses well past the (patched) ceiling must be+ # refused by the bounded read, not silently expanded into memory.+ mocker.patch("pipelex_api.bundle.MAX_BUNDLE_TOTAL_BYTES", 1024)+ with pytest.raises(ApiError) as exc:+ decode_bundle_archive(bundle_b64=_zip_b64([("bomb.txt", "A" * (1024 * 64))]))+ assert exc.value.status_code == 413+ assert exc.value.document["error_type"] == "PayloadTooLarge"++ def test_corrupt_zip_rejected(self):+ not_a_zip = base64.b64encode(b"this is definitely not a zip archive").decode("ascii")+ with pytest.raises(ApiError) as exc:+ decode_bundle_archive(bundle_b64=not_a_zip)+ assert exc.value.status_code == 422+ assert exc.value.document["error_type"] == "InvalidBundle"++ def test_invalid_base64_rejected(self):+ with pytest.raises(ApiError) as exc:+ decode_bundle_archive(bundle_b64="!!! not base64 !!!")+ assert exc.value.status_code == 400+ assert exc.value.document["error_type"] == "InvalidBase64"++ def test_oversized_base64_rejected_before_decode(self, mocker: MockerFixture):+ # The cheap length check on the still-encoded string fires before any b64decode.+ mocker.patch("pipelex_api.bundle._MAX_BUNDLE_B64_CHARS", 16)+ decode_spy = mocker.spy(base64, "b64decode")+ with pytest.raises(ApiError) as exc:+ decode_bundle_archive(bundle_b64="A" * 64)+ assert exc.value.status_code == 413+ assert exc.value.document["error_type"] == "PayloadTooLarge"+ decode_spy.assert_not_called()diff --git a/api/tests/unit/test_bundle_materialize.py b/api/tests/unit/test_bundle_materialize.pydeleted file mode 100644index a94c79877..000000000--- a/api/tests/unit/test_bundle_materialize.py+++ /dev/null@@ -1,182 +0,0 @@-"""Unit tests for `pipelex_api.bundle` — method-bundle materialization and its ingest guards.--Covers both transport forms (`bundle_b64` zip and the `files` map), their-equivalence, and every guard: both-forms/empty rejection, path traversal-(absolute / `..` / backslash), the file-count and total-size ceilings, the-zip-bomb bounded-decompression guard, corrupt-zip and invalid-base64 handling,-and the guaranteed temp-dir cleanup on context exit.-"""--import base64-import io-import zipfile-from pathlib import Path--import pytest-from pytest_mock import MockerFixture--from pipelex_api.bundle import materialized_bundle, parse_bundle-from pipelex_api.errors import ApiError--_MAIN_MTHDS = """\-domain = "b"-main_pipe = "echo"--[pipe.echo]-type = "PipeFunc"-description = "e"-inputs = { text = "Text" }-output = "Text"-function_name = "pipe_func.echo"-"""-_PIPE_FUNC_PY = "def echo(working_memory):\n return 'hi'\n"---def _zip_b64(files: dict[str, str]) -> str:- buffer = io.BytesIO()- with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive:- for name, content in files.items():- archive.writestr(name, content)- return base64.b64encode(buffer.getvalue()).decode("ascii")---class TestBundleMaterialize:- def test_files_map_materializes_to_disk(self):- files = {"main.mthds": _MAIN_MTHDS, "pipe_func.py": _PIPE_FUNC_PY}- with materialized_bundle(bundle_b64=None, files=files) as bundle:- assert bundle.directory.is_dir()- assert bundle.has_python_sources is True- assert set(bundle.relpaths) == {"main.mthds", "pipe_func.py"}- assert (bundle.directory / "main.mthds").read_text() == _MAIN_MTHDS- assert (bundle.directory / "pipe_func.py").read_text() == _PIPE_FUNC_PY-- def test_zip_equivalent_to_files_map(self):- files = {"main.mthds": _MAIN_MTHDS, "structures/models.py": "class Foo: ...\n"}- with materialized_bundle(bundle_b64=None, files=files) as from_files:- files_listing = {relpath: (from_files.directory / relpath).read_text() for relpath in from_files.relpaths}- with materialized_bundle(bundle_b64=_zip_b64(files), files=None) as from_zip:- zip_listing = {relpath: (from_zip.directory / relpath).read_text() for relpath in from_zip.relpaths}- assert files_listing == zip_listing == files-- def test_nested_subdir_materializes(self):- files = {"main.mthds": _MAIN_MTHDS, "structures/models.py": "class Foo: ...\n"}- with materialized_bundle(bundle_b64=None, files=files) as bundle:- nested = bundle.directory / "structures" / "models.py"- assert nested.is_file()- assert "structures/models.py" in bundle.relpaths-- def test_no_python_sources_flag_when_only_mthds(self):- with materialized_bundle(bundle_b64=None, files={"main.mthds": _MAIN_MTHDS}) as bundle:- assert bundle.has_python_sources is False-- def test_directory_cleaned_up_on_exit(self):- with materialized_bundle(bundle_b64=None, files={"main.mthds": _MAIN_MTHDS}) as bundle:- directory = bundle.directory- assert directory.exists()- assert not directory.exists()-- def test_directory_cleaned_up_on_error(self):- captured: dict[str, object] = {}-- class _Boom(RuntimeError):- pass-- def _run() -> None:- with materialized_bundle(bundle_b64=None, files={"main.mthds": _MAIN_MTHDS}) as bundle:- captured["dir"] = bundle.directory- raise _Boom-- with pytest.raises(_Boom):- _run()- directory = captured["dir"]- assert isinstance(directory, Path)- assert not directory.exists()-- def test_both_forms_rejected(self):- with (- pytest.raises(ApiError) as exc,- materialized_bundle(bundle_b64=_zip_b64({"main.mthds": _MAIN_MTHDS}), files={"main.mthds": _MAIN_MTHDS}),- ):- pass- assert exc.value.status_code == 422- assert exc.value.document["error_type"] == "InvalidBundle"-- def test_neither_form_rejected(self):- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64=None, files=None):- pass- assert exc.value.status_code == 422- assert exc.value.document["error_type"] == "InvalidBundle"-- def test_empty_files_rejected(self):- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64=None, files={}):- pass- assert exc.value.status_code == 422- assert exc.value.document["error_type"] == "InvalidBundle"-- @pytest.mark.parametrize("bad_name", ["/etc/passwd", "../evil.py", "sub/../../evil.py", "a\\b.py", "C:passwd"])- def test_unsafe_entry_names_rejected(self, bad_name: str):- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64=None, files={bad_name: "x"}):- pass- assert exc.value.status_code == 422- assert exc.value.document["error_type"] == "InvalidBundle"-- def test_unsafe_entry_names_rejected_in_zip(self):- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64=_zip_b64({"../evil.py": "x"}), files=None):- pass- assert exc.value.status_code == 422- assert exc.value.document["error_type"] == "InvalidBundle"-- def test_file_count_ceiling(self, mocker: MockerFixture):- mocker.patch("pipelex_api.bundle.MAX_BUNDLE_FILES", 2)- files = {f"file_{index}.py": "x" for index in range(3)}- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64=None, files=files):- pass- assert exc.value.status_code == 413- assert exc.value.document["error_type"] == "PayloadTooLarge"-- def test_total_size_ceiling_files(self, mocker: MockerFixture):- mocker.patch("pipelex_api.bundle.MAX_BUNDLE_TOTAL_BYTES", 64)- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64=None, files={"big.py": "x" * 128}):- pass- assert exc.value.status_code == 413- assert exc.value.document["error_type"] == "PayloadTooLarge"-- def test_zip_bomb_bounded_by_total_size(self, mocker: MockerFixture):- # A highly compressible entry that decompresses well past the (patched) ceiling must be- # refused by the bounded read, not silently expanded into memory.- mocker.patch("pipelex_api.bundle.MAX_BUNDLE_TOTAL_BYTES", 1024)- bomb = _zip_b64({"bomb.txt": "A" * (1024 * 64)})- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64=bomb, files=None):- pass- assert exc.value.status_code == 413- assert exc.value.document["error_type"] == "PayloadTooLarge"-- def test_corrupt_zip_rejected(self):- not_a_zip = base64.b64encode(b"this is definitely not a zip archive").decode("ascii")- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64=not_a_zip, files=None):- pass- assert exc.value.status_code == 422- assert exc.value.document["error_type"] == "InvalidBundle"-- def test_invalid_base64_rejected(self):- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64="!!! not base64 !!!", files=None):- pass- assert exc.value.status_code == 400- assert exc.value.document["error_type"] == "InvalidBase64"-- def test_parse_reports_python_sources_without_writing(self):- # parse_bundle decodes + guards in memory only; nothing is written to disk yet.- with_py = parse_bundle(bundle_b64=None, files={"main.mthds": _MAIN_MTHDS, "pipe_func.py": _PIPE_FUNC_PY})- assert with_py.has_python_sources is True- without_py = parse_bundle(bundle_b64=None, files={"main.mthds": _MAIN_MTHDS})- assert without_py.has_python_sources is False-- def test_oversized_base64_rejected_before_decode(self, mocker: MockerFixture):- # The cheap length check on the still-encoded string fires before any b64decode.- mocker.patch("pipelex_api.bundle._MAX_BUNDLE_B64_CHARS", 16)- decode_spy = mocker.spy(base64, "b64decode")- with pytest.raises(ApiError) as exc, materialized_bundle(bundle_b64="A" * 64, files=None):- pass- assert exc.value.status_code == 413- assert exc.value.document["error_type"] == "PayloadTooLarge"- decode_spy.assert_not_called()diff --git a/api/tests/unit/test_bundle_resources_routes.py b/api/tests/unit/test_bundle_resources_routes.pyindex 8a67b5831..4a7fc5943 100644--- a/api/tests/unit/test_bundle_resources_routes.py+++ b/api/tests/unit/test_bundle_resources_routes.py@@ -9,10 +9,12 @@ The hermetic API environment has no document engine, so the step's engine is sturecords the bytes it reads through the method file store, which is the read the engine's checker would be handed."""+import asyncioimport base64import io+import jsonimport zipfile-from collections.abc import Mapping+from collections.abc import Callable, Mappingfrom pathlib import Pathfrom typing import Any@@ -22,6 +24,7 @@ from fastapi.testclient import TestClientfrom mthds.method_tree.digest import digest_of_contents, sha256_hex, tree_files_from_contents, tree_listingfrom pipelex.cogt.content_generation.assignment_models import RenderDocumentAssignmentfrom pipelex.cogt.doc_gen.doc_gen_setting import DocGenSetting+from pipelex.core.method_trees.exceptions import MethodFileUnavailableErrorfrom pipelex.core.method_trees.method_files import StoredTree, read_method_file, read_stored_treesfrom pipelex.pipe_operators.doc_gen.pipe_doc_gen import PipeDocGenfrom pipelex.pipe_run.delivery_assignment import DeliveryAssignment@@ -68,6 +71,19 @@ template_file = "{_TEMPLATE_PATH}"_INVOICE_INPUTS: dict[str, Any] = {"invoice": {"concept": "invoicing.Invoice", "content": {"number": "INV-1"}}}+# The stub package's manifest, starting the method at the invoice's print.+_INVOICE_PACKAGE_MANIFEST = """\+[package]+name = "documents"+address = "github.com/pipelex/methods"+version = "0.1.0"+description = "A stub package printing an invoice from the template it ships."+main_pipe = "print_invoice"++[exports.invoicing]+pipes = ["print_invoice"]+"""+def _b64(data: bytes) -> str:return base64.b64encode(data).decode("ascii")@@ -330,20 +346,22 @@ class TestBundleResourcesRoutes:("bundle_files", "resources", "tree_files"),[pytest.param(- {"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES}, None, {_TEMPLATE_PATH: _TEMPLATE_BYTES}, id="beside_the_bundle"+ {"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES},+ None,+ {"invoicing.mthds": _INVOICE_MTHDS.encode("utf-8"), _TEMPLATE_PATH: _TEMPLATE_BYTES},+ id="beside_the_bundle",),- # The bundle's other files are written where it ships them, and its step's file is keyed by the path the step- # names, both in the one tree the bundle's contents sit at the root of.+ # The archive is the method's tree: the bundle's step names its file from the bundle's own tree path.pytest.param({"methods/invoicing.mthds": _INVOICE_MTHDS, f"methods/{_TEMPLATE_PATH}": _TEMPLATE_BYTES},None,- {_TEMPLATE_PATH: _TEMPLATE_BYTES, f"methods/{_TEMPLATE_PATH}": _TEMPLATE_BYTES},+ {"methods/invoicing.mthds": _INVOICE_MTHDS.encode("utf-8"), f"methods/{_TEMPLATE_PATH}": _TEMPLATE_BYTES},id="in_a_directory",),pytest.param({"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES},{_TEMPLATE_PATH: _b64(_TEMPLATE_BYTES)},- {_TEMPLATE_PATH: _TEMPLATE_BYTES},+ {"invoicing.mthds": _INVOICE_MTHDS.encode("utf-8"), _TEMPLATE_PATH: _TEMPLATE_BYTES},id="also_in_resources",),],@@ -351,7 +369,7 @@ class TestBundleResourcesRoutes:def test_a_method_bundle_carries_the_office_template_it_ships_to_its_step(self, mocker: MockerFixture, bundle_files: dict[str, str | bytes], resources: dict[str, str] | None, tree_files: dict[str, bytes]) -> None:- """A template file shipped beside the bundle's `.mthds` file joins the tree of the submitted files as one sent in `resources` does."""+ """A method sent as an archive is a method tree: the job names that tree, its bundle and the template it ships, by its digest."""_stub_the_document_engine(mocker)orchestrator = _record_dispatches_to_a_temporal_like_orchestrator(mocker)body: dict[str, Any] = {"bundle_b64": _zip_b64(bundle_files), "inputs": _INVOICE_INPUTS}@@ -382,50 +400,130 @@ class TestBundleResourcesRoutes:assert response.status_code == 202, response.text(pipe_job,) = orchestrator.pipe_jobs- _assert_the_job_names_the_tree_of(pipe_job=pipe_job, files={"templates/invoice.html": html_template.encode("utf-8")})- _assert_the_orchestrator_copied_the_tree_of(orchestrator=orchestrator, files={"templates/invoice.html": html_template.encode("utf-8")})+ tree_files = {"invoicing.mthds": html_mthds.encode("utf-8"), "templates/invoice.html": html_template.encode("utf-8")}+ _assert_the_job_names_the_tree_of(pipe_job=pipe_job, files=tree_files)+ _assert_the_orchestrator_copied_the_tree_of(orchestrator=orchestrator, files=tree_files)assert "invoice.number }}" not in pipe_job.model_dump_json()- @pytest.mark.parametrize(- ("bundle_files", "resources"),- [- pytest.param(- {"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES}, {_TEMPLATE_PATH: _b64(b"another template")}, id="and_resources"- ),- pytest.param(- {- "a/invoicing.mthds": _INVOICE_MTHDS,- f"a/{_TEMPLATE_PATH}": _TEMPLATE_BYTES,- "b/invoicing.mthds": _INVOICE_MTHDS.replace('domain = "invoicing"', 'domain = "invoicing_b"').replace(- "invoicing.Invoice", "invoicing_b.Invoice"- ),- f"b/{_TEMPLATE_PATH}": b"another template",- },- None,- id="two_of_its_bundles",- ),- ],- )- def test_two_different_files_a_step_reads_under_one_path_are_refused(- self, mocker: MockerFixture, bundle_files: dict[str, str | bytes], resources: dict[str, str] | None- ) -> None:- """Every bundle of a request reads its files as if it sat in one directory, so one path holds one file."""+ def test_a_tree_sent_to_start_is_served_in_this_process_only_until_start_returns(self, mocker: MockerFixture) -> None:+ """Why an orchestrator copies a job's trees before its `start` returns: the runner drops them as soon as it does.++ The runner wrote the sent tree into a temporary directory and its library held it for the request; both are+ gone once `/start` answers, so a job still running in any process reads its files from where its orchestrator+ copied them.+ """_stub_the_document_engine(mocker)- body: dict[str, Any] = {"bundle_b64": _zip_b64(bundle_files), "inputs": _INVOICE_INPUTS}- if resources is not None:- body["resources"] = resources+ orchestrator = _record_dispatches_to_a_temporal_like_orchestrator(mocker)+ html_mthds = _INVOICE_MTHDS.replace('format = "docx"', 'format = "pdf"').replace(_TEMPLATE_PATH, "templates/invoice.html")+ files = {"invoicing.mthds": html_mthds, "templates/invoice.html": "<h1>{{ invoice.number }}</h1>"}++ response = _build_client().post("/v1/start", json={"files": files, "inputs": _INVOICE_INPUTS})++ assert response.status_code == 202, response.text+ (pipe_job,) = orchestrator.pipe_jobs+ crate = pipe_job.library_crate+ assert crate is not None+ assert [copied.tree_digest for copied in orchestrator.copied_trees] == list(crate.tree_digests())+ with pytest.raises(MethodFileUnavailableError):+ asyncio.run(read_stored_trees(tree_digests=crate.tree_digests(), read_scope=pipe_job.job_metadata.run_metadata.read_scope))++ def test_a_file_of_the_method_and_other_bytes_in_resources_under_one_path_are_refused(self, mocker: MockerFixture) -> None:+ """`resources` joins the method's tree, so other bytes under a path the tree holds are two files under one path."""+ _stub_the_document_engine(mocker)+ body: dict[str, Any] = {+ "bundle_b64": _zip_b64({"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES}),+ "inputs": _INVOICE_INPUTS,+ "resources": {_TEMPLATE_PATH: _b64(b"another template")},+ }detail = _assert_problem(_build_client().post("/v1/execute", json=body), status=422, error_type="InvalidResourcePath")assert f"'{_TEMPLATE_PATH}'" in detail- def test_a_template_a_method_bundle_ships_is_held_to_the_per_file_limit(self, mocker: MockerFixture) -> None:+ @pytest.mark.parametrize("sends_the_logo", [True, False])+ def test_a_file_only_a_template_s_relative_url_names_travels_in_the_method_s_tree(self, mocker: MockerFixture, sends_the_logo: bool) -> None:+ """No step names the logo, only the template's `<img src>` does: a tree keeps it, and the load's check finds it there.++ Beside inline contents, `resources` still refuses a file no step names, so such a method is sent as a tree.+ """+ _stub_the_document_engine(mocker)+ html_mthds = _INVOICE_MTHDS.replace('format = "docx"', 'format = "pdf"').replace(_TEMPLATE_PATH, "templates/invoice.html")+ files: dict[str, Any] = {+ "invoicing.mthds": html_mthds,+ "templates/invoice.html": '<img src="logo.png" alt="Acme"><h1>{{ invoice.number }}</h1>',+ }+ if sends_the_logo:+ files["templates/logo.png"] = {"base64": _b64(b"\x89PNG\r\n\x1a\n a logo")}++ response = _build_client().post("/v1/validate", json={"files": files})++ assert response.status_code == 200, response.text+ verdict = response.json()+ assert verdict["is_valid"] is sends_the_logo, response.text+ if not sends_the_logo:+ assert "logo.png" in json.dumps(verdict["validation_errors"])++ def test_two_bundles_of_a_method_each_read_the_file_beside_them(self, mocker: MockerFixture) -> None:+ """A step names its file from its bundle's own tree path, so two bundles shipping a template each under one relative path both run."""+ _stub_the_document_engine(mocker)+ orchestrator = _record_dispatches_to_a_temporal_like_orchestrator(mocker)+ mthds_b = _INVOICE_MTHDS.replace('domain = "invoicing"', 'domain = "invoicing_b"').replace('main_pipe = "print_invoice"\n', "")+ tree_files = {+ "a/invoicing.mthds": _INVOICE_MTHDS.encode("utf-8"),+ f"a/{_TEMPLATE_PATH}": _TEMPLATE_BYTES,+ "b/invoicing.mthds": mthds_b.encode("utf-8"),+ f"b/{_TEMPLATE_PATH}": b"another template",+ }++ response = _build_client().post("/v1/start", json={"bundle_b64": _zip_b64(tree_files), "inputs": _INVOICE_INPUTS})++ assert response.status_code == 202, response.text+ (pipe_job,) = orchestrator.pipe_jobs+ _assert_the_job_names_the_tree_of(pipe_job=pipe_job, files=tree_files)++ def test_a_method_ref_package_supplies_the_template_its_step_names(+ self, mocker: MockerFixture, install_method_package: Callable[..., Path]+ ) -> None:+ """A fetched package is a method tree read from its commit: its validation reads the template it ships, and a job names that tree."""+ read_templates = _stub_the_document_engine(mocker)+ install_method_package(+ files={"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES.decode("ascii")}, manifest_toml=_INVOICE_PACKAGE_MANIFEST+ )+ method_ref = f"{STUB_METHOD_ADDRESS}@v0.1.0"+ client = _build_client()++ validated = client.post("/v1/validate", json={"method_ref": method_ref})++ assert validated.status_code == 200, validated.text+ assert validated.json()["is_valid"] is True, validated.text+ assert read_templates+ assert set(read_templates) == {_TEMPLATE_BYTES}++ orchestrator = _record_dispatches_to_a_temporal_like_orchestrator(mocker)+ started = client.post("/v1/start", json={"method_ref": method_ref, "inputs": _INVOICE_INPUTS})++ assert started.status_code == 202, started.text+ (pipe_job,) = orchestrator.pipe_jobs+ tree_files = {+ "METHODS.toml": _INVOICE_PACKAGE_MANIFEST.encode("utf-8"),+ "invoicing.mthds": _INVOICE_MTHDS.encode("utf-8"),+ _TEMPLATE_PATH: _TEMPLATE_BYTES,+ }+ _assert_the_job_names_the_tree_of(pipe_job=pipe_job, files=tree_files)+ _assert_the_orchestrator_copied_the_tree_of(orchestrator=orchestrator, files=tree_files)+ assert started.json()["method_provenance"]["tree_digest"] == digest_of_contents(tree_files)++ def test_a_template_a_method_bundle_ships_is_held_to_the_per_file_limit_of_a_tree(self, mocker: MockerFixture) -> None:_stub_the_document_engine(mocker)- mocker.patch(f"{_RESOURCES_NS}.MAX_RESOURCE_FILE_BYTES", len(_TEMPLATE_BYTES) - 1)+ mocker.patch("pipelex_api.method_tree_request.MAX_METHOD_TREE_FILE_BYTES", len(_INVOICE_MTHDS.encode("utf-8")) + 1)response = _build_client().post("/v1/execute",- json={"bundle_b64": _zip_b64({"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES}), "inputs": _INVOICE_INPUTS},+ json={+ "bundle_b64": _zip_b64({"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: bytes(len(_INVOICE_MTHDS) + 2)}),+ "inputs": _INVOICE_INPUTS,+ },)- assert "per-file" in _assert_problem(response, status=413, error_type="PayloadTooLarge")+ detail = _assert_problem(response, status=422, error_type="MethodTreeTooLarge")+ assert f"'{_TEMPLATE_PATH}'" in detaildiff --git a/api/tests/unit/test_method_cache.py b/api/tests/unit/test_method_cache.pyindex 0fc08cafa..75d2dd3fb 100644--- a/api/tests/unit/test_method_cache.py+++ b/api/tests/unit/test_method_cache.py@@ -75,8 +75,11 @@ class TestMethodCloneCache:fetch_spy = mocker.spy(cache, "_package_from_clone")clone_spy = mocker.patch("pipelex_api.method_cache.fetch_method_package", wraps=fetch_method_package)- first = cache.get_or_fetch(ref=_REF, clone_url=clone_url)- second = cache.get_or_fetch(ref=_REF, clone_url=clone_url)+ with cache.pinned_package(ref=_REF, clone_url=clone_url) as first:+ pass++ with cache.pinned_package(ref=_REF, clone_url=clone_url) as second:+ passassert clone_spy.call_count == 1, "the second request must be served from the cache"assert fetch_spy.call_count == 2, "location + bounds re-run on every request"@@ -91,7 +94,8 @@ class TestMethodCloneCache:def test_a_moved_tag_resolves_to_a_fresh_clone(self, origin: tuple[str, Path], tmp_path: Path):clone_url, repo = origincache = MethodCloneCache(root_dir=tmp_path / "cache")- first = cache.get_or_fetch(ref=_REF, clone_url=clone_url)+ with cache.pinned_package(ref=_REF, clone_url=clone_url) as first:+ pass# Move the tag: a new commit, and v0.1.0 forced onto it. Never cache by tag alone —# the same reference must now fetch the NEW commit.@@ -100,7 +104,8 @@ class TestMethodCloneCache:_git("commit", "-m", "move the tag", cwd=repo)_git("tag", "-f", "-a", "v0.1.0", "-m", "moved tag", cwd=repo)- second = cache.get_or_fetch(ref=_REF, clone_url=clone_url)+ with cache.pinned_package(ref=_REF, clone_url=clone_url) as second:+ passassert second.commit_sha != first.commit_shaassert (tmp_path / "cache" / first.commit_sha).is_dir()assert (tmp_path / "cache" / second.commit_sha).is_dir()@@ -109,7 +114,8 @@ class TestMethodCloneCache:clone_url, repo = originmocker.patch("pipelex_api.method_cache.MAX_METHOD_CACHE_CLONES", 1)cache = MethodCloneCache(root_dir=tmp_path / "cache")- first = cache.get_or_fetch(ref=_REF, clone_url=clone_url)+ with cache.pinned_package(ref=_REF, clone_url=clone_url) as first:+ pass(repo / "methods" / "documents" / "extra.mthds").write_text(VALID_MTHDS.replace('"smoke"', '"extra"'), encoding="utf-8")_git("add", "-A", cwd=repo)@@ -120,7 +126,8 @@ class TestMethodCloneCache:os.utime(tmp_path / "cache" / first.commit_sha, (old, old))second_ref = parse_method_ref("github.com/pipelex/methods/[email protected]")- second = cache.get_or_fetch(ref=second_ref, clone_url=clone_url)+ with cache.pinned_package(ref=second_ref, clone_url=clone_url) as second:+ passassert not (tmp_path / "cache" / first.commit_sha).exists(), "the oldest entry is evicted past the count bound"assert (tmp_path / "cache" / second.commit_sha).is_dir(), "the newest entry always survives"@@ -129,7 +136,8 @@ class TestMethodCloneCache:clone_url, repo = originmocker.patch("pipelex_api.method_cache.MAX_METHOD_CACHE_AGE_SECONDS", 3600)cache = MethodCloneCache(root_dir=tmp_path / "cache")- first = cache.get_or_fetch(ref=_REF, clone_url=clone_url)+ with cache.pinned_package(ref=_REF, clone_url=clone_url) as first:+ passstale = time.time() - 7200os.utime(tmp_path / "cache" / first.commit_sha, (stale, stale))@@ -138,11 +146,42 @@ class TestMethodCloneCache:_git("commit", "-m", "second version", cwd=repo)_git("tag", "-a", "v0.2.0", "-m", "second tag", cwd=repo)second_ref = parse_method_ref("github.com/pipelex/methods/[email protected]")- second = cache.get_or_fetch(ref=second_ref, clone_url=clone_url)+ with cache.pinned_package(ref=second_ref, clone_url=clone_url) as second:+ passassert not (tmp_path / "cache" / first.commit_sha).exists(), "an entry unused past the age bound is evicted"assert (tmp_path / "cache" / second.commit_sha).is_dir()+ def test_a_clone_being_read_survives_an_eviction(self, origin: tuple[str, Path], tmp_path: Path, mocker: MockerFixture):+ """A concurrent request's miss evicts the oldest clone, yet never the one another request is still reading."""+ clone_url, repo = origin+ mocker.patch("pipelex_api.method_cache.MAX_METHOD_CACHE_CLONES", 1)+ cache = MethodCloneCache(root_dir=tmp_path / "cache")+ (repo / "methods" / "documents" / "extra.mthds").write_text(VALID_MTHDS.replace('"smoke"', '"extra"'), encoding="utf-8")+ _git("add", "-A", cwd=repo)+ _git("commit", "-m", "second version", cwd=repo)+ _git("tag", "-a", "v0.2.0", "-m", "second tag", cwd=repo)+ second_ref = parse_method_ref("github.com/pipelex/methods/[email protected]")++ with cache.pinned_package(ref=_REF, clone_url=clone_url) as first:+ old = time.time() - 120+ os.utime(first.clone_dir, (old, old))+ with cache.pinned_package(ref=second_ref, clone_url=clone_url) as second:+ pass+ assert first.clone_dir.is_dir(), "a pinned clone is never evicted"+ assert (first.package_dir / "documents.mthds").read_text(encoding="utf-8") == VALID_MTHDS++ (repo / "methods" / "documents" / "third.mthds").write_text(VALID_MTHDS.replace('"smoke"', '"third"'), encoding="utf-8")+ _git("add", "-A", cwd=repo)+ _git("commit", "-m", "third version", cwd=repo)+ _git("tag", "-a", "v0.3.0", "-m", "third tag", cwd=repo)+ with cache.pinned_package(ref=parse_method_ref("github.com/pipelex/methods/[email protected]"), clone_url=clone_url) as third:+ pass++ assert not first.clone_dir.exists(), "once nothing reads a clone, the bounds evict it again"+ assert not second.clone_dir.exists()+ assert third.clone_dir.is_dir()+def test_nonexistent_repository_is_a_fetch_error(self, tmp_path: Path):with pytest.raises(MethodFetchError, match="could not reach the repository"):resolve_remote_commit_sha(ref=_BARE_REF, clone_url=f"file://{tmp_path}/no-such-repo")diff --git a/api/tests/unit/test_method_ref_run.py b/api/tests/unit/test_method_ref_run.pyindex ca28e3d7c..2e17fade7 100644--- a/api/tests/unit/test_method_ref_run.py+++ b/api/tests/unit/test_method_ref_run.py@@ -1,11 +1,12 @@"""Route-level tests for `method_ref` as a run source on /execute and /start.The pipeline runner is mocked (as in `test_pipeline_bundle`) and the clone cache is stubbed-with a local directory tree (`install_method_package` in conftest), so these pin the API-layer's contract: the fetched package's `.mthds` files ride `mthds_contents`, the entry pipe-defaults to the manifest's `main_pipe` (a request `pipe_code` overrides), provenance-`{address, tag, commit_sha}` lands on the response, the exclusivity rules 422, the-execution-locus gate refuses Python where it must, and each resolution failure maps to its+with a local git repository (`install_method_package` in conftest), so these pin the API+layer's contract: the fetched package is read from its commit's git objects as a method tree,+written as the run's one library directory, the caller's own; the entry pipe defaults to the+manifest's `main_pipe` (a request `pipe_code` overrides); provenance `{address, tag,+commit_sha, tree_digest}` lands on the response; the exclusivity rules 422; the+execution-locus gate refuses Python where it must; and each resolution failure maps to itsdistinct `error_type` + status as RFC 7807 `problem+json`."""@@ -17,15 +18,20 @@ from typing import Anyimport pytestfrom fastapi import FastAPIfrom fastapi.testclient import TestClient+from mthds.method_tree.digest import digest_of_contentsfrom pipelex.methods.exceptions import MethodFetchErrorfrom pipelex.pipeline.pipeline_response import PipelexRunResultStart, RunStatefrom pytest_mock import MockerFixturefrom pipelex_api.exception_handlers import register_exception_handlersfrom pipelex_api.routes.pipelex.pipeline import router as pipeline_router-from tests.unit._constants import STUB_METHOD_ADDRESS, STUB_METHOD_COMMIT_SHA, VALID_MTHDS+from tests.unit._constants import STUB_METHOD_ADDRESS, STUB_METHOD_COMMIT_SHA, STUB_METHOD_MANIFEST, VALID_MTHDS_METHOD_REF = f"{STUB_METHOD_ADDRESS}@v0.1.0"+_TREE_NS = "pipelex_api.method_tree_request"++# The digest of the stub package's tree: its manifest and its one bundle.+_STUB_TREE_DIGEST = digest_of_contents({"METHODS.toml": STUB_METHOD_MANIFEST.encode("utf-8"), "documents.mthds": VALID_MTHDS.encode("utf-8")})_PIPE_FUNC_PY = "def echo(working_memory):\n return 'hi'\n"@@ -66,6 +72,7 @@ def _build_client(mocker: MockerFixture) -> tuple[TestClient, dict[str, Any]]:def _make_runner(**kwargs: Any) -> Any:library_dirs: list[str] | None = kwargs.get("library_dirs")+ snapshot["library_dirs_are_callers"] = kwargs.get("library_dirs_are_callers")runner = mocker.MagicMock()async def _execute(**run_kwargs: Any) -> Any:@@ -97,16 +104,20 @@ class TestMethodRefRun:client, snapshot = _build_client(mocker)response = client.post("/v1/execute", json={"method_ref": _METHOD_REF, "inputs": {"text": "hi"}})assert response.status_code == 200, response.text- # The package's `.mthds` took the proven inline path; no Python → no library dir.- assert snapshot["mthds_contents"] == [VALID_MTHDS]- assert snapshot["library_dirs"] is None- # The entry pipe came from the manifest's `main_pipe`.- assert snapshot["pipe_code"] == "echo"- # Provenance: resolved full address, the requested tag, the fetched commit SHA.+ # The package is the run's method tree, written whole as its one library directory, the caller's own.+ assert snapshot["mthds_contents"] is None+ assert snapshot["files"] == ["METHODS.toml", "documents.mthds"]+ assert snapshot["library_dirs_are_callers"] is True+ # The entry pipe came from the manifest's `main_pipe`, qualified by the bundle that declares it.+ assert snapshot["pipe_code"] == "smoke.echo"+ # The tree is removed once the request returns.+ assert not Path(snapshot["library_dirs"][0]).exists()+ # Provenance: resolved full address, the requested tag, the fetched commit SHA, the digest of the tree it ran.assert response.json()["method_provenance"] == {"address": STUB_METHOD_ADDRESS,"tag": "v0.1.0","commit_sha": STUB_METHOD_COMMIT_SHA,+ "tree_digest": _STUB_TREE_DIGEST,}@pytest.mark.parametrize(@@ -151,11 +162,16 @@ class TestMethodRefRun:client, snapshot = _build_client(mocker)response = client.post("/v1/start", json={"method_ref": STUB_METHOD_ADDRESS, "inputs": {"text": "hi"}})assert response.status_code == 202, response.text- assert snapshot["mthds_contents"] == [VALID_MTHDS]+ assert snapshot["files"] == ["METHODS.toml", "documents.mthds"]body = response.json()assert body["pipeline_run_id"] == "run-1"# A bare address: provenance records tag=None and the resolved HEAD commit.- assert body["method_provenance"] == {"address": STUB_METHOD_ADDRESS, "tag": None, "commit_sha": STUB_METHOD_COMMIT_SHA}+ assert body["method_provenance"] == {+ "address": STUB_METHOD_ADDRESS,+ "tag": None,+ "commit_sha": STUB_METHOD_COMMIT_SHA,+ "tree_digest": _STUB_TREE_DIGEST,+ }def test_inline_runs_carry_no_provenance(self, mocker: MockerFixture):client, _ = _build_client(mocker)@@ -196,7 +212,7 @@ class TestMethodRefRun:def test_fetch_failure_is_422_with_fetch_error_type(self, mocker: MockerFixture):stub_cache = mocker.MagicMock()- stub_cache.get_or_fetch.side_effect = MethodFetchError("Failed to fetch method 'github.com/pipelex/methods/documents': boom")+ stub_cache.pinned_package.side_effect = MethodFetchError("Failed to fetch method 'github.com/pipelex/methods/documents': boom")mocker.patch("pipelex_api.method_source.get_method_clone_cache", return_value=stub_cache)client, _ = _build_client(mocker)response = client.post("/v1/execute", json={"method_ref": STUB_METHOD_ADDRESS, "inputs": {}})@@ -206,29 +222,28 @@ class TestMethodRefRun:def test_python_package_forbidden_when_not_sandbox_hosted(self, mocker: MockerFixture, install_method_package: Callable[..., Path]):install_method_package(files={"documents.mthds": VALID_MTHDS, "funcs/pipe_func.py": _PIPE_FUNC_PY})- mocker.patch("pipelex_api.method_source.is_pipe_func_sandbox_hosted", return_value=False)+ mocker.patch(f"{_TREE_NS}.is_pipe_func_sandbox_hosted", return_value=False)client, _ = _build_client(mocker)response = client.post("/v1/execute", json={"method_ref": _METHOD_REF, "inputs": {}})assert response.status_code == 403, response.textassert response.headers["content-type"] == "application/problem+json"assert response.json()["error_type"] == "CustomCodeRequiresSandbox"- def test_python_package_splits_mthds_from_py_when_sandbox_hosted(self, mocker: MockerFixture, install_method_package: Callable[..., Path]):+ def test_python_package_is_written_whole_when_sandbox_hosted(self, mocker: MockerFixture, install_method_package: Callable[..., Path]):install_method_package(files={"documents.mthds": VALID_MTHDS, "funcs/pipe_func.py": _PIPE_FUNC_PY})- mocker.patch("pipelex_api.method_source.is_pipe_func_sandbox_hosted", return_value=True)+ mocker.patch(f"{_TREE_NS}.is_pipe_func_sandbox_hosted", return_value=True)client, snapshot = _build_client(mocker)response = client.post("/v1/execute", json={"method_ref": _METHOD_REF, "inputs": {}})assert response.status_code == 200, response.text- # `.mthds` rides mthds_contents; ONLY the Python lands in the temp library dir — the- # manifest is consumed (identity + main_pipe) and deliberately NOT materialized.- assert snapshot["mthds_contents"] == [VALID_MTHDS]- assert snapshot["files"] == ["funcs/pipe_func.py"]+ # The whole tree is written, its bundle, its manifest and its Python, for the load to read as one method.+ assert snapshot["mthds_contents"] is None+ assert snapshot["files"] == ["METHODS.toml", "documents.mthds", "funcs/pipe_func.py"]# The temp dir is cleaned once the request returns.assert not Path(snapshot["library_dirs"][0]).exists()def test_structures_package_refused_even_when_sandbox_hosted(self, mocker: MockerFixture, install_method_package: Callable[..., Path]):install_method_package(files={"documents.mthds": VALID_MTHDS, "structures/models.py": _STRUCTURES_PY})- mocker.patch("pipelex_api.method_source.is_pipe_func_sandbox_hosted", return_value=True)+ mocker.patch(f"{_TREE_NS}.is_pipe_func_sandbox_hosted", return_value=True)client, _ = _build_client(mocker)response = client.post("/v1/execute", json={"method_ref": _METHOD_REF, "inputs": {}})assert response.status_code == 403, response.textdiff --git a/api/tests/unit/test_method_ref_validate.py b/api/tests/unit/test_method_ref_validate.pyindex bfdfb8fa7..93416240f 100644--- a/api/tests/unit/test_method_ref_validate.py+++ b/api/tests/unit/test_method_ref_validate.py@@ -78,4 +78,4 @@ class TestMethodRefValidate:client = _build_client()response = client.post("/v1/validate", json={"method_ref": _METHOD_REF, "mthds_sources": ["main.mthds"]})assert response.status_code == 422, response.text- assert "package's own file names" in response.json()["detail"]+ assert "named by their tree paths" in response.json()["detail"]diff --git a/api/tests/unit/test_method_requirements_refusal.py b/api/tests/unit/test_method_requirements_refusal.pyindex 22888d739..2c0c31c62 100644--- a/api/tests/unit/test_method_requirements_refusal.py+++ b/api/tests/unit/test_method_requirements_refusal.py@@ -99,13 +99,15 @@ class _UnreachableValidator:async def validate_bundles(self,*,- mthds_contents: list[str], # noqa: ARG002+ mthds_contents: list[str] | None, # noqa: ARG002mthds_sources: list[str] | None, # noqa: ARG002allow_signatures: bool, # noqa: ARG002library_dirs: Sequence[Path] | None, # noqa: ARG002caller_identity: CallerIdentity | None, # noqa: ARG002graph_pipe_code: str | None, # noqa: ARG002submitted_files: Mapping[str, bytes] | None, # noqa: ARG002+ method_dir: Path | None, # noqa: ARG002+ methods_dirs: Sequence[Path] | None, # noqa: ARG002) -> ErrorReport:self.calls += 1msg = "A method refused for its requirements must not reach the validator."diff --git a/api/tests/unit/test_method_source_git.py b/api/tests/unit/test_method_source_git.pynew file mode 100644index 000000000..ee5737310--- /dev/null+++ b/api/tests/unit/test_method_source_git.py@@ -0,0 +1,62 @@+"""A fetched package is read from its commit's git objects, never from the clone's working tree.++The clone cache is stubbed with a local git repository (`install_method_package` in conftest). These pin what the+commit holds against what the working tree holds: a file left uncommitted is no file of the package, a committed+symbolic link is refused, named by its path in the package and the package's address, and the tree's digest rides the+provenance.+"""++import subprocess+import tempfile+from collections.abc import Callable+from pathlib import Path++import pytest+from mthds.method_tree.digest import digest_of_contents+from pipelex.methods.exceptions import MethodPackageSymlinkError++from pipelex_api.method_source import fetched_method+from tests.unit._constants import STUB_METHOD_ADDRESS, STUB_METHOD_MANIFEST, VALID_MTHDS++_METHOD_REF = f"{STUB_METHOD_ADDRESS}@v0.1.0"+++def _commit_all(*, package_dir: Path) -> None:+ clone_root = package_dir.parent.parent+ git_identity = ["-c", "user.name=stub", "-c", "[email protected]", "-c", "commit.gpgsign=false"]+ for args in (["add", "--all"], [*git_identity, "commit", "--quiet", "-m", "more"]):+ subprocess.run(["git", "-C", str(clone_root), *args], check=True, capture_output=True) # noqa: S603, S607+++class TestMethodSourceGit:+ def test_the_package_is_its_commits_files_and_its_digest_rides_the_provenance(self, install_method_package: Callable[..., Path]) -> None:+ package_dir = install_method_package(files={"documents.mthds": VALID_MTHDS, "templates/invoice.html": "<h1>Invoice</h1>"})+ # Written after the commit, in the clone's working tree only.+ (package_dir / "uncommitted.txt").write_text("not in the commit", encoding="utf-8")+ (package_dir / "documents.mthds").write_text("edited after the commit", encoding="utf-8")++ fetched = fetched_method(_METHOD_REF)++ contents = fetched.request_method.tree.contents+ assert contents == {+ "METHODS.toml": STUB_METHOD_MANIFEST.encode("utf-8"),+ "documents.mthds": VALID_MTHDS.encode("utf-8"),+ "templates/invoice.html": b"<h1>Invoice</h1>",+ }+ assert fetched.request_method.dependencies == {}+ assert fetched.provenance.tree_digest == digest_of_contents(contents)+ assert fetched.label == f"'{STUB_METHOD_ADDRESS}'"++ def test_a_committed_symbolic_link_is_refused_by_tree_path_and_address(self, install_method_package: Callable[..., Path]) -> None:+ package_dir = install_method_package(files={"documents.mthds": VALID_MTHDS})+ (package_dir / "linked.mthds").symlink_to("documents.mthds")+ _commit_all(package_dir=package_dir)++ with pytest.raises(MethodPackageSymlinkError) as exc_info:+ fetched_method(_METHOD_REF)++ message = str(exc_info.value)+ assert "linked.mthds" in message+ assert STUB_METHOD_ADDRESS in message+ assert "pipelex-package-" not in message+ assert tempfile.gettempdir() not in messagediff --git a/api/tests/unit/test_method_tree_runs.py b/api/tests/unit/test_method_tree_runs.pynew file mode 100644index 000000000..8396b5c6f--- /dev/null+++ b/api/tests/unit/test_method_tree_runs.py@@ -0,0 +1,330 @@+"""A method sent as a tree, each file inline or by hash, run and validated by the real routes and runner.++The in-process orchestrator runs the method, whose only step composes a text, so what the run answers is what the+engine loaded from the tree it was sent. Hash entries resolve in the caller's own blob store, the in-memory one the+conftest installs, filled through the blob routes as a sender fills it. A refusal of the caller's own tree is the+caller's to read: a 422 naming files by their tree paths, never by the directory the runner wrote the tree into.+"""++import base64+import json+import tempfile+from collections.abc import Generator+from pathlib import Path+from typing import Any++import pytest+from fastapi import FastAPI+from fastapi.testclient import TestClient+from mthds.method_tree.digest import sha256_hex+from pipelex.config import get_config+from pytest_mock import MockerFixture++from pipelex_api.api_config import ApiConfig+from pipelex_api.exception_handlers import register_exception_handlers+from pipelex_api.routes import router as api_router+from pipelex_api.routes.blob_upload import router as blob_upload_router++_PIPELINE_NS = "pipelex_api.routes.pipelex.pipeline"++_GREET_MTHDS = """\+domain = "greeting"+main_pipe = "greet"++[pipe.greet]+type = "PipeCompose"+description = "Greet"+inputs = { text = "Text" }+output = "Text"+template = "Hello $text"+"""++_BROKEN_MTHDS = 'domain = "broken"\n\n[pipe.oops\ntype = "PipeCompose"\n'++_DEPENDENCY_ADDRESS = "github.com/acme/methods/library"+_DEPENDENCY_MANIFEST = """\+[package]+name = "library"+address = "github.com/acme/methods"+version = "0.1.0"+description = "A dependency a request sends beside its method."+"""++_NOTES = b"Some notes \xe2\x80\x94 kept with the method"+_ICON = b"\x89PNG\r\n\x1a\n not text"++[email protected](name="sandbox_hosted_mode")+def sandbox_hosted_mode_fixture() -> Generator[None, None, None]:+ """Select a non-`direct` PipeFunc execution mode, which makes the deployment sandbox-hosted for the route and the loader alike."""+ pipe_func_config = get_config().interpreter.pipe_func+ previous = pipe_func_config.execution_mode+ pipe_func_config.execution_mode = "sandbox"+ try:+ yield+ finally:+ pipe_func_config.execution_mode = previous+++def _build_client(mocker: MockerFixture) -> TestClient:+ mocker.patch(+ f"{_PIPELINE_NS}.get_api_config", return_value=ApiConfig(orchestration_mode="direct", allow_request_orchestration_mode_override=False)+ )+ app = FastAPI()+ app.include_router(api_router, prefix="/v1")+ app.include_router(blob_upload_router, prefix="/v1")+ register_exception_handlers(app)+ return TestClient(app)+++def _hash_entry(data: bytes) -> dict[str, Any]:+ return {"sha256": sha256_hex(data), "size": len(data)}+++def _upload_all(client: TestClient, *blobs: bytes, read_scope: str | None = None) -> None:+ """Upload every blob the runner lacks, as a sender does before sending hash entries."""+ params = {"read_scope": read_scope} if read_scope is not None else None+ response = client.post("/v1/blobs/missing", json={"blobs": [_hash_entry(data) for data in blobs]}, params=params)+ assert response.status_code == 200, response.text+ by_hash = {sha256_hex(data): data for data in blobs}+ for blob in response.json()["missing"]:+ upload = blob["upload"]+ uploaded = client.request(upload["method"], upload["url"], content=by_hash[blob["sha256"]], headers=upload["headers"])+ assert uploaded.status_code == 204, uploaded.text+++def _run_output(body: dict[str, Any]) -> str:+ return json.dumps(body["pipe_output"]["working_memory"])+++def _assert_no_host_path(body: Any) -> None:+ dump = json.dumps(body)+ assert "pipelex-method-" not in dump+ for temporary_root in (tempfile.gettempdir(), str(Path(tempfile.gettempdir()).resolve())):+ assert temporary_root not in dump+++class TestMethodTreeRuns:+ def test_a_method_sent_with_every_entry_form_runs(self, mocker: MockerFixture) -> None:+ """A bundle by hash, a binary file in Base64 and a text file inline make one tree, run from its entry pipe."""+ client = _build_client(mocker)+ _upload_all(client, _GREET_MTHDS.encode("utf-8"))+ files = {+ "greet.mthds": _hash_entry(_GREET_MTHDS.encode("utf-8")),+ "assets/icon.png": {"base64": base64.b64encode(_ICON).decode("ascii")},+ "notes.txt": _NOTES.decode("utf-8"),+ }++ response = client.post("/v1/execute", json={"files": files, "inputs": {"text": "world"}})++ assert response.status_code == 200, response.text+ body = response.json()+ assert body["state"] == "COMPLETED"+ assert "Hello world" in _run_output(body)++ def test_a_hash_the_runner_does_not_hold_is_a_422_naming_it(self, mocker: MockerFixture) -> None:+ client = _build_client(mocker)++ response = client.post(+ "/v1/execute", json={"files": {"greet.mthds": _GREET_MTHDS, "notes.txt": _hash_entry(_NOTES)}, "inputs": {"text": "x"}}+ )++ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] == "MissingBlobs"+ assert problem["missing_blobs"] == [sha256_hex(_NOTES)]+ assert "/v1/blobs/missing" in problem["detail"]++ def test_a_size_other_than_the_held_bytes_is_a_mismatch_no_upload_repeats(self, mocker: MockerFixture) -> None:+ """A sender following the answers, asking which blobs are missing and uploading them, ends at the mismatch, never in a loop."""+ client = _build_client(mocker)+ _upload_all(client, _NOTES)+ wrong_entry = {"sha256": sha256_hex(_NOTES), "size": len(_NOTES) + 1}+ body = {"files": {"greet.mthds": _GREET_MTHDS, "notes.txt": wrong_entry}, "inputs": {"text": "x"}}++ first = client.post("/v1/execute", json=body)+ asked = client.post("/v1/blobs/missing", json={"blobs": [wrong_entry]})+ (target,) = asked.json()["missing"]+ upload = target["upload"]+ uploaded = client.request(upload["method"], upload["url"], content=_NOTES, headers=upload["headers"])+ again = client.post("/v1/execute", json=body)++ assert uploaded.status_code == 204, uploaded.text+ for response in (first, again):+ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] == "BlobMismatch"+ assert problem["mismatched_blobs"] == [{"sha256": sha256_hex(_NOTES), "held_size": len(_NOTES)}]++ def test_a_hash_resolves_only_in_the_callers_own_scope(self, mocker: MockerFixture) -> None:+ client = _build_client(mocker)+ _upload_all(client, _GREET_MTHDS.encode("utf-8"), read_scope="org-a")+ files = {"greet.mthds": _hash_entry(_GREET_MTHDS.encode("utf-8"))}++ own = client.post("/v1/execute", json={"files": files, "inputs": {"text": "a"}, "read_scope": "org-a", "storage_scope": "org-a"})+ other = client.post("/v1/execute", json={"files": files, "inputs": {"text": "b"}, "read_scope": "org-b", "storage_scope": "org-b"})++ assert own.status_code == 200, own.text+ assert "Hello a" in _run_output(own.json())+ assert other.status_code == 422, other.text+ assert other.json()["error_type"] == "MissingBlobs"++ def test_a_callers_bundle_that_does_not_parse_is_its_422_named_by_tree_path(self, mocker: MockerFixture) -> None:+ """The written tree is the caller's own library directory, so its refusal is the caller's 422, never the runner's 500."""+ client = _build_client(mocker)+ files = {"methods/greet.mthds": _GREET_MTHDS, "methods/broken.mthds": _BROKEN_MTHDS}++ response = client.post("/v1/execute", json={"files": files, "pipe_code": "greeting.greet", "inputs": {"text": "x"}})++ assert response.status_code == 422, response.text+ body = response.json()+ assert body["error_type"] == "ValidateBundleError"+ assert {item["source"] for item in body["validation_errors"]} == {"methods/broken.mthds"}+ _assert_no_host_path(body)++ def test_validate_a_method_tree_sent_as_files(self, mocker: MockerFixture) -> None:+ client = _build_client(mocker)++ response = client.post("/v1/validate", json={"files": {"greet.mthds": _GREET_MTHDS, "notes.txt": _NOTES.decode("utf-8")}})++ assert response.status_code == 200, response.text+ body = response.json()+ assert body["is_valid"] is True, body+ assert body["mthds_contents"] == [_GREET_MTHDS]+ assert body["default_pipe_ref"] == "greeting.greet"+ assert body["graph_spec"] is not None++ def test_validate_a_method_tree_whose_bundle_does_not_parse_is_an_invalid_verdict_named_by_tree_path(self, mocker: MockerFixture) -> None:+ client = _build_client(mocker)++ response = client.post("/v1/validate", json={"files": {"greet.mthds": _GREET_MTHDS, "methods/broken.mthds": _BROKEN_MTHDS}})++ assert response.status_code == 200, response.text+ body = response.json()+ assert body["is_valid"] is False+ assert {item["source"] for item in body["validation_errors"]} == {"methods/broken.mthds"}+ _assert_no_host_path(body)++ @pytest.mark.usefixtures("sandbox_hosted_mode")+ def test_validate_reads_every_python_file_of_the_tree_for_its_syntax(self, mocker: MockerFixture) -> None:+ """No step names the module, yet the validation of the whole tree finds its syntax error, by tree path and line."""+ client = _build_client(mocker)+ files = {"greet.mthds": _GREET_MTHDS, "helpers/broken.py": "def ok():\n return 1\n\ndef oops(:\n pass\n"}++ response = client.post("/v1/validate", json={"files": files})++ assert response.status_code == 200, response.text+ body = response.json()+ assert body["is_valid"] is False, body+ (item,) = body["validation_errors"]+ assert item["source"] == "helpers/broken.py"+ assert item["line"] == 4+ _assert_no_host_path(body)++ def test_python_in_a_tree_is_refused_where_it_could_not_run(self, mocker: MockerFixture) -> None:+ client = _build_client(mocker)++ response = client.post("/v1/validate", json={"files": {"greet.mthds": _GREET_MTHDS, "helpers/tool.py": "VALUE = 1\n"}})++ assert response.status_code == 403, response.text+ assert response.json()["error_type"] == "CustomCodeRequiresSandbox"++ def test_a_json_constant_outside_the_tree_is_never_a_server_error(self, mocker: MockerFixture) -> None:+ """`NaN` is the rest of the server's to judge where it stands outside a map of file entries, and a 422 inside one."""+ client = _build_client(mocker)+ files = json.dumps({"greet.mthds": _GREET_MTHDS})+ headers = {"content-type": "application/json"}++ inline = client.post("/v1/execute", content=f'{{"mthds_contents": {json.dumps([_GREET_MTHDS])}, "inputs": {{"text": NaN}}}}', headers=headers)+ beside_a_tree = client.post("/v1/execute", content=f'{{"files": {files}, "inputs": {{"text": NaN}}}}', headers=headers)+ inside_the_tree = client.post("/v1/execute", content='{"files": {"greet.mthds": NaN}, "inputs": {"text": "x"}}', headers=headers)++ for response in (inline, beside_a_tree):+ assert response.status_code == 422, response.text+ assert response.json()["error_type"] == "WrongScalarKindError"+ assert inside_the_tree.status_code == 422, inside_the_tree.text+ assert inside_the_tree.json()["error_type"] == "InvalidMethodFiles"++ @pytest.mark.parametrize("encoding", ["utf-16", "utf-32", "utf-8-sig"])+ def test_validate_reads_a_tree_from_a_body_in_any_json_encoding(self, mocker: MockerFixture, encoding: str) -> None:+ client = _build_client(mocker)+ body = json.dumps({"files": {"greet.mthds": _GREET_MTHDS, "notes.txt": _NOTES.decode("utf-8")}}).encode(encoding)++ response = client.post("/v1/validate", content=body, headers={"content-type": "application/json"})++ assert response.status_code == 200, response.text+ assert response.json()["is_valid"] is True, response.text++ def test_a_bundle_that_is_not_utf8_text_is_the_callers_located_diagnostic(self, mocker: MockerFixture) -> None:+ client = _build_client(mocker)+ files = {"greet.mthds": _GREET_MTHDS, "bad.mthds": {"base64": "/w=="}}++ run = client.post("/v1/execute", json={"files": files, "pipe_code": "greeting.greet", "inputs": {"text": "x"}})+ run_from_the_entry = client.post("/v1/execute", json={"files": files, "inputs": {"text": "x"}})+ validated = client.post("/v1/validate", json={"files": files})++ assert run.status_code == 422, run.text+ assert run.json()["error_type"] == "ValidateBundleError"+ assert [(item["source"], item["line"], item["column"]) for item in run.json()["validation_errors"]] == [("bad.mthds", 1, 1)]+ assert run_from_the_entry.status_code == 422, run_from_the_entry.text+ assert run_from_the_entry.json()["error_type"] == "MethodEntryError"+ assert "'bad.mthds'" in run_from_the_entry.json()["detail"]+ assert validated.status_code == 200, validated.text+ verdict = validated.json()+ assert verdict["is_valid"] is False+ assert [item["source"] for item in verdict["validation_errors"]] == ["bad.mthds"]+ _assert_no_host_path(verdict)++ @pytest.mark.usefixtures("sandbox_hosted_mode")+ def test_validate_reports_a_python_file_holding_a_nul_byte_at_its_tree_path(self, mocker: MockerFixture) -> None:+ """Python 3.11 refuses a NUL byte with a ValueError where later versions raise a SyntaxError: both are the file's verdict."""+ client = _build_client(mocker)++ response = client.post("/v1/validate", json={"files": {"greet.mthds": _GREET_MTHDS, "helpers/nul.py": "x = 1\u0000\n"}})++ assert response.status_code == 200, response.text+ body = response.json()+ assert body["is_valid"] is False, body+ assert [item["source"] for item in body["validation_errors"]] == ["helpers/nul.py"]++ def test_files_sent_in_resources_are_held_to_the_tree_limits(self, mocker: MockerFixture) -> None:+ mocker.patch("pipelex_api.method_tree_request.MAX_METHOD_TREE_FILES", 2)+ client = _build_client(mocker)+ files = {"greet.mthds": _GREET_MTHDS, "notes.txt": _NOTES.decode("utf-8")}++ within = client.post("/v1/validate", json={"files": files})+ past = client.post("/v1/validate", json={"files": files, "resources": {"extra.txt": base64.b64encode(b"one more").decode("ascii")}})++ assert within.status_code == 200, within.text+ assert past.status_code == 422, past.text+ assert past.json()["error_type"] == "MethodTreeTooLarge"+ assert "`resources`" in past.json()["detail"]++ @pytest.mark.parametrize(+ ("limit", "value"),+ [("MAX_METHOD_DEPENDENCIES", 0), ("MAX_METHOD_REQUEST_TOTAL_BYTES", len(_GREET_MTHDS) + 10)],+ )+ def test_a_method_and_its_dependencies_are_held_to_limits_together(self, mocker: MockerFixture, limit: str, value: int) -> None:+ """Each tree passes its own limits, yet the request carries too many trees, or too many bytes across them."""+ mocker.patch(f"pipelex_api.method_tree_request.{limit}", value)+ client = _build_client(mocker)+ dependency = {"METHODS.toml": _DEPENDENCY_MANIFEST, "lib.mthds": _GREET_MTHDS.replace('"greeting"', '"library"')}++ response = client.post("/v1/validate", json={"files": {"greet.mthds": _GREET_MTHDS}, "dependencies": {_DEPENDENCY_ADDRESS: dependency}})++ assert response.status_code == 422, response.text+ assert response.json()["error_type"] == "MethodTreeTooLarge"++ @pytest.mark.parametrize("last_segment", ["..", "."])+ def test_a_nameless_dependency_keyed_by_an_address_no_directory_can_take_is_refused(self, mocker: MockerFixture, last_segment: str) -> None:+ client = _build_client(mocker)+ nameless_manifest = _DEPENDENCY_MANIFEST.replace('name = "library"\n', "")+ dependency = {"METHODS.toml": nameless_manifest, "lib.mthds": _GREET_MTHDS.replace('"greeting"', '"library"')}+ address = f"{_DEPENDENCY_ADDRESS.rsplit('/', 1)[0]}/{last_segment}"++ response = client.post("/v1/validate", json={"files": {"greet.mthds": _GREET_MTHDS}, "dependencies": {address: dependency}})++ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] == "InvalidMethodFiles"+ assert "cannot name a package's directory" in problem["detail"]diff --git a/api/tests/unit/test_openapi_contract.py b/api/tests/unit/test_openapi_contract.pyindex 73ac88bd8..3427fd7e7 100644--- a/api/tests/unit/test_openapi_contract.py+++ b/api/tests/unit/test_openapi_contract.py@@ -35,13 +35,15 @@ _OPERATION_KEYS = frozenset({"get", "put", "post", "delete", "options", "head",# rejections (422), and the server-fault floor (500).COMMON_STATUSES = (401, 413, 422, 500)-# `GET /v1/version` is the public protocol handshake: `pipelex_api.main` mounts it OUTSIDE the composite-# router (and outside the auth dependency), so it inherits none of the shared responses.-PUBLIC_PATHS = ("/v1/version",)+# `GET /v1/version` is the public protocol handshake, and `PUT /v1/blobs/{sha256}` the upload target a signed+# query authorizes: `pipelex_api.main` mounts both OUTSIDE the composite router (and outside the auth+# dependency), so they inherit none of the shared responses.+PUBLIC_PATHS = ("/v1/version", "/v1/blobs/{sha256}")# The MTHDS Protocol operations — and nothing else — carry `x-mthds-protocol: true`. This is the-# standard's five-operation surface (`execute`, `start`, `validate`, `models`, `version`; see the-# normative `mthds/docs/spec/openapi/mthds-protocol.openapi.yaml`), NOT everything this server+# standard's surface (`execute`, `start`, `validate`, `models`, `version`, and the blob routes a runner+# keeping a blob store serves; see the normative `mthds/docs/spec/openapi/mthds-protocol.openapi.yaml`),+# NOT everything this server# serves. `/resolve` and `/codegen` are deliberately absent: they are Pipelex API extensions. The# crate `/resolve` emits is standard-owned (the MTHDS Library Crate Format), but the route serving# it is ours, and the standard specifies no type projection at all.@@ -51,6 +53,8 @@ MTHDS_PROTOCOL_OPERATIONS = {("/v1/validate", "post"),("/v1/models", "get"),("/v1/version", "get"),+ ("/v1/blobs/missing", "post"),+ ("/v1/blobs/{sha256}", "put"),}# The extra statuses each route can produce on top of COMMON_STATUSES.diff --git a/api/tests/unit/test_pipeline_bundle.py b/api/tests/unit/test_pipeline_bundle.pyindex 8dd3183b6..1b26c5748 100644--- a/api/tests/unit/test_pipeline_bundle.py+++ b/api/tests/unit/test_pipeline_bundle.py@@ -1,13 +1,12 @@-"""Route-level tests for the method-bundle transport on /execute and /start.--The pipeline runner is mocked (as in `test_pipeline_routes`); these assert that-the API layer KEEPS the proven run path for a bundle — the bundle's `.mthds` text-is passed as `mthds_contents` (so the engine resolves `main_pipe` exactly as for a-plain run) — while ONLY the non-`.mthds` files (custom PipeFunc `.py`, etc.) are-materialized into a temporary `library_dirs` directory for source capture, while the-method packages it ships under `.mthds/methods/<name>/` leave both and are written-into their own temporary `methods_dirs` directory. They also assert the custom-Python-sandbox gate, the both-forms guard, and the shapes of `.mthds/` content refused.+"""Route-level tests for a method sent as a tree on /execute and /start: a `files` map, or a `bundle_b64` archive.++The pipeline runner is mocked (as in `test_pipeline_routes`); these assert that the API layer+reads the method as a method tree, by the one rule of the tree, and writes it WHOLE into a+temporary directory handed to the runner as its one library directory, the caller's own, its+entry pipe chosen by the MTHDS protocol's rule. The dependencies it carries, as `dependencies`+beside `files` or under `.mthds/methods/<name>/` in an archive, are written into a methods+directory of their own. They also assert the custom-Python sandbox gate, the both-forms guard,+and the refusals of a map that is not a tree, each naming the entry it refuses."""import base64@@ -35,32 +34,29 @@ from tests.unit._constants import ()_PIPE_FUNC_PY = "def echo(working_memory):\n return 'hi'\n"+_TREE_NS = "pipelex_api.method_tree_request"-# Each `.mthds/` shape a request is refused for, with the text the refusal must name: the entry, or the package directory.-_REFUSED_VENDORING_SHAPES: dict[str, tuple[dict[str, str], str]] = {- "a file under .mthds/ outside methods/": ({"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, ".mthds/foo": "x"}, ".mthds/foo"),+# The probe package as a dependency tree of its own, its files at its root.+_PROBE_TREE: dict[str, str] = {"METHODS.toml": VENDORED_PROBE_MANIFEST, "probe.mthds": VENDORED_PROBE_MTHDS}++# Each `.mthds/` shape an archive is refused for, with the text the refusal must name: an entry the tree's rule refuses, or+# the package directory a dependency tree must be.+_REFUSED_ARCHIVE_SHAPES: dict[str, tuple[dict[str, str], str]] = {+ "a file under .mthds/ outside methods/": ({"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, ".mthds/foo": "x"}, "'.mthds/foo'"),"a .mthds/methods/ store below the root": ({"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, "sub/.mthds/methods/probe/probe.mthds": VENDORED_PROBE_MTHDS},- "sub/.mthds/methods/probe/probe.mthds",+ "'sub/.mthds/methods/probe/probe.mthds'",),"a file directly under .mthds/methods/": ({"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, ".mthds/methods/probe.mthds": VENDORED_PROBE_MTHDS},- ".mthds/methods/probe.mthds",+ "'.mthds/methods/probe.mthds'",),"a package shipping its own .mthds/ store": ({**VENDORED_PROBE_FILES,".mthds/methods/probe/.mthds/methods/inner/inner.mthds": VENDORED_PROBE_MTHDS,},- ".mthds/methods/probe/.mthds/methods/inner/inner.mthds",- ),- "a package directory without METHODS.toml": (- {"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, ".mthds/methods/probe/probe.mthds": VENDORED_PROBE_MTHDS},- ".mthds/methods/probe/",- ),- "a package whose METHODS.toml is not a manifest": (- {**VENDORED_PROBE_FILES, ".mthds/methods/probe/METHODS.toml": "[package]\nname = 'probe'\n"},- ".mthds/methods/probe/METHODS.toml",+ "'.mthds/methods/inner/inner.mthds'",),"a hidden package directory": ({@@ -68,11 +64,19 @@ _REFUSED_VENDORING_SHAPES: dict[str, tuple[dict[str, str], str]] = {".mthds/methods/.probe/METHODS.toml": VENDORED_PROBE_MANIFEST,".mthds/methods/.probe/probe.mthds": VENDORED_PROBE_MTHDS,},- ".mthds/methods/.probe/",+ "'.mthds/methods/.probe/METHODS.toml'",),- "a request whose only .mthds files are vendored": (- {".mthds/methods/probe/METHODS.toml": VENDORED_PROBE_MANIFEST, ".mthds/methods/probe/probe.mthds": VENDORED_PROBE_MTHDS},- "no .mthds file of its own",+}++# A shipped package an archive is refused for, with the text the refusal must name.+_REFUSED_ARCHIVE_PACKAGES: dict[str, tuple[dict[str, str], str]] = {+ "a package directory without METHODS.toml": (+ {"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, ".mthds/methods/probe/probe.mthds": VENDORED_PROBE_MTHDS},+ ".mthds/methods/probe/",+ ),+ "a package whose METHODS.toml is not a manifest": (+ {**VENDORED_PROBE_FILES, ".mthds/methods/probe/METHODS.toml": "[package]\nname = 'probe'\n"},+ ".mthds/methods/probe/",),}@@ -85,7 +89,7 @@ def _probe_manifest(*, address: str, name: str | None) -> str:def _shipped_packages(manifests: dict[str, str]) -> dict[str, str]:- """A bundle calling the probe package and shipping one package per `{directory name: manifest}`."""+ """A bundle calling the probe package and shipping one package per `{directory name: manifest}` in an archive's layout."""files = {"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS}for directory, manifest in manifests.items():files[f".mthds/methods/{directory}/METHODS.toml"] = manifest@@ -93,36 +97,34 @@ def _shipped_packages(manifests: dict[str, str]) -> dict[str, str]:return files-# Two shipped packages a reference could not tell apart, each with the directories the refusal must name. The identity-# is the one a reference is matched against: the address, then the manifest's name or else the directory's, without case.-_DUPLICATE_IDENTITY_BUNDLES: dict[str, tuple[dict[str, str], tuple[str, ...]]] = {- "the same manifest twice": (_shipped_packages({"a": VENDORED_PROBE_MANIFEST, "b": VENDORED_PROBE_MANIFEST}), ("a", "b")),- "one named by its directory, in another case": (- _shipped_packages(- {- "first": _probe_manifest(address="github.com/invented/probe-lib", name="probe"),- "Probe": _probe_manifest(address="github.com/Invented/Probe-Lib", name=None),- }- ),- ("first", "Probe"),+# Two shipped packages a reference could not tell apart, with the address the refusal must name. The identity is the one a+# reference is matched against: the address, then the manifest's name or else the directory's, without case.+_DUPLICATE_IDENTITY_BUNDLES: dict[str, dict[str, str]] = {+ "the same manifest twice": _shipped_packages({"a": VENDORED_PROBE_MANIFEST, "b": VENDORED_PROBE_MANIFEST}),+ "one named by its directory, in another case": _shipped_packages(+ {+ "first": _probe_manifest(address="github.com/invented/probe-lib", name="probe"),+ "Probe": _probe_manifest(address="github.com/Invented/Probe-Lib", name=None),+ }),}-# Entries naming one file twice once their names are normalized, or a file that is another entry's directory, each with-# the path the refusal must name. Each name is spelled differently, so the entries fit a `files` map as well as a zip.-_COLLIDING_ENTRIES: dict[str, tuple[list[tuple[str, str]], str]] = {- "a file named twice": ([("main.mthds", VALID_MTHDS), ("notes.txt", "a"), ("./notes.txt", "b")], "'notes.txt'"),- "a shipped manifest named twice": (- [- *VENDORED_PROBE_FILES.items(),- (".mthds/methods/probe//METHODS.toml", _probe_manifest(address="github.com/invented/decoy", name="probe")),- ],- "'.mthds/methods/probe/METHODS.toml'",- ),+# Entries the tree's rule refuses, with the tree refusal code and the key the refusal must name. Each name is spelled+# differently, so the entries fit a `files` map as well as a zip.+_REFUSED_ENTRIES: dict[str, tuple[list[tuple[str, str]], str, str]] = {+ "a file named with a dot segment": ([("main.mthds", VALID_MTHDS), ("notes.txt", "a"), ("./notes.txt", "b")], "invalid_tree_path", "./notes.txt"),+ "a file named with an empty segment": ([("main.mthds", VALID_MTHDS), ("notes//a.txt", "a")], "invalid_tree_path", "notes//a.txt"),"a file that is another entry's directory": ([("main.mthds", VALID_MTHDS), ("helper.txt", "x"), ("helper.txt/inner.txt", "y")],- "'helper.txt'",+ "path_collision",+ "helper.txt",+ ),+ "a dotfile": ([("main.mthds", VALID_MTHDS), (".env", "x")], "excluded_name", ".env"),+ "a package in the local store layout": (+ [("main.mthds", VALID_MTHDS), *((f".mthds/methods/probe/{name}", text) for name, text in _PROBE_TREE.items())],+ "excluded_name",+ ".mthds/methods/probe/METHODS.toml",),}@@ -148,10 +150,9 @@ def _build_client(mocker: MockerFixture) -> tuple[TestClient, dict[str, Any]]:"""Wire an app whose ApiRunner is mocked and records what the run actually saw.The `snapshot` captures BOTH what the runner was asked to run — the- `mthds_contents` passed to `execute`/`start` — and the `library_dirs` it was- constructed with plus the files on disk at run time. Together they prove the- split: the `.mthds` travels as `mthds_contents`, only the `.py` hits disk, and- the temp dir is torn down after the request.+ `mthds_contents` and `pipe_code` passed to `execute`/`start` — and the+ `library_dirs` it was constructed with plus the files on disk at run time.+ Together they prove the tree was written whole and torn down after the request."""app = FastAPI()app.include_router(pipeline_router, prefix="/v1")@@ -174,8 +175,10 @@ def _build_client(mocker: MockerFixture) -> tuple[TestClient, dict[str, Any]]:library_dirs: list[str] | None = runner_kwargs.get("library_dirs")methods_dirs: list[Path] | None = runner_kwargs.get("methods_dirs")snapshot["library_dirs"] = library_dirs+ snapshot["library_dirs_are_callers"] = runner_kwargs.get("library_dirs_are_callers")snapshot["methods_dirs"] = methods_dirssnapshot["mthds_contents"] = run_kwargs.get("mthds_contents")+ snapshot["pipe_code"] = run_kwargs.get("pipe_code")if library_dirs:root = Path(library_dirs[0])snapshot["dir_exists"] = root.exists()@@ -208,55 +211,80 @@ def _build_client(mocker: MockerFixture) -> tuple[TestClient, dict[str, Any]]:return TestClient(app), snapshot+def _assert_tree_refusal(response: Any, *, code: str, named: str) -> None:+ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] == "InvalidMethodFiles", response.text+ assert problem["tree_refusal"]["code"] == code+ assert named in problem["tree_refusal"]["names"]+ assert repr(named) in problem["detail"] or f"'{named}'" in problem["detail"]++class TestPipelineBundle:- def test_mthds_only_bundle_rides_mthds_contents_no_library_dir(self, mocker: MockerFixture):- """A `.mthds`-only bundle takes the proven path: its text becomes `mthds_contents`- (so `main_pipe` resolves as always) and NO temp library dir is created.- """+ @pytest.mark.parametrize("transport", ["files", "bundle_b64"])+ @pytest.mark.parametrize("path", ["/v1/execute", "/v1/start"])+ def test_a_method_tree_is_written_whole_as_the_callers_library_dir(self, mocker: MockerFixture, transport: str, path: str):+ """The tree is written as it was sent, its bundles and every other file, and handed over as the caller's own method."""client, snapshot = _build_client(mocker)- response = client.post("/v1/execute", json={"files": {"main.mthds": VALID_MTHDS}, "inputs": {"text": "hi"}})- assert response.status_code == 200- assert snapshot["mthds_contents"] == [VALID_MTHDS]- assert snapshot["library_dirs"] is None+ files = {"main.mthds": VALID_MTHDS, "notes/readme.txt": "a note"}+ body: dict[str, Any] = {"files": files} if transport == "files" else {"bundle_b64": _zip_b64(files)}- def test_zip_bundle_rides_mthds_contents(self, mocker: MockerFixture):+ response = client.post(path, json={**body, "inputs": {"text": "hi"}})++ assert response.status_code in {200, 202}, response.text+ assert snapshot["mthds_contents"] is None+ assert snapshot["files"] == ["main.mthds", "notes/readme.txt"]+ assert snapshot["library_dirs_are_callers"] is True+ assert snapshot["dir_exists"] is True+ # The entry pipe is the one `main_pipe` the bundles declare, qualified by its bundle's domain.+ assert snapshot["pipe_code"] == "smoke.echo"+ assert snapshot["methods_dirs"] is None+ # The tree is removed once the request returns.+ assert not Path(snapshot["library_dirs"][0]).exists()++ def test_a_request_pipe_code_overrides_the_entry_pipe(self, mocker: MockerFixture):client, snapshot = _build_client(mocker)- bundle = _zip_b64({"main.mthds": VALID_MTHDS})- response = client.post("/v1/execute", json={"bundle_b64": bundle, "inputs": {"text": "hi"}})- assert response.status_code == 200- assert snapshot["mthds_contents"] == [VALID_MTHDS]- assert snapshot["library_dirs"] is None+ response = client.post("/v1/execute", json={"files": {"main.mthds": VALID_MTHDS}, "pipe_code": "echo", "inputs": {"text": "hi"}})+ assert response.status_code == 200, response.text+ assert snapshot["pipe_code"] == "echo"++ def test_a_tree_with_no_entry_pipe_and_no_pipe_code_is_refused(self, mocker: MockerFixture):+ client, snapshot = _build_client(mocker)+ files = {"a.mthds": VALID_MTHDS, "b.mthds": VALID_MTHDS.replace('domain = "smoke"', 'domain = "other"')}++ response = client.post("/v1/execute", json={"files": files, "inputs": {"text": "hi"}})++ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] == "MethodEntryError"+ assert "'a.mthds'" in problem["detail"]+ assert "'b.mthds'" in problem["detail"]+ assert snapshot == {}, "the runner was reached"def test_no_bundle_passes_request_mthds_contents(self, mocker: MockerFixture):client, snapshot = _build_client(mocker)response = client.post("/v1/execute", json={"pipe_code": "echo", "mthds_contents": [VALID_MTHDS], "inputs": {}})assert response.status_code == 200assert snapshot["library_dirs"] is None+ assert snapshot["library_dirs_are_callers"] is Falseassert snapshot["mthds_contents"] == [VALID_MTHDS]def test_python_bundle_forbidden_when_not_hosted(self, mocker: MockerFixture):- mocker.patch("pipelex_api.routes.pipelex.pipeline.is_pipe_func_sandbox_hosted", return_value=False)- client, _ = _build_client(mocker)+ mocker.patch(f"{_TREE_NS}.is_pipe_func_sandbox_hosted", return_value=False)+ client, snapshot = _build_client(mocker)response = client.post("/v1/execute", json={"files": {"main.mthds": VALID_MTHDS, "funcs/pipe_func.py": _PIPE_FUNC_PY}})assert response.status_code == 403assert response.headers["content-type"] == "application/problem+json"assert response.json()["error_type"] == "CustomCodeRequiresSandbox"+ assert snapshot == {}, "the runner was reached"- def test_python_bundle_splits_mthds_from_py_when_hosted(self, mocker: MockerFixture):- """The key fix: `.mthds` → `mthds_contents` (main_pipe path), ONLY the `.py`- is materialized to the temp `library_dirs` for source capture.- """- mocker.patch("pipelex_api.routes.pipelex.pipeline.is_pipe_func_sandbox_hosted", return_value=True)+ def test_python_bundle_is_written_with_its_tree_when_hosted(self, mocker: MockerFixture):+ mocker.patch(f"{_TREE_NS}.is_pipe_func_sandbox_hosted", return_value=True)client, snapshot = _build_client(mocker)response = client.post("/v1/execute", json={"files": {"main.mthds": VALID_MTHDS, "funcs/pipe_func.py": _PIPE_FUNC_PY}})- assert response.status_code == 200- # The .mthds took the proven path, not the disk.- assert snapshot["mthds_contents"] == [VALID_MTHDS]- # Only the Python landed in the library dir.- assert snapshot["files"] == ["funcs/pipe_func.py"]- assert snapshot["dir_exists"] is True- # Temp dir is cleaned once the request returns.- assert not Path(snapshot["library_dirs"][0]).exists()+ assert response.status_code == 200, response.text+ assert snapshot["mthds_contents"] is None+ assert snapshot["files"] == ["funcs/pipe_func.py", "main.mthds"]def test_both_forms_rejected_via_route(self, mocker: MockerFixture):client, _ = _build_client(mocker)@@ -267,15 +295,21 @@ class TestPipelineBundle:assert response.status_code == 422assert response.json()["error_type"] == "InvalidBundle"+ def test_a_tree_holding_no_bundle_is_refused(self, mocker: MockerFixture):+ client, snapshot = _build_client(mocker)+ response = client.post("/v1/execute", json={"files": {"notes.txt": "x"}, "pipe_code": "smoke.echo", "inputs": {}})+ assert response.status_code == 422, response.text+ assert response.json()["error_type"] == "InvalidBundle"+ assert snapshot == {}, "the runner was reached"+@pytest.mark.parametrize(- ("files", "named"),+ ("files", "code", "named"),[- pytest.param({"main.mthds": VALID_MTHDS, "notes.txt": "\ud800"}, "'notes.txt'", id="in-content"),- pytest.param({"main.mthds": VALID_MTHDS, "notes\ud800.txt": "x"}, r"'notes\ud800.txt'", id="in-name"),+ pytest.param({"main.mthds": VALID_MTHDS, "notes\ud800.txt": "x"}, "invalid_tree_path", r"notes\ud800.txt", id="in-name"),],)- def test_a_files_entry_that_is_not_utf8_text_is_refused(self, mocker: MockerFixture, files: dict[str, str], named: str):- """A lone surrogate rides JSON as an escape but has no UTF-8 form, so the entry is refused, never written."""+ def test_a_files_key_that_is_not_utf8_text_is_refused(self, mocker: MockerFixture, files: dict[str, str], code: str, named: str):+ """A lone surrogate rides JSON as an escape but has no UTF-8 form, so the key is no tree path, never written."""client, snapshot = _build_client(mocker)# The escaped `\ud800` is valid JSON text, which `json=` would refuse to encode.raw_body = json.dumps({"files": files, "inputs": {"text": "hi"}})@@ -284,16 +318,21 @@ class TestPipelineBundle:assert response.status_code == 422, response.textproblem = response.json()- assert problem["error_type"] == "InvalidBundle"- assert named in problem["detail"]+ assert problem["error_type"] == "InvalidMethodFiles"+ assert problem["tree_refusal"] == {"code": code, "names": [named]}assert snapshot == {}, "the runner was reached"- def test_start_bundle_rides_mthds_contents(self, mocker: MockerFixture):+ def test_a_text_entry_that_is_not_utf8_text_is_refused(self, mocker: MockerFixture):client, snapshot = _build_client(mocker)- response = client.post("/v1/start", json={"files": {"main.mthds": VALID_MTHDS}, "inputs": {"text": "hi"}})- assert response.status_code == 202- assert snapshot["mthds_contents"] == [VALID_MTHDS]- assert snapshot["library_dirs"] is None+ raw_body = json.dumps({"files": {"main.mthds": VALID_MTHDS, "notes.txt": "\ud800"}, "inputs": {"text": "hi"}})++ response = client.post("/v1/execute", content=raw_body, headers={"content-type": "application/json"})++ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] == "InvalidMethodFiles"+ assert problem["tree_refusal"]["names"] == ["notes.txt"]+ assert snapshot == {}, "the runner was reached"def test_bundle_and_mthds_contents_are_mutually_exclusive(self, mocker: MockerFixture):client, _ = _build_client(mocker)@@ -304,22 +343,26 @@ class TestPipelineBundle:assert response.status_code == 422assert "mutually exclusive" in response.text- @pytest.mark.parametrize("transport", ["files", "bundle_b64"])+ @pytest.mark.parametrize("transport", ["dependencies", "bundle_b64"])@pytest.mark.parametrize("path", ["/v1/execute", "/v1/start"])- def test_a_vendored_package_gets_its_own_methods_dir(self, mocker: MockerFixture, transport: str, path: str):- """A package under `.mthds/methods/<name>/` leaves `mthds_contents` and the library dir for a methods dir of its own."""- mocker.patch("pipelex_api.routes.pipelex.pipeline.is_pipe_func_sandbox_hosted", return_value=True)+ def test_a_dependency_gets_its_own_methods_dir(self, mocker: MockerFixture, transport: str, path: str):+ """A dependency tree, sent in `dependencies` or shipped in an archive's store, is written into a methods dir of its own."""+ mocker.patch(f"{_TREE_NS}.is_pipe_func_sandbox_hosted", return_value=True)client, snapshot = _build_client(mocker)- files = {**VENDORED_PROBE_FILES, "pipe_func.py": _PIPE_FUNC_PY}- body: dict[str, Any] = {"files": files} if transport == "files" else {"bundle_b64": _zip_b64(files)}+ body: dict[str, Any]+ if transport == "dependencies":+ body = {+ "files": {"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS, "pipe_func.py": _PIPE_FUNC_PY},+ "dependencies": {VENDORED_PROBE_ADDRESS: _PROBE_TREE},+ }+ else:+ body = {"bundle_b64": _zip_b64({**VENDORED_PROBE_FILES, "pipe_func.py": _PIPE_FUNC_PY})}response = client.post(path, json={**body, "inputs": {"text": "hi"}})assert response.status_code in {200, 202}, response.text- # The bundle's own file alone is the content; the package's bundle is not the caller's own domain.- assert snapshot["mthds_contents"] == [UNRESOLVED_PACKAGE_MTHDS]- # The library dir holds the bundle's own Python, and nothing of the package.- assert snapshot["files"] == ["pipe_func.py"]+ # The library dir holds the method's own tree, and nothing of the package.+ assert snapshot["files"] == ["bundle.mthds", "pipe_func.py"]# The package is in its own methods dir, laid out as `<name>/…`, a path no walk-up could take for a store.(methods_dir,) = snapshot["methods_dirs"]assert snapshot["methods_files"] == [["probe/METHODS.toml", "probe/probe.mthds"]]@@ -327,75 +370,110 @@ class TestPipelineBundle:# Cleaned once the request returns, like the library dir.assert not Path(methods_dir).exists()+ def test_a_dependency_keyed_by_another_address_than_its_manifests_is_refused(self, mocker: MockerFixture):+ client, snapshot = _build_client(mocker)+ body = {"files": {"bundle.mthds": UNRESOLVED_PACKAGE_MTHDS}, "dependencies": {"github.com/invented/elsewhere/probe": _PROBE_TREE}}++ response = client.post("/v1/execute", json={**body, "inputs": {"text": "hi"}})++ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] == "InvalidMethodFiles"+ assert VENDORED_PROBE_ADDRESS in problem["detail"]+ assert snapshot == {}, "the runner was reached"++ def test_dependencies_without_files_are_refused(self, mocker: MockerFixture):+ client, snapshot = _build_client(mocker)+ body = {"mthds_contents": [UNRESOLVED_PACKAGE_MTHDS], "dependencies": {VENDORED_PROBE_ADDRESS: _PROBE_TREE}}++ response = client.post("/v1/execute", json={**body, "inputs": {"text": "hi"}})++ assert response.status_code == 422, response.text+ assert "dependencies" in response.json()["detail"]+ assert snapshot == {}, "the runner was reached"+def test_a_bundle_shipping_no_package_gets_no_methods_dir(self, mocker: MockerFixture):client, snapshot = _build_client(mocker)response = client.post("/v1/execute", json={"files": {"main.mthds": VALID_MTHDS}, "inputs": {"text": "hi"}})assert response.status_code == 200assert snapshot["methods_dirs"] is None- @pytest.mark.parametrize("transport", ["files", "bundle_b64"])- @pytest.mark.parametrize("shape", list(_REFUSED_VENDORING_SHAPES))- def test_a_misplaced_mthds_entry_is_refused(self, mocker: MockerFixture, transport: str, shape: str):- """A `.mthds` path no reference could find is refused with the layout to use, never dropped."""- files, named = _REFUSED_VENDORING_SHAPES[shape]+ @pytest.mark.parametrize("shape", list(_REFUSED_ARCHIVE_SHAPES))+ def test_a_misplaced_mthds_entry_of_an_archive_is_refused_by_the_tree_rule(self, mocker: MockerFixture, shape: str):+ """An archive entry under a dot directory that is no package of its store is refused by the tree's rule, never dropped."""+ files, named = _REFUSED_ARCHIVE_SHAPES[shape]client, snapshot = _build_client(mocker)- body: dict[str, Any] = {"files": files} if transport == "files" else {"bundle_b64": _zip_b64(files)}- response = client.post("/v1/execute", json={**body, "inputs": {"text": "hi"}})+ response = client.post("/v1/execute", json={"bundle_b64": _zip_b64(files), "inputs": {"text": "hi"}})assert response.status_code == 422, response.textproblem = response.json()- assert problem["error_type"] == "InvalidBundle"+ assert problem["error_type"] == "InvalidMethodFiles"assert named in problem["detail"]- assert ".mthds/methods/<name>/" in problem["detail"]assert snapshot == {}, "the runner was reached"- @pytest.mark.parametrize("transport", ["files", "bundle_b64"])+ @pytest.mark.parametrize("shape", list(_REFUSED_ARCHIVE_PACKAGES))+ def test_a_shipped_package_with_no_valid_manifest_is_refused(self, mocker: MockerFixture, shape: str):+ files, named = _REFUSED_ARCHIVE_PACKAGES[shape]+ client, snapshot = _build_client(mocker)++ response = client.post("/v1/execute", json={"bundle_b64": _zip_b64(files), "inputs": {"text": "hi"}})++ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] == "InvalidMethodFiles"+ assert named in problem["detail"]+ assert snapshot == {}, "the runner was reached"++ def test_an_archive_whose_only_bundles_are_shipped_packages_is_refused(self, mocker: MockerFixture):+ client, snapshot = _build_client(mocker)+ files = {".mthds/methods/probe/METHODS.toml": VENDORED_PROBE_MANIFEST, ".mthds/methods/probe/probe.mthds": VENDORED_PROBE_MTHDS}++ response = client.post("/v1/execute", json={"bundle_b64": _zip_b64(files), "inputs": {"text": "hi"}})++ assert response.status_code == 422, response.text+ problem = response.json()+ assert problem["error_type"] in {"InvalidBundle", "InvalidMethodFiles"}+ assert snapshot == {}, "the runner was reached"+@pytest.mark.parametrize("bundle", list(_DUPLICATE_IDENTITY_BUNDLES))- def test_two_shipped_packages_of_one_identity_are_refused(self, mocker: MockerFixture, transport: str, bundle: str):- """Which of two packages a reference would match is no choice to make by directory order: the bundle is refused."""- files, directories = _DUPLICATE_IDENTITY_BUNDLES[bundle]+ def test_two_shipped_packages_of_one_identity_are_refused(self, mocker: MockerFixture, bundle: str):+ """Which of two packages a reference would match is no choice to make by directory order: the archive is refused."""+ files = _DUPLICATE_IDENTITY_BUNDLES[bundle]client, snapshot = _build_client(mocker)- body: dict[str, Any] = {"files": files} if transport == "files" else {"bundle_b64": _zip_b64(files)}- response = client.post("/v1/execute", json={**body, "inputs": {"text": "hi"}})+ response = client.post("/v1/execute", json={"bundle_b64": _zip_b64(files), "inputs": {"text": "hi"}})assert response.status_code == 422, response.textproblem = response.json()- assert problem["error_type"] == "InvalidBundle"+ assert problem["error_type"] == "InvalidMethodFiles"assert VENDORED_PROBE_ADDRESS.casefold() in problem["detail"].casefold()- for directory in directories:- assert f"'.mthds/methods/{directory}/'" in problem["detail"]assert snapshot == {}, "the runner was reached"@pytest.mark.parametrize("transport", ["files", "bundle_b64"])- @pytest.mark.parametrize("case", list(_COLLIDING_ENTRIES))- def test_entries_colliding_on_one_path_are_refused(self, mocker: MockerFixture, transport: str, case: str):- """Two entries of one file would let a check read one copy while the last one written lands on disk: refused."""- entries, named = _COLLIDING_ENTRIES[case]+ @pytest.mark.parametrize("case", list(_REFUSED_ENTRIES))+ def test_an_entry_the_tree_rule_refuses_is_refused_naming_it(self, mocker: MockerFixture, transport: str, case: str):+ """A key that is not a tree path, one the membership rule keeps out, or two entries of one path: refused, never written."""+ entries, code, named = _REFUSED_ENTRIES[case]+ if transport == "bundle_b64" and code == "excluded_name" and named.startswith(".mthds/methods/"):+ pytest.skip("An archive ships packages in the local store layout.")client, snapshot = _build_client(mocker)body: dict[str, Any] = {"files": dict(entries)} if transport == "files" else {"bundle_b64": _zip_b64_of_entries(entries)}response = client.post("/v1/execute", json={**body, "inputs": {"text": "hi"}})- assert response.status_code == 422, response.text- problem = response.json()- assert problem["error_type"] == "InvalidBundle"- assert named in problem["detail"]+ _assert_tree_refusal(response, code=code, named=named)assert snapshot == {}, "the runner was reached"def test_a_zip_member_named_twice_is_refused(self, mocker: MockerFixture):- """A zip may hold one member name twice, spelled the same."""+ """A zip may hold one member name twice, spelled the same: a duplicate key of the map it is read as."""client, snapshot = _build_client(mocker)with pytest.warns(UserWarning, match="Duplicate name"):bundle_b64 = _zip_b64_of_entries([("main.mthds", VALID_MTHDS), ("notes.txt", "a"), ("notes.txt", "b")])response = client.post("/v1/execute", json={"bundle_b64": bundle_b64, "inputs": {"text": "hi"}})- assert response.status_code == 422, response.text- problem = response.json()- assert problem["error_type"] == "InvalidBundle"- assert "'notes.txt'" in problem["detail"]+ _assert_tree_refusal(response, code="duplicate_key", named="notes.txt")assert snapshot == {}, "the runner was reached"def test_two_shipped_packages_of_one_repository_are_both_shipped(self, mocker: MockerFixture):@@ -408,7 +486,7 @@ class TestPipelineBundle:)client, snapshot = _build_client(mocker)- response = client.post("/v1/execute", json={"files": files, "inputs": {"text": "hi"}})+ response = client.post("/v1/execute", json={"bundle_b64": _zip_b64(files), "inputs": {"text": "hi"}})assert response.status_code == 200, response.text- assert snapshot["methods_files"] == [["probe/METHODS.toml", "probe/probe.mthds", "sibling/METHODS.toml", "sibling/probe.mthds"]]+ assert snapshot["methods_files"] == [["probe/METHODS.toml", "probe/probe.mthds"], ["sibling/METHODS.toml", "sibling/probe.mthds"]]diff --git a/api/tests/unit/test_validate_caller_identity.py b/api/tests/unit/test_validate_caller_identity.pyindex c7da86257..8d1363b6c 100644--- a/api/tests/unit/test_validate_caller_identity.py+++ b/api/tests/unit/test_validate_caller_identity.py@@ -58,13 +58,15 @@ class _RecordingBundleValidator:async def validate_bundles(self,*,- mthds_contents: list[str], # noqa: ARG002+ mthds_contents: list[str] | None, # noqa: ARG002mthds_sources: list[str] | None, # noqa: ARG002allow_signatures: bool, # noqa: ARG002library_dirs: Sequence[Path] | None, # noqa: ARG002caller_identity: CallerIdentity | None,graph_pipe_code: str | None, # noqa: ARG002submitted_files: Mapping[str, bytes] | None, # noqa: ARG002+ method_dir: Path | None, # noqa: ARG002+ methods_dirs: Sequence[Path] | None, # noqa: ARG002) -> ErrorReport:self.caller_identities.append(caller_identity)return ErrorReport(diff --git a/api/tests/unit/test_validate_default_pipe_ref.py b/api/tests/unit/test_validate_default_pipe_ref.pyindex 158ad78ff..f8ae4b066 100644--- a/api/tests/unit/test_validate_default_pipe_ref.py+++ b/api/tests/unit/test_validate_default_pipe_ref.py@@ -3,9 +3,10 @@The canonical `PipelexValidationReport` is manifest-blind: its `bundle_blueprint` is the batch'sprimary blueprint, so a package whose `METHODS.toml` names an entry pipe the closure would notdefault to validates with a report from which a consumer can only derive the WRONG entry pipe, or-none. These tests pin the route's own field, which applies the run routes' precedence — the fetched-manifest's `main_pipe`, else the closure's primary blueprint — so a by-address consumer projects a-signature for the pipe `/execute` actually defaults to.+none. These tests pin the route's own field, which applies the run routes' rule — for a method+tree, the MTHDS protocol's entry rule (the manifest's `main_pipe`, else the one `main_pipe` its+bundles declare); for inline contents, the closure's primary blueprint — so a by-address consumer+projects a signature for the pipe `/execute` actually defaults to."""from collections.abc import Callable@@ -105,12 +106,16 @@ class TestValidateDefaultPipeRef:assert response.status_code == 200, response.textassert response.json()["default_pipe_ref"] == "smoke.echo"- def test_a_manifest_without_main_pipe_falls_back_to_the_closures_declaration(self, install_method_package: Callable[..., Path]):+ def test_a_manifest_without_main_pipe_reports_null(self, install_method_package: Callable[..., Path]):+ # The protocol's entry rule: a method whose manifest declares no `main_pipe` has no pipe to start at, whatever+ # its bundles declare, so a selector-less run is refused and the field says nothing.install_method_package(files={"documents.mthds": VALID_MTHDS}, manifest_toml=STUB_METHOD_MANIFEST_NO_MAIN_PIPE)client = _build_client()response = client.post("/v1/validate", json={"method_ref": _METHOD_REF})assert response.status_code == 200, response.text- assert response.json()["default_pipe_ref"] == "smoke.echo"+ body = response.json()+ assert body["is_valid"] is True+ assert body["default_pipe_ref"] is Nonedef test_a_manifest_naming_a_pipe_the_closure_lacks_reports_null(self, install_method_package: Callable[..., Path]):# The manifest names `shout`; the package's closure holds only `smoke.echo`. A selector-less@@ -124,13 +129,13 @@ class TestValidateDefaultPipeRef:assert body["is_valid"] is Trueassert body["default_pipe_ref"] is None- def test_an_ambiguous_manifest_bare_code_reports_null(self, install_method_package: Callable[..., Path]):- # The manifest names `echo`, declared by both `smoke` and `twin`. The engine's entry lookup- # raises on an ambiguous bare code rather than picking a winner, so the field says nothing.+ def test_a_manifest_bare_code_two_domains_declare_resolves_in_the_domain_exporting_it(self, install_method_package: Callable[..., Path]):+ # The manifest names `echo`, declared by both `smoke` and `twin`, and exports it from `smoke` alone: the domain+ # exporting the entry pipe says which bundle defines it, so the run starts at `smoke.echo`.install_method_package(files={"smoke.mthds": VALID_MTHDS, "twin.mthds": COLLIDING_ECHO_LIST_MTHDS})client = _build_client()response = client.post("/v1/validate", json={"method_ref": _METHOD_REF})assert response.status_code == 200, response.textbody = response.json()assert body["is_valid"] is True- assert body["default_pipe_ref"] is None+ assert body["default_pipe_ref"] == "smoke.echo"diff --git a/api/tests/unit/test_validate_dispatch.py b/api/tests/unit/test_validate_dispatch.pyindex b896b66f1..c6d5044da 100644--- a/api/tests/unit/test_validate_dispatch.py+++ b/api/tests/unit/test_validate_dispatch.py@@ -43,13 +43,15 @@ class _StubBundleValidator:async def validate_bundles(self,*,- mthds_contents: list[str],+ mthds_contents: list[str] | None,mthds_sources: list[str] | None,allow_signatures: bool,library_dirs: Sequence[Path] | None,caller_identity: CallerIdentity | None,graph_pipe_code: str | None,submitted_files: Mapping[str, bytes] | None,+ method_dir: Path | None,+ methods_dirs: Sequence[Path] | None,) -> ErrorReport:self.calls.append({@@ -60,6 +62,8 @@ class _StubBundleValidator:"caller_identity": caller_identity,"graph_pipe_code": graph_pipe_code,"submitted_files": submitted_files,+ "method_dir": method_dir,+ "methods_dirs": methods_dirs,})if self._error is not None:diff --git a/api/tests/unit/test_validate_graph_target.py b/api/tests/unit/test_validate_graph_target.pyindex 8ae4338d1..c873f8cc1 100644--- a/api/tests/unit/test_validate_graph_target.py+++ b/api/tests/unit/test_validate_graph_target.py@@ -1,11 +1,11 @@"""The /validate graph is drawn from the pipe a selector-less run of the request would execute.-The runtime's graph arm defaults to the primary blueprint's `main_pipe`, which is manifest-blind:-a fetched package whose `METHODS.toml` names the entry pipe would be graphed from another pipe, or-not at all when its bundles declare no `main_pipe` of their own. The route hands the runtime the-manifest's `main_pipe` as the graph target, so `graph_spec` and `default_pipe_ref` name the same-pipe. These tests read the target twice: off the graph the real in-process validator draws, and-off the call a dispatched validator receives.+For inline contents the runtime's graph arm draws the primary blueprint's `main_pipe`. A method+tree, fetched by `method_ref` or sent as `files`, starts at its entry pipe by the MTHDS protocol's+rule, the manifest's `main_pipe` else the one its bundles declare: the route hands the runtime that+pipe as the graph target, so `graph_spec` and `default_pipe_ref` name the same pipe, and a tree+with no entry pipe gets neither. These tests read the target twice: off the graph the real+in-process validator draws, and off the call a dispatched validator receives."""from collections.abc import Callable, Mapping, Sequence@@ -46,13 +46,15 @@ class _RecordingBundleValidator:async def validate_bundles(self,*,- mthds_contents: list[str], # noqa: ARG002+ mthds_contents: list[str] | None, # noqa: ARG002mthds_sources: list[str] | None, # noqa: ARG002allow_signatures: bool, # noqa: ARG002library_dirs: Sequence[Path] | None, # noqa: ARG002caller_identity: CallerIdentity | None, # noqa: ARG002graph_pipe_code: str | None,submitted_files: Mapping[str, bytes] | None, # noqa: ARG002+ method_dir: Path | None, # noqa: ARG002+ methods_dirs: Sequence[Path] | None, # noqa: ARG002) -> ErrorReport:self.graph_pipe_codes.append(graph_pipe_code)return ErrorReport(@@ -118,14 +120,19 @@ class TestValidateGraphTarget:assert body["default_pipe_ref"] == "other.shout"assert _graphed_pipe_ref(body) == "other.shout"- def test_a_manifest_without_main_pipe_graphs_the_closures_declaration(self, install_method_package: Callable[..., Path]) -> None:+ def test_a_manifest_without_main_pipe_gets_no_graph(self, install_method_package: Callable[..., Path]) -> None:+ # The protocol's entry rule: a manifest declaring no `main_pipe` leaves the method no pipe to start at, whatever+ # its bundles declare, so no pipe is drawn.install_method_package(files={"documents.mthds": VALID_MTHDS}, manifest_toml=STUB_METHOD_MANIFEST_NO_MAIN_PIPE)client = _build_client()response = client.post("/v1/validate", json={"method_ref": _METHOD_REF})assert response.status_code == 200, response.text- assert _graphed_pipe_ref(response.json()) == "smoke.echo"+ body = response.json()+ assert body["is_valid"] is True+ assert body["default_pipe_ref"] is None+ assert body["graph_spec"] is Nonedef test_a_manifest_naming_a_pipe_the_closure_lacks_gets_no_graph(self, install_method_package: Callable[..., Path]) -> None:# The manifest names `shout`; the closure holds only `smoke.echo`. No run by this address@@ -150,11 +157,9 @@ class TestValidateGraphTarget:assert response.status_code == 200, response.textassert _graphed_pipe_ref(response.json()) == "smoke.echo"- def test_the_dispatched_validator_receives_the_manifests_bare_code(- self, mocker: MockerFixture, install_method_package: Callable[..., Path]- ) -> None:+ def test_the_dispatched_validator_receives_the_trees_entry_pipe(self, mocker: MockerFixture, install_method_package: Callable[..., Path]) -> None:# A worker-dispatched validator draws its graph on the worker, so the target must travel- # through the seam; it is the manifest's bare code, resolved there as a run resolves it.+ # through the seam; it is the tree's entry pipe, qualified by the bundle that defines it.install_method_package(files={"documents.mthds": NO_MAIN_PIPE_MTHDS}, manifest_toml=STUB_METHOD_MANIFEST_NOMAIN_ENTRY)validator = _register_recording_validator(mocker)client = _build_client()@@ -162,7 +167,7 @@ class TestValidateGraphTarget:response = client.post("/v1/validate", json={"method_ref": _METHOD_REF})assert response.status_code == 200, response.text- assert validator.graph_pipe_codes == ["echo"]+ assert validator.graph_pipe_codes == ["nomain.echo"]def test_the_dispatched_validator_keeps_its_default_target_for_inline_contents(self, mocker: MockerFixture) -> None:validator = _register_recording_validator(mocker)diff --git a/api/tests/unit/test_vendored_method_packages.py b/api/tests/unit/test_vendored_method_packages.pyindex 7ad968a05..708d9e4a1 100644--- a/api/tests/unit/test_vendored_method_packages.py+++ b/api/tests/unit/test_vendored_method_packages.py@@ -1,9 +1,10 @@-"""A request carries the method packages its bundle calls, under `.mthds/methods/<name>/` at the bundle's root.+"""A request carries the method packages its method calls: as `dependencies` beside `files`, or under `.mthds/methods/<name>/` in an archive.-The routes and the runner are real, and so is the in-process orchestrator of the run tests: what the run answers is-what the engine resolved. Fetch-on-miss is off and the installed store is an empty temporary directory, so the shipped-package is the only one that can answer, or, where a test installs a competing copy, the one that must win. The-invented package's entry composes a text naming its copy, which a dry run renders too, so the output tells which ran.+A fetched package carries them the second way, in its repository. The routes and the runner are real, and so is the+in-process orchestrator of the run tests: what the run answers is what the engine resolved. Fetch-on-miss is off and+the installed store is an empty temporary directory, so the shipped package is the only one that can answer, or, where+a test installs a competing copy, the one that must win. The invented package's entry composes a text naming its copy,+which a dry run renders too, so the output tells which ran."""import base64@@ -27,8 +28,9 @@ from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeRfrom pytest_mock import MockerFixturefrom pipelex_api.api_config import ApiConfig+from pipelex_api.errors import ApiErrorfrom pipelex_api.exception_handlers import register_exception_handlers-from pipelex_api.method_source import fetch_method_mthds_files, fetched_method_source+from pipelex_api.method_source import fetch_method_mthds_files, fetched_methodfrom pipelex_api.routes import router as api_routerfrom tests.unit._constants import (STUB_METHOD_ADDRESS,@@ -131,8 +133,18 @@ def _run_output(body: dict[str, Any]) -> str:return json.dumps(body["pipe_output"]["working_memory"])+_PROBE_PREFIX = ".mthds/methods/probe/"+++def _tree_body(files: dict[str, str]) -> dict[str, Any]:+ """The method of the files in the store's layout as a request sends it: its own `files`, and the probe in `dependencies`."""+ own = {tree_path: content for tree_path, content in files.items() if not tree_path.startswith(_PROBE_PREFIX)}+ probe = {tree_path.removeprefix(_PROBE_PREFIX): content for tree_path, content in files.items() if tree_path.startswith(_PROBE_PREFIX)}+ return {"files": own, "dependencies": {VENDORED_PROBE_ADDRESS: probe}}++def _execute_vendored(client: TestClient, *, files: dict[str, str]) -> Any:- return client.post("/v1/execute", json={"files": files, "inputs": {"text": "hello"}})+ return client.post("/v1/execute", json={**_tree_body(files), "inputs": {"text": "hello"}})def _zip_b64(files: Mapping[str, str | bytes]) -> str:@@ -182,7 +194,7 @@ class TestVendoredMethodPackages:orchestrator = _RecordingOrchestrator()client = _build_client(mocker, tmp_path=tmp_path, orchestrator=orchestrator)- response = client.post("/v1/start", json={"files": VENDORED_PROBE_FILES, "inputs": {"text": "hello"}})+ response = client.post("/v1/start", json={**_tree_body(VENDORED_PROBE_FILES), "inputs": {"text": "hello"}})assert response.status_code == 202, response.textassert orchestrator.dispatches == ["echo"]@@ -214,7 +226,7 @@ class TestVendoredMethodPackages:client = _build_client(mocker, tmp_path=tmp_path, orchestrator=orchestrator)files = {**VENDORED_PROBE_FILES, ".mthds/methods/probe/structures.py": _structures_py(sentinel=sentinel)}- response = client.post("/v1/start", json={"files": files, "inputs": {"text": "hello"}})+ response = client.post("/v1/start", json={**_tree_body(files), "inputs": {"text": "hello"}})assert response.status_code == 403, response.textbody = response.json()@@ -251,37 +263,65 @@ class TestVendoredMethodPackages:assert response.status_code == 422, response.textbody = response.json()- assert body["error_type"] == "InvalidBundle"- assert f"Method package '{STUB_METHOD_ADDRESS}'" in body["detail"]+ assert body["error_type"] == "InvalidMethodFiles"+ assert f"method package '{STUB_METHOD_ADDRESS}'" in body["detail"]assert VENDORED_PROBE_ADDRESS in body["detail"]- assert "'.mthds/methods/a/'" in body["detail"]assert "'.mthds/methods/b/'" in body["detail"]def test_a_method_ref_package_keeps_its_shipped_files_apart(self, install_method_package: Callable[..., Path]):install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **_package_vendoring_files()})- with fetched_method_source(_METHOD_REF) as fetched:- assert fetched.mthds_sources == ["smoke.mthds"]- assert fetched.mthds_contents == [UNRESOLVED_PACKAGE_MTHDS]- assert fetched.library_dirs is None- assert fetched.methods_dirs is not None- (methods_dir,) = fetched.methods_dirs- assert sorted(path.relative_to(methods_dir).as_posix() for path in methods_dir.rglob("*") if path.is_file()) == [- "probe/METHODS.toml",- "probe/probe.mthds",- ]- assert not methods_dir.exists()+ fetched = fetched_method(_METHOD_REF)++ assert sorted(fetched.request_method.tree.contents) == ["METHODS.toml", "smoke.mthds"]+ assert list(fetched.request_method.dependencies) == [VENDORED_PROBE_ADDRESS]+ assert sorted(fetched.request_method.dependencies[VENDORED_PROBE_ADDRESS].contents) == ["METHODS.toml", "probe.mthds"]+ assert fetched.provenance.tree_digest == fetched.request_method.tree.tree.digest++ tooling = fetch_method_mthds_files(_METHOD_REF)+ assert [item.source for item in tooling.files] == ["smoke.mthds"]++ def test_a_dot_directory_under_the_shipped_packages_is_no_package_of_a_method_ref(self, install_method_package: Callable[..., Path]):+ """A fetched package and an archive hold `.mthds/methods/<name>/` to one rule: a dot directory there is no package.++ A copy of the probe under `.mthds/methods/.cache/` would otherwise declare the probe's identity a second time.+ """+ hidden_copy = {+ f".mthds/methods/.cache/{relpath.removeprefix('.mthds/methods/probe/')}": content+ for relpath, content in _package_vendoring_files().items()+ }+ install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **_package_vendoring_files(), **hidden_copy})++ fetched = fetched_method(_METHOD_REF)++ assert list(fetched.request_method.dependencies) == [VENDORED_PROBE_ADDRESS]++ def test_the_tooling_read_keeps_the_tree_s_bundles_alone(self, install_method_package: Callable[..., Path]):+ """The tooling routes read the `.mthds` files the tree keeps, without the shipped packages a run needs whole."""+ no_manifest = {".mthds/methods/unfinished/draft.mthds": UNRESOLVED_PACKAGE_MTHDS}+ install_method_package(+ files={+ "smoke.mthds": UNRESOLVED_PACKAGE_MTHDS,+ "ignored/old.mthds": UNRESOLVED_PACKAGE_MTHDS,+ ".gitignore": "ignored/\n",+ "notes/readme.txt": "Not read by the tooling routes",+ **no_manifest,+ }+ )tooling = fetch_method_mthds_files(_METHOD_REF)+assert [item.source for item in tooling.files] == ["smoke.mthds"]+ assert tooling.main_pipe == "echo"+ with pytest.raises(ApiError, match="unfinished"):+ fetched_method(_METHOD_REF)- def test_validate_by_method_ref_validates_only_the_package_s_own_content(+ def test_validate_by_method_ref_resolves_the_dependency_it_ships(self, mocker: MockerFixture, tmp_path: Path, install_method_package: Callable[..., Path]):- """`/v1/validate` leaves a shipped package out of the content; the dependency resolves from the installed store there."""+ """`/v1/validate` loads the package's tree with the dependency it ships, as a run does, and echoes its own bundles only."""install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **_package_vendoring_files()})client = _build_client(mocker, tmp_path=tmp_path)- _install_competing_probe(tmp_path=tmp_path)response = client.post("/v1/validate", json={"method_ref": _METHOD_REF})@@ -290,10 +330,10 @@ class TestVendoredMethodPackages:assert body["is_valid"] is True, bodyassert body["mthds_contents"] == [UNRESOLVED_PACKAGE_MTHDS]- def test_validate_by_method_ref_does_not_search_the_shipped_package(+ def test_validate_by_method_ref_without_the_dependency_is_an_invalid_verdict(self, mocker: MockerFixture, tmp_path: Path, install_method_package: Callable[..., Path]):- install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS, **_package_vendoring_files()})+ install_method_package(files={"smoke.mthds": UNRESOLVED_PACKAGE_MTHDS})client = _build_client(mocker, tmp_path=tmp_path)response = client.post("/v1/validate", json={"method_ref": _METHOD_REF})diff --git a/docs/api-server/configuration.md b/docs/api-server/configuration.mdindex 8668f71a5..9967c80ed 100644--- a/docs/api-server/configuration.md+++ b/docs/api-server/configuration.md@@ -48,6 +48,27 @@ PIPELEX_ENV=dev# MAX_RESOURCE_FILE_KIB=5120 # decoded bytes per file (default 5 MiB)# MAX_RESOURCES_TOTAL_KIB=10240 # decoded bytes across the request (default 10 MiB)+# ── Methods sent as trees, and the blob store (`files`, `/v1/blobs/*`) ─────+# See pipe-run.md → "Sending a method as a tree" and "Sending files by hash".+# Each tree of a request is bounded on its own; over any limit is a 422+# MethodTreeTooLarge. All read at startup.+# MAX_METHOD_TREE_FILES=1000 # files in one tree+# MAX_METHOD_TREE_FILE_KIB=32768 # bytes per file, and per uploaded blob (default 32 MiB)+# MAX_METHOD_TREE_TOTAL_KIB=65536 # bytes across one tree (default 64 MiB)+# A method and the dependency trees it sends beside it are also bounded together:+# MAX_METHOD_DEPENDENCIES=32 # dependency trees in one request+# MAX_METHOD_REQUEST_TOTAL_KIB=131072 # bytes across the method's trees (default 128 MiB)+# BLOB_UPLOAD_URL_TTL_SECONDS=3600 # how long an upload target stays valid+# The address clients reach this API at, which upload targets start with. Set it+# behind a proxy that terminates TLS or serves the API under a path prefix, since+# the URL a request arrives at is then not the one a client can upload to. Unset,+# a target starts with the URL of the request that asked for it.+# PUBLIC_BASE_URL=https://runner.example.com+# The key the upload targets are signed with. Unset, each process draws its own,+# so a deployment serving one API through several instances sets it, the same on+# every instance. Blobs live in the configured storage under <scope>/blobs/.+# BLOB_UPLOAD_SIGNING_KEY=<a-long-random-secret>+# ── Running a method by address (`method_ref`) ────────────────────────────# See pipe-run.md → "Running a method by address". All read at startup.#diff --git a/docs/api-server/error-responses.md b/docs/api-server/error-responses.mdindex 443ed0351..dce911102 100644--- a/docs/api-server/error-responses.md+++ b/docs/api-server/error-responses.md@@ -35,6 +35,9 @@ A failure response looks like this:- `error_category` — finer classification when the originating error provides one (currently used by inference errors — see [`InferenceErrorCategory`](https://docs.pipelex.com/latest/errors/) upstream).- `user_action` — structured suggestion of what the caller should do next, when the error can author one.- `model`, `provider`, `provider_metadata` — populated when the failure originated in an upstream inference call. **Stripped under STRICT disclosure** (see [Disclosure modes](#disclosure-modes)).+- `tree_refusal` — on a **422** `InvalidMethodFiles`, the MTHDS standard's refusal of a method tree the request sent: `{"code": "<refusal code>", "names": [...]}`, the code one of the standard's tree refusal codes, such as `invalid_tree_path`, `duplicate_key`, `excluded_name` or `malformed_entry`, and `names` the keys, paths or addresses it refuses. A name holding a code point UTF-8 cannot encode, such as a lone surrogate a JSON escape can carry, is written with that code point backslash-escaped. See [Pipe Run](pipe-run.md#sending-a-method-as-a-tree).+- `missing_blobs` — on a **422** `MissingBlobs`, the SHA-256 hashes the request named by hash entries that this runner does not hold at their stated size, to upload through the blob routes before sending the request again. See [Sending files by hash](pipe-run.md#sending-files-by-hash).+- `mismatched_blobs` — on a **422** `BlobMismatch` refusing a request's hash entries, each hash this runner holds at another size than its entry states, as `{"sha256": "…", "held_size": n}`: the entry's size is the one to correct, since uploading the bytes again would not change the size they are held at.- `validation_errors` — structured per-error list carried by a `ValidateBundleError`. On the run routes (`/execute`, `/start`) it rides this 422 problem document; on the diagnostic routes (`/validate`, `/resolve`, `/codegen`, `/pipe-io`) it rides the **200 `is_valid: false`** body instead (see [Pipe Validate](pipe-validate.md)). See [Structured validation errors](#structured-validation-errors).## Structured validation errors@@ -157,14 +160,14 @@ A few specific statuses bypass the domain mapping:- **400** — a well-formed request this deployment cannot serve. `error_type = "StartRequiresAsyncOrchestration"`: `POST /v1/start` is fire-and-forget by nature, and this deployment's orchestrator is blocking-only (the in-process `direct` default), so it refuses honestly rather than blocking and acking — use `POST /v1/execute` instead.- **400** — `error_type = "BadRequest"`: the client disconnected before its request body was received in full. The client is gone before the answer, so this one is seen in the server's logs, at warning level.- **401** — missing/invalid bearer token. `WWW-Authenticate: Bearer` is set. Only reachable when the deployment enables auth (`AUTH_MODE=api_key` or `AUTH_MODE=jwt`).-- **403** — authenticated but not authorized, or refused by deployment policy. The cases: a storage-ownership mismatch; `error_type = "OrchestrationModeOverrideForbidden"` — the request asked for an `orchestration_mode` this deployment does not allow overriding per request (`allow_request_orchestration_mode_override = false`); `error_type = "CustomCodeRequiresSandbox"` — the method (an uploaded bundle, or a fetched `method_ref` package) ships custom Python `.py` files and this deployment is not sandbox-hosted, so running caller-supplied code in-process is refused; `error_type = "MethodStructuresRefusedError"` — a fetched `method_ref` package declares in-process Python structure classes (`StructuredContent` subclasses), which hosted execution always refuses, sandbox or not — express the types as MTHDS concepts with inline structures instead (see [Pipe Run](pipe-run.md#running-a-method-by-address-method_ref)).+- **403** — authenticated but not authorized, or refused by deployment policy. The cases: a storage-ownership mismatch; `error_type = "OrchestrationModeOverrideForbidden"` — the request asked for an `orchestration_mode` this deployment does not allow overriding per request (`allow_request_orchestration_mode_override = false`); `error_type = "BlobUploadNotAuthorized"` — a `PUT /v1/blobs/{sha256}` without a valid upload target: its signature is missing, forged, expired, or signed for another hash or scope, so ask `POST /v1/blobs/missing` for a fresh target; `error_type = "CustomCodeRequiresSandbox"` — the method (a tree sent with the request, or a fetched `method_ref` package) holds custom Python `.py` files and this deployment is not sandbox-hosted, so running caller-supplied code in-process is refused; `error_type = "MethodStructuresRefusedError"` — a fetched `method_ref` package declares in-process Python structure classes (`StructuredContent` subclasses), which hosted execution always refuses, sandbox or not — express the types as MTHDS concepts with inline structures instead (see [Pipe Run](pipe-run.md#running-a-method-by-address-method_ref)).- **404** — `error_type = "MethodPackageNotFoundError"`: the `method_ref` repository was fetched, but no package in it matches the requested address by manifest identity. The `detail` lists the package addresses the repository does contain, so the caller can correct the selector.- **409** — `error_type = "PipelineManagerAlreadyExistsError"`: the submitted `pipeline_run_id` is already registered for a run that is still in flight on this server. Completed and failed runs free their id, so this only fires for genuinely concurrent duplicates — resubmit after the in-flight run finishes, or pick a fresh id. Only `POST /v1/start` accepts a client-supplied `pipeline_run_id`, so only `/start` can produce it.-- **413** — request body exceeds the configured size limit (`MAX_REQUEST_BODY_MIB`, 100 MiB by default), or a part of it exceeds its own: a method bundle over `MAX_BUNDLE_FILES` or `MAX_BUNDLE_TOTAL_KIB`, or `resources` over `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` or `MAX_RESOURCES_TOTAL_KIB` (`error_type = "PayloadTooLarge"`).+- **413** — request body exceeds the configured size limit (`MAX_REQUEST_BODY_MIB`, 100 MiB by default), or a part of it exceeds its own: a `bundle_b64` archive over `MAX_BUNDLE_FILES` or `MAX_BUNDLE_TOTAL_KIB`, or `resources` over `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` or `MAX_RESOURCES_TOTAL_KIB` (`error_type = "PayloadTooLarge"`).- **429** — an upstream inference provider rate-limited the run. `Retry-After` is set when the originating error carries `provider_metadata.retry_after_seconds`. Only `POST /v1/execute` runs inference, so only `/execute` can produce it.- **501** — a request shape the published contract accepts but this server cannot serve. `error_type = "AsyncExecutionNotEnabledError"`: this deployment does not provide async pipeline execution (`POST /v1/start`). `error_type = "MethodRefNotSupported"`: the request selected its closure by a **registry-form** `method_ref` — not a `github.com/...` address — and no server-side method registry resolves those yet. Address-form references are fetched and resolved server-side (see [Pipe Run](pipe-run.md#running-a-method-by-address-method_ref)); use one of those, or submit inline `files[]`. Both 501s are permanent under the current deployment — do not retry.-The `method_ref` resolution errors are also mapped explicitly rather than through the domain table: `MethodRefParseError`, `MethodFetchError`, `MethodPackageAmbiguityError`, `MethodPackageTooLargeError` and `MethodPackageSymlinkError` are **422** (the caller can fix the reference, pin a tag, shrink the package, or remove the symlink it carries), `MethodPackageNotFoundError` is the **404** above, and `MethodStructuresRefusedError` is a **403**, whether the method was fetched by `method_ref` or sent as a bundle. These pipelex `MethodRefError` subclasses do not classify an `error_domain`, so without the explicit mapping they would fall through to the 500 default — which would misreport caller-fixable failures as server faults.+The `method_ref` resolution errors are also mapped explicitly rather than through the domain table: `MethodRefParseError`, `MethodFetchError`, `MethodPackageAmbiguityError`, `MethodPackageTooLargeError` and `MethodPackageSymlinkError` are **422** (the caller can fix the reference, pin a tag, shrink the package, or remove the symlink it carries), `MethodPackageNotFoundError` is the **404** above, and `MethodStructuresRefusedError` is a **403**, whether the method was fetched by `method_ref` or sent as a tree. These pipelex `MethodRefError` subclasses do not classify an `error_domain`, so without the explicit mapping they would fall through to the 500 default — which would misreport caller-fixable failures as server faults.Every route that takes a JSON body refuses one whose arrays and objects nest more than 128 levels deep, the body's own envelope included, with a **422** `InvalidJSON` whose `detail` says the body is nested too deeply. The depth is checked before the body is parsed, so the answer does not depend on the Python version or the stack size the server runs with. The bound is fixed rather than configurable, and real inputs stay far below it.@@ -174,6 +177,8 @@ A pipe selection the server cannot make is an input **422** named for its failurThe run routes' extension fields are validated before any method is fetched or any library loaded, and a failure there is a **422** whose `error_type` names the field that failed: `InvalidCallbackUrls` for `callback_urls`, `InvalidStorageScope` for `storage_scope`, `InvalidReadScope` for `read_scope`, `InvalidAnalyticsGroups` for `analytics_groups` (see [Pipe Run](pipe-run.md#host-supplied-run-context-storage_scope-and-analytics_groups)). A storage scope that does not lie under the run's read scope, whether either was sent or defaulted, is an `InvalidReadScope` as well. A failure on another extension field, or on more than one at once, carries the generic `ValidationError`, and the `detail` names every field that failed. `POST /v1/validate` takes `analytics_groups` too and classifies it the same way: a body failing on that field alone answers `InvalidAnalyticsGroups`, and one failing on it and another field answers `ValidationError`.+A method sent as a tree, in `files` or `bundle_b64`, is read under the MTHDS standard's tree rule before anything loads, on the run routes and `/validate` alike (see [Pipe Run](pipe-run.md#sending-a-method-as-a-tree)). A key or an entry the rule refuses, two shipped packages a reference could not tell apart, or `dependencies` that do not fit the request is a **422** `InvalidMethodFiles` carrying `tree_refusal` when the standard's reader refused it; a hash entry this runner does not hold is a **422** `MissingBlobs` carrying `missing_blobs`, and one it holds at another size than the entry states a **422** `BlobMismatch` carrying `mismatched_blobs`; a tree past `MAX_METHOD_TREE_FILES`, `MAX_METHOD_TREE_FILE_KIB` or `MAX_METHOD_TREE_TOTAL_KIB`, or a method past `MAX_METHOD_DEPENDENCIES` or `MAX_METHOD_REQUEST_TOTAL_KIB` across its trees, is a **422** `MethodTreeTooLarge`. The blob routes answer in the same vocabulary: a body whose SHA-256 is not the one its path names, or one hash listed with two sizes, is a **422** `BlobMismatch`, and a blob past the per-file limit a **422** `MethodTreeTooLarge`. Every one of these carries `retryable: false`. Bytes the blob store holds under a hash that do not hash to it are a **500** `BlobStoreCorrupted`: the deployment's storage is damaged, and its operator removes the object for the blob to be uploaded again.+The files a request sends in `resources` are checked before the bundle loads and again once it is parsed (see [Pipe Validate](pipe-validate.md#sending-the-files-the-steps-read-resources)). A key that cannot name a file inside the bundle is a **422** `InvalidResourcePath`, a value that is not base64 a **400** `InvalidBase64`, a file none of the bundle's steps names a **422** `BundleResourceNotNamedError`, and files that cannot make one method tree, such as two keys equal once case-folded, a **422** `BundleResourcePathError`, on every route, the diagnostic ones included: both are the request's fault, not the bundle's, and every one of these refusals carries `retryable: false`, since the same request is refused again.The HTTP status is the source of truth for success vs failure — there is no `success: true/false` field anywhere in the envelope.diff --git a/docs/api-server/index.md b/docs/api-server/index.mdindex 32ea198c6..be004e675 100644--- a/docs/api-server/index.md+++ b/docs/api-server/index.md@@ -19,7 +19,7 @@ MTHDS Protocol ⊂ Pipelex API (this server) ⊂ Pipelex hosted API(the standard) (protocol + extensions) (+ durable runs, catalog, account)```-- **MTHDS Protocol** — five routes: `POST /execute`, `POST /start`, `POST /validate`, `GET /models`, `GET /version`. Tagged `x-mthds-protocol: true` in the [committed OpenAPI artifact](openapi/pipelex-api.openapi.yaml), and **only** those five — the flag is how a conformance suite or a third-party runner extracts the portable subset.+- **MTHDS Protocol** — the routes `POST /execute`, `POST /start`, `POST /validate`, `GET /models`, `GET /version`, and the blob routes `POST /blobs/missing` and `PUT /blobs/{sha256}`, which the protocol makes optional and this server serves, declaring `blob_store: true` on `/version`. Tagged `x-mthds-protocol: true` in the [committed OpenAPI artifact](openapi/pipelex-api.openapi.yaml), and **only** those — the flag is how a conformance suite or a third-party runner extracts the portable subset.- **Pipelex API (this server)** — the protocol verbatim, plus the Pipelex extensions: resolve and codegen (`/resolve`, `/codegen`), pipe I/O (`/pipe-io`), the model reference check (`/models/check`), and editor tooling (`/lint`, `/format`). `/upload` and `/resolve-storage-url` were non-contract convenience routes and have been **removed** — see [Storage Transport](storage-transport.md) for where they went and why.- **Pipelex hosted API** (`api.pipelex.com/v1`) — everything here, same shapes, plus durable runs, the method catalog, and account management.@@ -142,13 +142,15 @@ Every failure is an RFC 7807 `application/problem+json` problem document — see- `GET /` — Service identity banner (no auth required)- `GET /health` — Health check (no auth required)-- `GET /v1/version` — MTHDS Protocol version handshake (no auth required): `{protocol_version, implementation, implementation_version, runtime_version}`. Replaces the former `/pipelex_version` and `/api_version` routes.+- `GET /v1/version` — MTHDS Protocol version handshake (no auth required): `{protocol_version, implementation, implementation_version, runtime_version, blob_store}`. Replaces the former `/pipelex_version` and `/api_version` routes.### Pipe RunExecute pipelines with flexible input formats, either synchronously or asynchronously.- `POST /v1/execute` — Run a pipeline and wait for completion (200 + full result)- `POST /v1/start` — Start a pipeline execution without waiting (202 + `StartAck`)+- `POST /v1/blobs/missing` — Which of a method's files, by hash, this runner lacks, each with a signed upload target+- `PUT /v1/blobs/{sha256}` — Upload one blob to its signed target (no bearer token: the signature authorizes it)[Learn more →](pipe-run.md)diff --git a/docs/api-server/openapi/pipelex-api.openapi.yaml b/docs/api-server/openapi/pipelex-api.openapi.yamlindex bd1bc266d..2dccc95fe 100644--- a/docs/api-server/openapi/pipelex-api.openapi.yaml+++ b/docs/api-server/openapi/pipelex-api.openapi.yaml@@ -51,7 +51,13 @@ paths:identifies this runner, `implementation_version` is this server package''s- version, and `runtime_version` is the underlying pipelex runtime version.'+ version, and `runtime_version` is the underlying pipelex runtime version.++ `blob_store` is true: this runner keeps a blob store over its storage++ provider and serves `POST /v1/blobs/missing` and `PUT /v1/blobs/{sha256}`,++ so a sender may name a file of `files` by its hash.'operationId: get_version_v1_version_getresponses:'200':@@ -68,6 +74,105 @@ paths:schema:$ref: '#/components/schemas/ProblemDocument'x-mthds-protocol: true+ /v1/blobs/{sha256}:+ put:+ tags:+ - blobs+ summary: Upload one blob's bytes to the target POST /blobs/missing named.+ description: 'Store one blob''s bytes under its SHA-256, as the upload target `POST /v1/blobs/missing` named (MTHDS+ Protocol).+++ The body is the blob''s bytes and nothing else. A target whose signature does not hold, or that has expired, is a++ 403 `BlobUploadNotAuthorized`, as is a query that is not one this runner signs, such as an expiry that is not a++ decimal integer; a body past `MAX_METHOD_TREE_FILE_KIB` a 422 `MethodTreeTooLarge`; and a body whose SHA-256 is not++ the path''s a 422 `BlobMismatch`. A blob is never changed: uploading bytes the runner already holds answers 204 and++ changes nothing.'+ operationId: put_blob_v1_blobs__sha256__put+ parameters:+ - name: sha256+ in: path+ required: true+ schema:+ type: string+ pattern: ^[0-9a-f]{64}$+ description: The SHA-256 of the body, in lowercase hexadecimal.+ title: Sha256+ description: The SHA-256 of the body, in lowercase hexadecimal.+ - name: scope+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ - type: 'null'+ description: The blob store's scope, as the signed target names it.+ title: Scope+ description: The blob store's scope, as the signed target names it.+ - name: expires+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ - type: 'null'+ description: When the target expires, in seconds since the epoch.+ title: Expires+ description: When the target expires, in seconds since the epoch.+ - name: signature+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ - type: 'null'+ description: The target's signature.+ title: Signature+ description: The target's signature.+ responses:+ '204':+ description: Successful Response+ '403':+ description: '`BlobUploadNotAuthorized` — the upload does not carry the signed query an upload target of `POST /v1/blobs/missing`+ names, its signature does not hold for this hash and scope, or it has expired. Ask `POST /v1/blobs/missing` again+ for a fresh target.'+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument'+ '413':+ description: Request body exceeds the deployment's size limit (`MAX_REQUEST_BODY_MIB`, 100 MiB by default).+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument'+ '422':+ description: 'The request could not be processed: a malformed body, a field failing validation, or an `input`-domain+ pipelex error (a `.mthds` bundle the caller must fix). Note that on the diagnostic routes an *invalid bundle*+ is a **200** verdict, not a 422 — see each route''s response contract.'+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument'+ '500':+ description: A `config`-domain or `runtime`-domain failure the caller cannot fix (a missing env var, a bad TOML+ override, a backend fault), or an unclassified error sanitized by the catch-all handler. Report the `request_id`.+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument'+ x-mthds-protocol: true+ requestBody:+ required: true+ content:+ application/octet-stream:+ schema:+ type: string+ format: binary/v1/execute:post:tags:@@ -93,6 +198,38 @@ paths:application/json:schema:$defs:+ Base64FileEntry:+ properties:+ base64:+ type: string+ title: Base64+ description: The file's bytes in canonical padded standard Base64.+ additionalProperties: false+ type: object+ required:+ - base64+ title: Base64FileEntry+ description: A file entry carrying the file's bytes in canonical padded standard Base64.+ HashFileEntry:+ properties:+ sha256:+ type: string+ pattern: ^[0-9a-f]{64}$+ title: Sha256+ description: The SHA-256 of the file's bytes, in lowercase hexadecimal.+ size:+ type: integer+ minimum: 0.0+ title: Size+ description: The file's size in bytes, an integer written with no fraction or exponent.+ additionalProperties: false+ type: object+ required:+ - sha256+ - size+ title: HashFileEntry+ description: A file entry naming, by their SHA-256 and their size, bytes the receiver already holds in its+ blob store.StuffContentAbstract:properties: {}type: object@@ -148,21 +285,50 @@ paths:- type: string- type: 'null'title: Bundle B64- description: PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — base64-encoded zip of the whole method- bundle (`.mthds` + `.py` + `structures/*.py` + `requirements.txt`), materialized into a temporary library- directory before the run so custom PipeFunc Python travels with the method. The method packages the bundle- calls by address may travel with it under `.mthds/methods/<name>/`, each with its `METHODS.toml`; they- are searched before the runner's installed copies, for this run alone. Mutually exclusive with `files`.- Custom `.py` is only honored on a sandbox-hosted deployment.+ description: PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — base64-encoded zip of the method's+ folder, read as a map of its entries, so the archive's names are judged by the method tree's rule as the+ keys of `files` are. The method packages it calls by address may travel with it under `.mthds/methods/<name>/`,+ each with its `METHODS.toml`; they are searched before the runner's installed copies, for this run alone.+ Mutually exclusive with `files`. Custom `.py` is only honored on a sandbox-hosted deployment.files:anyOf:- additionalProperties:- type: string+ anyOf:+ - type: string+ - $ref: '#/$defs/Base64FileEntry'+ - $ref: '#/$defs/HashFileEntry'type: object- type: 'null'title: Files- description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method bundle as a `{relative_path:- text}` map (the unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.'+ description: 'The method as a map of file entries, the MTHDS standard''s form of a method tree that travels:+ each key is a tree path (relative, forward slashes, in Unicode NFC, no `.` or `..` segment, no segment+ starting with a dot), and each value is the file''s UTF-8 text, `{"base64": ...}` holding its bytes, or+ `{"sha256": ..., "size": ...}` naming bytes this runner already holds (uploaded with `POST /v1/blobs/missing`+ then `PUT /v1/blobs/{sha256}`). The map is read from the request''s JSON text: a key given twice, a key+ the membership rule keeps out of every tree (`.mthds/` included), colliding keys and a malformed entry+ are a 422 `InvalidMethodFiles` whose `tree_refusal` names the code and the paths, and hash entries the+ runner does not hold are one 422 `MissingBlobs` whose `missing_blobs` lists every such hash. Its `.mthds`+ files are the method''s bundles and every other file is one a step may read, such as a `PipeDocGen` template+ or a `PipeFunc` module. With no `pipe_code`, the run starts at the `main_pipe` of the `METHODS.toml` at+ the tree''s root, or else at the one `main_pipe` its bundles declare. Custom `.py` is only honored on+ a sandbox-hosted deployment.'+ dependencies:+ anyOf:+ - additionalProperties:+ additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/$defs/Base64FileEntry'+ - $ref: '#/$defs/HashFileEntry'+ type: object+ type: object+ - type: 'null'+ title: Dependencies+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method packages the method calls+ by address that this runner cannot fetch, such as one an author has not published, as `{"<address>": <map+ of file entries>}`: each value is a tree of its own, read as `files` is, with a `METHODS.toml` at its+ root declaring the address it is keyed by. They are searched before the runner''s installed copies, for+ this request alone. Accompanies `files`.'method_ref:anyOf:- type: string@@ -187,14 +353,15 @@ paths:title: Resourcesdescription: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`(an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative- path, exactly as the step names it, to the file''s bytes in standard base64. Every bundle of the request- is read as sitting in one directory. A path must be relative and written with forward slashes, with no- `..` segment, backslash or colon. Every file must be named by a step, or the request is refused with a- 422 `BundleResourceNotNamedError`. By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded,- counting the files a method bundle ships for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,- `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. A file both sent here and shipped in a method bundle- must be the same. The files make one method tree with the bundle at its root, which the run names by its- digest: no file''s bytes travel with the run, each being read where a step uses it.'+ path, exactly as the step names it, to the file''s bytes in standard base64. A path must be relative and+ written with forward slashes, with no `..` segment, backslash or colon. Beside inline `mthds_contents`,+ every bundle of the request is read as sitting in one directory, and every file must be named by a step,+ or the request is refused with a 422 `BundleResourceNotNamedError`. Beside a method tree, `files` or `bundle_b64`,+ each file joins the tree at its path from the tree''s root, where a file the tree already holds must have+ the same bytes. By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment+ sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.+ The files make one method tree with the bundles, which the run names by its digest: no file''s bytes travel+ with the run, each being read where a step uses it.'orchestration_mode:anyOf:- type: string@@ -366,6 +533,38 @@ paths:application/json:schema:$defs:+ Base64FileEntry:+ properties:+ base64:+ type: string+ title: Base64+ description: The file's bytes in canonical padded standard Base64.+ additionalProperties: false+ type: object+ required:+ - base64+ title: Base64FileEntry+ description: A file entry carrying the file's bytes in canonical padded standard Base64.+ HashFileEntry:+ properties:+ sha256:+ type: string+ pattern: ^[0-9a-f]{64}$+ title: Sha256+ description: The SHA-256 of the file's bytes, in lowercase hexadecimal.+ size:+ type: integer+ minimum: 0.0+ title: Size+ description: The file's size in bytes, an integer written with no fraction or exponent.+ additionalProperties: false+ type: object+ required:+ - sha256+ - size+ title: HashFileEntry+ description: A file entry naming, by their SHA-256 and their size, bytes the receiver already holds in its+ blob store.StuffContentAbstract:properties: {}type: object@@ -421,21 +620,50 @@ paths:- type: string- type: 'null'title: Bundle B64- description: PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — base64-encoded zip of the whole method- bundle (`.mthds` + `.py` + `structures/*.py` + `requirements.txt`), materialized into a temporary library- directory before the run so custom PipeFunc Python travels with the method. The method packages the bundle- calls by address may travel with it under `.mthds/methods/<name>/`, each with its `METHODS.toml`; they- are searched before the runner's installed copies, for this run alone. Mutually exclusive with `files`.- Custom `.py` is only honored on a sandbox-hosted deployment.+ description: PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — base64-encoded zip of the method's+ folder, read as a map of its entries, so the archive's names are judged by the method tree's rule as the+ keys of `files` are. The method packages it calls by address may travel with it under `.mthds/methods/<name>/`,+ each with its `METHODS.toml`; they are searched before the runner's installed copies, for this run alone.+ Mutually exclusive with `files`. Custom `.py` is only honored on a sandbox-hosted deployment.files:anyOf:- additionalProperties:- type: string+ anyOf:+ - type: string+ - $ref: '#/$defs/Base64FileEntry'+ - $ref: '#/$defs/HashFileEntry'type: object- type: 'null'title: Files- description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method bundle as a `{relative_path:- text}` map (the unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.'+ description: 'The method as a map of file entries, the MTHDS standard''s form of a method tree that travels:+ each key is a tree path (relative, forward slashes, in Unicode NFC, no `.` or `..` segment, no segment+ starting with a dot), and each value is the file''s UTF-8 text, `{"base64": ...}` holding its bytes, or+ `{"sha256": ..., "size": ...}` naming bytes this runner already holds (uploaded with `POST /v1/blobs/missing`+ then `PUT /v1/blobs/{sha256}`). The map is read from the request''s JSON text: a key given twice, a key+ the membership rule keeps out of every tree (`.mthds/` included), colliding keys and a malformed entry+ are a 422 `InvalidMethodFiles` whose `tree_refusal` names the code and the paths, and hash entries the+ runner does not hold are one 422 `MissingBlobs` whose `missing_blobs` lists every such hash. Its `.mthds`+ files are the method''s bundles and every other file is one a step may read, such as a `PipeDocGen` template+ or a `PipeFunc` module. With no `pipe_code`, the run starts at the `main_pipe` of the `METHODS.toml` at+ the tree''s root, or else at the one `main_pipe` its bundles declare. Custom `.py` is only honored on+ a sandbox-hosted deployment.'+ dependencies:+ anyOf:+ - additionalProperties:+ additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/$defs/Base64FileEntry'+ - $ref: '#/$defs/HashFileEntry'+ type: object+ type: object+ - type: 'null'+ title: Dependencies+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method packages the method calls+ by address that this runner cannot fetch, such as one an author has not published, as `{"<address>": <map+ of file entries>}`: each value is a tree of its own, read as `files` is, with a `METHODS.toml` at its+ root declaring the address it is keyed by. They are searched before the runner''s installed copies, for+ this request alone. Accompanies `files`.'method_ref:anyOf:- type: string@@ -460,14 +688,15 @@ paths:title: Resourcesdescription: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`(an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative- path, exactly as the step names it, to the file''s bytes in standard base64. Every bundle of the request- is read as sitting in one directory. A path must be relative and written with forward slashes, with no- `..` segment, backslash or colon. Every file must be named by a step, or the request is refused with a- 422 `BundleResourceNotNamedError`. By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded,- counting the files a method bundle ships for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,- `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. A file both sent here and shipped in a method bundle- must be the same. The files make one method tree with the bundle at its root, which the run names by its- digest: no file''s bytes travel with the run, each being read where a step uses it.'+ path, exactly as the step names it, to the file''s bytes in standard base64. A path must be relative and+ written with forward slashes, with no `..` segment, backslash or colon. Beside inline `mthds_contents`,+ every bundle of the request is read as sitting in one directory, and every file must be named by a step,+ or the request is refused with a 422 `BundleResourceNotNamedError`. Beside a method tree, `files` or `bundle_b64`,+ each file joins the tree at its path from the tree''s root, where a file the tree already holds must have+ the same bytes. By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment+ sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.+ The files make one method tree with the bundles, which the run names by its digest: no file''s bytes travel+ with the run, each being read where a step uses it.'pipeline_run_id:anyOf:- type: string@@ -1270,6 +1499,91 @@ paths:application/problem+json:schema:$ref: '#/components/schemas/ProblemDocument'+ /v1/blobs/missing:+ post:+ tags:+ - blobs+ summary: Which of these blobs the runner lacks, and where to upload each.+ description: 'List every blob of the request this runner does not hold, each with a signed upload target (MTHDS Protocol).+++ A blob it holds at the stated size is left out, and an empty list means there is nothing to upload; bytes held++ under a hash at another size are listed, for the upload to write over. When the request names the tree the blobs++ make up by its `digest`, this runner has recorded that tree as held in full, which it does only after reading every++ file of a tree a request sent, and every listed blob is a file of that tree, the answer is an empty list at once.++ A hash given twice with different sizes is a 422 `BlobMismatch`, and a blob or a set of blobs past this runner''s++ tree limits a 422 `MethodTreeTooLarge`, both refused before anything is uploaded. Each upload target is this runner''s+ own++ `PUT /v1/blobs/{sha256}`, at `PUBLIC_BASE_URL` when the deployment sets it, its query signed for that hash and++ the caller''s scope, valid for `BLOB_UPLOAD_URL_TTL_SECONDS`, to send with exactly the headers it names.'+ operationId: find_missing_blobs_v1_blobs_missing_post+ parameters:+ - name: read_scope+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ - type: 'null'+ description: 'PIPELEX-API EXTENSION — the blob store''s scope, as a run''s `read_scope`: a multi-tenant host sends+ the one of the organization it serves. Omit it and the caller''s own id is the scope.'+ title: Read Scope+ description: 'PIPELEX-API EXTENSION — the blob store''s scope, as a run''s `read_scope`: a multi-tenant host sends+ the one of the organization it serves. Omit it and the caller''s own id is the scope.'+ requestBody:+ required: true+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/MissingBlobsRequest'+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/MissingBlobsResult'+ '401':+ description: Missing or invalid bearer token. Only reachable when the deployment enables auth (`AUTH_MODE=api_key`+ or `AUTH_MODE=jwt`).+ headers:+ WWW-Authenticate:+ description: Authentication challenge — always `Bearer`.+ schema:+ type: string+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument'+ '413':+ description: Request body exceeds the deployment's size limit (`MAX_REQUEST_BODY_MIB`, 100 MiB by default).+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument'+ '422':+ description: 'The request could not be processed: a malformed body, a field failing validation, or an `input`-domain+ pipelex error (a `.mthds` bundle the caller must fix). Note that on the diagnostic routes an *invalid bundle*+ is a **200** verdict, not a 422 — see each route''s response contract.'+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument'+ '500':+ description: A `config`-domain or `runtime`-domain failure the caller cannot fix (a missing env var, a bad TOML+ override, a backend fault), or an unclassified error sanitized by the catch-all handler. Report the `request_id`.+ content:+ application/problem+json:+ schema:+ $ref: '#/components/schemas/ProblemDocument'+ x-mthds-protocol: true/:get:tags:@@ -1388,6 +1702,18 @@ components:- opaque- autotitle: Background+ Base64FileEntry:+ properties:+ base64:+ type: string+ title: Base64+ description: The file's bytes in canonical padded standard Base64.+ additionalProperties: false+ type: object+ required:+ - base64+ title: Base64FileEntry+ description: A file entry carrying the file's bytes in canonical padded standard Base64.BindingStepBlueprint:properties:from:@@ -1414,6 +1740,26 @@ components:under `total_amount`, as the concept the path derives through the declared structures. A binding steplives in a PipeSequence''s `steps` only, and carries exactly these two fields.'+ BlobRef:+ properties:+ sha256:+ type: string+ pattern: ^[0-9a-f]{64}$+ title: Sha256+ description: The SHA-256 of the bytes, in lowercase hexadecimal.+ size:+ type: integer+ minimum: 0.0+ title: Size+ description: The length of the bytes, a JSON integer written without a fraction or an exponent. A size written 13.0+ is refused, although JSON Schema's integer type admits it.+ additionalProperties: false+ type: object+ required:+ - sha256+ - size+ title: BlobRef+ description: A blob, named by its SHA-256 in lowercase hexadecimal and its length in bytes.BooleanField:properties:title:@@ -1572,14 +1918,14 @@ components:title: Resourcesdescription: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`(an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly- as the step names it, to the file''s bytes in standard base64. Every bundle of the request is read as sitting- in one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or- colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`.- By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded, counting the files a method bundle ships- for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.- A file both sent here and shipped in a method bundle must be the same. The files make one method tree with the- bundle at its root, which the run names by its digest: no file''s bytes travel with the run, each being read where- a step uses it.'+ as the step names it, to the file''s bytes in standard base64. A path must be relative and written with forward+ slashes, with no `..` segment, backslash or colon. Beside inline `mthds_contents`, every bundle of the request+ is read as sitting in one directory, and every file must be named by a step, or the request is refused with a+ 422 `BundleResourceNotNamedError`. Beside a method tree, `files` or `bundle_b64`, each file joins the tree at+ its path from the tree''s root, where a file the tree already holds must have the same bytes. By default at most+ 32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,+ `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. The files make one method tree with the bundles, which+ the run names by its digest: no file''s bytes travel with the run, each being read where a step uses it.'kind:$ref: '#/components/schemas/CodegenRouteKind'description: What to project. `types` projects the crate's concept set into typed models.@@ -2805,6 +3151,25 @@ components:- viatitle: HandleMatchdescription: What a bare handle reference is in one category.+ HashFileEntry:+ properties:+ sha256:+ type: string+ pattern: ^[0-9a-f]{64}$+ title: Sha256+ description: The SHA-256 of the file's bytes, in lowercase hexadecimal.+ size:+ type: integer+ minimum: 0.0+ title: Size+ description: The file's size in bytes, an integer written with no fraction or exponent.+ additionalProperties: false+ type: object+ required:+ - sha256+ - size+ title: HashFileEntry+ description: A file entry naming, by their SHA-256 and their size, bytes the receiver already holds in its blob store.HealthResponse:properties:status:@@ -3637,12 +4002,73 @@ components:commit_sha:type: stringtitle: Commit Sha+ tree_digest:+ anyOf:+ - type: string+ - type: 'null'+ title: Tree Digest+ description: The digest of the package's method tree, `sha256:` and 64 lowercase hex characters, as read from the+ commit's git objects, set by a runner that reads the package as a tree, and null otherwise.type: objectrequired:- address- commit_shatitle: MethodProvenancedescription: 'What a fetched-method run must record: address, tag, and the resolved commit SHA.'+ MissingBlob:+ properties:+ sha256:+ type: string+ pattern: ^[0-9a-f]{64}$+ title: Sha256+ size:+ type: integer+ minimum: 0.0+ title: Size+ description: The length of the bytes, a JSON integer written without a fraction or an exponent. A size written 13.0+ is refused, although JSON Schema's integer type admits it.+ upload:+ $ref: '#/components/schemas/UploadTarget'+ type: object+ required:+ - sha256+ - size+ - upload+ title: MissingBlob+ description: A blob the runner lacks, with where to upload it.+ MissingBlobsRequest:+ properties:+ blobs:+ items:+ $ref: '#/components/schemas/BlobRef'+ type: array+ title: Blobs+ description: The blobs to check, each hash listed once.+ digest:+ anyOf:+ - type: string+ pattern: ^sha256:[0-9a-f]{64}$+ - type: 'null'+ title: Digest+ description: The tree digest of the method tree the blobs make up (optional).+ additionalProperties: true+ type: object+ required:+ - blobs+ title: MissingBlobsRequest+ description: The blobs a sender is about to send by hash, and optionally the tree they make up.+ MissingBlobsResult:+ properties:+ missing:+ items:+ $ref: '#/components/schemas/MissingBlob'+ type: array+ title: Missing+ type: object+ required:+ - missing+ title: MissingBlobsResult+ description: The blobs of the request the runner does not hold.ModelCategory:type: stringenum:@@ -3961,14 +4387,14 @@ components:title: Resourcesdescription: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`(an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly- as the step names it, to the file''s bytes in standard base64. Every bundle of the request is read as sitting- in one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or- colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`.- By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded, counting the files a method bundle ships- for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.- A file both sent here and shipped in a method bundle must be the same. The files make one method tree with the- bundle at its root, which the run names by its digest: no file''s bytes travel with the run, each being read where- a step uses it.'+ as the step names it, to the file''s bytes in standard base64. A path must be relative and written with forward+ slashes, with no `..` segment, backslash or colon. Beside inline `mthds_contents`, every bundle of the request+ is read as sitting in one directory, and every file must be named by a step, or the request is refused with a+ 422 `BundleResourceNotNamedError`. Beside a method tree, `files` or `bundle_b64`, each file joins the tree at+ its path from the tree''s root, where a file the tree already holds must have the same bytes. By default at most+ 32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,+ `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. The files make one method tree with the bundles, which+ the run names by its digest: no file''s bytes travel with the run, each being read where a step uses it.'type: objecttitle: MthdsFilesRequestdescription: 'Shared closure selector for the resolve/codegen routes: inline `files[]` XOR a `method_ref`.@@ -5040,14 +5466,14 @@ components:title: Resourcesdescription: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`(an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly- as the step names it, to the file''s bytes in standard base64. Every bundle of the request is read as sitting- in one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or- colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`.- By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded, counting the files a method bundle ships- for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.- A file both sent here and shipped in a method bundle must be the same. The files make one method tree with the- bundle at its root, which the run names by its digest: no file''s bytes travel with the run, each being read where- a step uses it.'+ as the step names it, to the file''s bytes in standard base64. A path must be relative and written with forward+ slashes, with no `..` segment, backslash or colon. Beside inline `mthds_contents`, every bundle of the request+ is read as sitting in one directory, and every file must be named by a step, or the request is refused with a+ 422 `BundleResourceNotNamedError`. Beside a method tree, `files` or `bundle_b64`, each file joins the tree at+ its path from the tree''s root, where a file the tree already holds must have the same bytes. By default at most+ 32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,+ `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. The files make one method tree with the bundles, which+ the run names by its digest: no file''s bytes travel with the run, each being read where a step uses it.'pipe_ref:anyOf:- type: string@@ -6204,6 +6630,10 @@ components:- type: string- type: 'null'title: Runtime Version+ blob_store:+ type: boolean+ title: Blob Store+ default: falseadditionalProperties: truetype: objectrequired:@@ -7504,6 +7934,30 @@ components:The mandatory escape hatch that makes a total derivation truthful: a renderer falls back to rawentry against the slot''s `json_schema` in the pipe I/O contract.'+ UploadTarget:+ properties:+ method:+ type: string+ title: Method+ description: The HTTP method, such as PUT.+ url:+ type: string+ title: Url+ description: 'Where to send the bytes: this runner''s own `PUT /v1/blobs/{sha256}`, with a signed query.'+ headers:+ additionalProperties:+ type: string+ type: object+ title: Headers+ description: The headers to send, exactly.+ type: object+ required:+ - method+ - url+ - headers+ title: UploadTarget+ description: 'Where to upload one blob: send its bytes as the whole body of a request with this method to this URL,+ with exactly these headers.'UserAction:properties:kind:@@ -7602,7 +8056,8 @@ components:type: stringtype: arraytitle: Mthds Contents- description: The MTHDS contents that were validated (echo of the request)+ description: 'The MTHDS contents that were validated: the request''s `mthds_contents`, or the texts of a method+ tree''s `.mthds` bundles in tree path order.'message:type: stringtitle: Message@@ -7614,11 +8069,10 @@ components:- type: 'null'title: Default Pipe Refdescription: 'The qualified `domain.pipe_code` a caller gets by omitting the pipe selector — the pipe a selector-less- run of THIS request would execute. On a `method_ref` request it is the fetched package manifest''s `main_pipe`- (the package author''s declared entry pipe, qualified against the closure); otherwise, and when the manifest declares- none, it is the closure''s primary blueprint''s `main_pipe` qualified by its domain. `null` when no entry pipe- is determined: no blueprint declares `main_pipe`, or a manifest names a pipe the closure does not declare (or- declares ambiguously), in which case a selector-less run by this address would fail to resolve it too.'+ run of THIS request would execute. For a method tree (`files` or a `method_ref`) it is the tree''s entry pipe+ by the MTHDS protocol''s rule: the `main_pipe` of the `METHODS.toml` at its root, else the one `main_pipe` its+ bundles declare. For inline `mthds_contents` it is the closure''s primary blueprint''s `main_pipe` qualified by+ its domain. `null` when no entry pipe is determined, in which case a selector-less run would be refused too.'rendered_markdown:anyOf:- type: string@@ -7685,7 +8139,45 @@ components:- type: 'null'title: Mthds Contentsdescription: MTHDS contents to load (always an array, even for a single file). Exactly one of `mthds_contents` /- `method_ref`.+ `files` / `method_ref`.+ files:+ anyOf:+ - additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/components/schemas/Base64FileEntry'+ - $ref: '#/components/schemas/HashFileEntry'+ type: object+ - type: 'null'+ title: Files+ description: 'The method as a map of file entries, the MTHDS standard''s form of a method tree that travels: each+ key is a tree path (relative, forward slashes, in Unicode NFC, no `.` or `..` segment, no segment starting with+ a dot), and each value is the file''s UTF-8 text, `{"base64": ...}` holding its bytes, or `{"sha256": ..., "size":+ ...}` naming bytes this runner already holds (uploaded with `POST /v1/blobs/missing` then `PUT /v1/blobs/{sha256}`).+ The map is read from the request''s JSON text: a key given twice, a key the membership rule keeps out of every+ tree (`.mthds/` included), colliding keys and a malformed entry are a 422 `InvalidMethodFiles` whose `tree_refusal`+ names the code and the paths, and hash entries the runner does not hold are one 422 `MissingBlobs` whose `missing_blobs`+ lists every such hash. Its `.mthds` files are the method''s bundles and every other file is one a step may read,+ such as a `PipeDocGen` template or a `PipeFunc` module. With no `pipe_code`, the run starts at the `main_pipe`+ of the `METHODS.toml` at the tree''s root, or else at the one `main_pipe` its bundles declare. Custom `.py` is+ only honored on a sandbox-hosted deployment. The whole tree is validated, its Python included, by syntax. Exactly+ one of `mthds_contents` / `files` / `method_ref`.'+ dependencies:+ anyOf:+ - additionalProperties:+ additionalProperties:+ anyOf:+ - type: string+ - $ref: '#/components/schemas/Base64FileEntry'+ - $ref: '#/components/schemas/HashFileEntry'+ type: object+ type: object+ - type: 'null'+ title: Dependencies+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the method packages the method calls by address+ that this runner cannot fetch, such as one an author has not published, as `{"<address>": <map of file entries>}`:+ each value is a tree of its own, read as `files` is, with a `METHODS.toml` at its root declaring the address it+ is keyed by. They are searched before the runner''s installed copies, for this request alone. Accompanies `files`.'method_ref:anyOf:- type: string@@ -7695,10 +8187,10 @@ components:title: Method Refdescription: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — validate a published method by referenceinstead of inline contents. Address form: `github.com/<owner>/<repo>[/<selector>][@<tag>]`, resolved by THIS runner- through the same fetch path as a `method_ref` run; the package''s `.mthds` files feed the validation with their- real relative paths as per-file sources. Exactly one of `mthds_contents` / `method_ref`. A resolution failure- (fetch, package location, the custom-Python policy) is a non-2xx `problem+json` — never an `is_valid: false` verdict,- which is reserved for actual MTHDS content.'+ through the same fetch path as a `method_ref` run; the package''s tree, read from its commit, is validated as+ a `files` tree is. Exactly one of `mthds_contents` / `files` / `method_ref`. A resolution failure (fetch, package+ location, the custom-Python policy) is a non-2xx `problem+json` — never an `is_valid: false` verdict, which is+ reserved for actual MTHDS content.'allow_signatures:type: booleantitle: Allow Signatures@@ -7724,14 +8216,14 @@ components:title: Resourcesdescription: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`(an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly- as the step names it, to the file''s bytes in standard base64. Every bundle of the request is read as sitting- in one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or- colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`.- By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded, counting the files a method bundle ships- for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.- A file both sent here and shipped in a method bundle must be the same. The files make one method tree with the- bundle at its root, which the run names by its digest: no file''s bytes travel with the run, each being read where- a step uses it.'+ as the step names it, to the file''s bytes in standard base64. A path must be relative and written with forward+ slashes, with no `..` segment, backslash or colon. Beside inline `mthds_contents`, every bundle of the request+ is read as sitting in one directory, and every file must be named by a step, or the request is refused with a+ 422 `BundleResourceNotNamedError`. Beside a method tree, `files` or `bundle_b64`, each file joins the tree at+ its path from the tree''s root, where a file the tree already holds must have the same bytes. By default at most+ 32 files, 5120 KiB each and 10240 KiB in all, decoded; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,+ `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. The files make one method tree with the bundles, which+ the run names by its digest: no file''s bytes travel with the run, each being read where a step uses it.'render:items:type: string@@ -7761,6 +8253,14 @@ components:to a worker; any plugin-provided token is accepted and an unregistered one is refused at dispatch. Honored onlywhen the deployment sets `allow_request_orchestration_mode_override = true` in its `api.toml`; otherwise a tokenthat differs from the deployment default is refused with a 403. Omitted → the default.'+ read_scope:+ anyOf:+ - type: string+ - type: 'null'+ title: Read Scope+ description: 'PIPELEX-API EXTENSION (not part of the MTHDS Protocol) — the scope of the blob store a hash entry+ of `files` resolves in, as a run''s `read_scope`: a multi-tenant host sends the one of the organization it serves.+ Omit it and the caller''s own id is the scope.'type: objecttitle: ValidateRequestdescription: 'The shared `mthds_contents` + `allow_signatures` payload, plus optional per-file sources.diff --git a/docs/api-server/pipe-run.md b/docs/api-server/pipe-run.mdindex aeb9d0eb1..a5e28f663 100644--- a/docs/api-server/pipe-run.md+++ b/docs/api-server/pipe-run.md@@ -35,10 +35,11 @@ Execute a Pipelex pipeline with flexible inputs and wait for completion.- `output_name` (string, optional): Name for the output slot- `output_multiplicity` (string, optional): Output multiplicity setting (`"single"`, `"variable"`, or a specific number)- `dynamic_output_concept_ref` (string, optional): Override output concept ref-- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of a whole method bundle, which may carry the method packages it calls — see [Shipping a method bundle](#shipping-a-method-bundle-custom-pipefunc). Mutually exclusive with `files`.-- `files` (dict[str, str], optional): **Pipelex-API extension.** The same bundle as a `{relative_path: text}` map (the unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.-- `method_ref` (string, optional): **Pipelex-API extension.** Run a published method by address instead of inline source — see [Running a method by address](#running-a-method-by-address-method_ref). Mutually exclusive with `mthds_contents` and with a method bundle; `pipe_code` may accompany it to override the package's `main_pipe`.-- `resources` (dict[str, str], optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](pipe-validate.md#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` or a method bundle; beside `method_ref`, or with no bundle at all, it is a 422. The files make one method tree with the bundles at its root, and the job names that tree by its digest and carries none of its files: the step that reads one, on an orchestrator that runs the job remotely such as Temporal included, reads it through the deployment's method file store.+- `files` (object, optional): The method as a map of file entries, the MTHDS standard's method tree: each key a tree path, each value the file's text, `{"base64": "…"}`, or `{"sha256": "…", "size": n}` naming bytes this runner holds — see [Sending a method as a tree](#sending-a-method-as-a-tree). Mutually exclusive with `mthds_contents`, `bundle_b64` and `method_ref`.+- `dependencies` (object, optional): **Pipelex-API extension.** The method packages the method calls by address that the runner cannot fetch, each a map of file entries keyed by its address — see [Sending the method packages the method calls](#sending-the-method-packages-the-method-calls). Accompanies `files`.+- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of the method's folder, read as a map of its entries, which may carry the method packages it calls under `.mthds/methods/<name>/`. Mutually exclusive with `files`.+- `method_ref` (string, optional): **Pipelex-API extension.** Run a published method by address instead of inline source — see [Running a method by address](#running-a-method-by-address-method_ref). Mutually exclusive with `mthds_contents` and with a method sent as a tree; `pipe_code` may accompany it to override the package's `main_pipe`.+- `resources` (dict[str, str], optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](pipe-validate.md#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` or a method sent as a tree, whose tree each file joins at its path from the root; beside `method_ref`, or with no bundle at all, it is a 422. The job names the method's tree by its digest and carries none of its files: the step that reads one, on an orchestrator that runs the job remotely such as Temporal included, reads it through the deployment's method file store.- `storage_scope` (string, optional): **Pipelex-API extension.** The prefix every object this run writes lands under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).- `read_scope` (string, optional): **Pipelex-API extension.** The prefix every stored object this run reads must lie under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).- `analytics_groups` (dict[str, str], optional): **Pipelex-API extension.** The groups this run's telemetry belongs to, such as `{"organization": "org_acme"}` — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).@@ -116,10 +117,11 @@ Start a pipeline execution and get its `pipeline_run_id` back with a `202` ack.- `output_name` (string, optional): Name for the output slot- `output_multiplicity` (string, optional): Output multiplicity setting (`"single"`, `"variable"`, or a specific number)- `dynamic_output_concept_ref` (string, optional): Override output concept ref-- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of a whole method bundle, which may carry the method packages it calls — see [Shipping a method bundle](#shipping-a-method-bundle-custom-pipefunc). Mutually exclusive with `files`.-- `files` (dict[str, str], optional): **Pipelex-API extension.** The same bundle as a `{relative_path: text}` map (the unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.-- `method_ref` (string, optional): **Pipelex-API extension.** Run a published method by address instead of inline source — see [Running a method by address](#running-a-method-by-address-method_ref). Mutually exclusive with `mthds_contents` and with a method bundle; `pipe_code` may accompany it to override the package's `main_pipe`.-- `resources` (dict[str, str], optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](pipe-validate.md#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` or a method bundle; beside `method_ref`, or with no bundle at all, it is a 422. The files make one method tree with the bundles at its root, and the job names that tree by its digest and carries none of its files: the step that reads one, on an orchestrator that runs the job remotely such as Temporal included, reads it through the deployment's method file store.+- `files` (object, optional): The method as a map of file entries, the MTHDS standard's method tree: each key a tree path, each value the file's text, `{"base64": "…"}`, or `{"sha256": "…", "size": n}` naming bytes this runner holds — see [Sending a method as a tree](#sending-a-method-as-a-tree). Mutually exclusive with `mthds_contents`, `bundle_b64` and `method_ref`.+- `dependencies` (object, optional): **Pipelex-API extension.** The method packages the method calls by address that the runner cannot fetch, each a map of file entries keyed by its address — see [Sending the method packages the method calls](#sending-the-method-packages-the-method-calls). Accompanies `files`.+- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of the method's folder, read as a map of its entries, which may carry the method packages it calls under `.mthds/methods/<name>/`. Mutually exclusive with `files`.+- `method_ref` (string, optional): **Pipelex-API extension.** Run a published method by address instead of inline source — see [Running a method by address](#running-a-method-by-address-method_ref). Mutually exclusive with `mthds_contents` and with a method sent as a tree; `pipe_code` may accompany it to override the package's `main_pipe`.+- `resources` (dict[str, str], optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](pipe-validate.md#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` or a method sent as a tree, whose tree each file joins at its path from the root; beside `method_ref`, or with no bundle at all, it is a 422. The job names the method's tree by its digest and carries none of its files: the step that reads one, on an orchestrator that runs the job remotely such as Temporal included, reads it through the deployment's method file store.- `storage_scope` (string, optional): **Pipelex-API extension.** The prefix every object this run writes lands under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).- `read_scope` (string, optional): **Pipelex-API extension.** The prefix every stored object this run reads must lie under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).- `analytics_groups` (dict[str, str], optional): **Pipelex-API extension.** The groups this run's telemetry belongs to, such as `{"organization": "org_acme"}` — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).@@ -209,45 +211,66 @@ The receiver-side secret must be the same value. In typical deployments both sid----## Shipping a method bundle (custom PipeFunc)+## Sending a method as a tree-`mthds_contents` carries only the `.mthds` text. When your method uses a **custom `PipeFunc`** — your own Python function — the code has to travel with the method too. Both `/execute` and `/start` accept the whole bundle in one of two mutually-exclusive forms:+`mthds_contents` carries only `.mthds` text. A method is more than its bundles: a custom `PipeFunc`'s Python, a `PipeDocGen` step's template, a `requirements.txt`, a `METHODS.toml`. Both `/execute` and `/start`, and [`/validate`](pipe-validate.md), take the whole method as a **method tree**, the MTHDS standard's unit of a method, in one of two mutually exclusive forms:-- `bundle_b64`: a base64-encoded **zip** of the bundle directory (`.mthds` + `pipe_func.py` + an optional `requirements.txt`).-- `files`: the same content as a `{relative_path: text}` **map** (the unzipped equivalent) — handy for JSON clients that would rather not zip.--The server materializes the bundle into a temporary library directory for the run and tears it down afterward. The pipe to run comes from the bundle's `main_pipe` (or an explicit `pipe_code`, to pick which pipe in the bundle to run). A bundle carries its own `.mthds`, so it is **mutually exclusive with inline `mthds_contents`** — sending both is a `422` (they would load into one library with no dedup and a shared domain would collide).--**Example (`files` form):**+- `files`: a **map of file entries**, the standard's form. Each key is the file's tree path, and each value one of three entries: the file's text as a JSON string, `{"base64": "…"}` for bytes that are not text, or `{"sha256": "…", "size": n}` naming bytes this runner already holds in its [blob store](#sending-files-by-hash). One request may mix all three.+- `bundle_b64`: a base64-encoded **zip** of the method's folder, a Pipelex-API extension, read as a map of its entries.```json{"files": {- "main.mthds": "domain = \"demo\"\nmain_pipe = \"crunch\"\n\n[pipe.crunch]\ntype = \"PipeFunc\"\ndescription = \"Crunch numbers\"\ninputs = { data = \"Text\" }\noutput = \"Text\"\nfunction_name = \"crunch\"\n",- "pipe_func.py": "def crunch(working_memory):\n return \"...\"\n"+ "main.mthds": "domain = \"demo\"\nmain_pipe = \"crunch\"\n\n[pipe.crunch]\ntype = \"PipeFunc\"\ndescription = \"Crunch numbers\"\ninputs = { data = \"Text\" }\noutput = \"Text\"\nfunction_name = \"pipe_func.crunch\"\n",+ "pipe_func.py": "def crunch(working_memory):\n return \"...\"\n",+ "templates/logo.png": { "base64": "iVBORw0KGgo…" },+ "templates/invoice.docx": { "sha256": "3e9b68b71cc667168399f0bccfb64b2d16314cdc0ed78fac4819aba0dda121a0", "size": 18342 }},"inputs": { "data": "1,2,3" }}```-**Template files shipped in the bundle.** A file a step reads beside its bundle, such as a `PipeDocGen` step's `template_file`, can ship in the bundle, relative to the `.mthds` file whose step names it, as it sits beside a bundle on disk. The bundle's `.mthds` files load as content, so the server sends each such file with them as if it had come in [`resources`](pipe-validate.md#sending-the-files-the-steps-read-resources): it is counted against the same limits as `resources`, together with them, and it joins the one method tree of the request's files, which the run names by its digest and a remote orchestrator copies into its deployment's method file store as it dispatches the run. Every `.mthds` file of the bundle is read as sitting in one directory, so two different files a step reads under one path, shipped beside two `.mthds` files or shipped and also sent in `resources`, are a `422 InvalidResourcePath`; the same file in both is accepted.+**One rule for every path.** The map is the tree: the runner reads it with the standard's reader and holds every key to the tree's one rule, whatever form the method came in. A key that is not a tree path (absolute, with a `.` or `..` segment, an empty segment, a backslash, a colon), a key the membership rule keeps out of every tree (a dotfile or a file under a dot directory such as `.mthds/`, `.git/` or `.env`, `__pycache__/`, `node_modules/`), one key given twice, two keys naming one file once normalized, or a file that is another entry's directory is a `422 InvalidMethodFiles` whose `tree_refusal` member gives the standard's refusal `code` and the `names` it refuses; a malformed entry, such as a size written `13.0`, is refused the same way. A zip's member names are judged by the same rule, so a name given twice in an archive is a duplicate key.++**The tree is written whole and loaded as the caller's.** The runner writes the tree into a temporary directory, exactly as the map holds it, and loads it as one library directory that belongs to the caller: every bundle is a member of the tree and is named by its tree path, every other file is one a step may read where the bundle sits, a `PipeDocGen` template beside its bundle or a `PipeFunc` module at the tree's import root, and a refusal of the load is the caller's `422`, or an `is_valid: false` verdict on `/validate`, naming files by their tree paths and never by the directory on the server. The directory is removed once the request returns, and the run's job names the tree by its digest, so an orchestrator that runs the job remotely reads its files from the deployment's method file store.++**The entry pipe.** With no `pipe_code`, the run starts at the pipe the MTHDS protocol's rule names: the `main_pipe` of the `METHODS.toml` at the tree's root, in the bundle of a domain exporting it, or, in a tree with no manifest, the one `main_pipe` its bundles declare. A manifest declaring none, or bundles declaring none or several, is a `422 MethodEntryError` naming the bundles by their tree paths; name the pipe with `pipe_code` instead.++**Limits.** A tree is bounded before any of its bytes is read, on the sizes its entries state, and again once read: at most `MAX_METHOD_TREE_FILES` files (default 1,000), `MAX_METHOD_TREE_FILE_KIB` per file (default 32 MiB) and `MAX_METHOD_TREE_TOTAL_KIB` in all (default 64 MiB), each tree of a request counted on its own, and, across the method's tree and the dependency trees beside it, at most `MAX_METHOD_DEPENDENCIES` dependency trees (default 32) and `MAX_METHOD_REQUEST_TOTAL_KIB` in all (default 128 MiB), refused with a `422 MethodTreeTooLarge` naming the limit. A `bundle_b64` archive is also bounded as an archive: a member-count ceiling (`MAX_BUNDLE_FILES`) and a decompressed-size ceiling (`MAX_BUNDLE_TOTAL_KIB`), each a `413 PayloadTooLarge`, its decompression bounded so a zip bomb cannot expand past the ceiling, and an oversized `bundle_b64` refused on its encoded length before it is decoded. Supplying both `bundle_b64` and `files`, an empty archive or a corrupt zip is a `422 InvalidBundle`, invalid base64 a `400 InvalidBase64`, and a tree holding no `.mthds` bundle of its own a `422 InvalidBundle`.++**Files sent in `resources` join the tree.** Beside a tree, each file of [`resources`](pipe-validate.md#sending-the-files-the-steps-read-resources) is added at its path from the tree's root: a file the tree already holds must have the same bytes, or it is a `422 InvalidResourcePath` naming the path, and a file no step names is kept like any other file of the tree. The tree they make is held to the tree's limits again, so files that take it past one are a `422 MethodTreeTooLarge`.++**Sandbox-hosted only for custom Python.** A tree that holds any `.py` file is honored **only on a sandbox-hosted deployment**, where the load reads the source without importing it and execution happens in an isolated sandbox. On a deployment that is not sandbox-hosted it is refused with `403 CustomCodeRequiresSandbox` before anything is written: running caller-supplied code in-process is never done implicitly. A tree without Python is accepted on any deployment.-**Ingest guards.** Bundles are bounded at ingest and rejected with a clear error (never a silent truncation):+**No Python structure classes.** A structure class (a `StructuredContent` subclass) can only be used by importing its module into the runner's own process, which a sandbox-hosted deployment never does, so a tree whose Python declares one is refused with a `403` (`MethodStructuresRefusedError`) naming each file and class, before anything is loaded. The rule is the same however the method arrives: sent as a tree, or fetched by `method_ref`. Declare the types as MTHDS concepts with inline structures instead; a PipeFunc returns them by importing the classes the sandbox generates from those concepts (`from structures import <domain>__<Concept>`). The `structures` module that `pipelex build structures` writes is accepted as long as it is left as generated, and it is not sent to the sandbox, which generates its own.-- A hard **file-count** ceiling (`MAX_BUNDLE_FILES`) and a **total decompressed-size** ceiling (`MAX_BUNDLE_TOTAL_KIB`) → `413 PayloadTooLarge`. The zip path bounds actual decompression, so a zip bomb cannot expand past the ceiling, and an oversized `bundle_b64` is refused on its encoded length *before* it is decoded into memory.-- **Path safety:** entry names that are absolute, use `..` traversal, use backslashes, or carry a Windows drive/`:` form → `422 InvalidBundle`.-- **One entry per path:** two entries naming one file once their names are normalized (`a/./b` and `a//b` are `a/b`), which a zip can also hold as one member name twice, and a file that is also the directory of another entry → `422 InvalidBundle` naming the path.-- Supplying **both** `bundle_b64` and `files`, an empty bundle, a corrupt zip, or a `files` entry whose name or text holds a lone surrogate (which JSON can escape but UTF-8 cannot encode) → `422 InvalidBundle`; invalid base64 → `400 InvalidBase64`.+**Requirements are checked against the sandbox.** A `requirements.txt` at the root of the tree declares the packages its Python imports, one PEP 508 requirement per line (see [Method Trees](../building-methods/method-trees.md#python-packages-requirementstxt)). Nothing is installed for a run: the packages of the deployment's PipeFunc sandbox are the whole offer, and a method asking for a package the sandbox does not hold, or for a version it does not hold, is refused with a `422` (`MethodRequirementsNotMetError`) naming each requirement and the version the sandbox holds, before the method loads and before anything is dispatched. The refusal names the file by its tree path, `requirements.txt`, and no path on the server. A line the standard refuses, such as `-r base.txt`, a URL or a comment after a requirement, is a `422` (`MethodRequirementsFileError`) naming the line. A deployment that states no package set for its sandbox refuses every method declaring requirements with a `500` (`PipeFuncPackageSetUnstatedError`), a fault of its configuration, and so does a method of the server's own library directories whose requirements are not met (`HostLibraryRequirementsError`), whose message is withheld from the caller. A `method_ref` package is checked the same way, on a run as on [`/validate`](pipe-validate.md), where the refusal is an `is_valid: false` verdict located on the requirement's line.-**Sandbox-hosted only for custom Python.** A bundle that ships any `.py` is honored **only on a sandbox-hosted deployment**, where the load path reads the source without importing it and execution happens in an isolated sandbox. On a non-hosted deployment such a bundle is refused with `403 CustomCodeRequiresSandbox` — running caller-supplied code in-process is never done implicitly. A bundle that carries only `.mthds` (no `.py`) is accepted on any deployment.+### Sending files by hash-**No Python structure classes.** A structure class (a `StructuredContent` subclass) can only be used by importing its module into the runner's own process, which a sandbox-hosted deployment never does, so a bundle whose Python declares one is refused with a `403` (`MethodStructuresRefusedError`) naming each file and class, before anything is loaded. The rule is the same however the method arrives: inline as a bundle, or fetched by `method_ref`. Declare the types as MTHDS concepts with inline structures instead; a PipeFunc returns them by importing the classes the sandbox generates from those concepts (`from structures import <domain>__<Concept>`). The `structures` module that `pipelex build structures` writes is accepted as long as it is left as generated, and it is not sent to the sandbox, which generates its own.+This runner keeps a **blob store**, declared by `blob_store: true` in [`GET /v1/version`](index.md), so a sender uploads each file at most once and then sends the method by hash: a second run of an unchanged method uploads nothing, and editing one template uploads one file. These are the MTHDS Protocol's blob routes.-**Requirements are checked against the sandbox.** A `requirements.txt` at the root of the bundle declares the packages its Python imports, one PEP 508 requirement per line (see [Method Trees](../building-methods/method-trees.md#python-packages-requirementstxt)). Nothing is installed for a run: the packages of the deployment's PipeFunc sandbox are the whole offer, and a bundle asking for a package the sandbox does not hold, or for a version it does not hold, is refused with a `422` (`MethodRequirementsNotMetError`) naming each requirement and the version the sandbox holds, before the bundle's files load and before anything is dispatched. The refusal names the file by its path in the bundle, `requirements.txt`, and no path on the server. A line the standard refuses, such as `-r base.txt`, a URL or a comment after a requirement, is a `422` (`MethodRequirementsFileError`) naming the line. A deployment that states no package set for its sandbox refuses every bundle declaring requirements with a `500` (`PipeFuncPackageSetUnstatedError`), a fault of its configuration, and so does a method of the server's own library directories whose requirements are not met (`HostLibraryRequirementsError`), whose message is withheld from the caller. A `method_ref` package is checked the same way, on a run as on [`/validate`](pipe-validate.md), where the refusal is an `is_valid: false` verdict located on the requirement's line.+1. The sender lists its method's tree and computes each file's SHA-256 and the tree's digest.+2. It asks `POST /v1/blobs/missing` which blobs the runner lacks, sending `{"blobs": [{"sha256": "…", "size": n}, …], "digest": "sha256:…"}`, the `digest` being optional.+3. It uploads each missing blob as the answer says: the blob's bytes as the whole body of a request with the answer's `method` to its `url`, carrying exactly its `headers`.+4. It sends the run, or the validation, with a hash entry for each file it chose to send by hash.-### Shipping the method packages the bundle calls+The answer is `{"missing": [{"sha256": "…", "size": n, "upload": {"method": "PUT", "url": "…", "headers": {"content-type": "application/octet-stream"}}}, …]}`, listing only the blobs the runner does not hold; an empty list means there is nothing to upload. Each target is the runner's own `PUT /v1/blobs/{sha256}`, its query signed for that hash and the caller's scope and valid for `BLOB_UPLOAD_URL_TTL_SECONDS` (default an hour). The target starts with `PUBLIC_BASE_URL` when the deployment sets it, the address its clients reach the API at, which a deployment behind a proxy terminating TLS or adding a path prefix sets, and otherwise with the URL the request arrived at. The upload route needs no bearer token, since the signed target is its authorization: a missing, malformed, forged or expired signature is a `403 BlobUploadNotAuthorized`, a body whose SHA-256 is not the path's a `422 BlobMismatch`, and a body past `MAX_METHOD_TREE_FILE_KIB` a `422 MethodTreeTooLarge`. A blob is written whole and never changed, so uploading bytes the runner holds answers `204` and changes nothing. On `POST /v1/blobs/missing`, one hash listed with two sizes is a `422 BlobMismatch`, and blobs past the tree limits a `422 MethodTreeTooLarge`, both before anything is uploaded.-A bundle can call a pipe of another method package by its address, such as `github.com/Pipelex/methods/documents->documents.extract_page_contents_and_views`. A `files` or `bundle_b64` request can carry that package with the bundle, so the run needs no copy installed on the server and no fetch. Put each package under `.mthds/methods/<name>/` at the bundle's root, with its `METHODS.toml` at `.mthds/methods/<name>/METHODS.toml`, which is the layout of a local method store:+**Sizes.** A blob is held at a size, which the runner reads from the storage's metadata without reading the bytes. `POST /v1/blobs/missing` lists a blob the runner does not hold at the size the sender states, bytes held short included, which only a storage that does not publish a write whole could leave and which the upload then writes over. On a run or a validation, every hash entry is looked up by size before any byte is read: a hash the runner holds at another size than the entry states is a `422 BlobMismatch` whose `mismatched_blobs` member gives each hash with its `held_size`, since the bytes under a hash have one size and uploading them again would change nothing, so the entry's size is the one to correct. Only bytes held at the size their entry states are read, a bounded number at a time, and checked against their hash; bytes that do not hash to their key mean the deployment's storage is damaged, a `500 BlobStoreCorrupted` that its operator repairs by removing the object, for it to be uploaded again.++**The digest shortcut.** Once the runner has read a request's tree in full, it records the tree's listing beside its blobs, after every file it names is held, and an upload never records one. Asking `POST /v1/blobs/missing` with the digest of a recorded tree then answers an empty list in one lookup, however many files the tree has, when every blob listed is a file of that tree; a blob outside it is looked up as if no digest were sent.++**Scopes.** Blobs live in the deployment's configured storage, under `<scope>/blobs/<sha256>`, and recorded listings under `<scope>/trees/<digest>`. The scope is the request's `read_scope`, which a multi-tenant host sends for the organization it serves (a query parameter on `POST /v1/blobs/missing`, a body field on the run and validate routes), else the caller's own id, so a hash entry resolves only in the caller's own blobs and asking whether a hash is held tells a caller nothing of another's files. A hash entry the runner does not hold, at the size it states, is a `422 MissingBlobs` whose `missing_blobs` member lists the hashes to upload before sending the request again.++**Signing key.** Each process draws its own signing key unless `BLOB_UPLOAD_SIGNING_KEY` is set. A deployment serving one API through several instances must set it, the same on every instance, since an upload may reach another instance than the one that named its target.++### Sending the method packages the method calls++A method can call a pipe of another method package by its address, such as `github.com/Pipelex/methods/documents->documents.extract_page_contents_and_views`. A request can carry that package with the method, so the run needs no copy installed on the server and no fetch:++- **With `files`**, send each package in `dependencies`, a map from the package's address to the package's own map of file entries, its `METHODS.toml` at its root declaring that address. A package keyed by another address than its manifest declares is a `422 InvalidMethodFiles`, and so is `dependencies` without `files`.+- **In a `bundle_b64` archive**, put each package under `.mthds/methods/<name>/`, with its `METHODS.toml` at `.mthds/methods/<name>/METHODS.toml`, which is the layout of a local method store. The store's `.provenance.json` sidecar is left out of the package.```textbundle.mthds@@ -255,22 +278,15 @@ bundle.mthds.mthds/methods/documents/documents.mthds```-- **A reference finds the package by its manifest.** The package that answers is the one whose `METHODS.toml` declares the referenced address and name, whatever its directory is called. A reference pinned with `@<tag>` is answered by the shipped copy too, and the server logs a warning when the copy's `version` is not that tag.-- **The shipped copy comes first.** The server searches the shipped packages before the copies installed on it and before fetch-on-miss, so a shipped package wins over an installed copy of the same address, and it runs even where fetching is disabled or the address is not one the server can fetch. It serves this run alone: nothing is installed.-- **It is not the bundle's own content.** A shipped package's files are kept out of the bundle's `.mthds` content and out of its library directory, so its domains never merge with the bundle's, and a bundle whose only `.mthds` files are shipped ones is refused as having no `.mthds` file of its own.-- **Its Python never runs in the server's process.** A `.py` file in a shipped package counts for the sandbox gate like any other, so a deployment that is not sandbox-hosted refuses the bundle with `403 CustomCodeRequiresSandbox`. On a sandbox-hosted deployment, a shipped package whose Python declares a structure class is refused with `403 MethodStructuresRefusedError` naming the package's address, as a package fetched by address is.+- **A reference finds the package by its manifest.** The package that answers is the one whose `METHODS.toml` declares the referenced address and name. A reference pinned with `@<tag>` is answered by the shipped copy too, and the server logs a warning when the copy's `version` is not that tag.+- **The shipped copy comes first.** The server searches the shipped packages before the copies installed on it and before fetch-on-miss, so a shipped package wins over an installed copy of the same address, and it runs even where fetching is disabled or the address is not one the server can fetch. It serves this request alone: nothing is installed.+- **It is a tree of its own.** Each package is written into a methods directory of its own, never into the method's tree, so its domains never merge with the method's, and a method whose only bundles are shipped ones is refused as holding no bundle of its own. Its keys are held to the tree's rule as the method's are.+- **Its Python never runs in the server's process.** A `.py` file in a shipped package counts for the sandbox gate like any other. On a sandbox-hosted deployment, a shipped package whose Python declares a structure class is refused with `403 MethodStructuresRefusedError` naming the package's address, as a package fetched by address is.- **A shipped bundle that does not parse refuses the run** with the `422` verdict, whose items name the file as `<address>/<path inside the package>`, never by a path on the server.-Any other `.mthds` path in the bundle is refused with `422 InvalidBundle`, naming the entry and the expected layout, rather than dropped, and so are shipped packages a reference could not tell apart:--- a file under `.mthds/` outside `.mthds/methods/<name>/`, such as `.mthds/foo` or a file directly under `.mthds/methods/`;-- a `.mthds/methods/` directory below the bundle's root, such as `sub/.mthds/methods/…`;-- a `.mthds/` directory inside a shipped package, since a package's own dependencies are not loaded;-- a package directory whose name starts with a dot;-- a package directory without a `METHODS.toml`, or whose `METHODS.toml` is not a valid manifest;-- two package directories whose manifests declare the same address and name, compared without regard to case as a reference is matched, the name being the directory's when the manifest has none. The refusal names the address and each directory, since nothing but their order would decide which one answers.+Two shipped packages a reference could not tell apart, whose manifests declare the same address and name compared without regard to case, the name being the directory's when the manifest has none, are a `422 InvalidMethodFiles` naming the address, since nothing but their order would decide which one answers. In an archive, a package directory without a valid `METHODS.toml` is refused the same way, and any other entry under `.mthds/`, such as a file directly under `.mthds/methods/`, one under a directory whose name no package's directory can take, such as a dot directory, or a package's own `.mthds/` store, is refused by the tree's rule; a package fetched by `method_ref` holds its `.mthds/methods/` to the same rule, its other entries left out of its tree. A package whose manifest declares no name takes the last segment of its address, which must then be a name a directory can take: a dependency keyed by an address ending in `.` or `..` with a nameless manifest is a `422 InvalidMethodFiles`.-Shipping packages is a feature of the run routes. `POST /v1/validate` takes `mthds_contents` or `method_ref`, and a package reference in the validated content resolves there through the server's installed copies and fetch-on-miss only.+The validation of a method sent as a tree, or fetched by `method_ref`, loads its shipped packages too, so `/validate` resolves a dependency the run would resolve.---@@ -287,13 +303,13 @@ Both `/execute` and `/start` accept **`method_ref`** — a globally resolvable a**The reference grammar is `<address>[@<tag>]`.** The address is `github.com/<owner>/<repo>` optionally followed by a package selector within the repository (`github.com` only for now). A bare address means the repository's default branch at HEAD; `@<tag>` pins the repository at that **git tag** (recommended form `vX.Y.Z`) — branch names are refused, so a reference can never silently track a moving branch. Full browser URLs (`https://github.com/...`, including `/tree/<branch>/...` deep links) are accepted and normalized.-**Resolution.** The server fetches the repository (shallow clone, at the tag when one is named), records the **commit SHA** of what was fetched, and locates the requested package **by manifest identity**: it scans the clone for `METHODS.toml` manifests and selects the one whose declared address (plus `name`, for a package in a library repo) equals the requested address — never by directory path. The package's `.mthds` files then run exactly like an inline submission; its non-`.mthds` files travel like a bundle's, and so do the packages it ships under its own `.mthds/methods/<name>/` ([Shipping the method packages the bundle calls](#shipping-the-method-packages-the-bundle-calls)), under the same refusals. `POST /v1/validate` by `method_ref` keeps those shipped packages out of the validated content without searching them, so a dependency the package ships resolves there through the installed copies and fetch-on-miss only.+**Resolution.** The server fetches the repository (shallow clone, at the tag when one is named), records the **commit SHA** of what was fetched, and locates the requested package **by manifest identity**: it scans the clone for `METHODS.toml` manifests and selects the one whose declared address (plus `name`, for a package in a library repo) equals the requested address — never by directory path. It then reads the package's directory **from the commit's git objects**, never from the clone's working tree, and lists it as a method tree under the standard's rule: the package is a tree like one [sent as `files`](#sending-a-method-as-a-tree), written whole and loaded as the caller's, its templates and Python beside its bundles, and the packages it ships under its own `.mthds/methods/<name>/` become dependency trees ([Sending the method packages the method calls](#sending-the-method-packages-the-method-calls)), under the same refusals.-**Entry pipe.** The pipe to run defaults to the package manifest's `main_pipe`; an explicit `pipe_code` in the request overrides it (to run another pipe from the fetched package).+**Entry pipe.** The pipe to run is the tree's entry pipe by the protocol's rule, the package manifest's `main_pipe`; an explicit `pipe_code` in the request overrides it (to run another pipe from the fetched package).-**Provenance.** The response carries `method_provenance = {address, tag, commit_sha}` — on `/execute` only for `method_ref` runs, on the `/start` ack as a nullable field. Tags can move; the recorded SHA is what keeps a run explainable after the fact.+**Provenance.** The response carries `method_provenance = {address, tag, commit_sha, tree_digest}` — on `/execute` only for `method_ref` runs, on the `/start` ack as a nullable field. Tags can move; the recorded SHA is what keeps a run explainable after the fact, and `tree_digest` names the exact tree the run loaded.-**Custom Python follows the bundle rules, plus one more.** What decides is *where the code would execute*: `.mthds` content is data, always acceptable. On a deployment that is **not** sandbox-hosted, a fetched package carrying any `.py` is refused with `403 CustomCodeRequiresSandbox`. On a sandbox-hosted deployment, PipeFunc `.py` is accepted (captured as text, executed in the isolated sandbox) — but a package declaring **Python structure classes** (`StructuredContent` subclasses) is refused with a `403` (`MethodStructuresRefusedError`), exactly as a bundle is: see [No Python structure classes](#shipping-a-method-bundle-custom-pipefunc).+**Custom Python follows the tree rules, plus one more.** What decides is *where the code would execute*: `.mthds` content is data, always acceptable. On a deployment that is **not** sandbox-hosted, a fetched package carrying any `.py` is refused with `403 CustomCodeRequiresSandbox`. On a sandbox-hosted deployment, PipeFunc `.py` is accepted (captured as text, executed in the isolated sandbox) — but a package declaring **Python structure classes** (`StructuredContent` subclasses) is refused with a `403` (`MethodStructuresRefusedError`), exactly as a tree sent with the request is: see [No Python structure classes](#sending-a-method-as-a-tree).**Errors.** Every resolution failure is a distinct RFC 7807 `problem+json` with the originating class as `error_type` (see [Error responses](error-responses.md)):@@ -308,7 +324,7 @@ Both `/execute` and `/start` accept **`method_ref`** — a globally resolvable a| Package declares Python structure classes | 403 | `MethodStructuresRefusedError` || Package ships `.py` on a non-sandbox deployment | 403 | `CustomCodeRequiresSandbox` |-**The clone cache.** Fetched repositories are cached on local disk **keyed by resolved commit SHA** — never by tag. Each request first resolves the reference to its current SHA with a cheap `git ls-remote` (no clone); a cached SHA is reused, a new one (a moved tag, an advanced HEAD) triggers a fresh clone. The cache is per server instance and bounded by count, total bytes, and age.+**The clone cache.** Fetched repositories are cached on local disk **keyed by resolved commit SHA** — never by tag. Each request first resolves the reference to its current SHA with a cheap `git ls-remote` (no clone); a cached SHA is reused, a new one (a moved tag, an advanced HEAD) triggers a fresh clone. The cache is per server instance and bounded by count, total bytes, and age, oldest first; a clone a request is still reading is never evicted, though it counts toward the bounds. That protection holds within one process, so a deployment running several worker processes gives each its own `METHOD_CACHE_DIR`.**Environment knobs.** The cache: `MAX_METHOD_CACHE_CLONES` (default 64), `MAX_METHOD_CACHE_TOTAL_KIB` (default 512 MiB), `MAX_METHOD_CACHE_AGE_HOURS` (default 24), `METHOD_CACHE_DIR` (default: a fixed directory under the system temp dir). The fetch itself is bounded by the pipelex runtime's fetched-package ceilings: `PIPELEX_MAX_FETCHED_PACKAGE_FILES` (default 256), `PIPELEX_MAX_FETCHED_PACKAGE_TOTAL_KIB` (default 8 MiB), and the manifest-scan bounds `PIPELEX_MAX_SCANNED_MANIFESTS` (default 100 manifests considered per fetched repository) and `PIPELEX_MAX_MANIFEST_FILE_KIB` (default 256 KiB per `METHODS.toml` read during package location). All are read once at process start.diff --git a/docs/api-server/pipe-validate.md b/docs/api-server/pipe-validate.mdindex 3c6abd39b..069522768 100644--- a/docs/api-server/pipe-validate.md+++ b/docs/api-server/pipe-validate.md@@ -4,7 +4,7 @@ Validate MTHDS content by parsing, loading, and dry-running pipes without execut**Endpoint:** `POST /v1/validate`-`/validate` is a **diagnostic endpoint**: any verdict the validator can produce — valid, invalid, or valid-but-not-runnable — rides an HTTP **200**, discriminated in the body on `is_valid`. "Invalid, here are the problems" is the *successful product* of the call, not a transport failure. Non-2xx is reserved for the cases where **no verdict could be produced** (a malformed request body, an `mthds_sources` length mismatch, a `method_ref` that could not be resolved, auth, or a server fault).+`/validate` is a **diagnostic endpoint**: any verdict the validator can produce — valid, invalid, or valid-but-not-runnable — rides an HTTP **200**, discriminated in the body on `is_valid`. "Invalid, here are the problems" is the *successful product* of the call, not a transport failure. Non-2xx is reserved for the cases where **no verdict could be produced** (a malformed request body, an `mthds_sources` length mismatch, a method tree that cannot be read, a `method_ref` that could not be resolved, auth, or a server fault).**Request Body:**@@ -16,11 +16,14 @@ Validate MTHDS content by parsing, loading, and dry-running pipes without execut**Request Fields:**-- `mthds_contents` (list[str], optional): MTHDS contents to validate (always an array, even for a single file). Exactly one of `mthds_contents` / `method_ref` must be provided-- `method_ref` (string, optional): **Pipelex-API extension.** Validate a published method by address — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — resolved by this server through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the repository is fetched at the tag, the package located by manifest identity, and its `.mthds` files validated with their **real relative paths** feeding the per-file sources (so `validation_errors[].source` names the owning file). The method packages the package ships under its own `.mthds/methods/<name>/` are not part of the validated content, and validation does not search them: a call into one resolves here through the server's installed copies and fetch-on-miss only, while a [run](pipe-run.md#shipping-the-method-packages-the-bundle-calls) uses the shipped copy first. Exactly one of `mthds_contents` / `method_ref`; `mthds_sources` cannot accompany it. A package holding Python and a `requirements.txt` has its requirements checked on this server against the package set of the deployment's PipeFunc sandbox before anything is validated, and a requirement the sandbox does not meet, or a line the standard refuses, makes the verdict `is_valid: false` with one item per line, whose `source` is `requirements.txt` and whose `line` is the line's number+- `mthds_contents` (list[str], optional): MTHDS contents to validate (always an array, even for a single file). Exactly one of `mthds_contents` / `files` / `method_ref` must be provided+- `files` (object, optional): The method as a map of file entries, the MTHDS standard's method tree, each value the file's text, `{"base64": "…"}` or `{"sha256": "…", "size": n}`, read as on a run — see [Validating a method tree](#validating-a-method-tree). Exactly one of `mthds_contents` / `files` / `method_ref`+- `dependencies` (object, optional): **Pipelex-API extension.** The method packages the method calls by address that the runner cannot fetch, each a map of file entries keyed by its address, as on a [run](pipe-run.md#sending-the-method-packages-the-method-calls). Accompanies `files`+- `method_ref` (string, optional): **Pipelex-API extension.** Validate a published method by address — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — resolved by this server through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the repository is fetched at the tag, the package located by manifest identity, and its tree read from the commit's git objects and validated as a tree sent in `files` is — see [Validating a method tree](#validating-a-method-tree). Exactly one of `mthds_contents` / `files` / `method_ref`; `mthds_sources` cannot accompany it. A package holding Python and a `requirements.txt` has its requirements checked on this server against the package set of the deployment's PipeFunc sandbox before anything is validated, and a requirement the sandbox does not meet, or a line the standard refuses, makes the verdict `is_valid: false` with one item per line, whose `source` is `requirements.txt` and whose `line` is the line's number+- `read_scope` (string, optional): **Pipelex-API extension.** The scope of the blob store a hash entry of `files` resolves in, as a run's `read_scope`; omitted, the caller's own id is the scope — see [Sending files by hash](pipe-run.md#sending-files-by-hash)- `allow_signatures` (boolean, optional, default `false`): controls only the **sweep mechanics** for `PipeSignature` placeholders — whether signature pipes are mock-run during the dry-run sweep and therefore listed in `validated_pipes`. It does **not** change the verdict: an unimplemented signature is never a rejection, it is a *runnability fact* reported via `pending_signatures` + `is_runnable` in both modes-- `mthds_sources` (list[str] | null, optional): per-file sources, parallel to `mthds_contents` — see [Sourcing submitted files](#sourcing-submitted-files). When present it must match `mthds_contents` in length (a mismatch is a 422 request error)-- `resources` (object | null, optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` only; beside `method_ref` it is a 422+- `mthds_sources` (list[str] | null, optional): per-file sources, parallel to `mthds_contents` — see [Sourcing submitted files](#sourcing-submitted-files). When present it must match `mthds_contents` in length (a mismatch is a 422 request error), and it accompanies `mthds_contents` alone, since a tree's files are named by their tree paths+- `resources` (object | null, optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` or `files`, whose tree each file joins; beside `method_ref` it is a 422- `render` (list[str], optional, default `[]`): opt-in *rendered text* to attach to the verdict — see [Opt-in extras](#opt-in-extras-render-and-views)- `views` (list[str], optional, default `[]`): opt-in *structured views* to attach to the verdict — see [Opt-in extras](#opt-in-extras-render-and-views)- `orchestration_mode` (string | null, optional): per-request backend override for the validation dispatch — see [Where validation runs](#where-validation-runs). Honored only when the deployment allows it; a forbidden override is a 403@@ -156,6 +159,15 @@ Neither axis is part of the verdict contract: a machine consumer branches on theThe submit path carries bundle text, not file paths, so by default the runtime cannot tell the client which file an error belongs to — `source` comes back `null`. Send `mthds_sources` parallel to `mthds_contents` to fix this: each source is the logical identity of that content (e.g. the file's path relative to the submitted directory), and the runtime threads it onto the corresponding `blueprint.source`. The source then rides back on both arms — `bundle_blueprint.source` on the valid arm, and `validation_errors[].source` on the invalid arm — so a multi-file editor client can map a cross-file diagnostic to the file that owns it. Omit `mthds_sources` (or send `null`) and behavior is exactly as before. The list, when present, must be the same length as `mthds_contents`; a mismatch is a request-shape 422 (it is the caller's wiring bug, caught before the validation sweep runs). A source is a label for diagnostics and nothing more: even one naming a real file on the server is never a directory the server reads from, so a step's template file comes from [`resources`](#sending-the-files-the-steps-read-resources) alone, and an address-based reference is not searched for above it.+## Validating a method tree++A method sent in `files`, or fetched by `method_ref`, is validated whole, as it would run: the runner reads the tree under the standard's rule, writes it into a temporary directory, and validates it as one library directory of the caller's, with the packages it ships in `dependencies` (or, for a `method_ref` package, under its own `.mthds/methods/<name>/`) loaded beside it, so a call into one resolves as on a run. Every bundle is named by its tree path, so `validation_errors[].source` names the owning file, and no item names a path on the server.++- **Python is checked by syntax.** Every `.py` file of the tree is read and compiled, never imported or run, whether or not a step names it: a file that does not parse, or is not UTF-8, makes the verdict `is_valid: false` with an item whose `source` is the file's tree path and whose `line` is the line of the error. A `PipeFunc` step's function is then located in the tree and its form checked statically. As on a run, a tree holding Python is refused with a `403` (`CustomCodeRequiresSandbox`) on a deployment that is not sandbox-hosted.+- **The default pipe and the graph follow the tree's entry pipe.** `default_pipe_ref` is the pipe the protocol's rule names, the `main_pipe` of the root `METHODS.toml` or the one `main_pipe` its bundles declare, and the graph is drawn from it. A tree with no entry pipe validates with `default_pipe_ref: null` and `graph_spec: null`.+- **Refusals of the tree itself produce no verdict**: a key the tree's rule refuses or a malformed entry is a **422** `InvalidMethodFiles`, a hash the runner does not hold a **422** `MissingBlobs`, and a tree past its limits a **422** `MethodTreeTooLarge`, as on a [run](pipe-run.md#sending-a-method-as-a-tree).+- **The `mthds_contents` echo** holds the text of the tree's own bundles.+## Sending the files the steps read (`resources`)A bundle sent as content has no directory, so a file one of its steps reads from beside the bundle, such as a `PipeDocGen` step's `template_file`, is sent with it in `resources`. The same field, with the same rules, is accepted by [`/v1/execute` and `/v1/start`](pipe-run.md), [`/v1/pipe-io`](pipe-io.md), and [`/v1/resolve` and `/v1/codegen`](codegen.md#selecting-the-closure).@@ -167,10 +179,10 @@ A bundle sent as content has no directory, so a file one of its steps reads from}```-- **Each key is the path a step names**, relative to the bundle and written with forward slashes, exactly as in its `template_file`. `.` segments and repeated slashes collapse, so `./templates/invoice.docx` names the same file; every bundle of the request is read as sitting in one directory. An empty or absolute path, a `..` segment, a backslash, a colon, or two keys naming the same file is a **422** `InvalidResourcePath`, refused before any value is decoded. A method bundle sent to `/v1/execute` or `/v1/start` may ship the file instead, beside the `.mthds` file whose step names it, and a file both shipped and sent must be the same (see [Pipe Run](pipe-run.md#shipping-a-method-bundle-custom-pipefunc)).+- **Each key is the path a step names**, relative to the bundle and written with forward slashes, exactly as in its `template_file`. `.` segments and repeated slashes collapse, so `./templates/invoice.docx` names the same file; every bundle of the request is read as sitting in one directory. An empty or absolute path, a `..` segment, a backslash, a colon, or two keys naming the same file is a **422** `InvalidResourcePath`, refused before any value is decoded. A method sent as a tree may hold the file instead, beside the bundle whose step names it; beside a tree, each key of `resources` joins it at that path from the tree's root, and a file the tree already holds must have the same bytes (see [Pipe Run](pipe-run.md#sending-a-method-as-a-tree)).- **Each value is the file's bytes** in standard base64 (RFC 4648, with padding). A value that does not decode is a **400** `InvalidBase64`.-- **Every file must be one a step names.** A file no step of the bundle reads is a **422** `BundleResourceNotNamedError`, whose `detail` lists the unnamed paths: a misspelled key fails loudly rather than leaving its step without its file. This is a request error, never an `is_valid: false` verdict.-- **The sizes are bounded**: at most `MAX_RESOURCES_PER_REQUEST` files (32 by default), each at most `MAX_RESOURCE_FILE_KIB` once decoded (5 MiB by default), and at most `MAX_RESOURCES_TOTAL_KIB` across the request (10 MiB by default), counting the files a method bundle ships for its steps. A request over any of them is a **413** `PayloadTooLarge`, an oversized file refused on its encoded length before it is decoded. The whole body stays under `MAX_REQUEST_BODY_MIB` as well.+- **Beside inline contents, every file must be one a step names.** A file no step of the bundle reads is a **422** `BundleResourceNotNamedError`, whose `detail` lists the unnamed paths: a misspelled key fails loudly rather than leaving its step without its file. This is a request error, never an `is_valid: false` verdict. Beside a tree, a file no step names is kept like any other file of the tree.+- **The sizes are bounded**: at most `MAX_RESOURCES_PER_REQUEST` files (32 by default), each at most `MAX_RESOURCE_FILE_KIB` once decoded (5 MiB by default), and at most `MAX_RESOURCES_TOTAL_KIB` across the request (10 MiB by default). A request over any of them is a **413** `PayloadTooLarge`, an oversized file refused on its encoded length before it is decoded. The whole body stays under `MAX_REQUEST_BODY_MIB` as well.- **The files make one method tree**, with the request's bundles at its root. A run names that tree by its digest, as it names the tree of a method loaded from a directory, and carries none of the files in its payloads, a remote orchestrator's included: the step that reads one reads it from the tree, through the deployment's method file store. Two files colliding in one tree, such as two keys equal once case-folded, are a **422** `BundleResourcePathError`.The files are checked as files on disk are. An `.html` template file is checked against the step's inputs when the bundle loads, and the document engine's template check runs on an office template file in the validation's dry run, where its errors are validation errors on the step. A step whose `template_file` could not be a key, being empty or absolute, or holding a `..` segment, a backslash, a colon or a NUL character, is refused when its bundle is parsed, whatever the request sends: the verdict's one item is a `blueprint_validation` item located at the step by its `pipe_code` and by `field_path` `pipe.<code>`, whose message names the field without repeating the path. A step whose `template_file` could be a key but was not sent is refused at load, and the verdict's `validation_errors[]` says the file was not supplied with its bundle and names the key it is expected under.@@ -191,7 +203,8 @@ A validation is not a run, but it still emits telemetry: the sweep's `pipe_dry_rOnly conditions where the endpoint could not produce a verdict are non-2xx, rendered as [RFC 7807 problem documents](error-responses.md):-- **422** — a malformed request body, an `mthds_sources` / `mthds_contents` length mismatch, both/neither of `mthds_contents` / `method_ref`, or a malformed `analytics_groups` (`error_type` `InvalidAnalyticsGroups`) — request-shape errors caught before the runtime.+- **422** — a malformed request body, an `mthds_sources` / `mthds_contents` length mismatch, more or fewer than one of `mthds_contents` / `files` / `method_ref`, or a malformed `analytics_groups` (`error_type` `InvalidAnalyticsGroups`) — request-shape errors caught before the runtime.+- **422** on a method tree — a key or entry the tree's rule refuses (`InvalidMethodFiles`), a hash entry the runner does not hold (`MissingBlobs`), or a tree past its limits (`MethodTreeTooLarge`) — see [Validating a method tree](#validating-a-method-tree).- **422**, **413** and **400** on `resources` — a path that cannot name a file inside the bundle (`InvalidResourcePath`), a file no step names (`BundleResourceNotNamedError`), files that cannot make one tree (`BundleResourcePathError`), a file or a request over its size limit (`PayloadTooLarge`), or a value that is not base64 (`InvalidBase64`) — see [Sending the files the steps read](#sending-the-files-the-steps-read-resources).- **401 / 403** — unauthenticated / forbidden (including a per-request `orchestration_mode` override the deployment does not allow).- **`method_ref` resolution failures** — a selector that could not be resolved produced no verdict, so it is never rendered as `is_valid: false`: a malformed reference or a failed fetch is a **422**, no matching package in the repository a **404**, and the custom-Python policy (the sandbox gate, the structures refusal) a **403** — the same statuses and `error_type`s as on the run routes (see the [error table](pipe-run.md#running-a-method-by-address-method_ref)).diff --git a/docs/building-methods/packages.md b/docs/building-methods/packages.mdindex edb74f95c..9eeaef5cc 100644--- a/docs/building-methods/packages.md+++ b/docs/building-methods/packages.md@@ -135,7 +135,7 @@ An address-based reference resolves against the **installed methods** — `~/.mtA few behaviors worth knowing:- **Installed wins over a fetch.** Once a method is installed, it is used as-is — including when a reference pins a different `@<tag>` than what is installed (a warning tells you so). Remove the installed directory to re-fetch.-- **A package can travel with the bundle.** A bundle sent over HTTP to a Pipelex API server as `files` or `bundle_b64` can carry the packages it calls under `.mthds/methods/<name>/` at its root, each with its `METHODS.toml`. The server searches them before its installed copies and before fetch-on-miss, for that run alone, so a shipped package wins over an installed copy and runs where fetching is disabled; see [Shipping the method packages the bundle calls](../api-server/pipe-run.md#shipping-the-method-packages-the-bundle-calls). On disk, the same layout beside the bundle is one of the local stores above.+- **A package can travel with the method.** A method sent over HTTP to a Pipelex API server as `files` can carry the packages it calls in `dependencies`, each a map of file entries keyed by its address, and one sent as a `bundle_b64` archive under `.mthds/methods/<name>/` at its root, each with its `METHODS.toml`. The server searches them before its installed copies and before fetch-on-miss, for that request alone, so a shipped package wins over an installed copy and runs where fetching is disabled; see [Sending the method packages the method calls](../api-server/pipe-run.md#sending-the-method-packages-the-method-calls). On disk, the `.mthds/methods/<name>/` layout beside the bundle is one of the local stores above.- **A package bundle that does not parse refuses the load.** When a bundle of a referenced package does not parse, the bundle calling it is refused with the parser's items, each naming the file as `<address>/<path inside the package>`, never by its path on the machine, whether the package was shipped with the bundle, installed or fetched.- **A miss that cannot be bridged refuses the bundle, never a silent pass.** Fetch disabled, an address this runtime cannot fetch, or a failed fetch (a repository that does not exist, a `@tag` that is not a tag, a repository holding no package at that address) refuses the load with one `unresolved_package_dependency` item per package, located on the first pipe that calls it, with the reference as written in `missing_pipe_code` and the reason and the remedy in its message. It is the same invalid verdict as any other mistake in the bundle: `pipelex validate` reports it, `POST /v1/validate` answers `200` with `is_valid: false`, and a run is refused with a `422` before any pipe runs. Every package of the bundle is tried before the refusal, so two bad references give two items.- **Hosted parity warning.** A fetched method declaring Python structure classes runs locally but triggers the same warning as a direct CLI fetch: hosted execution accepts MTHDS concepts and sandboxed PipeFuncs, not in-process Python. On a sandbox-hosted deployment the fetch refuses such a package outright, with the same rule-naming error. The refusal is not specific to fetching: a sandbox-hosted deployment refuses every method whose Python declares structure classes, however it arrived (see [Python classes](concepts/python-classes.md)).diff --git a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md b/docs/building-methods/pipes/pipe-operators/PipeDocGen.mdindex 158e2be4b..4de6d77d6 100644--- a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md+++ b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md@@ -164,7 +164,7 @@ body { font-family: Brand; }- **A URL the template writes is checked when the method loads.** `pipelex validate`, a dry run and every load check each literal relative URL of the template and of the stylesheets it links, such as `<img src="logo.png">`, so a missing or misspelt file fails before any print. A URL the template builds from the run's inputs is checked when the document prints, and a refused one fails the print with a `TemplateUrlRefusedError`.- **A URL with a scheme is left as written.** An `https://` URL is fetched through the SSRF guard, a `data:` URL is decoded, and a `pipelex-storage://` URL is read under the run's read scope, as before, so a template's split between the method's files and the caller's is the URL's own: relative for the method, a stored URI for a caller's file or an earlier output. A URL with no path, such as `#top`, names no file.- **A file name holding `?`, `#` or `%` is written percent-encoded**, `100%25.png` for `100%.png`, since a URL's query and fragment are dropped before it resolves and a `%` starts an escape.-- **A bundle loaded from a string** resolves its inline template's relative URLs from the root of the files sent with it, and with no file sent beside it, it belongs to no method tree, so a relative URL its template writes is refused when the method loads, and one built from the run's inputs fails the print. Load such a method from its folder.+- **A bundle loaded from a string** resolves its inline template's relative URLs from the root of the files sent with it, and with no file sent beside it, it belongs to no method tree, so a relative URL its template writes is refused when the method loads, and one built from the run's inputs fails the print. Load such a method from its folder, or send it to the API server as a tree, in `files` (see [Pipe Run](../../../api-server/pipe-run.md#sending-a-method-as-a-tree)), where every file of the folder travels with it.The engine that prints HTML joins each relative URL to a base URL Pipelex hands it and reads the file through Pipelex ([Document Engine Plugins](../../../under-the-hood/document-engine-plugins.md#relative-urls-the-files-of-the-templates-method-tree)).@@ -172,11 +172,11 @@ The engine that prints HTML joins each relative URL to a base URL Pipelex handsAn `.xlsx`, `.docx` or `.pptx` template file is filled by its engine from the inputs as plain data, so it sees fields, not the sigils and filters of HTML templates. In plain data a native value is its value, and a verdict is its verdict: a `YesNo` is its boolean, a `Choice` its option key and a `Rating` its level, so the file reads `team` rather than `team.choice`, and a verdict's probability or confidence does not reach it. A `pdf` template reads the verdict whole, `team.choice` and `team.confidence` included, as a `PipeCompose` template does. Its contract is documented with the plugin. When the plugin's engine checks templates, `pipelex validate` and the dry run compare the file with the inputs' concepts: a misspelled name is reported before any run.-A bundle loaded from a directory, a dependency package's included, finds its template file beside its own file, and never among the files sent with a request. A bundle loaded from a string has no directory, so its template files come with it: the API's routes take them in the request's `resources`, each keyed by exactly the path its step names, and the Python entry points take them as `submitted_files` (see [Template files sent with a bundle](#template-files-sent-with-a-bundle)). A string-loaded step whose file was not supplied is refused at load, saying so, whatever source label the bundle carries: only a bundle file the runtime read itself has a directory beside it, so a label naming a path, or a `source` written in the bundle's text, never makes the runtime read its own disk. An inline `template` works on every load path.+A bundle loaded from a directory, a dependency package's included, finds its template file beside its own file, and never among the files sent with a request. A bundle loaded from a string has no directory, so its template files come with it: the API's routes take the whole method as a tree in `files`, or its bundles as strings with their template files in the request's `resources`, each keyed by exactly the path its step names, and the Python entry points take them as `submitted_files` (see [Template files sent with a bundle](#template-files-sent-with-a-bundle)). A string-loaded step whose file was not supplied is refused at load, saying so, whatever source label the bundle carries: only a bundle file the runtime read itself has a directory beside it, so a label naming a path, or a `source` written in the bundle's text, never makes the runtime read its own disk. An inline `template` works on every load path.### Template files sent with a bundle-When a bundle is sent as content rather than read from a directory, as it is through the [API server](../../../api-server/pipe-validate.md), every template file a step reads travels beside it:+A method sent to the [API server](../../../api-server/pipe-run.md#sending-a-method-as-a-tree) as a tree, in `files`, carries its folder whole: its bundles, its template files and every image, stylesheet and font its templates name, each at its path in the method, whether a step names it or not, and it loads as if read from its folder. When the bundles are sent as strings instead, in `mthds_contents`, every template file a step reads travels beside them:```json{@@ -187,7 +187,7 @@ When a bundle is sent as content rather than read from a directory, as it is thr- **The key is the path the step names**, relative to the bundle and written with forward slashes: a step with `template_file = "templates/invoice.docx"` reads the file sent as `templates/invoice.docx`, and `./templates/invoice.docx` or `templates//invoice.docx`, as a key or as the path a step names, names the same file. Every bundle of the request is read as sitting in one directory. An absolute path, a `..` segment, a backslash and a colon are refused, as a key and as the path a step names: a step whose `template_file` could not be a key is refused when its bundle is parsed, whatever the request sends, so every file a string-loaded step can name can be sent with it.- **The value is the file's bytes** in standard base64.-- **Every file sent must be one a step names.** A file no step reads is refused, so a misspelled key fails loudly rather than leaving its step without a template. A file a template names by a relative URL, such as its logo, is named by no step, so it cannot be sent this way: such a method is loaded from its folder.+- **Every file sent must be one a step names.** A file no step reads is refused, so a misspelled key fails loudly rather than leaving its step without a template. A file a template names by a relative URL, such as its logo, is named by no step, so it cannot be sent this way: send such a method as a tree, in `files`, or load it from its folder.- **The files are checked like files on disk.** An `.html` template file is checked against the inputs when the method loads, and the engine's template check runs on an office template file at validation and in the dry run.The files sent with a bundle make one method tree, with the bundles sent as strings at its root: the run names that tree by its digest, as it names the tree of a bundle loaded from a directory, and carries none of its files. The print stage reads a template file from the tree, under the run's read scope, to print it or to check it, wherever it runs: in the process that loaded the method, the tree is the one its library holds; on a remote orchestrator such as Temporal, the deployment's method file store serves it. No template file rides in the run's payloads, so none is bounded by them.diff --git a/docs/under-the-hood/storage-provider-plugins.md b/docs/under-the-hood/storage-provider-plugins.mdindex dd4f047ac..a9a1584d6 100644--- a/docs/under-the-hood/storage-provider-plugins.md+++ b/docs/under-the-hood/storage-provider-plugins.md@@ -147,7 +147,7 @@ The duplicate detection is the same fail-loud `_add` helper the inference/orchesA third-party storage plugin is a distribution that:-1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light);+1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light); `_store` publishes the bytes whole or not at all, never leaving an object cut short under its key, and a provider that can read a byte range or an object's size without its bytes also overrides `_load_head` and `_load_size`, whose defaults load the whole object;2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_storage_provider(method="<token>", factory=...)` — and nothing else;3. advertises itself under the `pipelex.plugins.kernel` entry-point group — a storage provider is kernel-layer, and the group is how a plugin declares that (see [Inference Backend Plugins](inference-backend-plugins.md#shipping-it-as-an-out-of-tree-plugin) for what each group means and what happens if you pick the wrong one; the pre-split `pipelex.plugins` group is retired and now fails startup):diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex c73a50dc6..c7be4e507 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -67,6 +67,7 @@ from pipelex.libraries.method_trees import ()from pipelex.libraries.pipe.exceptions import PipeLibraryErrorfrom pipelex.libraries.visibility_utils import check_visibility_for_blueprints, make_visibility_checker+from pipelex.libraries.written_trees import WRITTEN_METHOD_TREESfrom pipelex.methods.exceptions import MethodDependencyFetchError, MethodFetchDisabledErrorfrom pipelex.methods.fetch_on_miss import find_vendored_method, resolve_address_based_methodfrom pipelex.methods.structures_check import ensure_no_structured_content_in_library_sources@@ -685,7 +686,10 @@ class LibraryManager(LibraryManagerAbstract):library_id: str,library_dirs: list[Path] | None = None,library_file_paths: list[Path] | None = None,+ methods_dirs: list[Path] | None = None,) -> list[PipeAbstract]:+ # `methods_dirs` are laid out like `.mthds/methods/` and hold the packages a caller ships with the method these+ # directories hold, searched first for an address-based reference of their bundles and never written.# Ensure libraries exist for this library_idif library_id not in self._libraries:msg = f"Trying to load a library that does not exist: '{library_id}'"@@ -780,7 +784,7 @@ class LibraryManager(LibraryManagerAbstract):"Loading the MTHDS files into the library",fields={"library_id": library_id, "mthds_paths": [str(bundle_file.path) for bundle_file in bundle_files]},)- return self._load_mthds_files_into_library(library_id=library_id, bundle_files=bundle_files)+ return self._load_mthds_files_into_library(library_id=library_id, bundle_files=bundle_files, methods_dirs=methods_dirs)@overridedef is_crate_loaded(self, *, library_id: str, fingerprint: str) -> bool:@@ -1268,7 +1272,9 @@ class LibraryManager(LibraryManagerAbstract):LOCAL_METHOD_FILES.hold_tree(holder=library_id, tree=tree)return live_trees- def _load_mthds_files_into_library(self, *, library_id: str, bundle_files: list[LibraryBundleFile]) -> list[PipeAbstract]:+ def _load_mthds_files_into_library(+ self, *, library_id: str, bundle_files: list[LibraryBundleFile], methods_dirs: list[Path] | None = None+ ) -> list[PipeAbstract]:"""Load MTHDS files into a specific library.This method:@@ -1279,21 +1285,27 @@ class LibraryManager(LibraryManagerAbstract):Args:library_id: The ID of the library to load intobundle_files: The bundle files to load, each the member of its method tree+ methods_dirs: Directories laid out like ``.mthds/methods/`` holding packages for this load alone"""valid_mthds_paths = [bundle_file.path for bundle_file in bundle_files]blueprints: list[PipelexBundleBlueprint] = []for bundle_file in bundle_files:mthds_file_path = bundle_file.path+ # A bundle of a tree a host wrote out for a caller is named by its tree path, never by the host's directory.+ bundle_source = (+ bundle_file.tree_bundle.tree_path+ if bundle_file.tree_bundle.tree_path is not None and WRITTEN_METHOD_TREES.holds_tree(tree=bundle_file.tree_bundle.tree)+ else str(mthds_file_path)+ )try:- blueprint = MthdsParser.make_pipelex_bundle_blueprint(bundle_path=mthds_file_path)- blueprint.source = str(mthds_file_path)+ blueprint = MthdsParser.make_pipelex_bundle_blueprint(bundle_path=mthds_file_path, mthds_source=bundle_source)blueprint.place_in_tree(tree_bundle=bundle_file.tree_bundle)except FileNotFoundError as file_not_found_error:- msg = f"Could not find MTHDS bundle at '{mthds_file_path}'"+ msg = f"Could not find MTHDS bundle at '{bundle_source}'"raise LibraryLoadingError(msg) from file_not_found_errorexcept MthdsParserError as parser_error:# Forward BLUEPRINT validation errors from interpreter- msg = f"Could not load MTHDS bundle from '{mthds_file_path}' because of: {parser_error.message}"+ msg = f"Could not load MTHDS bundle from '{bundle_source}' because of: {parser_error.message}"raise LibraryLoadingError(message=msg,blueprint_validation_errors=parser_error.validation_errors,@@ -1330,7 +1342,7 @@ class LibraryManager(LibraryManagerAbstract):library.loaded_mthds_paths.append(resolved_path)try:- return self.load_from_blueprints(library_id=library_id, blueprints=blueprints)+ return self.load_from_blueprints(library_id=library_id, blueprints=blueprints, methods_dirs=methods_dirs)except ValidationError as validation_error:# Categorize before wrapping. `translate_to_validate_bundle_error` has its own# `except ValidationError` arm that categorizes, but it never sees this error: wrapping itdiff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.pyindex 08c3f8cd3..7ff05c58e 100644--- a/pipelex/libraries/library_manager_abstract.py+++ b/pipelex/libraries/library_manager_abstract.py@@ -188,5 +188,6 @@ class LibraryManagerAbstract(ABC):library_id: str,library_dirs: list[Path] | None = None,library_file_paths: list[Path] | None = None,+ methods_dirs: list[Path] | None = None,) -> list[PipeAbstract]:passdiff --git a/pipelex/libraries/method_trees.py b/pipelex/libraries/method_trees.pyindex c4240f816..a9441eb1c 100644--- a/pipelex/libraries/method_trees.py+++ b/pipelex/libraries/method_trees.py@@ -17,6 +17,9 @@ whose own tree is empty, one holding nothing but nested methods, still has its nEach tree is listed under `[interpreter.method_tree_limits]`, this machine's own limits, read once per load: the listerstops before hashing any file once a tree passes one, so a directory handed over by mistake is refused in moments.++A directory a host wrote a tree out as, and holds in `WRITTEN_METHOD_TREES`, is never listed: every listing of it returns+the tree as written, so the membership rule is not applied twice (`pipelex.libraries.written_trees`)."""from collections.abc import Generator, Iterable@@ -48,6 +51,7 @@ from pipelex.core.method_trees.method_file_store import HeldFilefrom pipelex.core.method_trees.tree_bundle import TreeBundlefrom pipelex.hosted.execution import get_or_load_pipelex_configfrom pipelex.libraries.exceptions import HostLibraryRequirementsError, HostLibraryTreeError, MethodRequirementsError, MethodTreeLoadError+from pipelex.libraries.written_trees import WRITTEN_METHOD_TREESLIMITS_ADVICE = ("Hand over the method's own folder rather than a directory above it, or raise the limits of "@@ -82,6 +86,9 @@ def list_local_method_tree(*, handed_over: Path, limits: TreeLimits | None = NonRaises:MethodTreeLoadError: The standard refused the tree, it passed a limit, or it could not be read."""+ written = WRITTEN_METHOD_TREES.tree_written_at(directory=handed_over)+ if written is not None:+ return writtentry:return list_method_tree(handed_over, limits=limits or local_tree_limits())except TreeLimitError as exc:@@ -113,6 +120,9 @@ def list_package_method_tree(*, root: Path, package_address: str, limits: TreeLiRaises:MethodTreeLoadError: The standard refused the tree, it passed a limit, or it could not be read."""+ written = WRITTEN_METHOD_TREES.tree_written_at(directory=root)+ if written is not None:+ return writtentry:return list_method_tree_at_root(root, limits=limits or local_tree_limits())except (MethodTreeRefusedError, TreeLimitError, TreeSourceError) as exc:@@ -362,6 +372,9 @@ def list_library_trees(*, library_dir: Path, limits: TreeLimits | None = None) -if not library_dir.exists():log.debug(f"A library directory that does not exist was skipped: {library_dir}")return []+ written = WRITTEN_METHOD_TREES.tree_written_at(directory=library_dir)+ if written is not None:+ return [written]active_limits = limits or local_tree_limits()try:tree = list_method_tree(library_dir, limits=active_limits)diff --git a/pipelex/libraries/written_trees.py b/pipelex/libraries/written_trees.pynew file mode 100644index 000000000..418dfed14--- /dev/null+++ b/pipelex/libraries/written_trees.py@@ -0,0 +1,113 @@+"""Method trees a host writes out to its own disk to load them, read back as written.++A server receives a method as a map of file entries, or reads a fetched package from its commit's git objects, and+writes the tree's files into a directory of its own to load them. Every file of that copy is a file of the tree, and+the membership rule is never applied to it again (`mthds/docs/spec/method-tree.md`, "A tree written out is read back as+written"): the map was the tree, and a second pass of the rule could drop a file the map holds, a file a `.gitignore`+of the tree names say.++While the writer holds a tree in `WRITTEN_METHOD_TREES`, every listing of its directory, as a library directory or as a+dependency package's root, returns the tree as written (`pipelex.libraries.method_trees`), and the loader names each+of its bundles by its tree path, never by the directory, which is a path on this host the caller never chose.+"""++import threading+from collections.abc import Generator, Mapping+from contextlib import contextmanager+from pathlib import Path++from mthds.method_tree.exceptions import MethodTreeRefusedError, TreeSourceError+from mthds.method_tree.models import MethodTree+from mthds.method_tree.written_tree import read_written_tree++from pipelex.base_exceptions import PipelexUnexpectedError+++class WrittenMethodTrees:+ """The method trees this process wrote out to load them, by the directory each was written as.++ Process-global like the local method file store: a request writes its tree, holds it here for the load, and releases+ it when the request ends, so a listing of the directory meanwhile returns the tree as written.+ """++ def __init__(self) -> None:+ self._trees: dict[Path, MethodTree] = {}+ self._lock = threading.Lock()++ def tree_written_at(self, *, directory: Path) -> MethodTree | None:+ """The tree held as written at `directory`, or None when no tree written there is held."""+ with self._lock:+ if not self._trees:+ return None+ resolved = directory.resolve()+ with self._lock:+ return self._trees.get(resolved)++ def holds_tree(self, *, tree: MethodTree) -> bool:+ """Whether `tree` is a tree held as written, so its files are named by their tree paths."""+ if tree.directory is None:+ return False+ return self.tree_written_at(directory=tree.directory) is tree++ @contextmanager+ def holding(self, *, tree: MethodTree) -> Generator[MethodTree, None, None]:+ """Hold a tree written out for as long as the block runs, and release it on the way out.++ Raises:+ PipelexUnexpectedError: The tree was not read from a directory, or another tree written at its directory is held.+ """+ if tree.directory is None:+ msg = "A method tree is held as written only when it was read back from the directory it was written as."+ raise PipelexUnexpectedError(msg)+ resolved = tree.directory.resolve()+ with self._lock:+ if resolved in self._trees:+ msg = f"A method tree written at '{resolved}' is already held."+ raise PipelexUnexpectedError(msg)+ self._trees[resolved] = tree+ try:+ yield tree+ finally:+ with self._lock:+ self._trees.pop(resolved, None)+++WRITTEN_METHOD_TREES = WrittenMethodTrees()+++def write_method_tree(*, contents: Mapping[str, bytes], destination: Path, expected_digest: str) -> MethodTree:+ """Write a tree's files at their tree paths as `destination`, and read the copy back as written.++ `destination` must not exist, or be an empty directory. The copy is read back as written (`read_written_tree`), and+ its digest must be the tree's: the files were checked as a tree before they were written, so a copy that reads back+ otherwise is this host's fault, never the caller's.++ Args:+ contents: Each file's bytes by its tree path, as a map of file entries carried them.+ destination: The directory to write the tree as.+ expected_digest: The digest of the tree the contents make.++ Returns:+ The tree as written, listed from `destination` resolved.++ Raises:+ PipelexUnexpectedError: The copy reads back as another tree, or as no tree at all.+ OSError: A file could not be written.+ """+ destination.mkdir(parents=True, exist_ok=True)+ if any(destination.iterdir()):+ msg = f"Cannot write a method tree at '{destination}': it already holds files."+ raise PipelexUnexpectedError(msg)+ for tree_path, data in contents.items():+ target = destination.joinpath(*tree_path.split("/"))+ target.parent.mkdir(parents=True, exist_ok=True)+ target.write_bytes(data)+ try:+ written = read_written_tree(destination.resolve())+ except (MethodTreeRefusedError, TreeSourceError) as exc:+ msg = f"The method tree written at '{destination}' cannot be read back: {exc.message}"+ raise PipelexUnexpectedError(msg) from exc+ if written.digest != expected_digest:+ msg = f"The method tree written at '{destination}' reads back as {written.digest}, not as the tree it was written from, {expected_digest}."+ raise PipelexUnexpectedError(msg)+ return writtendiff --git a/pipelex/methods/fetching.py b/pipelex/methods/fetching.pyindex 382f671a2..4564ac5c4 100644--- a/pipelex/methods/fetching.py+++ b/pipelex/methods/fetching.py@@ -12,7 +12,7 @@ from pathlib import Pathfrom mthds.package.exceptions import VCSFetchErrorfrom mthds.package.manifest.schema import MethodsManifestfrom mthds.package.vcs_resolver import clone_at_version, clone_default_branch-from pydantic import BaseModel, ConfigDict+from pydantic import BaseModel, ConfigDict, Fieldfrom pipelex.methods.exceptions import MethodFetchError, MethodPackageSymlinkError, MethodPackageTooLargeErrorfrom pipelex.methods.fetch_limits import MAX_FETCHED_PACKAGE_FILES, MAX_FETCHED_PACKAGE_TOTAL_BYTES@@ -33,6 +33,13 @@ class MethodProvenance(BaseModel):address: strtag: str | None = Nonecommit_sha: str+ tree_digest: str | None = Field(+ default=None,+ description=(+ "The digest of the package's method tree, `sha256:` and 64 lowercase hex characters, as read from the commit's "+ "git objects, set by a runner that reads the package as a tree, and null otherwise."+ ),+ )class FetchedMethodPackage(BaseModel):diff --git a/pipelex/migration/function_names.py b/pipelex/migration/function_names.pyindex 5e1ffb2ba..f4dfc1fac 100644--- a/pipelex/migration/function_names.py+++ b/pipelex/migration/function_names.py@@ -276,7 +276,9 @@ def _marked_functions(*, files: Mapping[str, str]) -> list[_MarkedFunction]:continuetry:module = ast.parse(files[tree_path], filename=tree_path)- except SyntaxError:+ except (SyntaxError, ValueError, RecursionError, MemoryError):+ # A source that does not parse declares no function: a NUL byte is a ValueError on Python 3.11, and+ # nesting deep enough to exhaust the parser a RecursionError or a MemoryError.continuemarker_refs = marker_references(module=module)if not marker_refs:diff --git a/pipelex/mthds_parsing/parser.py b/pipelex/mthds_parsing/parser.pyindex e3b040ed6..fc99e3944 100644--- a/pipelex/mthds_parsing/parser.py+++ b/pipelex/mthds_parsing/parser.py@@ -14,6 +14,16 @@ from pipelex.tools.misc.toml_utils import load_toml_from_content, load_toml_fromfrom pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error+def _decode_error_position(exc: UnicodeDecodeError) -> tuple[int, int]:+ """The line and the column, both counted from 1 as tomli counts them, of the first byte that did not decode.++ The bytes before it decode, so the column counts the characters before it on its line.+ """+ before = exc.object[: exc.start].decode("utf-8")+ line_start = before.rfind("\n") + 1+ return before.count("\n") + 1, len(before) - line_start + 1++class MthdsParser(BaseModel):"""MTHDS -> PipelexBundleBlueprint"""@@ -37,7 +47,8 @@ class MthdsParser(BaseModel):both on the validated blueprint AND on any blueprint-validation error thecategorizer reads off that dict — mirroring the real file path the diskpath already records, and overriding any ``source`` the bundle text- itself declared. Ignored when ``bundle_path`` is provided (the path wins).+ itself declared. With ``bundle_path``, it names the file in place of its path on+ this host, as a host names a file of a tree it wrote out for a caller by its tree path.``file_origin`` says where the bundle's steps find the files they name, and so the rule a filereference, such as a ``PipeDocGen`` step's ``template_file``, is held to when the blueprint validates.@@ -52,7 +63,7 @@ class MthdsParser(BaseModel):try:if bundle_path is not None:blueprint_dict = load_toml_from_path(path=str(bundle_path))- blueprint_dict[PIPELEX_BUNDLE_BLUEPRINT_SOURCE_FIELD] = str(bundle_path)+ blueprint_dict[PIPELEX_BUNDLE_BLUEPRINT_SOURCE_FIELD] = mthds_source if mthds_source is not None else str(bundle_path)validation_context = BundleValidationContext(file_origin=BundleFileOrigin.METHOD_TREE)elif mthds_content is not None:validation_context = BundleValidationContext(file_origin=file_origin)@@ -66,8 +77,12 @@ class MthdsParser(BaseModel):msg = "Either 'bundle_path' or 'mthds_content' must be provided for the MthdsParser to make a PipelexBundleBlueprint"raise MthdsParserError(msg)except TomlError as exc:- msg = f"TOML syntax error at line {exc.lineno}, column {exc.colno}: {exc.message}"- source = str(bundle_path) if bundle_path is not None else mthds_source+ toml_message = exc.message+ if bundle_path is not None and mthds_source is not None:+ # The file is named by its source, never by its path on this host, the error's message included.+ toml_message = toml_message.replace(str(bundle_path), mthds_source)+ msg = f"TOML syntax error at line {exc.lineno}, column {exc.colno}: {toml_message}"+ source = str(bundle_path) if bundle_path is not None and mthds_source is None else mthds_sourceraise MthdsParserError(msg,validation_errors=[@@ -80,6 +95,15 @@ class MthdsParser(BaseModel):)],) from exc+ except UnicodeDecodeError as exc:+ # tomli reads a file as bytes and decodes them as UTF-8, so a file of a tree that is not text fails here.+ source = mthds_source if mthds_source is not None else str(bundle_path)+ line, column = _decode_error_position(exc)+ msg = f"The MTHDS file is not UTF-8 text: the byte at line {line}, column {column} does not decode."+ raise MthdsParserError(+ msg,+ validation_errors=[PipelexBundleBlueprintValidationErrorData(source=source, message=msg, line=line, column=column)],+ ) from excif not blueprint_dict:msg = "Could not make 'PipelexBundleBlueprint': no blueprint found in the MTHDS file"diff --git a/pipelex/pipe_operators/func/pipe_func_definition_check.py b/pipelex/pipe_operators/func/pipe_func_definition_check.pyindex 0d887ffb6..e8c98c39b 100644--- a/pipelex/pipe_operators/func/pipe_func_definition_check.py+++ b/pipelex/pipe_operators/func/pipe_func_definition_check.py@@ -30,6 +30,11 @@ def _parsed_module(*, source: str, module_label: str) -> ast.Module | str:return ast.parse(source, filename=module_label)except SyntaxError as exc:return f"The module {module_label} does not parse as Python: {exc.msg} (line {exc.lineno})."+ except ValueError as exc:+ # Python 3.11 refuses a source holding a NUL byte with a ValueError where later versions raise a SyntaxError.+ return f"The module {module_label} does not parse as Python: {exc}."+ except (RecursionError, MemoryError):+ return f"The module {module_label} does not parse as Python: it nests too deeply."def pipe_func_definition_violation(*, source: str, function: str, module_label: str) -> str | None:diff --git a/pipelex/pipeline/direct_bundle_validator.py b/pipelex/pipeline/direct_bundle_validator.pyindex 4fafe771a..59857ba11 100644--- a/pipelex/pipeline/direct_bundle_validator.py+++ b/pipelex/pipeline/direct_bundle_validator.py@@ -38,13 +38,15 @@ class DirectBundleValidator:async def validate_bundles(self,*,- mthds_contents: list[str],+ mthds_contents: list[str] | None,mthds_sources: list[str] | None,allow_signatures: bool,library_dirs: Sequence[Path] | None,caller_identity: CallerIdentity | None,graph_pipe_code: str | None,submitted_files: Mapping[str, bytes] | None,+ method_dir: Path | None,+ methods_dirs: Sequence[Path] | None,) -> PipelexValidationReport | ErrorReport:try:return await validate_bundles_in_process(@@ -56,6 +58,8 @@ class DirectBundleValidator:log_context="API validate",caller_identity=caller_identity,submitted_files=submitted_files,+ method_dir=method_dir,+ methods_dirs=methods_dirs,)except ValidateBundleError as exc:# An invalid bundle is a produced verdict, not a fault: surface its structureddiff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.pyindex 3b640982c..362a4b358 100644--- a/pipelex/pipeline/execution_seams.py+++ b/pipelex/pipeline/execution_seams.py@@ -114,9 +114,10 @@ def acquire_library(the caller's to fix and its paths are not the caller's to read, so by default they load untranslated.``methods_dirs`` are directories laid out like ``.mthds/methods/`` holding the packages the caller ships with the- ``mthds_contents`` (on a host, the ones a request carries under ``.mthds/methods/<name>/``). An address-based- reference of the contents is looked up there first, before the installed stores and fetch-on-miss, and a package- found there is never installed; the bundles of a package are named by its address, never by these directories.+ ``mthds_contents`` or with the method of its own ``library_dirs`` (on a host, the dependency trees a request carries+ beside its method's tree). An address-based reference of those bundles is looked up there first, before the+ installed stores and fetch-on-miss, and a package found there is never installed; the bundles of a package are+ named by its address, never by these directories.``bundle_uris``, parallel to ``mthds_contents``, name the files on this host the caller read them from, such as thebundle file a CLI runs. A bundle already loaded from the library directories under that path is skipped, and one@@ -157,7 +158,7 @@ def acquire_library(translate_to_validate_bundle_error() if library_dirs_are_callers else reporting_host_library_tree_faults())with library_dirs_translation:- library_manager.load_libraries(library_id=library_id, library_dirs=effective_dirs)+ library_manager.load_libraries(library_id=library_id, library_dirs=effective_dirs, methods_dirs=methods_dirs)else:log.verbose(f"No library directories to load ({source_label})")diff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.pyindex aab053990..7419a4da1 100644--- a/pipelex/pipeline/runner.py+++ b/pipelex/pipeline/runner.py@@ -134,6 +134,9 @@ class PipelexVersionInfo(VersionInfo):implementation: strimplementation_version: strruntime_version: str | None = None+ # Whether this runner keeps a blob store and serves the MTHDS protocol's blob routes, `POST /blobs/missing` and+ # `PUT /blobs/{sha256}`, so a sender may name a file of `files` by its hash. A runtime in process keeps none.+ blob_store: bool = False_VALIDATE_EXTRA_GRAPH_PIPE_CODE = "graph_pipe_code"@@ -198,8 +201,9 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):# `acquire_library`.library_dirs_are_callers: bool = False,# Directories laid out like `.mthds/methods/` holding the method packages a caller ships with the contents it- # runs, searched first for an address-based reference and never written. A run reads them; `validate` does- # not, and resolves such a reference through the installed stores and fetch-on-miss only.+ # runs, searched first for an address-based reference and never written. A run reads them, and so does the+ # validation of a method directory; the validation of contents does not, and resolves such a reference through+ # the installed stores and fetch-on-miss only.methods_dirs: list[Path] | None = None,# The files a caller submits with the contents it runs or validates that their steps read, such as a# PipeDocGen step's `template_file`, keyed by canonical bundle-relative path (`canonical_submitted_files`).diff --git a/pipelex/pipeline/validate_bundle.py b/pipelex/pipeline/validate_bundle.pyindex fa9866289..a5641689a 100644--- a/pipelex/pipeline/validate_bundle.py+++ b/pipelex/pipeline/validate_bundle.py@@ -1,9 +1,11 @@+import astimport asynciofrom collections.abc import Mappingfrom contextlib import AbstractContextManager, nullcontextfrom pathlib import Pathfrom typing import Sequence+from mthds.method_tree.models import MethodTreefrom pydantic import BaseModel# TypedDict from typing_extensions, not typing: pydantic rejects typing.TypedDict as a model@@ -12,6 +14,7 @@ from typing_extensions import TypedDictfrom pipelex import logfrom pipelex.base_exceptions import PipelexUnexpectedError+from pipelex.core.exceptions import PipelexBundleBlueprintValidationErrorDatafrom pipelex.core.method_trees.file_references import BundleFileOriginfrom pipelex.core.qualified_ref import QualifiedReffrom pipelex.interpreter_hub import (@@ -23,7 +26,7 @@ from pipelex.interpreter_hub import ()from pipelex.libraries.bundle_resources import check_submitted_filesfrom pipelex.libraries.library_utils import get_pipelex_mthds_files_from_dirs-from pipelex.libraries.method_trees import reporting_host_library_tree_faults+from pipelex.libraries.method_trees import list_local_method_tree, reporting_host_library_tree_faultsfrom pipelex.libraries.pipe.exceptions import EntryPipeNotFoundErrorfrom pipelex.mthds_parsing.parser import MthdsParserfrom pipelex.mthds_parsing.pipelex_bundle_blueprint import PipelexBundleBlueprint@@ -33,6 +36,8 @@ from pipelex.pipeline.exceptions import ValidateBundleErrorfrom pipelex.pipeline.validate_bundle_translation import translate_to_validate_bundle_error, withholding_host_library_filesfrom pipelex.system.caller_identity import CallerIdentity+PYTHON_SUFFIX = ".py"+class ValidateBundleResult(BaseModel):blueprints: list[PipelexBundleBlueprint]@@ -129,8 +134,16 @@ async def validate_bundle(library_dirs_are_callers: bool = False,collect_dry_run_warnings: bool = False,submitted_files: Mapping[str, bytes] | None = None,+ method_dir: Path | None = None,+ methods_dirs: Sequence[Path] | None = None,) -> ValidateBundleResult:- """Load one bundle into a fresh library and dry-run its pipes.+ """Load one bundle, or one method's tree, into a fresh library and dry-run its pipes.++ Exactly one of ``mthds_file_path``, ``mthds_contents`` and ``method_dir`` names what is validated.+ ``method_dir`` is the directory of a method's tree, the caller's own, such as the tree a host wrote out for a+ request: every bundle of the tree is loaded as a member of it and dry-run, every refusal is the caller's verdict and+ names the tree's files as the tree does, and ``methods_dirs``, laid out like ``.mthds/methods/``, hold the packages+ the caller ships with it, searched first for an address-based reference. No other library directory loads beside it.``submitted_files`` are the files submitted with ``mthds_contents`` that their steps read, such as a``PipeDocGen`` step's ``template_file``, keyed by canonical bundle-relative path (see@@ -156,6 +169,7 @@ async def validate_bundle([mthds_contents is not None,mthds_file_path is not None,+ method_dir is not None,])if provided_params == 0:@@ -163,10 +177,16 @@ async def validate_bundle(# input. It is not a content verdict the end user can fix → PipelexUnexpectedError (→ 500,# redacted under STRICT), matching the mthds_sources-mismatch guard below — never a# ValidateBundleError, which would 200/422 a host-wiring bug as if the bundle were invalid.- msg = "At least one of mthds_contents or mthds_file_path must be provided to validate_bundle"+ msg = "One of mthds_contents, mthds_file_path or method_dir must be provided to validate_bundle"raise PipelexUnexpectedError(msg)if provided_params > 1:- msg = "Only one of mthds_contents or mthds_file_path can be provided to validate_bundle, not both"+ msg = "Only one of mthds_contents, mthds_file_path or method_dir can be provided to validate_bundle"+ raise PipelexUnexpectedError(msg)+ if method_dir is not None and library_dirs is not None:+ msg = "A method's tree validates alone: no library directory loads beside method_dir"+ raise PipelexUnexpectedError(msg)+ if methods_dirs is not None and method_dir is None:+ msg = "methods_dirs accompany method_dir, the method whose packages they hold"raise PipelexUnexpectedError(msg)if mthds_contents is not None and not mthds_contents:# An EMPTY list is not None, so it passes the provided-params check above — without this@@ -196,6 +216,20 @@ async def validate_bundle(try:set_current_library(library_id=library_id)+ if method_dir is not None:+ await asyncio.sleep(0) # Yield to event loop (keeps function async-compatible)+ result = await _validate_method_dir(+ method_dir=method_dir,+ methods_dirs=methods_dirs,+ library_id=library_id,+ allow_signatures=allow_signatures,+ dry_run_pipe_codes=dry_run_pipe_codes,+ caller_identity=caller_identity,+ collect_dry_run_warnings=collect_dry_run_warnings,+ )+ success = True+ return result+# Load libraries from resolved directories before loading the bundleeffective_dirs, source_label = resolve_library_dirs(library_dirs)@@ -306,6 +340,87 @@ async def validate_bundle(library_manager.teardown(library_id=library_id)+def _ensure_tree_python_parses(*, tree: MethodTree) -> None:+ """Refuse a method tree holding a Python file that does not parse, naming each such file by its tree path.++ Validation covers the whole tree, its Python included, by syntax alone: no module is imported, so a file no step+ calls is checked too, and a `PipeFunc`'s own module is then held to the static check of its function.++ Raises:+ ValidateBundleError: One item per Python file that is not UTF-8 text or does not parse, located on its line.+ """+ items: list[PipelexBundleBlueprintValidationErrorData] = []+ for tree_path in tree.tree_paths:+ if not tree_path.endswith(PYTHON_SUFFIX):+ continue+ try:+ source = tree.file_location(tree_path).read_bytes().decode("utf-8")+ except UnicodeDecodeError:+ items.append(PipelexBundleBlueprintValidationErrorData(source=tree_path, message=f"The Python file '{tree_path}' is not UTF-8 text."))+ continue+ try:+ ast.parse(source, filename=tree_path)+ except SyntaxError as exc:+ items.append(+ PipelexBundleBlueprintValidationErrorData(+ source=tree_path,+ message=f"The Python file '{tree_path}' does not parse: {exc.msg} (line {exc.lineno}).",+ line=exc.lineno,+ column=exc.offset,+ )+ )+ except ValueError as exc:+ # Python 3.11 refuses a source holding a NUL byte with a ValueError where later versions raise a SyntaxError.+ items.append(PipelexBundleBlueprintValidationErrorData(source=tree_path, message=f"The Python file '{tree_path}' does not parse: {exc}."))+ except (RecursionError, MemoryError):+ # Nesting deep enough to exhaust the parser, which Python's own import refuses the same way.+ message = f"The Python file '{tree_path}' does not parse: it nests too deeply."+ items.append(PipelexBundleBlueprintValidationErrorData(source=tree_path, message=message))+ if items:+ raise ValidateBundleError(message=" ".join(item.message for item in items), pipelex_bundle_blueprint_validation_errors=items)+++async def _validate_method_dir(+ *,+ method_dir: Path,+ methods_dirs: Sequence[Path] | None,+ library_id: str,+ allow_signatures: bool,+ dry_run_pipe_codes: list[str] | None,+ caller_identity: CallerIdentity | None,+ collect_dry_run_warnings: bool,+) -> ValidateBundleResult:+ """Load the tree of a caller's method into the open library and dry-run its pipes (see ``validate_bundle``)."""+ library_manager = get_library_manager()+ library = library_manager.get_library(library_id=library_id)+ with translate_to_validate_bundle_error():+ _ensure_tree_python_parses(tree=list_local_method_tree(handed_over=method_dir))+ loaded_pipes = library_manager.load_libraries(+ library_id=library_id,+ library_dirs=[method_dir],+ methods_dirs=list(methods_dirs) if methods_dirs is not None else None,+ )+ blueprints = library_manager.get_accumulated_blueprints(library_id)+ if not blueprints:+ msg = "The method holds no .mthds bundle, so it has nothing to validate."+ raise ValidateBundleError(message=msg)+ dry_run_results = await BundleValidator().validate_pipes(+ pipes=_pipes_to_dry_run(loaded_pipes, dry_run_pipe_codes=dry_run_pipe_codes),+ library_id=library_id,+ allow_signatures=allow_signatures,+ caller_identity=caller_identity,+ # The caller's own method: every failure names its file, by its path in the tree.+ source_pipe_refs=None,+ collect_dry_run_warnings=collect_dry_run_warnings,+ )+ return ValidateBundleResult(+ blueprints=blueprints,+ pipes=loaded_pipes,+ dry_run_result=dry_run_results,+ pending_signatures=build_pending_signatures(library.pipe_library.get_pipes_dict()),+ )++async def validate_bundles_from_directory(directory: Path, *, allow_signatures: bool = False) -> ValidateBundleResult:# A directory whose tree cannot be listed is the invalid verdict, as a bundle of it that does not load is.with translate_to_validate_bundle_error():diff --git a/pipelex/pipeline/validate_in_process.py b/pipelex/pipeline/validate_in_process.pyindex 032ae97d7..ed1373571 100644--- a/pipelex/pipeline/validate_in_process.py+++ b/pipelex/pipeline/validate_in_process.py@@ -26,14 +26,18 @@ from typing import TYPE_CHECKINGfrom pipelex import logfrom pipelex.base_exceptions import PipelexError, ValidationErrorItemfrom pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager, set_current_library+from pipelex.libraries.method_trees import list_local_method_tree+from pipelex.mthds_parsing.pipelex_bundle_blueprint import PipelexBundleBlueprintfrom pipelex.pipe_run.dry_run_in_process import best_effort_graph_specfrom pipelex.pipe_run.dry_run_warning_collector import collecting_dry_run_warningsfrom pipelex.pipeline.advisory_warnings import build_advisory_warningsfrom pipelex.pipeline.blueprint_selection import collect_entry_pipe_refs, select_primary_blueprintfrom pipelex.pipeline.build_pipe_io_artifacts import build_pipe_io_artifactsfrom pipelex.pipeline.controller_taint import collect_controller_taint_analyses+from pipelex.pipeline.exceptions import MethodEntryErrorfrom pipelex.pipeline.input_form import qualify_current_library_cratefrom pipelex.pipeline.liftable_pipes import LiftablePipeEntry, build_liftable_pipes+from pipelex.pipeline.method_entry import select_method_entryfrom pipelex.pipeline.usage_stats_reporting import MethodValidationReportfrom pipelex.pipeline.validate_bundle import validate_bundlefrom pipelex.pipeline.validation_report import PipelexValidationReport, build_validation_report@@ -44,9 +48,25 @@ if TYPE_CHECKING:from pipelex.graph.graphspec import GraphSpec+def _graph_target_ref(*, graph_pipe_code: str | None, method_dir: Path | None, blueprints: list[PipelexBundleBlueprint]) -> str | None:+ """The pipe the graph arm draws: the explicit target, else a method tree's entry pipe, else the batch's `main_pipe`.++ A method tree starts at its entry pipe by the MTHDS protocol's rule (`select_method_entry`), so its graph is that+ pipe's, and a tree with no entry pipe, which a run naming no pipe refuses, gets no graph.+ """+ if graph_pipe_code is not None:+ return graph_pipe_code+ if method_dir is None:+ return select_primary_blueprint(blueprints).main_pipe_ref+ try:+ return select_method_entry(tree=list_local_method_tree(handed_over=method_dir)).pipe_ref+ except MethodEntryError:+ return None++async def validate_bundles_in_process(*,- mthds_contents: list[str],+ mthds_contents: list[str] | None = None,mthds_sources: list[str] | None = None,library_dirs: Sequence[Path] | None = None,allow_signatures: bool = False,@@ -55,6 +75,8 @@ async def validate_bundles_in_process(caller_identity: CallerIdentity | None = None,library_dirs_are_callers: bool = False,submitted_files: Mapping[str, bytes] | None = None,+ method_dir: Path | None = None,+ methods_dirs: Sequence[Path] | None = None,) -> PipelexValidationReport:"""Parse, validate, and dry-run MTHDS bundles in-process; assemble the canonical report.@@ -67,7 +89,8 @@ async def validate_bundles_in_process(graph-arm domain failure degrades to `graph_spec=None` with validation still successful.Args:- mthds_contents: MTHDS contents to load (always a list, even for one file).+ mthds_contents: MTHDS contents to load (always a list, even for one file), unless `method_dir` names the+ method instead.mthds_sources: Optional per-content sources threaded onto `blueprint.source`(length must match `mthds_contents` — `validate_bundle` enforces it). `None`for the local protocol runtime, which receives only sourceless content strings@@ -75,9 +98,10 @@ async def validate_bundles_in_process(library_dirs: Resolved library directories to load before the bundle.allow_signatures: Tolerate unimplemented pipe signatures (strict by default).graph_pipe_code: Optional explicit pipe target for the best-effort graph arm.- When omitted, the graph arm keeps the existing default of targeting the- selected bundle `main_pipe`. Bare and qualified refs are accepted according- to the loaded pipe library's normal resolution rules.+ When omitted, the graph arm targets the entry pipe of `method_dir` by the MTHDS+ protocol's rule, drawing no graph when it has none, or for contents the selected+ bundle `main_pipe`. Bare and qualified refs are accepted according to the loaded+ pipe library's normal resolution rules.log_context: Label the graph-arm degradation and teardown-failure lines carry in their ``caller`` field.caller_identity: Who asked for the validation, when the host knows it. It is theambient caller for the whole pass — the sweep's telemetry event and every dry run,@@ -89,6 +113,9 @@ async def validate_bundles_in_process(submitted_files: The files submitted with ``mthds_contents`` that their steps read, such as a``PipeDocGen`` template file, keyed by canonical bundle-relative path. The dry run reads them, so thetemplate check runs on them.+ method_dir: The directory of the caller's method tree, validated whole in place of `mthds_contents`, such as+ the tree a host wrote out for a request (see `validate_bundle`).+ methods_dirs: Directories laid out like `.mthds/methods/` holding the packages shipped with `method_dir`.Returns:PipelexValidationReport with the structural artifacts of a valid bundle.@@ -106,13 +133,15 @@ async def validate_bundles_in_process(log_context=log_context,library_dirs_are_callers=library_dirs_are_callers,submitted_files=submitted_files,+ method_dir=method_dir,+ methods_dirs=methods_dirs,validation_report=validation_report,)async def _validate_bundles_in_scope(*,- mthds_contents: list[str],+ mthds_contents: list[str] | None,mthds_sources: list[str] | None,library_dirs: Sequence[Path] | None,allow_signatures: bool,@@ -120,6 +149,8 @@ async def _validate_bundles_in_scope(log_context: str,library_dirs_are_callers: bool,submitted_files: Mapping[str, bytes] | None,+ method_dir: Path | None,+ methods_dirs: Sequence[Path] | None,validation_report: MethodValidationReport,) -> PipelexValidationReport:"""The body of :func:`validate_bundles_in_process`, run inside the caller's scope and timed by `validation_report`."""@@ -148,6 +179,8 @@ async def _validate_bundles_in_scope(library_dirs_are_callers=library_dirs_are_callers,collect_dry_run_warnings=True,submitted_files=submitted_files,+ method_dir=method_dir,+ methods_dirs=methods_dirs,)# Capture the validation library id ONCE, right after validate_bundle leaves it# current — the graph arm and the finally must target the SAME library even if@@ -170,7 +203,7 @@ async def _validate_bundles_in_scope(dry_run_result=result.dry_run_result,)validation_report.count_warnings(count=len(warnings))- graph_target_ref = graph_pipe_code if graph_pipe_code is not None else select_primary_blueprint(result.blueprints).main_pipe_ref+ graph_target_ref = _graph_target_ref(graph_pipe_code=graph_pipe_code, method_dir=method_dir, blueprints=result.blueprints)# The graph arm dry-runs its target once more, and finds again what the sweep found, which `warnings`# already carries: it is collected and dropped here rather than logged as if nothing reported it.with collecting_dry_run_warnings():diff --git a/pipelex/plugins/bundle_validator_registry.py b/pipelex/plugins/bundle_validator_registry.pyindex 00d8b2fbf..85c1fa8e5 100644--- a/pipelex/plugins/bundle_validator_registry.py+++ b/pipelex/plugins/bundle_validator_registry.py@@ -81,18 +81,29 @@ class BundleValidatorProtocol(Protocol):``None`` meaning none were submitted: a validator that dropped them would refuse a valid bundlefor a missing file. A worker-dispatched arm hands them to its worker through the method filestore, the crate naming their tree by digest.++ ``method_dir`` names the caller's method as a tree in place of ``mthds_contents``, which is then+ ``None``: the directory of a tree the host wrote out on its own disk, such as a request's map of+ file entries or a fetched package, held as written (``pipelex.libraries.written_trees``). Every+ bundle of the tree is validated, its Python included, and every refusal is the caller's verdict,+ naming the tree's files by their tree paths. ``methods_dirs``, laid out like ``.mthds/methods/``,+ hold the dependency trees the caller ships with it. Both are required for the same reason as+ ``caller_identity``, ``None`` meaning the method came as contents. A worker-dispatched arm reads+ the tree's files on the host, where it is called, and hands them to its worker as trees."""async def validate_bundles(self,*,- mthds_contents: list[str],+ mthds_contents: list[str] | None,mthds_sources: list[str] | None,allow_signatures: bool,library_dirs: "Sequence[Path] | None",caller_identity: CallerIdentity | None,graph_pipe_code: str | None,submitted_files: "Mapping[str, bytes] | None",+ method_dir: "Path | None",+ methods_dirs: "Sequence[Path] | None",) -> BundleValidationVerdict: ...diff --git a/pipelex/plugins/contract.py b/pipelex/plugins/contract.pyindex b56a3b972..c3566422e 100644--- a/pipelex/plugins/contract.py+++ b/pipelex/plugins/contract.py@@ -57,7 +57,9 @@ if TYPE_CHECKING:## v7 changed what a run hands an orchestration plugin, now that a run's library crate names each method tree by its# digest and carries no file's bytes. A bundle validator's ``validate_bundles`` receives ``submitted_files``, a mapping-# from the path a step names to the file's bytes, where it received ``bundle_resources``; a PipeFunc executor's+# from the path a step names to the file's bytes, where it received ``bundle_resources``, and ``method_dir`` and+# ``methods_dirs``, a method's tree and its dependency trees written out on the host, in place of ``mthds_contents``,+# which may now be ``None``; a PipeFunc executor's# transported request names its tree by ``tree_digest`` where it carried the call's ``files``, and the transported# entrypoint takes the directory the call's files were unpacked into as a required argument; and an orchestrator that# runs a job out of process copies the trees the job's crate names before its ``execute`` or ``start`` returnsdiff --git a/pipelex/plugins/orchestrator_registry.py b/pipelex/plugins/orchestrator_registry.pyindex 54e1e2231..e65f9631f 100644--- a/pipelex/plugins/orchestrator_registry.py+++ b/pipelex/plugins/orchestrator_registry.py@@ -42,7 +42,9 @@ class OrchestratorProtocol(Protocol):``read_stored_trees(tree_digests=pipe_job.library_crate.tree_digests(), read_scope=...)``(``pipelex.core.method_trees.method_files``) reads each one as a store keeps it, its listingunder the tree's digest and its files under their hashes, checked. An orchestrator running- the job in this process copies nothing.+ the job in this process, to its end before its ``execute`` returns, copies nothing; one that+ keeps a job running after its ``start`` returned copies the trees as an out-of-process one+ does, wherever the job runs, since the library holding them is gone by then."""supports_fire_and_forget: booldiff --git a/pipelex/tools/storage/gcp_storage_provider.py b/pipelex/tools/storage/gcp_storage_provider.pyindex 161e3767d..02494ac97 100644--- a/pipelex/tools/storage/gcp_storage_provider.py+++ b/pipelex/tools/storage/gcp_storage_provider.py@@ -186,6 +186,41 @@ class GcpStorageProvider(StorageProviderAbstract):"""return await asyncio.to_thread(self._load_head_sync, key, nb_bytes=nb_bytes)+ def _load_size_sync(self, key: str) -> int:+ """Synchronous implementation of the size read for use with to_thread.++ Raises:+ StorageFileNotFoundError: If the object does not exist.+ StorageGcpError: If the GCS operation fails.+ """+ bucket = self._get_bucket()++ from google.api_core.exceptions import ( # type: ignore[import-untyped] # ruff: ignore[import-outside-top-level] - optional dependency, lazy import+ GoogleAPIError,+ )++ try:+ # Fetches the object's metadata alone, None when there is no such object.+ blob = bucket.get_blob(key)+ except GoogleAPIError as exc: # pyright: ignore[reportUnknownVariableType]+ msg = f"Failed to read the size of the GCS object '{key}': {exc}"+ raise StorageGcpError(msg) from exc+ if blob is None:+ msg = f"Object not found in GCS: '{key}'"+ raise StorageFileNotFoundError(msg)+ size: int = blob.size+ return size++ @override+ async def _load_size(self, key: str) -> int:+ """The size of a GCS object, read from its metadata instead of its bytes.++ Raises:+ StorageFileNotFoundError: If the object does not exist.+ StorageGcpError: If the GCS operation fails.+ """+ return await asyncio.to_thread(self._load_size_sync, key)+def _store_sync(self, data: bytes, *, key: str, content_type: str | None) -> None:"""Synchronous implementation of store for use with to_thread.diff --git a/pipelex/tools/storage/in_memory_storage_provider.py b/pipelex/tools/storage/in_memory_storage_provider.pyindex 32d371fc1..7486e9e94 100644--- a/pipelex/tools/storage/in_memory_storage_provider.py+++ b/pipelex/tools/storage/in_memory_storage_provider.py@@ -43,6 +43,18 @@ class InMemoryStorageProvider(RootModel[InMemoryStorageRoot], StorageProviderAbsreturn StoredData(data=data, mime_type=mime_type)+ @override+ async def _load_size(self, key: str) -> int:+ """The length of the bytes held under `key`.++ Raises:+ StorageFileNotFoundError: If no data exists for the key.+ """+ if key not in self.root:+ msg = f"File not found: '{key}'"+ raise StorageFileNotFoundError(msg)+ return len(self.root[key])+@overrideasync def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:"""Store bytes in memory.diff --git a/pipelex/tools/storage/local_storage_provider.py b/pipelex/tools/storage/local_storage_provider.pyindex aa0c5bf80..1af6e967d 100644--- a/pipelex/tools/storage/local_storage_provider.py+++ b/pipelex/tools/storage/local_storage_provider.py@@ -1,4 +1,7 @@+import contextlib+import osfrom pathlib import Path+from uuid import uuid4import aiofilesfrom typing_extensions import override@@ -99,9 +102,37 @@ class LocalStorageProvider(StorageProviderAbstract):return StoredData(data=data, mime_type=mime_type)+ @override+ async def _load_size(self, key: str) -> int:+ """The size of a file, read from its metadata without opening it.++ Raises:+ StorageFileNotFoundError: If the file does not exist.+ StorageInvalidUriError: If the key is invalid.+ StorageLocalError: If the key names something that is not a file, or the filesystem cannot stat it.+ """+ file_path = self._validate_key(key)+ try:+ stat_result = file_path.stat()+ except FileNotFoundError as exc:+ msg = f"File not found: '{key}'"+ raise StorageFileNotFoundError(msg) from exc+ except OSError as exc:+ msg = f"Failed to read the size of file '{key}': {exc}"+ raise StorageLocalError(msg) from exc+ if not file_path.is_file():+ msg = f"Failed to read the size of '{key}': it is not a file"+ raise StorageLocalError(msg)+ return stat_result.st_size+@overrideasync def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:- """Store bytes to a file.+ """Store bytes to a file, published whole or not at all.++ The bytes are written to a temporary file beside the target, flushed to disk, then renamed onto the target,+ which is atomic within one filesystem: a reader, or a process checking whether the key exists, never sees a+ file a full disk or a killed process cut short. Two writers of one key each write their own temporary file,+ and the last rename wins whole.Args:data: The bytes to store.@@ -113,16 +144,24 @@ class LocalStorageProvider(StorageProviderAbstract):StorageLocalError: If the filesystem write fails (e.g. disk full, permission denied)."""file_path = self._validate_key(key)+ temporary_path = file_path.with_name(f".{file_path.name}.{uuid4().hex}.tmp")try:# Create parent directories if they don't existfile_path.parent.mkdir(parents=True, exist_ok=True)- async with aiofiles.open(file_path, "wb") as file_handle: # pyright: ignore[reportUnknownMemberType]+ async with aiofiles.open(temporary_path, "wb") as file_handle: # pyright: ignore[reportUnknownMemberType]await file_handle.write(data)+ await file_handle.flush()+ os.fsync(file_handle.fileno())+ temporary_path.replace(file_path)except OSError as exc:msg = f"Failed to write file '{key}': {exc}"raise StorageLocalError(msg) from exc+ finally:+ # Gone once renamed; a write that failed leaves it behind, and removing it is best effort.+ with contextlib.suppress(OSError):+ temporary_path.unlink(missing_ok=True)@overrideasync def public_url(self, uri: str) -> str:diff --git a/pipelex/tools/storage/s3_storage_provider.py b/pipelex/tools/storage/s3_storage_provider.pyindex 394cbe454..87ab659cc 100644--- a/pipelex/tools/storage/s3_storage_provider.py+++ b/pipelex/tools/storage/s3_storage_provider.py@@ -186,6 +186,45 @@ class S3StorageProvider(StorageProviderAbstract):msg = f"S3 backend error for key '{key}': {type(exc).__name__}"raise StorageS3Error(msg) from exc+ @override+ async def _load_size(self, key: str) -> int:+ """The size of an S3 object, read with a HEAD request instead of its bytes.++ Args:+ key: Storage key (without scheme prefix).++ Returns:+ The object's `ContentLength`.++ Raises:+ StorageFileNotFoundError: If the object does not exist.+ StorageS3Error: If the S3 operation fails (any other ClientError or BotoCoreError).+ """+ from botocore.exceptions import ( # ruff: ignore[import-outside-top-level] - optional dependency, lazy import+ BotoCoreError,+ ClientError,+ )++ session = self._get_session()+ client_config = self._get_client_config()++ async with session.create_client(**client_config) as client:+ try:+ response = await client.head_object(Bucket=self._bucket_name, Key=key)+ except ClientError as exc:+ error_code = (exc.response.get("Error") or {}).get("Code", "Unknown")+ # A HEAD answer has no body, so a missing object comes back as the bare status code.+ if error_code in {"404", "NoSuchKey", "NotFound"}:+ msg = f"Object not found in S3: '{key}'"+ raise StorageFileNotFoundError(msg) from exc+ msg = f"S3 ClientError ({error_code}) for key '{key}'"+ raise StorageS3Error(msg) from exc+ except BotoCoreError as exc:+ msg = f"S3 backend error for key '{key}': {type(exc).__name__}"+ raise StorageS3Error(msg) from exc+ content_length: int = response["ContentLength"]+ return content_length+@overrideasync def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:"""Store bytes to an S3 object.diff --git a/pipelex/tools/storage/storage_provider_abstract.py b/pipelex/tools/storage/storage_provider_abstract.pyindex 469da08de..6eea7c643 100644--- a/pipelex/tools/storage/storage_provider_abstract.py+++ b/pipelex/tools/storage/storage_provider_abstract.py@@ -124,6 +124,34 @@ class StorageProviderAbstract(ABC):stored_data = await self._load_with_metadata(key)return stored_data.data[:nb_bytes]+ async def load_size(self, uri: str) -> int:+ """The size of a stored object in bytes, read without its bytes where the provider can.++ Args:+ uri: Full URI including PIPELEX_STORAGE_SCHEME prefix.++ Returns:+ The object's length in bytes.+ """+ key = self._strip_scheme(uri)+ return await self._load_size(key)++ async def _load_size(self, key: str) -> int:+ """The size of an object by key.++ This default loads the whole object and measures it, so a provider that cannot read an object's+ metadata alone, an external storage plugin included, keeps working unchanged. A provider that can+ overrides it, and a caller bounding what it reads relies on that override.++ Args:+ key: Storage key (without scheme prefix).++ Returns:+ The object's length in bytes.+ """+ stored_data = await self._load_with_metadata(key)+ return len(stored_data.data)+async def store(self, data: bytes, *, key: str, content_type: str | None = None) -> str:"""Store data and return full URI with scheme.diff --git a/subject_grants.toml b/subject_grants.tomlindex 1e4a3cf79..4c664aa7c 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -3948,6 +3948,10 @@ rationale = "Predicate on the file path; single operand."param = "bundle_path"rationale = "Factory from source: parses the bundle at the path; options stay keyword."+["pipelex/mthds_parsing/parser.py::_decode_error_position"]+param = "exc"+rationale = "Reads the position out of the one exception it is handed; single operand."+["pipelex/mthds_parsing/pipelex_bundle_blueprint.py::PipelexBundleBlueprint._collect_local_refs_from_concept"]param = "concept_code"rationale = "Verb-object: collects refs from the concept named; context stays keyword."@@ -6524,6 +6528,10 @@ rationale = "Keyed accessor by the key; single operand."param = "key"rationale = "Keyed accessor by the storage key, called positionally through asyncio.to_thread."+["pipelex/tools/storage/gcp_storage_provider.py::GcpStorageProvider._load_size_sync"]+param = "key"+rationale = "Keyed accessor by the storage key, called positionally through asyncio.to_thread."+["pipelex/tools/storage/gcp_storage_provider.py::GcpStorageProvider._load_with_metadata_sync"]param = "key"rationale = "Parses/reads the key; single operand."@@ -6552,6 +6560,10 @@ rationale = "Keyed accessor by the key; single operand."param = "key"rationale = "Keyed accessor by the storage key; single operand."+["pipelex/tools/storage/storage_provider_abstract.py::StorageProviderAbstract._load_size"]+param = "key"+rationale = "Keyed accessor by the storage key; single operand, as _load_head."+["pipelex/tools/storage/storage_provider_abstract.py::StorageProviderAbstract._load_with_metadata"]param = "key"rationale = "Parses/reads the key; single operand."@@ -6572,6 +6584,10 @@ rationale = "Keyed accessor by the uri; single operand."param = "uri"rationale = "Keyed accessor by the uri; single operand."+["pipelex/tools/storage/storage_provider_abstract.py::StorageProviderAbstract.load_size"]+param = "uri"+rationale = "Keyed accessor by the uri; single operand, as load_head."+["pipelex/tools/storage/storage_provider_abstract.py::StorageProviderAbstract.load_with_metadata"]param = "uri"rationale = "Parses/reads the uri; single operand."diff --git a/tests/integration/pipelex/pipeline/test_validate_method_dir.py b/tests/integration/pipelex/pipeline/test_validate_method_dir.pynew file mode 100644index 000000000..58ccac969--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_validate_method_dir.py@@ -0,0 +1,86 @@+from collections.abc import Generator+from pathlib import Path++import pytest+from mthds.method_tree.digest import digest_of_contents+from mthds.method_tree.models import MethodTree++from pipelex.libraries.written_trees import WRITTEN_METHOD_TREES, write_method_tree+from pipelex.pipeline.exceptions import ValidateBundleError+from pipelex.pipeline.validate_bundle import validate_bundle+from pipelex.pipeline.validate_in_process import validate_bundles_in_process++_GREET_MTHDS = b"""\+domain = "greeting"+main_pipe = "greet"++[pipe.greet]+type = "PipeCompose"+description = "Greet"+inputs = { text = "Text" }+output = "Text"+template = "Hello $text"+"""++_SECOND_GREET_MTHDS = _GREET_MTHDS.replace(b'domain = "greeting"', b'domain = "greeting_too"')++_BROKEN_PYTHON = b"def ok():\n return 1\n\n\ndef oops(:\n pass\n"++[email protected](name="held_tree")+def held_tree_fixture(tmp_path: Path) -> Generator[Path, None, None]:+ """The default tree, written and held as a host holds a request's tree, its directory yielded."""+ contents = {"greet.mthds": _GREET_MTHDS, "notes/readme.txt": b"notes"}+ written = write_method_tree(contents=contents, destination=tmp_path / "tree", expected_digest=digest_of_contents(contents))+ with WRITTEN_METHOD_TREES.holding(tree=written):+ yield tmp_path / "tree"+++def _written(*, tmp_path: Path, contents: dict[str, bytes]) -> tuple[MethodTree, Path]:+ """A tree written out as a host writes a request's tree, and the directory it was written as."""+ destination = tmp_path / "tree"+ return write_method_tree(contents=contents, destination=destination, expected_digest=digest_of_contents(contents)), destination++[email protected](loop_scope="class")+class TestValidateMethodDir:+ async def test_a_held_tree_validates_whole_its_bundles_named_by_tree_path(self, held_tree: Path) -> None:+ result = await validate_bundle(method_dir=held_tree)++ assert [blueprint.source for blueprint in result.blueprints] == ["greet.mthds"]+ assert [pipe.code for pipe in result.pipes] == ["greet"]++ async def test_every_python_file_of_the_tree_is_read_for_its_syntax(self, tmp_path: Path) -> None:+ """No step names the module, yet the tree's validation refuses it, by tree path and line, before anything loads."""+ tree, method_dir = _written(tmp_path=tmp_path, contents={"greet.mthds": _GREET_MTHDS, "helpers/broken.py": _BROKEN_PYTHON})+ with WRITTEN_METHOD_TREES.holding(tree=tree), pytest.raises(ValidateBundleError) as exc_info:+ await validate_bundle(method_dir=method_dir)++ (item,) = exc_info.value.pipelex_bundle_blueprint_validation_errors or []+ assert item.source == "helpers/broken.py"+ assert item.line == 5+ assert str(tmp_path) not in exc_info.value.message++ async def test_a_python_file_that_is_not_utf8_is_refused_by_tree_path(self, tmp_path: Path) -> None:+ tree, method_dir = _written(tmp_path=tmp_path, contents={"greet.mthds": _GREET_MTHDS, "helpers/latin.py": b"NAME = '\xe9'\n"})+ with WRITTEN_METHOD_TREES.holding(tree=tree), pytest.raises(ValidateBundleError) as exc_info:+ await validate_bundle(method_dir=method_dir)++ (item,) = exc_info.value.pipelex_bundle_blueprint_validation_errors or []+ assert item.source == "helpers/latin.py"+ assert "not UTF-8" in item.message++ async def test_the_graph_is_the_trees_entry_pipe(self, held_tree: Path) -> None:+ report = await validate_bundles_in_process(method_dir=held_tree, library_dirs_are_callers=True)++ assert report.graph_spec is not None+ assert report.graph_spec.pipeline_ref.domain == "greeting"+ assert report.graph_spec.pipeline_ref.main_pipe == "greet"++ async def test_a_tree_with_no_entry_pipe_validates_without_a_graph(self, tmp_path: Path) -> None:+ """Two bundles declaring a main_pipe and no manifest leave a run naming no pipe nothing to start at, so nothing is drawn."""+ tree, method_dir = _written(tmp_path=tmp_path, contents={"a.mthds": _GREET_MTHDS, "b.mthds": _SECOND_GREET_MTHDS})+ with WRITTEN_METHOD_TREES.holding(tree=tree):+ report = await validate_bundles_in_process(method_dir=method_dir, library_dirs_are_callers=True)++ assert report.graph_spec is Nonediff --git a/tests/unit/pipelex/libraries/test_written_method_trees.py b/tests/unit/pipelex/libraries/test_written_method_trees.pynew file mode 100644index 000000000..c260f946c--- /dev/null+++ b/tests/unit/pipelex/libraries/test_written_method_trees.py@@ -0,0 +1,76 @@+from pathlib import Path++import pytest+from mthds.method_tree.digest import digest_of_contents++from pipelex.base_exceptions import PipelexUnexpectedError+from pipelex.libraries.method_trees import list_library_trees, list_local_method_tree, list_package_method_tree+from pipelex.libraries.written_trees import WRITTEN_METHOD_TREES, WrittenMethodTrees, write_method_tree++_CONTENTS: dict[str, bytes] = {+ "main.mthds": b'domain = "written"\n',+ "templates/invoice.html": b"<h1>{{ number }}</h1>",+}+++class TestWrittenMethodTrees:+ def test_a_tree_is_written_at_its_tree_paths_and_read_back_as_written(self, tmp_path: Path) -> None:+ contents = dict(_CONTENTS)+ destination = tmp_path / "tree"++ written = write_method_tree(contents=contents, destination=destination, expected_digest=digest_of_contents(contents))++ assert written.digest == digest_of_contents(contents)+ assert written.directory == destination.resolve()+ assert sorted(written.tree_paths) == ["main.mthds", "templates/invoice.html"]+ assert (destination / "templates" / "invoice.html").read_bytes() == contents["templates/invoice.html"]++ def test_a_copy_reading_back_as_another_tree_is_the_hosts_fault(self, tmp_path: Path) -> None:+ contents = {"main.mthds": _CONTENTS["main.mthds"]}++ with pytest.raises(PipelexUnexpectedError, match="reads back as"):+ write_method_tree(contents=contents, destination=tmp_path / "tree", expected_digest=digest_of_contents({"other.mthds": b"x"}))++ def test_a_destination_already_holding_files_is_refused(self, tmp_path: Path) -> None:+ destination = tmp_path / "tree"+ destination.mkdir()+ (destination / "stray.txt").write_text("stray", encoding="utf-8")+ contents = {"main.mthds": _CONTENTS["main.mthds"]}++ with pytest.raises(PipelexUnexpectedError, match="already holds files"):+ write_method_tree(contents=contents, destination=destination, expected_digest=digest_of_contents(contents))++ def test_a_held_tree_is_every_listing_of_its_directory_until_released(self, tmp_path: Path) -> None:+ contents = {"main.mthds": _CONTENTS["main.mthds"]}+ destination = tmp_path / "tree"+ written = write_method_tree(contents=contents, destination=destination, expected_digest=digest_of_contents(contents))+ registry = WrittenMethodTrees()++ with registry.holding(tree=written):+ assert registry.tree_written_at(directory=destination) is written+ assert registry.holds_tree(tree=written) is True++ assert registry.tree_written_at(directory=destination) is None+ assert registry.holds_tree(tree=written) is False++ def test_the_process_registry_answers_every_lister_while_the_tree_is_held(self, tmp_path: Path) -> None:+ contents = {"main.mthds": _CONTENTS["main.mthds"]}+ destination = tmp_path / "tree"+ written = write_method_tree(contents=contents, destination=destination, expected_digest=digest_of_contents(contents))+ # A file written after the tree, which a listing from disk would read, and the held tree does not hold.+ (destination / "late.txt").write_text("late", encoding="utf-8")++ with WRITTEN_METHOD_TREES.holding(tree=written):+ assert list_local_method_tree(handed_over=destination) is written+ assert list_library_trees(library_dir=destination) == [written]+ assert list_package_method_tree(root=destination, package_address="github.com/acme/written") is written++ assert "late.txt" in list_local_method_tree(handed_over=destination).tree_paths++ def test_one_directory_is_held_once(self, tmp_path: Path) -> None:+ contents = {"main.mthds": _CONTENTS["main.mthds"]}+ written = write_method_tree(contents=contents, destination=tmp_path / "tree", expected_digest=digest_of_contents(contents))+ registry = WrittenMethodTrees()++ with registry.holding(tree=written), pytest.raises(PipelexUnexpectedError, match="already held"), registry.holding(tree=written):+ passdiff --git a/tests/unit/pipelex/mthds_parsing/test_parser.py b/tests/unit/pipelex/mthds_parsing/test_parser.pyindex da3885a7d..ebb0db044 100644--- a/tests/unit/pipelex/mthds_parsing/test_parser.py+++ b/tests/unit/pipelex/mthds_parsing/test_parser.py@@ -1,3 +1,5 @@+from pathlib import Path+import pytestfrom pipelex import log, pretty_print@@ -37,3 +39,50 @@ class TestMthdsParser:assert len(err.validation_errors) == 1assert err.validation_errors[0].source == "broken.mthds"assert "TOML syntax error" in err.validation_errors[0].message++ def test_a_bundle_read_from_a_path_is_named_by_its_source_when_given(self, tmp_path: Path):+ """A host reading a tree it wrote out names each bundle by its tree path, never by the file it wrote."""+ bundle_path = tmp_path / "written" / "methods" / "greet.mthds"+ bundle_path.parent.mkdir(parents=True)+ bundle_path.write_text('domain = "greeting"\n', encoding="utf-8")++ blueprint = MthdsParser.make_pipelex_bundle_blueprint(bundle_path=bundle_path, mthds_source="methods/greet.mthds")++ assert blueprint.source == "methods/greet.mthds"++ def test_a_toml_error_of_a_bundle_read_from_a_path_names_its_source_not_the_path(self, tmp_path: Path):+ bundle_path = tmp_path / "written" / "methods" / "broken.mthds"+ bundle_path.parent.mkdir(parents=True)+ bundle_path.write_text('domain = "broken"\n[pipe.oops\n', encoding="utf-8")++ with pytest.raises(MthdsParserError) as exc_info:+ MthdsParser.make_pipelex_bundle_blueprint(bundle_path=bundle_path, mthds_source="methods/broken.mthds")++ (item,) = exc_info.value.validation_errors or []+ assert item.source == "methods/broken.mthds"+ assert "'methods/broken.mthds'" in item.message+ assert str(tmp_path) not in item.message+ assert str(tmp_path) not in exc_info.value.message++ @pytest.mark.parametrize(+ ("content", "line", "column"),+ [+ pytest.param(b"\xff", 1, 1, id="first_byte"),+ pytest.param(b'domain = "x"\ndefinition = "caf\xe9"\n', 2, 18, id="later_line"),+ ],+ )+ def test_a_bundle_file_that_is_not_utf8_is_a_located_parser_error(self, tmp_path: Path, content: bytes, line: int, column: int):+ """A file of a tree that is not text is the caller's invalid bundle, located at its first byte that does not decode."""+ bundle_path = tmp_path / "written" / "bad.mthds"+ bundle_path.parent.mkdir(parents=True)+ bundle_path.write_bytes(content)++ with pytest.raises(MthdsParserError) as exc_info:+ MthdsParser.make_pipelex_bundle_blueprint(bundle_path=bundle_path, mthds_source="bad.mthds")++ assert exc_info.value.validation_errors is not None+ (item,) = exc_info.value.validation_errors+ assert item.source == "bad.mthds"+ assert (item.line, item.column) == (line, column)+ assert "not UTF-8 text" in item.message+ assert str(tmp_path) not in item.messagediff --git a/tests/unit/pipelex/pipeline/test_direct_bundle_validator.py b/tests/unit/pipelex/pipeline/test_direct_bundle_validator.pyindex b9a9398b8..60087ba13 100644--- a/tests/unit/pipelex/pipeline/test_direct_bundle_validator.py+++ b/tests/unit/pipelex/pipeline/test_direct_bundle_validator.py@@ -43,6 +43,8 @@ class TestDirectBundleValidator:caller_identity=CallerIdentity(user_id="user-42", extras={"organization": "org_acme"}),graph_pipe_code="extract_document_markdown",submitted_files=submitted_files,+ method_dir=None,+ methods_dirs=None,)assert verdict is report@@ -55,6 +57,8 @@ class TestDirectBundleValidator:log_context="API validate",caller_identity=CallerIdentity(user_id="user-42", extras={"organization": "org_acme"}),submitted_files=submitted_files,+ method_dir=None,+ methods_dirs=None,)async def test_validate_bundle_error_becomes_the_invalid_arm(self, mocker: MockerFixture) -> None:@@ -73,6 +77,8 @@ class TestDirectBundleValidator:caller_identity=None,graph_pipe_code=None,submitted_files=None,+ method_dir=None,+ methods_dirs=None,)assert isinstance(verdict, ErrorReport)@@ -95,4 +101,6 @@ class TestDirectBundleValidator:caller_identity=None,graph_pipe_code=None,submitted_files=None,+ method_dir=None,+ methods_dirs=None,)diff --git a/tests/unit/pipelex/pipeline/test_runner_validate_plumbing.py b/tests/unit/pipelex/pipeline/test_runner_validate_plumbing.pyindex 087f1ecc9..919f1db66 100644--- a/tests/unit/pipelex/pipeline/test_runner_validate_plumbing.py+++ b/tests/unit/pipelex/pipeline/test_runner_validate_plumbing.py@@ -97,6 +97,8 @@ class TestRunnerValidatePlumbing:library_dirs_are_callers=True,collect_dry_run_warnings=True,submitted_files=submitted_files,+ method_dir=None,+ methods_dirs=None,)async def test_no_library_dirs_passes_none_and_strict_default(self, mocker: MockerFixture) -> None:@@ -114,6 +116,8 @@ class TestRunnerValidatePlumbing:library_dirs_are_callers=False,collect_dry_run_warnings=True,submitted_files=None,+ method_dir=None,+ methods_dirs=None,)async def test_validation_runs_with_the_protocols_caller_in_scope(self, mocker: MockerFixture) -> None:diff --git a/tests/unit/pipelex/plugins/test_bundle_validator_registry.py b/tests/unit/pipelex/plugins/test_bundle_validator_registry.pyindex d984d927c..9fbeb9f80 100644--- a/tests/unit/pipelex/plugins/test_bundle_validator_registry.py+++ b/tests/unit/pipelex/plugins/test_bundle_validator_registry.py@@ -34,13 +34,15 @@ class _FakeBundleValidator:async def validate_bundles(self,*,- mthds_contents: list[str],+ mthds_contents: list[str] | None,mthds_sources: list[str] | None,allow_signatures: bool,library_dirs: Sequence[Path] | None,caller_identity: CallerIdentity | None,graph_pipe_code: str | None,submitted_files: Mapping[str, bytes] | None,+ method_dir: Path | None,+ methods_dirs: Sequence[Path] | None,) -> PipelexValidationReport | ErrorReport:raise NotImplementedErrordiff --git a/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py b/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.pyindex 3a6759bd3..21c53a019 100644--- a/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py+++ b/tests/unit/pipelex/tools/storage/test_gcp_storage_provider.py@@ -319,3 +319,30 @@ class TestGcpStorageProvider:with pytest.raises(StorageFileNotFoundError, match=r"missing/file\.pdf"):await gcp_provider_no_signed_urls.load_head(uri=f"{PIPELEX_STORAGE_SCHEME}missing/file.pdf", nb_bytes=8192)++ async def test_load_size_reads_the_object_s_metadata(+ self,+ mocker: MockerFixture,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ mock_gcp_storage: dict[str, Any],+ ) -> None:+ """A size read fetches the object's metadata alone, never its bytes."""+ held = mocker.MagicMock()+ held.size = 4096+ mock_gcp_storage["bucket"].get_blob.return_value = held++ size = await gcp_provider_no_signed_urls.load_size(uri=f"{PIPELEX_STORAGE_SCHEME}test/file.pdf")++ assert size == 4096+ mock_gcp_storage["bucket"].get_blob.assert_called_once_with("test/file.pdf")+ mock_gcp_storage["blob"].download_as_bytes.assert_not_called()++ async def test_load_size_of_a_missing_object_raises_not_found(+ self,+ gcp_provider_no_signed_urls: GcpStorageProvider,+ mock_gcp_storage: dict[str, Any],+ ) -> None:+ mock_gcp_storage["bucket"].get_blob.return_value = None++ with pytest.raises(StorageFileNotFoundError, match=r"missing/file\.pdf"):+ await gcp_provider_no_signed_urls.load_size(uri=f"{PIPELEX_STORAGE_SCHEME}missing/file.pdf")diff --git a/tests/unit/pipelex/tools/storage/test_local_storage_provider.py b/tests/unit/pipelex/tools/storage/test_local_storage_provider.pyindex dfdd585c3..c47980a7f 100644--- a/tests/unit/pipelex/tools/storage/test_local_storage_provider.py+++ b/tests/unit/pipelex/tools/storage/test_local_storage_provider.py@@ -222,3 +222,35 @@ class TestLocalStorageProvider:await provider.load(uri=f"{PIPELEX_STORAGE_SCHEME}{key}")assert key in str(exc_info.value)++ @pytest.mark.parametrize("held_before", [None, b"the bytes held before"])+ async def test_a_write_cut_short_leaves_the_key_as_it_was(self, tmp_path: Path, mocker: MockerFixture, held_before: bytes | None) -> None:+ """A write that fails midway, as on a full disk, publishes nothing: the key holds what it held, and no temporary file stays."""+ provider = LocalStorageProvider(root_path=tmp_path)+ key = "blobs/one.bin"+ if held_before is not None:+ await provider.store(data=held_before, key=key)+ real_replace = mocker.patch.object(Path, "replace", autospec=True, side_effect=OSError(28, "No space left on device"))++ with pytest.raises(StorageLocalError, match="No space left on device"):+ await provider.store(data=b"x" * 10_000, key=key)++ real_replace.assert_called_once()+ if held_before is None:+ assert not (tmp_path / key).exists()+ else:+ assert (tmp_path / key).read_bytes() == held_before+ assert sorted(path.name for path in (tmp_path / "blobs").iterdir()) == ([] if held_before is None else ["one.bin"])++ async def test_a_write_retried_after_a_failure_lands_whole(self, tmp_path: Path, mocker: MockerFixture) -> None:+ provider = LocalStorageProvider(root_path=tmp_path)+ key = "blobs/one.bin"+ mocker.patch("pipelex.tools.storage.local_storage_provider.os.fsync", side_effect=OSError(5, "Input/output error"))+ with pytest.raises(StorageLocalError):+ await provider.store(data=b"payload", key=key)+ mocker.stopall()++ await provider.store(data=b"payload", key=key)++ assert (tmp_path / key).read_bytes() == b"payload"+ assert [path.name for path in (tmp_path / "blobs").iterdir()] == ["one.bin"]diff --git a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.pyindex 221602fee..87a527b63 100644--- a/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py+++ b/tests/unit/pipelex/tools/storage/test_s3_storage_provider.py@@ -497,3 +497,48 @@ class TestS3StorageProvider:with pytest.raises(StorageS3Error, match="AccessDenied"):await s3_provider_no_signed_urls.load_head(uri=f"{PIPELEX_STORAGE_SCHEME}test/file.pdf", nb_bytes=8192)++ async def test_load_size_reads_the_content_length_of_a_head_request(+ self,+ mocker: MockerFixture,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aiobotocore: dict[str, Any],+ ) -> None:+ """A size read asks S3 for the object's metadata alone, never its bytes."""+ mock_aiobotocore["client"].head_object = mocker.AsyncMock(return_value={"ContentLength": 4096})++ size = await s3_provider_no_signed_urls.load_size(uri=f"{PIPELEX_STORAGE_SCHEME}test/file.pdf")++ assert size == 4096+ mock_aiobotocore["client"].head_object.assert_called_once_with(Bucket=S3_TEST_BUCKET, Key="test/file.pdf")+ mock_aiobotocore["client"].get_object.assert_not_called()++ @pytest.mark.parametrize("error_code", ["404", "NoSuchKey", "NotFound"])+ async def test_load_size_of_a_missing_object_raises_not_found(+ self,+ mocker: MockerFixture,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aiobotocore: dict[str, Any],+ error_code: str,+ ) -> None:+ """A HEAD answer has no body, so S3 names a missing object by its bare status code."""+ missing = ClientError({"Error": {"Code": error_code, "Message": "Not Found"}}, "HeadObject")+ mock_aiobotocore["client"].head_object = mocker.AsyncMock(side_effect=missing)++ with pytest.raises(StorageFileNotFoundError, match=r"missing/file\.pdf"):+ await s3_provider_no_signed_urls.load_size(uri=f"{PIPELEX_STORAGE_SCHEME}missing/file.pdf")++ async def test_load_size_maps_other_errors(+ self,+ mocker: MockerFixture,+ s3_provider_no_signed_urls: S3StorageProvider,+ mock_aiobotocore: dict[str, Any],+ ) -> None:+ access_denied = ClientError({"Error": {"Code": "403", "Message": "Forbidden"}}, "HeadObject")+ mock_aiobotocore["client"].head_object = mocker.AsyncMock(side_effect=access_denied)+ with pytest.raises(StorageS3Error, match="403"):+ await s3_provider_no_signed_urls.load_size(uri=f"{PIPELEX_STORAGE_SCHEME}test/file.pdf")++ mock_aiobotocore["client"].head_object = mocker.AsyncMock(side_effect=EndpointConnectionError(endpoint_url="https://s3.example.com"))+ with pytest.raises(StorageS3Error):+ await s3_provider_no_signed_urls.load_size(uri=f"{PIPELEX_STORAGE_SCHEME}test/file.pdf")diff --git a/tests/unit/pipelex/tools/storage/test_storage_load_size.py b/tests/unit/pipelex/tools/storage/test_storage_load_size.pynew file mode 100644index 000000000..b6dcf5f77--- /dev/null+++ b/tests/unit/pipelex/tools/storage/test_storage_load_size.py@@ -0,0 +1,69 @@+"""The size read every storage provider offers: from metadata where it can, else by loading the object.++A provider that can read an object's size without its bytes (local, in memory, S3, GCS) overrides it; one that cannot,+including an external storage plugin written before the size read existed, keeps working through the default.+"""++from pathlib import Path++import pytest+from pytest_mock import MockerFixture+from typing_extensions import override++from pipelex.tools.storage.exceptions import StorageFileNotFoundError, StorageLocalError+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider+from pipelex.tools.storage.local_storage_provider import LocalStorageProvider+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract, StoredData++PAYLOAD = b"%PDF-1.7\n" + b"x" * 20_000+++class _LoadOnlyProvider(StorageProviderAbstract):+ """A provider implementing only the abstract reads and writes, as an older external plugin does."""++ def __init__(self) -> None:+ self.objects: dict[str, bytes] = {}++ @override+ async def _load_with_metadata(self, key: str) -> StoredData:+ if key not in self.objects:+ msg = f"File not found: '{key}'"+ raise StorageFileNotFoundError(msg)+ return StoredData(data=self.objects[key], mime_type=None)++ @override+ async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ self.objects[key] = data++ @override+ async def public_url(self, uri: str) -> str | None:+ return None++[email protected](loop_scope="class")+class TestStorageLoadSize:+ async def test_every_provider_reads_the_size_of_what_it_stored(self, tmp_path: Path) -> None:+ for provider in [InMemoryStorageProvider(), LocalStorageProvider(root_path=tmp_path), _LoadOnlyProvider()]:+ uri = await provider.store(data=PAYLOAD, key="size/file.pdf")++ assert await provider.load_size(uri=uri) == len(PAYLOAD)++ async def test_a_missing_object_raises_not_found(self, tmp_path: Path) -> None:+ for provider in [InMemoryStorageProvider(), LocalStorageProvider(root_path=tmp_path), _LoadOnlyProvider()]:+ with pytest.raises(StorageFileNotFoundError):+ await provider.load_size(uri=f"{PIPELEX_STORAGE_SCHEME}missing/file.pdf")++ async def test_the_local_size_is_read_without_opening_the_file(self, tmp_path: Path, mocker: MockerFixture) -> None:+ provider = LocalStorageProvider(root_path=tmp_path)+ uri = await provider.store(data=PAYLOAD, key="size/file.pdf")+ opened = mocker.patch("pipelex.tools.storage.local_storage_provider.aiofiles.open")++ assert await provider.load_size(uri=uri) == len(PAYLOAD)+ opened.assert_not_called()++ async def test_a_local_key_naming_a_directory_is_a_local_storage_error(self, tmp_path: Path) -> None:+ provider = LocalStorageProvider(root_path=tmp_path)+ (tmp_path / "a_directory").mkdir()++ with pytest.raises(StorageLocalError, match="a_directory"):+ await provider.load_size(uri=f"{PIPELEX_STORAGE_SCHEME}a_directory")