Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Bundle-template-files · L-261011-d4535d (#1422)

Louis Choquel9ddbc3a875a3
diff --git a/.drift/acks/hub-layering-convention.toml b/.drift/acks/hub-layering-convention.toml
index dc444bf42..e6fdf4150 100644
--- a/.drift/acks/hub-layering-convention.toml
+++ b/.drift/acks/hub-layering-convention.toml
@@ -1,10 +1,10 @@
contract = "hub-layering-convention"
-digest = "sha256:a6c9b398552db5a716bbd6fe02c68c4f10a1bcae770e232be643c55b52585880"
-reviewed_by = "Claude (Opus 5.5)"
-reviewed_at = "2026-10-10T11:27:08Z"
-rationale = "clean-pass: runtime_hub.py rewords two warnings (a second set_config, and a live run forced dry under a keyless boot, whose advice now rides user_action) and imports USER_ACTION_FIELD from pipelex.tools.log.log_fields, a kernel-layer leaf; no accessor was added, moved or removed and KERNEL_LAYER_PACKAGES is unchanged. Ran the page's two mechanical checks from the code: every public function and class defined at module level in runtime_hub.py and interpreter_hub.py appears in the partition tables, and every package of the guard's KERNEL_LAYER_PACKAGES is named under the layer rule. check-hub-layering passes with the new import. Nothing was stale."
+digest = "sha256:a936b7ba24211f227bcc1b60046075b56522c24b7f6de366ee2fa091ce821267"
+reviewed_by = "Louis Choquel"
+reviewed_at = "2026-10-12T02:37:48Z"
+rationale = "real-catch: interpreter_hub.py gains the module-level does_pipe_run_leave_process (and InterpreterHub.set_pipe_run takes leaves_process, stored beside the pipe run), so the run row of the partition table now names it. No import was added to either hub, KERNEL_LAYER_PACKAGES is unchanged, and every public module-level function and class of both hubs is in the tables."
[trigger_files]
"pipelex/cli/dev_cli/commands/hub_layering_guard.py" = "blob:fda2237457fe0b6830ff2ae8f720698735bec4c3"
-"pipelex/interpreter_hub.py" = "blob:327e75d0fbb5bdf46f6cbf06024f83840f0106d0"
+"pipelex/interpreter_hub.py" = "blob:023f8452daf76dde409afc7c0966b149b11edabb"
"pipelex/runtime_hub.py" = "blob:65beba670c857897858b75fd17bbbc03ba84e750"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2398f6b46..2dab360bc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,12 +4,27 @@
### Added
-- **Template-check warnings in the validation verdict**: a warning the engine's template check finds in a `PipeDocGen` step's office template file, such as an Excel defined name or a PowerPoint shape name that matches no input field, now rides the verdict's advisory `warnings` on `pipelex validate`, `pipelex-agent validate` and the API server's `POST /v1/validate`, where it was only logged. It is a `dry_run` item with the new `error_type` `template_check_warning`, the step's `pipe_code` and `domain_code`, and two new optional locators of `ValidationErrorItem`, `template_file` and `template_location`, which the server's OpenAPI schema publishes, a step of the server's own library directories naming no file of theirs; an error finding still fails the validation, `validate pipe <code>` on both CLIs carries these warnings alone, and a dry run that returns no verdict, such as `pipelex run --dry-run`, `pipelex fix bundle` or `pipelex build inputs`, still logs them.
+- **`resources` on the API server's routes**: `POST /v1/validate`, `/v1/execute`, `/v1/start`, `/v1/pipe-io`, `/v1/resolve` and `/v1/codegen` take the files a bundle sent as content reads, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64, so a caller with no local pipelex can validate, dry-run and run a method that prints with an `.xlsx`, `.docx`, `.pptx` or `.html` template file. A key that cannot name a file inside the bundle answers `422 InvalidResourcePath`, a file no step names `422 BundleResourceNotNamedError`, a value that is not base64 `400 InvalidBase64`, and a request over `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` or `MAX_RESOURCES_TOTAL_KIB` (32 files, 5 MiB each and 10 MiB in all by default) `413 PayloadTooLarge`; `resources` beside a `method_ref` is a `422`. A method bundle sent as `bundle_b64` or `files` may ship such a file instead, beside the `.mthds` file whose step names it, and it reaches the step the same way, under the same limits; two different files under one path, shipped twice or shipped and sent, answer `422 InvalidResourcePath`, as two spellings of one key do before any value is decoded. On an orchestrator other than `direct`, such as Temporal, a run stores each office template file under its `storage_scope` at `assets/bundle/` before it is dispatched, so the job carries a `pipelex-storage://` reference rather than the bytes, while the `.html` template files, which travel inline with the method, may total 512 KiB at most, a fixed bound refused with `413 BundleResourcesTooLargeError` once the bundle is parsed.
+- **Template files sent with a bundle loaded from content strings**: `PipelexMTHDSProtocol`, `validate_bundle`, `validate_bundles_in_process`, `acquire_library`, `pipeline_run_setup` and `resolve_crate_from_contents` take `bundle_resources`, built with `make_bundle_resources` from the bytes of each file keyed by the path its step names. The library holds them and its `LibraryCrate` carries them, outside its fingerprint, so a `PipeDocGen` step finds its template file wherever the bundle is loaded: an `.html` one is checked against the inputs at load, the engine's template check runs on an office one at validation and in the dry run, and the print stage reads it, from the run's storage when the crate carries a reference. Before a run, live or dry, leaves the process, `prepare_job_for_remote_dispatch` stores its office template files under the run's storage scope; `PipelexMTHDSProtocol.execute` calls it when the hub's pipe run is a boot orchestrator's, and a dry run's template check reads a stored file under the run's read scope, as the print stage does. A step of a bundle the runtime read from its own disk, such as the host's library, still reads only beside it, so a file sent under the same path never stands in for its template. A file no step names raises `BundleResourceNotNamedError`, a path outside the bundle `BundleResourcePathError`, and `.html` template files totalling more than `MAX_LOAD_STAGE_RESOURCES_BYTES` (512 KiB), which travel inline with the method wherever it runs, `BundleResourcesTooLargeError`.
+- **Template-check warnings in the validation verdict**: a warning the engine's template check finds in a `PipeDocGen` step's office template file, such as an Excel defined name or a PowerPoint shape name that matches no input field, now rides the verdict's advisory `warnings` on `pipelex validate`, `pipelex-agent validate` and the API server's `POST /v1/validate`, where it was only logged. It is a `dry_run` item with the new `error_type` `template_check_warning`, the step's `pipe_code` and `domain_code`, and two new optional locators of `ValidationErrorItem`, `template_file` and `template_location`, the latter unset when the engine names no place, which the server's OpenAPI schema publishes, a step of the server's own library directories naming no file of theirs; an error finding still fails the validation, `validate pipe <code>` on both CLIs carries these warnings alone, and a dry run that returns no verdict, such as `pipelex run --dry-run`, `pipelex fix bundle` or `pipelex build inputs`, still logs them.
### Changed
+- **`BundleValidatorProtocol.validate_bundles` takes `bundle_resources` (Breaking)**: a bundle validator registered for an orchestration mode receives the files sent with the contents, or `None`, and hands them to the validation it runs, so a remote validator implements the parameter and carries the files to its worker.
+- **`InterpreterHub.set_pipe_run` takes `leaves_process` (Breaking)**: whoever sets the hub's pipe run says whether it hands each job to an orchestrator outside the process, as a boot orchestrator's does, which `does_pipe_run_leave_process` reads back so a run's submitted files are stored before it leaves.
+- **`DocumentComposition.make_render_job` is asynchronous (Breaking)**: it takes the run's `storage_provider`, since a composition's template file may be a `BundleResource` read back from storage, and a `TEMPLATE_FILE` composition carries either `template_path` or `template_resource`. A stored template file's URI is among the composition's `referenced_uris()`, so the run's read scope authorizes it before the print stage reads it.
+- **A `PipeDocGen` step's `template_file` must be a path inside its bundle (Breaking)**: an empty or absolute path, a Windows root, a drive, a UNC share, a `..` segment, a backslash or a colon is refused when the blueprint validates, by the rule the files sent with a bundle are keyed by, so every template file a step names can be sent with its bundle; an absolute path naming a file inside the bundle's directory, a path such as `templates/../invoice.docx` and a file name holding a colon used to load. The refusal does not repeat the path, so a host library's step can no longer put the host's own path into a caller's verdict through its template file. Beside a server's own library directories, the verdict also withholds the template files their steps name, as their bundle files already were, both when such a step is refused while they load and when the engine's template check refuses its template in the dry run of a caller's pipe that runs it; a template file the caller's own content names is kept.
+- **A string-loaded `PipeDocGen` step without its template file says it was not supplied**: the load refusal now names the path the file is expected under and says to send it with the bundle, where it used to say the bundle was not loaded from a file on disk.
- **`build_advisory_warnings` and `collect_advisory_warnings` take the sweep's `dry_run_result` (Breaking)**: the composition point of the advisory warnings reads what the dry runs found from the per-pipe status map, so a caller passes the map its validation sweep returned. `DryRunOutput` carries those findings in its new `warnings` when the sweep is asked to collect them with the new `collect_dry_run_warnings` of `validate_bundle`, `BundleValidator.validate_pipes` and `BundleValidator.validate_current_library`; a sweep that is not asked leaves them to the log, as before.
+### Fixed
+
+- **Contents named by `bundle_uris` read their template files beside their file**: a bundle passed as `mthds_contents` with the path of the file it was read from in `bundle_uris`, to `PipelexMTHDSProtocol`, `acquire_library` or `pipeline_run_setup`, and not already loaded from the library directories, now loads as read from that file, so a `PipeDocGen` step's `template_file` is found beside it, where it was refused as if the bundle had no directory, and an address-based reference is looked for in a `.mthds/methods/` directory above it.
+
+### Security
+
+- **A bundle loaded from content strings never reads the runtime's disk**: a `PipeDocGen` step of such a bundle read its `template_file` beside whatever its source named, so a caller's `mthds_sources` on `/v1/validate`, a `files[].source` on `/v1/resolve`, `/v1/codegen` or `/v1/pipe-io`, or a `source` key written on the step in the bundle's text could make the runner read a file from its own disk, and an address-based reference searched `.mthds/methods/` directories above that label too. Both now start only from a bundle file the runtime read itself, which the parser records as `PipelexBundleBlueprint.host_bundle_file` and the crate carries as `LibraryCrate.host_bundle_files`, outside its fingerprint and its normalized form; a content-string bundle reads only the files sent with it, and a step's `source` is overwritten whenever the pipe is built.
+
## [v0.84.0] - 2026-10-12
### Highlights
diff --git a/api/pipelex_api/bundle_resources.py b/api/pipelex_api/bundle_resources.py
new file mode 100644
index 000000000..c299a2a60
--- /dev/null
+++ b/api/pipelex_api/bundle_resources.py
@@ -0,0 +1,148 @@
+"""Decode the files a request submits with its bundle that its steps read, such as a template file.
+
+They come in two ways. `resources` is a JSON object mapping each file's bundle-relative path, the exact path a step
+names (a `PipeDocGen` step's `template_file`), to the file's bytes in standard base64, beside `mthds_contents` or a
+method bundle. A method bundle (`bundle_b64` or `files`) may also ship the file beside its `.mthds` file, which
+`files_its_steps_read` picks out. Every bundle of the request is read as sitting in one directory, so one path holds
+one file. The decoded files go to the runtime as `BundleResources`, which refuses a file none of the bundle's steps
+names once the bundle is parsed.
+
+Refusals, in the order they are checked:
+
+- more `resources` than `MAX_RESOURCES_PER_REQUEST` → 413 `PayloadTooLarge`;
+- a path that cannot name a file inside the bundle (absolute, a `..` segment, a backslash, a colon, empty, or a
+ second spelling of a path already submitted) → 422 `InvalidResourcePath`, before any value is decoded;
+- more files than `MAX_RESOURCES_PER_REQUEST`, counting those the method bundle ships → 413 `PayloadTooLarge`;
+- a file whose encoded length already exceeds the per-file ceiling → 413 `PayloadTooLarge`, before decoding;
+- a value that is not valid base64 → 400 `InvalidBase64`, as `bundle_b64` is;
+- two different files under one path, sent in `resources` and shipped in the bundle, or shipped beside two of its
+ `.mthds` files → 422 `InvalidResourcePath`;
+- a decoded or shipped file over `MAX_RESOURCE_FILE_BYTES`, or a running total over `MAX_RESOURCES_TOTAL_BYTES` → 413.
+"""
+
+import base64
+import binascii
+from collections.abc import Iterable, Mapping
+from pathlib import PurePosixPath
+from typing import NoReturn
+
+from pipelex import log
+from pipelex.core.bundle_resources.bundle_resource import (
+ BundleResources,
+ canonical_resource_path,
+ canonical_resource_path_or_none,
+ make_bundle_resources,
+)
+from pipelex.core.bundle_resources.exceptions import BundleResourcePathError
+from pipelex.mthds_parsing.exceptions import MthdsParserError
+from pipelex.mthds_parsing.parser import MthdsParser
+from pipelex.tools.log.error_fields import error_fields
+
+from pipelex_api.error_types import ErrorType
+from pipelex_api.errors import raise_bad_request, raise_payload_too_large, raise_validation_error
+from pipelex_api.limits import MAX_RESOURCE_FILE_BYTES, MAX_RESOURCES_PER_REQUEST, MAX_RESOURCES_TOTAL_BYTES
+
+# Base64 inflates by 4/3, so a file within the ceiling encodes to at most this many characters (+4 covers padding).
+# Checking the encoded length first refuses an oversized file before it is decoded into memory.
+_MAX_RESOURCE_B64_CHARS = MAX_RESOURCE_FILE_BYTES * 4 // 3 + 4
+
+
+def _refuse_two_files_under_one_path(*, path: str, where: str) -> NoReturn:
+ msg = (
+ f"Two different files are named '{path}', {where}. Every bundle of a request reads its files as if it sat in one "
+ "directory, so a path names one file: send one of them, or rename one and the step that reads it."
+ )
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_RESOURCE_PATH)
+
+
+def files_its_steps_read(
+ *, mthds_entries: Iterable[tuple[PurePosixPath, str]], other_entries: Iterable[tuple[PurePosixPath, bytes]]
+) -> dict[str, bytes]:
+ """The files a method bundle ships beside its `.mthds` files that their steps read, keyed by the path a step names.
+
+ Each `.mthds` file is parsed for the files its steps name, such as a `PipeDocGen` step's `template_file`, which is
+ looked up relative to that file's own directory in the bundle. The bundle's `.mthds` files load as content strings,
+ so these files travel with them as the `resources` a request sends do, wherever the run goes. A file a step names
+ that the bundle does not ship is left out, for `resources` to supply or the load to refuse saying so, and a `.mthds`
+ file that does not parse is skipped, for the load to give its verdict on.
+
+ Raises (as a 422 `InvalidResourcePath`): two steps naming one path, beside two `.mthds` files that ship different files.
+ """
+ shipped = dict(other_entries)
+ files: dict[str, bytes] = {}
+ for mthds_relpath, mthds_content in mthds_entries:
+ try:
+ blueprint = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=mthds_content)
+ except MthdsParserError as exc:
+ log.debug("A method bundle's file did not parse, so the files its steps read are left to its load", fields=error_fields(exc=exc))
+ continue
+ for pipe_blueprint in (blueprint.pipe or {}).values():
+ for named_path in pipe_blueprint.bundle_resource_reads:
+ canonical_path = canonical_resource_path_or_none(path=named_path)
+ if canonical_path is None:
+ continue
+ data = shipped.get(mthds_relpath.parent / canonical_path)
+ if data is None:
+ continue
+ known = files.get(canonical_path)
+ if known is not None and known != data:
+ _refuse_two_files_under_one_path(path=canonical_path, where="shipped beside two of the bundle's .mthds files")
+ files[canonical_path] = data
+ return files
+
+
+def decode_bundle_resources(*, resources: dict[str, str] | None, bundle_files: Mapping[str, bytes] | None = None) -> BundleResources | None:
+ """The files the bundle's steps read, decoded and keyed by canonical path, or None when there are none.
+
+ `resources` are the ones the request sends, in base64. `bundle_files` are the ones its method bundle ships, as
+ `files_its_steps_read` picks them out. Both are held to the same limits, together, and a file in both must be the same.
+ """
+ if not resources and not bundle_files:
+ return None
+ sent = resources or {}
+ shipped = bundle_files or {}
+ if len(sent) > MAX_RESOURCES_PER_REQUEST:
+ raise_payload_too_large(message=f"resources exceeds the {MAX_RESOURCES_PER_REQUEST}-file limit (got {len(sent)})")
+ # Every path first, so a second spelling of one file is refused as such before any value is decoded.
+ sent_by_path: dict[str, tuple[str, str]] = {}
+ for path, encoded in sent.items():
+ try:
+ canonical_path = canonical_resource_path(path=path)
+ except BundleResourcePathError as exc:
+ raise_validation_error(message=exc.message, error_type=ErrorType.INVALID_RESOURCE_PATH)
+ if canonical_path in sent_by_path:
+ msg = f"Two files submitted with the bundle name the same file '{canonical_path}'."
+ raise_validation_error(message=msg, error_type=ErrorType.INVALID_RESOURCE_PATH)
+ sent_by_path[canonical_path] = (path, encoded)
+ file_count = len(sent_by_path.keys() | shipped.keys())
+ if file_count > MAX_RESOURCES_PER_REQUEST:
+ msg = (
+ f"The files the bundle's steps read, in resources and in the bundle, exceed the {MAX_RESOURCES_PER_REQUEST}-file limit (got {file_count})"
+ )
+ raise_payload_too_large(message=msg)
+ files: dict[str, bytes] = {}
+ total_bytes = 0
+ for canonical_path, (path, encoded) in sent_by_path.items():
+ if len(encoded) > _MAX_RESOURCE_B64_CHARS:
+ raise_payload_too_large(message=f"The file '{path}' in resources exceeds the {MAX_RESOURCE_FILE_BYTES // 1024} KiB per-file limit")
+ try:
+ data = base64.b64decode(encoded, validate=True)
+ except (binascii.Error, ValueError) as decode_error:
+ # DEBUG: the refusal below is a caller's mistake, which the error handler already logs; this keeps the cause.
+ log.debug("A submitted file's base64 could not be decoded", fields=error_fields(exc=decode_error))
+ raise_bad_request(message=f"The file '{path}' in resources is not valid base64", error_type=ErrorType.INVALID_BASE64)
+ shipped_data = shipped.get(canonical_path)
+ if shipped_data is not None and shipped_data != data:
+ _refuse_two_files_under_one_path(path=canonical_path, where="one sent in resources and one shipped in the bundle")
+ files[canonical_path] = data
+ for canonical_path, data in shipped.items():
+ files.setdefault(canonical_path, data)
+ for canonical_path, data in files.items():
+ if len(data) > MAX_RESOURCE_FILE_BYTES:
+ raise_payload_too_large(
+ message=f"The file '{canonical_path}' a step reads exceeds the {MAX_RESOURCE_FILE_BYTES // 1024} KiB per-file limit"
+ )
+ total_bytes += len(data)
+ if total_bytes > MAX_RESOURCES_TOTAL_BYTES:
+ raise_payload_too_large(message=f"The files the bundle's steps read exceed the {MAX_RESOURCES_TOTAL_BYTES // 1024} KiB total-size limit")
+ return make_bundle_resources(files=files)
diff --git a/api/pipelex_api/error_types.py b/api/pipelex_api/error_types.py
index a3705efe5..27048d71c 100644
--- a/api/pipelex_api/error_types.py
+++ b/api/pipelex_api/error_types.py
@@ -61,6 +61,10 @@ class ErrorType(StrEnum):
# A run request carried a malformed method bundle (`bundle_b64` / `files`): a corrupt zip, an
# unsafe entry name (absolute path or `..` traversal), or both transport forms supplied at once.
INVALID_BUNDLE = "InvalidBundle"
+ # A request's `resources` map keys a submitted file by a path that cannot name a file inside the
+ # bundle: absolute, with a `..` segment, a backslash or a colon, empty, or a second spelling of a
+ # path already submitted. A 422: the key is the path a step names, relative to the bundle.
+ INVALID_RESOURCE_PATH = "InvalidResourcePath"
# A method carrying custom Python (`.py` in a bundle, or in a fetched `method_ref` package)
# reached a deployment that is NOT sandbox-hosted. Running customer code in-process is
# refused (403): custom code is a sandbox-hosted capability only. Use a sandbox-hosted
diff --git a/api/pipelex_api/exception_handlers.py b/api/pipelex_api/exception_handlers.py
index 4158f0481..c05d824a6 100644
--- a/api/pipelex_api/exception_handlers.py
+++ b/api/pipelex_api/exception_handlers.py
@@ -341,6 +341,11 @@ _ERROR_TYPE_STATUS_OVERRIDES: dict[str, int] = {
# sandboxed PipeFuncs, not in-process Python) → 403, the same status as the bundle
# transport's custom-code gate.
"MethodStructuresRefusedError": 403,
+ # The files submitted in `resources` that the bundle's steps read when the method loads, its HTML templates,
+ # total more than they may, since they travel inline with the method wherever it runs. Only the parsed bundle
+ # says which files those are, so the library refuses them rather than this server's decoding: the request
+ # entity itself is too large → 413, as the decoder's own size refusals are.
+ "BundleResourcesTooLargeError": 413,
}
diff --git a/api/pipelex_api/limits.py b/api/pipelex_api/limits.py
index 2884d681e..dd287e60e 100644
--- a/api/pipelex_api/limits.py
+++ b/api/pipelex_api/limits.py
@@ -25,6 +25,9 @@ DEFAULT_MAX_CALLBACK_URLS = 5
DEFAULT_MAX_CALLBACK_URL_LEN = 2048
DEFAULT_MAX_BUNDLE_FILES = 128 # entries in a materialized method bundle (.mthds + .py + requirements.txt)
DEFAULT_MAX_BUNDLE_TOTAL_KIB = 8 * 1024 # 8 MiB decompressed across the whole bundle (zip-bomb guard)
+DEFAULT_MAX_RESOURCES_PER_REQUEST = 32 # files submitted with a bundle that its steps read (`resources`)
+DEFAULT_MAX_RESOURCE_FILE_KIB = 5 * 1024 # 5 MiB decoded per submitted file, such as an office template
+DEFAULT_MAX_RESOURCES_TOTAL_KIB = 10 * 1024 # 10 MiB decoded across a request's submitted files
DEFAULT_MAX_METHOD_CACHE_CLONES = 64 # cached method-package clones (one per resolved commit SHA)
DEFAULT_MAX_METHOD_CACHE_TOTAL_KIB = 512 * 1024 # 512 MiB across all cached clones
DEFAULT_MAX_METHOD_CACHE_AGE_HOURS = 24 # a cached clone unused for this long is evicted
@@ -43,6 +46,10 @@ MAX_CALLBACK_URL_LEN = get_positive_int_env(env_var="MAX_CALLBACK_URL_LEN", defa
MAX_BUNDLE_FILES = get_positive_int_env(env_var="MAX_BUNDLE_FILES", default=DEFAULT_MAX_BUNDLE_FILES)
MAX_BUNDLE_TOTAL_BYTES = get_positive_int_env(env_var="MAX_BUNDLE_TOTAL_KIB", default=DEFAULT_MAX_BUNDLE_TOTAL_KIB) * 1024
+MAX_RESOURCES_PER_REQUEST = get_positive_int_env(env_var="MAX_RESOURCES_PER_REQUEST", default=DEFAULT_MAX_RESOURCES_PER_REQUEST)
+MAX_RESOURCE_FILE_BYTES = get_positive_int_env(env_var="MAX_RESOURCE_FILE_KIB", default=DEFAULT_MAX_RESOURCE_FILE_KIB) * 1024
+MAX_RESOURCES_TOTAL_BYTES = get_positive_int_env(env_var="MAX_RESOURCES_TOTAL_KIB", default=DEFAULT_MAX_RESOURCES_TOTAL_KIB) * 1024
+
MAX_METHOD_CACHE_CLONES = get_positive_int_env(env_var="MAX_METHOD_CACHE_CLONES", default=DEFAULT_MAX_METHOD_CACHE_CLONES)
MAX_METHOD_CACHE_TOTAL_BYTES = get_positive_int_env(env_var="MAX_METHOD_CACHE_TOTAL_KIB", default=DEFAULT_MAX_METHOD_CACHE_TOTAL_KIB) * 1024
MAX_METHOD_CACHE_AGE_SECONDS = get_positive_int_env(env_var="MAX_METHOD_CACHE_AGE_HOURS", default=DEFAULT_MAX_METHOD_CACHE_AGE_HOURS) * 3600
diff --git a/api/pipelex_api/routes/pipelex/crate_ops.py b/api/pipelex_api/routes/pipelex/crate_ops.py
index 456fe5896..7916d110d 100644
--- a/api/pipelex_api/routes/pipelex/crate_ops.py
+++ b/api/pipelex_api/routes/pipelex/crate_ops.py
@@ -27,6 +27,7 @@ from pipelex.pipeline.resolve_bundle import resolve_crate_from_contents
from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
from pydantic import BaseModel, Field
+from pipelex_api.bundle_resources import decode_bundle_resources
from pipelex_api.error_types import ErrorType
from pipelex_api.errors import raise_not_implemented
from pipelex_api.method_source import fetch_method_mthds_files
@@ -151,6 +152,8 @@ def resolve_requested_crate(request_data: MthdsFilesRequest) -> ResolvedClosure:
crate = resolve_crate_from_contents(
mthds_contents=[item.content for item in selection.files],
mthds_sources=[item.source for item in selection.files],
+ # The files the closure's steps read, such as a PipeDocGen template file, which the load reads.
+ bundle_resources=decode_bundle_resources(resources=request_data.resources),
)
return ResolvedClosure(crate=crate, files=selection.files, manifest_main_pipe=selection.manifest_main_pipe)
diff --git a/api/pipelex_api/routes/pipelex/pipeline.py b/api/pipelex_api/routes/pipelex/pipeline.py
index 431d48030..cfbe4c43c 100644
--- a/api/pipelex_api/routes/pipelex/pipeline.py
+++ b/api/pipelex_api/routes/pipelex/pipeline.py
@@ -3,7 +3,7 @@ from __future__ import annotations
import hashlib
import hmac
from contextlib import ExitStack, contextmanager
-from pathlib import Path
+from pathlib import Path, PurePosixPath
from typing import TYPE_CHECKING, Annotated, Any, NamedTuple, cast
from fastapi import APIRouter, Depends, Request
@@ -14,11 +14,13 @@ from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.methods.structures_check import ensure_no_structured_content_python
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, StorageTarget, WebhookTarget
from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
+from pipelex.pipe_run.remote_dispatch import prepare_job_for_remote_dispatch
from pipelex.pipeline.pipeline_response import PipelexRunResultExecute, PipelexRunResultStart, RunState
from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup
from pipelex.pipeline.runner import PipelexMTHDSProtocol
from pipelex.reporting.usage_records import apply_tokens_usage_wire_shape
from pipelex.runtime_bridge.exceptions import MissingBundleValidatorError, MissingOrchestratorError
+from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE
from pipelex.runtime_bridge.primitives.hydration import hydrate_working_memory
from pipelex.runtime_hub import get_bundle_validator_registry, get_orchestrator_registry
from pipelex.system.environment import get_required_env
@@ -35,6 +37,7 @@ from pipelex_api.bundle import (
parse_bundle,
partition_bundle_entries,
)
+from pipelex_api.bundle_resources import decode_bundle_resources, files_its_steps_read
from pipelex_api.error_types import ErrorType
from pipelex_api.errors import raise_bad_request, raise_forbidden, raise_validation_error
from pipelex_api.json_body import decode_json_body
@@ -227,6 +230,18 @@ def _pipe_output_from_run_output(run_output: PipelexPipeRunOutput) -> PipeOutput
)
+async def _prepare_job_for_dispatch(*, pipe_job: PipeJob, orchestration_mode: str) -> PipeJob:
+ """The job as the mode's orchestrator receives it: prepared to leave this process for every mode but `direct`.
+
+ The files submitted with a bundle travel in the job's crate. An orchestrator other than the in-process `direct`
+ one, such as Temporal, runs the job outside this process, so the files its steps read only when they run move to
+ the run's storage first, dry runs included (`prepare_job_for_remote_dispatch`). A `direct` run keeps them inline.
+ """
+ if orchestration_mode == DIRECT_ORCHESTRATION_MODE:
+ return pipe_job
+ return await prepare_job_for_remote_dispatch(pipe_job=pipe_job)
+
+
class _OrchestratorPipeRun(PipeRunProtocol):
"""Adapts a mode-selected orchestrator to the `PipeRunProtocol` the base `execute` drives.
@@ -241,12 +256,14 @@ class _OrchestratorPipeRun(PipeRunProtocol):
synchronous and always drives this arm, so there is no delivery axis to thread here.
"""
- def __init__(self, *, orchestrator: OrchestratorProtocol) -> None:
+ def __init__(self, *, orchestrator: OrchestratorProtocol, orchestration_mode: str) -> None:
self._orchestrator = orchestrator
+ self._orchestration_mode = orchestration_mode
@override
async def run(self, pipe_job: PipeJob, *, delivery_assignment: DeliveryAssignment | None = None) -> PipeOutput:
- run_output = await self._orchestrator.execute(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+ dispatched_job = await _prepare_job_for_dispatch(pipe_job=pipe_job, orchestration_mode=self._orchestration_mode)
+ run_output = await self._orchestrator.execute(pipe_job=dispatched_job, delivery_assignment=delivery_assignment)
return _pipe_output_from_run_output(run_output)
@@ -313,7 +330,7 @@ class ApiRunner(PipelexMTHDSProtocol):
# PipeRun, then delegate to the base execute, which owns the full run lifecycle. The
# ApiRunner is constructed per request, so mutating _pipe_run here is request-scoped.
# `/execute` is synchronous, so it drives the orchestrator's BLOCKING `execute` arm.
- self._pipe_run = _OrchestratorPipeRun(orchestrator=orchestrator)
+ self._pipe_run = _OrchestratorPipeRun(orchestrator=orchestrator, orchestration_mode=orchestration_mode)
return await super().execute(
pipe_code=pipe_code,
mthds_contents=mthds_contents,
@@ -417,6 +434,8 @@ class ApiRunner(PipelexMTHDSProtocol):
extras=self.extras,
# The packages the request ships, which the base `execute` threads too.
methods_dirs=self.methods_dirs,
+ # The files submitted with the contents, such as a template file, which the base `execute` threads too.
+ bundle_resources=self.bundle_resources,
pipeline_run_id=pipeline_run_id,
request_id=request_id,
)
@@ -455,7 +474,8 @@ class ApiRunner(PipelexMTHDSProtocol):
# `PipelexPipeRunInput` (which carries no request_id / output_multiplicity /
# dynamic_output_concept_ref and skips run registration + telemetry). `start` genuinely
# enqueues the job and returns a `PipelexPipeDispatchAck` (ids only) immediately.
- dispatch_ack = await orchestrator.start(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+ dispatched_job = await _prepare_job_for_dispatch(pipe_job=pipe_job, orchestration_mode=orchestration_mode)
+ dispatch_ack = await orchestrator.start(pipe_job=dispatched_job, delivery_assignment=delivery_assignment)
return PipelexRunResultStart(
pipeline_run_id=resolved_pipeline_run_id,
@@ -517,6 +537,8 @@ class ApiRunner(PipelexMTHDSProtocol):
# done for the caller this runner was built for — the user and groups a run would state.
caller_identity=self.caller_identity,
graph_pipe_code=graph_pipe_code,
+ # The files submitted with the contents, which the validation's dry run reads, the template check included.
+ bundle_resources=self.bundle_resources,
)
# The core seam types its valid arm at the protocol-level ValidationReport (a leaf type)
# to stay import-acyclic in core; every registered validator in fact produces the canonical
@@ -701,6 +723,8 @@ class _BundleRunSource(NamedTuple):
mthds_contents: list[str] | None
library_dirs: list[str] | None
methods_dirs: list[Path] | None
+ bundle_files: dict[str, bytes]
+ """The files a method bundle ships beside its `.mthds` files that their steps read, keyed by the path a step names."""
def _ensure_vendored_packages_hold_no_structures(*, methods_dir: Path, packages: tuple[VendoredMethodPackage, ...]) -> None:
@@ -746,7 +770,7 @@ def _bundle_run_source(run_request: RunRequest) -> Generator[_BundleRunSource, N
rather than executing untrusted code.
"""
if run_request.bundle_b64 is None and run_request.files is None:
- yield _BundleRunSource(mthds_contents=run_request.mthds_contents, library_dirs=None, methods_dirs=None)
+ yield _BundleRunSource(mthds_contents=run_request.mthds_contents, library_dirs=None, methods_dirs=None, bundle_files={})
return
# Parse + guard in memory FIRST, then apply the sandbox-hosted gate BEFORE any disk write —
# a bundle destined for a 403 on a non-hosted deployment never touches the filesystem.
@@ -758,12 +782,13 @@ def _bundle_run_source(run_request: RunRequest) -> Generator[_BundleRunSource, N
# (`.py`, `requirements.txt`) → a temp `library_dirs` entry the load path source-captures; the packages it
# ships → a temp methods directory.
partition = partition_bundle_entries(parsed.entries)
- mthds_contents: list[str] = []
+ mthds_entries: list[tuple[PurePosixPath, str]] = []
for relpath, content in partition.mthds_entries:
try:
- mthds_contents.append(content.decode("utf-8"))
+ mthds_entries.append((relpath, content.decode("utf-8")))
except UnicodeDecodeError:
raise_validation_error(message=f"Bundle .mthds file '{relpath}' is not valid UTF-8.", error_type=ErrorType.INVALID_BUNDLE)
+ mthds_contents = [content for _, content in mthds_entries]
if not mthds_contents:
msg = "Method bundle contains no .mthds file."
if partition.vendored_packages:
@@ -772,6 +797,8 @@ def _bundle_run_source(run_request: RunRequest) -> Generator[_BundleRunSource, N
"packages it ships, and the bundle's own .mthds files go outside '.mthds/'."
)
raise_validation_error(message=msg, error_type=ErrorType.INVALID_BUNDLE)
+ # The `.mthds` files load as content strings, so a template file shipped beside one travels with it as a resource.
+ bundle_files = files_its_steps_read(mthds_entries=mthds_entries, other_entries=partition.library_entries)
with ExitStack() as stack:
library_dirs: list[str] | None = None
if partition.library_entries:
@@ -783,7 +810,7 @@ def _bundle_run_source(run_request: RunRequest) -> Generator[_BundleRunSource, N
if is_pipe_func_sandbox_hosted():
_ensure_vendored_packages_hold_no_structures(methods_dir=methods_dir, packages=partition.vendored_packages)
methods_dirs = [methods_dir]
- yield _BundleRunSource(mthds_contents=mthds_contents, library_dirs=library_dirs, methods_dirs=methods_dirs)
+ yield _BundleRunSource(mthds_contents=mthds_contents, library_dirs=library_dirs, methods_dirs=methods_dirs, bundle_files=bundle_files)
class _ResolvedRunSource(NamedTuple):
@@ -794,6 +821,8 @@ class _ResolvedRunSource(NamedTuple):
methods_dirs: list[Path] | None
pipe_code: str | None
method_provenance: MethodProvenance | None
+ bundle_files: dict[str, bytes]
+ """The files a method bundle ships that its steps read, which join the request's `resources`."""
@contextmanager
@@ -817,6 +846,7 @@ def _run_source(run_request: RunRequest) -> Generator[_ResolvedRunSource, None,
methods_dirs=fetched.methods_dirs,
pipe_code=run_request.pipe_code or fetched.main_pipe,
method_provenance=fetched.provenance,
+ bundle_files={},
)
return
with _bundle_run_source(run_request) as bundle_source:
@@ -826,6 +856,7 @@ def _run_source(run_request: RunRequest) -> Generator[_ResolvedRunSource, None,
methods_dirs=bundle_source.methods_dirs,
pipe_code=run_request.pipe_code,
method_provenance=None,
+ bundle_files=bundle_source.bundle_files,
)
@@ -878,6 +909,7 @@ async def execute(request: Request) -> JSONResponse:
extras=extras.analytics_groups,
library_dirs=source.library_dirs,
methods_dirs=source.methods_dirs,
+ bundle_resources=decode_bundle_resources(resources=run_request.resources, bundle_files=source.bundle_files),
)
response = await runner.execute(
pipe_code=source.pipe_code,
@@ -970,6 +1002,7 @@ async def start(
extras=extras.analytics_groups,
library_dirs=source.library_dirs,
methods_dirs=source.methods_dirs,
+ bundle_resources=decode_bundle_resources(resources=run_request.resources, bundle_files=source.bundle_files),
)
start_result = await runner.start(
pipe_code=source.pipe_code,
diff --git a/api/pipelex_api/routes/pipelex/validate.py b/api/pipelex_api/routes/pipelex/validate.py
index 7666f204d..01f307b79 100644
--- a/api/pipelex_api/routes/pipelex/validate.py
+++ b/api/pipelex_api/routes/pipelex/validate.py
@@ -12,13 +12,14 @@ from pipelex.pipeline.validation_report import PipelexValidationReport
from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
from pydantic import BaseModel, Field, model_validator
+from pipelex_api.bundle_resources import decode_bundle_resources
from pipelex_api.errors import raise_validation_error
from pipelex_api.exception_handlers import problem_response_from_error_report
from pipelex_api.json_body import JsonBodyRoute
from pipelex_api.method_source import fetched_method_source
from pipelex_api.openapi_responses import PROBLEM_403_RUN_POLICY, PROBLEM_404_METHOD_PACKAGE
from pipelex_api.routes.pipelex.pipeline import ApiRunner, get_request_user_id
-from pipelex_api.schemas.models import CallerAnalyticsGroupsMixin, MthdsContentsRequest
+from pipelex_api.schemas.models import RESOURCES_FIELD_DESCRIPTION, CallerAnalyticsGroupsMixin, MthdsContentsRequest
router = APIRouter(tags=["validate"], route_class=JsonBodyRoute)
@@ -87,6 +88,7 @@ class ValidateRequest(MthdsContentsRequest, CallerAnalyticsGroupsMixin):
"at the owning file. Must match `mthds_contents` in length when present."
),
)
+ resources: dict[str, str] | None = Field(default=None, description=RESOURCES_FIELD_DESCRIPTION)
render: list[str] = Field(
default_factory=list,
description=(
@@ -124,6 +126,10 @@ class ValidateRequest(MthdsContentsRequest, CallerAnalyticsGroupsMixin):
if self.mthds_sources is not None and self.method_ref is not None:
msg = "mthds_sources accompanies inline mthds_contents; a method_ref's sources are the package's own file names"
raise ValueError(msg)
+ # `resources` are the files the inline contents' steps read; a `method_ref`'s files are the package's own.
+ if self.resources and self.method_ref is not None:
+ msg = "resources accompany inline mthds_contents; a method_ref's files are the package's own"
+ raise ValueError(msg)
# A caller-supplied length mismatch is a request-shape bug → caught here as a 422.
# Without this guard it reaches the runtime's `validate_bundle`, which treats the
# mismatch as an internal host error (500) — the wrong status for caller input.
@@ -396,7 +402,11 @@ async def validate_mthds(request: Request, request_data: ValidateRequest) -> JSO
validated_contents = request_data.mthds_contents
# Inline contents carry no manifest, so the closure's own declaration is the whole chain.
manifest_main_pipe = None
- verdict = await ApiRunner(user_id=user_id, extras=request_data.analytics_groups).validate_verdict(
+ verdict = await ApiRunner(
+ user_id=user_id,
+ extras=request_data.analytics_groups,
+ bundle_resources=decode_bundle_resources(resources=request_data.resources),
+ ).validate_verdict(
mthds_contents=request_data.mthds_contents,
mthds_sources=request_data.mthds_sources,
allow_signatures=request_data.allow_signatures,
diff --git a/api/pipelex_api/schemas/models.py b/api/pipelex_api/schemas/models.py
index 12df7f4ed..20d63e3e1 100644
--- a/api/pipelex_api/schemas/models.py
+++ b/api/pipelex_api/schemas/models.py
@@ -15,6 +15,7 @@ from urllib.parse import urlparse
from mthds.protocol.exceptions import PipelineRequestError
from mthds.protocol.pipe_output import VariableMultiplicity
from mthds.protocol.pipeline_inputs import PipelineInputs
+from pipelex.core.bundle_resources.bundle_resource import MAX_LOAD_STAGE_RESOURCES_BYTES
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.methods.fetching import MethodProvenance
from pipelex.pipeline.pipeline_response import PipelexRunResultExecute, PipelexRunResultStart
@@ -25,6 +26,9 @@ from pydantic import BaseModel, ConfigDict, Field, field_validator, model_valida
from pydantic.functional_validators import SkipValidation
from pipelex_api.limits import (
+ DEFAULT_MAX_RESOURCE_FILE_KIB,
+ DEFAULT_MAX_RESOURCES_PER_REQUEST,
+ DEFAULT_MAX_RESOURCES_TOTAL_KIB,
MAX_CALLBACK_URL_LEN,
MAX_CALLBACK_URLS,
MAX_METHOD_REF_LEN,
@@ -41,6 +45,21 @@ def _ensure_mthds_file_within_bytes_limit(content: str) -> None:
raise ValueError(msg)
+RESOURCES_FIELD_DESCRIPTION = (
+ "The files submitted with the bundle that its steps read, such as a PipeDocGen step's `template_file` (an "
+ "`.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file's bundle-relative path, exactly "
+ "as the step names it, to the file's bytes in standard base64. Every bundle of the request is read as sitting in "
+ "one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or "
+ "colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`. "
+ f"By default at most {DEFAULT_MAX_RESOURCES_PER_REQUEST} files, {DEFAULT_MAX_RESOURCE_FILE_KIB} KiB each and "
+ f"{DEFAULT_MAX_RESOURCES_TOTAL_KIB} KiB in all, decoded, counting the files a method bundle ships for its steps; a "
+ "deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. A file "
+ "both sent here and shipped in a method bundle must be the same. The `.html` template files, which travel inline with the "
+ f"method wherever it runs, may total {MAX_LOAD_STAGE_RESOURCES_BYTES // 1024} KiB at most, a fixed bound, or the request "
+ "is refused with a 413 `BundleResourcesTooLargeError`."
+)
+
+
class RunRequest(BaseModel):
"""Body of `POST /execute` — this server's typed request model.
@@ -83,6 +102,9 @@ class RunRequest(BaseModel):
by manifest identity); mutually exclusive with `mthds_contents` and with a
method bundle. `pipe_code` may accompany it to override the manifest's
`main_pipe`.
+ resources: The files the submitted bundle's steps read, such as a PipeDocGen
+ step's `template_file`, as `{bundle_relative_path: base64}`. Accompanies inline
+ `mthds_contents` or a method bundle, never a `method_ref`.
"""
model_config = ConfigDict(extra="allow")
@@ -127,6 +149,7 @@ class RunRequest(BaseModel):
"MTHDS concepts and sandboxed PipeFuncs, not in-process Python."
),
)
+ resources: dict[str, str] | None = Field(default=None, description=RESOURCES_FIELD_DESCRIPTION)
@model_validator(mode="before")
@classmethod
@@ -155,6 +178,15 @@ class RunRequest(BaseModel):
if has_method_ref and has_bundle:
msg = "method_ref and a method bundle (bundle_b64 / files) are mutually exclusive; send one or the other."
raise PipelineRequestError(msg)
+ # `resources` are the files the submitted bundle's steps read, so they accompany a bundle the request
+ # carries: inline contents or a method bundle. A fetched package's files are its own.
+ if values.get("resources"):
+ if has_method_ref:
+ msg = "resources accompany inline mthds_contents or a method bundle; a method_ref's files are the package's own."
+ raise PipelineRequestError(msg)
+ if not values.get("mthds_contents") and not values.get("mthds_content") and not has_bundle:
+ msg = "resources accompany the bundle whose steps read them: send them with mthds_contents or a method bundle."
+ raise PipelineRequestError(msg)
if values.get("pipe_code") is None and not values.get("mthds_contents") and not has_bundle and not has_method_ref:
msg = (
"pipe_code and mthds_contents cannot both be empty. Either: both are provided, or if there are no mthds_contents, "
@@ -231,6 +263,7 @@ class RunRequest(BaseModel):
bundle_b64=request_body.get("bundle_b64"),
files=request_body.get("files"),
method_ref=request_body.get("method_ref"),
+ resources=request_body.get("resources"),
)
@@ -565,11 +598,16 @@ class MthdsFilesRequest(BaseModel):
),
)
+ resources: dict[str, str] | None = Field(default=None, description=RESOURCES_FIELD_DESCRIPTION)
+
@model_validator(mode="after")
def _exactly_one_selector(self) -> Self:
if (self.files is None) == (self.method_ref is None):
msg = "provide exactly one of `files` or `method_ref`"
raise ValueError(msg)
+ if self.resources and self.method_ref is not None:
+ msg = "resources accompany inline `files`; a method_ref's files are the package's own"
+ raise ValueError(msg)
return self
diff --git a/api/tests/unit/test_bundle_resources_routes.py b/api/tests/unit/test_bundle_resources_routes.py
new file mode 100644
index 000000000..0c112a04e
--- /dev/null
+++ b/api/tests/unit/test_bundle_resources_routes.py
@@ -0,0 +1,408 @@
+"""The `resources` a request submits beside its `mthds_contents`: the files its steps read, keyed by the path a step names.
+
+Pins the wire across the routes that load inline contents: the files reach the step that reads them, a file no step
+names is refused as the request's fault, a path that cannot name a file inside the bundle is refused, and the count,
+per-file and total limits are enforced before anything loads, and the files read when the method loads within a bound of
+their own once it is parsed. A run handed to an orchestrator other than `direct` carries its office templates by
+reference to the run's storage, never as bytes in the job.
+
+The hermetic API environment has no document engine, so the step's engine is stubbed and its template check records
+the bytes it reads, which is the read the engine's checker would be handed.
+"""
+
+import base64
+import io
+import zipfile
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Any
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+from pipelex.cogt.doc_gen.doc_gen_setting import DocGenSetting
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.pipe_operators.doc_gen.pipe_doc_gen import PipeDocGen
+from pipelex.pipe_operators.doc_gen.template_file_resource import bundle_resource_of_template_file
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.plugins.orchestrator_registry import OrchestratorRegistry
+from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeRunOutput
+from pipelex.system.job_metadata import JobMetadata
+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
+from pytest_mock import MockerFixture
+
+from pipelex_api.api_config import ApiConfig
+from pipelex_api.exception_handlers import register_exception_handlers
+from pipelex_api.routes import router as api_router
+from tests.unit._constants import STUB_METHOD_ADDRESS
+
+_PIPELINE_NS = "pipelex_api.routes.pipelex.pipeline"
+_REMOTE_DISPATCH_NS = "pipelex.pipe_run.remote_dispatch"
+_RESOURCES_NS = "pipelex_api.bundle_resources"
+_LIBRARY_RESOURCES_NS = "pipelex.libraries.bundle_resources"
+
+_TEMPLATE_PATH = "templates/invoice.docx"
+_TEMPLATE_BYTES = b"PK\x03\x04 a docx template"
+
+_INVOICE_MTHDS = f"""
+domain = "invoicing"
+description = "Printing an invoice"
+main_pipe = "print_invoice"
+
+[concept.Invoice]
+description = "An invoice to print"
+
+[concept.Invoice.structure]
+number = {{ type = "text", description = "The invoice number", required = true }}
+
+[concept.InvoiceFile]
+description = "The invoice as a file"
+refines = "Document"
+
+[pipe.print_invoice]
+type = "PipeDocGen"
+description = "Print the invoice in Word"
+inputs = {{ invoice = "Invoice" }}
+output = "InvoiceFile"
+format = "docx"
+template_file = "{_TEMPLATE_PATH}"
+"""
+
+_INVOICE_INPUTS: dict[str, Any] = {"invoice": {"concept": "invoicing.Invoice", "content": {"number": "INV-1"}}}
+
+
+def _b64(data: bytes) -> str:
+ return base64.b64encode(data).decode("ascii")
+
+
+def _build_client() -> TestClient:
+ app = FastAPI()
+ app.include_router(api_router, prefix="/v1")
+ register_exception_handlers(app)
+ return TestClient(app)
+
+
+def _stub_the_document_engine(mocker: MockerFixture) -> list[bytes]:
+ """Resolve every PipeDocGen step to an engine, and record the template bytes each template check finds for its step.
+
+ The check looks the step's template file up in the library it runs in, where the files sent with the bundle are held.
+ """
+ read_templates: list[bytes] = []
+
+ async def _record_template_check(pipe: PipeDocGen, *, job_metadata: JobMetadata, working_memory: WorkingMemory) -> None: # noqa: ARG001
+ assert pipe.template_file is not None
+ resource = bundle_resource_of_template_file(template_file=pipe.template_file)
+ assert resource is not None, f"the library holds no file for the template file '{pipe.template_file}'"
+ read_templates.append(await resource.read_bytes(storage_provider=InMemoryStorageProvider()))
+
+ mocker.patch.object(PipeDocGen, "resolve_engine", return_value=DocGenSetting(model="stub-docx"))
+ mocker.patch.object(PipeDocGen, "check_template_file", autospec=True, side_effect=_record_template_check)
+ return read_templates
+
+
+class _RecordingTemporalStub:
+ """An async-capable orchestrator standing in for Temporal: it records the job it is handed and acks it."""
+
+ supports_fire_and_forget = True
+
+ def __init__(self) -> None:
+ self.pipe_jobs: list[PipeJob] = []
+
+ async def execute(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput:
+ msg = "/start drives the fire-and-forget `start` arm; `execute` must not be reached."
+ raise NotImplementedError(msg)
+
+ async def start(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeDispatchAck: # noqa: ARG002
+ self.pipe_jobs.append(pipe_job)
+ return PipelexPipeDispatchAck(pipeline_run_id=pipe_job.job_metadata.run_metadata.pipeline_run_id, workflow_id="wf-1")
+
+
+def _record_dispatches_to_a_temporal_like_orchestrator(mocker: MockerFixture) -> tuple[_RecordingTemporalStub, InMemoryStorageProvider]:
+ """Make Temporal the deployment's orchestrator, recording each job it is handed, over an in-memory run storage."""
+ storage_provider = InMemoryStorageProvider()
+ mocker.patch(f"{_REMOTE_DISPATCH_NS}.get_storage_provider", return_value=storage_provider)
+ mocker.patch(
+ f"{_PIPELINE_NS}.get_api_config", return_value=ApiConfig(orchestration_mode="temporal", allow_request_orchestration_mode_override=False)
+ )
+ orchestrator = _RecordingTemporalStub()
+ mocker.patch(f"{_PIPELINE_NS}.get_orchestrator_registry", return_value=OrchestratorRegistry({"temporal": orchestrator}))
+ return orchestrator, storage_provider
+
+
+def _zip_b64(files: Mapping[str, str | bytes]) -> str:
+ """A `bundle_b64` of the files, which, unlike a `files` map, can carry bytes that are not text."""
+ buffer = io.BytesIO()
+ with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive:
+ for name, content in files.items():
+ archive.writestr(name, content)
+ return base64.b64encode(buffer.getvalue()).decode("ascii")
+
+
+def _contents_body(*, route: str, resources: dict[str, str], mthds: str = _INVOICE_MTHDS) -> dict[str, Any]:
+ """The route's body carrying the bundle inline with the given resources: the crate routes take `files[]`."""
+ match route:
+ case "/v1/validate":
+ return {"mthds_contents": [mthds], "resources": resources}
+ case "/v1/execute":
+ return {"mthds_contents": [mthds], "inputs": _INVOICE_INPUTS, "resources": resources}
+ case "/v1/codegen":
+ return {"files": [{"content": mthds, "source": "invoicing.mthds"}], "kind": "types", "target": "ts-zod", "resources": resources}
+ case _:
+ return {"files": [{"content": mthds, "source": "invoicing.mthds"}], "resources": resources}
+
+
+def _assert_problem(response: Any, *, status: int, error_type: str) -> str:
+ assert response.status_code == status, response.text
+ assert response.headers["content-type"].startswith("application/problem+json")
+ problem = response.json()
+ assert problem["error_type"] == error_type, response.text
+ assert "is_valid" not in problem
+ detail: str = problem["detail"]
+ return detail
+
+
+class TestBundleResourcesRoutes:
+ def test_validate_hands_the_step_the_submitted_template_file(self, mocker: MockerFixture) -> None:
+ """The validation's dry run reads the template from the submitted bytes, keyed by the path the step names."""
+ read_templates = _stub_the_document_engine(mocker)
+
+ response = _build_client().post(
+ "/v1/validate",
+ json={"mthds_contents": [_INVOICE_MTHDS], "resources": {f"./{_TEMPLATE_PATH}": _b64(_TEMPLATE_BYTES)}},
+ )
+
+ assert response.status_code == 200, response.text
+ assert response.json()["is_valid"] is True, response.text
+ # The validation dry-runs the step for its verdict and again for the graph it draws: every read is the submitted file.
+ assert read_templates
+ assert set(read_templates) == {_TEMPLATE_BYTES}
+
+ def test_validate_without_the_template_file_is_an_invalid_verdict_saying_it_was_not_supplied(self, mocker: MockerFixture) -> None:
+ _stub_the_document_engine(mocker)
+
+ response = _build_client().post("/v1/validate", json={"mthds_contents": [_INVOICE_MTHDS]})
+
+ assert response.status_code == 200, response.text
+ body = response.json()
+ assert body["is_valid"] is False
+ assert "which was not supplied with its bundle" in str(body["validation_errors"])
+
+ @pytest.mark.parametrize("route", ["/v1/validate", "/v1/execute", "/v1/pipe-io", "/v1/resolve", "/v1/codegen"])
+ def test_a_file_no_step_names_is_refused_on_every_route_that_loads_contents(self, mocker: MockerFixture, route: str) -> None:
+ _stub_the_document_engine(mocker)
+ body = _contents_body(route=route, resources={_TEMPLATE_PATH: _b64(_TEMPLATE_BYTES), "templates/logo.png": _b64(b"png")})
+
+ detail = _assert_problem(_build_client().post(route, json=body), status=422, error_type="BundleResourceNotNamedError")
+
+ assert "'templates/logo.png'" in detail
+
+ @pytest.mark.parametrize("route", ["/v1/validate", "/v1/execute", "/v1/pipe-io", "/v1/resolve", "/v1/codegen"])
+ def test_html_templates_over_the_load_stage_bound_are_refused_with_a_413_naming_the_limit(self, mocker: MockerFixture, route: str) -> None:
+ """An HTML template travels inline with the method wherever it runs, so the files read at load are bounded in total."""
+ _stub_the_document_engine(mocker)
+ mocker.patch(f"{_LIBRARY_RESOURCES_NS}.MAX_LOAD_STAGE_RESOURCES_BYTES", 1024)
+ html_mthds = _INVOICE_MTHDS.replace('format = "docx"', 'format = "pdf"').replace(_TEMPLATE_PATH, "invoice.html")
+ body = _contents_body(route=route, resources={"invoice.html": _b64(b"x" * 1025)}, mthds=html_mthds)
+
+ detail = _assert_problem(_build_client().post(route, json=body), status=413, error_type="BundleResourcesTooLargeError")
+
+ assert "('invoice.html')" in detail
+ assert "1 KiB limit" in detail
+
+ @pytest.mark.parametrize(
+ ("path", "reason"),
+ [
+ ("/etc/invoice.docx", "absolute"),
+ ("../invoice.docx", "'..' segment"),
+ ("templates\\invoice.docx", "backslash"),
+ ("C:invoice.docx", "colon"),
+ ],
+ )
+ def test_a_path_that_cannot_name_a_file_inside_the_bundle_is_refused(self, path: str, reason: str) -> None:
+ response = _build_client().post("/v1/validate", json={"mthds_contents": [_INVOICE_MTHDS], "resources": {path: _b64(_TEMPLATE_BYTES)}})
+
+ assert reason in _assert_problem(response, status=422, error_type="InvalidResourcePath")
+
+ @pytest.mark.parametrize("second_value", [pytest.param(_b64(b"b"), id="both_base64"), pytest.param("not base64!", id="second_not_base64")])
+ def test_two_spellings_of_one_file_are_refused_before_either_is_decoded(self, second_value: str) -> None:
+ response = _build_client().post(
+ "/v1/validate",
+ json={"mthds_contents": [_INVOICE_MTHDS], "resources": {_TEMPLATE_PATH: _b64(b"a"), f"./{_TEMPLATE_PATH}": second_value}},
+ )
+
+ assert "name the same file" in _assert_problem(response, status=422, error_type="InvalidResourcePath")
+
+ def test_a_value_that_is_not_base64_is_refused(self) -> None:
+ response = _build_client().post("/v1/validate", json={"mthds_contents": [_INVOICE_MTHDS], "resources": {_TEMPLATE_PATH: "not base64!"}})
+
+ _assert_problem(response, status=400, error_type="InvalidBase64")
+
+ @pytest.mark.parametrize(
+ ("limit_patches", "resources", "expected"),
+ [
+ pytest.param({"MAX_RESOURCES_PER_REQUEST": 1}, {"a.docx": _b64(b"a"), "b.docx": _b64(b"b")}, "1-file limit", id="count"),
+ pytest.param({"MAX_RESOURCE_FILE_BYTES": 2048, "_MAX_RESOURCE_B64_CHARS": 1024}, {"a.docx": _b64(bytes(1000))}, "per-file", id="encoded"),
+ pytest.param({"MAX_RESOURCE_FILE_BYTES": 1024}, {"a.docx": _b64(bytes(1025))}, "per-file", id="decoded"),
+ pytest.param({"MAX_RESOURCES_TOTAL_BYTES": 1024}, {"a.docx": _b64(bytes(600)), "b.docx": _b64(bytes(600))}, "total-size", id="total"),
+ ],
+ )
+ def test_the_limits_are_enforced_before_the_bundle_loads(
+ self, mocker: MockerFixture, limit_patches: dict[str, int], resources: dict[str, str], expected: str
+ ) -> None:
+ for name, value in limit_patches.items():
+ mocker.patch(f"{_RESOURCES_NS}.{name}", value)
+
+ response = _build_client().post("/v1/validate", json={"mthds_contents": [_INVOICE_MTHDS], "resources": resources})
+
+ assert expected in _assert_problem(response, status=413, error_type="PayloadTooLarge")
+
+ @pytest.mark.parametrize("route", ["/v1/validate", "/v1/execute", "/v1/pipe-io", "/v1/resolve", "/v1/codegen"])
+ def test_resources_beside_a_method_ref_are_refused(self, route: str) -> None:
+ """A published method's files are its package's own, so a caller cannot submit them beside its reference."""
+ body = _contents_body(route=route, resources={_TEMPLATE_PATH: _b64(_TEMPLATE_BYTES)})
+ body.pop("mthds_contents", None)
+ body.pop("files", None)
+ body["method_ref"] = f"{STUB_METHOD_ADDRESS}@v0.1.0"
+
+ detail = _assert_problem(_build_client().post(route, json=body), status=422, error_type="ValidationError")
+
+ assert "method_ref" in detail
+
+ @pytest.mark.parametrize("route", ["/v1/validate", "/v1/pipe-io"])
+ def test_a_source_label_naming_a_file_on_the_runner_never_reads_beside_it(self, mocker: MockerFixture, tmp_path: Path, route: str) -> None:
+ """A caller's absolute source naming a real bundle file, with the template file beside it, reaches nothing on the runner's disk."""
+ read_templates = _stub_the_document_engine(mocker)
+ bundle_path = tmp_path / "invoicing.mthds"
+ bundle_path.write_text(_INVOICE_MTHDS, encoding="utf-8")
+ (tmp_path / "templates").mkdir()
+ (tmp_path / "templates" / "invoice.docx").write_bytes(_TEMPLATE_BYTES)
+ body: dict[str, Any] = (
+ {"mthds_contents": [_INVOICE_MTHDS], "mthds_sources": [str(bundle_path)]}
+ if route == "/v1/validate"
+ else {"files": [{"content": _INVOICE_MTHDS, "source": str(bundle_path)}]}
+ )
+
+ response = _build_client().post(route, json=body)
+
+ assert response.status_code == 200, response.text
+ verdict = response.json()
+ assert verdict["is_valid"] is False
+ assert "which was not supplied with its bundle" in str(verdict["validation_errors"])
+ assert read_templates == []
+
+ def test_resources_with_no_bundle_to_read_them_are_refused(self) -> None:
+ response = _build_client().post("/v1/execute", json={"pipe_code": "print_invoice", "resources": {_TEMPLATE_PATH: _b64(_TEMPLATE_BYTES)}})
+
+ assert "resources accompany the bundle" in _assert_problem(response, status=422, error_type="ValidationError")
+
+ def test_a_run_handed_to_a_remote_orchestrator_carries_its_template_by_reference(self, mocker: MockerFixture) -> None:
+ """`/start` on a Temporal-like orchestrator: the job's crate names the stored file, and the run's storage holds its bytes."""
+ _stub_the_document_engine(mocker)
+ orchestrator, storage_provider = _record_dispatches_to_a_temporal_like_orchestrator(mocker)
+
+ response = _build_client().post(
+ "/v1/start",
+ json={"mthds_contents": [_INVOICE_MTHDS], "inputs": _INVOICE_INPUTS, "resources": {_TEMPLATE_PATH: _b64(_TEMPLATE_BYTES)}},
+ )
+
+ assert response.status_code == 202, response.text
+ (pipe_job,) = orchestrator.pipe_jobs
+ assert pipe_job.library_crate is not None
+ stored = pipe_job.library_crate.bundle_resources[_TEMPLATE_PATH]
+ assert stored.content_b64 is None
+ assert stored.uri is not None
+ assert stored.uri.startswith(f"pipelex-storage://{pipe_job.job_metadata.run_metadata.storage_scope}/assets/bundle/")
+ assert storage_provider.root[stored.uri.removeprefix("pipelex-storage://")] == _TEMPLATE_BYTES
+ assert _b64(_TEMPLATE_BYTES) not in pipe_job.model_dump_json()
+
+ @pytest.mark.parametrize(
+ ("bundle_files", "resources"),
+ [
+ pytest.param({"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES}, None, id="beside_the_bundle"),
+ pytest.param({"methods/invoicing.mthds": _INVOICE_MTHDS, f"methods/{_TEMPLATE_PATH}": _TEMPLATE_BYTES}, None, id="in_a_directory"),
+ pytest.param(
+ {"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES}, {_TEMPLATE_PATH: _b64(_TEMPLATE_BYTES)}, id="also_in_resources"
+ ),
+ ],
+ )
+ def test_a_method_bundle_carries_the_office_template_it_ships_to_its_step(
+ self, mocker: MockerFixture, bundle_files: dict[str, str | bytes], resources: dict[str, str] | None
+ ) -> None:
+ """A template file shipped beside the bundle's `.mthds` file reaches its step as one sent in `resources` does, by reference."""
+ _stub_the_document_engine(mocker)
+ orchestrator, storage_provider = _record_dispatches_to_a_temporal_like_orchestrator(mocker)
+ body: dict[str, Any] = {"bundle_b64": _zip_b64(bundle_files), "inputs": _INVOICE_INPUTS}
+ if resources is not None:
+ body["resources"] = resources
+
+ response = _build_client().post("/v1/start", json=body)
+
+ assert response.status_code == 202, response.text
+ (pipe_job,) = orchestrator.pipe_jobs
+ assert pipe_job.library_crate is not None
+ assert set(pipe_job.library_crate.bundle_resources) == {_TEMPLATE_PATH}
+ stored = pipe_job.library_crate.bundle_resources[_TEMPLATE_PATH]
+ assert stored.uri is not None
+ assert storage_provider.root[stored.uri.removeprefix("pipelex-storage://")] == _TEMPLATE_BYTES
+
+ def test_a_method_bundle_s_files_map_carries_the_html_template_it_ships_inline(self, mocker: MockerFixture) -> None:
+ """An HTML template is read while the method loads, so it travels inline in the job, from a `files` map as from a zip."""
+ _stub_the_document_engine(mocker)
+ orchestrator, _ = _record_dispatches_to_a_temporal_like_orchestrator(mocker)
+ html_mthds = _INVOICE_MTHDS.replace('format = "docx"', 'format = "pdf"').replace(_TEMPLATE_PATH, "templates/invoice.html")
+ html_template = "<h1>{{ invoice.number }}</h1>"
+
+ response = _build_client().post(
+ "/v1/start",
+ json={"files": {"invoicing.mthds": html_mthds, "templates/invoice.html": html_template}, "inputs": _INVOICE_INPUTS},
+ )
+
+ assert response.status_code == 202, response.text
+ (pipe_job,) = orchestrator.pipe_jobs
+ assert pipe_job.library_crate is not None
+ assert pipe_job.library_crate.bundle_resources["templates/invoice.html"].inline_bytes() == html_template.encode("utf-8")
+
+ @pytest.mark.parametrize(
+ ("bundle_files", "resources"),
+ [
+ pytest.param(
+ {"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES}, {_TEMPLATE_PATH: _b64(b"another template")}, id="and_resources"
+ ),
+ pytest.param(
+ {
+ "a/invoicing.mthds": _INVOICE_MTHDS,
+ f"a/{_TEMPLATE_PATH}": _TEMPLATE_BYTES,
+ "b/invoicing.mthds": _INVOICE_MTHDS.replace('domain = "invoicing"', 'domain = "invoicing_b"').replace(
+ "invoicing.Invoice", "invoicing_b.Invoice"
+ ),
+ f"b/{_TEMPLATE_PATH}": b"another template",
+ },
+ None,
+ id="two_of_its_bundles",
+ ),
+ ],
+ )
+ def test_two_different_files_a_step_reads_under_one_path_are_refused(
+ self, mocker: MockerFixture, bundle_files: dict[str, str | bytes], resources: dict[str, str] | None
+ ) -> None:
+ """Every bundle of a request reads its files as if it sat in one directory, so one path holds one file."""
+ _stub_the_document_engine(mocker)
+ body: dict[str, Any] = {"bundle_b64": _zip_b64(bundle_files), "inputs": _INVOICE_INPUTS}
+ if resources is not None:
+ body["resources"] = resources
+
+ detail = _assert_problem(_build_client().post("/v1/execute", json=body), status=422, error_type="InvalidResourcePath")
+
+ assert f"'{_TEMPLATE_PATH}'" in detail
+
+ def test_a_template_a_method_bundle_ships_is_held_to_the_per_file_limit(self, mocker: MockerFixture) -> None:
+ _stub_the_document_engine(mocker)
+ mocker.patch(f"{_RESOURCES_NS}.MAX_RESOURCE_FILE_BYTES", len(_TEMPLATE_BYTES) - 1)
+
+ response = _build_client().post(
+ "/v1/execute",
+ json={"bundle_b64": _zip_b64({"invoicing.mthds": _INVOICE_MTHDS, _TEMPLATE_PATH: _TEMPLATE_BYTES}), "inputs": _INVOICE_INPUTS},
+ )
+
+ assert "per-file" in _assert_problem(response, status=413, error_type="PayloadTooLarge")
diff --git a/api/tests/unit/test_validate_caller_identity.py b/api/tests/unit/test_validate_caller_identity.py
index 2766d9765..0eab6beec 100644
--- a/api/tests/unit/test_validate_caller_identity.py
+++ b/api/tests/unit/test_validate_caller_identity.py
@@ -20,6 +20,7 @@ import pytest
from fastapi import FastAPI, Request, Response
from fastapi.testclient import TestClient
from pipelex.base_exceptions import ErrorReport, ValidationErrorCategory, ValidationErrorItem
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.pipeline.validate_in_process import validate_bundles_in_process
from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistry
from pipelex.system.caller_identity import CallerIdentity
@@ -64,6 +65,7 @@ class _RecordingBundleValidator:
library_dirs: Sequence[Path] | None, # noqa: ARG002
caller_identity: CallerIdentity | None,
graph_pipe_code: str | None, # noqa: ARG002
+ bundle_resources: BundleResources | None, # noqa: ARG002
) -> ErrorReport:
self.caller_identities.append(caller_identity)
return ErrorReport(
diff --git a/api/tests/unit/test_validate_dispatch.py b/api/tests/unit/test_validate_dispatch.py
index 24454ae3f..34d98df3b 100644
--- a/api/tests/unit/test_validate_dispatch.py
+++ b/api/tests/unit/test_validate_dispatch.py
@@ -18,6 +18,7 @@ import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pipelex.base_exceptions import ErrorDomain, ErrorReport, PipelexConfigError, ValidationErrorCategory, ValidationErrorItem
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistry
from pipelex.runtime_bridge.exceptions import MissingBundleValidatorError
from pipelex.system.caller_identity import CallerIdentity
@@ -49,6 +50,7 @@ class _StubBundleValidator:
library_dirs: Sequence[Path] | None,
caller_identity: CallerIdentity | None,
graph_pipe_code: str | None,
+ bundle_resources: BundleResources | None,
) -> ErrorReport:
self.calls.append(
{
@@ -58,6 +60,7 @@ class _StubBundleValidator:
"library_dirs": library_dirs,
"caller_identity": caller_identity,
"graph_pipe_code": graph_pipe_code,
+ "bundle_resources": bundle_resources,
}
)
if self._error is not None:
diff --git a/api/tests/unit/test_validate_graph_target.py b/api/tests/unit/test_validate_graph_target.py
index 2db29b62e..22e2155f4 100644
--- a/api/tests/unit/test_validate_graph_target.py
+++ b/api/tests/unit/test_validate_graph_target.py
@@ -15,6 +15,7 @@ from typing import Any
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pipelex.base_exceptions import ErrorReport, ValidationErrorCategory, ValidationErrorItem
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistry
from pipelex.system.caller_identity import CallerIdentity
from pytest_mock import MockerFixture
@@ -52,6 +53,7 @@ class _RecordingBundleValidator:
library_dirs: Sequence[Path] | None, # noqa: ARG002
caller_identity: CallerIdentity | None, # noqa: ARG002
graph_pipe_code: str | None,
+ bundle_resources: BundleResources | None, # noqa: ARG002
) -> ErrorReport:
self.graph_pipe_codes.append(graph_pipe_code)
return ErrorReport(
diff --git a/docs/api-server/codegen.md b/docs/api-server/codegen.md
index dd6adf4a4..ddbb7d3c9 100644
--- a/docs/api-server/codegen.md
+++ b/docs/api-server/codegen.md
@@ -15,6 +15,8 @@ Both endpoints accept the same closure selector — exactly one of:
Providing neither or both is a request-shape `422`.
+Beside inline `files`, both endpoints accept `resources`: the files the closure's steps read, such as a `PipeDocGen` step's `template_file`, keyed by the path the step names, each in standard base64, with the rules and refusals of [`/v1/validate`](pipe-validate.md#sending-the-files-the-steps-read-resources). A step that names a template file the request did not send cannot be built, so the closure is invalid. `resources` beside a `method_ref` is a request-shape `422`: a published method's files are its package's own.
+
## Resolve
**Endpoint:** `POST /v1/resolve`
diff --git a/docs/api-server/configuration.md b/docs/api-server/configuration.md
index 8288567d5..8668f71a5 100644
--- a/docs/api-server/configuration.md
+++ b/docs/api-server/configuration.md
@@ -41,6 +41,13 @@ PIPELEX_ENV=dev
# A JSON body may also nest its arrays and objects at most 128 levels deep. That bound is
# fixed, not configurable (see error-responses.md).
+# The files a request sends beside its bundle in `resources`, such as a PipeDocGen step's
+# template file (see pipe-validate.md → "Sending the files the steps read"). Over any of
+# them is a 413. Read at startup.
+# MAX_RESOURCES_PER_REQUEST=32 # files in one request
+# MAX_RESOURCE_FILE_KIB=5120 # decoded bytes per file (default 5 MiB)
+# MAX_RESOURCES_TOTAL_KIB=10240 # decoded bytes across the request (default 10 MiB)
+
# ── Running a method by address (`method_ref`) ────────────────────────────
# See pipe-run.md → "Running a method by address". All read at startup.
#
diff --git a/docs/api-server/error-responses.md b/docs/api-server/error-responses.md
index 174ad1f7c..26ea40195 100644
--- a/docs/api-server/error-responses.md
+++ b/docs/api-server/error-responses.md
@@ -46,7 +46,7 @@ When a bundle fails validation, the `ValidateBundleError` carries a `validation_
| `category` | The failure family — one of `blueprint_validation`, `pipe_factory`, `pipe_validation`, `dry_run`. |
| `message` | Human-readable description of this specific error. |
| `error_type` | Finer error subtype within the category, when the source error provides one. |
-| `source` | The owning file of the error, when the runtime could attribute it to one: on `blueprint_validation` and `pipe_validation` items, and on a `dry_run` item, where it is the file of the pipe whose dry run failed. On the in-memory submit path it is the matching `mthds_sources[i]` (see [Sourcing submitted files](pipe-validate.md)), so it is absent when the caller sent no sources. Absent on `pipe_factory` items and on the parse-level residual described below. Beside a server's own library directories, an item never names one of their files. |
+| `source` | The owning file of the error, when the runtime could attribute it to one: on `blueprint_validation` and `pipe_validation` items, and on a `dry_run` item, where it is the file of the pipe whose dry run failed. On the in-memory submit path it is the matching `mthds_sources[i]` (see [Sourcing submitted files](pipe-validate.md)), so it is absent when the caller sent no sources. Absent on `pipe_factory` items and on the parse-level residual described below. Beside a server's own library directories, an item never names one of their files, nor a template file one of their `PipeDocGen` steps names. |
| `pipe_code`, `concept_code`, `domain_code` | The pipe / concept / domain the error is about, when applicable. On a `dry_run` item they name the innermost pipe that failed, not the controller the failure passed through. |
| `field_path`, `field_name` | The offending field within the bundle, when the error localizes to one. |
| `variable_names`, `missing_concept_code`, `missing_pipe_code`, `declared_concepts` | Extra context for specific failure shapes (undefined variables, an unresolved concept, pipe or package reference, the set of concepts that were declared). On an `unresolved_package_dependency` item, `missing_pipe_code` is the reference as written, `<address>->domain.code`, so the package's address is the part before `->`. |
@@ -160,7 +160,7 @@ A few specific statuses bypass the domain mapping:
- **403** — authenticated but not authorized, or refused by deployment policy. The cases: a storage-ownership mismatch; `error_type = "OrchestrationModeOverrideForbidden"` — the request asked for an `orchestration_mode` this deployment does not allow overriding per request (`allow_request_orchestration_mode_override = false`); `error_type = "CustomCodeRequiresSandbox"` — the method (an uploaded bundle, or a fetched `method_ref` package) ships custom Python `.py` files and this deployment is not sandbox-hosted, so running caller-supplied code in-process is refused; `error_type = "MethodStructuresRefusedError"` — a fetched `method_ref` package declares in-process Python structure classes (`StructuredContent` subclasses), which hosted execution always refuses, sandbox or not — express the types as MTHDS concepts with inline structures instead (see [Pipe Run](pipe-run.md#running-a-method-by-address-method_ref)).
- **404** — `error_type = "MethodPackageNotFoundError"`: the `method_ref` repository was fetched, but no package in it matches the requested address by manifest identity. The `detail` lists the package addresses the repository does contain, so the caller can correct the selector.
- **409** — `error_type = "PipelineManagerAlreadyExistsError"`: the submitted `pipeline_run_id` is already registered for a run that is still in flight on this server. Completed and failed runs free their id, so this only fires for genuinely concurrent duplicates — resubmit after the in-flight run finishes, or pick a fresh id. Only `POST /v1/start` accepts a client-supplied `pipeline_run_id`, so only `/start` can produce it.
-- **413** — request body exceeds the configured size limit (`MAX_REQUEST_BODY_MIB`, 100 MiB by default).
+- **413** — request body exceeds the configured size limit (`MAX_REQUEST_BODY_MIB`, 100 MiB by default), or a part of it exceeds its own: a method bundle over `MAX_BUNDLE_FILES` or `MAX_BUNDLE_TOTAL_KIB`, or `resources` over `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` or `MAX_RESOURCES_TOTAL_KIB` (`error_type = "PayloadTooLarge"`).
- **429** — an upstream inference provider rate-limited the run. `Retry-After` is set when the originating error carries `provider_metadata.retry_after_seconds`. Only `POST /v1/execute` runs inference, so only `/execute` can produce it.
- **501** — a request shape the published contract accepts but this server cannot serve. `error_type = "AsyncExecutionNotEnabledError"`: this deployment does not provide async pipeline execution (`POST /v1/start`). `error_type = "MethodRefNotSupported"`: the request selected its closure by a **registry-form** `method_ref` — not a `github.com/...` address — and no server-side method registry resolves those yet. Address-form references are fetched and resolved server-side (see [Pipe Run](pipe-run.md#running-a-method-by-address-method_ref)); use one of those, or submit inline `files[]`. Both 501s are permanent under the current deployment — do not retry.
@@ -174,6 +174,8 @@ A pipe selection the server cannot make is an input **422** named for its failur
The run routes' extension fields are validated before any method is fetched or any library loaded, and a failure there is a **422** whose `error_type` names the field that failed: `InvalidCallbackUrls` for `callback_urls`, `InvalidStorageScope` for `storage_scope`, `InvalidReadScope` for `read_scope`, `InvalidAnalyticsGroups` for `analytics_groups` (see [Pipe Run](pipe-run.md#host-supplied-run-context-storage_scope-and-analytics_groups)). A storage scope that does not lie under the run's read scope, whether either was sent or defaulted, is an `InvalidReadScope` as well. A failure on another extension field, or on more than one at once, carries the generic `ValidationError`, and the `detail` names every field that failed. `POST /v1/validate` takes `analytics_groups` too and classifies it the same way: a body failing on that field alone answers `InvalidAnalyticsGroups`, and one failing on it and another field answers `ValidationError`.
+The files a request sends in `resources` are checked before the bundle loads and again once it is parsed (see [Pipe Validate](pipe-validate.md#sending-the-files-the-steps-read-resources)). A key that cannot name a file inside the bundle is a **422** `InvalidResourcePath`, a value that is not base64 a **400** `InvalidBase64`, a file none of the bundle's steps names a **422** `BundleResourceNotNamedError`, and `.html` template files that total more than the 512 KiB the files read when the method loads may carry a **413** `BundleResourcesTooLargeError`, on every route, the diagnostic ones included: both are the request's fault, not the bundle's.
+
The HTTP status is the source of truth for success vs failure — there is no `success: true/false` field anywhere in the envelope.
Which statuses a given route can actually produce is documented per operation in the [committed OpenAPI artifact](openapi/pipelex-api.openapi.yaml), each as an `application/problem+json` `ProblemDocument`.
diff --git a/docs/api-server/openapi/pipelex-api.openapi.yaml b/docs/api-server/openapi/pipelex-api.openapi.yaml
index b29fc2b6b..d685eb33d 100644
--- a/docs/api-server/openapi/pipelex-api.openapi.yaml
+++ b/docs/api-server/openapi/pipelex-api.openapi.yaml
@@ -178,6 +178,23 @@ paths:
`pipe_code` may accompany it to override the manifest''s `main_pipe`. A package shipping custom Python
is only honored on a sandbox-hosted deployment, and a package declaring Python structure classes is always
refused — hosted execution accepts MTHDS concepts and sandboxed PipeFuncs, not in-process Python.'
+ resources:
+ anyOf:
+ - additionalProperties:
+ type: string
+ type: object
+ - type: 'null'
+ title: Resources
+ description: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`
+ (an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative
+ path, exactly as the step names it, to the file''s bytes in standard base64. Every bundle of the request
+ is read as sitting in one directory. A path must be relative and written with forward slashes, with no
+ `..` segment, backslash or colon. Every file must be named by a step, or the request is refused with a
+ 422 `BundleResourceNotNamedError`. By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded,
+ counting the files a method bundle ships for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,
+ `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. A file both sent here and shipped in a method bundle
+ must be the same. The `.html` template files, which travel inline with the method wherever it runs, may
+ total 512 KiB at most, a fixed bound, or the request is refused with a 413 `BundleResourcesTooLargeError`.'
orchestration_mode:
anyOf:
- type: string
@@ -434,6 +451,23 @@ paths:
`pipe_code` may accompany it to override the manifest''s `main_pipe`. A package shipping custom Python
is only honored on a sandbox-hosted deployment, and a package declaring Python structure classes is always
refused — hosted execution accepts MTHDS concepts and sandboxed PipeFuncs, not in-process Python.'
+ resources:
+ anyOf:
+ - additionalProperties:
+ type: string
+ type: object
+ - type: 'null'
+ title: Resources
+ description: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`
+ (an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative
+ path, exactly as the step names it, to the file''s bytes in standard base64. Every bundle of the request
+ is read as sitting in one directory. A path must be relative and written with forward slashes, with no
+ `..` segment, backslash or colon. Every file must be named by a step, or the request is refused with a
+ 422 `BundleResourceNotNamedError`. By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded,
+ counting the files a method bundle ships for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`,
+ `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`. A file both sent here and shipped in a method bundle
+ must be the same. The `.html` template files, which travel inline with the method wherever it runs, may
+ total 512 KiB at most, a fixed bound, or the request is refused with a 413 `BundleResourcesTooLargeError`.'
pipeline_run_id:
anyOf:
- type: string
@@ -1529,6 +1563,23 @@ components:
title: Method Ref
description: Reference to a published method, resolving to its package's `.mthds` files. Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`)
is fetched and resolved server-side; the registry form stays reserved (501 until a method registry exists).
+ resources:
+ anyOf:
+ - additionalProperties:
+ type: string
+ type: object
+ - type: 'null'
+ title: Resources
+ description: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`
+ (an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly
+ as the step names it, to the file''s bytes in standard base64. Every bundle of the request is read as sitting
+ in one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or
+ colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`.
+ By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded, counting the files a method bundle ships
+ for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.
+ A file both sent here and shipped in a method bundle must be the same. The `.html` template files, which travel
+ inline with the method wherever it runs, may total 512 KiB at most, a fixed bound, or the request is refused with
+ a 413 `BundleResourcesTooLargeError`.'
kind:
$ref: '#/components/schemas/CodegenRouteKind'
description: What to project. `types` projects the crate's concept set into typed models.
@@ -3897,6 +3948,23 @@ components:
title: Method Ref
description: Reference to a published method, resolving to its package's `.mthds` files. Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`)
is fetched and resolved server-side; the registry form stays reserved (501 until a method registry exists).
+ resources:
+ anyOf:
+ - additionalProperties:
+ type: string
+ type: object
+ - type: 'null'
+ title: Resources
+ description: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`
+ (an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly
+ as the step names it, to the file''s bytes in standard base64. Every bundle of the request is read as sitting
+ in one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or
+ colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`.
+ By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded, counting the files a method bundle ships
+ for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.
+ A file both sent here and shipped in a method bundle must be the same. The `.html` template files, which travel
+ inline with the method wherever it runs, may total 512 KiB at most, a fixed bound, or the request is refused with
+ a 413 `BundleResourcesTooLargeError`.'
type: object
title: MthdsFilesRequest
description: 'Shared closure selector for the resolve/codegen routes: inline `files[]` XOR a `method_ref`.
@@ -4734,8 +4802,9 @@ components:
out by itself (the auto-layout), for every format but `pptx`. `template` is an inline HTML and Jinja2 template, for
- `pdf` only; `template_file` is a path relative to the bundle''s file: an `.html` for `pdf`, an `.xlsx` for `xlsx`,
- a
+ `pdf` only; `template_file` is a path relative to the bundle''s file, inside its bundle and written with forward
+
+ slashes: an `.html` for `pdf`, an `.xlsx` for `xlsx`, a
`.docx` for `docx` and a `.pptx` for `pptx`, which `pptx` requires. `filename` is a Jinja expression over the
@@ -4950,6 +5019,23 @@ components:
title: Method Ref
description: Reference to a published method, resolving to its package's `.mthds` files. Address form (`github.com/<owner>/<repo>[/<selector>][@<tag>]`)
is fetched and resolved server-side; the registry form stays reserved (501 until a method registry exists).
+ resources:
+ anyOf:
+ - additionalProperties:
+ type: string
+ type: object
+ - type: 'null'
+ title: Resources
+ description: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`
+ (an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly
+ as the step names it, to the file''s bytes in standard base64. Every bundle of the request is read as sitting
+ in one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or
+ colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`.
+ By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded, counting the files a method bundle ships
+ for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.
+ A file both sent here and shipped in a method bundle must be the same. The `.html` template files, which travel
+ inline with the method wherever it runs, may total 512 KiB at most, a fixed bound, or the request is refused with
+ a 413 `BundleResourcesTooLargeError`.'
pipe_ref:
anyOf:
- type: string
@@ -7616,6 +7702,23 @@ components:
description: Optional per-file sources, parallel to `mthds_contents`. When provided, each entry is threaded onto
the corresponding bundle's `source` so server-side validation errors carry a `source` pointing at the owning file.
Must match `mthds_contents` in length when present.
+ resources:
+ anyOf:
+ - additionalProperties:
+ type: string
+ type: object
+ - type: 'null'
+ title: Resources
+ description: 'The files submitted with the bundle that its steps read, such as a PipeDocGen step''s `template_file`
+ (an `.xlsx`, `.docx`, `.pptx` or `.html` template): a JSON object mapping each file''s bundle-relative path, exactly
+ as the step names it, to the file''s bytes in standard base64. Every bundle of the request is read as sitting
+ in one directory. A path must be relative and written with forward slashes, with no `..` segment, backslash or
+ colon. Every file must be named by a step, or the request is refused with a 422 `BundleResourceNotNamedError`.
+ By default at most 32 files, 5120 KiB each and 10240 KiB in all, decoded, counting the files a method bundle ships
+ for its steps; a deployment sets its own with `MAX_RESOURCES_PER_REQUEST`, `MAX_RESOURCE_FILE_KIB` and `MAX_RESOURCES_TOTAL_KIB`.
+ A file both sent here and shipped in a method bundle must be the same. The `.html` template files, which travel
+ inline with the method wherever it runs, may total 512 KiB at most, a fixed bound, or the request is refused with
+ a 413 `BundleResourcesTooLargeError`.'
render:
items:
type: string
diff --git a/docs/api-server/pipe-io.md b/docs/api-server/pipe-io.md
index 9949c6f2a..a00c612f2 100644
--- a/docs/api-server/pipe-io.md
+++ b/docs/api-server/pipe-io.md
@@ -18,6 +18,7 @@ The body is the [crate routes' closure selector](codegen.md#selecting-the-closur
- `pipe_ref` (string, optional): the qualified ref `domain.pipe_code` of the pipe to describe. Omitted, the selection chain below decides.
- `all_pipes` (boolean, default `false`): describe every pipe the closure loads instead of the selected one.
- `include_files` (boolean, default `false`): echo the resolved closure's `.mthds` files on the valid arm.
+- `resources` (object, optional): the files the closure's steps read, such as a `PipeDocGen` step's `template_file`, keyed by the path the step names, each in standard base64, as on [`/v1/validate`](pipe-validate.md#sending-the-files-the-steps-read-resources). It accompanies `files` only. A string-loaded step whose template file was not sent makes the closure invalid.
There is no `views` field, because the valid arm always carries all three artifacts. The hosted API also accepts a catalog `method_id`, which the platform resolves into `files[]` before forwarding, so this server never declares it.
diff --git a/docs/api-server/pipe-run.md b/docs/api-server/pipe-run.md
index 6d5e11463..c1eb3c626 100644
--- a/docs/api-server/pipe-run.md
+++ b/docs/api-server/pipe-run.md
@@ -38,6 +38,7 @@ Execute a Pipelex pipeline with flexible inputs and wait for completion.
- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of a whole method bundle, which may carry the method packages it calls — see [Shipping a method bundle](#shipping-a-method-bundle-custom-pipefunc). Mutually exclusive with `files`.
- `files` (dict[str, str], optional): **Pipelex-API extension.** The same bundle as a `{relative_path: text}` map (the unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.
- `method_ref` (string, optional): **Pipelex-API extension.** Run a published method by address instead of inline source — see [Running a method by address](#running-a-method-by-address-method_ref). Mutually exclusive with `mthds_contents` and with a method bundle; `pipe_code` may accompany it to override the package's `main_pipe`.
+- `resources` (dict[str, str], optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](pipe-validate.md#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` or a method bundle; beside `method_ref`, or with no bundle at all, it is a 422. On an orchestrator that runs the job remotely, such as Temporal, each office template file is stored under the run's `storage_scope` (`assets/bundle/`) before the run is dispatched, so the job carries a reference rather than the bytes; the `.html` template files, which travel inline, are bounded to 512 KiB in total.
- `storage_scope` (string, optional): **Pipelex-API extension.** The prefix every object this run writes lands under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).
- `read_scope` (string, optional): **Pipelex-API extension.** The prefix every stored object this run reads must lie under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).
- `analytics_groups` (dict[str, str], optional): **Pipelex-API extension.** The groups this run's telemetry belongs to, such as `{"organization": "org_acme"}` — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).
@@ -118,6 +119,7 @@ Start a pipeline execution and get its `pipeline_run_id` back with a `202` ack.
- `bundle_b64` (string, optional): **Pipelex-API extension.** Base64-encoded zip of a whole method bundle, which may carry the method packages it calls — see [Shipping a method bundle](#shipping-a-method-bundle-custom-pipefunc). Mutually exclusive with `files`.
- `files` (dict[str, str], optional): **Pipelex-API extension.** The same bundle as a `{relative_path: text}` map (the unzipped equivalent of `bundle_b64`). Mutually exclusive with `bundle_b64`.
- `method_ref` (string, optional): **Pipelex-API extension.** Run a published method by address instead of inline source — see [Running a method by address](#running-a-method-by-address-method_ref). Mutually exclusive with `mthds_contents` and with a method bundle; `pipe_code` may accompany it to override the package's `main_pipe`.
+- `resources` (dict[str, str], optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](pipe-validate.md#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` or a method bundle; beside `method_ref`, or with no bundle at all, it is a 422. On an orchestrator that runs the job remotely, such as Temporal, each office template file is stored under the run's `storage_scope` (`assets/bundle/`) before the run is dispatched, so the job carries a reference rather than the bytes; the `.html` template files, which travel inline, are bounded to 512 KiB in total.
- `storage_scope` (string, optional): **Pipelex-API extension.** The prefix every object this run writes lands under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).
- `read_scope` (string, optional): **Pipelex-API extension.** The prefix every stored object this run reads must lie under — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).
- `analytics_groups` (dict[str, str], optional): **Pipelex-API extension.** The groups this run's telemetry belongs to, such as `{"organization": "org_acme"}` — see [Host-supplied run context](#host-supplied-run-context-storage_scope-and-analytics_groups).
@@ -228,6 +230,8 @@ The server materializes the bundle into a temporary library directory for the ru
}
```
+**Template files shipped in the bundle.** A file a step reads beside its bundle, such as a `PipeDocGen` step's `template_file`, can ship in the bundle, relative to the `.mthds` file whose step names it, as it sits beside a bundle on disk. The bundle's `.mthds` files load as content, so the server sends each such file with them as if it had come in [`resources`](pipe-validate.md#sending-the-files-the-steps-read-resources): it is counted against the same limits as `resources`, together with them, an office template file is stored with the run when the run goes to a remote orchestrator, and an `.html` one is bounded with the files read when the method loads. Every `.mthds` file of the bundle is read as sitting in one directory, so two different files a step reads under one path, shipped beside two `.mthds` files or shipped and also sent in `resources`, are a `422 InvalidResourcePath`; the same file in both is accepted.
+
**Ingest guards.** Bundles are bounded at ingest and rejected with a clear error (never a silent truncation):
- A hard **file-count** ceiling (`MAX_BUNDLE_FILES`) and a **total decompressed-size** ceiling (`MAX_BUNDLE_TOTAL_KIB`) → `413 PayloadTooLarge`. The zip path bounds actual decompression, so a zip bomb cannot expand past the ceiling, and an oversized `bundle_b64` is refused on its encoded length *before* it is decoded into memory.
diff --git a/docs/api-server/pipe-validate.md b/docs/api-server/pipe-validate.md
index 8a80d3aeb..fb78dfa54 100644
--- a/docs/api-server/pipe-validate.md
+++ b/docs/api-server/pipe-validate.md
@@ -20,6 +20,7 @@ Validate MTHDS content by parsing, loading, and dry-running pipes without execut
- `method_ref` (string, optional): **Pipelex-API extension.** Validate a published method by address — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — resolved by this server through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the repository is fetched at the tag, the package located by manifest identity, and its `.mthds` files validated with their **real relative paths** feeding the per-file sources (so `validation_errors[].source` names the owning file). The method packages the package ships under its own `.mthds/methods/<name>/` are not part of the validated content, and validation does not search them: a call into one resolves here through the server's installed copies and fetch-on-miss only, while a [run](pipe-run.md#shipping-the-method-packages-the-bundle-calls) uses the shipped copy first. Exactly one of `mthds_contents` / `method_ref`; `mthds_sources` cannot accompany it
- `allow_signatures` (boolean, optional, default `false`): controls only the **sweep mechanics** for `PipeSignature` placeholders — whether signature pipes are mock-run during the dry-run sweep and therefore listed in `validated_pipes`. It does **not** change the verdict: an unimplemented signature is never a rejection, it is a *runnability fact* reported via `pending_signatures` + `is_runnable` in both modes
- `mthds_sources` (list[str] | null, optional): per-file sources, parallel to `mthds_contents` — see [Sourcing submitted files](#sourcing-submitted-files). When present it must match `mthds_contents` in length (a mismatch is a 422 request error)
+- `resources` (object | null, optional): **Pipelex-API extension.** The files the bundle's steps read, such as a `PipeDocGen` step's `template_file`, as a map from the path the step names to the file's bytes in standard base64 — see [Sending the files the steps read](#sending-the-files-the-steps-read-resources). It accompanies `mthds_contents` only; beside `method_ref` it is a 422
- `render` (list[str], optional, default `[]`): opt-in *rendered text* to attach to the verdict — see [Opt-in extras](#opt-in-extras-render-and-views)
- `views` (list[str], optional, default `[]`): opt-in *structured views* to attach to the verdict — see [Opt-in extras](#opt-in-extras-render-and-views)
- `orchestration_mode` (string | null, optional): per-request backend override for the validation dispatch — see [Where validation runs](#where-validation-runs). Honored only when the deployment allows it; a forbidden override is a 403
@@ -153,7 +154,26 @@ Neither axis is part of the verdict contract: a machine consumer branches on the
## Sourcing submitted files
-The submit path carries bundle text, not file paths, so by default the runtime cannot tell the client which file an error belongs to — `source` comes back `null`. Send `mthds_sources` parallel to `mthds_contents` to fix this: each source is the logical identity of that content (e.g. the file's path relative to the submitted directory), and the runtime threads it onto the corresponding `blueprint.source`. The source then rides back on both arms — `bundle_blueprint.source` on the valid arm, and `validation_errors[].source` on the invalid arm — so a multi-file editor client can map a cross-file diagnostic to the file that owns it. Omit `mthds_sources` (or send `null`) and behavior is exactly as before. The list, when present, must be the same length as `mthds_contents`; a mismatch is a request-shape 422 (it is the caller's wiring bug, caught before the validation sweep runs).
+The submit path carries bundle text, not file paths, so by default the runtime cannot tell the client which file an error belongs to — `source` comes back `null`. Send `mthds_sources` parallel to `mthds_contents` to fix this: each source is the logical identity of that content (e.g. the file's path relative to the submitted directory), and the runtime threads it onto the corresponding `blueprint.source`. The source then rides back on both arms — `bundle_blueprint.source` on the valid arm, and `validation_errors[].source` on the invalid arm — so a multi-file editor client can map a cross-file diagnostic to the file that owns it. Omit `mthds_sources` (or send `null`) and behavior is exactly as before. The list, when present, must be the same length as `mthds_contents`; a mismatch is a request-shape 422 (it is the caller's wiring bug, caught before the validation sweep runs). A source is a label for diagnostics and nothing more: even one naming a real file on the server is never a directory the server reads from, so a step's template file comes from [`resources`](#sending-the-files-the-steps-read-resources) alone, and an address-based reference is not searched for above it.
+
+## Sending the files the steps read (`resources`)
+
+A bundle sent as content has no directory, so a file one of its steps reads from beside the bundle, such as a `PipeDocGen` step's `template_file`, is sent with it in `resources`. The same field, with the same rules, is accepted by [`/v1/execute` and `/v1/start`](pipe-run.md), [`/v1/pipe-io`](pipe-io.md), and [`/v1/resolve` and `/v1/codegen`](codegen.md#selecting-the-closure).
+
+```json
+{
+ "mthds_contents": ["domain = \"invoicing\"\n...\n[pipe.print_invoice]\ntype = \"PipeDocGen\"\n...\nformat = \"docx\"\ntemplate_file = \"templates/invoice.docx\""],
+ "resources": {"templates/invoice.docx": "UEsDBBQABgAIAAAAIQ..."}
+}
+```
+
+- **Each key is the path a step names**, relative to the bundle and written with forward slashes, exactly as in its `template_file`. `.` segments and repeated slashes collapse, so `./templates/invoice.docx` names the same file; every bundle of the request is read as sitting in one directory. An empty or absolute path, a `..` segment, a backslash, a colon, or two keys naming the same file is a **422** `InvalidResourcePath`, refused before any value is decoded. A method bundle sent to `/v1/execute` or `/v1/start` may ship the file instead, beside the `.mthds` file whose step names it, and a file both shipped and sent must be the same (see [Pipe Run](pipe-run.md#shipping-a-method-bundle-custom-pipefunc)).
+- **Each value is the file's bytes** in standard base64 (RFC 4648, with padding). A value that does not decode is a **400** `InvalidBase64`.
+- **Every file must be one a step names.** A file no step of the bundle reads is a **422** `BundleResourceNotNamedError`, whose `detail` lists the unnamed paths: a misspelled key fails loudly rather than leaving its step without its file. This is a request error, never an `is_valid: false` verdict.
+- **The sizes are bounded**: at most `MAX_RESOURCES_PER_REQUEST` files (32 by default), each at most `MAX_RESOURCE_FILE_KIB` once decoded (5 MiB by default), and at most `MAX_RESOURCES_TOTAL_KIB` across the request (10 MiB by default), counting the files a method bundle ships for its steps. A request over any of them is a **413** `PayloadTooLarge`, an oversized file refused on its encoded length before it is decoded. The whole body stays under `MAX_REQUEST_BODY_MIB` as well.
+- **The files read when the method loads are bounded too**: the `.html` template files, which travel inline with the method wherever it runs, a remote orchestrator's payloads included, may total 512 KiB at most once decoded. That bound is fixed, sized well under Temporal's 2 MB payload limit, and checked once the bundle is parsed, since only its steps say which files those are: a request over it is a **413** `BundleResourcesTooLargeError` naming the files and the limit. An office template file is not counted, since a run handed to a remote orchestrator carries it by reference to the run's storage.
+
+The files are checked as files on disk are. An `.html` template file is checked against the step's inputs when the bundle loads, and the document engine's template check runs on an office template file in the validation's dry run, where its errors are validation errors on the step. A step whose `template_file` was not sent is refused at load, and the verdict's `validation_errors[]` says the file was not supplied with its bundle and names the key it is expected under.
## Where validation runs
@@ -172,6 +192,7 @@ A validation is not a run, but it still emits telemetry: the sweep's `pipe_dry_r
Only conditions where the endpoint could not produce a verdict are non-2xx, rendered as [RFC 7807 problem documents](error-responses.md):
- **422** — a malformed request body, an `mthds_sources` / `mthds_contents` length mismatch, both/neither of `mthds_contents` / `method_ref`, or a malformed `analytics_groups` (`error_type` `InvalidAnalyticsGroups`) — request-shape errors caught before the runtime.
+- **422**, **413** and **400** on `resources` — a path that cannot name a file inside the bundle (`InvalidResourcePath`), a file no step names (`BundleResourceNotNamedError`), a file or a request over its size limit (`PayloadTooLarge`), `.html` template files over the load-stage bound (`BundleResourcesTooLargeError`), or a value that is not base64 (`InvalidBase64`) — see [Sending the files the steps read](#sending-the-files-the-steps-read-resources).
- **401 / 403** — unauthenticated / forbidden (including a per-request `orchestration_mode` override the deployment does not allow).
- **`method_ref` resolution failures** — a selector that could not be resolved produced no verdict, so it is never rendered as `is_valid: false`: a malformed reference or a failed fetch is a **422**, no matching package in the repository a **404**, and the custom-Python policy (the sandbox gate, the structures refusal) a **403** — the same statuses and `error_type`s as on the run routes (see the [error table](pipe-run.md#running-a-method-by-address-method_ref)).
- **5xx** — a server fault (including a host-wiring programmer error, surfaced as `PipelexUnexpectedError`).
diff --git a/docs/building-methods/pipes/executing-pipelines.md b/docs/building-methods/pipes/executing-pipelines.md
index adf13fc13..13c480b55 100644
--- a/docs/building-methods/pipes/executing-pipelines.md
+++ b/docs/building-methods/pipes/executing-pipelines.md
@@ -92,7 +92,7 @@ When using `PipelexMTHDSProtocol`, you can control library behavior with these p
- **`mthds_contents`**: When provided to `PipelexMTHDSProtocol.execute()`, Pipelex parses these strings and loads them into the library in addition to any resolved library directories. To run with only this content, disable directory loading explicitly (e.g. `library_dirs=[]`). This is useful for dynamic pipeline execution without file-based definitions.
-- **`bundle_uris`**: Optional file paths identifying the bundles passed as `mthds_contents` (matched by position). When a URI matches a bundle already loaded from the library directories, that content is skipped instead of being loaded twice. Without `bundle_uris`, every `mthds_contents` entry is loaded unconditionally — so passing content that duplicates a directory-loaded bundle raises a duplicate-declaration error.
+- **`bundle_uris`**: Optional file paths identifying the bundles passed as `mthds_contents` (matched by position). When a URI matches a bundle already loaded from the library directories, that content is skipped instead of being loaded twice. Without `bundle_uris`, every `mthds_contents` entry is loaded unconditionally — so passing content that duplicates a directory-loaded bundle raises a duplicate-declaration error. A content whose URI names a file on this machine is loaded as read from that file, so its steps read the files beside it, such as a `PipeDocGen` step's `template_file`, and an address-based reference is looked for in a `.mthds/methods/` directory above it, exactly as when its directory loads; without `bundle_uris`, a content string has no directory, and its template files are passed as `bundle_resources`.
!!! info "Python Structure Classes"
If your concepts use Python `StructuredContent` classes instead of inline structures, those Python files must be in the directories specified by `library_dirs`. Pipelex auto-discovers and registers these classes during library loading. Learn more about [Python StructuredContent Classes](../concepts/python-classes.md).
diff --git a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
index 4c08aeba9..5cc4c2b14 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
@@ -81,7 +81,7 @@ To print the same invoice on the plugin's `pipelex-pdf`, the step names it: `mod
| `format` | string | `pdf`, `xlsx`, `docx` or `pptx`. It sets the file's type and suffix. | Yes |
| `model` | string | The document engine that prints the file, a `doc_gen` model, alias or preset, such as `reportlab-pdf` or, with the plugin, `pipelex-pdf`. It defaults to the model deck's engine for the format and source. | No |
| `template` | string | An inline HTML and Jinja2 template, for `pdf` only. A `pdf` with a template is printed by the plugin. | No |
-| `template_file` | string | A template file beside the bundle, relative to the bundle's file: `.html` for `pdf`, `.xlsx`, `.docx` or `.pptx`. It excludes `template`. `pptx` requires one. | No |
+| `template_file` | string | A template file relative to the bundle's file, inside its bundle and written with forward slashes: `.html` for `pdf`, `.xlsx`, `.docx` or `.pptx`. An empty or absolute path, a `..` segment, a backslash and a colon are refused, the rule the files sent with a bundle are keyed by, so any template file can be sent with its bundle. Read beside the bundle when it is loaded from a directory, and from the files sent with it when it is loaded from a string. It excludes `template`. `pptx` requires one. | No |
| `filename` | string | A Jinja expression over the inputs for the file's name, without its suffix, such as `invoice-{{ invoice.number }}`. It defaults to the pipe's code. | No |
### An LLM report as a PDF
@@ -121,7 +121,37 @@ A `pdf` step's `template`, or its `.html` `template_file`, is rendered with exac
An `.xlsx`, `.docx` or `.pptx` template file is filled by its engine from the inputs as plain data, so it sees fields, not the sigils and filters of HTML templates. In plain data a native value is its value, and a verdict is its verdict: a `YesNo` is its boolean, a `Choice` its option key and a `Rating` its level, so the file reads `team` rather than `team.choice`, and a verdict's probability or confidence does not reach it. A `pdf` template reads the verdict whole, `team.choice` and `team.confidence` included, as a `PipeCompose` template does. Its contract is documented with the plugin. When the plugin's engine checks templates, `pipelex validate` and the dry run compare the file with the inputs' concepts: a misspelled name is reported before any run.
-A template file is found beside the bundle, so it works for a bundle loaded from a directory, a dependency package's included. A bundle loaded from a string, as the hosted API does, cannot name one: the step is refused at load. An inline `template` works on every load path.
+A bundle loaded from a directory, a dependency package's included, finds its template file beside its own file, and never among the files sent with a request. A bundle loaded from a string has no directory, so its template files come with it: the API's routes take them in the request's `resources`, each keyed by exactly the path its step names, and the Python entry points take them as `bundle_resources` (see [Template files sent with a bundle](#template-files-sent-with-a-bundle)). A string-loaded step whose file was not supplied is refused at load, saying so, whatever source label the bundle carries: only a bundle file the runtime read itself has a directory beside it, so a label naming a path, or a `source` written in the bundle's text, never makes the runtime read its own disk. An inline `template` works on every load path.
+
+### Template files sent with a bundle
+
+When a bundle is sent as content rather than read from a directory, as it is through the [API server](../../../api-server/pipe-validate.md), every template file a step reads travels beside it:
+
+```json
+{
+ "mthds_contents": ["domain = \"invoicing\"\n..."],
+ "resources": {"templates/invoice.docx": "UEsDBBQABgAIAAAAIQ..."}
+}
+```
+
+- **The key is the path the step names**, relative to the bundle and written with forward slashes: a step with `template_file = "templates/invoice.docx"` reads the file sent as `templates/invoice.docx`, and `./templates/invoice.docx` names the same file. Every bundle of the request is read as sitting in one directory. An absolute path, a `..` segment, a backslash and a colon are refused, as they are in a `template_file`.
+- **The value is the file's bytes** in standard base64.
+- **Every file sent must be one a step names.** A file no step reads is refused, so a misspelled key fails loudly rather than leaving its step without a template.
+- **The files are checked like files on disk.** An `.html` template file is checked against the inputs when the method loads, and the engine's template check runs on an office template file at validation and in the dry run.
+
+A run that a remote orchestrator such as Temporal executes, a dry run included, stores each office template file in the run's storage before the run leaves the process, so the run carries a reference rather than the bytes, and the step reads the file back under the run's read scope, to print it or to check it. An HTML template file travels with the method, since it is read while the method loads, so the HTML template files sent with a bundle may total 512 KiB at most: keep images and fonts out of them, at URLs they reference.
+
+From Python, the same files are passed as `bundle_resources`, built with `make_bundle_resources`:
+
+```python
+from pathlib import Path
+
+from pipelex.core.bundle_resources.bundle_resource import make_bundle_resources
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+
+resources = make_bundle_resources(files={"templates/invoice.docx": Path("templates/invoice.docx").read_bytes()})
+result = await PipelexMTHDSProtocol(bundle_resources=resources).execute(mthds_contents=[bundle_text], inputs=inputs)
+```
## Running and iterating
diff --git a/docs/contribute/hub-layering.md b/docs/contribute/hub-layering.md
index 56e9adc90..c305e09cf 100644
--- a/docs/contribute/hub-layering.md
+++ b/docs/contribute/hub-layering.md
@@ -62,7 +62,7 @@ There is deliberately **no** `pipelex.hub`. It was deleted rather than kept as a
| current-library contextvar | `set_current_library`, `get_current_library`, `get_current_library_id_or_none`, `clear_current_library`, `scoped_current_library` |
| library dirs | `resolve_library_dirs`, `get_default_library_dirs` |
| pipe router | `get_pipe_router`, `set_pipe_router`, `teardown_current_pipe_router`, `scoped_pipe_router` |
-| run | `get_pipe_run`, `get_pipeline_manager`, `get_pipeline` |
+| run | `get_pipe_run`, `does_pipe_run_leave_process`, `get_pipeline_manager`, `get_pipeline` |
| pipe func | `get_pipe_func_executor`, `scoped_pipe_func_executor`, `get_pipe_func_executor_registry` |
Setters follow their getters onto the matching container.
diff --git a/docs/errors/authoring-and-language.md b/docs/errors/authoring-and-language.md
index a71452327..6e8b047ed 100644
--- a/docs/errors/authoring-and-language.md
+++ b/docs/errors/authoring-and-language.md
@@ -17,6 +17,10 @@ own page. Classes are grouped by subsystem.
## Core language
+- [`BundleResourceError`](bundle-resource-error.md) — Bundle resource
+- [`BundleResourceNotNamedError`](bundle-resource-not-named-error.md) — Bundle resource not named
+- [`BundleResourcePathError`](bundle-resource-path-error.md) — Bundle resource path
+- [`BundleResourcesTooLargeError`](bundle-resources-too-large-error.md) — Bundle resources too large
- [`ConceptCodeError`](concept-code-error.md) — Concept code
- [`ConceptError`](concept-error.md) — Concept error
- [`ConceptFactoryError`](concept-factory-error.md) — Concept factory
diff --git a/docs/errors/bundle-resource-error.md b/docs/errors/bundle-resource-error.md
new file mode 100644
index 000000000..446aaa425
--- /dev/null
+++ b/docs/errors/bundle-resource-error.md
@@ -0,0 +1,21 @@
+---
+title: "Bundle resource"
+description: "Reference for the `BundleResourceError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Bundle resource
+
+Raised when a bundle resource cannot be read: stored bytes that no longer match their digest, or a read the resource cannot serve.
+
+| Field | Value |
+|---|---|
+| `error_type` | `BundleResourceError` |
+| `title` | Bundle resource |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/bundle-resource-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.core.bundle_resources.exceptions` |
+| Parent class | [`PipelexError`](pipelex-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/errors/bundle-resource-not-named-error.md b/docs/errors/bundle-resource-not-named-error.md
new file mode 100644
index 000000000..928b0268d
--- /dev/null
+++ b/docs/errors/bundle-resource-not-named-error.md
@@ -0,0 +1,21 @@
+---
+title: "Bundle resource not named"
+description: "Reference for the `BundleResourceNotNamedError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Bundle resource not named
+
+Raised when a file submitted with a bundle is named by none of its steps.
+
+| Field | Value |
+|---|---|
+| `error_type` | `BundleResourceNotNamedError` |
+| `title` | Bundle resource not named |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/bundle-resource-not-named-error/` |
+| `error_domain` | `input` |
+| Defined in | `pipelex.core.bundle_resources.exceptions` |
+| Parent class | [`PipelexError`](pipelex-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/errors/bundle-resource-path-error.md b/docs/errors/bundle-resource-path-error.md
new file mode 100644
index 000000000..3514bfa9e
--- /dev/null
+++ b/docs/errors/bundle-resource-path-error.md
@@ -0,0 +1,21 @@
+---
+title: "Bundle resource path"
+description: "Reference for the `BundleResourcePathError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Bundle resource path
+
+Raised when a file submitted with a bundle is keyed by a path that cannot name a file inside the bundle.
+
+| Field | Value |
+|---|---|
+| `error_type` | `BundleResourcePathError` |
+| `title` | Bundle resource path |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/bundle-resource-path-error/` |
+| `error_domain` | `input` |
+| Defined in | `pipelex.core.bundle_resources.exceptions` |
+| Parent class | [`PipelexError`](pipelex-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/errors/bundle-resources-too-large-error.md b/docs/errors/bundle-resources-too-large-error.md
new file mode 100644
index 000000000..837a0a9a0
--- /dev/null
+++ b/docs/errors/bundle-resources-too-large-error.md
@@ -0,0 +1,21 @@
+---
+title: "Bundle resources too large"
+description: "Reference for the `BundleResourcesTooLargeError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Bundle resources too large
+
+Raised when the files a bundle's steps read when the method loads, such as HTML templates, are too large in total.
+
+| Field | Value |
+|---|---|
+| `error_type` | `BundleResourcesTooLargeError` |
+| `title` | Bundle resources too large |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/bundle-resources-too-large-error/` |
+| `error_domain` | `input` |
+| Defined in | `pipelex.core.bundle_resources.exceptions` |
+| Parent class | [`PipelexError`](pipelex-error.md) |
+
+[Back to Error Reference](index.md)
diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.md
index 5d0a6bbf9..f3c130b09 100644
--- a/docs/under-the-hood/distributed-content-generation.md
+++ b/docs/under-the-hood/distributed-content-generation.md
@@ -172,6 +172,8 @@ What gets stored vs. what crosses the boundary, by content type:
| Search structured | Nothing (JSON is small) | Raw `dict`, re-validated against the output class on the submitter |
| Judgment answers | Nothing (verdicts are small) | `dict[str, JudgmentAnswer]` keyed as the questions were |
+The traffic also runs the other way. A template file sent with a bundle loaded from content strings, such as a `PipeDocGen` step's `.docx` template, travels in the run's `LibraryCrate` as a `BundleResource`, keyed by the path the step names. Before a run, live or dry, is handed to an orchestrator that runs it outside the process, the submitter stores each file a step reads only when it runs under `{storage_scope}/assets/bundle/<sha256>/<name>` and replaces its bytes in the crate with the `pipelex-storage://` URI (`prepare_job_for_remote_dispatch` in `pipelex/pipe_run/remote_dispatch.py`, over `relocate_run_resources` in `pipelex/libraries/bundle_resources.py`). Every dispatcher calls it as the job leaves: `PipelexMTHDSProtocol.execute` when the hub's pipe run is a boot orchestrator's, which the interpreter hub records when the plugin claims the slot (`does_pipe_run_leave_process`), and the API server for every orchestration mode but `direct`. The worker's library holds the reference, the step's composition carries it, and the print stage reads the bytes back through the storage provider, checking them against their digest, after the read scope has authorized the URI like any other the composition declares; a dry run's template check authorizes it the same way before it reads. An `.html` template file stays inline, since it is read while the crate loads, where the workflow may do no IO, so the files read at load are bounded when the bundle loads: `MAX_LOAD_STAGE_RESOURCES_BYTES`, 512 KiB in total, well under Temporal's 2 MB payload limit, refused with `BundleResourcesTooLargeError`. An in-process run keeps every file inline and stores nothing, and so does a run whose storage scope is the dry-run placeholder.
+
Each host-runtime plugin dispatches these through its own activities and queue routing; for the Temporal realization (the `act_*` activity set, per-activity task queues), see our Temporal plugin's own docs.
### What a leaf may read: the read scope
diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.md
index af31959d4..b8190100c 100644
--- a/docs/under-the-hood/error-model.md
+++ b/docs/under-the-hood/error-model.md
@@ -89,7 +89,7 @@ The **structured-info invariant is total**: every invalid verdict carries a non-
Besides `category` and `message`, each item carries whatever identity fields its stage produced — `error_type`, `pipe_code`, `concept_code`, `domain_code`, `field_path`, `field_name`, `variable_names`, `missing_concept_code`, `declared_concepts` (on an `unresolved_concept` item, the bare codes of the concepts the validated bundle declares in the domain the reference was looked up in, native concepts excluded, and never those a library loaded before it, which may be a host's own), the TOML position `line` and `column`, the unknown-model locators `model_reference`, `model_type` and `suggestions`, the template-check locators `template_file` and `template_location` (on a `template_check_warning`, which rides `warnings`), and a `source` (the declaring file path, or the per-content source the in-memory load path was given) that hands a consumer the owning file for cross-file diagnostic placement. When the error has a deterministic remedy, the item also carries a [`suggested_fix`](#suggested_fix-structured-deterministic-fixes).
-**A verdict names no path on the host.** An item and the verdict's message are kept verbatim under STRICT disclosure, so neither names where the host keeps a file. A bundle of a package the method depends on by address is named by the package's address and the bundle's path inside the package, such as `github.com/acme/harbour-methods/tides/notices/tide_notices.mthds`, never by the directory the host installed the package in: that is the `source` of an item located inside the dependency and the file a duplicate declaration inside it names, on every surface, a local one included. When submitted content is validated or run, the library directories loaded beside it are the host's (installed libraries, the defaults, `PIPELEXPATH`) unless the caller says they are its own, so the verdict names none of their files: an item located in one carries no `source`, nor a `field_path` naming it, and is found by its `pipe_code` and `domain_code`, a warning found at a pipe of theirs carries no `template_file` (the sweep withholds it, `BundleValidator`'s `source_pipe_refs`), and a message that named one, or named the module a Python file of theirs was imported as (two `@pipe_func` functions of the same name, for instance), reads `<host library file>` in its place (`withholding_host_library_files` in `pipelex/pipeline/validate_bundle_translation.py`). A name only counts standing on its own, never inside a longer one, and a source the caller submitted is never withheld. A bundle validated from a file on the caller's own disk, and content validated or run against library directories that are the caller's (`library_dirs_are_callers`, an option of `validate_bundle`, of the in-process validator and of the local runtime `PipelexMTHDSProtocol`, whose `validate` and `execute` both read it), keep the full path of every file, since there the directories are the caller's own. `pipelex fix bundle --diff` leaves a dependency's address-named source as it is.
+**A verdict names no path on the host.** An item and the verdict's message are kept verbatim under STRICT disclosure, so neither names where the host keeps a file. A bundle of a package the method depends on by address is named by the package's address and the bundle's path inside the package, such as `github.com/acme/harbour-methods/tides/notices/tide_notices.mthds`, never by the directory the host installed the package in: that is the `source` of an item located inside the dependency and the file a duplicate declaration inside it names, on every surface, a local one included. When submitted content is validated or run, the library directories loaded beside it are the host's (installed libraries, the defaults, `PIPELEXPATH`) unless the caller says they are its own, so the verdict names none of their files: an item located in one carries no `source`, nor a `field_path` naming it, and is found by its `pipe_code` and `domain_code`, a warning found at a pipe of theirs carries no `template_file` (the sweep withholds it, `BundleValidator`'s `source_pipe_refs`), and a message that named one, named the template file a `PipeDocGen` step of theirs names (a step refused while they load, or one whose template the engine's check refuses in the dry run of a caller's pipe that runs it), or named the module a Python file of theirs was imported as (two `@pipe_func` functions of the same name, for instance), reads `<host library file>` in its place (`withholding_host_library_files` in `pipelex/pipeline/validate_bundle_translation.py`). A name only counts standing on its own, never inside a longer one, and neither a source the caller submitted nor a template file its own content names is ever withheld. A step can only name its template file relative to its bundle, so no message about one names a path on the host. A bundle validated from a file on the caller's own disk, and content validated or run against library directories that are the caller's (`library_dirs_are_callers`, an option of `validate_bundle`, of the in-process validator and of the local runtime `PipelexMTHDSProtocol`, whose `validate` and `execute` both read it), keep the full path of every file, since there the directories are the caller's own. `pipelex fix bundle --diff` leaves a dependency's address-named source as it is.
**Every refusal of the bundle is a verdict.** A validator answers either a verdict — valid, or invalid with located items — or *no verdict could be produced*, which is reserved for a failure of the tool or its environment. So a refusal raised while loading or validating a bundle becomes an item, never a no-verdict fault. After its class-specific arms, the shared cascade (`translate_to_validate_bundle_error` in `pipelex/pipeline/validate_bundle_translation.py`) turns any other `PipelexError` whose report is `input`-domained into a verdict with one item: `pipe_validation`, with the `pipe_code`, `domain_code` and `source`, when the library load located it on the pipe it was building, and `blueprint_validation` otherwise. That item carries no `error_type`, since a refusal with a closed code has an arm of its own. It keeps the refusal's message only when the refusal authored that message as caller-facing copy, and otherwise names the refusal's title, because the verdict as a whole is kept verbatim under STRICT disclosure. The location comes from the load loop in `LibraryManager.load_from_crate`, the one place that holds a pipe's code, domain and file when building it fails: it raises such a refusal again as a `PipeLoadRefusalError` naming the pipe and the file, `from` the original. A `config` or `runtime` fault, an unclassified one, a `SecurityError`, a `PipeNotFoundError` (which has its own not-found handler) and anything that is not a `PipelexError` still propagate as no verdict. The rule therefore reaches exactly the refusals whose class declares the `input` domain. The pipe factories' refusals do: `PipeLLMFactoryError`, `PipeComposeFactoryError`, `PipeExtractFactoryError`, `PipeConditionFactoryError` and `PipeParallelFactoryError` are raised only while a pipe is built, about its own blueprint, so each is `input`-domained and caller-facing, and a `PipeExtract` whose input is neither an image nor a document validates to one item on that pipe, saying which input to redeclare. `PipeImgGenFactoryError` is not among them: it is raised while an image-generation pipe runs, not while it is built.
diff --git a/pipelex/cogt/content_generation/doc_gen_generate.py b/pipelex/cogt/content_generation/doc_gen_generate.py
index aa13bcbb4..82d6be609 100644
--- a/pipelex/cogt/content_generation/doc_gen_generate.py
+++ b/pipelex/cogt/content_generation/doc_gen_generate.py
@@ -155,7 +155,7 @@ async def _print_document(*, render_assignment: RenderDocumentAssignment, storag
)
engine_name = inference_model.name
worker = DocGenWorkerFactory.make_doc_gen_worker(inference_model=inference_model, reporting_delegate=get_report_delegate())
- render_job = composition.make_render_job()
+ render_job = await composition.make_render_job(storage_provider=storage_provider)
resources = RunRenderResources(
storage_provider=storage_provider,
read_scope=render_assignment.job_metadata.run_metadata.read_scope,
diff --git a/pipelex/cogt/doc_gen/document_composition.py b/pipelex/cogt/doc_gen/document_composition.py
index 348c2273c..5e7be6cb7 100644
--- a/pipelex/cogt/doc_gen/document_composition.py
+++ b/pipelex/cogt/doc_gen/document_composition.py
@@ -6,6 +6,8 @@ from pydantic import BaseModel, ConfigDict, Field, model_validator
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource
from pipelex.cogt.doc_gen.layout_tree import LayoutDocument
from pipelex.cogt.doc_gen.render_job import RenderJob
+from pipelex.core.bundle_resources.bundle_resource import BundleResource
+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
from pipelex.tools.uri.uri_read_scope import UriReference
@@ -14,7 +16,10 @@ class DocumentComposition(BaseModel):
The compose stage is pure, so it can run where the pipe runs, inside a workflow; this is what crosses to
the print stage, which reads the template file and prints. It differs from the engine's `RenderJob` by
- naming the template file by its path rather than carrying its bytes: reading a file is the print stage's IO.
+ naming the template file rather than carrying its bytes: reading a file is the print stage's IO. A template
+ file beside a bundle loaded from disk is named by its path; one submitted with a bundle loaded from content
+ strings is its bundle resource, inline in a run that stays in its process and a reference to the run's storage
+ in one handed to a remote orchestrator, so the bytes never ride in the composition across a workflow boundary.
"""
model_config = ConfigDict(extra="forbid")
@@ -26,6 +31,9 @@ class DocumentComposition(BaseModel):
layout: LayoutDocument | None = None
html: str | None = None
template_path: str | None = Field(default=None, description="The template file's resolved path, for the print stage to read")
+ template_resource: BundleResource | None = Field(
+ default=None, description="The template file submitted with the bundle, for the print stage to read"
+ )
template_name: str | None = Field(default=None, description="The template file as the method names it")
data: dict[str, Any] = Field(default_factory=dict)
@@ -36,7 +44,7 @@ class DocumentComposition(BaseModel):
case DocGenSource.HTML:
return self.html is not None
case DocGenSource.TEMPLATE_FILE:
- return self.template_path is not None
+ return self.template_path is not None or self.template_resource is not None
@model_validator(mode="after")
def validate_payload(self) -> Self:
@@ -49,26 +57,39 @@ class DocumentComposition(BaseModel):
if extra_payloads:
msg = f"A document composed {self.source.desc} carries only its own payload, not the {', '.join(extra_payloads)} one."
raise ValueError(msg)
+ if self.template_path is not None and self.template_resource is not None:
+ msg = "A template file is read from disk or submitted with its bundle, not both."
+ raise ValueError(msg)
return self
def referenced_uris(self) -> list[UriReference]:
"""The files the print stage will read for the document, which it authorizes against the run's read scope first.
- The images of the layout tree. Composed HTML may name files too, but only an engine laying it out finds
- them, and it reads each through `RenderResources`, which applies the same check.
+ The images of the layout tree, and a template file submitted with the bundle and stored with the run.
+ Composed HTML may name files too, but only an engine laying it out finds them, and it reads each through
+ `RenderResources`, which applies the same check.
"""
- if self.layout is None:
- return []
- return [
- UriReference(uri=image_url, position=f"image {index} of the document '{self.filename}'")
- for index, image_url in enumerate(self.layout.image_urls(), start=1)
- ]
+ references: list[UriReference] = []
+ if self.template_resource is not None and self.template_resource.uri is not None:
+ references.append(UriReference(uri=self.template_resource.uri, position=f"the template file '{self.template_name}'"))
+ if self.layout is not None:
+ references.extend(
+ UriReference(uri=image_url, position=f"image {index} of the document '{self.filename}'")
+ for index, image_url in enumerate(self.layout.image_urls(), start=1)
+ )
+ return references
+
+ async def make_render_job(self, *, storage_provider: StorageProviderAbstract) -> RenderJob:
+ """The engine's job: this composition with the template file's bytes read in, when there is one.
- def make_render_job(self) -> RenderJob:
- """The engine's job: this composition with the template file's bytes read in, when there is one."""
+ A stored template file is read through the run's storage provider; the caller authorizes the read against the
+ run's read scope first, with `referenced_uris`.
+ """
template: bytes | None = None
if self.template_path is not None:
template = Path(self.template_path).read_bytes()
+ elif self.template_resource is not None:
+ template = await self.template_resource.read_bytes(storage_provider=storage_provider)
return RenderJob(
format=self.format,
source=self.source,
diff --git a/pipelex/core/bundle_resources/__init__.py b/pipelex/core/bundle_resources/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/core/bundle_resources/bundle_resource.py b/pipelex/core/bundle_resources/bundle_resource.py
new file mode 100644
index 000000000..0955b057e
--- /dev/null
+++ b/pipelex/core/bundle_resources/bundle_resource.py
@@ -0,0 +1,199 @@
+"""A bundle's resources: the files beside its `.mthds` text that its steps read, such as a `PipeDocGen` template file.
+
+A bundle loaded from a directory finds them beside its file on disk. A bundle submitted as content strings, which is
+how the runner's routes receive one, has no directory, so the files come with it: each is keyed by the
+bundle-relative path a step names, read as if every bundle of the submission sat in one directory. The library holds
+them for its pipes to read, and its crate carries them wherever the library is loaded again.
+
+A resource carries its bytes inline, base64-encoded so that it serializes to JSON like the rest of a crate, or by
+reference to the run's storage. A file a step reads when the method loads, such as an HTML template, which is checked
+against the step's inputs, always travels inline, like the bundle's own text, so the files read at load are bounded in
+total (`MAX_LOAD_STAGE_RESOURCES_BYTES`). A file a step reads only when it runs, such as an office template, which only
+its engine opens, moves to the run's storage before a run is handed to an orchestrator outside this process, so a run
+that crosses to another process carries a reference rather than the bytes.
+"""
+
+import base64
+import binascii
+import hashlib
+from collections.abc import Mapping
+from enum import StrEnum
+from pathlib import PurePosixPath
+from typing import Self
+
+from pydantic import BaseModel, ConfigDict, Field, model_validator
+
+from pipelex.core.bundle_resources.exceptions import BundleResourceError, BundleResourcePathError
+from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME, StorageProviderAbstract
+
+MAX_LOAD_STAGE_RESOURCES_BYTES = 512 * 1024
+"""The most the files a bundle's steps read when the method loads, such as its HTML templates, may total, in bytes.
+
+They travel inline in the method's crate wherever it runs, a remote orchestrator's payloads included, where the remote
+side reads them while it loads the crate and may do no IO, so they cannot move to the run's storage as an office
+template does. Sized well under Temporal's 2 MB payload limit, with room for the third that base64 adds and for the
+method's own text. A fixed bound, not a setting: raising it would let a run that passed every check fail at dispatch.
+"""
+
+
+class BundleResourceReadStage(StrEnum):
+ """When a step reads a file of its bundle, which decides how the file travels with the bundle's crate."""
+
+ LOAD = "load"
+ """Read while the method loads, such as an HTML template checked against the step's inputs: it travels inline."""
+
+ RUN = "run"
+ """Read only when the step runs, such as an office template its engine fills: it may travel by reference."""
+
+ @property
+ def is_read_at_load(self) -> bool:
+ """Whether the file is read while the method loads, so it travels inline with the method wherever it runs."""
+ match self:
+ case BundleResourceReadStage.LOAD:
+ return True
+ case BundleResourceReadStage.RUN:
+ return False
+
+
+def resource_path_refusal(*, path: str) -> str | None:
+ """Why `path` cannot name a file inside a bundle, or None when it can.
+
+ The one rule for a bundle-relative path, held by the key of a file submitted with a bundle and by the path a step
+ names, such as a `PipeDocGen` step's `template_file`, so any file a step names can be submitted with its bundle.
+ The reason never repeats the path, which a host library's step would otherwise put in front of a caller.
+ """
+ if not path.strip():
+ return "it is empty"
+ if "\\" in path:
+ return "it holds a backslash; write the path with forward slashes"
+ if ":" in path:
+ return "it holds a colon"
+ if "\x00" in path:
+ return "it holds a NUL character"
+ pure_path = PurePosixPath(path)
+ if pure_path.is_absolute():
+ return "it is absolute; write it relative to the bundle"
+ if ".." in pure_path.parts:
+ return "it leaves the bundle through a '..' segment"
+ if not pure_path.parts:
+ return "it names no file"
+ return None
+
+
+def canonical_resource_path_or_none(*, path: str) -> str | None:
+ """The one spelling of a bundle-relative path, or None when it cannot name a file inside the bundle.
+
+ `.` segments and repeated slashes collapse, so `./invoice.xlsx` and `invoice.xlsx` name one file.
+ """
+ if resource_path_refusal(path=path) is not None:
+ return None
+ return PurePosixPath(path).as_posix()
+
+
+def canonical_resource_path(*, path: str) -> str:
+ """The one spelling of a bundle-relative path, the key a resource is held and looked up under.
+
+ POSIX separators only, relative, and inside the bundle: an absolute path, a `..` segment, a backslash, a colon
+ and an empty path are refused. `.` segments and repeated slashes collapse, so `./invoice.xlsx` and
+ `invoice.xlsx` name one file.
+
+ Raises:
+ BundleResourcePathError: the path cannot name a file inside the bundle.
+ """
+ refusal = resource_path_refusal(path=path)
+ if refusal is not None:
+ msg = f"The file '{path}' submitted with the bundle cannot be one of its files: {refusal}."
+ raise BundleResourcePathError(msg)
+ return PurePosixPath(path).as_posix()
+
+
+class BundleResource(BaseModel):
+ """One file of a bundle, with its bytes inline or stored, and their digest either way."""
+
+ model_config = ConfigDict(frozen=True, extra="forbid")
+
+ sha256: str = Field(description="The SHA-256 of the file's bytes, in hexadecimal")
+ size: int = Field(ge=0, description="The file's size in bytes")
+ content_b64: str | None = Field(default=None, description="The file's bytes, base64-encoded, when they travel inline")
+ uri: str | None = Field(default=None, description="The pipelex-storage URI of the file's bytes, when they were stored")
+
+ @model_validator(mode="after")
+ def validate_carrier(self) -> Self:
+ if (self.content_b64 is None) == (self.uri is None):
+ msg = "A bundle resource carries its bytes either inline or by reference, exactly one of the two."
+ raise ValueError(msg)
+ if self.uri is not None and not self.uri.startswith(PIPELEX_STORAGE_SCHEME):
+ msg = f"A bundle resource stored by reference names a '{PIPELEX_STORAGE_SCHEME}' URI."
+ raise ValueError(msg)
+ return self
+
+ @classmethod
+ def make_inline(cls, *, content: bytes) -> Self:
+ return cls(
+ sha256=hashlib.sha256(content).hexdigest(),
+ size=len(content),
+ content_b64=base64.b64encode(content).decode("ascii"),
+ )
+
+ def stored_at(self, *, uri: str) -> Self:
+ """The same file, now carried by reference to where its bytes were stored."""
+ return self.model_copy(update={"content_b64": None, "uri": uri})
+
+ @property
+ def is_inline(self) -> bool:
+ return self.content_b64 is not None
+
+ def inline_bytes(self) -> bytes:
+ """The file's bytes, which it carries inline.
+
+ Raises:
+ BundleResourceError: the file was stored, so its bytes are read through the run's storage.
+ """
+ if self.content_b64 is None:
+ msg = "The bundle file was stored with its run, so it is read through the run's storage provider."
+ raise BundleResourceError(msg)
+ try:
+ return base64.b64decode(self.content_b64, validate=True)
+ except binascii.Error as exc:
+ msg = "The bundle file's inline bytes are not valid base64."
+ raise BundleResourceError(msg) from exc
+
+ async def read_bytes(self, *, storage_provider: StorageProviderAbstract) -> bytes:
+ """The file's bytes, inline or read back from storage and checked against their digest.
+
+ A stored file is read from its own `pipelex-storage://` URI only. The caller authorizes that read against the
+ run's read scope first, as it does every file a step reads.
+
+ Raises:
+ BundleResourceError: the stored bytes no longer match the digest the file was submitted with.
+ """
+ if self.content_b64 is not None:
+ return self.inline_bytes()
+ if self.uri is None:
+ msg = "A bundle resource carries its bytes either inline or by reference."
+ raise BundleResourceError(msg)
+ data = await storage_provider.load(self.uri)
+ if hashlib.sha256(data).hexdigest() != self.sha256:
+ msg = "The stored bundle file no longer matches the file submitted with the bundle."
+ raise BundleResourceError(msg)
+ return data
+
+
+BundleResources = dict[str, BundleResource]
+"""A bundle's files, keyed by their canonical bundle-relative path."""
+
+
+def make_bundle_resources(*, files: Mapping[str, bytes]) -> BundleResources:
+ """The files submitted with a bundle, keyed by their canonical path and carried inline.
+
+ Raises:
+ BundleResourcePathError: a path cannot name a file inside the bundle, or two paths name the same file.
+ """
+ resources: BundleResources = {}
+ for path, content in files.items():
+ canonical_path = canonical_resource_path(path=path)
+ if canonical_path in resources:
+ msg = f"Two files submitted with the bundle name the same file '{canonical_path}'."
+ raise BundleResourcePathError(msg)
+ resources[canonical_path] = BundleResource.make_inline(content=content)
+ return resources
diff --git a/pipelex/core/bundle_resources/exceptions.py b/pipelex/core/bundle_resources/exceptions.py
new file mode 100644
index 000000000..0e11952bf
--- /dev/null
+++ b/pipelex/core/bundle_resources/exceptions.py
@@ -0,0 +1,40 @@
+from pipelex.base_exceptions import ErrorDomain, PipelexError
+
+
+class BundleResourceError(PipelexError):
+ """Raised when a bundle resource cannot be read: stored bytes that no longer match their digest, or a read the resource cannot serve."""
+
+
+class BundleResourcePathError(PipelexError):
+ """Raised when a file submitted with a bundle is keyed by a path that cannot name a file inside the bundle.
+
+ A resource's key is the path a step names, relative to the bundle and written with forward slashes: an absolute
+ path, a `..` segment, a backslash, a drive colon and an empty path are refused, and so are two keys naming the
+ same file. The path is the caller's own, so the message is theirs to read.
+ """
+
+ error_domain = ErrorDomain.INPUT
+ _authors_caller_facing_message = True
+
+
+class BundleResourceNotNamedError(PipelexError):
+ """Raised when a file submitted with a bundle is named by none of its steps.
+
+ Every file a caller submits must be one a step reads, such as a `PipeDocGen` step's `template_file`, so a
+ misspelt key fails loudly rather than leaving the step without its file. The message names the caller's paths.
+ """
+
+ error_domain = ErrorDomain.INPUT
+ _authors_caller_facing_message = True
+
+
+class BundleResourcesTooLargeError(PipelexError):
+ """Raised when the files a bundle's steps read when the method loads, such as HTML templates, are too large in total.
+
+ They travel inline with the method wherever it runs, so their total is bounded (`MAX_LOAD_STAGE_RESOURCES_BYTES`),
+ whereas a file read only when its step runs, such as an office template, travels by reference to the run's storage.
+ The message names the caller's paths and the limit.
+ """
+
+ error_domain = ErrorDomain.INPUT
+ _authors_caller_facing_message = True
diff --git a/pipelex/interpreter_hub.py b/pipelex/interpreter_hub.py
index 327e75d0f..023f8452d 100644
--- a/pipelex/interpreter_hub.py
+++ b/pipelex/interpreter_hub.py
@@ -61,6 +61,7 @@ class InterpreterHub:
# run
self._pipe_router: PipeRouterProtocol | None = None
self._pipe_run: PipeRunProtocol | None = None
+ self._pipe_run_leaves_process = False
self._pipe_func_executor: PipeFuncExecutorProtocol | None = None
self._pipe_func_executor_registry: PipeFuncExecutorRegistry | None = None
# pipeline
@@ -107,8 +108,14 @@ class InterpreterHub:
def set_pipe_router(self, pipe_router: "PipeRouterProtocol"):
self._pipe_router = pipe_router
- def set_pipe_run(self, pipe_run: "PipeRunProtocol") -> None:
+ def set_pipe_run(self, pipe_run: "PipeRunProtocol", *, leaves_process: bool) -> None:
+ """Set the pipe run a run uses when its caller injects none, and say whether it runs the job in this process.
+
+ `leaves_process` is true for the pipe run a boot orchestrator's plugin claimed, which hands each job to that
+ orchestrator, such as Temporal, outside this process, and false for core's own, which runs it here.
+ """
self._pipe_run = pipe_run
+ self._pipe_run_leaves_process = leaves_process
def set_pipe_func_executor(self, pipe_func_executor: PipeFuncExecutorProtocol):
self._pipe_func_executor = pipe_func_executor
@@ -174,6 +181,10 @@ class InterpreterHub:
raise RuntimeError(msg)
return self._pipe_run
+ def does_pipe_run_leave_process(self) -> bool:
+ """Whether the pipe run `get_required_pipe_run` returns hands each job to an orchestrator outside this process."""
+ return self._pipe_run_leaves_process
+
def get_required_pipe_func_executor(self) -> PipeFuncExecutorProtocol:
if self._pipe_func_executor is None:
msg = "PipeFuncExecutor is not initialized"
@@ -491,6 +502,10 @@ def get_pipe_run() -> "PipeRunProtocol":
return get_interpreter_hub().get_required_pipe_run()
+def does_pipe_run_leave_process() -> bool:
+ return get_interpreter_hub().does_pipe_run_leave_process()
+
+
def get_pipeline_manager() -> PipelineManagerAbstract:
return get_interpreter_hub().get_required_pipeline_manager()
diff --git a/pipelex/libraries/bundle_resources.py b/pipelex/libraries/bundle_resources.py
new file mode 100644
index 000000000..00f4102eb
--- /dev/null
+++ b/pipelex/libraries/bundle_resources.py
@@ -0,0 +1,108 @@
+"""The files submitted with bundles loaded from content strings, checked against their steps and moved with their runs.
+
+`pipelex.core.bundle_resources.bundle_resource` says what a bundle resource is. This module holds what the library
+layer does with them: refusing a file no step names and files read at load that are too large, and moving the files
+read only when a step runs to the run's storage before the run leaves the process, so the crate a remote orchestrator
+receives carries references rather than the bytes.
+"""
+
+import mimetypes
+import re
+from collections.abc import Iterable, Mapping
+from pathlib import PurePosixPath
+
+from pipelex.core.bundle_resources.bundle_resource import (
+ MAX_LOAD_STAGE_RESOURCES_BYTES,
+ BundleResource,
+ BundleResourceReadStage,
+ canonical_resource_path_or_none,
+)
+from pipelex.core.bundle_resources.exceptions import BundleResourceNotNamedError, BundleResourcesTooLargeError
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.mthds_parsing.pipelex_bundle_blueprint import PipeBlueprintUnion, PipelexBundleBlueprint
+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
+
+# The run's storage leaf for the files its caller supplied, under which the bundle's files are stored by digest.
+BUNDLE_RESOURCES_STORAGE_PREFIX = "assets/bundle"
+_UNSAFE_KEY_CHARACTERS = re.compile(r"[^A-Za-z0-9._-]+")
+
+
+def bundle_resource_reads(*, pipe_blueprints: Iterable[PipeBlueprintUnion]) -> dict[str, BundleResourceReadStage]:
+ """The bundle files these pipes read, keyed by canonical path, and when each is first read.
+
+ A path no file inside a bundle can have, such as one with a `..` segment, is left out: no submitted file can
+ answer it, and a step's blueprint refuses to name one. A file read at load by one step and at run by another
+ counts as read at load, which is the stricter way to carry it.
+ """
+ reads: dict[str, BundleResourceReadStage] = {}
+ for pipe_blueprint in pipe_blueprints:
+ for named_path, stage in pipe_blueprint.bundle_resource_reads.items():
+ canonical_path = canonical_resource_path_or_none(path=named_path)
+ if canonical_path is None:
+ continue
+ known_stage = reads.get(canonical_path)
+ if known_stage is None or not known_stage.is_read_at_load:
+ reads[canonical_path] = stage
+ return reads
+
+
+def check_bundle_resources(*, blueprints: Iterable[PipelexBundleBlueprint], bundle_resources: Mapping[str, BundleResource]) -> None:
+ """Refuse a file submitted with the bundles that none of their steps names, and files read at load that are too large.
+
+ The files the steps read when the method loads, such as HTML templates, travel inline with the method wherever it
+ runs, so their total is bounded by `MAX_LOAD_STAGE_RESOURCES_BYTES`.
+
+ Raises:
+ BundleResourceNotNamedError: a submitted file is named by no step of the bundles.
+ BundleResourcesTooLargeError: the files the steps read when the method loads total more than the bound.
+ """
+ if not bundle_resources:
+ return
+ pipe_blueprints = [pipe_blueprint for blueprint in blueprints for pipe_blueprint in (blueprint.pipe or {}).values()]
+ reads = bundle_resource_reads(pipe_blueprints=pipe_blueprints)
+ unnamed_paths = sorted(set(bundle_resources) - set(reads))
+ if unnamed_paths:
+ listed = ", ".join(f"'{path}'" for path in unnamed_paths)
+ msg = (
+ f"The bundle was submitted with files none of its steps names: {listed}. Submit only the files a step reads, "
+ "each keyed by the path the step names, such as a PipeDocGen step's 'template_file', relative to the bundle."
+ )
+ raise BundleResourceNotNamedError(msg)
+ load_stage_paths = sorted(path for path, stage in reads.items() if stage.is_read_at_load and path in bundle_resources)
+ load_stage_bytes = sum(bundle_resources[path].size for path in load_stage_paths)
+ if load_stage_bytes > MAX_LOAD_STAGE_RESOURCES_BYTES:
+ listed = ", ".join(f"'{path}'" for path in load_stage_paths)
+ msg = (
+ f"The files the bundle's steps read when the method loads ({listed}) total {load_stage_bytes} bytes, over the "
+ f"{MAX_LOAD_STAGE_RESOURCES_BYTES // 1024} KiB limit. Such a file, an HTML template, travels inline with the method "
+ "wherever it runs, so keep it small: move images and fonts out of it, to URLs it references."
+ )
+ raise BundleResourcesTooLargeError(msg)
+
+
+async def relocate_run_resources(*, crate: LibraryCrate, storage_provider: StorageProviderAbstract, storage_scope: str) -> LibraryCrate:
+ """The crate with every file read only when a step runs stored under the run's scope and carried by reference.
+
+ Called before a run leaves the process (`prepare_job_for_remote_dispatch`), so the job an orchestrator such as
+ Temporal receives carries a reference to each office template rather than its bytes, and the step reads the bytes
+ back from the run's storage under the run's read scope, when it prints or when a dry run checks the template. A
+ file read when the method loads stays inline, since the remote side reads it while it loads the crate, where it may
+ do no IO. The files are stored by digest, so a run that submits the same file twice stores it once, and a file
+ already stored is kept as it is.
+ """
+ if not crate.bundle_resources:
+ return crate
+ run_stage_paths = {path for path, stage in bundle_resource_reads(pipe_blueprints=crate.pipes.values()).items() if not stage.is_read_at_load}
+ relocated: dict[str, BundleResource] = {}
+ for path, resource in crate.bundle_resources.items():
+ if path not in run_stage_paths or not resource.is_inline:
+ relocated[path] = resource
+ continue
+ file_name = _UNSAFE_KEY_CHARACTERS.sub("-", PurePosixPath(path).name).strip(".-") or "file"
+ uri = await storage_provider.store(
+ resource.inline_bytes(),
+ key=f"{storage_scope}/{BUNDLE_RESOURCES_STORAGE_PREFIX}/{resource.sha256}/{file_name}",
+ content_type=mimetypes.guess_type(file_name)[0],
+ )
+ relocated[path] = resource.stored_at(uri=uri)
+ return crate.model_copy(update={"bundle_resources": relocated})
diff --git a/pipelex/libraries/library.py b/pipelex/libraries/library.py
index 56b8abc40..da6c13fb7 100644
--- a/pipelex/libraries/library.py
+++ b/pipelex/libraries/library.py
@@ -7,6 +7,7 @@ from pydantic import BaseModel, Field, PrivateAttr
from pipelex import log
from pipelex.base_exceptions import PipelexUnexpectedError
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.exceptions import PipesAndConceptValidationErrorData
from pipelex.core.qualified_ref import QualifiedRef, QualifiedRefError
from pipelex.libraries.concept.concept_library import ConceptLibrary
@@ -123,6 +124,10 @@ class Library(BaseModel):
pipe_library: PipeLibrary
loaded_mthds_paths: list[Path] = Field(default_factory=empty_list_factory_of(Path))
dependency_libraries: dict[str, "Library"] = Field(default_factory=dict)
+ bundle_resources: BundleResources = Field(default_factory=dict)
+ """The files submitted with the bundles this library loaded from content strings, keyed by canonical path, for
+ their pipes to read: a `PipeDocGen` step of such a bundle reads its `template_file` here and nowhere else, while a
+ step of a bundle this host read from a file reads beside it on disk."""
_class_registry: ClassRegistry | None = PrivateAttr(default=None)
def get_class_registry(self) -> ClassRegistry | None:
@@ -131,6 +136,22 @@ class Library(BaseModel):
def set_class_registry(self, class_registry: ClassRegistry) -> None:
self._class_registry = class_registry
+ def add_bundle_resources(self, *, bundle_resources: BundleResources) -> None:
+ """Hold the files submitted with a bundle, for its pipes to read.
+
+ A file the library already holds under the same path is kept when the new one has the same bytes.
+
+ Raises:
+ LibraryError: a file the library already holds under the same path has different bytes.
+ """
+ for path, resource in bundle_resources.items():
+ held = self.bundle_resources.get(path)
+ if held is None:
+ self.bundle_resources[path] = resource
+ elif held.sha256 != resource.sha256:
+ msg = f"Two different files were submitted under the same path '{path}' for one library."
+ raise LibraryError(msg)
+
def get_domain_library(self) -> DomainLibrary:
return self.domain_library
@@ -181,6 +202,7 @@ class Library(BaseModel):
self.concept_library.teardown()
self.domain_library.teardown()
self.loaded_mthds_paths = []
+ self.bundle_resources = {}
# Drop this library's dynamically generated structure classes with the library itself.
# The registry object would be garbage anyway once the manager forgets the Library, but
# a caller holding its own reference must not keep resolving a torn-down library's classes.
diff --git a/pipelex/libraries/library_crate.py b/pipelex/libraries/library_crate.py
index bde4ab2d2..765d1d5b2 100644
--- a/pipelex/libraries/library_crate.py
+++ b/pipelex/libraries/library_crate.py
@@ -3,6 +3,7 @@ import json
from pydantic import BaseModel, ConfigDict, Field
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.domains.domain_blueprint import DomainBlueprint
from pipelex.mthds_parsing.pipelex_bundle_blueprint import ElaborationMetadata, PipeBlueprintUnion
@@ -50,6 +51,27 @@ class LibraryCrate(BaseModel):
folding source into it would break the structural-idempotency contract in load_from_crate.
"""
+ bundle_resources: BundleResources = Field(default_factory=dict)
+ """canonical bundle-relative path -> a file submitted with a bundle loaded from content strings.
+
+ Carries the files a content-string bundle's steps read, such as a PipeDocGen `template_file`, wherever the
+ crate is loaded again, a Temporal workflow included: a file read when the method loads travels inline, and one
+ read only when its step runs may travel by reference to the run's storage (see `relocate_run_resources`).
+ Empty for a library loaded from directories, whose steps read beside their files on disk. Like
+ `python_sources`, it is transport-only: EXCLUDED from the fingerprint and from the normalized crate.
+ """
+
+ host_bundle_files: dict[str, str] = Field(default_factory=dict)
+ """pipe_ref -> the bundle file the pipe was read from, on the host that assembled the crate, for a pipe whose bundle
+ that host read from a file (a library directory, a bundle file).
+
+ A pipe reads a file beside its bundle, such as a PipeDocGen `template_file`, only through this map, never through
+ `source_map`: a source is a label, which a caller sends with content strings and a bundle's own text can write, so
+ a pipe loaded from content strings has no entry and reads only the files submitted with it (`bundle_resources`).
+ Transport-only, like `python_sources`: EXCLUDED from the fingerprint and from the normalized crate, so a host path
+ never reaches a caller through `/resolve`.
+ """
+
fingerprint: str = ""
"""SHA-256 hex digest of the serialized concepts + pipes content (source is intentionally excluded)."""
diff --git a/pipelex/libraries/library_crate_factory.py b/pipelex/libraries/library_crate_factory.py
index 0e26f614b..424a0dd5b 100644
--- a/pipelex/libraries/library_crate_factory.py
+++ b/pipelex/libraries/library_crate_factory.py
@@ -1,5 +1,6 @@
from typing import TYPE_CHECKING, NamedTuple
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.domains.domain_blueprint import DomainBlueprint
from pipelex.libraries.collision_messages import duplicate_ref_msg
@@ -35,6 +36,7 @@ class LibraryCrateFactory:
blueprints: list[PipelexBundleBlueprint],
*,
python_sources: dict[str, str] | None = None,
+ bundle_resources: BundleResources | None = None,
) -> LibraryCrate:
"""Build a LibraryCrate from parsed bundle blueprints.
@@ -52,6 +54,8 @@ class LibraryCrateFactory:
python_sources: Optional relpath -> Python source text captured (without importing) in
sandbox-hosted load mode. Attached verbatim to the crate; intentionally NOT folded
into the fingerprint (the fingerprint represents library structure for dedupe).
+ bundle_resources: Optional canonical path -> file submitted with a bundle loaded from content strings.
+ Attached verbatim to the crate and, like ``python_sources``, NOT folded into the fingerprint.
Returns:
A LibraryCrate with all content merged and fingerprinted
@@ -60,11 +64,13 @@ class LibraryCrateFactory:
pipes: dict[str, PipeBlueprintUnion] = {}
domains: dict[str, DomainBlueprint] = {}
source_map: dict[str, str] = {}
+ host_bundle_files: dict[str, str] = {}
elaboration_metadata: dict[str, ElaborationMetadata] = {}
for blueprint in blueprints:
domain_code = blueprint.domain
source = blueprint.source
+ host_bundle_file = str(blueprint.host_bundle_file) if blueprint.host_bundle_file is not None else None
# Domain metadata: order-independent, omission-quiet merge (see merge_domain_metadata_field).
if domain_code not in domains:
@@ -143,6 +149,11 @@ class LibraryCrateFactory:
else:
source_map.pop(pipe_ref, None)
if winner.blueprint is pipe_blueprint:
+ # The bundle file a step reads beside follows the winner too.
+ if host_bundle_file is not None:
+ host_bundle_files[pipe_ref] = host_bundle_file
+ else:
+ host_bundle_files.pop(pipe_ref, None)
cls._track_elaboration(
elaboration_metadata=elaboration_metadata, pipe_ref=pipe_ref, elaboration=blueprint.get_elaboration_for(pipe_code)
)
@@ -150,6 +161,8 @@ class LibraryCrateFactory:
pipes[pipe_ref] = pipe_blueprint
if source:
source_map[pipe_ref] = source
+ if host_bundle_file is not None:
+ host_bundle_files[pipe_ref] = host_bundle_file
cls._track_elaboration(
elaboration_metadata=elaboration_metadata, pipe_ref=pipe_ref, elaboration=blueprint.get_elaboration_for(pipe_code)
)
@@ -164,8 +177,10 @@ class LibraryCrateFactory:
pipes=pipes,
domains=domains,
source_map=source_map,
+ host_bundle_files=host_bundle_files,
elaboration_metadata=elaboration_metadata,
python_sources=python_sources or {},
+ bundle_resources=dict(bundle_resources) if bundle_resources else {},
fingerprint=fingerprint,
)
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index cda2f29f8..4acd25668 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -18,6 +18,7 @@ from pipelex import log
from pipelex.base_exceptions import PipelexError, SecurityError, error_domain_is_input, iter_cause_chain
from pipelex.cogt.exceptions import ModelChoiceNotFoundError
from pipelex.config import is_pipe_func_sandbox_hosted
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.helpers import make_qualified_structure_class_name
@@ -80,7 +81,7 @@ if TYPE_CHECKING:
MTHDS_METHODS_DIRNAME = ".mthds/methods"
-def _find_methods_dirs_from_blueprints(blueprints: list[PipelexBundleBlueprint]) -> list[Path]:
+def find_methods_dirs_from_blueprints(blueprints: list[PipelexBundleBlueprint]) -> list[Path]:
"""Find .mthds/methods/ directories by walking up from blueprint source paths.
When a bundle file is located outside the CWD (e.g. validating a
@@ -90,8 +91,8 @@ def _find_methods_dirs_from_blueprints(blueprints: list[PipelexBundleBlueprint])
additional .mthds/methods/ directories.
Args:
- blueprints: The parsed bundle blueprints (their ``source`` field
- holds the original file path)
+ blueprints: The parsed bundle blueprints; only those read from a file on this host
+ (``host_bundle_file``) are walked up from
Returns:
Deduplicated list of existing .mthds/methods/ directories found
@@ -99,9 +100,11 @@ def _find_methods_dirs_from_blueprints(blueprints: list[PipelexBundleBlueprint])
result: list[Path] = []
seen: set[Path] = set()
for blueprint in blueprints:
- if not blueprint.source:
+ # Only from a file this host read the bundle from: a bundle parsed from a string carries a label at most, which
+ # a caller chose, so walking up from it would search this host's directories wherever the caller pointed.
+ if blueprint.host_bundle_file is None:
continue
- current = Path(blueprint.source).resolve().parent
+ current = blueprint.host_bundle_file.resolve().parent
while True:
candidate = current / MTHDS_METHODS_DIRNAME
resolved_candidate = candidate.resolve()
@@ -540,13 +543,23 @@ class LibraryManager(LibraryManagerAbstract):
accumulated = self._blueprints.get(library_id)
if not accumulated:
return None
+ library = self._libraries.get(library_id)
crate = LibraryCrateFactory.make_from_blueprints(
blueprints=accumulated,
python_sources=self._library_sources.get(library_id),
+ bundle_resources=library.bundle_resources if library is not None else None,
)
self._crate_cache[library_id] = crate
return crate
+ @override
+ def add_bundle_resources(self, *, library_id: str, bundle_resources: BundleResources) -> None:
+ if not bundle_resources:
+ return
+ self.get_library(library_id=library_id).add_bundle_resources(bundle_resources=bundle_resources)
+ # The cached crate was built without them.
+ self._crate_cache.pop(library_id, None)
+
############################################################
# Private methods
############################################################
@@ -723,6 +736,11 @@ class LibraryManager(LibraryManagerAbstract):
if library_id not in self._blueprints:
self._crate_cache[library_id] = crate
+ # The files a transported crate carries for its content-string bundles, held before the pipes are built,
+ # since a pipe reading one, such as a PipeDocGen step with an HTML template file, reads it as it is built.
+ if crate.bundle_resources:
+ self.get_library(library_id=library_id).add_bundle_resources(bundle_resources=crate.bundle_resources)
+
# Fingerprint idempotency: skip if this crate was already loaded into this library
fingerprint = crate.fingerprint
loaded_set = self._loaded_fingerprints.setdefault(library_id, set())
@@ -790,10 +808,11 @@ class LibraryManager(LibraryManagerAbstract):
concept_codes_for_domain = domain_concept_codes.get(domain_code, [])
source = crate.source_map.get(pipe_ref)
- # A pipe that reads a file beside its bundle (a PipeDocGen `template_file`) resolves it
- # against the bundle's own file, which the crate knows and the pipe's blueprint does not.
- if source is not None and pipe_blueprint.source is None:
- pipe_blueprint = pipe_blueprint.model_copy(update={"source": source})
+ # A pipe that reads a file beside its bundle (a PipeDocGen `template_file`) resolves it against the
+ # bundle file the host that assembled the crate read it from, and against nothing else: the pipe's
+ # `source` is overwritten with that file, or cleared, so neither a caller's label (`source_map`) nor a
+ # `source` the bundle's text wrote on the pipe can point the read at a directory on this host.
+ pipe_blueprint = pipe_blueprint.model_copy(update={"source": crate.host_bundle_files.get(pipe_ref)})
with _locating_pipe_build_refusals(
pipe_code=pipe_code, domain_code=domain_code, source=source, elaboration=crate.elaboration_metadata.get(pipe_ref)
):
@@ -1418,8 +1437,8 @@ class LibraryManager(LibraryManagerAbstract):
# bundle's file on this host, which its blueprint carries into the factory and no further. The
# package's address stays the source every refusal names, since the host's path is not the caller's.
bundle_file = bundle_files_by_source.get(dependency_source) if dependency_source is not None else None
- if bundle_file is not None and pipe_blueprint.source is None:
- pipe_blueprint = pipe_blueprint.model_copy(update={"source": str(bundle_file)})
+ # Overwritten either way, so a `source` the package's own text wrote on the pipe never names a directory.
+ pipe_blueprint = pipe_blueprint.model_copy(update={"source": str(bundle_file) if bundle_file is not None else None})
try:
# The same location the main load path attaches, so a dependency pipe's refusal
# names the dependency's own file.
@@ -1514,7 +1533,7 @@ class LibraryManager(LibraryManagerAbstract):
return
# Derive extra search dirs from bundle source paths
- extra_search_dirs = _find_methods_dirs_from_blueprints(blueprints)
+ extra_search_dirs = find_methods_dirs_from_blueprints(blueprints)
unresolved_items: list[PipesAndConceptValidationErrorData] = []
for full_address, (reference, blueprint) in first_references.items():
diff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.py
index 7de66929a..53050a259 100644
--- a/pipelex/libraries/library_manager_abstract.py
+++ b/pipelex/libraries/library_manager_abstract.py
@@ -4,6 +4,7 @@ from typing import TYPE_CHECKING
from kajson.class_registry import ClassRegistry
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.libraries.library_crate import LibraryCrate
from pipelex.mthds_parsing.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
@@ -97,6 +98,18 @@ class LibraryManagerAbstract(ABC):
A LibraryCrate built from all accumulated blueprints, or None
"""
+ @abstractmethod
+ def add_bundle_resources(self, *, library_id: str, bundle_resources: BundleResources) -> None:
+ """Hold the files submitted with bundles loaded from content strings, for their pipes to read.
+
+ Called before the bundles load, since a pipe that reads one, such as a `PipeDocGen` step with an HTML
+ `template_file`, reads it while it is built. The library's crate carries them from then on.
+
+ Args:
+ library_id: The library the bundles load into
+ bundle_resources: The files, keyed by canonical bundle-relative path
+ """
+
@abstractmethod
def get_accumulated_blueprints(self, library_id: str) -> list[PipelexBundleBlueprint]:
"""The bundle blueprints accumulated into a library, in load order.
diff --git a/pipelex/mthds_parsing/parser.py b/pipelex/mthds_parsing/parser.py
index bcf467105..4cf380426 100644
--- a/pipelex/mthds_parsing/parser.py
+++ b/pipelex/mthds_parsing/parser.py
@@ -94,6 +94,10 @@ class MthdsParser(BaseModel):
) from exc
try:
- return BundleElaborator.elaborate(bundle=pipelex_bundle_blueprint)
+ elaborated_blueprint = BundleElaborator.elaborate(bundle=pipelex_bundle_blueprint)
except BundleElaboratorError as exc:
raise MthdsParserError(message=str(exc)) from exc
+ # Only a bundle read from a file here may read beside it: a content string's `mthds_source` is a label.
+ if bundle_path is not None:
+ elaborated_blueprint.mark_read_from_host_file(bundle_file=Path(bundle_path))
+ return elaborated_blueprint
diff --git a/pipelex/mthds_parsing/pipelex_bundle_blueprint.py b/pipelex/mthds_parsing/pipelex_bundle_blueprint.py
index f669564b5..6ff6068b0 100644
--- a/pipelex/mthds_parsing/pipelex_bundle_blueprint.py
+++ b/pipelex/mthds_parsing/pipelex_bundle_blueprint.py
@@ -1,7 +1,8 @@
from enum import StrEnum
+from pathlib import Path
from typing import Annotated, Any, NamedTuple, cast
-from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator, model_validator
+from pydantic import BaseModel, ConfigDict, Field, PrivateAttr, ValidationInfo, field_validator, model_validator
from pydantic.json_schema import SkipJsonSchema
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
@@ -123,6 +124,22 @@ class PipelexBundleBlueprint(BaseModel):
# next step recorded in `docs/under-the-hood/build-time-elaboration.md`.
elaboration_metadata: SkipJsonSchema[dict[str, ElaborationMetadata] | None] = Field(default=None, exclude=True)
+ # The file on this host the bundle was read from, set by the parser when it reads one and by a loader that reads a
+ # bundle's file itself. Never serialized, and never taken from `source`: `source` is a label, which a caller sends
+ # with content strings (`mthds_sources`) and the bundle's own text can write, so it names no directory to read.
+ # This is what lets a step read a file beside its bundle, such as a PipeDocGen `template_file`, and lets an
+ # address-based reference find a `.mthds/methods/` directory above it; a bundle parsed from a string has none.
+ _host_bundle_file: Path | None = PrivateAttr(default=None)
+
+ @property
+ def host_bundle_file(self) -> Path | None:
+ """The file on this host the bundle was read from, or None for a bundle parsed from a string."""
+ return self._host_bundle_file
+
+ def mark_read_from_host_file(self, *, bundle_file: Path) -> None:
+ """Record that this host read the bundle from `bundle_file`, which its steps may read beside."""
+ self._host_bundle_file = bundle_file
+
@field_validator("domain", mode="before")
@classmethod
def validate_domain_syntax(cls, domain_code: str) -> str:
diff --git a/pipelex/pipe_machinery/pipe_blueprint.py b/pipelex/pipe_machinery/pipe_blueprint.py
index c294e468e..a36ca14dc 100644
--- a/pipelex/pipe_machinery/pipe_blueprint.py
+++ b/pipelex/pipe_machinery/pipe_blueprint.py
@@ -6,6 +6,7 @@ from typing import Any, Self, cast, final
from pydantic import BaseModel, ConfigDict, Field, SerializerFunctionWrapHandler, field_validator, model_serializer, model_validator
+from pipelex.core.bundle_resources.bundle_resource import BundleResourceReadStage
from pipelex.core.concepts.exceptions import ConceptStringError
from pipelex.core.concepts.validation import validate_concept_ref_or_code
from pipelex.core.pipes.exceptions import PipeValidationError
@@ -228,6 +229,9 @@ class PipeBlueprint(ABC, BaseModel):
model_config = ConfigDict(extra="forbid")
source: str | None = None
+ """Not an authoring field. The library manager overwrites it, just before the pipe is built, with the bundle file this
+ host read the pipe's bundle from, or None, so whatever a bundle's text writes here is never read (see
+ `LibraryCrate.host_bundle_files`)."""
pipe_category: Any = Field(exclude=True) # Technical field for Union discrimination, not user-facing
type: Any # TODO: Find a better way to handle this.
description: str
@@ -286,6 +290,16 @@ class PipeBlueprint(ABC, BaseModel):
"""
return set()
+ @property
+ def bundle_resource_reads(self) -> dict[str, BundleResourceReadStage]:
+ """The files of its bundle this pipe reads, keyed by the bundle-relative path it names, and when it reads each.
+
+ Overridden by a pipe that reads a file beside its bundle, such as a `PipeDocGen` step's `template_file`.
+ A bundle submitted as content strings brings exactly these files with it, and the one read only when the
+ step runs may travel with the run by reference.
+ """
+ return {}
+
@property
def ordered_pipe_dependencies(self) -> list[str] | None:
"""Return ordered dependencies if order matters (e.g., for PipeSequence steps).
diff --git a/pipelex/pipe_operators/doc_gen/pipe_doc_gen.py b/pipelex/pipe_operators/doc_gen/pipe_doc_gen.py
index ccf4c273b..8db53efab 100644
--- a/pipelex/pipe_operators/doc_gen/pipe_doc_gen.py
+++ b/pipelex/pipe_operators/doc_gen/pipe_doc_gen.py
@@ -9,6 +9,7 @@ from typing_extensions import override
from pipelex import log
from pipelex.base_exceptions import ValidationErrorCategory, ValidationErrorItem
+from pipelex.cogt.content_generation.read_authorization import authorize_assignment_reads
from pipelex.cogt.doc_gen.doc_gen_engine import require_doc_gen_engine_declared, resolve_doc_gen_setting
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource
from pipelex.cogt.doc_gen.doc_gen_setting import DocGenModelChoice, DocGenSetting
@@ -22,6 +23,7 @@ from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_deck_check import check_doc_gen_choice_with_deck
from pipelex.cogt.templating.template_preprocessor import rewrite_template_sigils
from pipelex.cogt.templating.template_rendering import render_template
+from pipelex.core.bundle_resources.bundle_resource import BundleResource
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.pipe_output import PipeOutput
@@ -33,16 +35,18 @@ from pipelex.kernel.templating_style_ops import resolve_templating_style
from pipelex.pipe_machinery.template_guard_lint import lint_authored_template
from pipelex.pipe_operators.doc_gen.exceptions import PipeDocGenRunError, PipeDocGenTemplateCheckError, PipeDocGenUndefinedValueError
from pipelex.pipe_operators.doc_gen.template_field_check import check_template_field_paths
+from pipelex.pipe_operators.doc_gen.template_file_resource import bundle_resource_of_template_file
from pipelex.pipe_operators.pipe_operator import PipeOperator
from pipelex.pipe_run.dry_run_warning_collector import report_dry_run_warning
from pipelex.pipe_run.pipe_run_params import PipeRunParams
-from pipelex.runtime_hub import get_class_registry, get_content_generator, get_model_deck, get_report_delegate
+from pipelex.runtime_hub import get_class_registry, get_content_generator, get_model_deck, get_report_delegate, get_storage_provider
from pipelex.system.job_metadata import JobMetadata
from pipelex.tools.jinja2.exceptions import Jinja2TemplateRenderError
from pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables
from pipelex.tools.jinja2.template_category import TemplateCategory
from pipelex.tools.misc.string_utils import get_root_from_dotted_path
from pipelex.tools.templating.templating_style import TemplatingStyle
+from pipelex.tools.uri.uri_read_scope import UriReference
from pipelex.validation_error_types import TemplateCheckErrorType
_UNSAFE_FILENAME_CHARACTERS = re.compile(r"[^\w.\-]+")
@@ -85,7 +89,7 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
@property
def source(self) -> DocGenSource:
- return DocGenSource.for_step(doc_gen_format=self.doc_gen_format, has_template=self.template is not None or self.template_path is not None)
+ return DocGenSource.for_step(doc_gen_format=self.doc_gen_format, has_template=self.template is not None or self.template_file is not None)
@override
def needed_inputs(self, *, visited_pipes: set[str] | None = None) -> InputStuffSpecs:
@@ -260,10 +264,47 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
filename=filename,
title=title,
template_path=self.template_path,
+ template_resource=self._template_resource(),
template_name=self.template_file,
data={name: plain_data(content) for name, content in named_contents},
)
+ def _template_resource(self) -> BundleResource | None:
+ """The office template submitted with the bundle, or None when the step reads it beside its bundle on disk.
+
+ Looked up when the step composes, in the library the step runs in: the submitter's own, or the one a remote
+ worker loaded from the run's crate, whose copy may be a reference to the run's storage.
+ """
+ if self.template_path is not None or self.template_file is None:
+ return None
+ resource = bundle_resource_of_template_file(template_file=self.template_file)
+ if resource is None:
+ msg = f"PipeDocGen '{self.code}' names the template file '{self.template_file}', which the library it runs in does not hold."
+ raise PipeDocGenRunError(msg)
+ return resource
+
+ async def _read_template_file(self, *, job_metadata: JobMetadata) -> bytes:
+ """The office template's bytes, from beside the bundle on disk or from the files submitted with it.
+
+ A submitted file the run stored before it left the process is read back under the run's read scope, as the
+ print stage reads it.
+
+ Raises:
+ UriReadRefusedError: the stored file lies outside the run's read scope.
+ """
+ if self.template_path is not None:
+ return Path(self.template_path).read_bytes()
+ resource = self._template_resource()
+ if resource is None:
+ msg = f"PipeDocGen '{self.code}' has no template file to read."
+ raise PipeDocGenRunError(msg)
+ if resource.uri is not None:
+ authorize_assignment_reads(
+ job_metadata=job_metadata,
+ uri_references=[UriReference(uri=resource.uri, position=f"the template file '{self.template_file}'")],
+ )
+ return await resource.read_bytes(storage_provider=get_storage_provider())
+
####################################################################################################################
# Template file check, at the dry run
####################################################################################################################
@@ -277,7 +318,7 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
shapes[input_name] = shape_of_input(content_class=content_class, is_list=stuff_spec.is_multiple())
return shapes
- async def check_template_file(self, *, working_memory: WorkingMemory) -> None:
+ async def check_template_file(self, *, job_metadata: JobMetadata, working_memory: WorkingMemory) -> None:
"""Run the engine's template checker on the step's template file, when its engine has one.
The inputs in memory are the dry run's mocks, so the checker can fill the template in memory with them.
@@ -286,15 +327,17 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
Raises:
PipeDocGenTemplateCheckError: the checker reported errors.
+ UriReadRefusedError: the template file was stored outside the run's read scope.
"""
- if self.template_path is None or self.template_file is None:
+ # Only an office template file is the engine's to check; an HTML one was checked against the inputs at load.
+ if self.template is not None or self.template_file is None:
return
doc_gen_setting = self.resolve_engine()
inference_model = get_model_deck().get_required_inference_model(model_handle=doc_gen_setting.model, model_type=ModelType.DOC_GEN)
worker = DocGenWorkerFactory.make_doc_gen_worker(inference_model=inference_model, reporting_delegate=get_report_delegate())
request = TemplateCheckRequest(
format=self.doc_gen_format,
- template=Path(self.template_path).read_bytes(),
+ template=await self._read_template_file(job_metadata=job_metadata),
template_name=self.template_file,
inputs=self._input_shapes(),
data={name: plain_data(content) for name, content in self._named_contents(working_memory)},
@@ -310,22 +353,26 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
raise PipeDocGenTemplateCheckError(msg)
def _report_template_warning(self, *, finding: TemplateFinding, template_file: str) -> None:
- """Report a warning of the template check as an advisory item, located at this step, its file and the place in it."""
- location_suffix = f" ({finding.location})" if finding.location else ""
+ """Report a warning of the template check as an advisory item, located at this step, its file and the place in it.
+
+ An engine that answers an empty location names no place, so the item's `template_location` is left unset.
+ """
+ location = finding.location or None
+ location_suffix = f" ({location})" if location is not None else ""
warning = ValidationErrorItem(
category=ValidationErrorCategory.DRY_RUN,
error_type=TemplateCheckErrorType.TEMPLATE_CHECK_WARNING,
pipe_code=self.code,
domain_code=self.domain_code,
template_file=template_file,
- template_location=finding.location,
+ template_location=location,
message=f"PipeDocGen '{self.code}', template file '{template_file}': {finding.message}{location_suffix}",
)
if report_dry_run_warning(warning=warning):
return
finding_fields: dict[str, str] = {"pipe_code": self.code, "template_file": template_file, "finding_message": finding.message}
- if finding.location:
- finding_fields["finding_location"] = finding.location
+ if location is not None:
+ finding_fields["finding_location"] = location
log.warning("The template check of a PipeDocGen found a warning in its template file", fields=finding_fields)
####################################################################################################################
@@ -395,7 +442,7 @@ class PipeDocGen(PipeOperator[PipeDocGenOutput]):
# The live path with the dry mode riding into the leaf: the compose stage runs for real against the mock
# inputs, which checks the templates, and the leaf prints and stores nothing. The template file's checker
# runs first, since only a dry run has inputs to fill it with and nothing to spend.
- await self.check_template_file(working_memory=working_memory)
+ await self.check_template_file(job_metadata=job_metadata, working_memory=working_memory)
return await self._live_run_operator_pipe(
job_metadata=job_metadata,
working_memory=working_memory,
diff --git a/pipelex/pipe_operators/doc_gen/pipe_doc_gen_blueprint.py b/pipelex/pipe_operators/doc_gen/pipe_doc_gen_blueprint.py
index 11044cd87..71fcf66e0 100644
--- a/pipelex/pipe_operators/doc_gen/pipe_doc_gen_blueprint.py
+++ b/pipelex/pipe_operators/doc_gen/pipe_doc_gen_blueprint.py
@@ -7,6 +7,7 @@ from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat
from pipelex.cogt.doc_gen.doc_gen_setting import DocGenModelChoice
from pipelex.cogt.templating.exceptions import TemplateSigilSyntaxError
from pipelex.cogt.templating.template_preprocessor import preprocess_template
+from pipelex.core.bundle_resources.bundle_resource import BundleResourceReadStage, resource_path_refusal
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.pipe_machinery.pipe_blueprint import PipeBlueprint
from pipelex.tools.jinja2.exceptions import Jinja2TemplateSyntaxError
@@ -45,7 +46,8 @@ class PipeDocGenBlueprint(PipeBlueprint):
It calls no AI model: it lays out or fills what its inputs already hold. With no template, the pipe lays its inputs
out by itself (the auto-layout), for every format but `pptx`. `template` is an inline HTML and Jinja2 template, for
- `pdf` only; `template_file` is a path relative to the bundle's file: an `.html` for `pdf`, an `.xlsx` for `xlsx`, a
+ `pdf` only; `template_file` is a path relative to the bundle's file, inside its bundle and written with forward
+ slashes: an `.html` for `pdf`, an `.xlsx` for `xlsx`, a
`.docx` for `docx` and a `.pptx` for `pptx`, which `pptx` requires. `filename` is a Jinja expression over the
inputs; the suffix is added. The output must be a single `Document`, or a concept refining it. `model` names the
document engine that prints it, a model of the `doc_gen` family such as `reportlab-pdf` or, with the Pipelex
@@ -65,6 +67,13 @@ class PipeDocGenBlueprint(PipeBlueprint):
@model_validator(mode="after")
def validate_template_sources(self) -> Self:
+ # The rule the files submitted with a bundle are keyed by, so a template file can be submitted whatever loads the
+ # bundle. The message never repeats the path: a host library's step would put the host's own path in front of a caller.
+ if self.template_file is not None:
+ refusal = resource_path_refusal(path=self.template_file)
+ if refusal is not None:
+ msg = f"PipeDocGen's 'template_file' is a path relative to the bundle's file, inside its bundle, and this one cannot be: {refusal}."
+ raise ValueError(msg)
if self.template is not None and self.template_file is not None:
msg = "PipeDocGen cannot have both 'template' and 'template_file': use one or the other."
raise ValueError(msg)
@@ -89,6 +98,16 @@ class PipeDocGenBlueprint(PipeBlueprint):
raise ValueError(msg)
return self
+ @property
+ @override
+ def bundle_resource_reads(self) -> dict[str, BundleResourceReadStage]:
+ # An HTML template is read and checked against the inputs when the method loads; an office template is read
+ # only when the step prints, or when a dry run hands it to the engine's checker.
+ if self.template_file is None:
+ return {}
+ stage = BundleResourceReadStage.LOAD if self.format.is_template_html else BundleResourceReadStage.RUN
+ return {self.template_file: stage}
+
@override
def validate_inputs(self):
declared_inputs = set(self.input_names)
diff --git a/pipelex/pipe_operators/doc_gen/pipe_doc_gen_factory.py b/pipelex/pipe_operators/doc_gen/pipe_doc_gen_factory.py
index 58cbf5c38..3e1877ab8 100644
--- a/pipelex/pipe_operators/doc_gen/pipe_doc_gen_factory.py
+++ b/pipelex/pipe_operators/doc_gen/pipe_doc_gen_factory.py
@@ -1,8 +1,10 @@
+from collections.abc import Callable
from pathlib import Path
from typing import Any
from typing_extensions import override
+from pipelex.core.bundle_resources.bundle_resource import BundleResource
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
@@ -11,14 +13,17 @@ from pipelex.pipe_machinery.pipe_factory import PipeFactoryProtocol
from pipelex.pipe_operators.doc_gen.exceptions import PipeDocGenFactoryError
from pipelex.pipe_operators.doc_gen.pipe_doc_gen import PipeDocGen
from pipelex.pipe_operators.doc_gen.pipe_doc_gen_blueprint import PipeDocGenBlueprint, template_required_roots
+from pipelex.pipe_operators.doc_gen.template_file_resource import bundle_resource_of_template_file
from pipelex.tools.jinja2.template_category import TemplateCategory
def bundle_dir_of_source(source: str | None) -> Path | None:
"""The directory of the bundle file a pipe was loaded from, or None for a bundle loaded from a string.
- A content-string load, such as the hosted API's `api://bundle-0.mthds`, has no directory, so nothing beside
- the bundle can be found from it.
+ The library manager sets a pipe blueprint's `source` to the bundle file this host read the bundle from just before
+ the pipe is built, and clears it otherwise (`LibraryCrate.host_bundle_files`), so the label a caller sends with
+ content strings, or a `source` the bundle's own text writes, never reaches here: a content-string load has no
+ directory, and nothing beside it is read from this host's disk.
"""
if source is None or "://" in source:
return None
@@ -54,16 +59,23 @@ class PipeDocGenFactory(PipeFactoryProtocol[PipeDocGenBlueprint, PipeDocGen]):
template = blueprint.template
template_path: str | None = None
if blueprint.template_file is not None:
- resolved_path = cls._resolve_template_file(pipe_code=pipe_code, template_file=blueprint.template_file, source=blueprint.source)
+ bundle_dir = bundle_dir_of_source(blueprint.source)
+ read_template_bytes: Callable[[], bytes]
+ if bundle_dir is None:
+ # A bundle loaded from content strings reads only the files submitted with it, never this host's disk. An
+ # office template is read from them when the step prints, so the pipe keeps only its name, and the run
+ # carries the file in its crate.
+ read_template_bytes = cls._submitted_template_file(pipe_code=pipe_code, template_file=blueprint.template_file).inline_bytes
+ else:
+ # A bundle this host read from a file finds its template file beside it, and never among submitted files.
+ resolved_path = cls._resolve_template_file(pipe_code=pipe_code, template_file=blueprint.template_file, bundle_dir=bundle_dir)
+ read_template_bytes = resolved_path.read_bytes
+ if not blueprint.format.is_template_html:
+ template_path = str(resolved_path)
if blueprint.format.is_template_html:
- try:
- template = resolved_path.read_text(encoding="utf-8")
- except (OSError, UnicodeError) as exc:
- msg = f"PipeDocGen '{pipe_code}' could not read the template file '{blueprint.template_file}' as UTF-8 text."
- raise PipeDocGenFactoryError(msg) from exc
+ # An HTML template is read and checked now, from disk as from the submitted bytes, as written.
+ template = cls._html_template_file_text(pipe_code=pipe_code, template_file=blueprint.template_file, read_bytes=read_template_bytes)
cls._check_html_template_file(pipe_code=pipe_code, template=template, inputs=inputs, template_file=blueprint.template_file)
- else:
- template_path = str(resolved_path)
return PipeDocGen(
domain_code=domain_code,
@@ -80,20 +92,12 @@ class PipeDocGenFactory(PipeFactoryProtocol[PipeDocGenBlueprint, PipeDocGen]):
)
@classmethod
- def _resolve_template_file(cls, *, pipe_code: str, template_file: str, source: str | None) -> Path:
+ def _resolve_template_file(cls, *, pipe_code: str, template_file: str, bundle_dir: Path) -> Path:
"""The template file's path, beside the bundle and inside its directory, checked to exist.
The messages name the template file as the method does and never the host's path to it: the bundle a
refusal is located in already says which one, and a dependency's install path is not the caller's to read.
"""
- bundle_dir = bundle_dir_of_source(source)
- if bundle_dir is None:
- msg = (
- f"PipeDocGen '{pipe_code}' names the template file '{template_file}', but its bundle was not loaded from a file on disk, "
- "so there is no directory to find the template in. Template files are not supported there yet: they work for bundles "
- "loaded from a directory. Use an inline 'template' for a pdf, or no template for the auto-layout."
- )
- raise PipeDocGenFactoryError(msg)
resolved_path = (bundle_dir / template_file).resolve()
if not resolved_path.is_relative_to(bundle_dir):
msg = f"PipeDocGen '{pipe_code}' names the template file '{template_file}', which is outside its bundle's directory."
@@ -103,6 +107,29 @@ class PipeDocGenFactory(PipeFactoryProtocol[PipeDocGenBlueprint, PipeDocGen]):
raise PipeDocGenFactoryError(msg)
return resolved_path
+ @classmethod
+ def _submitted_template_file(cls, *, pipe_code: str, template_file: str) -> BundleResource:
+ """The template file submitted with a bundle loaded from content strings, which the current library holds."""
+ resource = bundle_resource_of_template_file(template_file=template_file)
+ if resource is None:
+ msg = (
+ f"PipeDocGen '{pipe_code}' names the template file '{template_file}', which was not supplied with its "
+ "bundle. A bundle loaded from content strings has no directory to find the file in, so the file must be "
+ f"submitted with it, keyed by the same path, '{template_file}'. A bundle loaded from a directory "
+ "finds it beside its file."
+ )
+ raise PipeDocGenFactoryError(msg)
+ return resource
+
+ @classmethod
+ def _html_template_file_text(cls, *, pipe_code: str, template_file: str, read_bytes: Callable[[], bytes]) -> str:
+ """An HTML template file's text, decoded from its bytes as written, line endings included."""
+ try:
+ return read_bytes().decode("utf-8")
+ except (OSError, UnicodeError) as exc:
+ msg = f"PipeDocGen '{pipe_code}' could not read the template file '{template_file}' as UTF-8 text."
+ raise PipeDocGenFactoryError(msg) from exc
+
@classmethod
def _check_html_template_file(cls, *, pipe_code: str, template: str, inputs: InputStuffSpecs, template_file: str) -> None:
"""Check an HTML template file at load, as an inline template is checked when its blueprint validates."""
diff --git a/pipelex/pipe_operators/doc_gen/template_file_resource.py b/pipelex/pipe_operators/doc_gen/template_file_resource.py
new file mode 100644
index 000000000..659e52128
--- /dev/null
+++ b/pipelex/pipe_operators/doc_gen/template_file_resource.py
@@ -0,0 +1,15 @@
+from pipelex.core.bundle_resources.bundle_resource import BundleResource, canonical_resource_path_or_none
+from pipelex.interpreter_hub import get_library
+
+
+def bundle_resource_of_template_file(*, template_file: str) -> BundleResource | None:
+ """The file submitted with the bundle under the path a step's `template_file` names, or None when none was.
+
+ Looked up in the current library, which holds the files submitted with the bundles it loaded from content strings
+ and the ones a transported crate carries: the factory reads an HTML template from it when the step is built, and
+ the step reads an office template from it when it composes, so the run's own copy is the one it prints.
+ """
+ canonical_path = canonical_resource_path_or_none(path=template_file)
+ if canonical_path is None:
+ return None
+ return get_library().bundle_resources.get(canonical_path)
diff --git a/pipelex/pipe_run/remote_dispatch.py b/pipelex/pipe_run/remote_dispatch.py
new file mode 100644
index 000000000..a417ee183
--- /dev/null
+++ b/pipelex/pipe_run/remote_dispatch.py
@@ -0,0 +1,34 @@
+"""A pipe job as an orchestrator outside this process receives it.
+
+A job handed to an orchestrator other than the in-process one, such as Temporal, travels as that orchestrator's
+payloads, which it bounds: Temporal refuses a payload over 2 MB. The files submitted with the job's bundles travel in
+its crate, so before the job leaves, every file a step reads only when it runs, such as an office template, moves to the
+run's storage and the crate carries a reference to it. The files read when the method loads, such as an HTML template,
+stay inline, and their size is bounded when the bundle loads (`MAX_LOAD_STAGE_RESOURCES_BYTES`).
+
+Every dispatcher calls `prepare_job_for_remote_dispatch` at the moment the job leaves the process:
+`PipelexMTHDSProtocol.execute` when the hub's pipe run belongs to a boot orchestrator's plugin, and a host dispatching
+through an orchestrator of the registry, such as the API server, for every mode but `direct`.
+"""
+
+from pipelex.libraries.bundle_resources import relocate_run_resources
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.runtime_hub import get_storage_provider
+from pipelex.system.storage_scope import DRY_RUN_STORAGE_SCOPE
+
+
+async def prepare_job_for_remote_dispatch(*, pipe_job: PipeJob) -> PipeJob:
+ """The job with the files its steps read only when they run stored under the run's scope and carried by reference.
+
+ A dry run's files move too, since a dry run hands its office templates to the engine's checker, which reads them
+ back under the run's read scope as a print does. A run whose storage scope is the dry-run placeholder stores
+ nothing, so its files stay inline. The job is returned unchanged when it carries no submitted file.
+ """
+ crate = pipe_job.library_crate
+ if crate is None or not crate.bundle_resources:
+ return pipe_job
+ storage_scope = pipe_job.job_metadata.run_metadata.storage_scope
+ if storage_scope == DRY_RUN_STORAGE_SCOPE:
+ return pipe_job
+ relocated_crate = await relocate_run_resources(crate=crate, storage_provider=get_storage_provider(), storage_scope=storage_scope)
+ return pipe_job.model_copy(update={"library_crate": relocated_crate})
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index e2d3d5c1f..dd7d83a82 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -196,10 +196,13 @@ class Pipelex(RuntimeBoot):
)
# --- Pipe Run --------------------------------------------------------------------------
- # No explicit param for pipe_run: plugin slot-claim thunk > core default.
+ # No explicit param for pipe_run: plugin slot-claim thunk > core default. A boot orchestrator's pipe run hands
+ # each job to that orchestrator, outside this process, which is what decides how the job's files travel
+ # (`prepare_job_for_remote_dispatch`); core's runs it here.
self.interpreter_hub.set_pipe_run(
- self._resolve_hub_slot(slot=HubSlot.PIPE_RUN, default=lambda: PipeRun(pipe_router=self.interpreter_hub.get_required_pipe_router()))
+ self._resolve_hub_slot(slot=HubSlot.PIPE_RUN, default=lambda: PipeRun(pipe_router=self.interpreter_hub.get_required_pipe_router())),
+ leaves_process=self._is_hub_slot_claimed(slot=HubSlot.PIPE_RUN),
)
@override
diff --git a/pipelex/pipeline/direct_bundle_validator.py b/pipelex/pipeline/direct_bundle_validator.py
index e18572be7..ecec1d84d 100644
--- a/pipelex/pipeline/direct_bundle_validator.py
+++ b/pipelex/pipeline/direct_bundle_validator.py
@@ -22,6 +22,7 @@ if TYPE_CHECKING:
from pathlib import Path
from pipelex.base_exceptions import ErrorReport
+ from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.pipeline.validation_report import PipelexValidationReport
from pipelex.system.caller_identity import CallerIdentity
@@ -44,6 +45,7 @@ class DirectBundleValidator:
library_dirs: Sequence[Path] | None,
caller_identity: CallerIdentity | None,
graph_pipe_code: str | None,
+ bundle_resources: BundleResources | None,
) -> PipelexValidationReport | ErrorReport:
try:
return await validate_bundles_in_process(
@@ -54,6 +56,7 @@ class DirectBundleValidator:
graph_pipe_code=graph_pipe_code,
log_context="API validate",
caller_identity=caller_identity,
+ bundle_resources=bundle_resources,
)
except ValidateBundleError as exc:
# An invalid bundle is a produced verdict, not a fault: surface its structured
diff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.py
index f30b8c2a3..664fe06a1 100644
--- a/pipelex/pipeline/execution_seams.py
+++ b/pipelex/pipeline/execution_seams.py
@@ -27,6 +27,8 @@ from typing import TYPE_CHECKING
from mthds.protocol.pipeline_inputs import PipelineInputs
from pipelex import log
+from pipelex.base_exceptions import PipelexUnexpectedError
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
@@ -39,6 +41,7 @@ from pipelex.interpreter_hub import (
set_current_library,
)
from pipelex.kernel.memory_ops import shape_inputs
+from pipelex.libraries.bundle_resources import check_bundle_resources
from pipelex.mthds_parsing.parser import MthdsParser
from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
from pipelex.pipe_run.pipe_job import PipeJob
@@ -69,6 +72,7 @@ def acquire_library(
bundle_uris: list[str] | None = None,
library_dirs_are_callers: bool = False,
methods_dirs: list[Path] | None = None,
+ bundle_resources: BundleResources | None = None,
) -> tuple[str, str | None]:
"""Set the current library, open it, and load dirs + blueprints into it.
@@ -102,10 +106,25 @@ def acquire_library(
reference of the contents is looked up there first, before the installed stores and fetch-on-miss, and a package
found there is never installed; the bundles of a package are named by its address, never by these directories.
+ ``bundle_uris``, parallel to ``mthds_contents``, name the files on this host the caller read them from, such as the
+ bundle file a CLI runs. A bundle already loaded from the library directories under that path is skipped, and one
+ whose file exists is loaded as read from it: its steps read beside it, such as a ``PipeDocGen`` step's
+ ``template_file``, and an address-based reference is searched for in a ``.mthds/methods/`` directory above it, as
+ when its directory loads. They are the host's own paths, never a label a remote caller sends: a server passes none.
+
+ ``bundle_resources`` are the files submitted with the ``mthds_contents`` that their steps read, such as a
+ ``PipeDocGen`` step's ``template_file``, keyed by canonical bundle-relative path. The library holds them before the
+ contents load, and its crate carries them with the run. A file none of the contents' steps names is refused with a
+ ``BundleResourceNotNamedError``, and files read at load too large to travel with the method with a
+ ``BundleResourcesTooLargeError``.
+
Returns the ``library_id`` and the bundle's qualified ``main_pipe`` ref
(``domain.pipe_code``) when an ``mthds_contents`` bundle declares one, else
``None``. Pipe resolution is the caller's responsibility.
"""
+ if bundle_resources and not mthds_contents:
+ msg = "bundle_resources accompany mthds_contents: a bundle loaded from a directory reads the files beside it on disk"
+ raise PipelexUnexpectedError(msg)
library_manager = get_library_manager()
prev_library_id = get_current_library_id_or_none()
# Adopt the canonical id open_library keyed the Library under: when given a falsy library_id it
@@ -132,12 +151,18 @@ def acquire_library(
if mthds_contents:
# Beside a host's library directories, the verdict names none of their files, which are paths on the host.
host_library_withholding: AbstractContextManager[None] = (
- nullcontext() if library_dirs_are_callers else withholding_host_library_files(library_dirs=effective_dirs)
+ nullcontext()
+ if library_dirs_are_callers
+ else withholding_host_library_files(library_dirs=effective_dirs, caller_contents=mthds_contents)
)
with host_library_withholding, translate_to_validate_bundle_error():
all_blueprints = [MthdsParser.make_pipelex_bundle_blueprint(mthds_content=content) for content in mthds_contents]
+ if bundle_resources:
+ check_bundle_resources(blueprints=all_blueprints, bundle_resources=bundle_resources)
+ library_manager.add_bundle_resources(library_id=library_id, bundle_resources=bundle_resources)
- # Filter out blueprints whose URIs are already loaded (e.g. via PIPELEXPATH).
+ # Filter out blueprints whose URIs are already loaded (e.g. via PIPELEXPATH), and record the file on this
+ # host each of the others was read from, so its steps read beside it as when its directory loads.
blueprints_to_load: list[PipelexBundleBlueprint] = list(all_blueprints)
if bundle_uris:
current_library = library_manager.get_library(library_id=library_id)
@@ -150,6 +175,8 @@ def acquire_library(
if resolved_uri in current_library.loaded_mthds_paths:
log.verbose(f"Bundle '{uri}' was skipped, already loaded from library directories")
else:
+ if resolved_uri.is_file():
+ blueprint.mark_read_from_host_file(bundle_file=resolved_uri)
blueprints_to_load.append(blueprint)
if blueprints_to_load:
diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.py
index 4ce72e06f..56ac749bc 100644
--- a/pipelex/pipeline/pipeline_run_setup.py
+++ b/pipelex/pipeline/pipeline_run_setup.py
@@ -5,6 +5,7 @@ from mthds.protocol.pipeline_inputs import PipelineInputs
from pipelex import log
from pipelex.config import get_config
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.graph.graph_tracer_manager import GraphTracerManager
@@ -71,6 +72,7 @@ async def pipeline_run_setup(
inputs_base_dir: Path | None = None,
library_dirs_are_callers: bool = False,
methods_dirs: list[Path] | None = None,
+ bundle_resources: BundleResources | None = None,
) -> tuple[PipeJob, str, str]:
"""Set up a pipeline for execution.
@@ -106,7 +108,8 @@ async def pipeline_run_setup(
List of URIs identifying the bundles. Used to detect if bundles were already
loaded from library directories (e.g., via PIPELEXPATH) to avoid duplicate
domain registration. If all resolved absolute paths are already in the loaded
- MTHDS paths, the ``mthds_contents`` loading will be skipped.
+ MTHDS paths, the ``mthds_contents`` loading will be skipped. A content whose URI names a file on this host
+ loads as read from that file, so its steps read beside it, such as a ``PipeDocGen`` step's ``template_file``.
inputs:
Inputs passed to the pipeline. Can be either a ``PipelineInputs`` dictionary
or a ``WorkingMemory`` instance.
@@ -176,6 +179,12 @@ async def pipeline_run_setup(
``mthds_contents``, for this run alone. An address-based reference is looked up there before the
installed stores and fetch-on-miss, and a package found there is never installed. See
:func:`acquire_library`.
+ bundle_resources:
+ The files submitted with ``mthds_contents`` that their steps read, such as a ``PipeDocGen``
+ step's ``template_file``, keyed by canonical bundle-relative path. The run library holds them, and
+ the job's crate carries them inline; a host that hands the job to a remote orchestrator moves the
+ ones read only when a step runs to the run's storage first (``relocate_run_resources``). See
+ :func:`acquire_library`.
Returns:
-------
@@ -306,6 +315,7 @@ async def pipeline_run_setup(
bundle_uris=bundle_uris,
library_dirs_are_callers=library_dirs_are_callers,
methods_dirs=methods_dirs,
+ bundle_resources=bundle_resources,
)
library_acquired = True
diff --git a/pipelex/pipeline/resolve_bundle.py b/pipelex/pipeline/resolve_bundle.py
index b8b90ca58..2924d1865 100644
--- a/pipelex/pipeline/resolve_bundle.py
+++ b/pipelex/pipeline/resolve_bundle.py
@@ -17,8 +17,10 @@ from typing import Sequence
from mthds.package.manifest.schema import MTHDS_STANDARD_VERSION
from pipelex.base_exceptions import PipelexUnexpectedError
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager, set_current_library
+from pipelex.libraries.bundle_resources import check_bundle_resources
from pipelex.libraries.crate_normalization import normalize_crate
from pipelex.libraries.library_crate import LibraryCrate
from pipelex.libraries.pipe.exceptions import PipeLibraryError
@@ -27,7 +29,12 @@ from pipelex.mthds_parsing.pipelex_bundle_blueprint import PipelexBundleBlueprin
from pipelex.pipeline.validate_bundle_translation import translate_to_validate_bundle_error
-def resolve_crate_from_contents(*, mthds_contents: list[str], mthds_sources: Sequence[str | None] | None = None) -> LibraryCrate:
+def resolve_crate_from_contents(
+ *,
+ mthds_contents: list[str],
+ mthds_sources: Sequence[str | None] | None = None,
+ bundle_resources: BundleResources | None = None,
+) -> LibraryCrate:
"""Assemble, validate, and normalize a library crate from in-memory MTHDS contents.
Mirrors ``validate_bundle``'s in-memory arm and its **loaded-on-success contract**: on
@@ -42,6 +49,9 @@ def resolve_crate_from_contents(*, mthds_contents: list[str], mthds_sources: Seq
onto each blueprint so diagnostics carry a real ``source``; a ``None`` entry leaves
that content source-less. Must match ``mthds_contents`` in length when provided — a
mismatch is a host-wiring bug.
+ bundle_resources: The files submitted with the contents that their steps read, such as a ``PipeDocGen``
+ step's ``template_file``, keyed by canonical bundle-relative path. The load reads them; the normalized
+ crate, a standard artifact, does not carry them.
Returns:
The normalized library crate (fully qualified, natives materialized, fingerprint set).
@@ -73,6 +83,9 @@ def resolve_crate_from_contents(*, mthds_contents: list[str], mthds_sources: Seq
for content, source in zip(mthds_contents, content_sources, strict=True)
]
_reject_address_based_dependencies(blueprints=blueprints)
+ if bundle_resources:
+ check_bundle_resources(blueprints=blueprints, bundle_resources=bundle_resources)
+ library_manager.add_bundle_resources(library_id=library_id, bundle_resources=bundle_resources)
# load_from_blueprints ends in load_from_crate, whose last step is
# library.validate_library() — so a crate read after it satisfies the D6
# "built only from a valid library" contract.
diff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.py
index e6db6c9ff..5339e2616 100644
--- a/pipelex/pipeline/runner.py
+++ b/pipelex/pipeline/runner.py
@@ -20,12 +20,14 @@ from pipelex.config import get_config
from pipelex.graph.graph_tracer_manager import GraphTracerManager
from pipelex.interpreter_hub import (
clear_current_library,
+ does_pipe_run_leave_process,
get_current_library_id_or_none,
get_library_manager,
get_pipe_run,
get_pipeline_manager,
set_current_library,
)
+from pipelex.pipe_run.remote_dispatch import prepare_job_for_remote_dispatch
from pipelex.pipeline.exceptions import PipeExecutionError, PipelineExecutionError
from pipelex.pipeline.pipeline_response import PipelexRunResultExecute, PipelexRunResultStart, RunState
from pipelex.pipeline.pipeline_run_setup import pipeline_run_setup
@@ -46,6 +48,7 @@ if TYPE_CHECKING:
from mthds.protocol.pipeline_inputs import PipelineInputs
from mthds.protocol.working_memory import WorkingMemoryAbstract
+ from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
@@ -188,6 +191,8 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):
# a label here would put every such run into one shared entity.
extras: dict[str, str] | None = None,
execution_config: PipelineExecutionConfig | None = None,
+ # The pipe run to hand each job to instead of the hub's. One that hands the job to an orchestrator outside this
+ # process prepares it with `prepare_job_for_remote_dispatch` first, as the hub's path does here.
pipe_run: PipeRunProtocol | None = None,
inputs_base_dir: Path | None = None,
# Whose `library_dirs` are: the caller's own on a local CLI run, so a refusal while loading
@@ -198,11 +203,16 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):
# runs, searched first for an address-based reference and never written. A run reads them; `validate` does
# not, and resolves such a reference through the installed stores and fetch-on-miss only.
methods_dirs: list[Path] | None = None,
+ # The files a caller submits with the contents it runs or validates that their steps read, such as a
+ # PipeDocGen step's `template_file`, keyed by canonical bundle-relative path (`make_bundle_resources`).
+ # A bundle loaded from a directory reads beside itself on disk instead.
+ bundle_resources: BundleResources | None = None,
):
self.library_id = library_id
self.library_dirs = library_dirs
self.library_dirs_are_callers = library_dirs_are_callers
self.methods_dirs = methods_dirs
+ self.bundle_resources = bundle_resources
self.bundle_uris = bundle_uris
self.pipe_run_mode = pipe_run_mode
self.is_mock_usage = is_mock_usage
@@ -350,9 +360,14 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):
inputs_base_dir=self.inputs_base_dir,
library_dirs_are_callers=self.library_dirs_are_callers,
methods_dirs=self.methods_dirs,
+ bundle_resources=self.bundle_resources,
request_id=request_id,
)
effective_pipe_run = self._pipe_run or get_pipe_run()
+ if self._pipe_run is None and does_pipe_run_leave_process():
+ # The hub's pipe run hands the job to the boot orchestrator, outside this process, so the files its
+ # steps read when they run travel by reference. An injected pipe run dispatches the job itself.
+ pipe_job = await prepare_job_for_remote_dispatch(pipe_job=pipe_job)
with method_source_scope(source=self._method_source(mthds_contents=mthds_contents)):
pipe_output = await effective_pipe_run.run(pipe_job, delivery_assignment=delivery_assignment)
except PipelexError as exc:
@@ -540,6 +555,7 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):
caller_identity=self.caller_identity,
# The same ownership a run of this runtime reads: the caller's own directories keep their paths.
library_dirs_are_callers=self.library_dirs_are_callers,
+ bundle_resources=self.bundle_resources,
)
@override
diff --git a/pipelex/pipeline/validate_bundle.py b/pipelex/pipeline/validate_bundle.py
index 6dc7f6d04..91bf6d6c4 100644
--- a/pipelex/pipeline/validate_bundle.py
+++ b/pipelex/pipeline/validate_bundle.py
@@ -11,6 +11,7 @@ from typing_extensions import TypedDict
from pipelex import log
from pipelex.base_exceptions import PipelexUnexpectedError
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.interpreter_hub import (
clear_current_library,
@@ -19,6 +20,7 @@ from pipelex.interpreter_hub import (
resolve_library_dirs,
set_current_library,
)
+from pipelex.libraries.bundle_resources import check_bundle_resources
from pipelex.libraries.library_utils import get_pipelex_mthds_files_from_dirs
from pipelex.libraries.pipe.exceptions import EntryPipeNotFoundError
from pipelex.mthds_parsing.parser import MthdsParser
@@ -124,9 +126,16 @@ async def validate_bundle(
caller_identity: CallerIdentity | None = None,
library_dirs_are_callers: bool = False,
collect_dry_run_warnings: bool = False,
+ bundle_resources: BundleResources | None = None,
) -> ValidateBundleResult:
"""Load one bundle into a fresh library and dry-run its pipes.
+ ``bundle_resources`` are the files submitted with ``mthds_contents`` that their steps read, such as a
+ ``PipeDocGen`` step's ``template_file``, keyed by canonical bundle-relative path (see
+ ``make_bundle_resources``). A file none of the submitted steps names is refused with a
+ ``BundleResourceNotNamedError``, and files read at load that total more than ``MAX_LOAD_STAGE_RESOURCES_BYTES`` with
+ a ``BundleResourcesTooLargeError``, neither a verdict. A bundle file reads beside itself on disk and takes none.
+
``caller_identity`` is who asked for the validation, when the host knows it; the dry-run
sweep is attributed to that caller (see ``BundleValidator.validate_pipes``). ``None``
inherits the caller already in scope, and with none the sweep belongs to nobody.
@@ -175,6 +184,10 @@ async def validate_bundle(
msg = "mthds_sources, when provided, must be a per-item source list matching mthds_contents in length"
raise PipelexUnexpectedError(msg)
+ if bundle_resources and mthds_contents is None:
+ msg = "bundle_resources accompany mthds_contents: a bundle file reads the files beside it on disk"
+ raise PipelexUnexpectedError(msg)
+
library_manager = get_library_manager()
library_id, library = library_manager.open_library()
success = False
@@ -192,7 +205,9 @@ async def validate_bundle(
host_library_withholding: AbstractContextManager[None]
if mthds_contents is not None and not library_dirs_are_callers:
host_library_withholding = withholding_host_library_files(
- library_dirs=effective_dirs, caller_sources=[source for source in mthds_sources or [] if source is not None]
+ library_dirs=effective_dirs,
+ caller_sources=[source for source in mthds_sources or [] if source is not None],
+ caller_contents=mthds_contents,
)
else:
host_library_withholding = nullcontext()
@@ -212,6 +227,9 @@ async def validate_bundle(
MthdsParser.make_pipelex_bundle_blueprint(mthds_content=content, mthds_source=source)
for content, source in zip(mthds_contents, content_sources, strict=True)
]
+ if bundle_resources:
+ check_bundle_resources(blueprints=loaded_blueprints, bundle_resources=bundle_resources)
+ library_manager.add_bundle_resources(library_id=library_id, bundle_resources=bundle_resources)
loaded_pipes = library_manager.load_from_blueprints(library_id=library_id, blueprints=loaded_blueprints)
dry_run_results = await BundleValidator().validate_pipes(
pipes=_pipes_to_dry_run(loaded_pipes, dry_run_pipe_codes=dry_run_pipe_codes),
diff --git a/pipelex/pipeline/validate_bundle_translation.py b/pipelex/pipeline/validate_bundle_translation.py
index 0f5d5b9ce..11bbc96a1 100644
--- a/pipelex/pipeline/validate_bundle_translation.py
+++ b/pipelex/pipeline/validate_bundle_translation.py
@@ -9,14 +9,17 @@ below the validation service in the import graph (``validate_bundle`` → ``bund
``execution_seams``), and both must reach the same translation.
"""
-from collections.abc import Collection, Generator, Sequence
+import tomllib
+from collections.abc import Collection, Generator, Iterable, Sequence
from contextlib import contextmanager
from pathlib import Path
+from typing import NamedTuple, cast
from pydantic import ValidationError
from pipelex.base_exceptions import PipelexError, SecurityError, error_domain_is_input
from pipelex.config import get_config
+from pipelex.core.bundle_resources.exceptions import BundleResourceNotNamedError, BundleResourcesTooLargeError
from pipelex.core.exceptions import DryRunFailureErrorData, PipelexBundleBlueprintValidationErrorData, PipesAndConceptValidationErrorData
from pipelex.core.pipes.exceptions import (
PipeFactoryError,
@@ -93,8 +96,9 @@ def translate_to_validate_bundle_error() -> Generator[None, None, None]:
as a crash or a no-verdict fault. Only a failure of the tool or its environment — a ``config`` or
``runtime`` fault, an unclassified one, anything that is not a ``PipelexError`` — propagates as
no verdict, along with the refusals that must keep their own class: ``PipeNotFoundError`` (its
- dedicated not-found handler), a ``SecurityError`` (never absorbed by a domain handler), and a
- ``ValidateBundleError`` already produced (never re-wrapped).
+ dedicated not-found handler), a ``SecurityError`` (never absorbed by a domain handler), a
+ ``ValidateBundleError`` already produced (never re-wrapped), and a ``BundleResourceNotNamedError`` or a
+ ``BundleResourcesTooLargeError``, which refuse the files submitted with the bundle rather than the bundle.
"""
try:
yield
@@ -192,10 +196,13 @@ def translate_to_validate_bundle_error() -> Generator[None, None, None]:
message=model_choice_error.message,
pipe_validation_errors=[categorize_pipe_operator_model_choice_error(model_choice_error=model_choice_error)],
) from model_choice_error
- except (ValidateBundleError, SecurityError):
- # Both are PipelexErrors the general arm below would otherwise catch. A verdict already produced
+ except (ValidateBundleError, SecurityError, BundleResourceNotNamedError, BundleResourcesTooLargeError):
+ # All are PipelexErrors the general arm below would otherwise catch. A verdict already produced
# (e.g. by a nested load) passes through as it is: re-wrapping it would flatten its items into one.
- # A security refusal is never absorbed into a domain answer, verdict included.
+ # A security refusal is never absorbed into a domain answer, verdict included. A file submitted with
+ # the bundle that none of its steps names, or files read at load too large to travel with it, are a
+ # refusal of the request, not of the bundle, so they keep their own class rather than becoming a
+ # verdict about a bundle that may well be valid.
raise
except PipelexError as refusal:
verdict = _make_refusal_verdict(refusal=refusal)
@@ -241,7 +248,9 @@ def _make_refusal_verdict(*, refusal: PipelexError) -> ValidateBundleError | Non
@contextmanager
-def withholding_host_library_files(*, library_dirs: Sequence[Path], caller_sources: Collection[str] = ()) -> Generator[None, None, None]:
+def withholding_host_library_files(
+ *, library_dirs: Sequence[Path], caller_sources: Collection[str] = (), caller_contents: Collection[str] = ()
+) -> Generator[None, None, None]:
"""Keep the files of a host's own library directories out of the verdict the block raises.
A verdict is caller-facing as a whole, and STRICT disclosure hands its message and its items to the caller
@@ -254,7 +263,10 @@ def withholding_host_library_files(*, library_dirs: Sequence[Path], caller_sourc
domain, and a message reads ``HOST_LIBRARY_FILE_PLACEHOLDER`` where it named one. The names are the ones the
load gives: each bundle file in the spelling it gives a bundle's ``source`` and resolved, and each Python file's
module name. A name the caller submitted as a content's source (``caller_sources``) is the caller's own and is
- never withheld, even where a host file is spelled the same.
+ never withheld, even where a host file is spelled the same. The template files the host's ``PipeDocGen`` steps
+ name are withheld too, in the spelling the steps give them, since their refusals, at load or in the dry run of a
+ caller's pipe that runs one, name them; a template file the caller's own content names (``caller_contents``)
+ stays, so the caller's own steps keep naming theirs.
A bundle validated from a file on the caller's own disk, or validated or run with library directories that
are the caller's, keeps every path: there the directories are the caller's to read. A package the content
@@ -263,7 +275,9 @@ def withholding_host_library_files(*, library_dirs: Sequence[Path], caller_sourc
try:
yield
except ValidateBundleError as verdict:
- host_library_names = _host_library_names(library_dirs=library_dirs) - set(caller_sources)
+ host_files = _host_library_files(library_dirs=library_dirs)
+ host_library_names = host_files.names - set(caller_sources)
+ host_library_names |= host_files.template_files - _template_files_named_in(mthds_texts=caller_contents)
if not host_library_names:
raise
withheld_verdict = verdict.withholding_files(withheld_files=host_library_names, placeholder=HOST_LIBRARY_FILE_PLACEHOLDER)
@@ -272,15 +286,62 @@ def withholding_host_library_files(*, library_dirs: Sequence[Path], caller_sourc
raise withheld_verdict from verdict.__cause__
-def _host_library_names(*, library_dirs: Sequence[Path]) -> set[str]:
- """Every name the library load gives a file of ``library_dirs``.
+def _template_files_named_in(*, mthds_texts: Iterable[str]) -> set[str]:
+ """The ``template_file`` every pipe of the given bundle texts names, read from the TOML alone.
+
+ The bundles are read as TOML rather than parsed into blueprints, because the bundle whose refusal is being
+ withheld may be the one that does not parse, and its steps' template files are still the ones its messages name.
+ """
+ template_files: set[str] = set()
+ for mthds_text in mthds_texts:
+ try:
+ document = tomllib.loads(mthds_text)
+ except tomllib.TOMLDecodeError:
+ continue
+ pipe_tables: object = document.get("pipe")
+ if not isinstance(pipe_tables, dict):
+ continue
+ for pipe_table in cast("dict[str, object]", pipe_tables).values():
+ if not isinstance(pipe_table, dict):
+ continue
+ template_file = cast("dict[str, object]", pipe_table).get("template_file")
+ if isinstance(template_file, str):
+ template_files.add(template_file)
+ return template_files
+
+
+def _bundle_text_or_none(*, library_file: Path) -> str | None:
+ """A host bundle file's text, or None for a file the load could not read either, whose refusal names no template file."""
+ try:
+ return library_file.read_text(encoding="utf-8")
+ except (OSError, UnicodeDecodeError):
+ return None
+
+
+class _HostLibraryFiles(NamedTuple):
+ """What a verdict must not name of the host's library directories."""
+
+ names: set[str]
+ """Every name the library load gives a file of the directories."""
+
+ template_files: set[str]
+ """The template files the steps of their bundles name, as the steps spell them."""
+
+
+def _host_library_files(*, library_dirs: Sequence[Path]) -> _HostLibraryFiles:
+ """Every name the library load gives a file of ``library_dirs``, and the template files their steps name, in one walk.
A bundle file is named as listed, which is the ``source`` the load gives it, and resolved; a Python file by
- the module name the function registry imports it under, which is spelled from its resolved path.
+ the module name the function registry imports it under, which is spelled from its resolved path. Each bundle file
+ is read once, for the template files its steps name.
"""
names: set[str] = set()
+ host_texts: list[str] = []
for library_file in get_pipelex_mthds_files_from_dirs(set(library_dirs)):
names.add(str(library_file))
+ host_text = _bundle_text_or_none(library_file=library_file)
+ if host_text is not None:
+ host_texts.append(host_text)
try:
names.add(str(library_file.resolve()))
except (OSError, RuntimeError):
@@ -294,4 +355,4 @@ def _host_library_names(*, library_dirs: Sequence[Path]) -> set[str]:
names.add(convert_file_path_to_module_path(python_file))
except (OSError, RuntimeError):
continue
- return names
+ return _HostLibraryFiles(names=names, template_files=_template_files_named_in(mthds_texts=host_texts))
diff --git a/pipelex/pipeline/validate_in_process.py b/pipelex/pipeline/validate_in_process.py
index bd36ae643..cd6d890b4 100644
--- a/pipelex/pipeline/validate_in_process.py
+++ b/pipelex/pipeline/validate_in_process.py
@@ -25,6 +25,7 @@ from typing import TYPE_CHECKING
from pipelex import log
from pipelex.base_exceptions import PipelexError, ValidationErrorItem
+from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager, set_current_library
from pipelex.pipe_run.dry_run_in_process import best_effort_graph_spec
from pipelex.pipe_run.dry_run_warning_collector import collecting_dry_run_warnings
@@ -54,6 +55,7 @@ async def validate_bundles_in_process(
log_context: str = "validate",
caller_identity: CallerIdentity | None = None,
library_dirs_are_callers: bool = False,
+ bundle_resources: BundleResources | None = None,
) -> PipelexValidationReport:
"""Parse, validate, and dry-run MTHDS bundles in-process; assemble the canonical report.
@@ -85,6 +87,9 @@ async def validate_bundles_in_process(
library_dirs_are_callers: Whether ``library_dirs`` are the caller's own, whose files the
verdict may name; by default they are a host's, whose files it withholds (see
``validate_bundle``).
+ bundle_resources: The files submitted with ``mthds_contents`` that their steps read, such as a
+ ``PipeDocGen`` template file, keyed by canonical bundle-relative path. The dry run reads them, so the
+ template check runs on them.
Returns:
PipelexValidationReport with the structural artifacts of a valid bundle.
@@ -101,6 +106,7 @@ async def validate_bundles_in_process(
graph_pipe_code=graph_pipe_code,
log_context=log_context,
library_dirs_are_callers=library_dirs_are_callers,
+ bundle_resources=bundle_resources,
validation_report=validation_report,
)
@@ -114,6 +120,7 @@ async def _validate_bundles_in_scope(
graph_pipe_code: str | None,
log_context: str,
library_dirs_are_callers: bool,
+ bundle_resources: BundleResources | None,
validation_report: MethodValidationReport,
) -> PipelexValidationReport:
"""The body of :func:`validate_bundles_in_process`, run inside the caller's scope and timed by `validation_report`."""
@@ -141,6 +148,7 @@ async def _validate_bundles_in_scope(
allow_signatures=allow_signatures,
library_dirs_are_callers=library_dirs_are_callers,
collect_dry_run_warnings=True,
+ bundle_resources=bundle_resources,
)
# Capture the validation library id ONCE, right after validate_bundle leaves it
# current — the graph arm and the finally must target the SAME library even if
diff --git a/pipelex/plugins/bundle_validator_registry.py b/pipelex/plugins/bundle_validator_registry.py
index 7eac883cc..ed7a9e1fc 100644
--- a/pipelex/plugins/bundle_validator_registry.py
+++ b/pipelex/plugins/bundle_validator_registry.py
@@ -36,6 +36,8 @@ if TYPE_CHECKING:
from collections.abc import Sequence
from pathlib import Path
+ from pipelex.core.bundle_resources.bundle_resource import BundleResources
+
# The verdict a validator returns: valid arm = the protocol validation report (at runtime
# always the canonical PipelexValidationReport subtype), invalid arm = the structured
# ErrorReport. Bound at runtime (not just under TYPE_CHECKING) so an external validator
@@ -73,6 +75,14 @@ class BundleValidatorProtocol(Protocol):
dropped it would graph a different pipe than the one the host asked for. A target that
does not resolve degrades the graph to ``None``, like any other graph-arm failure, and
never changes the verdict.
+
+ ``bundle_resources`` are the files submitted with ``mthds_contents`` that their steps read,
+ such as a ``PipeDocGen`` step's ``template_file``, keyed by canonical bundle-relative path,
+ each carrying its bytes inline. The validation loads the bundles with them and its dry run
+ reads them, the template check included. It is required for the same reason as
+ ``caller_identity``, with ``None`` meaning none were submitted: a validator that dropped them
+ would refuse a valid bundle for a missing file. A worker-dispatched arm carries them to its
+ worker, by reference to storage rather than inline when they are large.
"""
async def validate_bundles(
@@ -84,6 +94,7 @@ class BundleValidatorProtocol(Protocol):
library_dirs: "Sequence[Path] | None",
caller_identity: CallerIdentity | None,
graph_pipe_code: str | None,
+ bundle_resources: "BundleResources | None",
) -> BundleValidationVerdict: ...
diff --git a/pipelex/runtime_boot.py b/pipelex/runtime_boot.py
index db9d4ec7d..6c296fd47 100644
--- a/pipelex/runtime_boot.py
+++ b/pipelex/runtime_boot.py
@@ -669,6 +669,10 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
return cast("_HubSlotImplT", factory())
return default()
+ def _is_hub_slot_claimed(self, *, slot: HubSlot) -> bool:
+ """Whether a plugin claimed the slot, so that `_resolve_hub_slot` resolves it to the plugin's implementation."""
+ return self._plugin_registrar is not None and slot in self._plugin_registrar.slot_claims
+
def _teardown_plugin_callbacks(self) -> None:
"""Run the plugin-contributed teardown callbacks, LIFO.
diff --git a/subject_grants.toml b/subject_grants.toml
index f7568a117..a9df7958d 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -3816,7 +3816,7 @@ rationale = "Verb-object: loads the ref-to-entry mapping in topological order; s
param = "mthds_version_constraint"
rationale = "Predicate-style check on the version constraint; single operand."
-["pipelex/libraries/library_manager.py::_find_methods_dirs_from_blueprints"]
+["pipelex/libraries/library_manager.py::find_methods_dirs_from_blueprints"]
param = "blueprints"
rationale = "Derivation: finds dirs from the blueprints; single operand."
diff --git a/tests/data/errors/error_identity.txt b/tests/data/errors/error_identity.txt
index bc4392536..043dcd040 100644
--- a/tests/data/errors/error_identity.txt
+++ b/tests/data/errors/error_identity.txt
@@ -24,6 +24,10 @@ BindingPathUnresolvedError | Binding path unresolved | https://docs.pipelex.com/
BindingStepRunError | Binding step run | https://docs.pipelex.com/latest/errors/binding-step-run-error/
BrokenPluginError | Broken plugin | https://docs.pipelex.com/latest/errors/broken-plugin-error/
BundleElaboratorError | Bundle elaborator | https://docs.pipelex.com/latest/errors/bundle-elaborator-error/
+BundleResourceError | Bundle resource | https://docs.pipelex.com/latest/errors/bundle-resource-error/
+BundleResourceNotNamedError | Bundle resource not named | https://docs.pipelex.com/latest/errors/bundle-resource-not-named-error/
+BundleResourcePathError | Bundle resource path | https://docs.pipelex.com/latest/errors/bundle-resource-path-error/
+BundleResourcesTooLargeError | Bundle resources too large | https://docs.pipelex.com/latest/errors/bundle-resources-too-large-error/
CodegenError | Codegen error | https://docs.pipelex.com/latest/errors/codegen-error/
CodegenLockError | Codegen lock error | https://docs.pipelex.com/latest/errors/codegen-lock-error/
CodegenStampError | Codegen stamp error | https://docs.pipelex.com/latest/errors/codegen-stamp-error/
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/conftest.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/conftest.py
index b09025099..e6ddef85c 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/conftest.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/conftest.py
@@ -5,12 +5,18 @@ from pathlib import Path
import pytest
from pipelex.cogt.doc_gen.exceptions import BUILT_IN_DOC_GEN_MODEL
+from pipelex.interpreter_hub import get_interpreter_hub
from pipelex.plugins.inference_backend_registry import InferenceFamily
from pipelex.plugins.plugin_model_declarations import PluginModelDeclarations
from pipelex.runtime_hub import get_inference_backend_registry, get_models_manager, get_runtime_hub
from pipelex.system.configuration.config_loader import config_manager
from pipelex.tools.misc.toml_utils import load_toml_with_tomlkit, save_toml_to_path
-from tests.integration.pipelex.pipes.operator.pipe_doc_gen.doc_gen_helpers import StubEngines, make_doc_gen_backends, make_models_manager
+from tests.integration.pipelex.pipes.operator.pipe_doc_gen.doc_gen_helpers import (
+ StubEngines,
+ WorkerLikePipeRun,
+ make_doc_gen_backends,
+ make_models_manager,
+)
@pytest.fixture
@@ -35,6 +41,20 @@ def stub_engines() -> Iterator[StubEngines]:
get_runtime_hub().set_models_manager(models_manager=booted_models_manager)
+def remote_worker() -> Iterator[WorkerLikePipeRun]:
+ """Install a worker-like pipe run as the hub's, one that hands each job outside the process, as a boot orchestrator's does."""
+ interpreter_hub = get_interpreter_hub()
+ booted_pipe_run = interpreter_hub.get_required_pipe_run()
+ booted_leaves_process = interpreter_hub.does_pipe_run_leave_process()
+ worker = WorkerLikePipeRun(in_process_pipe_run=booted_pipe_run)
+ interpreter_hub.set_pipe_run(worker, leaves_process=True)
+ try:
+ yield worker
+ finally:
+ interpreter_hub.set_pipe_run(booted_pipe_run, leaves_process=booted_leaves_process)
+
+
@pytest.fixture
def no_engines() -> Iterator[None]:
"""A runtime with no document engine, as a host that disables the built-in one and installs no plugin."""
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/doc_gen_helpers.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/doc_gen_helpers.py
index f4224d479..5f4a8cf31 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/doc_gen_helpers.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/doc_gen_helpers.py
@@ -1,5 +1,5 @@
from pathlib import Path
-from typing import Any
+from typing import TYPE_CHECKING, Any
import pytest
from typing_extensions import override
@@ -13,6 +13,11 @@ from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.models.model_manager import ModelManager
from pipelex.config import get_config
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.interpreter_hub import get_current_library, get_library_manager, set_current_library
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
from pipelex.pipeline.exceptions import ValidateBundleError
from pipelex.pipeline.validate_bundle import validate_bundle
from pipelex.plugins.contract import PLUGIN_API_VERSION
@@ -25,6 +30,9 @@ from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.runtime_hub import get_secrets_provider
from tests.integration.pipelex.pipes.operator.pipe_doc_gen.test_data import PipeDocGenTestData
+if TYPE_CHECKING:
+ from pipelex.libraries.library_crate import LibraryCrate
+
# The engines the stubs stand in for: the built-in pdf from the layout, and two of the document generation plugin's,
# a pdf from HTML and a docx, which the stub plugin declares as the real plugin would.
BUILT_IN_STUB_SDK = "reportlab"
@@ -157,3 +165,44 @@ def write_bundle(*, directory: Path, step_fields: str) -> Path:
bundle_path = directory / "invoice.mthds"
bundle_path.write_text(PipeDocGenTestData.bundle(step_fields=step_fields), encoding="utf-8")
return bundle_path
+
+
+class WorkerLikePipeRun(PipeRunProtocol):
+ """Runs a job the way a remote worker does: loads the crate it was handed into a library of its own and runs its own pipe.
+
+ Installed as the hub's pipe run, as a boot orchestrator's plugin installs its own (the `remote_worker` fixture), it
+ receives the job only after the submitter prepared it to leave the process. Its pipe, rebuilt from the crate, finds
+ only a reference to its office template in the library it runs in, and reads the bytes back from the run's storage,
+ as a Temporal activity does. `worker_read_scope`, when given, replaces the run's read scope on the worker.
+ """
+
+ def __init__(self, *, in_process_pipe_run: PipeRunProtocol, worker_read_scope: str | None = None) -> None:
+ self._in_process_pipe_run = in_process_pipe_run
+ self.worker_read_scope = worker_read_scope
+ self.received_crates: list[LibraryCrate] = []
+
+ @override
+ async def run(self, pipe_job: PipeJob, *, delivery_assignment: DeliveryAssignment | None = None) -> PipeOutput:
+ crate = pipe_job.library_crate
+ assert crate is not None
+ self.received_crates.append(crate)
+ library_manager = get_library_manager()
+ submitter_library_id = get_current_library()
+ worker_library_id = f"worker-{pipe_job.job_metadata.run_metadata.pipeline_run_id}"
+ worker_library = library_manager.open_fresh_library(library_id=worker_library_id)
+ submitter_class_registry = library_manager.get_library(library_id=submitter_library_id).get_class_registry()
+ if submitter_class_registry is not None:
+ worker_library.set_class_registry(submitter_class_registry)
+ set_current_library(library_id=worker_library_id)
+ try:
+ library_manager.load_from_crate(library_id=worker_library_id, crate=crate, is_crate_prevalidated=True)
+ worker_pipe = library_manager.get_library(library_id=worker_library_id).pipe_library.get_required_pipe(pipe_code=pipe_job.pipe.pipe_ref)
+ job_metadata = pipe_job.job_metadata
+ if self.worker_read_scope is not None:
+ run_metadata = job_metadata.run_metadata.model_copy(update={"read_scope": self.worker_read_scope})
+ job_metadata = job_metadata.model_copy(update={"run_metadata": run_metadata})
+ worker_job = pipe_job.model_copy(update={"pipe": worker_pipe, "job_metadata": job_metadata})
+ return await self._in_process_pipe_run.run(worker_job, delivery_assignment=delivery_assignment)
+ finally:
+ set_current_library(library_id=submitter_library_id)
+ library_manager.teardown(library_id=worker_library_id)
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_data.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_data.py
index 9d283a245..220786bb7 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_data.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_data.py
@@ -64,6 +64,21 @@ output = "InvoicePdf"
PDF_LAYOUT_STEP: ClassVar[str] = 'format = "pdf"\nfilename = "invoice-{{ invoice.number }}"'
+ # A bundle of another domain whose sequence runs the test bundle's step, which a library directory holds: the step
+ # is dry-run only inside the sequence, so what it finds reaches the verdict from a pipe the sweep did not sweep itself.
+ CALLER_BUNDLE_MTHDS: ClassVar[str] = """
+domain = "doc_gen_caller"
+description = "Billing a customer with a printed invoice"
+main_pipe = "bill_customer"
+
+[pipe.bill_customer]
+type = "PipeSequence"
+description = "Print the customer's invoice"
+inputs = { invoice = "doc_gen_tests.Invoice" }
+output = "doc_gen_tests.InvoicePdf"
+steps = [{ pipe = "doc_gen_tests.print_invoice", result = "invoice_pdf" }]
+"""
+
@classmethod
def bundle(cls, *, step_fields: str) -> str:
return cls.BUNDLE_MTHDS.replace(cls.STEP_SLOT, step_fields)
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_bundle_resources.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_bundle_resources.py
new file mode 100644
index 000000000..636e02d0f
--- /dev/null
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_bundle_resources.py
@@ -0,0 +1,265 @@
+"""A bundle loaded from content strings brings the template files its PipeDocGen steps read, as the runner's routes receive it."""
+
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from pipelex.base_exceptions import PipelexError
+from pipelex.cogt.doc_gen.doc_gen_format import DocGenSource
+from pipelex.cogt.doc_gen.template_check import TemplateFinding, TemplateFindingSeverity
+from pipelex.core.bundle_resources.bundle_resource import BundleResources, make_bundle_resources
+from pipelex.core.bundle_resources.exceptions import BundleResourceNotNamedError
+from pipelex.core.stuffs.document_content import DocumentContent
+from pipelex.pipeline.exceptions import ValidateBundleError
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.pipeline.validate_bundle import validate_bundle
+from pipelex.system.pipe_run_mode import PipeRunMode
+from pipelex.system.storage_scope import DRY_RUN_STORAGE_SCOPE, LOCAL_STORAGE_SCOPE
+from tests.integration.pipelex.pipes.operator.pipe_doc_gen.doc_gen_helpers import StubEngines, WorkerLikePipeRun, write_bundle
+from tests.integration.pipelex.pipes.operator.pipe_doc_gen.test_data import PipeDocGenTestData
+
+_DOCX_TEMPLATE_STEP = 'format = "docx"\ntemplate_file = "templates/invoice.docx"'
+_DOCX_BYTES = b"PK docx template bytes"
+
+
+def _docx_resources() -> BundleResources:
+ return make_bundle_resources(files={"./templates/invoice.docx": _DOCX_BYTES})
+
+
[email protected](loop_scope="class")
+class TestPipeDocGenBundleResources:
+ async def test_a_bundle_validates_with_its_template_file_and_the_check_runs_on_its_bytes(self, stub_engines: StubEngines) -> None:
+ """The dry run hands the engine's checker the bytes submitted with the bundle, found by the path the step names."""
+ await validate_bundle(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields=_DOCX_TEMPLATE_STEP)],
+ bundle_resources=_docx_resources(),
+ )
+
+ (request,) = stub_engines.check_requests
+ assert request.template == _DOCX_BYTES
+ assert request.template_name == "templates/invoice.docx"
+
+ async def test_the_check_s_errors_on_a_submitted_template_fail_the_validation(self, stub_engines: StubEngines) -> None:
+ stub_engines.findings = [TemplateFinding(severity=TemplateFindingSeverity.ERROR, message="The tag {{ invoice.totl }} names no field")]
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields=_DOCX_TEMPLATE_STEP)],
+ bundle_resources=_docx_resources(),
+ )
+
+ assert "invoice.totl" in str(exc_info.value.to_error_report().model_dump())
+
+ @pytest.mark.usefixtures("stub_engines")
+ async def test_a_template_file_not_supplied_is_refused_saying_so(self) -> None:
+ """The refusal is the bundle's verdict, located on the step, and names the path the file is expected under."""
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(mthds_contents=[PipeDocGenTestData.bundle(step_fields=_DOCX_TEMPLATE_STEP)])
+
+ report = str(exc_info.value.to_error_report().model_dump())
+ assert "names the template file 'templates/invoice.docx', which was not supplied with its bundle" in report
+
+ @pytest.mark.usefixtures("stub_engines")
+ async def test_a_file_no_step_names_is_refused_as_the_request_s_fault_not_as_a_verdict(self) -> None:
+ resources = make_bundle_resources(files={"templates/invoice.docx": _DOCX_BYTES, "templates/logo.png": b"png"})
+
+ with pytest.raises(BundleResourceNotNamedError) as exc_info:
+ await validate_bundle(mthds_contents=[PipeDocGenTestData.bundle(step_fields=_DOCX_TEMPLATE_STEP)], bundle_resources=resources)
+
+ assert "'templates/logo.png'" in exc_info.value.message
+
+ @pytest.mark.usefixtures("stub_engines")
+ async def test_a_submitted_html_template_is_read_and_checked_when_the_method_loads(self) -> None:
+ """An HTML template file is checked against the inputs at load, as one beside a bundle on disk is."""
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields='format = "pdf"\ntemplate_file = "invoice.html"')],
+ bundle_resources=make_bundle_resources(files={"invoice.html": b"<h1>{{ invoice.numbr }}</h1>"}),
+ )
+
+ assert "numbr" in str(exc_info.value.to_error_report().model_dump())
+
+ async def test_a_run_prints_with_the_submitted_html_template(self, stub_engines: StubEngines) -> None:
+ await PipelexMTHDSProtocol(bundle_resources=make_bundle_resources(files={"invoice.html": b"<h1>{{ invoice.number }}</h1>"})).execute(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields='format = "pdf"\ntemplate_file = "invoice.html"')],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+
+ (job,) = stub_engines.engine.jobs
+ assert job.source == DocGenSource.HTML
+ assert job.html == "<h1>INV-2026-0142</h1>"
+
+ async def test_a_run_hands_the_engine_the_submitted_template_file(self, stub_engines: StubEngines) -> None:
+ result = await PipelexMTHDSProtocol(bundle_resources=_docx_resources()).execute(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields=_DOCX_TEMPLATE_STEP)],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+
+ document = result.pipe_output.main_stuff_as(content_type=DocumentContent)
+ assert document.filename is not None
+ assert document.filename.endswith(".docx")
+ (job,) = stub_engines.engine.jobs
+ assert job.source == DocGenSource.TEMPLATE_FILE
+ assert job.template == _DOCX_BYTES
+ assert job.template_name == "templates/invoice.docx"
+ assert job.data["invoice"]["number"] == "INV-2026-0142"
+
+ async def test_a_dry_run_checks_the_submitted_template_file_and_prints_nothing(self, stub_engines: StubEngines) -> None:
+ result = await PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, bundle_resources=_docx_resources()).execute(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields=_DOCX_TEMPLATE_STEP)],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+
+ document = result.pipe_output.main_stuff_as(content_type=DocumentContent)
+ assert document.filename is not None
+ assert document.filename.endswith(".docx")
+ (request,) = stub_engines.check_requests
+ assert request.template == _DOCX_BYTES
+ assert stub_engines.engine.jobs == []
+
+ @pytest.mark.parametrize(
+ ("pipe_run_mode", "storage_scope", "is_stored"),
+ [
+ pytest.param(PipeRunMode.LIVE, LOCAL_STORAGE_SCOPE, True, id="live"),
+ pytest.param(PipeRunMode.DRY, LOCAL_STORAGE_SCOPE, True, id="dry"),
+ pytest.param(PipeRunMode.DRY, DRY_RUN_STORAGE_SCOPE, False, id="dry_run_placeholder_scope"),
+ ],
+ )
+ async def test_a_run_handed_to_a_remote_worker_reads_its_template_from_the_run_s_storage(
+ self,
+ stub_engines: StubEngines,
+ remote_worker: WorkerLikePipeRun,
+ pipe_run_mode: PipeRunMode,
+ storage_scope: str,
+ is_stored: bool,
+ ) -> None:
+ """The job leaves the process carrying a reference, and the worker's step reads the stored bytes, to print or to check.
+
+ A run whose storage scope is the dry-run placeholder stores nothing, so its template travels inline.
+ """
+ await PipelexMTHDSProtocol(pipe_run_mode=pipe_run_mode, storage_scope=storage_scope, bundle_resources=_docx_resources()).execute(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields=_DOCX_TEMPLATE_STEP)],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+
+ (received_crate,) = remote_worker.received_crates
+ received = received_crate.bundle_resources["templates/invoice.docx"]
+ assert (received.uri is not None) == is_stored
+ assert received.is_inline != is_stored
+ templates_read = [request.template for request in stub_engines.check_requests] + [job.template for job in stub_engines.engine.jobs]
+ assert templates_read == [_DOCX_BYTES]
+
+ async def test_a_worker_reads_no_stored_template_outside_the_run_s_read_scope_even_to_check_it(
+ self, stub_engines: StubEngines, remote_worker: WorkerLikePipeRun
+ ) -> None:
+ """The dry run's template check reads a stored template under the run's read scope, as the print stage does."""
+ remote_worker.worker_read_scope = "another-tenant"
+
+ with pytest.raises(PipelexError) as exc_info:
+ await PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, bundle_resources=_docx_resources()).execute(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields=_DOCX_TEMPLATE_STEP)],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+
+ assert "Refused to read the template file 'templates/invoice.docx'" in str(exc_info.value.to_error_report().model_dump())
+ assert stub_engines.check_requests == []
+
+ @pytest.mark.parametrize(
+ ("step_fields", "template_name", "template_bytes"),
+ [
+ pytest.param(_DOCX_TEMPLATE_STEP, "templates/invoice.docx", _DOCX_BYTES, id="office"),
+ pytest.param('format = "pdf"\ntemplate_file = "invoice.html"', "invoice.html", b"<h1>{{ invoice.number }}</h1>", id="html"),
+ ],
+ )
+ async def test_a_caller_s_source_label_never_reads_beside_it_on_this_host(
+ self, stub_engines: StubEngines, tmp_path: Path, step_fields: str, template_name: str, template_bytes: bytes
+ ) -> None:
+ """A label naming a real bundle file on this host, beside which the template file exists, still reads nothing from disk."""
+ bundle_path = write_bundle(directory=tmp_path, step_fields=step_fields)
+ (tmp_path / template_name).parent.mkdir(parents=True, exist_ok=True)
+ (tmp_path / template_name).write_bytes(template_bytes)
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(mthds_contents=[bundle_path.read_text(encoding="utf-8")], mthds_sources=[str(bundle_path)])
+
+ assert "which was not supplied with its bundle" in str(exc_info.value.to_error_report().model_dump())
+ assert stub_engines.check_requests == []
+
+ @pytest.mark.parametrize("pipe_run_mode", [pytest.param(PipeRunMode.LIVE, id="live"), pytest.param(PipeRunMode.DRY, id="dry")])
+ async def test_a_bundle_its_caller_read_from_a_file_here_reads_its_template_beside_that_file(
+ self, stub_engines: StubEngines, tmp_path: Path, pipe_run_mode: PipeRunMode
+ ) -> None:
+ """A caller that read the bundle's file itself, and names it in `bundle_uris`, runs it as a bundle loaded from that file.
+
+ Nothing loads the bundle's directory, so the contents are what loads, and their step reads the template beside the file.
+ """
+ bundle_path = write_bundle(directory=tmp_path, step_fields=_DOCX_TEMPLATE_STEP)
+ (tmp_path / "templates").mkdir()
+ (tmp_path / "templates" / "invoice.docx").write_bytes(_DOCX_BYTES)
+
+ await PipelexMTHDSProtocol(bundle_uris=[str(bundle_path)], pipe_run_mode=pipe_run_mode).execute(
+ mthds_contents=[bundle_path.read_text(encoding="utf-8")],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+
+ match pipe_run_mode:
+ case PipeRunMode.LIVE:
+ assert [job.template for job in stub_engines.engine.jobs] == [_DOCX_BYTES]
+ case PipeRunMode.DRY:
+ assert [check_request.template for check_request in stub_engines.check_requests] == [_DOCX_BYTES]
+ assert stub_engines.engine.jobs == []
+
+ async def test_a_source_the_bundle_s_text_writes_on_its_step_never_reads_beside_it_on_this_host(
+ self, stub_engines: StubEngines, tmp_path: Path
+ ) -> None:
+ """The step's own `source` key is overwritten when the pipe is built, on the validation path and on the run path."""
+ bundle_path = write_bundle(directory=tmp_path, step_fields=_DOCX_TEMPLATE_STEP)
+ (tmp_path / "templates").mkdir()
+ (tmp_path / "templates" / "invoice.docx").write_bytes(_DOCX_BYTES)
+ content = PipeDocGenTestData.bundle(step_fields=f'{_DOCX_TEMPLATE_STEP}\nsource = "{bundle_path}"')
+
+ with pytest.raises(ValidateBundleError) as validate_exc_info:
+ await validate_bundle(mthds_contents=[content])
+ with pytest.raises(ValidateBundleError) as run_exc_info:
+ await PipelexMTHDSProtocol().execute(mthds_contents=[content], inputs=PipeDocGenTestData.INVOICE_INPUTS)
+
+ for exc_info in (validate_exc_info, run_exc_info):
+ assert "which was not supplied with its bundle" in str(exc_info.value.to_error_report().model_dump())
+ assert stub_engines.check_requests == []
+ assert stub_engines.engine.jobs == []
+
+ @pytest.mark.parametrize("is_remote", [pytest.param(False, id="in_process"), pytest.param(True, id="remote_worker")])
+ async def test_a_step_read_from_a_file_here_reads_beside_it_and_never_a_submitted_file(
+ self, request: pytest.FixtureRequest, stub_engines: StubEngines, tmp_path: Path, is_remote: bool
+ ) -> None:
+ """A host library's step and a submitted step naming the same template path each read their own file, here and on a worker.
+
+ A worker's library holds the crate's submitted files before it builds any pipe, the host library's included,
+ so the host step must not take the submitted file for its own there either.
+ """
+ host_dir = tmp_path / "host"
+ host_dir.mkdir()
+ step_fields = 'format = "docx"\ntemplate_file = "invoice.docx"'
+ (host_dir / "invoice.mthds").write_text(
+ PipeDocGenTestData.bundle(step_fields=step_fields).replace("doc_gen_tests", "doc_gen_host"), encoding="utf-8"
+ )
+ (host_dir / "invoice.docx").write_bytes(b"host template")
+ submitted = make_bundle_resources(files={"invoice.docx": b"submitted template"})
+ host_inputs: dict[str, Any] = {"invoice": {**PipeDocGenTestData.INVOICE_INPUTS["invoice"], "concept": "doc_gen_host.Invoice"}}
+ runs: list[tuple[str, dict[str, Any]]] = [
+ ("doc_gen_host.print_invoice", host_inputs),
+ ("doc_gen_tests.print_invoice", PipeDocGenTestData.INVOICE_INPUTS),
+ ]
+
+ if is_remote:
+ request.getfixturevalue("remote_worker")
+
+ for pipe_code, inputs in runs:
+ await PipelexMTHDSProtocol(library_dirs=[str(host_dir)], bundle_resources=submitted).execute(
+ pipe_code=pipe_code,
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields=step_fields)],
+ inputs=inputs,
+ )
+
+ assert [job.template for job in stub_engines.engine.jobs] == [b"host template", b"submitted template"]
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_host_template_files.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_host_template_files.py
new file mode 100644
index 000000000..7f3db0b54
--- /dev/null
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_host_template_files.py
@@ -0,0 +1,94 @@
+from collections.abc import Iterator
+from pathlib import Path
+
+import pytest
+
+from pipelex.cogt.doc_gen.template_check import TemplateFinding, TemplateFindingSeverity
+from pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager, set_current_library
+from pipelex.pipeline.exceptions import ValidateBundleError
+from pipelex.pipeline.validate_bundle_translation import HOST_LIBRARY_FILE_PLACEHOLDER
+from pipelex.pipeline.validate_in_process import validate_bundles_in_process
+from tests.integration.pipelex.pipes.operator.pipe_doc_gen.doc_gen_helpers import StubEngines, write_bundle
+from tests.integration.pipelex.pipes.operator.pipe_doc_gen.test_data import PipeDocGenTestData
+
+_HOST_TEMPLATE_FILE = "templates/host-invoice.docx"
+
+
[email protected](autouse=True)
+def restore_the_outer_library() -> Iterator[None]:
+ """Tear down whatever library a refused validation left current, and restore the one the test started in."""
+ outer_library_id = get_current_library_id_or_none()
+ yield
+ leaked_library_id = get_current_library_id_or_none()
+ if leaked_library_id is not None and leaked_library_id != outer_library_id:
+ get_library_manager().teardown(library_id=leaked_library_id)
+ if outer_library_id is not None:
+ set_current_library(library_id=outer_library_id)
+ else:
+ clear_current_library()
+
+
+async def _caller_verdict(*, mthds_content: str, host_dir: Path) -> str:
+ """The invalid verdict, as JSON, of submitted content validated beside a host's library directory."""
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundles_in_process(mthds_contents=[mthds_content], library_dirs=[host_dir])
+ return exc_info.value.to_error_report().model_dump_json()
+
+
[email protected](loop_scope="class")
+class TestPipeDocGenHostTemplateFiles:
+ async def test_a_host_step_s_template_check_error_names_no_template_file(self, tmp_path: Path, stub_engines: StubEngines) -> None:
+ """A caller's sequence runs a host step whose template the engine refuses: the verdict says so without naming the file."""
+ (tmp_path / "templates").mkdir()
+ (tmp_path / _HOST_TEMPLATE_FILE).write_bytes(b"docx bytes")
+ write_bundle(directory=tmp_path, step_fields=f'format = "docx"\ntemplate_file = "{_HOST_TEMPLATE_FILE}"')
+ stub_engines.findings = [TemplateFinding(severity=TemplateFindingSeverity.ERROR, message="The tag {{ invoice.totl }} names no field")]
+
+ verdict = await _caller_verdict(mthds_content=PipeDocGenTestData.CALLER_BUNDLE_MTHDS, host_dir=tmp_path)
+
+ assert "invoice.totl" in verdict
+ assert HOST_LIBRARY_FILE_PLACEHOLDER in verdict
+ assert "host-invoice" not in verdict
+
+ @pytest.mark.parametrize(
+ "step_fields",
+ [
+ f'format = "docx"\ntemplate_file = "{_HOST_TEMPLATE_FILE}"',
+ 'format = "docx"\ntemplate_file = "templates/host-invoice.xlsx"',
+ ],
+ ids=["missing_file", "wrong_suffix"],
+ )
+ @pytest.mark.usefixtures("stub_engines")
+ async def test_a_host_step_s_load_refusal_names_no_template_file(self, tmp_path: Path, step_fields: str) -> None:
+ """A host step refused while the host's library loads beside submitted content names no template file of it."""
+ write_bundle(directory=tmp_path, step_fields=step_fields)
+
+ verdict = await _caller_verdict(mthds_content=PipeDocGenTestData.CALLER_BUNDLE_MTHDS, host_dir=tmp_path)
+
+ assert HOST_LIBRARY_FILE_PLACEHOLDER in verdict
+ assert "host-invoice" not in verdict
+
+ @pytest.mark.usefixtures("stub_engines")
+ async def test_a_host_step_naming_its_template_file_by_an_absolute_path_names_no_host_path(self, tmp_path: Path) -> None:
+ """The blueprint refuses the absolute path without repeating it, so the verdict carries nothing of the host's disk."""
+ (tmp_path / "invoice.docx").write_bytes(b"docx bytes")
+ write_bundle(directory=tmp_path, step_fields=f'format = "docx"\ntemplate_file = "{(tmp_path / "invoice.docx").resolve()}"')
+
+ verdict = await _caller_verdict(mthds_content=PipeDocGenTestData.CALLER_BUNDLE_MTHDS, host_dir=tmp_path)
+
+ assert "a path relative to the bundle's file, inside its bundle, and this one cannot be: it is absolute" in verdict
+ assert str(tmp_path.resolve()) not in verdict
+ assert str(tmp_path) not in verdict
+
+ @pytest.mark.usefixtures("stub_engines")
+ async def test_a_caller_step_keeps_naming_a_template_file_a_host_step_names_too(self, tmp_path: Path) -> None:
+ """A template file the submitted content names is the caller's own, so its refusal keeps naming it."""
+ (tmp_path / "invoice.docx").write_bytes(b"docx bytes")
+ write_bundle(directory=tmp_path, step_fields='format = "docx"\ntemplate_file = "invoice.docx"')
+ own_bundle = PipeDocGenTestData.bundle(step_fields='format = "docx"\ntemplate_file = "invoice.docx"').replace("doc_gen_tests", "doc_gen_own")
+
+ verdict = await _caller_verdict(mthds_content=own_bundle, host_dir=tmp_path)
+
+ assert "which was not supplied with its bundle" in verdict
+ assert "'invoice.docx'" in verdict
+ assert HOST_LIBRARY_FILE_PLACEHOLDER not in verdict
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py
index 5677abb1d..e00e53b63 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_load_refusals.py
@@ -64,10 +64,10 @@ class TestPipeDocGenLoadRefusals:
assert "built into Pipelex and disabled" in report
@pytest.mark.usefixtures("stub_engines")
- async def test_a_template_file_in_a_bundle_loaded_from_a_string_is_refused(self) -> None:
- """A bundle loaded from a string has no directory to find a template file in."""
+ async def test_a_template_file_not_supplied_with_a_bundle_loaded_from_a_string_is_refused(self) -> None:
+ """A bundle loaded from a string has no directory to find a template file in, so the file must come with it."""
report = await refusal_report(step_fields='format = "docx"\ntemplate_file = "invoice.docx"')
- assert "not loaded from a file on disk" in report
+ assert "which was not supplied with its bundle" in report
@pytest.mark.usefixtures("stub_engines")
@pytest.mark.parametrize(
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_files.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_files.py
index e1cc36b8b..8e34d299b 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_files.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_files.py
@@ -4,14 +4,30 @@ import pytest
from pipelex.cogt.doc_gen.input_shape import InputShapeKind
from pipelex.cogt.doc_gen.template_check import TemplateFinding, TemplateFindingSeverity
+from pipelex.core.bundle_resources.bundle_resource import make_bundle_resources
+from pipelex.pipe_operators.doc_gen.pipe_doc_gen import PipeDocGen
from pipelex.pipeline.exceptions import ValidateBundleError
-from pipelex.pipeline.validate_bundle import validate_bundle
+from pipelex.pipeline.validate_bundle import ValidateBundleResult, validate_bundle
from tests.integration.pipelex.pipes.operator.pipe_doc_gen.doc_gen_helpers import StubEngines, write_bundle
from tests.integration.pipelex.pipes.operator.pipe_doc_gen.test_data import PipeDocGenTestData
_DOCX_TEMPLATE_STEP = 'format = "docx"\ntemplate_file = "invoice.docx"'
+async def _validate_with_html_template_file(*, directory: Path, load: str, template_bytes: bytes) -> ValidateBundleResult:
+ """Validate the test bundle with a pdf step naming `invoice.html`, loaded from its directory or sent as content with the file."""
+ step_fields = 'format = "pdf"\ntemplate_file = "invoice.html"'
+ match load:
+ case "from_its_directory":
+ (directory / "invoice.html").write_bytes(template_bytes)
+ return await validate_bundle(mthds_file_path=write_bundle(directory=directory, step_fields=step_fields))
+ case _:
+ return await validate_bundle(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields=step_fields)],
+ bundle_resources=make_bundle_resources(files={"invoice.html": template_bytes}),
+ )
+
+
@pytest.mark.asyncio(loop_scope="class")
class TestPipeDocGenTemplateFiles:
@pytest.mark.usefixtures("stub_engines")
@@ -19,15 +35,20 @@ class TestPipeDocGenTemplateFiles:
("template_file", "expected"),
[
("invoice.docx", "does not exist"),
- ("../outside.docx", "outside its bundle's directory"),
+ ("../outside.docx", "it leaves the bundle through a '..' segment"),
+ ("linked.docx", "outside its bundle's directory"),
],
- ids=["missing_file", "outside_the_bundle"],
+ ids=["missing_file", "outside_the_bundle", "linked_outside_the_bundle"],
)
async def test_a_template_file_is_found_inside_the_bundle_s_directory(self, tmp_path: Path, template_file: str, expected: str) -> None:
- """A template file must exist beside the bundle, and a path out of the bundle's directory is refused even when the file exists."""
+ """A template file must exist beside the bundle, and a path out of the bundle's directory is refused even when the file exists.
+
+ The blueprint refuses a path written out of the bundle, and the load one that leads out of it through a link.
+ """
bundle_dir = tmp_path / "bundle"
bundle_dir.mkdir()
(tmp_path / "outside.docx").write_bytes(b"docx")
+ (bundle_dir / "linked.docx").symlink_to(tmp_path / "outside.docx")
bundle_path = write_bundle(directory=bundle_dir, step_fields=f'format = "docx"\ntemplate_file = "{template_file}"')
with pytest.raises(ValidateBundleError) as exc_info:
@@ -36,16 +57,40 @@ class TestPipeDocGenTemplateFiles:
assert expected in str(exc_info.value.to_error_report().model_dump())
@pytest.mark.usefixtures("stub_engines")
- async def test_an_html_template_file_that_is_not_utf8_is_refused_naming_it(self, tmp_path: Path) -> None:
- """A template saved in another encoding is refused at load, naming the file, rather than crashing the load."""
- (tmp_path / "invoice.html").write_bytes("<p>Facture émise</p>".encode("latin-1"))
- bundle_path = write_bundle(directory=tmp_path, step_fields='format = "pdf"\ntemplate_file = "invoice.html"')
+ async def test_a_template_file_named_by_an_absolute_path_is_refused_without_repeating_it(self, tmp_path: Path) -> None:
+ """Even an absolute path to a file inside the bundle's directory is refused, and the refusal never carries the host's path."""
+ (tmp_path / "invoice.docx").write_bytes(b"docx")
+ absolute_template = tmp_path / "invoice.docx"
+ bundle_path = write_bundle(directory=tmp_path, step_fields=f'format = "docx"\ntemplate_file = "{absolute_template}"')
with pytest.raises(ValidateBundleError) as exc_info:
await validate_bundle(mthds_file_path=bundle_path)
+ report = str(exc_info.value.to_error_report().model_dump())
+ assert "a path relative to the bundle's file, inside its bundle, and this one cannot be: it is absolute" in report
+ assert str(absolute_template) not in report
+
+ @pytest.mark.usefixtures("stub_engines")
+ @pytest.mark.parametrize("load", ["from_its_directory", "from_content"])
+ async def test_an_html_template_file_that_is_not_utf8_is_refused_naming_it(self, tmp_path: Path, load: str) -> None:
+ """A template saved in another encoding is refused at load, naming the file, rather than crashing the load, however it came."""
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await _validate_with_html_template_file(directory=tmp_path, load=load, template_bytes="<p>Facture émise</p>".encode("latin-1"))
+
assert "could not read the template file 'invoice.html' as UTF-8 text" in str(exc_info.value.to_error_report().model_dump())
+ @pytest.mark.usefixtures("stub_engines")
+ @pytest.mark.parametrize("load", ["from_its_directory", "from_content"])
+ async def test_an_html_template_file_reads_the_same_whichever_way_its_bundle_loads(self, tmp_path: Path, load: str) -> None:
+ """The step holds the file's text as written, its line endings included, read from disk as from the submitted bytes."""
+ template_bytes = b"<h1>{{ invoice.number }}</h1>\r\n<p>Due on receipt</p>\r\n"
+
+ result = await _validate_with_html_template_file(directory=tmp_path, load=load, template_bytes=template_bytes)
+
+ (pipe,) = result.pipes
+ assert isinstance(pipe, PipeDocGen)
+ assert pipe.template == template_bytes.decode("utf-8")
+
async def test_the_engine_s_checker_findings_fail_the_dry_run(self, tmp_path: Path, stub_engines: StubEngines) -> None:
"""The dry run hands the checker the file's bytes, the inputs' shapes and the mock data, and an error finding fails it."""
(tmp_path / "invoice.docx").write_bytes(b"docx bytes")
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_warnings.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_warnings.py
index 5f79a12db..771a3b6f8 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_warnings.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_template_warnings.py
@@ -24,6 +24,7 @@ from pipelex.pipeline.validate_in_process import validate_bundles_in_process
from pipelex.pipeline.validation_render import format_validate_markdown
from pipelex.validation_error_types import TemplateCheckErrorType
from tests.integration.pipelex.pipes.operator.pipe_doc_gen.doc_gen_helpers import StubEngines, write_bundle
+from tests.integration.pipelex.pipes.operator.pipe_doc_gen.test_data import PipeDocGenTestData
if TYPE_CHECKING:
from pytest_mock import MockType
@@ -50,20 +51,7 @@ _NOTES_LOG_FIELDS = {
"finding_location": "the field notes on page 2",
}
-# A bundle of another domain whose sequence runs the templated step, which a library directory holds: the step is
-# dry-run only inside the sequence, so its warning reaches the verdict from a pipe the sweep did not sweep itself.
-_CALLER_MTHDS = """
-domain = "doc_gen_caller"
-description = "Billing a customer with a printed invoice"
-main_pipe = "bill_customer"
-
-[pipe.bill_customer]
-type = "PipeSequence"
-description = "Print the customer's invoice"
-inputs = { invoice = "doc_gen_tests.Invoice" }
-output = "doc_gen_tests.InvoicePdf"
-steps = [{ pipe = "doc_gen_tests.print_invoice", result = "invoice_pdf" }]
-"""
+_CALLER_MTHDS = PipeDocGenTestData.CALLER_BUNDLE_MTHDS
@pytest.fixture(autouse=True)
@@ -116,19 +104,31 @@ class TestPipeDocGenTemplateWarnings:
assert len(stub_engines.check_requests) == 2
assert _template_check_log_calls(warning_spy) == []
- @pytest.mark.parametrize("spelling", ["relative", "absolute"])
- async def test_the_protocol_report_withholds_a_host_template_file(self, tmp_path: Path, stub_engines: StubEngines, spelling: str) -> None:
+ async def test_a_finding_with_an_empty_location_is_reported_with_no_location(self, tmp_path: Path, stub_engines: StubEngines) -> None:
+ """An engine that answers an empty location says nowhere in particular, so the item's `template_location` is unset."""
+ _write_templated_bundle(directory=tmp_path, stub_engines=stub_engines)
+ stub_engines.findings = [_NOTES_FINDING.model_copy(update={"location": ""})]
+
+ report = await validate_bundles_in_process(mthds_contents=[_CALLER_MTHDS], library_dirs=[tmp_path], library_dirs_are_callers=True)
+
+ template_warnings = [warning for warning in report.warnings if warning.error_type == TemplateCheckErrorType.TEMPLATE_CHECK_WARNING]
+ assert template_warnings == [
+ _NOTES_WARNING.model_copy(
+ update={"template_location": None, "message": "PipeDocGen 'print_invoice', template file 'invoice.docx': The notes are never printed"}
+ )
+ ]
+
+ async def test_the_protocol_report_withholds_a_host_template_file(self, tmp_path: Path, stub_engines: StubEngines) -> None:
"""A step of the host's library directories is found by its code and domain, and its template file is not named.
Submitted content beside library directories that are the host's names none of their files, so the warning carries
- no `template_file` and its message reads the placeholder where it named one, however the step spells it, its
- absolute path on the host included. The engine's own words about the template stay.
+ no `template_file` and its message reads the placeholder where it named one. The engine's own words about the
+ template stay. A step cannot name its template file by an absolute path, which its blueprint refuses.
"""
template_path = (tmp_path / "invoice.docx").resolve()
template_path.write_bytes(b"docx bytes")
stub_engines.findings = [_NOTES_FINDING]
- template_file = str(template_path) if spelling == "absolute" else "invoice.docx"
- write_bundle(directory=tmp_path, step_fields=f'format = "docx"\ntemplate_file = "{template_file}"')
+ write_bundle(directory=tmp_path, step_fields=_DOCX_TEMPLATE_STEP)
report = await validate_bundles_in_process(mthds_contents=[_CALLER_MTHDS], library_dirs=[tmp_path])
diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py
index 81db5ff13..cd2ec817c 100644
--- a/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py
+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_read_coverage.py
@@ -44,6 +44,7 @@ from pipelex.cogt.judgment.judgment_models import JudgmentPrompt, YesNoQuestion
from pipelex.cogt.judgment.judgment_setting import JudgmentSetting
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.core.bundle_resources.bundle_resource import BundleResource
from pipelex.system.job_metadata import JobMetadata, RunMetadata
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.tools.uri.uri_read_scope import UriReference
@@ -62,7 +63,8 @@ URL_FIELD_INVENTORY: dict[type[BaseModel], set[str]] = {
ExtractAssignment: {"extract_input.image_uri", "extract_input.document_uri"},
RenderPageViewsAssignment: {"document_uri"},
# A section's blocks hold blocks again: the walk names the images inside one section and stops where a section would enclose itself.
- RenderDocumentAssignment: {"composition.layout.blocks.url", "composition.layout.blocks.blocks.url"},
+ # A template file submitted with the bundle and stored with the run is read back by the print stage too.
+ RenderDocumentAssignment: {"composition.layout.blocks.url", "composition.layout.blocks.blocks.url", "composition.template_resource.uri"},
SearchAssignment: set(),
SearchObjectAssignment: set(),
JudgmentAssignment: {"prompt.images.uri", "prompt.documents.uri"},
@@ -236,6 +238,27 @@ class TestAssignmentReadCoverage:
referenced_uris = sample.referenced_uris()
assert {uri_reference.uri for uri_reference in referenced_uris} == expected_uris, assignment_class.__name__
+ def test_a_document_printed_from_a_stored_template_file_declares_it(self) -> None:
+ """A composition holds a layout or a template file, never both, so the template file's field gets a sample of its own."""
+ stored_template = BundleResource.make_inline(content=b"template").stored_at(uri="pipelex-storage://s/assets/bundle/invoice.docx")
+ assignment = RenderDocumentAssignment(
+ job_metadata=_job_metadata(),
+ cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),
+ composition=DocumentComposition(
+ format=DocGenFormat.DOCX,
+ source=DocGenSource.TEMPLATE_FILE,
+ filename="invoice.docx",
+ title="Invoice",
+ template_resource=stored_template,
+ template_name="templates/invoice.docx",
+ ),
+ doc_gen_setting=DocGenSetting(model="pipelex-docx"),
+ )
+
+ referenced_uris = assignment.referenced_uris()
+
+ assert [uri_reference.uri for uri_reference in referenced_uris] == ["pipelex-storage://s/assets/bundle/invoice.docx"]
+
def test_the_extract_input_declares_an_image_too(self) -> None:
referenced_uris = ExtractInput(image_uri="pipelex-storage://s/scan.png").referenced_uris()
assert [uri_reference.uri for uri_reference in referenced_uris] == ["pipelex-storage://s/scan.png"]
diff --git a/tests/unit/pipelex/cogt/doc_gen/test_document_composition.py b/tests/unit/pipelex/cogt/doc_gen/test_document_composition.py
index aaf881f60..d52bd63c2 100644
--- a/tests/unit/pipelex/cogt/doc_gen/test_document_composition.py
+++ b/tests/unit/pipelex/cogt/doc_gen/test_document_composition.py
@@ -6,6 +6,8 @@ from pydantic import ValidationError
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat, DocGenSource
from pipelex.cogt.doc_gen.document_composition import DocumentComposition
from pipelex.cogt.doc_gen.layout_tree import ImageBlock, LayoutDocument, SectionBlock
+from pipelex.core.bundle_resources.bundle_resource import BundleResource
+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
class TestDocumentComposition:
@@ -46,7 +48,8 @@ class TestDocumentComposition:
)
assert composition.referenced_uris() == []
- def test_the_render_job_carries_the_template_file_s_bytes(self, tmp_path: Path) -> None:
+ @pytest.mark.asyncio
+ async def test_the_render_job_carries_the_template_file_s_bytes(self, tmp_path: Path) -> None:
template_file = tmp_path / "invoice.docx"
template_file.write_bytes(b"docx bytes")
composition = DocumentComposition(
@@ -59,12 +62,55 @@ class TestDocumentComposition:
data={"invoice": {"number": "INV-1"}},
)
- job = composition.make_render_job()
+ job = await composition.make_render_job(storage_provider=InMemoryStorageProvider())
assert job.template == b"docx bytes"
assert job.template_name == "invoice.docx"
assert job.data == {"invoice": {"number": "INV-1"}}
+ @pytest.mark.asyncio
+ async def test_a_template_file_submitted_with_its_bundle_is_read_from_its_resource(self) -> None:
+ """Inline, the bytes are the composition's own; stored, they are read back through the run's storage, under its read scope."""
+ storage_provider = InMemoryStorageProvider()
+ inline = BundleResource.make_inline(content=b"docx bytes")
+ uri = await storage_provider.store(b"docx bytes", key="org/run/assets/bundle/digest/invoice.docx")
+ stored = inline.stored_at(uri=uri)
+
+ for resource in (inline, stored):
+ composition = DocumentComposition(
+ format=DocGenFormat.DOCX,
+ source=DocGenSource.TEMPLATE_FILE,
+ filename="invoice.docx",
+ title="Invoice",
+ template_resource=resource,
+ template_name="invoice.docx",
+ )
+ job = await composition.make_render_job(storage_provider=storage_provider)
+ assert job.template == b"docx bytes"
+
+ stored_composition = DocumentComposition(
+ format=DocGenFormat.DOCX,
+ source=DocGenSource.TEMPLATE_FILE,
+ filename="invoice.docx",
+ title="Invoice",
+ template_resource=stored,
+ template_name="invoice.docx",
+ )
+ (uri_reference,) = stored_composition.referenced_uris()
+ assert uri_reference.uri == uri
+ assert uri_reference.position == "the template file 'invoice.docx'"
+
+ def test_a_template_file_read_from_disk_and_from_its_resource_is_refused(self) -> None:
+ with pytest.raises(ValidationError, match="not both"):
+ DocumentComposition(
+ format=DocGenFormat.DOCX,
+ source=DocGenSource.TEMPLATE_FILE,
+ filename="invoice.docx",
+ title="Invoice",
+ template_path="/nowhere/invoice.docx",
+ template_resource=BundleResource.make_inline(content=b"docx bytes"),
+ )
+
def test_a_composition_without_its_payload_is_refused(self) -> None:
with pytest.raises(ValidationError, match="carries its payload"):
DocumentComposition(format=DocGenFormat.DOCX, source=DocGenSource.TEMPLATE_FILE, filename="invoice.docx", title="Invoice")
diff --git a/tests/unit/pipelex/core/bundle_resources/test_bundle_resource.py b/tests/unit/pipelex/core/bundle_resources/test_bundle_resource.py
new file mode 100644
index 000000000..580a59c2d
--- /dev/null
+++ b/tests/unit/pipelex/core/bundle_resources/test_bundle_resource.py
@@ -0,0 +1,116 @@
+import base64
+import hashlib
+
+import pytest
+from pydantic import ValidationError
+
+from pipelex.core.bundle_resources.bundle_resource import (
+ BundleResource,
+ canonical_resource_path,
+ canonical_resource_path_or_none,
+ make_bundle_resources,
+)
+from pipelex.core.bundle_resources.exceptions import BundleResourceError, BundleResourcePathError
+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
+
+
+class TestCanonicalResourcePath:
+ @pytest.mark.parametrize(
+ ("path", "expected"),
+ [
+ ("invoice.docx", "invoice.docx"),
+ ("./invoice.docx", "invoice.docx"),
+ ("templates//invoice.docx", "templates/invoice.docx"),
+ ("templates/./invoice.docx", "templates/invoice.docx"),
+ ("templates/invoice.docx/", "templates/invoice.docx"),
+ ],
+ )
+ def test_a_relative_path_inside_the_bundle_has_one_spelling(self, path: str, expected: str) -> None:
+ """`.` segments and repeated slashes collapse, so two spellings of one file are one key."""
+ assert canonical_resource_path(path=path) == expected
+ assert canonical_resource_path_or_none(path=path) == expected
+
+ @pytest.mark.parametrize(
+ ("path", "reason"),
+ [
+ ("", "empty"),
+ (" ", "empty"),
+ (".", "names no file"),
+ ("/etc/invoice.docx", "absolute"),
+ ("../invoice.docx", "'..' segment"),
+ ("templates/../../invoice.docx", "'..' segment"),
+ ("templates\\invoice.docx", "backslash"),
+ ("C:invoice.docx", "colon"),
+ ("invoice\x00.docx", "NUL"),
+ ],
+ )
+ def test_a_path_that_cannot_name_a_file_inside_the_bundle_is_refused(self, path: str, reason: str) -> None:
+ """The message says why, and the lenient form answers None."""
+ with pytest.raises(BundleResourcePathError) as exc_info:
+ canonical_resource_path(path=path)
+ assert reason in exc_info.value.message
+ assert canonical_resource_path_or_none(path=path) is None
+
+
+class TestBundleResource:
+ def test_an_inline_resource_carries_its_bytes_and_their_digest(self) -> None:
+ resource = BundleResource.make_inline(content=b"docx bytes")
+
+ assert resource.is_inline
+ assert resource.size == len(b"docx bytes")
+ assert resource.sha256 == hashlib.sha256(b"docx bytes").hexdigest()
+ assert resource.content_b64 == base64.b64encode(b"docx bytes").decode("ascii")
+ assert resource.inline_bytes() == b"docx bytes"
+
+ def test_a_resource_round_trips_through_json(self) -> None:
+ """A crate carries its resources as JSON, so the bytes travel base64-encoded."""
+ resource = BundleResource.make_inline(content=bytes(range(256)))
+
+ assert BundleResource.model_validate_json(resource.model_dump_json()) == resource
+
+ @pytest.mark.parametrize(
+ "fields",
+ [
+ {"sha256": "abc", "size": 1},
+ {"sha256": "abc", "size": 1, "content_b64": "eA==", "uri": "pipelex-storage://run/assets/bundle/abc/x.docx"},
+ {"sha256": "abc", "size": 1, "uri": "https://example.com/x.docx"},
+ ],
+ ids=["neither", "both", "not_a_storage_uri"],
+ )
+ def test_a_resource_carries_its_bytes_one_way(self, fields: dict[str, object]) -> None:
+ """Inline or by a pipelex-storage reference, exactly one of the two."""
+ with pytest.raises(ValidationError):
+ BundleResource.model_validate(fields)
+
+ @pytest.mark.asyncio
+ async def test_a_stored_resource_is_read_back_and_checked_against_its_digest(self) -> None:
+ storage_provider = InMemoryStorageProvider()
+ resource = BundleResource.make_inline(content=b"docx bytes")
+ uri = await storage_provider.store(b"docx bytes", key="run/assets/bundle/digest/invoice.docx")
+ stored = resource.stored_at(uri=uri)
+
+ assert not stored.is_inline
+ assert await stored.read_bytes(storage_provider=storage_provider) == b"docx bytes"
+ with pytest.raises(BundleResourceError):
+ stored.inline_bytes()
+
+ @pytest.mark.asyncio
+ async def test_stored_bytes_that_changed_are_refused(self) -> None:
+ storage_provider = InMemoryStorageProvider()
+ uri = await storage_provider.store(b"other bytes", key="run/assets/bundle/digest/invoice.docx")
+ stored = BundleResource.make_inline(content=b"docx bytes").stored_at(uri=uri)
+
+ with pytest.raises(BundleResourceError, match="no longer matches"):
+ await stored.read_bytes(storage_provider=storage_provider)
+
+
+class TestMakeBundleResources:
+ def test_files_are_keyed_by_their_canonical_path(self) -> None:
+ resources = make_bundle_resources(files={"./invoice.docx": b"docx", "templates/report.html": b"<h1></h1>"})
+
+ assert set(resources) == {"invoice.docx", "templates/report.html"}
+ assert resources["invoice.docx"].inline_bytes() == b"docx"
+
+ def test_two_spellings_of_one_file_are_refused(self) -> None:
+ with pytest.raises(BundleResourcePathError, match="name the same file"):
+ make_bundle_resources(files={"invoice.docx": b"a", "./invoice.docx": b"b"})
diff --git a/tests/unit/pipelex/libraries/test_bundle_resources.py b/tests/unit/pipelex/libraries/test_bundle_resources.py
new file mode 100644
index 000000000..697d8e6b9
--- /dev/null
+++ b/tests/unit/pipelex/libraries/test_bundle_resources.py
@@ -0,0 +1,124 @@
+import pytest
+
+from pipelex.core.bundle_resources.bundle_resource import MAX_LOAD_STAGE_RESOURCES_BYTES, BundleResourceReadStage, make_bundle_resources
+from pipelex.core.bundle_resources.exceptions import BundleResourceNotNamedError, BundleResourcesTooLargeError
+from pipelex.libraries.bundle_resources import (
+ BUNDLE_RESOURCES_STORAGE_PREFIX,
+ bundle_resource_reads,
+ check_bundle_resources,
+ relocate_run_resources,
+)
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.libraries.library_crate_factory import LibraryCrateFactory
+from pipelex.mthds_parsing.parser import MthdsParser
+from pipelex.mthds_parsing.pipelex_bundle_blueprint import PipelexBundleBlueprint
+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
+
+_BUNDLE = """
+domain = "bundle_resources_tests"
+main_pipe = "print_report"
+
+[concept.Report]
+description = "A report"
+
+[concept.ReportFile]
+description = "The report as a file"
+refines = "Document"
+
+[pipe.print_report]
+type = "PipeDocGen"
+description = "Print the report in Word"
+inputs = { report = "Report" }
+output = "ReportFile"
+format = "docx"
+template_file = "./templates/report.docx"
+
+[pipe.print_summary]
+type = "PipeDocGen"
+description = "Print the summary as a PDF"
+inputs = { report = "Report" }
+output = "ReportFile"
+format = "pdf"
+template_file = "summary.html"
+"""
+
+
+def _blueprint() -> PipelexBundleBlueprint:
+ return MthdsParser.make_pipelex_bundle_blueprint(mthds_content=_BUNDLE)
+
+
+class TestBundleResources:
+ def test_each_named_file_is_keyed_by_its_canonical_path_with_when_it_is_read(self) -> None:
+ """An office template is read when its step runs, an HTML template when the method loads."""
+ reads = bundle_resource_reads(pipe_blueprints=(_blueprint().pipe or {}).values())
+
+ assert reads == {"templates/report.docx": BundleResourceReadStage.RUN, "summary.html": BundleResourceReadStage.LOAD}
+
+ def test_files_the_steps_name_pass_whatever_their_spelling(self) -> None:
+ resources = make_bundle_resources(files={"templates/report.docx": b"docx", "./summary.html": b"<h1></h1>"})
+
+ check_bundle_resources(blueprints=[_blueprint()], bundle_resources=resources)
+
+ def test_a_file_no_step_names_is_refused_naming_it(self) -> None:
+ resources = make_bundle_resources(files={"templates/report.docx": b"docx", "report.docx": b"docx", "logo.png": b"png"})
+
+ with pytest.raises(BundleResourceNotNamedError) as exc_info:
+ check_bundle_resources(blueprints=[_blueprint()], bundle_resources=resources)
+
+ assert "'logo.png', 'report.docx'" in exc_info.value.message
+ assert "templates/report.docx" not in exc_info.value.message
+
+ def test_files_read_at_load_over_the_bound_are_refused_naming_them_and_the_limit(self) -> None:
+ """An HTML template travels inline with the method wherever it runs, so its size is bounded when the bundle loads."""
+ resources = make_bundle_resources(files={"summary.html": b"x" * (MAX_LOAD_STAGE_RESOURCES_BYTES + 1)})
+
+ with pytest.raises(BundleResourcesTooLargeError) as exc_info:
+ check_bundle_resources(blueprints=[_blueprint()], bundle_resources=resources)
+
+ assert "('summary.html')" in exc_info.value.message
+ assert f"{MAX_LOAD_STAGE_RESOURCES_BYTES // 1024} KiB limit" in exc_info.value.message
+
+ def test_a_file_read_when_its_step_runs_is_not_bounded_by_the_load_stage_limit(self) -> None:
+ """An office template moves to the run's storage before the run leaves the process, so only the request bounds it."""
+ resources = make_bundle_resources(
+ files={"templates/report.docx": b"x" * (MAX_LOAD_STAGE_RESOURCES_BYTES + 1), "summary.html": b"<h1></h1>"},
+ )
+
+ check_bundle_resources(blueprints=[_blueprint()], bundle_resources=resources)
+
+ @pytest.mark.asyncio
+ async def test_a_file_read_when_its_step_runs_is_stored_and_carried_by_reference(self) -> None:
+ """The office template moves to the run's storage; the HTML template, read at load, stays inline."""
+ storage_provider = InMemoryStorageProvider()
+ resources = make_bundle_resources(files={"templates/report.docx": b"docx bytes", "summary.html": b"<h1></h1>"})
+ crate = LibraryCrateFactory.make_from_blueprints([_blueprint()], bundle_resources=resources)
+
+ relocated = await relocate_run_resources(crate=crate, storage_provider=storage_provider, storage_scope="org/method/run")
+
+ stored = relocated.bundle_resources["templates/report.docx"]
+ assert stored.uri == f"pipelex-storage://org/method/run/{BUNDLE_RESOURCES_STORAGE_PREFIX}/{stored.sha256}/report.docx"
+ assert stored.content_b64 is None
+ assert await stored.read_bytes(storage_provider=storage_provider) == b"docx bytes"
+ assert relocated.bundle_resources["summary.html"].is_inline
+ # The crate the run was built from is left as it was.
+ assert crate.bundle_resources["templates/report.docx"].is_inline
+ assert relocated.fingerprint == crate.fingerprint
+
+ @pytest.mark.asyncio
+ async def test_a_crate_without_resources_is_returned_as_it_is(self) -> None:
+ crate = LibraryCrateFactory.make_from_blueprints([_blueprint()])
+
+ relocated = await relocate_run_resources(crate=crate, storage_provider=InMemoryStorageProvider(), storage_scope="run")
+
+ assert relocated is crate
+
+ def test_resources_round_trip_and_stay_out_of_the_fingerprint(self) -> None:
+ """Like python_sources, the files travel with the crate but are not part of its structure."""
+ resources = make_bundle_resources(files={"templates/report.docx": b"docx bytes"})
+ bare = LibraryCrateFactory.make_from_blueprints([_blueprint()])
+ with_resources = LibraryCrateFactory.make_from_blueprints([_blueprint()], bundle_resources=resources)
+
+ restored = LibraryCrate.model_validate_json(with_resources.model_dump_json())
+
+ assert restored.bundle_resources == resources
+ assert bare.fingerprint == with_resources.fingerprint
diff --git a/tests/unit/pipelex/libraries/test_host_bundle_files.py b/tests/unit/pipelex/libraries/test_host_bundle_files.py
new file mode 100644
index 000000000..8c614f5d5
--- /dev/null
+++ b/tests/unit/pipelex/libraries/test_host_bundle_files.py
@@ -0,0 +1,75 @@
+from pathlib import Path
+
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.libraries.library_crate_factory import LibraryCrateFactory
+from pipelex.libraries.library_manager import MTHDS_METHODS_DIRNAME, find_methods_dirs_from_blueprints
+from pipelex.mthds_parsing.parser import MthdsParser
+
+
+def _bundle(*, domain: str = "host_files_tests", pipe_fields: str = "") -> str:
+ return f"""
+domain = "{domain}"
+main_pipe = "summarize"
+
+[pipe.summarize]
+type = "PipeLLM"
+description = "Summarize the text"
+inputs = {{ text = "Text" }}
+output = "Text"
+prompt = "Summarize $text"
+{pipe_fields}
+"""
+
+
+_BUNDLE = _bundle()
+
+
+def _write_bundle(*, directory: Path, mthds: str = _BUNDLE) -> Path:
+ bundle_path = directory / "bundle.mthds"
+ bundle_path.write_text(mthds, encoding="utf-8")
+ return bundle_path
+
+
+class TestHostBundleFiles:
+ def test_only_a_bundle_read_from_a_file_here_has_a_host_file(self, tmp_path: Path) -> None:
+ """A content string's label, however real a path it names, is not a file this host read the bundle from."""
+ bundle_path = _write_bundle(directory=tmp_path)
+
+ from_file = MthdsParser.make_pipelex_bundle_blueprint(bundle_path=bundle_path)
+ from_content = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=_BUNDLE, mthds_source=str(bundle_path))
+
+ assert from_file.host_bundle_file == bundle_path
+ assert from_content.host_bundle_file is None
+ assert from_content.source == str(bundle_path)
+
+ def test_the_crate_carries_the_host_file_of_its_file_loaded_pipes_only(self, tmp_path: Path) -> None:
+ """A `source` the bundle's text writes on a pipe, or a caller's label, never becomes a file to read beside."""
+ bundle_path = _write_bundle(directory=tmp_path)
+ other_dir = tmp_path / "other"
+ other_dir.mkdir()
+ content_with_a_written_source = _bundle(domain="host_files_other", pipe_fields=f'source = "{bundle_path}"')
+
+ crate = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[
+ MthdsParser.make_pipelex_bundle_blueprint(bundle_path=bundle_path),
+ MthdsParser.make_pipelex_bundle_blueprint(mthds_content=content_with_a_written_source, mthds_source=str(other_dir / "x.mthds")),
+ ]
+ )
+
+ assert crate.host_bundle_files == {"host_files_tests.summarize": str(bundle_path)}
+ assert crate.pipes["host_files_other.summarize"].source == str(bundle_path)
+ assert crate.fingerprint == LibraryCrate.model_validate_json(crate.model_dump_json()).fingerprint
+
+ def test_method_packages_are_searched_above_a_file_read_here_only(self, tmp_path: Path) -> None:
+ """Walking up from a caller's label would search this host's directories wherever the caller pointed."""
+ methods_dir = tmp_path / MTHDS_METHODS_DIRNAME
+ methods_dir.mkdir(parents=True)
+ bundle_dir = tmp_path / "project"
+ bundle_dir.mkdir()
+ bundle_path = _write_bundle(directory=bundle_dir)
+
+ from_file = MthdsParser.make_pipelex_bundle_blueprint(bundle_path=bundle_path)
+ from_content = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=_BUNDLE, mthds_source=str(bundle_path))
+
+ assert methods_dir.resolve() in find_methods_dirs_from_blueprints([from_file])
+ assert find_methods_dirs_from_blueprints([from_content]) == []
diff --git a/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py
index d50a5921f..c845eed11 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py
@@ -4,6 +4,7 @@ import pytest
from pydantic import ValidationError
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat
+from pipelex.core.bundle_resources.bundle_resource import BundleResourceReadStage, canonical_resource_path_or_none
from pipelex.pipe_operators.doc_gen.pipe_doc_gen_blueprint import PipeDocGenBlueprint
@@ -32,6 +33,17 @@ class TestPipeDocGenBlueprint:
("filename_reads_an_undeclared_input", {"filename": "invoice-{{ order.number }}"}, "Variable 'order'"),
("list_output", {"output": "Document[]"}, "produces one file"),
("broken_template", {"template": "{% for %}"}, "Could not parse"),
+ ("posix_absolute_template_file", {"format": "docx", "template_file": "/srv/methods/invoice.docx"}, "it is absolute"),
+ ("windows_rooted_template_file", {"format": "docx", "template_file": "\\methods\\invoice.docx"}, "it holds a backslash"),
+ ("drive_absolute_template_file", {"format": "docx", "template_file": "C:/methods/invoice.docx"}, "it holds a colon"),
+ ("drive_relative_template_file", {"format": "docx", "template_file": "C:invoice.docx"}, "it holds a colon"),
+ ("unc_template_file", {"format": "docx", "template_file": "\\\\server\\share\\invoice.docx"}, "it holds a backslash"),
+ ("posix_unc_template_file", {"format": "docx", "template_file": "//server/share/invoice.docx"}, "it is absolute"),
+ ("colon_in_the_name", {"format": "docx", "template_file": "invoice:v2.docx"}, "it holds a colon"),
+ ("backslash_in_the_name", {"format": "docx", "template_file": "templates\\invoice.docx"}, "it holds a backslash"),
+ ("leaves_the_bundle", {"format": "docx", "template_file": "../invoice.docx"}, "'..' segment"),
+ ("dot_dot_inside_the_bundle", {"format": "docx", "template_file": "templates/../invoice.docx"}, "'..' segment"),
+ ("empty_template_file", {"format": "docx", "template_file": " "}, "it is empty"),
],
)
def test_refused_blueprints(
@@ -43,6 +55,20 @@ class TestPipeDocGenBlueprint:
with pytest.raises(ValidationError, match=expected_message):
_blueprint(**overrides)
+ @pytest.mark.parametrize("template_file", ["invoice.docx", "./invoice.docx", "templates/invoice.docx", "templates//invoice.docx"])
+ def test_a_path_inside_the_bundle_is_accepted_and_could_be_submitted(self, template_file: str) -> None:
+ """A step's template file obeys the rule the files submitted with a bundle are keyed by, so it can always be submitted."""
+ blueprint = _blueprint(format="docx", template_file=template_file)
+ assert blueprint.template_file == template_file
+ assert canonical_resource_path_or_none(path=template_file) is not None
+
+ @pytest.mark.parametrize("template_file", ["/srv/methods/invoice.docx", "C:/srv/methods/invoice.docx", "../srv/methods/invoice.docx"])
+ def test_the_refusal_of_a_template_file_outside_its_bundle_never_repeats_it(self, template_file: str) -> None:
+ """A host library's step would otherwise put the host's own path in front of a caller."""
+ with pytest.raises(ValidationError) as exc_info:
+ _blueprint(format="docx", template_file=template_file)
+ assert all("srv/methods" not in error["msg"] for error in exc_info.value.errors())
+
def test_a_name_every_branch_sets_is_the_template_s_own(self) -> None:
template = (
"{% if invoice.paid %}{% set status = 'Paid' %}{% else %}{% set status = 'Due' %}{% endif %}<h1>{{ invoice.number }}: {{ status }}</h1>"
@@ -53,3 +79,17 @@ class TestPipeDocGenBlueprint:
def test_a_sigil_in_the_template_reads_its_input(self) -> None:
blueprint = _blueprint(template="<h1>$invoice.number</h1>")
assert blueprint.template == "<h1>$invoice.number</h1>"
+
+ @pytest.mark.parametrize(
+ ("overrides", "expected_reads"),
+ [
+ ({"template_file": "templates/invoice.html"}, {"templates/invoice.html": BundleResourceReadStage.LOAD}),
+ ({"format": "docx", "template_file": "invoice.docx"}, {"invoice.docx": BundleResourceReadStage.RUN}),
+ ({"format": "xlsx", "template_file": "./invoice.xlsx"}, {"./invoice.xlsx": BundleResourceReadStage.RUN}),
+ ({"template": "<p>{{ invoice.number }}</p>"}, {}),
+ ({}, {}),
+ ],
+ )
+ def test_the_file_it_reads_from_its_bundle_and_when(self, overrides: dict[str, Any], expected_reads: dict[str, BundleResourceReadStage]) -> None:
+ """An HTML template file is read while the method loads, to check it against the inputs; an office one only when the step prints."""
+ assert _blueprint(**overrides).bundle_resource_reads == expected_reads
diff --git a/tests/unit/pipelex/pipeline/test_direct_bundle_validator.py b/tests/unit/pipelex/pipeline/test_direct_bundle_validator.py
index 15d657973..9d4904a01 100644
--- a/tests/unit/pipelex/pipeline/test_direct_bundle_validator.py
+++ b/tests/unit/pipelex/pipeline/test_direct_bundle_validator.py
@@ -16,6 +16,7 @@ from typing import TYPE_CHECKING
import pytest
from pipelex.base_exceptions import ErrorReport
+from pipelex.core.bundle_resources.bundle_resource import make_bundle_resources
from pipelex.pipeline.direct_bundle_validator import DirectBundleValidator
from pipelex.pipeline.exceptions import ValidateBundleError
from pipelex.system.caller_identity import CallerIdentity
@@ -29,6 +30,7 @@ class TestDirectBundleValidator:
async def test_valid_sweep_is_returned_as_the_valid_arm(self, mocker: MockerFixture) -> None:
"""A successful in-process sweep's report is returned verbatim, with arguments forwarded."""
report = mocker.MagicMock(name="validation_report")
+ bundle_resources = make_bundle_resources(files={"invoice.docx": b"docx bytes"})
in_process_mock = mocker.patch(
"pipelex.pipeline.direct_bundle_validator.validate_bundles_in_process",
new=mocker.AsyncMock(return_value=report),
@@ -41,6 +43,7 @@ class TestDirectBundleValidator:
library_dirs=[Path("lib_dir")],
caller_identity=CallerIdentity(user_id="user-42", extras={"organization": "org_acme"}),
graph_pipe_code="extract_document_markdown",
+ bundle_resources=bundle_resources,
)
assert verdict is report
@@ -52,6 +55,7 @@ class TestDirectBundleValidator:
graph_pipe_code="extract_document_markdown",
log_context="API validate",
caller_identity=CallerIdentity(user_id="user-42", extras={"organization": "org_acme"}),
+ bundle_resources=bundle_resources,
)
async def test_validate_bundle_error_becomes_the_invalid_arm(self, mocker: MockerFixture) -> None:
@@ -69,6 +73,7 @@ class TestDirectBundleValidator:
library_dirs=None,
caller_identity=None,
graph_pipe_code=None,
+ bundle_resources=None,
)
assert isinstance(verdict, ErrorReport)
@@ -90,4 +95,5 @@ class TestDirectBundleValidator:
library_dirs=None,
caller_identity=None,
graph_pipe_code=None,
+ bundle_resources=None,
)
diff --git a/tests/unit/pipelex/pipeline/test_runner_validate_plumbing.py b/tests/unit/pipelex/pipeline/test_runner_validate_plumbing.py
index a98ee7477..40584e735 100644
--- a/tests/unit/pipelex/pipeline/test_runner_validate_plumbing.py
+++ b/tests/unit/pipelex/pipeline/test_runner_validate_plumbing.py
@@ -24,6 +24,7 @@ from typing import TYPE_CHECKING, Any, NamedTuple
import pytest
+from pipelex.core.bundle_resources.bundle_resource import make_bundle_resources
from pipelex.core.pipes.pipe_io_artifacts import PipeIOArtifacts
from pipelex.pipeline.runner import PipelexMTHDSProtocol
from pipelex.system.caller_identity import CallerIdentity, get_current_caller_identity
@@ -79,12 +80,13 @@ class TestRunnerValidatePlumbing:
)
async def test_library_dirs_converted_to_paths_and_allow_signatures_passthrough(self, mocker: MockerFixture) -> None:
- """Constructor library_dirs strings reach validate_bundle as Path objects; allow_signatures and their ownership forwarded verbatim.
+ """Constructor library_dirs strings reach validate_bundle as Paths; allow_signatures, their ownership and the files forwarded verbatim.
The protocol validation returns a verdict, so it asks the sweep to collect the dry runs' warnings for it.
"""
env = self._patch_env(mocker, library_ids=[None, "val-lib"])
- runner = PipelexMTHDSProtocol(library_dirs=["dir_alpha", "nested/dir_beta"], library_dirs_are_callers=True)
+ bundle_resources = make_bundle_resources(files={"templates/invoice.docx": b"template"})
+ runner = PipelexMTHDSProtocol(library_dirs=["dir_alpha", "nested/dir_beta"], library_dirs_are_callers=True, bundle_resources=bundle_resources)
await runner.validate(mthds_contents=["bundle-content"], allow_signatures=True)
@@ -95,6 +97,7 @@ class TestRunnerValidatePlumbing:
allow_signatures=True,
library_dirs_are_callers=True,
collect_dry_run_warnings=True,
+ bundle_resources=bundle_resources,
)
async def test_no_library_dirs_passes_none_and_strict_default(self, mocker: MockerFixture) -> None:
@@ -111,6 +114,7 @@ class TestRunnerValidatePlumbing:
allow_signatures=False,
library_dirs_are_callers=False,
collect_dry_run_warnings=True,
+ bundle_resources=None,
)
async def test_validation_runs_with_the_protocols_caller_in_scope(self, mocker: MockerFixture) -> None:
diff --git a/tests/unit/pipelex/pipeline/test_validate_bundle_error_withholding.py b/tests/unit/pipelex/pipeline/test_validate_bundle_error_withholding.py
index 3c77986f1..9dbfff282 100644
--- a/tests/unit/pipelex/pipeline/test_validate_bundle_error_withholding.py
+++ b/tests/unit/pipelex/pipeline/test_validate_bundle_error_withholding.py
@@ -2,6 +2,7 @@ import inspect
from pathlib import Path
import pytest
+from pytest_mock import MockerFixture
from pipelex.core.exceptions import (
DryRunFailureErrorData,
@@ -9,6 +10,7 @@ from pipelex.core.exceptions import (
PipelexBundleBlueprintValidationErrorData,
PipesAndConceptValidationErrorData,
)
+from pipelex.pipeline import validate_bundle_translation
from pipelex.pipeline.exceptions import ValidateBundleError
from pipelex.pipeline.validate_bundle_translation import HOST_LIBRARY_FILE_PLACEHOLDER, withholding_host_library_files
from pipelex.validation_error_types import PipeFactoryErrorType
@@ -106,3 +108,19 @@ class TestValidateBundleErrorWithholding:
(item,) = raised.value.to_error_report().validation_errors or []
assert item.source == callers_source
+
+ def test_the_host_s_files_and_the_template_files_its_steps_name_are_found_in_one_walk(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """The host's directories are listed once, and each bundle file read once, for both its name and its steps' template files."""
+ host_library_file = tmp_path / "harbour_board.mthds"
+ host_library_file.write_text(
+ 'domain = "harbour_board"\n\n[pipe.print_notice]\ntype = "PipeDocGen"\nformat = "docx"\ntemplate_file = "notice-template.docx"\n',
+ encoding="utf-8",
+ )
+ message = f"Pipe 'print_notice' in '{host_library_file}' could not read 'notice-template.docx'"
+ walk_spy = mocker.spy(validate_bundle_translation, "get_pipelex_mthds_files_from_dirs")
+
+ with pytest.raises(ValidateBundleError) as raised, withholding_host_library_files(library_dirs=[tmp_path]):
+ raise ValidateBundleError(message=message)
+
+ assert raised.value.message == f"Pipe 'print_notice' in '{HOST_LIBRARY_FILE_PLACEHOLDER}' could not read '{HOST_LIBRARY_FILE_PLACEHOLDER}'"
+ assert walk_spy.call_count == 1
diff --git a/tests/unit/pipelex/plugins/test_bundle_validator_registry.py b/tests/unit/pipelex/plugins/test_bundle_validator_registry.py
index 7a5da4b8d..e0132bf69 100644
--- a/tests/unit/pipelex/plugins/test_bundle_validator_registry.py
+++ b/tests/unit/pipelex/plugins/test_bundle_validator_registry.py
@@ -23,6 +23,7 @@ if TYPE_CHECKING:
from pathlib import Path
from pipelex.base_exceptions import ErrorReport
+ from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.pipeline.validation_report import PipelexValidationReport
from pipelex.system.caller_identity import CallerIdentity
from pipelex.system.configuration.configs import PipelexConfig
@@ -40,6 +41,7 @@ class _FakeBundleValidator:
library_dirs: Sequence[Path] | None,
caller_identity: CallerIdentity | None,
graph_pipe_code: str | None,
+ bundle_resources: BundleResources | None,
) -> PipelexValidationReport | ErrorReport:
raise NotImplementedError