Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix/Escaping-template-file-refusal · L-261016-7e017e (#1435)

Louis Choquel7e57346e9e53
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a98477543..8c9e27c7f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,6 +18,10 @@
- **`LibraryCrate.pipe_func_files` replaces `python_sources` (Breaking)**: a sandbox-hosted crate carries the files a `PipeFunc` call receives, keyed by Python root, the key of a method tree holding a `PipeFunc` step or `submitted` for the bundles passed as content, then by tree path: every file of the tree except its `.mthds` files and a generated `structures` module, a file that is not UTF-8 text in Base64. They are read as the library loads, so a crate built later reads no file. The `submitted` root holds the files of the library directories whose own tree holds no bundle, such as the directory a server writes a request's Python into; a directory holding a bundle submits nothing, so two methods' `README.md` or `requirements.txt` never clash, and two directories submitting different files at one path refuse the load with `LibraryError`. A structure class is refused in every tree the load lists, a package or repository nested in a library directory included.
- **A `PipeFunc` call carries its tree's files (Breaking)**: `PipeFuncExecutionRequest.files` holds the files of the called step's Python root at their tree paths, and `build_pipe_func_execution_request` moves them there from the crate, which no longer carries `pipe_func_files`. The direct executor's transported run writes them into a temporary directory, generates `structures` unless the method shipped one, imports the function by its import path under the process's hold on that directory and unloads its modules before removing it; `DirectPipeFuncExecutor.run_pipe_func_transported_at` and an optional third argument of the transported entrypoint take a directory a sandbox client filled itself.
+### Fixed
+
+- **A `template_file` no sent file could be keyed by is a blueprint item at its step**: a bundle loaded from content strings, as the API's routes and `validate_bundle` load one, whose `PipeDocGen` step's `template_file` holds a `..` segment or a NUL character, or names no file, is refused when it parses, whatever files the request sends, as one `blueprint_validation` item with `field_path` `pipe.<code>` whose message names the field and not the path, instead of a load refusal saying the file was not supplied. A bundle read from disk still resolves a `..` in its method tree.
+
## [v0.86.1] - 2026-10-16
### Added
diff --git a/docs/api-server/pipe-validate.md b/docs/api-server/pipe-validate.md
index d83930f84..62786775f 100644
--- a/docs/api-server/pipe-validate.md
+++ b/docs/api-server/pipe-validate.md
@@ -173,7 +173,7 @@ A bundle sent as content has no directory, so a file one of its steps reads from
- **The sizes are bounded**: at most `MAX_RESOURCES_PER_REQUEST` files (32 by default), each at most `MAX_RESOURCE_FILE_KIB` once decoded (5 MiB by default), and at most `MAX_RESOURCES_TOTAL_KIB` across the request (10 MiB by default), counting the files a method bundle ships for its steps. A request over any of them is a **413** `PayloadTooLarge`, an oversized file refused on its encoded length before it is decoded. The whole body stays under `MAX_REQUEST_BODY_MIB` as well.
- **The files read when the method loads are bounded too**: the `.html` template files, which travel inline with the method wherever it runs, a remote orchestrator's payloads included, may total 512 KiB at most once decoded. That bound is fixed, sized well under Temporal's 2 MB payload limit, and checked once the bundle is parsed, since only its steps say which files those are: a request over it is a **413** `BundleResourcesTooLargeError` naming the files and the limit. An office template file is not counted, since a run handed to a remote orchestrator carries it by reference to the run's storage.
-The files are checked as files on disk are. An `.html` template file is checked against the step's inputs when the bundle loads, and the document engine's template check runs on an office template file in the validation's dry run, where its errors are validation errors on the step. A step whose `template_file` was not sent is refused at load, and the verdict's `validation_errors[]` says the file was not supplied with its bundle and names the key it is expected under.
+The files are checked as files on disk are. An `.html` template file is checked against the step's inputs when the bundle loads, and the document engine's template check runs on an office template file in the validation's dry run, where its errors are validation errors on the step. A step whose `template_file` could not be a key, being empty or absolute, or holding a `..` segment, a backslash, a colon or a NUL character, is refused when its bundle is parsed, whatever the request sends: the verdict's one item is a `blueprint_validation` item located at the step by its `pipe_code` and by `field_path` `pipe.<code>`, whose message names the field without repeating the path. A step whose `template_file` could be a key but was not sent is refused at load, and the verdict's `validation_errors[]` says the file was not supplied with its bundle and names the key it is expected under.
## Where validation runs
diff --git a/docs/building-methods/pipes/executing-pipelines.md b/docs/building-methods/pipes/executing-pipelines.md
index 13c480b55..f4f3f2328 100644
--- a/docs/building-methods/pipes/executing-pipelines.md
+++ b/docs/building-methods/pipes/executing-pipelines.md
@@ -92,7 +92,7 @@ When using `PipelexMTHDSProtocol`, you can control library behavior with these p
- **`mthds_contents`**: When provided to `PipelexMTHDSProtocol.execute()`, Pipelex parses these strings and loads them into the library in addition to any resolved library directories. To run with only this content, disable directory loading explicitly (e.g. `library_dirs=[]`). This is useful for dynamic pipeline execution without file-based definitions.
-- **`bundle_uris`**: Optional file paths identifying the bundles passed as `mthds_contents` (matched by position). When a URI matches a bundle already loaded from the library directories, that content is skipped instead of being loaded twice. Without `bundle_uris`, every `mthds_contents` entry is loaded unconditionally — so passing content that duplicates a directory-loaded bundle raises a duplicate-declaration error. A content whose URI names a file on this machine is loaded as read from that file, so its steps read the files beside it, such as a `PipeDocGen` step's `template_file`, and an address-based reference is looked for in a `.mthds/methods/` directory above it, exactly as when its directory loads; without `bundle_uris`, a content string has no directory, and its template files are passed as `bundle_resources`.
+- **`bundle_uris`**: Optional file paths identifying the bundles passed as `mthds_contents` (matched by position). When a URI matches a bundle already loaded from the library directories, that content is skipped instead of being loaded twice. Without `bundle_uris`, every `mthds_contents` entry is loaded unconditionally — so passing content that duplicates a directory-loaded bundle raises a duplicate-declaration error. A content whose URI names a file on this machine is loaded as read from that file, so its steps read the files beside it, such as a `PipeDocGen` step's `template_file`, and an address-based reference is looked for in a `.mthds/methods/` directory above it, exactly as when its directory loads; without `bundle_uris`, a content string has no directory, and its template files are passed as `bundle_resources`, so a step naming one by a path no such file could be keyed by, such as one through a `..` segment, is refused when the content is parsed.
!!! info "Python Structure Classes"
If your concepts use Python `StructuredContent` classes instead of inline structures, those Python files must be in the directories specified by `library_dirs`. Pipelex auto-discovers and registers these classes during library loading. Learn more about [Python StructuredContent Classes](../concepts/python-classes.md).
diff --git a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
index cc36e3bab..f2e973a0b 100644
--- a/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
+++ b/docs/building-methods/pipes/pipe-operators/PipeDocGen.md
@@ -81,7 +81,7 @@ To print the same invoice on the plugin's `pipelex-pdf`, the step names it: `mod
| `format` | string | `pdf`, `xlsx`, `docx` or `pptx`. It sets the file's type and suffix. | Yes |
| `model` | string | The document engine that prints the file, a `doc_gen` model, alias or preset, such as `reportlab-pdf` or, with the plugin, `pipelex-pdf`. It defaults to the model deck's engine for the format and source. | No |
| `template` | string | An inline HTML and Jinja2 template, for `pdf` only. A `pdf` with a template is printed by the plugin. | No |
-| `template_file` | string | A template file relative to the bundle's file, written with forward slashes: `.html` for `pdf`, `.xlsx`, `.docx` or `.pptx`. An empty or absolute path, a URL, a backslash and a colon are refused. When the bundle is read from a folder, the file is the one its [method tree](../../method-trees.md#file-references) holds at that path, a `..` climbing to another folder of the tree included, and a file the tree keeps out, such as a gitignored one, is refused; a bundle file run alone with no `METHODS.toml` at or above it reads no other file. When the bundle is loaded from a string, the file is one of the files sent with it, where a `..` segment names none. It excludes `template`. `pptx` requires one. | No |
+| `template_file` | string | A template file relative to the bundle's file, written with forward slashes: `.html` for `pdf`, `.xlsx`, `.docx` or `.pptx`. An empty or absolute path, a URL, a backslash and a colon are refused. When the bundle is read from a folder, the file is the one its [method tree](../../method-trees.md#file-references) holds at that path, a `..` climbing to another folder of the tree included, and a file the tree keeps out, such as a gitignored one, is refused; a bundle file run alone with no `METHODS.toml` at or above it reads no other file. When the bundle is loaded from a string, the file is one of the files sent with it, each keyed by a path inside the bundle, so a path no such file could be keyed by, one holding a `..` segment or a NUL character, or naming no file, is refused when the bundle is parsed, as a validation error at the step that does not repeat the path. It excludes `template`. `pptx` requires one. | No |
| `filename` | string | A Jinja expression over the inputs for the file's name, without its suffix, such as `invoice-{{ invoice.number }}`. It defaults to the pipe's code. The rendered name is made into one every storage reader serves, the hosted API's included, spelled in ASCII with hyphens for the rest: see [The file's name](#the-files-name). | No |
### The file's name
@@ -145,7 +145,7 @@ When a bundle is sent as content rather than read from a directory, as it is thr
}
```
-- **The key is the path the step names**, relative to the bundle and written with forward slashes: a step with `template_file = "templates/invoice.docx"` reads the file sent as `templates/invoice.docx`, and `./templates/invoice.docx` names the same file. Every bundle of the request is read as sitting in one directory. An absolute path, a `..` segment, a backslash and a colon are refused.
+- **The key is the path the step names**, relative to the bundle and written with forward slashes: a step with `template_file = "templates/invoice.docx"` reads the file sent as `templates/invoice.docx`, and `./templates/invoice.docx` names the same file. Every bundle of the request is read as sitting in one directory. An absolute path, a `..` segment, a backslash and a colon are refused, as a key and as the path a step names: a step whose `template_file` could not be a key is refused when its bundle is parsed, whatever the request sends, so every file a string-loaded step can name can be sent with it.
- **The value is the file's bytes** in standard base64.
- **Every file sent must be one a step names.** A file no step reads is refused, so a misspelled key fails loudly rather than leaving its step without a template.
- **The files are checked like files on disk.** An `.html` template file is checked against the inputs when the method loads, and the engine's template check runs on an office template file at validation and in the dry run.
diff --git a/pipelex/core/method_trees/file_references.py b/pipelex/core/method_trees/file_references.py
index 50bb3abb3..31c354849 100644
--- a/pipelex/core/method_trees/file_references.py
+++ b/pipelex/core/method_trees/file_references.py
@@ -5,18 +5,82 @@ joined to the bundle's own tree path, a `..` may climb as long as it stays insid
file of the tree, which is a lookup in the set of tree paths the listing produced. So a file the membership rule keeps
out of the tree, a dotfile, a `.gitignore`d file or a file of a nested package, is invisible to the step, and nothing
outside the tree is ever opened.
+
+A bundle loaded from a content string belongs to no tree and reads only the files submitted with it, so the parser,
+told so by its caller, validates its blueprint under that origin (`BundleValidationContext`), and a reference none of
+those files could be keyed by is refused there, as a blueprint item at the step that names it.
"""
+from enum import StrEnum
+
from mthds.method_tree.exceptions import MethodTreeRefusedError
from mthds.method_tree.references import resolve_reference
from mthds.method_tree.vocabulary import ReferenceContext, TreeRefusalCode
+from pydantic import BaseModel, ConfigDict, Field
from pipelex.base_exceptions import PipelexUnexpectedError
+from pipelex.core.bundle_resources.bundle_resource import resource_path_refusal
from pipelex.core.method_trees.tree_bundle import TreeBundle
ONE_FILE_TREE_ADVICE = "Run the method's folder instead, or add a METHODS.toml to it."
+class BundleFileOrigin(StrEnum):
+ """Where a bundle's steps find the files they name, which decides the rule a file reference is held to.
+
+ A bundle read from this host's disk finds them in its method tree, where a `..` may climb to another folder of the
+ tree. A bundle loaded from a content string belongs to no tree: it reads only the files submitted with it, each
+ keyed by a path inside the bundle (`resource_path_refusal`), so a reference no such key could be, one holding a
+ `..` segment included, can never be supplied.
+ """
+
+ METHOD_TREE = "method_tree"
+ SUBMITTED_FILES = "submitted_files"
+
+
+class BundleValidationContext(BaseModel):
+ """The pydantic validation context the parser validates a bundle's blueprint with.
+
+ The validators that check a file reference, such as a `PipeDocGen` step's `template_file`, read it to know which
+ rule the reference is held to.
+ """
+
+ model_config = ConfigDict(frozen=True, extra="forbid")
+
+ file_origin: BundleFileOrigin = Field(strict=False)
+
+
+def bundle_file_origin(*, validation_context: object) -> BundleFileOrigin:
+ """Where the bundle under validation finds the files its steps name, as its validation context says.
+
+ A blueprint validated with no such context, such as one a loader parsed from its path, a crate's blueprint loaded
+ again or one built in code, is held to the method tree's rule, which checks a reference's form alone: its load then
+ resolves the reference in the bundle's tree, or, for a bundle in none, refuses a file not submitted with it.
+ """
+ if isinstance(validation_context, BundleValidationContext):
+ return validation_context.file_origin
+ return BundleFileOrigin.METHOD_TREE
+
+
+def file_reference_refusal(*, reference: str, file_origin: BundleFileOrigin) -> str | None:
+ """Why a path written in a bundle field cannot name a file where the bundle finds its files, or None when it can.
+
+ The form check comes first, whatever the origin. A bundle reading its files from its method tree is held to it
+ alone, its load deciding in the tree whether the path names a file. A bundle reading only the files submitted with
+ it is also held to the rule those files are keyed by, so a step naming a file that could never be submitted, such
+ as one through a `..` segment, is refused when its blueprint validates, before any file is looked up. The reason
+ never repeats the path.
+ """
+ form_refusal = file_reference_form_refusal(reference=reference)
+ if form_refusal is not None:
+ return form_refusal
+ match file_origin:
+ case BundleFileOrigin.METHOD_TREE:
+ return None
+ case BundleFileOrigin.SUBMITTED_FILES:
+ return resource_path_refusal(path=reference)
+
+
def file_reference_form_refusal(*, reference: str) -> str | None:
"""Why a path written in a bundle field cannot name a file of any method tree, or None when its form can.
diff --git a/pipelex/mthds_parsing/parser.py b/pipelex/mthds_parsing/parser.py
index 4cf380426..e3b040ed6 100644
--- a/pipelex/mthds_parsing/parser.py
+++ b/pipelex/mthds_parsing/parser.py
@@ -4,6 +4,7 @@ from typing import Any
from pydantic import BaseModel, ValidationError
from pipelex.core.exceptions import PipelexBundleBlueprintValidationErrorData
+from pipelex.core.method_trees.file_references import BundleFileOrigin, BundleValidationContext
from pipelex.mthds_parsing.bundle_elaborator import BundleElaborator
from pipelex.mthds_parsing.exceptions import BundleElaboratorError, MthdsParserError
from pipelex.mthds_parsing.pipelex_bundle_blueprint import PipelexBundleBlueprint
@@ -23,6 +24,7 @@ class MthdsParser(BaseModel):
*,
mthds_content: str | None = None,
mthds_source: str | None = None,
+ file_origin: BundleFileOrigin = BundleFileOrigin.METHOD_TREE,
) -> PipelexBundleBlueprint:
"""Parse one MTHDS source into a ``PipelexBundleBlueprint``.
@@ -36,13 +38,24 @@ class MthdsParser(BaseModel):
categorizer reads off that dict — mirroring the real file path the disk
path already records, and overriding any ``source`` the bundle text
itself declared. Ignored when ``bundle_path`` is provided (the path wins).
+
+ ``file_origin`` says where the bundle's steps find the files they name, and so the rule a file
+ reference, such as a ``PipeDocGen`` step's ``template_file``, is held to when the blueprint validates.
+ A caller loading ``mthds_content`` that no loader will place in a method tree passes ``SUBMITTED_FILES``:
+ such a bundle reads only the files submitted with it, so a step naming a file none of them could be keyed
+ by, such as one through a ``..`` segment, is refused here, as a blueprint item at the step, whatever the
+ request sends. ``METHOD_TREE``, the default, checks a reference's form alone, for the load to resolve it
+ in the bundle's tree; it is the only origin of a bundle read from ``bundle_path``, so it is ignored there.
"""
blueprint_dict: dict[str, Any]
+ validation_context: BundleValidationContext
try:
if bundle_path is not None:
blueprint_dict = load_toml_from_path(path=str(bundle_path))
blueprint_dict[PIPELEX_BUNDLE_BLUEPRINT_SOURCE_FIELD] = str(bundle_path)
+ validation_context = BundleValidationContext(file_origin=BundleFileOrigin.METHOD_TREE)
elif mthds_content is not None:
+ validation_context = BundleValidationContext(file_origin=file_origin)
blueprint_dict = load_toml_from_content(content=mthds_content)
# Seed the logical source unconditionally (``None`` when unnamed): this
# overrides any ``source`` the bundle text declared and keeps the success
@@ -76,7 +89,7 @@ class MthdsParser(BaseModel):
# ``source`` is already populated from the seeded ``blueprint_dict`` above
# (disk path → real file path; in-memory path → ``mthds_source``), so no
# post-validate re-assignment is needed.
- pipelex_bundle_blueprint = PipelexBundleBlueprint.model_validate(blueprint_dict)
+ pipelex_bundle_blueprint = PipelexBundleBlueprint.model_validate(blueprint_dict, context=validation_context)
except ValidationError as exc:
# TODO: Move this to the validate_bundle function
# Every error becomes an item, categorized or not, so none hides another; only a concept's
diff --git a/pipelex/pipe_operators/doc_gen/pipe_doc_gen_blueprint.py b/pipelex/pipe_operators/doc_gen/pipe_doc_gen_blueprint.py
index de6e5d1ee..0951bb65b 100644
--- a/pipelex/pipe_operators/doc_gen/pipe_doc_gen_blueprint.py
+++ b/pipelex/pipe_operators/doc_gen/pipe_doc_gen_blueprint.py
@@ -1,6 +1,6 @@
from typing import Literal, Self
-from pydantic import Field, model_validator
+from pydantic import Field, ValidationInfo, model_validator
from typing_extensions import override
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat
@@ -8,7 +8,7 @@ from pipelex.cogt.doc_gen.doc_gen_setting import DocGenModelChoice
from pipelex.cogt.templating.exceptions import TemplateSigilSyntaxError
from pipelex.cogt.templating.template_preprocessor import preprocess_template
from pipelex.core.bundle_resources.bundle_resource import BundleResourceReadStage
-from pipelex.core.method_trees.file_references import file_reference_form_refusal
+from pipelex.core.method_trees.file_references import BundleFileOrigin, bundle_file_origin, file_reference_refusal
from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
from pipelex.pipe_machinery.pipe_blueprint import PipeBlueprint
from pipelex.tools.jinja2.exceptions import Jinja2TemplateSyntaxError
@@ -67,14 +67,26 @@ class PipeDocGenBlueprint(PipeBlueprint):
filename: str | None = None
@model_validator(mode="after")
- def validate_template_sources(self) -> Self:
- # The form of a file reference, the one check a blueprint can make before its method tree is at hand: whether the
- # path names a file, a `..` climbing elsewhere in the tree included, is decided when the bundle loads, in its tree.
+ def validate_template_sources(self, info: ValidationInfo) -> Self:
+ # A bundle read into a method tree is held to the form of a file reference, the one check a blueprint can make
+ # before its tree is at hand: whether the path names a file, a `..` climbing elsewhere in the tree included, is
+ # decided when the bundle loads, in its tree. A bundle loaded from a content string reads only the files
+ # submitted with it, so its step is also held to the rule they are keyed by, here, before any file is looked up.
# The message never repeats the path: a host library's step would put the host's own path in front of a caller.
if self.template_file is not None:
- refusal = file_reference_form_refusal(reference=self.template_file)
+ file_origin = bundle_file_origin(validation_context=info.context)
+ refusal = file_reference_refusal(reference=self.template_file, file_origin=file_origin)
if refusal is not None:
- msg = f"PipeDocGen's 'template_file' is a path relative to the bundle's file, in its method, and this one cannot be: {refusal}."
+ match file_origin:
+ case BundleFileOrigin.METHOD_TREE:
+ msg = (
+ f"PipeDocGen's 'template_file' is a path relative to the bundle's file, in its method, and this one cannot be: {refusal}."
+ )
+ case BundleFileOrigin.SUBMITTED_FILES:
+ msg = (
+ "PipeDocGen's 'template_file' names a file submitted with its bundle, by a path relative to the bundle "
+ f"and inside it, and this one cannot be: {refusal}."
+ )
raise ValueError(msg)
if self.template is not None and self.template_file is not None:
msg = "PipeDocGen cannot have both 'template' and 'template_file': use one or the other."
diff --git a/pipelex/pipeline/execution_seams.py b/pipelex/pipeline/execution_seams.py
index 6b70cebcd..ab5328cfe 100644
--- a/pipelex/pipeline/execution_seams.py
+++ b/pipelex/pipeline/execution_seams.py
@@ -32,6 +32,7 @@ from pipelex.core.bundle_resources.bundle_resource import BundleResources
from pipelex.core.memory.absence import AbsenceKind, AbsenceRecord
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.method_trees.file_references import BundleFileOrigin
from pipelex.interpreter_hub import (
clear_current_library,
get_concept_library,
@@ -65,6 +66,17 @@ if TYPE_CHECKING:
from pipelex.system.trace_context import TraceContext
+def _resolved_bundle_uris(*, bundle_uris: Sequence[str]) -> list[Path]:
+ """Each bundle URI as a path on this host: resolved, or as written when it cannot be resolved."""
+ resolved_uris: list[Path] = []
+ for uri in bundle_uris:
+ try:
+ resolved_uris.append(Path(uri).resolve())
+ except (OSError, RuntimeError):
+ resolved_uris.append(Path(uri))
+ return resolved_uris
+
+
def acquire_library(
library_id: str,
*,
@@ -112,6 +124,8 @@ def acquire_library(
whose file exists is loaded as read from it: its steps read beside it, such as a ``PipeDocGen`` step's
``template_file``, and an address-based reference is searched for in a ``.mthds/methods/`` directory above it, as
when its directory loads. They are the host's own paths, never a label a remote caller sends: a server passes none.
+ Every other content belongs to no method tree and reads only the files submitted with it, so a step naming a file
+ none of them could be keyed by, such as one through a ``..`` segment, is refused when its blueprint validates.
``bundle_resources`` are the files submitted with the ``mthds_contents`` that their steps read, such as a
``PipeDocGen`` step's ``template_file``, keyed by canonical bundle-relative path. The library holds them before the
@@ -158,7 +172,18 @@ def acquire_library(
else withholding_host_library_files(library_dirs=effective_dirs, caller_contents=mthds_contents)
)
with host_library_withholding, translate_to_validate_bundle_error():
- all_blueprints = [MthdsParser.make_pipelex_bundle_blueprint(mthds_content=content) for content in mthds_contents]
+ # The file on this host each content was read from, where the caller names one that exists: that bundle
+ # is the file's, whose steps find the files they name in its method tree. Every other content belongs to
+ # no tree, and its steps read only the files submitted with it, so its blueprint validates under that rule.
+ resolved_uris = _resolved_bundle_uris(bundle_uris=bundle_uris or [])
+ host_bundle_files: list[Path | None] = [uri if uri.is_file() else None for uri in resolved_uris] or [None] * len(mthds_contents)
+ all_blueprints = [
+ MthdsParser.make_pipelex_bundle_blueprint(
+ mthds_content=content,
+ file_origin=BundleFileOrigin.SUBMITTED_FILES if host_bundle_file is None else BundleFileOrigin.METHOD_TREE,
+ )
+ for content, host_bundle_file in zip(mthds_contents, host_bundle_files, strict=True)
+ ]
if bundle_resources:
check_bundle_resources(blueprints=all_blueprints, bundle_resources=bundle_resources)
library_manager.add_bundle_resources(library_id=library_id, bundle_resources=bundle_resources)
@@ -169,16 +194,14 @@ def acquire_library(
if bundle_uris:
current_library = library_manager.get_library(library_id=library_id)
blueprints_to_load = []
- for blueprint, uri in zip(all_blueprints, bundle_uris, strict=True):
- try:
- resolved_uri = Path(uri).resolve()
- except (OSError, RuntimeError):
- resolved_uri = Path(uri)
+ for blueprint, uri, resolved_uri, host_bundle_file in zip(
+ all_blueprints, bundle_uris, resolved_uris, host_bundle_files, strict=True
+ ):
if resolved_uri in current_library.loaded_mthds_paths:
log.verbose(f"Bundle '{uri}' was skipped, already loaded from library directories")
else:
- if resolved_uri.is_file():
- blueprint.mark_read_from_host_file(bundle_file=resolved_uri)
+ if host_bundle_file is not None:
+ blueprint.mark_read_from_host_file(bundle_file=host_bundle_file)
blueprints_to_load.append(blueprint)
if blueprints_to_load:
diff --git a/pipelex/pipeline/resolve_bundle.py b/pipelex/pipeline/resolve_bundle.py
index 2924d1865..1b5377393 100644
--- a/pipelex/pipeline/resolve_bundle.py
+++ b/pipelex/pipeline/resolve_bundle.py
@@ -18,6 +18,7 @@ from mthds.package.manifest.schema import MTHDS_STANDARD_VERSION
from pipelex.base_exceptions import PipelexUnexpectedError
from pipelex.core.bundle_resources.bundle_resource import BundleResources
+from pipelex.core.method_trees.file_references import BundleFileOrigin
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.interpreter_hub import clear_current_library, get_current_library_id_or_none, get_library_manager, set_current_library
from pipelex.libraries.bundle_resources import check_bundle_resources
@@ -78,8 +79,9 @@ def resolve_crate_from_contents(
set_current_library(library_id=library_id)
content_sources: list[str | None] = list(mthds_sources) if mthds_sources is not None else [None] * len(mthds_contents)
with translate_to_validate_bundle_error():
+ # Contents belong to no method tree: their steps read only the files submitted with them.
blueprints = [
- MthdsParser.make_pipelex_bundle_blueprint(mthds_content=content, mthds_source=source)
+ MthdsParser.make_pipelex_bundle_blueprint(mthds_content=content, mthds_source=source, file_origin=BundleFileOrigin.SUBMITTED_FILES)
for content, source in zip(mthds_contents, content_sources, strict=True)
]
_reject_address_based_dependencies(blueprints=blueprints)
diff --git a/pipelex/pipeline/validate_bundle.py b/pipelex/pipeline/validate_bundle.py
index 8c9e008e9..dcaefbfaf 100644
--- a/pipelex/pipeline/validate_bundle.py
+++ b/pipelex/pipeline/validate_bundle.py
@@ -12,6 +12,7 @@ from typing_extensions import TypedDict
from pipelex import log
from pipelex.base_exceptions import PipelexUnexpectedError
from pipelex.core.bundle_resources.bundle_resource import BundleResources
+from pipelex.core.method_trees.file_references import BundleFileOrigin
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.interpreter_hub import (
clear_current_library,
@@ -229,8 +230,11 @@ async def validate_bundle(
log.verbose(f"No library directories to load ({source_label})")
if mthds_contents is not None:
content_sources: list[str | None] = list(mthds_sources) if mthds_sources is not None else [None] * len(mthds_contents)
+ # Contents belong to no method tree: their steps read only the files submitted with them.
loaded_blueprints = [
- MthdsParser.make_pipelex_bundle_blueprint(mthds_content=content, mthds_source=source)
+ MthdsParser.make_pipelex_bundle_blueprint(
+ mthds_content=content, mthds_source=source, file_origin=BundleFileOrigin.SUBMITTED_FILES
+ )
for content, source in zip(mthds_contents, content_sources, strict=True)
]
if bundle_resources:
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_data.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_data.py
index 220786bb7..6a7827853 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_data.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_data.py
@@ -79,6 +79,13 @@ output = "doc_gen_tests.InvoicePdf"
steps = [{ pipe = "doc_gen_tests.print_invoice", result = "invoice_pdf" }]
"""
+ # The domain of the same bundle loaded beside the test bundle, so the two declare no pipe or concept twice.
+ OTHER_DOMAIN: ClassVar[str] = "doc_gen_other"
+
@classmethod
def bundle(cls, *, step_fields: str) -> str:
return cls.BUNDLE_MTHDS.replace(cls.STEP_SLOT, step_fields)
+
+ @classmethod
+ def bundle_in_the_other_domain(cls, *, step_fields: str) -> str:
+ return cls.bundle(step_fields=step_fields).replace("doc_gen_tests", cls.OTHER_DOMAIN)
diff --git a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_bundle_resources.py b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_bundle_resources.py
index 636e02d0f..4bd19c663 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_bundle_resources.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_doc_gen/test_pipe_doc_gen_bundle_resources.py
@@ -5,28 +5,52 @@ from typing import Any
import pytest
-from pipelex.base_exceptions import PipelexError
+from pipelex.base_exceptions import PipelexError, ValidationErrorCategory
from pipelex.cogt.doc_gen.doc_gen_format import DocGenSource
from pipelex.cogt.doc_gen.template_check import TemplateFinding, TemplateFindingSeverity
from pipelex.core.bundle_resources.bundle_resource import BundleResources, make_bundle_resources
from pipelex.core.bundle_resources.exceptions import BundleResourceNotNamedError
from pipelex.core.stuffs.document_content import DocumentContent
from pipelex.pipeline.exceptions import ValidateBundleError
+from pipelex.pipeline.resolve_bundle import resolve_crate_from_contents
from pipelex.pipeline.runner import PipelexMTHDSProtocol
from pipelex.pipeline.validate_bundle import validate_bundle
from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.system.storage_scope import DRY_RUN_STORAGE_SCOPE, LOCAL_STORAGE_SCOPE
-from tests.integration.pipelex.pipes.operator.pipe_doc_gen.doc_gen_helpers import StubEngines, WorkerLikePipeRun, write_bundle
+from tests.integration.pipelex.pipes.operator.pipe_doc_gen.doc_gen_helpers import METHOD_MANIFEST, StubEngines, WorkerLikePipeRun, write_bundle
from tests.integration.pipelex.pipes.operator.pipe_doc_gen.test_data import PipeDocGenTestData
_DOCX_TEMPLATE_STEP = 'format = "docx"\ntemplate_file = "templates/invoice.docx"'
_DOCX_BYTES = b"PK docx template bytes"
+_ESCAPING_DOCX_STEP = 'format = "docx"\ntemplate_file = "../invoice.docx"'
+
+
def _docx_resources() -> BundleResources:
return make_bundle_resources(files={"./templates/invoice.docx": _DOCX_BYTES})
+def _bundle_file_climbing_in_its_method_tree(*, method_dir: Path) -> Path:
+ """Write a method whose bundle file, in `bundles/`, names its template through `..` in `templates/`, and return that file."""
+ (method_dir / "METHODS.toml").write_text(METHOD_MANIFEST, encoding="utf-8")
+ (method_dir / "templates").mkdir()
+ (method_dir / "templates" / "invoice.docx").write_bytes(_DOCX_BYTES)
+ (method_dir / "bundles").mkdir()
+ return write_bundle(directory=method_dir / "bundles", step_fields='format = "docx"\ntemplate_file = "../templates/invoice.docx"', as_method=False)
+
+
+async def _load_from_content(*, load: str, content: str, resources: BundleResources | None) -> None:
+ """Validate the content, resolve its crate or run it, as the runner's routes do with a bundle sent as a string."""
+ match load:
+ case "validate":
+ await validate_bundle(mthds_contents=[content], bundle_resources=resources)
+ case "resolve":
+ resolve_crate_from_contents(mthds_contents=[content], bundle_resources=resources)
+ case _:
+ await PipelexMTHDSProtocol(bundle_resources=resources).execute(mthds_contents=[content], inputs=PipeDocGenTestData.INVOICE_INPUTS)
+
+
@pytest.mark.asyncio(loop_scope="class")
class TestPipeDocGenBundleResources:
async def test_a_bundle_validates_with_its_template_file_and_the_check_runs_on_its_bytes(self, stub_engines: StubEngines) -> None:
@@ -60,6 +84,105 @@ class TestPipeDocGenBundleResources:
report = str(exc_info.value.to_error_report().model_dump())
assert "names the template file 'templates/invoice.docx', which was not supplied with its bundle" in report
+ @pytest.mark.usefixtures("stub_engines")
+ @pytest.mark.parametrize("template_file", ["../invoice.docx", "templates/../invoice.docx"], ids=["leaves_the_bundle", "dot_dot_inside"])
+ @pytest.mark.parametrize("load", ["validate", "resolve", "run"])
+ @pytest.mark.parametrize("sends_a_file", [False, True], ids=["sends_nothing", "sends_a_file"])
+ async def test_a_template_file_no_submitted_file_could_be_keyed_by_is_a_blueprint_item_at_the_step(
+ self, template_file: str, load: str, sends_a_file: bool
+ ) -> None:
+ """The step's file is held to the rule the files sent with its bundle are keyed by, when its blueprint validates.
+
+ Whatever the request sends, the step is the verdict's one item, a blueprint one located at the step, which
+ names the field and never repeats the path, rather than a load refusal saying the file was not supplied.
+ """
+ content = PipeDocGenTestData.bundle(step_fields=f'format = "docx"\ntemplate_file = "{template_file}"')
+ resources = make_bundle_resources(files={"invoice.docx": _DOCX_BYTES}) if sends_a_file else None
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await _load_from_content(load=load, content=content, resources=resources)
+
+ items = exc_info.value.to_error_report().validation_errors
+ assert items is not None
+ (item,) = items
+ assert item.category == ValidationErrorCategory.BLUEPRINT_VALIDATION
+ assert item.pipe_code == "print_invoice"
+ assert item.domain_code == "doc_gen_tests"
+ assert item.field_path == "pipe.print_invoice"
+ assert "PipeDocGen's 'template_file'" in item.message
+ assert "it leaves the bundle through a '..' segment" in item.message
+ assert template_file not in item.message
+
+ async def test_a_bundle_its_caller_read_from_a_file_here_climbs_through_dot_dot_in_its_method_tree(
+ self, stub_engines: StubEngines, tmp_path: Path
+ ) -> None:
+ """Contents read from a file named in `bundle_uris` are that file's bundle, whose step may climb through `..` in its tree."""
+ bundle_path = _bundle_file_climbing_in_its_method_tree(method_dir=tmp_path)
+
+ await PipelexMTHDSProtocol(bundle_uris=[str(bundle_path)], pipe_run_mode=PipeRunMode.DRY).execute(
+ mthds_contents=[bundle_path.read_text(encoding="utf-8")],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+
+ assert [check_request.template for check_request in stub_engines.check_requests] == [_DOCX_BYTES]
+
+ @pytest.mark.usefixtures("stub_engines")
+ @pytest.mark.parametrize("uri_kind", ["label", "stale_path", "directory"])
+ async def test_a_content_whose_uri_names_no_file_here_reads_only_the_files_submitted_with_it(self, tmp_path: Path, uri_kind: str) -> None:
+ """A URI naming no file on this host, a label, a path to no file or a directory, gives its content no method tree to climb in."""
+ bundle_uri = {"label": "invoice-printer", "stale_path": str(tmp_path / "gone.mthds"), "directory": str(tmp_path)}[uri_kind]
+
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await PipelexMTHDSProtocol(bundle_uris=[bundle_uri], pipe_run_mode=PipeRunMode.DRY).execute(
+ mthds_contents=[PipeDocGenTestData.bundle(step_fields=_ESCAPING_DOCX_STEP)],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+
+ (item,) = exc_info.value.to_error_report().validation_errors or []
+ assert item.category == ValidationErrorCategory.BLUEPRINT_VALIDATION
+ assert item.field_path == "pipe.print_invoice"
+ assert "it leaves the bundle through a '..' segment" in item.message
+
+ @pytest.mark.parametrize("file_first", [True, False], ids=["file_first", "label_first"])
+ async def test_in_a_mixed_list_the_file_read_here_climbs_in_its_tree_and_the_other_content_does_not(
+ self, stub_engines: StubEngines, tmp_path: Path, file_first: bool
+ ) -> None:
+ """Each content takes the origin its own URI gives it, wherever it sits in the list.
+
+ Beside a content under a label whose step names no file, the file's step climbs through `..` in its tree. Beside one
+ whose step climbs, that step is the verdict's one item, and the file's own `..` is not refused.
+ """
+ bundle_path = _bundle_file_climbing_in_its_method_tree(method_dir=tmp_path)
+ read_here = (bundle_path.read_text(encoding="utf-8"), str(bundle_path))
+ under_a_label = (PipeDocGenTestData.bundle_in_the_other_domain(step_fields=PipeDocGenTestData.PDF_LAYOUT_STEP), "invoice-printer")
+ escaping_under_a_label = (PipeDocGenTestData.bundle_in_the_other_domain(step_fields=_ESCAPING_DOCX_STEP), "invoice-printer")
+
+ def _ordered(other: tuple[str, str]) -> list[tuple[str, str]]:
+ return [read_here, other] if file_first else [other, read_here]
+
+ entries = _ordered(under_a_label)
+ await PipelexMTHDSProtocol(bundle_uris=[uri for _, uri in entries], pipe_run_mode=PipeRunMode.DRY).execute(
+ pipe_code="doc_gen_tests.print_invoice",
+ mthds_contents=[content for content, _ in entries],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+ assert [check_request.template for check_request in stub_engines.check_requests] == [_DOCX_BYTES]
+
+ entries = _ordered(escaping_under_a_label)
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await PipelexMTHDSProtocol(bundle_uris=[uri for _, uri in entries], pipe_run_mode=PipeRunMode.DRY).execute(
+ pipe_code="doc_gen_tests.print_invoice",
+ mthds_contents=[content for content, _ in entries],
+ inputs=PipeDocGenTestData.INVOICE_INPUTS,
+ )
+ (item,) = exc_info.value.to_error_report().validation_errors or []
+ assert (item.category, item.domain_code, item.pipe_code) == (
+ ValidationErrorCategory.BLUEPRINT_VALIDATION,
+ PipeDocGenTestData.OTHER_DOMAIN,
+ "print_invoice",
+ )
+ assert "it leaves the bundle through a '..' segment" in item.message
+
@pytest.mark.usefixtures("stub_engines")
async def test_a_file_no_step_names_is_refused_as_the_request_s_fault_not_as_a_verdict(self) -> None:
resources = make_bundle_resources(files={"templates/invoice.docx": _DOCX_BYTES, "templates/logo.png": b"png"})
diff --git a/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py
index c6d808aa7..ce85583e2 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_doc_gen/test_pipe_doc_gen_blueprint.py
@@ -5,13 +5,15 @@ from pydantic import ValidationError
from pipelex.cogt.doc_gen.doc_gen_format import DocGenFormat
from pipelex.core.bundle_resources.bundle_resource import BundleResourceReadStage, canonical_resource_path_or_none
+from pipelex.core.method_trees.file_references import BundleFileOrigin, BundleValidationContext
from pipelex.pipe_operators.doc_gen.pipe_doc_gen_blueprint import PipeDocGenBlueprint
-def _blueprint(**overrides: Any) -> PipeDocGenBlueprint:
+def _blueprint(*, file_origin: BundleFileOrigin | None = None, **overrides: Any) -> PipeDocGenBlueprint:
kwargs: dict[str, Any] = {"description": "Print the invoice", "inputs": {"invoice": "Invoice"}, "output": "Document", "format": "pdf"}
kwargs.update(overrides)
- return PipeDocGenBlueprint.model_validate(kwargs)
+ context = BundleValidationContext(file_origin=file_origin) if file_origin is not None else None
+ return PipeDocGenBlueprint.model_validate(kwargs, context=context)
class TestPipeDocGenBlueprint:
@@ -61,10 +63,37 @@ class TestPipeDocGenBlueprint:
assert blueprint.template_file == template_file
assert canonical_resource_path_or_none(path=template_file) is not None
+ @pytest.mark.parametrize("file_origin", [None, BundleFileOrigin.METHOD_TREE], ids=["no_context", "method_tree"])
@pytest.mark.parametrize("template_file", ["../invoice.docx", "templates/../invoice.docx", "../shared/templates/invoice.docx"])
- def test_a_path_climbing_through_dot_dot_is_left_to_the_method_tree(self, template_file: str) -> None:
+ def test_a_path_climbing_through_dot_dot_is_left_to_the_method_tree(self, template_file: str, file_origin: BundleFileOrigin | None) -> None:
"""A `..` may climb to a file elsewhere in the method's tree, which only the load, holding the tree, can tell."""
- blueprint = _blueprint(format="docx", template_file=template_file)
+ blueprint = _blueprint(file_origin=file_origin, format="docx", template_file=template_file)
+ assert blueprint.template_file == template_file
+
+ @pytest.mark.parametrize(
+ ("template_file", "expected_reason"),
+ [
+ ("../invoice.docx", "it leaves the bundle through a '..' segment"),
+ ("templates/../invoice.docx", "it leaves the bundle through a '..' segment"),
+ ("templates/invoice\x00.docx", "it holds a NUL character"),
+ ("/srv/methods/invoice.docx", "it is absolute"),
+ ("C:invoice.docx", "it holds a colon"),
+ ],
+ ids=["leaves_the_bundle", "dot_dot_inside", "nul_character", "absolute", "colon"],
+ )
+ def test_a_bundle_reading_submitted_files_refuses_a_path_none_could_be_keyed_by(self, template_file: str, expected_reason: str) -> None:
+ """A bundle loaded from a string reads only the files submitted with it, so its step is held to the rule they are keyed by."""
+ with pytest.raises(ValidationError) as exc_info:
+ _blueprint(file_origin=BundleFileOrigin.SUBMITTED_FILES, format="docx", template_file=template_file)
+
+ (error,) = exc_info.value.errors()
+ assert "PipeDocGen's 'template_file' names a file submitted with its bundle" in error["msg"]
+ assert expected_reason in error["msg"]
+ assert template_file not in error["msg"]
+
+ @pytest.mark.parametrize("template_file", ["invoice.docx", "./invoice.docx", "templates/invoice.docx", "templates//invoice.docx"])
+ def test_a_bundle_reading_submitted_files_accepts_a_path_inside_it(self, template_file: str) -> None:
+ blueprint = _blueprint(file_origin=BundleFileOrigin.SUBMITTED_FILES, format="docx", template_file=template_file)
assert blueprint.template_file == template_file
@pytest.mark.parametrize("template_file", ["/srv/methods/invoice.docx", "C:/srv/methods/invoice.docx", "srv\\methods\\invoice.docx"])