Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Hosted-structures-refusal · L-260824-ea92b8 (#1300)

Louis Choquelb2ffab84e62f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 01bd93e4d..72151a5a3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,12 +18,14 @@
- **An include in a method template is a render error**: `{% include %}`, `{% extends %}` and `{% import %}` in a template rendered without a loader raise `Jinja2TemplateRenderError`, where they leaked Jinja's own `TemplateNotFound`.
- **A template reads a `Composite`'s parts by name**: `{{ combo.summary }}`, `{{ combo['summary'] }}` and `combo.get('summary')` resolve the named parts of a `Composite` output, and `iter_keys` lists them; they used to render empty, and only the raw `_content`, now refused, reached them. Like any private name, a part whose name starts with an underscore is refused when a template reads it on an input.
- **A template's arithmetic error is a render error**: a division by zero, or a `range` longer than the sandbox's cap, raises `Jinja2TemplateRenderError` where the raw Python exception escaped, which validation reported as an internal error rather than as the method's.
+- **A sandboxed PipeFunc's imports resolve to the same file in every process**: the sandbox searches a method's root directory first and then its subdirectories in sorted order, where the order followed the hash of their paths, which changes from one process to the next. A module name used in two directories of a method, such as a `structures.py` shipped in a subdirectory beside the `structures` module the sandbox generates at the root, could resolve to either file. A temporary directory reached through a symlink, as on macOS, no longer lets a file's own directory jump ahead of that order or stay on `sys.path` after the run.
### Security
- **Every template renders in a sandbox (Breaking)**: prompts, compose and construct templates, image and search prompts and condition expressions may read data and call methods of plain values (strings, numbers, dates, containers) only, so a template that reaches for a Python internal, calls a method of a Pipelex or pydantic object, or reads a name starting with an underscore other than an input's `_stuff_name`, `_content_class`, `_concept_code` and `_stuff_code` raises `Jinja2TemplateSecurityError` instead of rendering, and validation refuses the visible cases with the new `template_private_name` error. `_content` and the `stuff` property are no longer readable on an input, and bracket access and `get` on an input resolve only its content fields and metadata fields; a plain dict's keys stay readable with brackets, `record['_id']` included, and a `namespace()` holds data only, refusing a callable. The `jinja2` floor rises to 3.1.6, the release carrying Jinja's own sandbox fix for the `attr` filter.
- **Dotted paths refuse underscore segments**: a PipeCompose construct `from` path such as `note.__class__.__init__.__globals__.sys.modules.os.environ` walked into Python internals with plain `getattr` and could copy the process environment into a method's output. A `from` path or `list_to_dict_keyed_by` name with a segment starting with an underscore is now refused at validation, and every dotted path the runtime walks, batch lists and prompt image references included, refuses one.
- **A scoped run reads only under its read scope, and never the local disk**: on a run with a read scope, a `pipelex-storage://` key is read only when it lies under that scope, compared segment by segment, and a bare path or a `file://` URI is never read, whichever value carries it — an input, a `PipeCompose` construct that assembles a URL from text, a model's structured output. Every content-generation leaf checks the URLs it will read before the dry-run branch, and the input seam checks a CSV input and every file it would upload or sign; a refused read raises `UriReadRefusedError`, answered as a 422, which names where the URL sat without quoting it. A run whose read scope is `None` reads as before.
+- **A sandbox-hosted load imports no method Python, and refuses structure classes (Breaking)**: on a deployment whose PipeFunc execution mode is not `direct`, loading a library directory captures every `.py` file as source for the sandbox and imports none of them, where a file declaring a `StructuredContent` subclass was imported into the loading process and its module-level code ran. A load whose Python declares such a class, however the method arrived, now raises `MethodStructuresRefusedError` naming each file and class by its relative path, the error a fetched package already got: declare the types as MTHDS concepts with inline structures, which a PipeFunc imports in the sandbox from the generated `structures` module. The module `pipelex build structures` writes is accepted while it is left as generated, by this refusal and by the fetched-package one, and is not sent to the sandbox, which generates its own from the same concepts. A `direct` load is unchanged.
## [v0.69.0] - 2026-09-28
diff --git a/docs/building-methods/concepts/python-classes.md b/docs/building-methods/concepts/python-classes.md
index ff188b7d8..77f1b35bd 100644
--- a/docs/building-methods/concepts/python-classes.md
+++ b/docs/building-methods/concepts/python-classes.md
@@ -44,6 +44,13 @@ Classes inheriting from `StructuredContent` are automatically discovered and reg
**Best practice:** Keep your `StructuredContent` classes in dedicated modules (e.g., `*_struct.py` files) with minimal module-level code, or ensure module-level code is safe to execute during discovery.
+!!! warning "Not available on the hosted API"
+ Python structure classes are an open-source and self-hosted feature. Using one means importing its module, and the hosted API never imports a method's Python into its own process. It refuses any method whose Python declares a `StructuredContent` subclass, whether the method was sent inline, saved to the catalog or fetched by address. The refusal is a [`MethodStructuresRefusedError`](../../errors/method-structures-refused-error.md) naming each file and class. Python that only holds PipeFuncs is accepted, because PipeFuncs run in an isolated sandbox.
+
+ To run such a method on the hosted API, declare its types as MTHDS concepts with [inline structures](inline-structures.md). A PipeFunc can still return them: in the sandbox, Pipelex generates a `structures` module from the method's concepts, so the function imports the class it returns with `from structures import <domain>__<Concept>`.
+
+ The `structures` module that `pipelex build structures` writes into a bundle is a copy of those same concepts, so the hosted API accepts it as long as it is left as generated. The API neither imports it nor sends it to the sandbox, which generates its own from the concepts with domain-qualified class names, so a PipeFunc imports them as shown above. Edit the file and it is refused like any hand-written class.
+
## With Custom Validation
```python
diff --git a/docs/building-methods/packages.md b/docs/building-methods/packages.md
index 647c64f67..f68b7c2d3 100644
--- a/docs/building-methods/packages.md
+++ b/docs/building-methods/packages.md
@@ -117,7 +117,7 @@ A few behaviors worth knowing:
- **Installed wins.** Once a method is installed, it is used as-is — including when a reference pins a different `@<tag>` than what is installed (a warning tells you so). Remove the installed directory to re-fetch.
- **A miss that cannot be bridged is a loud diagnostic, never a silent pass**: fetch disabled, an unfetchable address, or a failed fetch each raise an error naming the address and the remedy.
-- **Hosted parity warning.** A fetched method declaring Python structure classes runs locally but triggers the same warning as a direct CLI fetch: hosted execution accepts MTHDS concepts and sandboxed PipeFuncs, not in-process Python. On a sandbox-hosted deployment the fetch refuses such a package outright, with the same rule-naming error.
+- **Hosted parity warning.** A fetched method declaring Python structure classes runs locally but triggers the same warning as a direct CLI fetch: hosted execution accepts MTHDS concepts and sandboxed PipeFuncs, not in-process Python. On a sandbox-hosted deployment the fetch refuses such a package outright, with the same rule-naming error. The refusal is not specific to fetching: a sandbox-hosted deployment refuses every method whose Python declares structure classes, however it arrived (see [Python classes](concepts/python-classes.md)).
To **disable network fetches at load time**, set the switch in your `pipelex.toml` (or use the environment variable, which takes precedence):
diff --git a/docs/errors/method-structures-refused-error.md b/docs/errors/method-structures-refused-error.md
index 4612ae6bc..fd6b2e9b7 100644
--- a/docs/errors/method-structures-refused-error.md
+++ b/docs/errors/method-structures-refused-error.md
@@ -7,7 +7,7 @@ description: "Reference for the `MethodStructuresRefusedError` Pipelex error cla
# Method structures refused
-The fetched package declares in-process Python structure classes, which hosted execution refuses.
+The method declares Python structure classes, which hosted execution refuses to import into its own process.
| Field | Value |
|---|---|
diff --git a/pipelex/codegen/emitters/target.py b/pipelex/codegen/emitters/target.py
index cfa0b6e16..3374b5161 100644
--- a/pipelex/codegen/emitters/target.py
+++ b/pipelex/codegen/emitters/target.py
@@ -38,6 +38,14 @@ class CodegenTarget(StrEnum):
PYTHON_PYDANTIC = "python-pydantic"
TS_ZOD = "ts-zod"
+ @property
+ def is_python_structures(self) -> bool:
+ match self:
+ case CodegenTarget.PYTHON_STRUCTURES:
+ return True
+ case CodegenTarget.PYTHON_PYDANTIC | CodegenTarget.TS_ZOD:
+ return False
+
class EmittedFile(BaseModel):
"""One generated file: a filename relative to the output root and its full content."""
diff --git a/pipelex/codegen/stamp.py b/pipelex/codegen/stamp.py
index 3bc1fa86a..a618273f8 100644
--- a/pipelex/codegen/stamp.py
+++ b/pipelex/codegen/stamp.py
@@ -235,6 +235,9 @@ def _parse_options(options_raw: str) -> dict[str, str] | None:
loaded = json.loads(options_raw, parse_constant=_reject_json_constant)
except ValueError: # JSONDecodeError is a subclass, so this one clause covers malformed JSON too
return None
+ except RecursionError:
+ # A deeply nested value exhausts the parser's stack: not a stamp this module could have written.
+ return None
if not isinstance(loaded, dict):
return None
typed = cast("dict[str, object]", loaded)
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index d00eb0b6e..2dfa9b34a 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -49,6 +49,7 @@ from pipelex.libraries.library_utils import (
from pipelex.libraries.pipe.exceptions import PipeLibraryError
from pipelex.libraries.visibility_utils import check_visibility_for_blueprints, make_visibility_checker
from pipelex.methods.fetch_on_miss import resolve_address_based_method
+from pipelex.methods.structures_check import ensure_no_structured_content_in_library_sources, is_generated_structures_module
from pipelex.mthds_parsing.exceptions import MthdsParserError
from pipelex.mthds_parsing.handle_pipe_errors import categorize_pipe_validation_error
from pipelex.mthds_parsing.parser import MthdsParser
@@ -495,29 +496,29 @@ class LibraryManager(LibraryManagerAbstract):
# Import modules and register in global registries
# Import from user directories
- is_sandbox_hosted = is_pipe_func_sandbox_hosted()
- for library_dir in all_dirs:
- # Only import files that contain StructuredContent subclasses (uses AST pre-check).
- # Kept in hosted mode too: concepts declared as `structure = "ClassName"` resolve that
- # class from the registry at load time, so the structure classes must be present. These
- # are pydantic data classes, not the arbitrary PipeFunc bodies the hosted invariant guards.
- ClassRegistryUtils.import_modules_in_folder(
- folder_path=library_dir,
- base_class_names=[StructuredContent.__name__],
- force_include_dirs=[Path(builder_pkg.__file__).parent],
- )
- if is_sandbox_hosted:
- # Sandbox-hosted mode: never import/register the customer's PipeFunc bodies in this
- # process. Capture every .py as source text (no import) so it can travel to the
- # sandbox, where it is registered and executed instead. No force-include here (unlike
- # the direct branch): the only force-included dir is pipelex's own builder package,
- # which must NOT travel in a customer crate — the sandbox has pipelex installed.
- # Accumulate across dirs, but
- # fail loud on a relpath collision: the sandbox writes sources flat by relpath, so two
- # dirs sharing a path would otherwise silently clobber one customer's code and run the
- # wrong PipeFunc body.
- captured_sources = self._library_sources.setdefault(library_id, {})
- for relpath, source in FuncRegistryUtils.read_py_sources(folder_path=library_dir).items():
+ if is_pipe_func_sandbox_hosted():
+ # Sandbox-hosted mode imports no Python from a library directory, whatever its origin (an
+ # inline bundle, a stored method, a host directory): importing a file runs its module-level
+ # code in this process. Every .py is captured as source text instead, to travel to the
+ # sandbox, where it is imported and executed. A structure class cannot travel that way: it
+ # would have to be imported here to back a concept, so the load is refused, naming each file
+ # and class, before any source is kept. The refusal is a static scan, which a dynamically
+ # built class escapes; such a file then travels as source and still never executes here.
+ sources_by_dir = {library_dir: FuncRegistryUtils.read_py_sources(folder_path=library_dir) for library_dir in all_dirs}
+ ensure_no_structured_content_in_library_sources(sources_by_dir=sources_by_dir)
+ # No force-include here (unlike the direct branch): the only force-included dir is pipelex's
+ # own builder package, which must NOT travel in a customer crate — the sandbox has pipelex
+ # installed. Accumulate across dirs, but fail loud on a relpath collision: the sandbox writes
+ # sources flat by relpath, so two dirs sharing a path would otherwise silently clobber one
+ # customer's code and run the wrong PipeFunc body.
+ captured_sources = self._library_sources.setdefault(library_id, {})
+ for dir_sources in sources_by_dir.values():
+ for relpath, source in dir_sources.items():
+ if is_generated_structures_module(source=source):
+ # The refusal accepted it as a copy of the method's own concepts, and the sandbox
+ # generates its own from those concepts, qualified. Shipping this one too would give
+ # the sandbox two `structures` modules, and a PipeFunc's import would reach either.
+ continue
if relpath in captured_sources and captured_sources[relpath] != source:
msg = (
f"Duplicate PipeFunc source path '{relpath}' across library dirs while loading library "
@@ -526,9 +527,17 @@ class LibraryManager(LibraryManagerAbstract):
)
raise LibraryError(msg)
captured_sources[relpath] = source
- else:
- # Local/direct mode (unchanged): import files that contain @pipe_func decorated
- # functions (uses AST pre-check) and register them in the process-global func_registry.
+ else:
+ for library_dir in all_dirs:
+ # Only import files that contain StructuredContent subclasses (uses AST pre-check), so
+ # concepts declared as `structure = "ClassName"` resolve that class from the registry.
+ ClassRegistryUtils.import_modules_in_folder(
+ folder_path=library_dir,
+ base_class_names=[StructuredContent.__name__],
+ force_include_dirs=[Path(builder_pkg.__file__).parent],
+ )
+ # Import files that contain @pipe_func decorated functions (uses AST pre-check) and
+ # register them in the process-global func_registry.
FuncRegistryUtils.register_funcs_in_folder(
folder_path=library_dir,
force_include_dirs=[Path(builder_pkg.__file__).parent],
diff --git a/pipelex/methods/exceptions.py b/pipelex/methods/exceptions.py
index 342251214..8f0c1fdc6 100644
--- a/pipelex/methods/exceptions.py
+++ b/pipelex/methods/exceptions.py
@@ -39,7 +39,11 @@ class MethodPackageSymlinkError(MethodRefError):
class MethodStructuresRefusedError(MethodRefError):
- """The fetched package declares in-process Python structure classes, which hosted execution refuses."""
+ """The method declares Python structure classes, which hosted execution refuses to import into its own process.
+
+ Raised for a fetched package before it is installed, and for any sandbox-hosted library load,
+ whatever the content's origin: an inline bundle, a stored method or a host directory.
+ """
class MethodInstallError(MethodRefError):
diff --git a/pipelex/methods/structures_check.py b/pipelex/methods/structures_check.py
index e44fa8b63..c0a6037c3 100644
--- a/pipelex/methods/structures_check.py
+++ b/pipelex/methods/structures_check.py
@@ -1,27 +1,45 @@
-"""The structures-refusal check, reusable by the CLI, the runner, and the loader.
-
-The rule (execution locus decides): `.mthds` content is data — always acceptable. PipeFunc
-`.py` executes in the network-blocked sandbox — acceptable on sandbox-hosted deployments.
-`structures/*.py` is imported into the runner's own process, so a fetched package that
-declares `StructuredContent` subclasses is refused, loudly, with an error that names the
-rule. The discrimination is what the AST declares — never mere `.py` presence: PipeFunc-only
-Python is supported.
+"""The structures-refusal check, shared by the CLI, the fetcher and the library loader.
+
+The rule (execution locus decides): `.mthds` content is data, always acceptable. PipeFunc `.py`
+executes in the network-blocked sandbox, acceptable on sandbox-hosted deployments. A class
+subclassing `StructuredContent` would have to be imported into the runner's own process to back a
+concept, so hosted execution refuses any method content that declares one, loudly, with an error
+that names the rule. The discrimination is what the AST declares, never mere `.py` presence:
+PipeFunc-only Python is supported.
+
+Two doors apply it. A fetched package is scanned on disk before it is installed
+(`ensure_no_structured_content_python`). Every sandbox-hosted library load, whatever the content's
+origin (an inline bundle, a stored method, a host directory), scans the sources it would ship to
+the sandbox before it keeps any of them (`ensure_no_structured_content_in_library_sources`). Both
+run the same walk over one source string, so the two refusals cannot drift apart.
+
+One file declaring such classes is not refused: the `python-structures` projection that
+`pipelex build structures` writes, left as generated. It is a copy of the method's own MTHDS
+concepts, which is exactly what the refusal asks for. A hosted load imports it no more than any other
+file, and does not ship it either, since the sandbox generates its own from the same concepts. Its
+stamp is not a signature, anyone can compute its hash, so the exemption is sound only because
+nothing it lets through is imported into the runner; a hand-edited copy is refused.
The scan is alias-aware: a base written as `StructuredContent`, imported under another name
(`from ... import StructuredContent as SC`), or reached as an attribute
-(`module.StructuredContent`, whatever the module was imported as) is caught. It remains a
-static pre-check, and honestly so: dynamic tricks (rebinding through assignments, metaclass
-construction, `type(...)` calls) are out of its scope — its job is that straightforward
-declarations and straightforward aliasing cannot defeat the refusal. A subclass of a caught
-class in the same package is covered transitively, because the file declaring the base is
-itself refused and refusal applies to the whole package.
+(`module.StructuredContent`, whatever the module was imported as) is caught. It remains a static
+pre-check, and honestly so: dynamic tricks (rebinding through assignments, metaclass construction,
+`type(...)` calls) are out of its scope. Its job is that straightforward declarations and
+straightforward aliasing cannot defeat the refusal. On a sandbox-hosted load that is enough,
+because the loader imports none of the customer's Python: a file that escapes the scan travels to
+the sandbox as source and never executes in the runner. A subclass of a class declared elsewhere
+escapes it the same way, whether its base comes from the exempt generated module or from Pipelex
+itself: it is never imported here, so a concept naming it fails to resolve its structure class
+instead of getting this refusal.
"""
import ast
+from collections.abc import Mapping
from pathlib import Path
from pydantic import BaseModel, ConfigDict
+from pipelex.codegen.stamp import compute_content_hash, parse_stamped
from pipelex.methods.exceptions import MethodStructuresRefusedError
STRUCTURE_BASE_CLASS_NAME = "StructuredContent"
@@ -73,13 +91,62 @@ class StructuredContentViolation(BaseModel):
class_names: list[str]
+def structured_content_class_names_in_source(*, source: str, filename: str) -> list[str]:
+ """Class names declared in one Python source whose bases resolve to `StructuredContent`.
+
+ This is the one walk both refusals run, over the text itself, so the fetch-time scan of a
+ directory and the load-time scan of captured sources cannot drift apart. A source that does not
+ parse declares nothing: it cannot be imported either, so it cannot smuggle a structure class
+ into the process. That includes a source nested deeply enough to exhaust the parser, which
+ Python's own import rejects the same way.
+
+ Args:
+ source: The Python source text.
+ filename: The name reported by the parser, never shown to a caller.
+
+ Returns:
+ The declared subclass names, in source order; empty when the source does not parse.
+ """
+ try:
+ tree = ast.parse(source, filename=filename)
+ except (SyntaxError, ValueError):
+ return []
+ except (RecursionError, MemoryError):
+ # A deep attribute chain exhausts the AST builder's recursion, a deep run of unary operators
+ # overflows the parser's stack ("Parser stack overflowed"): too complex to import, like a syntax error.
+ return []
+ return _structured_content_class_names(tree=tree)
+
+
+def is_generated_structures_module(*, source: str) -> bool:
+ """Whether a source is the stamped `python-structures` projection, unedited below its stamp.
+
+ That projection is what `pipelex build structures` writes into a bundle: the method's own MTHDS
+ concepts rendered as `StructuredContent` classes. The stamp's hash proves only that the body was
+ not edited after stamping, not who stamped it, so this is never a trust decision: it spares an
+ author who generated their types a refusal whose remedy they already applied.
+ """
+ parsed = parse_stamped(source, comment_prefix="#")
+ if parsed is None:
+ return False
+ return parsed.stamp.target.is_python_structures and parsed.stamp.content_hash == compute_content_hash(parsed.body)
+
+
+def _refused_class_names(*, source: str, filename: str) -> list[str]:
+ """The structure classes a source would be refused for: none for the generated projection, else the walk's."""
+ if is_generated_structures_module(source=source):
+ return []
+ return structured_content_class_names_in_source(source=source, filename=filename)
+
+
def scan_structured_content_classes(*, package_dir: Path) -> list[StructuredContentViolation]:
"""AST-scan a package directory for `.py` files declaring `StructuredContent` subclasses.
Alias-aware: bases reached through `from ... import StructuredContent as SC` or through
an attribute access (`module.StructuredContent`) are caught alongside the literal name.
- Files that cannot be parsed are skipped: an unparseable file cannot be imported either,
- so it cannot smuggle a structure class into the process.
+ Files that cannot be read or parsed are skipped: such a file cannot be imported either,
+ so it cannot smuggle a structure class into the process. The generated `python-structures`
+ projection, left as generated, is not a violation (`is_generated_structures_module`).
Args:
package_dir: The package directory to scan.
@@ -94,17 +161,35 @@ def scan_structured_content_classes(*, package_dir: Path) -> list[StructuredCont
continue
try:
source = py_file.read_text(encoding="utf-8")
- tree = ast.parse(source, filename=str(py_file))
- except (OSError, SyntaxError, ValueError):
- # Unparseable or unreadable file: it cannot be imported either, so it cannot
- # smuggle a structure class into the process.
+ except (OSError, ValueError):
continue
- class_names = _structured_content_class_names(tree=tree)
+ class_names = _refused_class_names(source=source, filename=str(py_file))
if class_names:
violations.append(StructuredContentViolation(relative_path=relative.as_posix(), class_names=class_names))
return violations
+def scan_structured_content_sources(*, sources: Mapping[str, str]) -> list[StructuredContentViolation]:
+ """Scan captured Python sources, keyed by relative path, for `StructuredContent` subclasses.
+
+ The sandbox-hosted loader calls this on the very mapping it would ship, so the bytes it refuses
+ are exactly the bytes that would travel, with no second read of the disk. The generated
+ `python-structures` projection, left as generated, is not a violation (`is_generated_structures_module`).
+
+ Args:
+ sources: POSIX relative path to source text, as `FuncRegistryUtils.read_py_sources` returns it.
+
+ Returns:
+ One violation per offending source, ordered by relative path.
+ """
+ violations: list[StructuredContentViolation] = []
+ for relative_path in sorted(sources):
+ class_names = _refused_class_names(source=sources[relative_path], filename=relative_path)
+ if class_names:
+ violations.append(StructuredContentViolation(relative_path=relative_path, class_names=class_names))
+ return violations
+
+
def describe_structured_content_violations(*, violations: list[StructuredContentViolation]) -> str:
"""Render violations as a compact, human-readable listing."""
return "; ".join(f"{violation.relative_path} defines {', '.join(violation.class_names)}" for violation in violations)
@@ -131,3 +216,37 @@ def ensure_no_structured_content_python(*, package_dir: Path, package_address: s
f"with inline structures instead of Python classes."
)
raise MethodStructuresRefusedError(msg)
+
+
+def ensure_no_structured_content_in_library_sources(*, sources_by_dir: Mapping[Path, Mapping[str, str]]) -> None:
+ """Refuse a sandbox-hosted library load whose Python declares structure classes.
+
+ Every directory is scanned before anything is raised, so one refusal names every offending file
+ of the load. The message names files by their path relative to their library directory, never
+ the directory itself: on a host that directory is a temporary one, which is not the caller's to read.
+
+ Args:
+ sources_by_dir: For each library directory of the load, the Python sources it would ship,
+ keyed by POSIX relative path.
+
+ Raises:
+ MethodStructuresRefusedError: If any source declares a `StructuredContent` subclass.
+ The message names the rule and teaches the fix.
+ """
+ violations: list[StructuredContentViolation] = []
+ for sources in sources_by_dir.values():
+ for violation in scan_structured_content_sources(sources=sources):
+ # Two directories shipping the same file would otherwise be listed twice.
+ if violation not in violations:
+ violations.append(violation)
+ if not violations:
+ return
+ details = describe_structured_content_violations(violations=violations)
+ msg = (
+ f"This method declares Python structure classes ({details}), which would be imported into the runner's "
+ f"own process. Refused: {STRUCTURES_REFUSAL_RULE}. Declare these types as MTHDS concepts with inline "
+ f"structures; a PipeFunc can return them by importing the classes the sandbox generates from those "
+ f"concepts (`from structures import <domain>__<Concept>`). A structures module generated by "
+ f"`pipelex build structures` is accepted as long as it is left as generated."
+ )
+ raise MethodStructuresRefusedError(msg)
diff --git a/pipelex/pipe_operators/func/direct_pipe_func_executor.py b/pipelex/pipe_operators/func/direct_pipe_func_executor.py
index 0d10dadd8..6c0575d44 100644
--- a/pipelex/pipe_operators/func/direct_pipe_func_executor.py
+++ b/pipelex/pipe_operators/func/direct_pipe_func_executor.py
@@ -28,6 +28,23 @@ from pipelex.system.registries.func_registry_utils import FuncRegistryUtils
_TRANSPORTED_LIBRARY_ID = "pipe_func_transported_lib"
+def bundle_import_dirs(*, workdir: Path) -> list[Path]:
+ """The directories a materialized bundle's modules import from, in `sys.path` search order.
+
+ The bundle root comes first and the subdirectories holding a `.py` file follow in sorted order, so a
+ module name that two of them share resolves to the same file in every process. In particular, the
+ `structures` module generated at the root wins over any `structures.py` a subdirectory ships.
+
+ Args:
+ workdir: The directory the bundle's sources were materialized into.
+
+ Returns:
+ The root followed by its source subdirectories, each listed once.
+ """
+ subdirs = {py_file.parent for py_file in workdir.rglob("*.py")} - {workdir}
+ return [workdir, *sorted(subdirs)]
+
+
class DirectPipeFuncExecutor(PipeFuncExecutorProtocol):
"""The ``direct`` execution mode: resolve the function from the process-global registry and run it here.
@@ -76,7 +93,10 @@ class DirectPipeFuncExecutor(PipeFuncExecutorProtocol):
isolation boundary. The output rides back wrapped in a working memory so it round-trips with its
concept (and thus its dynamic-class identity), rebound against the receiver's registry.
"""
- workdir = Path(tempfile.mkdtemp(prefix="pipelex_pipe_func_"))
+ # Resolved, because importing a file puts its resolved directory on sys.path: under a symlinked
+ # temp dir (`/var` is `/private/var` on macOS) those entries would jump ahead of the bundle's
+ # import order and escape the prune below.
+ workdir = Path(tempfile.mkdtemp(prefix="pipelex_pipe_func_")).resolve()
try:
return await self._run_transported_from_workdir(request=request, workdir=workdir)
finally:
@@ -128,9 +148,7 @@ class DirectPipeFuncExecutor(PipeFuncExecutorProtocol):
# workdir (resolves `from funcs.helpers import ...`) and every subdirectory holding a .py file
# (resolves the sibling-relative `from helpers import ...`); the caller's finally prunes these
# entries once the import phase is over, so they never outlive the run.
- source_dirs = {workdir} | {py_file.parent for py_file in workdir.rglob("*.py")}
- for source_dir in source_dirs:
- sys.path.insert(0, str(source_dir))
+ sys.path[0:0] = [str(source_dir) for source_dir in bundle_import_dirs(workdir=workdir)]
# Register the customer's structure classes and @pipe_func functions from their real source,
# before rehydration seeds a per-run registry from the global one and before the function lookup.
diff --git a/pipelex/system/registries/func_registry_utils.py b/pipelex/system/registries/func_registry_utils.py
index cf197cb04..5212979dd 100644
--- a/pipelex/system/registries/func_registry_utils.py
+++ b/pipelex/system/registries/func_registry_utils.py
@@ -125,8 +125,10 @@ class FuncRegistryUtils:
runner/worker from ever executing customer code.
Discovery mirrors ``register_funcs_in_folder`` (same ``find_files_in_dir`` + ``excluded_dirs``)
- so the captured set matches what the local path would have imported. Both PipeFunc bodies and
- structure classes are captured, since the sandbox needs the customer's real classes too.
+ so the captured set matches what the local path would have imported. Every file is captured,
+ PipeFunc bodies and their helpers alike. A structure class never travels: the sandbox-hosted load
+ scans this very mapping and refuses a library that declares one before keeping any of it
+ (``ensure_no_structured_content_in_library_sources``).
Args:
folder_path: Path to the folder containing Python files.
diff --git a/tests/integration/pipelex/libraries/conftest.py b/tests/integration/pipelex/libraries/conftest.py
new file mode 100644
index 000000000..d2fa4f4eb
--- /dev/null
+++ b/tests/integration/pipelex/libraries/conftest.py
@@ -0,0 +1,24 @@
+from collections.abc import Generator
+
+import pytest
+
+from pipelex.config import get_config
+
+
+def sandbox_hosted_mode() -> Generator[None, None, None]:
+ """Select a non-``direct`` execution mode for the duration of a test, then restore it.
+
+ Flipping config (not monkeypatching the helper) exercises the genuine
+ is_pipe_func_sandbox_hosted() read at every call site (loader + validators) together: any
+ non-``direct`` mode is sandbox-hosted. The concrete sandbox modes live in the out-of-tree closed
+ plugin, so this open-core test uses a neutral ``"sandbox"`` token — it only flips the hosted flag
+ (the executor already on the hub is never re-resolved here), so no registered mode is needed.
+ """
+ pipe_func_config = get_config().interpreter.pipe_func
+ previous = pipe_func_config.execution_mode
+ pipe_func_config.execution_mode = "sandbox"
+ try:
+ yield
+ finally:
+ pipe_func_config.execution_mode = previous
diff --git a/tests/integration/pipelex/libraries/test_hosted_pipe_func_load.py b/tests/integration/pipelex/libraries/test_hosted_pipe_func_load.py
index b06532822..040e41011 100644
--- a/tests/integration/pipelex/libraries/test_hosted_pipe_func_load.py
+++ b/tests/integration/pipelex/libraries/test_hosted_pipe_func_load.py
@@ -1,10 +1,8 @@
import tempfile
-from collections.abc import Generator
from pathlib import Path
import pytest
-from pipelex.config import get_config
from pipelex.interpreter_hub import get_library_manager, scoped_current_library
from pipelex.libraries.exceptions import LibraryError
from pipelex.system.registries.func_registry import func_registry
@@ -36,25 +34,6 @@ async def make_note_in_sandbox(working_memory: WorkingMemory) -> TextContent:
"""
-def sandbox_hosted_mode() -> Generator[None, None, None]:
- """Select a non-``direct`` execution mode for the duration of a test, then restore it.
-
- Flipping config (not monkeypatching the helper) exercises the genuine
- is_pipe_func_sandbox_hosted() read at every call site (loader + validators) together: any
- non-``direct`` mode is sandbox-hosted. The concrete sandbox modes live in the out-of-tree closed
- plugin, so this open-core test uses a neutral ``"sandbox"`` token — it only flips the hosted flag
- (the executor already on the hub is never re-resolved here), so no registered mode is needed.
- """
- pipe_func_config = get_config().interpreter.pipe_func
- previous = pipe_func_config.execution_mode
- pipe_func_config.execution_mode = "sandbox"
- try:
- yield
- finally:
- pipe_func_config.execution_mode = previous
-
-
class TestHostedPipeFuncLoad:
"""Sandbox-hosted load: the customer's PipeFunc .py is captured as crate source and is NEVER
registered/executed in this process, yet the method still loads (validators skip the callable).
diff --git a/tests/integration/pipelex/libraries/test_hosted_structures_refusal.py b/tests/integration/pipelex/libraries/test_hosted_structures_refusal.py
new file mode 100644
index 000000000..14373693d
--- /dev/null
+++ b/tests/integration/pipelex/libraries/test_hosted_structures_refusal.py
@@ -0,0 +1,318 @@
+"""A sandbox-hosted library load imports no customer Python, and refuses one that declares structure classes.
+
+Importing a file runs its module-level code, so on a sandbox-hosted deployment the loader captures every
+`.py` as source for the sandbox and imports none of it. A `StructuredContent` subclass would have to be
+imported into the runner's own process to back a concept, so a load whose sources declare one is refused
+with `MethodStructuresRefusedError`, the error a fetched package already gets. Each structure file here
+carries a module-level sentinel that writes a marker file, so a test can tell whether the file executed.
+"""
+
+import re
+import sys
+from collections.abc import Callable
+from pathlib import Path
+
+import pytest
+from mthds.package.manifest.schema import MTHDS_STANDARD_VERSION
+
+from pipelex.codegen.emission import write_stamped_projection
+from pipelex.codegen.emitters.target import CodegenKind, CodegenTarget
+from pipelex.codegen.emitters.types_emitter import emit_types
+from pipelex.codegen.stamp import apply_stamp
+from pipelex.core.concepts.exceptions import ConceptFactoryError
+from pipelex.interpreter_hub import get_concept_library, get_library_manager, scoped_current_library
+from pipelex.libraries.crate_normalization import normalize_crate
+from pipelex.methods.exceptions import MethodStructuresRefusedError
+
+SENTINEL_LINE = 'open(r"{marker}", "w").write("ran")\n'
+
+STRUCTURE_FILE_PY = """\
+from pipelex.core.stuffs.structured_content import StructuredContent
+
+
+class {class_name}(StructuredContent):
+ total: float
+
+
+"""
+
+ALIASED_STRUCTURE_FILE_PY = """\
+from pipelex.core.stuffs.structured_content import StructuredContent as SC
+
+
+class {class_name}(SC):
+ total: float
+
+
+"""
+
+ATTRIBUTE_STRUCTURE_FILE_PY = """\
+import pipelex.core.stuffs.structured_content as sc_module
+
+
+class {class_name}(sc_module.StructuredContent):
+ total: float
+
+
+"""
+
+PIPE_FUNC_WITH_OWN_CLASS_PY = """\
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.system.registries.func_registry import pipe_func
+
+
+class {class_name}(StructuredContent):
+ total: float
+
+
+@pipe_func(name="make_hosted_refusal_total")
+async def make_hosted_refusal_total(working_memory: WorkingMemory) -> {class_name}:
+ return {class_name}(total=1.0)
+
+
+"""
+
+DYNAMIC_STRUCTURE_FILE_PY = """\
+from pipelex.core.stuffs.structured_content import StructuredContent
+
+{class_name} = type("{class_name}", (StructuredContent,), {{"__annotations__": {{"total": float}}}})
+
+
+"""
+
+INLINE_STRUCTURE_MTHDS = """\
+domain = "hosted_generated"
+description = "A concept with an inline structure"
+
+[concept.Receipt]
+description = "A receipt"
+
+[concept.Receipt.structure]
+total = { type = "number", description = "The total", required = true }
+"""
+
+CLASS_BACKED_MTHDS = """\
+domain = "hosted_refusal"
+description = "A concept backed by a Python class"
+
+[concept.Invoice]
+description = "An invoice"
+structure = "{class_name}"
+"""
+
+
+def _write_structure_file(*, path: Path, template: str, class_name: str, marker: Path) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(template.format(class_name=class_name) + SENTINEL_LINE.format(marker=marker), encoding="utf-8")
+
+
+def _is_imported(*, path: Path) -> bool:
+ """Whether any module in this process was loaded from *path*."""
+ resolved = path.resolve()
+ for module in list(sys.modules.values()):
+ module_file = getattr(module, "__file__", None)
+ if module_file and Path(module_file).resolve() == resolved:
+ return True
+ return False
+
+
+class TestHostedStructuresRefusal:
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_structure_file_is_refused_and_never_executed(self, tmp_path: Path, load_empty_library: Callable[[], str]):
+ library_dir = tmp_path / "bundle"
+ structure_file = library_dir / "structures" / "invoice.py"
+ marker = tmp_path / "sentinel.ran"
+ _write_structure_file(path=structure_file, template=STRUCTURE_FILE_PY, class_name="HostedRefusalInvoice", marker=marker)
+
+ library_id = load_empty_library()
+ with pytest.raises(MethodStructuresRefusedError) as exc_info:
+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[library_dir])
+
+ message = str(exc_info.value)
+ assert "structures/invoice.py defines HostedRefusalInvoice" in message
+ assert "not in-process Python" in message
+ assert "from structures import <domain>__<Concept>" in message
+ assert not marker.exists(), "the structure file's module-level code ran in the loading process"
+ assert not _is_imported(path=structure_file)
+ # Refused before anything was kept: no crate carries the source.
+ assert get_library_manager().get_crate(library_id=library_id) is None
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ @pytest.mark.parametrize(
+ ("template", "class_name"),
+ [
+ (ALIASED_STRUCTURE_FILE_PY, "HostedRefusalAliasedInvoice"),
+ (ATTRIBUTE_STRUCTURE_FILE_PY, "HostedRefusalAttributeInvoice"),
+ (PIPE_FUNC_WITH_OWN_CLASS_PY, "HostedRefusalPipeFuncTotal"),
+ ],
+ ids=["aliased-import", "attribute-base", "pipe-func-declaring-its-class"],
+ )
+ def test_every_declared_form_is_refused(
+ self,
+ tmp_path: Path,
+ load_empty_library: Callable[[], str],
+ template: str,
+ class_name: str,
+ ):
+ library_dir = tmp_path / "bundle"
+ py_file = library_dir / "customer.py"
+ marker = tmp_path / "sentinel.ran"
+ _write_structure_file(path=py_file, template=template, class_name=class_name, marker=marker)
+
+ library_id = load_empty_library()
+ with pytest.raises(MethodStructuresRefusedError, match=re.escape(f"customer.py defines {class_name}")):
+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[library_dir])
+
+ assert not marker.exists()
+ assert not _is_imported(path=py_file)
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_dynamic_class_escapes_the_scan_but_never_executes(self, tmp_path: Path, load_empty_library: Callable[[], str]):
+ """A class built by `type(...)` escapes the static scan, so the load is not refused. It is not imported either:
+ the file travels as source, its module-level code never runs here, and a concept naming the class finds none.
+ """
+ class_name = "HostedRefusalDynamicInvoice"
+ library_dir = tmp_path / "bundle"
+ py_file = library_dir / "structures" / "invoice.py"
+ marker = tmp_path / "sentinel.ran"
+ _write_structure_file(path=py_file, template=DYNAMIC_STRUCTURE_FILE_PY, class_name=class_name, marker=marker)
+ (library_dir / "invoice.mthds").write_text(CLASS_BACKED_MTHDS.format(class_name=class_name), encoding="utf-8")
+
+ library_id = load_empty_library()
+ with pytest.raises(ConceptFactoryError, match=f"Structure class '{class_name}'.*not a registered subclass of StuffContent"):
+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[library_dir])
+
+ assert not marker.exists(), "the dynamically built class's module-level code ran in the loading process"
+ assert not _is_imported(path=py_file)
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_structure_file_under_an_excluded_dir_is_not_refused(self, tmp_path: Path, load_empty_library: Callable[[], str]):
+ """A file under an excluded directory is neither shipped nor imported, so it is not refused."""
+ library_dir = tmp_path / "bundle"
+ py_file = library_dir / ".venv" / "lib" / "invoice.py"
+ marker = tmp_path / "sentinel.ran"
+ _write_structure_file(path=py_file, template=STRUCTURE_FILE_PY, class_name="HostedRefusalVenvInvoice", marker=marker)
+
+ library_id = load_empty_library()
+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[library_dir])
+
+ assert not marker.exists()
+ assert not _is_imported(path=py_file)
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_one_refusal_lists_every_directory(self, tmp_path: Path, load_empty_library: Callable[[], str]):
+ dir_a = tmp_path / "a"
+ dir_b = tmp_path / "b"
+ marker = tmp_path / "sentinel.ran"
+ _write_structure_file(path=dir_a / "structures" / "invoice.py", template=STRUCTURE_FILE_PY, class_name="HostedRefusalA", marker=marker)
+ _write_structure_file(path=dir_b / "models" / "receipt.py", template=STRUCTURE_FILE_PY, class_name="HostedRefusalB", marker=marker)
+
+ library_id = load_empty_library()
+ with pytest.raises(MethodStructuresRefusedError) as exc_info:
+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[dir_a, dir_b])
+
+ message = str(exc_info.value)
+ assert "structures/invoice.py defines HostedRefusalA" in message
+ assert "models/receipt.py defines HostedRefusalB" in message
+ assert not marker.exists()
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_refusal_names_relative_paths_only(self, tmp_path: Path, load_empty_library: Callable[[], str]):
+ """On a host the library directory is a temporary one, which is not the caller's to read."""
+ library_dir = tmp_path / "bundle"
+ marker = tmp_path / "sentinel.ran"
+ _write_structure_file(
+ path=library_dir / "structures" / "invoice.py", template=STRUCTURE_FILE_PY, class_name="HostedRefusalPathInvoice", marker=marker
+ )
+
+ library_id = load_empty_library()
+ with pytest.raises(MethodStructuresRefusedError) as exc_info:
+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[library_dir])
+
+ message = str(exc_info.value)
+ assert str(tmp_path) not in message
+ assert str(tmp_path.resolve()) not in message
+ assert tmp_path.name not in message
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_generated_structures_module_is_accepted_and_not_shipped(self, tmp_path: Path, load_empty_library: Callable[[], str]):
+ """The module `pipelex build structures` writes into a bundle copies the method's own MTHDS concepts, so a
+ hosted load accepts it as generated. It is never imported here, and it does not travel either: the sandbox
+ generates its own from the same concepts, and two `structures` modules there would shadow each other.
+ """
+ library_dir = tmp_path / "bundle"
+ library_dir.mkdir()
+ (library_dir / "receipt.mthds").write_text(INLINE_STRUCTURE_MTHDS, encoding="utf-8")
+ library_manager = get_library_manager()
+ generation_library_id, _ = library_manager.open_library()
+ try:
+ with scoped_current_library(library_id=generation_library_id):
+ library_manager.load_libraries(library_id=generation_library_id, library_dirs=[library_dir])
+ crate = library_manager.get_crate(library_id=generation_library_id)
+ assert crate is not None
+ normalized = normalize_crate(crate, mthds_version=MTHDS_STANDARD_VERSION)
+ write_stamped_projection(
+ emit_types(normalized, target=CodegenTarget.PYTHON_STRUCTURES),
+ output_dir=library_dir / "structures",
+ crate_fingerprint=normalized.fingerprint,
+ engine_version="0.0.0-test",
+ kind=CodegenKind.TYPES,
+ target=CodegenTarget.PYTHON_STRUCTURES,
+ )
+ finally:
+ library_manager.teardown(library_id=generation_library_id)
+ generated_file = library_dir / "structures" / "structures.py"
+ generated_source = generated_file.read_text(encoding="utf-8")
+ assert "(StructuredContent)" in generated_source
+ helper_source = "GREETING = 'hello'\n"
+ (library_dir / "helpers.py").write_text(helper_source, encoding="utf-8")
+
+ library_id = load_empty_library()
+ library_manager.load_libraries(library_id=library_id, library_dirs=[library_dir])
+
+ hosted_crate = library_manager.get_crate(library_id=library_id)
+ assert hosted_crate is not None
+ assert "structures/structures.py" not in hosted_crate.python_sources
+ assert hosted_crate.python_sources.get("helpers.py") == helper_source
+ assert not _is_imported(path=generated_file)
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_edited_generated_structures_module_is_refused(self, tmp_path: Path, load_empty_library: Callable[[], str]):
+ """A generated module edited below its stamp may carry a hand-written class, so it is refused like one."""
+ library_dir = tmp_path / "bundle"
+ (library_dir / "structures").mkdir(parents=True)
+ body = STRUCTURE_FILE_PY.format(class_name="HostedRefusalEditedInvoice")
+ stamped = apply_stamp(
+ body,
+ crate_fingerprint="0" * 64,
+ engine_version="0.0.0-test",
+ kind=CodegenKind.TYPES,
+ target=CodegenTarget.PYTHON_STRUCTURES,
+ pipe_ref=None,
+ options={},
+ comment_prefix="#",
+ )
+ (library_dir / "structures" / "structures.py").write_text(stamped + " extra: str = ''\n", encoding="utf-8")
+
+ library_id = load_empty_library()
+ with pytest.raises(MethodStructuresRefusedError, match=re.escape("structures/structures.py defines HostedRefusalEditedInvoice")):
+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[library_dir])
+
+ def test_direct_mode_imports_the_structure_class(self, tmp_path: Path, load_empty_library: Callable[[], str]):
+ """Structure classes stay an open-source and self-hosted feature: a direct-mode load imports the file and
+ the concept resolves to its class.
+ """
+ class_name = "HostedRefusalDirectModeInvoice"
+ library_dir = tmp_path / "bundle"
+ (library_dir / "structures").mkdir(parents=True)
+ (library_dir / "structures" / "invoice.py").write_text(STRUCTURE_FILE_PY.format(class_name=class_name), encoding="utf-8")
+ (library_dir / "invoice.mthds").write_text(CLASS_BACKED_MTHDS.format(class_name=class_name), encoding="utf-8")
+
+ library_id = load_empty_library()
+ get_library_manager().load_libraries(library_id=library_id, library_dirs=[library_dir])
+
+ concept_library = get_concept_library()
+ invoice_class = concept_library.get_structure_class(concept=concept_library.get_required_concept("hosted_refusal.Invoice"))
+ assert invoice_class.__name__ == class_name
+ assert _is_imported(path=library_dir / "structures" / "invoice.py")
diff --git a/tests/unit/pipelex/codegen/test_stamp.py b/tests/unit/pipelex/codegen/test_stamp.py
index 14f08101b..deb5b6af8 100644
--- a/tests/unit/pipelex/codegen/test_stamp.py
+++ b/tests/unit/pipelex/codegen/test_stamp.py
@@ -125,6 +125,13 @@ class TestStamp:
tampered = stamped.replace("# options: {}", f"# options: {options_value}")
assert parse_stamped(tampered, comment_prefix="#") is None
+ def test_deeply_nested_options_are_not_a_stamp(self) -> None:
+ # A hosted load parses the stamp of every customer `.py` it scans, so a nesting deep enough to
+ # exhaust the JSON parser's stack must read as "no stamp", not escape as a RecursionError.
+ stamped = self._stamp("# a\nclass A:\n pass\n", comment_prefix="#")
+ tampered = stamped.replace("# options: {}", "# options: " + "[" * 100_000)
+ assert parse_stamped(tampered, comment_prefix="#") is None
+
def test_standard_options_json_still_parses(self) -> None:
empty = parse_stamped(self._stamp("# a\n", comment_prefix="#"), comment_prefix="#")
assert empty is not None
diff --git a/tests/unit/pipelex/methods/test_structures_check.py b/tests/unit/pipelex/methods/test_structures_check.py
index a6cab4a58..cf14eaa44 100644
--- a/tests/unit/pipelex/methods/test_structures_check.py
+++ b/tests/unit/pipelex/methods/test_structures_check.py
@@ -2,15 +2,23 @@
from __future__ import annotations
+from pathlib import PurePosixPath
from typing import TYPE_CHECKING
import pytest
+from pipelex.codegen.emitters.target import CodegenKind, CodegenTarget
+from pipelex.codegen.stamp import apply_stamp
from pipelex.methods.exceptions import MethodStructuresRefusedError
from pipelex.methods.structures_check import (
STRUCTURES_REFUSAL_RULE,
+ StructuredContentViolation,
+ ensure_no_structured_content_in_library_sources,
ensure_no_structured_content_python,
+ is_generated_structures_module,
scan_structured_content_classes,
+ scan_structured_content_sources,
+ structured_content_class_names_in_source,
)
if TYPE_CHECKING:
@@ -67,6 +75,19 @@ class Helper(SC):
"""
+def _stamped(body: str, *, target: CodegenTarget) -> str:
+ return apply_stamp(
+ body,
+ crate_fingerprint="0" * 64,
+ engine_version="0.0.0-test",
+ kind=CodegenKind.TYPES,
+ target=target,
+ pipe_ref=None,
+ options={},
+ comment_prefix="#",
+ )
+
+
class TestStructuresCheck:
"""Tests for the AST-based StructuredContent refusal — never gating on mere .py presence."""
@@ -154,3 +175,104 @@ class TestStructuresCheck:
assert len(violations) == 1
assert violations[0].relative_path == "structures/models.py"
+
+ def test_source_walk_matches_the_directory_scan(self, tmp_path: Path) -> None:
+ """Both refusals run one walk, so a directory and its captured sources yield the same violations."""
+ modules = {
+ "structures/invoice.py": STRUCTURES_MODULE,
+ "aliased.py": ALIASED_IMPORT_MODULE,
+ "attribute_base.py": ATTRIBUTE_BASE_MODULE,
+ "funcs.py": PIPE_FUNC_MODULE,
+ "unrelated.py": UNRELATED_ALIAS_MODULE,
+ }
+ for relative_path, source in modules.items():
+ file_path = tmp_path / PurePosixPath(relative_path)
+ file_path.parent.mkdir(parents=True, exist_ok=True)
+ file_path.write_text(source, encoding="utf-8")
+
+ assert scan_structured_content_sources(sources=modules) == scan_structured_content_classes(package_dir=tmp_path)
+
+ def test_unparseable_source_declares_nothing(self) -> None:
+ assert structured_content_class_names_in_source(source="def broken(:\n", filename="broken.py") == []
+ assert structured_content_class_names_in_source(source="x = 1\0\n", filename="nul.py") == []
+
+ @pytest.mark.parametrize(
+ "source",
+ [
+ pytest.param("x = a" + ".a" * 200_000 + "\n", id="attribute_chain_exhausts_recursion"),
+ pytest.param("x = " + "-" * 200_000 + "1\n", id="unary_run_overflows_parser_stack"),
+ ],
+ )
+ def test_source_too_deep_to_parse_declares_nothing(self, source: str) -> None:
+ # Every sandbox-hosted load scans every customer `.py`, so a source that exhausts the parser
+ # must read as declaring nothing, not escape the load as a RecursionError or MemoryError.
+ assert structured_content_class_names_in_source(source=source, filename="deep.py") == []
+
+ def test_sources_are_reported_in_path_order(self) -> None:
+ violations = scan_structured_content_sources(sources={"z.py": STRUCTURES_MODULE, "a.py": ALIASED_IMPORT_MODULE})
+
+ assert violations == [
+ StructuredContentViolation(relative_path="a.py", class_names=["Invoice"]),
+ StructuredContentViolation(relative_path="z.py", class_names=["Invoice", "LineItem"]),
+ ]
+
+ def test_pipe_func_only_sources_load(self, tmp_path: Path) -> None:
+ ensure_no_structured_content_in_library_sources(sources_by_dir={tmp_path: {"funcs.py": PIPE_FUNC_MODULE, "helper.py": PLAIN_MODULE}})
+
+ def test_library_refusal_names_files_rule_and_route(self, tmp_path: Path) -> None:
+ """The load-time message names each file by its relative path, the rule, and the MTHDS route for a PipeFunc."""
+ dir_a = tmp_path / "a"
+ dir_b = tmp_path / "b"
+
+ with pytest.raises(MethodStructuresRefusedError) as exc_info:
+ ensure_no_structured_content_in_library_sources(
+ sources_by_dir={
+ dir_a: {"structures/invoice.py": STRUCTURES_MODULE, "funcs.py": PIPE_FUNC_MODULE},
+ dir_b: {"models/receipt.py": ALIASED_IMPORT_MODULE},
+ }
+ )
+
+ message = str(exc_info.value)
+ assert "structures/invoice.py defines Invoice, LineItem; models/receipt.py defines Invoice" in message
+ assert STRUCTURES_REFUSAL_RULE in message
+ assert "MTHDS concepts with inline structures" in message
+ assert "from structures import <domain>__<Concept>" in message
+ assert "funcs.py" not in message
+ assert str(tmp_path) not in message
+
+ def test_the_same_file_in_two_directories_is_listed_once(self, tmp_path: Path) -> None:
+ with pytest.raises(MethodStructuresRefusedError) as exc_info:
+ ensure_no_structured_content_in_library_sources(
+ sources_by_dir={
+ tmp_path / "a": {"structures/invoice.py": STRUCTURES_MODULE},
+ tmp_path / "b": {"structures/invoice.py": STRUCTURES_MODULE},
+ }
+ )
+
+ assert str(exc_info.value).count("structures/invoice.py") == 1
+
+ def test_generated_structures_module_is_not_a_violation(self, tmp_path: Path) -> None:
+ """The `python-structures` projection, left as generated, copies MTHDS concepts: neither scan flags it."""
+ generated = _stamped(STRUCTURES_MODULE, target=CodegenTarget.PYTHON_STRUCTURES)
+ structures_dir = tmp_path / "structures"
+ structures_dir.mkdir()
+ (structures_dir / "structures.py").write_text(generated, encoding="utf-8")
+
+ assert is_generated_structures_module(source=generated)
+ assert scan_structured_content_classes(package_dir=tmp_path) == []
+ assert scan_structured_content_sources(sources={"structures/structures.py": generated}) == []
+ ensure_no_structured_content_in_library_sources(sources_by_dir={tmp_path: {"structures/structures.py": generated}})
+
+ @pytest.mark.parametrize(
+ "source",
+ [
+ _stamped(STRUCTURES_MODULE, target=CodegenTarget.PYTHON_STRUCTURES) + "\n\nclass Extra(StructuredContent):\n note: str\n",
+ _stamped(STRUCTURES_MODULE, target=CodegenTarget.PYTHON_PYDANTIC),
+ _stamped(STRUCTURES_MODULE, target=CodegenTarget.PYTHON_STRUCTURES).replace("# content_hash: ", "# content_hash: 0"),
+ STRUCTURES_MODULE,
+ ],
+ ids=["edited-below-the-stamp", "another-projection", "mismatched-hash", "unstamped"],
+ )
+ def test_only_an_unedited_structures_projection_is_exempt(self, source: str) -> None:
+ assert not is_generated_structures_module(source=source)
+ assert scan_structured_content_sources(sources={"structures/structures.py": source}) != []
diff --git a/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py b/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py
index 4e7cc1f6e..ef043e4e7 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_func/test_direct_executor_workdir.py
@@ -12,7 +12,7 @@ from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.interpreter_hub import get_library_manager, scoped_current_library, set_current_library
from pipelex.libraries.library_crate import LibraryCrate
from pipelex.pipe_operators.func import direct_pipe_func_executor
-from pipelex.pipe_operators.func.direct_pipe_func_executor import DirectPipeFuncExecutor
+from pipelex.pipe_operators.func.direct_pipe_func_executor import DirectPipeFuncExecutor, bundle_import_dirs
from pipelex.pipe_operators.func.pipe_func_execution_dtos import PipeFuncExecutionRequest
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.system.job_metadata import JobMetadata, RunMetadata
@@ -50,6 +50,19 @@ async def greet_it(working_memory: WorkingMemory) -> greet_demo__Greeting:
"""
+# A PipeFunc file, so the transported run imports it by path, which puts its resolved directory on sys.path.
+_SYMLINK_PROBE_FUNC = """\
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.system.registries.func_registry import pipe_func
+
+
+@pipe_func(name="symlink_probe_func")
+async def symlink_probe_func(working_memory: WorkingMemory) -> TextContent:
+ return TextContent(text="probe")
+"""
+
+
class TestDirectExecutorWorkdir:
@pytest.mark.asyncio
async def test_transported_run_cleans_up_workdir_on_failure(self, tmp_path: Path, mocker: MockerFixture):
@@ -102,6 +115,45 @@ class TestDirectExecutorWorkdir:
stale_entries = [entry for entry in sys.path if Path(entry).is_relative_to(workdir)]
assert stale_entries == []
+ @pytest.mark.asyncio
+ async def test_transported_run_restores_sys_path_under_a_symlinked_temp_dir(self, tmp_path: Path, mocker: MockerFixture):
+ """Importing a file puts its resolved directory on sys.path, so a temp dir reached through a symlink
+ (`/var` on macOS) must still leave no entry behind.
+ """
+ real_workdir = tmp_path / "real_workdir"
+ real_workdir.mkdir()
+ linked_workdir = tmp_path / "linked_workdir"
+ linked_workdir.symlink_to(real_workdir, target_is_directory=True)
+ mocker.patch("tempfile.mkdtemp", return_value=str(linked_workdir))
+
+ request = PipeFuncExecutionRequest(
+ crate=LibraryCrate(python_sources={"funcs/probe.py": _SYMLINK_PROBE_FUNC}),
+ working_memory_raw={},
+ pipe_code="missing_pipe",
+ function_name="missing_func",
+ job_metadata=JobMetadata(run_metadata=RunMetadata(storage_scope="test/scope", read_scope=None, user_id="user", pipeline_run_id="run")),
+ pipe_run_params=PipeRunParams(run_mode=PipeRunMode.DRY, pipe_stack_limit=10, batch_max_concurrency=None),
+ )
+
+ with scoped_current_library(library_id=direct_pipe_func_executor._TRANSPORTED_LIBRARY_ID): # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+ try:
+ with pytest.raises((PipelexError, ValidationError)):
+ await DirectPipeFuncExecutor().run_pipe_func_transported(request=request)
+ finally:
+ get_library_manager().teardown(library_id=direct_pipe_func_executor._TRANSPORTED_LIBRARY_ID) # ruff: ignore[private-member-access] # pyright: ignore[reportPrivateUsage]
+
+ stale_entries = [entry for entry in sys.path if Path(entry).is_relative_to(real_workdir) or Path(entry).is_relative_to(linked_workdir)]
+ assert stale_entries == []
+
+ def test_import_dirs_search_the_root_first_then_sorted_subdirs(self, tmp_path: Path):
+ """A module name two bundle directories share resolves to the same file in every process, root first."""
+ for relpath in ("zeta/structures.py", "alpha/helpers.py", "funcs.py", "alpha/deep/more.py", "notes/readme.txt"):
+ file_path = tmp_path / relpath
+ file_path.parent.mkdir(parents=True, exist_ok=True)
+ file_path.write_text("x = 1\n", encoding="utf-8")
+
+ assert bundle_import_dirs(workdir=tmp_path) == [tmp_path, tmp_path / "alpha", tmp_path / "alpha" / "deep", tmp_path / "zeta"]
+
@pytest.mark.asyncio
async def test_transported_run_generates_concept_structures(self):
"""A PipeFunc that imports its output concept structure runs even though the method ships no