Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Offer-factory-sees-requirements · L-261017-1dc9dc (#1455)

Louis Choquelb2fb7f58f02f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 269447111..d83a7421e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,7 @@
- **Validating a method's tree judges only its own pipes (Breaking)**: `validate_bundle(method_dir=...)`, and so `POST /v1/validate` on a method's files and `validate bundle <folder>`, dry-runs and reports only the bundles of the method's own tree; a nested package below it with its own `METHODS.toml`, the packages it depends on and the `-L` directories still load for its bundles to use, but their pipes are no longer judged with it.
- **`/v1/resolve`, `/v1/codegen` and `/v1/pipe-io` take the method tree (Breaking)**: the three routes take the method as `/v1/validate` and the run routes do, `files` being the MTHDS protocol's map of file entries with `dependencies`, `resources` and `read_scope` beside it, or a `method_ref` whose package's whole tree is loaded with the packages it ships, so a call into another package by its address resolves when that package was shipped, while one nothing shipped is the invalid verdict, never looked up among the runner's installed methods or fetched, and a `METHODS.toml` at the root of a sent tree names the default pipe of `/v1/pipe-io`, which reads it only for a request naming no pipe. A `files` list of `{content, source}` items is refused with a `422` `InvalidMethodFiles`, the tree's refusals and the custom-Python `403`s are those of `/v1/validate`, `include_files` echoes the method's `.mthds` bundles as a map from tree path to text, and the request models `MthdsFilesRequest`, `MthdsFileItem` and `MthdsPipeRequest` are removed. The routes load the method in a worker thread, so a slow load never holds up the server's other requests.
- **A bundle validator receives the read scope, plugin API version 8 (Breaking)**: `BundleValidatorProtocol.validate_bundles` takes `read_scope`, the scope the host named for the validation, `None` when it named none, so a validator dispatching the validation to another process hands the method's trees to its worker through that scope's blob store and the worker reads them by digest under it, as a run's worker reads under the run's read scope. The API server's `POST /v1/validate` hands its validator the request's `read_scope` on every arm, never the caller's own id, and `PLUGIN_API_VERSION` moves to 8, so a plugin declaring `targets_api = 7` is refused at discovery with `PluginApiVersionMismatchError`.
+- **A PipeFunc package-offer factory receives the method's requirements, under plugin API version 8 (Breaking)**: a factory registered with `PluginRegistrar.add_pipe_func_package_offer` is called with a `PipeFuncPackageOfferRequest`, the `pipe_func` config and the requirements of the one method tree being judged, where it received the config alone, so a sandbox mode building an environment for each method states the environment it builds for them, or answers a `PipeFuncPackageRefusal` when none can hold them, which refuses the method with a `MethodRequirementsNotMetError` carrying the plugin's reason. Each tree judged in one check gets the offer stated for its own requirements, and the factory stays synchronous, so a plugin whose answer takes time caches it by the requirements. The requirements arrive with their markers unevaluated, and the new `requirements_needed_in` (`pipelex.libraries.method_requirements`) picks out those whose markers hold where the mode would run the method, the only ones a refusal may rest on; `parse_requirements_text`, `RequirementLine` and `REQUIREMENTS_FILE_NAME` move to `pipelex.core.method_trees.requirements_file`.
### Fixed
diff --git a/api/tests/unit/test_method_requirements_refusal.py b/api/tests/unit/test_method_requirements_refusal.py
index 8ccd4ccac..f961c4dcc 100644
--- a/api/tests/unit/test_method_requirements_refusal.py
+++ b/api/tests/unit/test_method_requirements_refusal.py
@@ -23,10 +23,14 @@ from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistry
from pipelex.plugins.orchestrator_registry import OrchestratorRegistry
-from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOffer, PipeFuncPackageOfferRegistry
+from pipelex.plugins.pipe_func_package_offer_registry import (
+ PipeFuncPackageOffer,
+ PipeFuncPackageOfferRegistry,
+ PipeFuncPackageOfferRequest,
+ PipeFuncPackageRefusal,
+)
from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeRunOutput
from pipelex.system.caller_identity import CallerIdentity
-from pipelex.system.configuration.pipe_func_config import PipeFuncConfig
from pytest_mock import MockerFixture
from pipelex_api.api_config import ApiConfig
@@ -37,6 +41,8 @@ from tests.unit._constants import STUB_METHOD_ADDRESS
_PIPELINE_NS = "pipelex_api.routes.pipelex.pipeline"
_SANDBOX_ENVIRONMENT = "the Daytona sandbox snapshot 'pipelex-base-test'"
_ABSENT_PACKAGE = "pipelex-requirements-test-absent"
+_UNRESOLVABLE_REQUIREMENT = "pandas<2"
+_REFUSAL_REASON = "uv explains: Because pandas<2 conflicts with the base snapshot's pandas==2.2.3, the requirements are unsatisfiable."
_PLAIN_MTHDS = """\
domain = "plain_demo"
@@ -115,21 +121,23 @@ class _UnreachableValidator:
raise AssertionError(msg)
[email protected](name="sandbox_hosted_mode")
-def sandbox_hosted_mode_fixture(mocker: MockerFixture) -> Generator[None, None, None]:
- """Run PipeFunc steps in the neutral `sandbox` mode, whose plugin states a small package set."""
- offer = PipeFuncPackageOffer(
+def _stated_offer(request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
+ """A small package set, whatever the requirements, except for a method asking for `pandas<2`, which no environment holds."""
+ if any(line.text == _UNRESOLVABLE_REQUIREMENT for line in request.requirements):
+ return PipeFuncPackageRefusal(environment="the Daytona sandbox", reason=_REFUSAL_REASON)
+ return PipeFuncPackageOffer(
environment=_SANDBOX_ENVIRONMENT,
distributions={"pandas": "2.2.3", "openpyxl": "3.1.5"},
marker_environment={"python_version": "3.12", "sys_platform": "linux"},
)
- def stated_offer(_config: PipeFuncConfig) -> PipeFuncPackageOffer | None:
- return offer
[email protected](name="sandbox_hosted_mode")
+def sandbox_hosted_mode_fixture(mocker: MockerFixture) -> Generator[None, None, None]:
+ """Run PipeFunc steps in the neutral `sandbox` mode, whose plugin states a small package set or refuses `pandas<2`."""
mocker.patch(
"pipelex.libraries.package_environments.get_pipe_func_package_offer_registry",
- return_value=PipeFuncPackageOfferRegistry({"sandbox": stated_offer}),
+ return_value=PipeFuncPackageOfferRegistry({"sandbox": _stated_offer}),
)
pipe_func_config = get_config().interpreter.pipe_func
previous = pipe_func_config.execution_mode
@@ -171,6 +179,24 @@ class TestMethodRequirementsRefusal:
assert "rooted at" not in body["detail"]
assert orchestrator.dispatches == []
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_a_run_whose_requirements_the_deployment_refuses_carries_its_reason_before_dispatch(self, mocker: MockerFixture) -> None:
+ orchestrator = _RecordingOrchestrator()
+ client = _build_client(mocker, orchestrator=orchestrator, validator=_UnreachableValidator())
+
+ files = {"main.mthds": _PIPE_FUNC_MTHDS, "funcs/pipe_func.py": _PIPE_FUNC_PY, "requirements.txt": f"{_UNRESOLVABLE_REQUIREMENT}\n"}
+ response = client.post("/v1/start", json={"files": files, "inputs": {"data": "1,2,3"}})
+
+ assert response.status_code == 422, response.text
+ body = response.json()
+ assert body["error_type"] == "MethodRequirementsNotMetError"
+ # Caller-facing under STRICT disclosure: the deployment's own reason, and no directory of the runner.
+ assert (
+ body["detail"]
+ == f"The method needs Python packages that the Daytona sandbox cannot offer, as its requirements.txt declares: {_REFUSAL_REASON}"
+ )
+ assert orchestrator.dispatches == []
+
@pytest.mark.usefixtures("sandbox_hosted_mode")
def test_a_run_whose_requirements_the_sandbox_meets_dispatches(self, mocker: MockerFixture) -> None:
orchestrator = _RecordingOrchestrator()
diff --git a/docs/api-server/pipe-run.md b/docs/api-server/pipe-run.md
index 828b3390e..e6bfdf354 100644
--- a/docs/api-server/pipe-run.md
+++ b/docs/api-server/pipe-run.md
@@ -248,7 +248,7 @@ The receiver-side secret must be the same value. In typical deployments both sid
**No Python structure classes.** A structure class (a `StructuredContent` subclass) can only be used by importing its module into the runner's own process, which a sandbox-hosted deployment never does, so a tree whose Python declares one is refused with a `403` (`MethodStructuresRefusedError`) naming each file and class, before anything is loaded. The rule is the same however the method arrives: sent as a tree, or fetched by `method_ref`. Declare the types as MTHDS concepts with inline structures instead; a PipeFunc returns them by importing the classes the sandbox generates from those concepts (`from structures import <domain>__<Concept>`). The `structures` module that `pipelex build structures` writes is accepted as long as it is left as generated, and it is not sent to the sandbox, which generates its own.
-**Requirements are checked against the sandbox.** A `requirements.txt` at the root of the tree declares the packages its Python imports, one PEP 508 requirement per line (see [Method Trees](../building-methods/method-trees.md#python-packages-requirementstxt)). Nothing is installed for a run: the packages of the deployment's PipeFunc sandbox are the whole offer, and a method asking for a package the sandbox does not hold, or for a version it does not hold, is refused with a `422` (`MethodRequirementsNotMetError`) naming each requirement and the version the sandbox holds, before the method loads and before anything is dispatched. The refusal names the file by its tree path, `requirements.txt`, and no path on the server. A line the standard refuses, such as `-r base.txt`, a URL or a comment after a requirement, is a `422` (`MethodRequirementsFileError`) naming the line. A deployment that states no package set for its sandbox refuses every method declaring requirements with a `500` (`PipeFuncPackageSetUnstatedError`), a fault of its configuration, and so does a method of the server's own library directories whose requirements are not met (`HostLibraryRequirementsError`), whose message is withheld from the caller. A `method_ref` package is checked the same way, on a run as on [`/validate`](pipe-validate.md), where the refusal is an `is_valid: false` verdict located on the requirement's line.
+**Requirements are checked against the sandbox.** A `requirements.txt` at the root of the tree declares the packages its Python imports, one PEP 508 requirement per line (see [Method Trees](../building-methods/method-trees.md#python-packages-requirementstxt)). Nothing is installed into a running sandbox: the packages the deployment states its PipeFunc sandbox offers for the method's requirements are the whole offer, a fixed package set or an environment the deployment builds for those requirements, and a method asking for a package the sandbox does not offer, or for a version it does not offer, is refused with a `422` (`MethodRequirementsNotMetError`) naming each requirement and the version offered, before the method loads and before anything is dispatched. A deployment that can build no environment holding the requirements refuses them with the same `422`, carrying its own explanation. The refusal names the file by its tree path, `requirements.txt`, and no path on the server. A line the standard refuses, such as `-r base.txt`, a URL or a comment after a requirement, is a `422` (`MethodRequirementsFileError`) naming the line. A deployment that states no package set for its sandbox refuses every method declaring requirements with a `500` (`PipeFuncPackageSetUnstatedError`), a fault of its configuration, and so does a method of the server's own library directories whose requirements are not met (`HostLibraryRequirementsError`), whose message is withheld from the caller. A `method_ref` package is checked the same way, on a run as on [`/validate`](pipe-validate.md), where the refusal is an `is_valid: false` verdict located on the requirement's line.
### Sending files by hash
diff --git a/docs/api-server/pipe-validate.md b/docs/api-server/pipe-validate.md
index 98ca9a909..95c595591 100644
--- a/docs/api-server/pipe-validate.md
+++ b/docs/api-server/pipe-validate.md
@@ -19,7 +19,7 @@ Validate MTHDS content by parsing, loading, and dry-running pipes without execut
- `mthds_contents` (list[str], optional): MTHDS contents to validate (always an array, even for a single file). Exactly one of `mthds_contents` / `files` / `method_ref` must be provided
- `files` (object, optional): The method as a map of file entries, the MTHDS standard's method tree, each value the file's text, `{"base64": "…"}` or `{"sha256": "…", "size": n}`, read as on a run — see [Validating a method tree](#validating-a-method-tree). Exactly one of `mthds_contents` / `files` / `method_ref`
- `dependencies` (object, optional): **Pipelex-API extension.** The method packages the method calls by address that the runner cannot fetch, each a map of file entries keyed by its address, as on a [run](pipe-run.md#sending-the-method-packages-the-method-calls). Accompanies `files`
-- `method_ref` (string, optional): **Pipelex-API extension.** Validate a published method by address — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — resolved by this server through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the repository is fetched at the tag, the package located by manifest identity, and its tree read from the commit's git objects and validated as a tree sent in `files` is — see [Validating a method tree](#validating-a-method-tree). Exactly one of `mthds_contents` / `files` / `method_ref`; `mthds_sources` cannot accompany it. A package holding Python and a `requirements.txt` has its requirements checked on this server against the package set of the deployment's PipeFunc sandbox before anything is validated, and a requirement the sandbox does not meet, or a line the standard refuses, makes the verdict `is_valid: false` with one item per line, whose `source` is `requirements.txt` and whose `line` is the line's number
+- `method_ref` (string, optional): **Pipelex-API extension.** Validate a published method by address — `github.com/<owner>/<repo>` plus an optional package selector and `@<tag>` — resolved by this server through the same fetch path as a [`method_ref` run](pipe-run.md#running-a-method-by-address-method_ref): the repository is fetched at the tag, the package located by manifest identity, and its tree read from the commit's git objects and validated as a tree sent in `files` is — see [Validating a method tree](#validating-a-method-tree). Exactly one of `mthds_contents` / `files` / `method_ref`; `mthds_sources` cannot accompany it. A package holding Python and a `requirements.txt` has its requirements checked on this server against the packages the deployment's PipeFunc sandbox offers for them before anything is validated, and a requirement the sandbox does not meet, or a line the standard refuses, makes the verdict `is_valid: false` with one item per line, whose `source` is `requirements.txt` and whose `line` is the line's number; a deployment refusing the requirements with its own explanation, no environment it can build holding them, makes it `is_valid: false` with one item on `requirements.txt` and no line
- `read_scope` (string, optional): **Pipelex-API extension.** The scope the validation reads the method under, as a run's `read_scope`, which a multi-tenant host sends for the organization it serves; [a `jwt` caller may send only a scope under its own id](pipe-run.md#host-supplied-run-context-storage_scope-and-analytics_groups), else a `403 ReadScopeForbidden`. A hash entry of `files` resolves in that scope's blob store — see [Sending files by hash](pipe-run.md#sending-files-by-hash) — and the runner hands the scope to its validator, so a validation dispatched to a worker can hand it the method through that store — see [Where validation runs](#where-validation-runs). Omitted, a hash entry resolves in the caller's own id's store, but only a validation carrying it records the tree it read in the blob store, and the validator is handed no scope, so a dry run without one leaves nothing stored
- `allow_signatures` (boolean, optional, default `false`): controls only the **sweep mechanics** for `PipeSignature` placeholders — whether signature pipes are mock-run during the dry-run sweep and therefore listed in `validated_pipes`. It does **not** change the verdict: an unimplemented signature is never a rejection, it is a *runnability fact* reported via `pending_signatures` + `is_runnable` in both modes
- `mthds_sources` (list[str] | null, optional): per-file sources, parallel to `mthds_contents` — see [Sourcing submitted files](#sourcing-submitted-files). When present it must match `mthds_contents` in length (a mismatch is a 422 request error), and it accompanies `mthds_contents` alone, since a tree's files are named by their tree paths
diff --git a/docs/building-methods/method-trees.md b/docs/building-methods/method-trees.md
index b525eeacf..508039400 100644
--- a/docs/building-methods/method-trees.md
+++ b/docs/building-methods/method-trees.md
@@ -56,7 +56,7 @@ Pipelex refuses, naming each such line, a requirement naming a URL (`name @ http
When a tree holding Python and a root `requirements.txt` loads, Pipelex checks every requirement against the Python environment its PipeFunc steps run in, before importing any of the tree's Python and before building any of its pipes, and refuses the method when one is not met, naming each requirement and why. A requirement whose marker is false in that environment is not needed there and is not checked. A tree with no Python is not checked, since nothing in it could import a package. A package your method calls by its address is checked the same way when it loads, its refusal naming the package by its address. Pipelex never installs anything.
- **Locally**, a PipeFunc runs in the interpreter Pipelex runs in, so the requirements are checked against the packages installed there, extras included. The refusal names the method's folder and gives the command that installs what is missing into that interpreter, such as `uv pip install --python /path/to/python 'pandas>=2.2,<3'`. When Pipelex was installed as a uv tool, add the packages to the tool instead, which keeps them across upgrades: `uv tool install pipelex --with 'pandas>=2.2,<3'`.
-- **On a sandbox-hosted deployment**, such as the hosted API, a PipeFunc runs in the deployment's sandbox, whose packages are the whole offer: nothing is installed for a method. The requirements are checked against the package set the deployment states for its sandbox, and a method asking for a package it does not hold, or for a version it does not hold, is refused when it loads and when it is validated, naming each requirement. A requirement with extras cannot be checked there and is refused, so list the packages the extra brings directly.
+- **On a sandbox-hosted deployment**, such as the hosted API, a PipeFunc runs in the deployment's sandbox, whose packages are the whole offer: nothing is installed into a running sandbox. The requirements are checked against the packages the deployment states its sandbox offers for them, a fixed package set, or an environment the deployment builds for your method's requirements, and a method asking for a package it does not offer, or for a version it does not offer, is refused when it loads and when it is validated, naming each requirement. A deployment that can build no environment holding your requirements, such as one whose packages would conflict with a version its sandbox pins, refuses them with its own explanation. A requirement with extras cannot be checked there and is refused, so list the packages the extra brings directly.
Python declaring a structure class is refused on a sandbox-hosted deployment whatever its requirements: see [Python classes](concepts/python-classes.md).
diff --git a/docs/under-the-hood/pipe-func-requirements.md b/docs/under-the-hood/pipe-func-requirements.md
index 991036035..db7f78382 100644
--- a/docs/under-the-hood/pipe-func-requirements.md
+++ b/docs/under-the-hood/pipe-func-requirements.md
@@ -1,6 +1,6 @@
---
title: "PipeFunc Requirements"
-description: "How Pipelex checks a method's requirements.txt against the Python environment its PipeFunc steps run in, where the check runs on a sandbox-hosted deployment, and the plugin seam through which a deployment states its sandbox's packages."
+description: "How Pipelex checks a method's requirements.txt against the Python environment its PipeFunc steps run in, where the check runs on a sandbox-hosted deployment, and the plugin seam through which a deployment states the packages its sandbox offers for a method's requirements."
---
# PipeFunc Requirements
@@ -21,12 +21,14 @@ A requirement whose marker is false in the environment is not needed there and i
## The environment the check reads
-`resolve_pipe_func_package_environment` in `pipelex/libraries/package_environments.py` picks the environment from `interpreter.pipe_func.execution_mode`:
+`resolve_pipe_func_package_environment` in `pipelex/libraries/package_environments.py` picks the environment from `interpreter.pipe_func.execution_mode`, for one tree's requirements at a time:
| Execution mode | Environment | What it reads |
| --- | --- | --- |
-| `direct` | `InterpreterPackageEnvironment` | The distributions installed in the interpreter Pipelex runs in, through `importlib.metadata`, and this interpreter's own marker environment. Extras are followed through each distribution's `Requires-Dist`. |
-| any other mode | `OfferedPackageEnvironment` | The package set the plugin providing the mode states for its sandbox, and nothing else. A marker variable the offer does not state cannot be judged, and a requirement with extras cannot be followed, so both are refused rather than judged against this process's own values. |
+| `direct` | `InterpreterPackageEnvironment` | The distributions installed in the interpreter Pipelex runs in, through `importlib.metadata`, and this interpreter's own marker environment, whatever the requirements. Extras are followed through each distribution's `Requires-Dist`. |
+| any other mode | `OfferedPackageEnvironment` | The package set the plugin providing the mode states its sandbox offers for the tree's requirements, and nothing else: a fixed package set, or an environment the plugin builds for those requirements. A marker variable the offer does not state cannot be judged, and a requirement with extras cannot be followed, so both are refused rather than judged against this process's own values. The plugin may instead refuse the requirements with its own reason, when no environment it can build holds them, and the method is then refused with that reason. |
+
+Each tree declaring requirements is judged against the environment resolved for its own requirements, so when one check judges several trees, such as the library directories of one load, each gets the offer stated for it and never another tree's.
In the `direct` mode, a method's Python is imported into the process that loads it, when its steps first bind their functions, and that is the process the check judges. A distributed orchestrator, such as our Temporal plugin, never carries a method's Python to its worker: the library crate names the step's method tree by its digest, and in a sandbox mode the call's files are read through the method file store by whoever runs it, for the sandbox to import them, never this process. A worker in the `direct` mode therefore runs only the functions its own deployment installed, whose packages are that deployment's to install, and the API server refuses a caller's Python outright unless it is sandbox-hosted (`403 CustomCodeRequiresSandbox`). So no worker needs to repeat the check.
@@ -53,21 +55,26 @@ A server loads its own library directories (`PIPELEXPATH`, the instance defaults
## The seam: stating the sandbox's packages
-Pipelex cannot read a sandbox it never runs in, and it must not depend on the deployment that provides one, so the plugin providing an execution mode's executor states that mode's package set through its registrar, keyed by the same mode token:
+Pipelex cannot read a sandbox it never runs in, and it must not depend on the deployment that provides one, so the plugin providing an execution mode's executor states that mode's packages through its registrar, keyed by the same mode token. The plugin is asked for one method's requirements at a time, so a mode whose sandbox is not one fixed package set, such as one building an environment for each method, states the environment it would run that method in:
```python
-from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOffer
-from pipelex.system.configuration.pipe_func_config import PipeFuncConfig
+from pipelex.libraries.method_requirements import requirements_needed_in
+from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOffer, PipeFuncPackageOfferRequest, PipeFuncPackageRefusal
-def _sandbox_package_offer(pipe_func_config: PipeFuncConfig) -> PipeFuncPackageOffer | None:
- package_set = read_my_sandbox_package_set() # the plugin's own reading of its image's lock, cached
- if package_set is None:
+def _sandbox_package_offer(request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
+ base = read_my_sandbox_package_set() # the plugin's own reading of its image's lock, cached
+ if base is None:
return None
+ # Only what the sandbox needs: a requirement whose marker is false in its interpreter is neither built nor refused.
+ needed = requirements_needed_in(requirements=request.requirements, marker_environment=base.marker_environment)
+ environment = environment_for(needed) # resolved over the base, cached by the requirements' texts
+ if environment.unresolvable:
+ return PipeFuncPackageRefusal(environment="the Example sandbox", reason=environment.explanation)
return PipeFuncPackageOffer(
- environment=f"the Example sandbox image '{package_set.image}'",
- distributions={package.name: package.version for package in package_set.packages},
- marker_environment=package_set.marker_environment,
+ environment=f"the Example sandbox image '{environment.image}'",
+ distributions={package.name: package.version for package in environment.packages},
+ marker_environment=base.marker_environment,
)
@@ -77,8 +84,14 @@ class ExampleSandboxPlugin:
registrar.add_pipe_func_package_offer(mode="example", factory=_sandbox_package_offer)
```
-- **`PipeFuncPackageOffer`** (`pipelex/plugins/pipe_func_package_offer_registry.py`) is closed, frozen plain data: `environment`, the place as a refusal names it to the method's author; `distributions`, each distribution's name with its exact installed version, the names compared normalized; and `marker_environment`, the PEP 508 marker values of the sandbox's interpreter. A variable left out of `marker_environment`, such as `python_full_version` when the image pins only its minor version, makes every requirement whose marker reads it refused.
-- **The factory** receives the `PipeFuncConfig` and is called each time a method declaring requirements loads, never at registration, so it may read the plugin's own configuration and should cache what it computes. It answers `None` when it cannot state the offer.
+A mode with one fixed package set ignores the requirements and answers its package set whatever they are.
+
+- **The request**, `PipeFuncPackageOfferRequest` (`pipelex/plugins/pipe_func_package_offer_registry.py`), carries the `PipeFuncConfig` and `requirements`, every requirement the method's `requirements.txt` declares, in the file's order, each a `RequirementLine` (`pipelex/core/method_trees/requirements_file.py`: its line number, its text as written and the parsed PEP 508 requirement). A plugin judging them against a package set of its own uses Pipelex's judgment, `unmet_requirements` with an `OfferedPackageEnvironment`, so it sets aside exactly what the check would find met.
+- **The markers are the factory's to evaluate.** The requirements arrive with their markers unevaluated, since Pipelex cannot evaluate a marker before it knows the environment the mode would run the method in. A requirement whose marker is false in that environment is not needed there, and the check skips it, so the factory evaluates the markers against the marker environment of the place it would offer, builds nothing for a requirement the place does not need, and never refuses for one: a refusal rests only on requirements whose markers hold there. `requirements_needed_in` (`pipelex/libraries/method_requirements.py`) picks them out as the check would, keeping a requirement whose marker reads a variable the marker environment does not state, or cannot be evaluated, which the check refuses rather than skips.
+- **`PipeFuncPackageOffer`** is closed, frozen plain data: `environment`, the place as a refusal names it to the method's author; `distributions`, each distribution's name with its exact installed version, the names compared normalized; and `marker_environment`, the PEP 508 marker values of the sandbox's interpreter. A variable left out of `marker_environment`, such as `python_full_version` when the image pins only its minor version, makes every requirement whose marker reads it refused. Pipelex judges every requirement against the offer, one built for them included, so an offer that does not meet them refuses the method as a fixed package set would, naming each requirement.
+- **`PipeFuncPackageRefusal`** is the answer when no environment of the mode can hold the requirements the place needs, such as a version conflicting with a package the sandbox's base pins, or a package with no wheel for the sandbox: `environment`, the place as the refusal names it, and `reason`, the plugin's own words, such as its resolver's explanation. The method is refused with `MethodRequirementsNotMetError`, an `input` fault of the method's whose message, "The method needs Python packages that the Example sandbox cannot offer, as its requirements.txt declares: …", carries the reason, with one item on the file and no line, since the reason is about the requirements together. The reason reaches the method's author, a server's caller included, so it names no path on the host.
+- **The factory is synchronous, and caching by the requirements is the plugin's job.** The check runs inside the library load, which is synchronous and runs on whatever thread its caller runs it on: a worker thread for the API server's check of a served method, and the thread of a running event loop for a load inside an asynchronous run or validation. An asynchronous factory could not be awaited there without blocking that same thread, so it would buy nothing short of making every load asynchronous. Pipelex calls the factory for each tree declaring requirements each time a check judges it, never at registration, and keeps none of its answers, so a load that judges a tree at more than one of its steps asks again. A factory whose answer takes time, such as one resolving the requirements with `uv`, which takes seconds the first time, bounds the work with a timeout of its own and caches its answers by the requirements' texts, so only a process's first check of a requirement set pays for it: on the API server, the check of a served method runs first, off the event loop, and the load after it reads the cache. A factory may be called on a thread running an event loop, so it never calls `asyncio.run`.
+- **A fault of the plugin's own is not a refusal.** A resolver the plugin cannot reach, or one outlasting its timeout, says nothing about the method: the factory raises it as an error of the plugin's, which Pipelex lets through, rather than answering a `PipeFuncPackageRefusal`, which would blame the method's author.
- **It fails closed.** A mode with no registered offer, or whose factory answers `None`, meets no requirement: a method declaring one is refused with `PipeFuncPackageSetUnstatedError`, a `config`-domain fault of the deployment whose message is not caller-facing, rather than loading and failing inside the sandbox at its first import. Its message says which of the two happened: no plugin registers an offer for the mode, or the registered one could not state the packages, whose remedy is in the plugin's configuration. A method declaring none still loads.
- **It is interpreter-layer**, like `add_pipe_func_executor`: a plugin published under `pipelex.plugins.kernel` that calls it is refused with `PluginLayerViolationError`. A second plugin stating the same mode is refused with `DuplicatePipeFuncPackageOfferError`, naming both.
@@ -86,14 +99,14 @@ class ExampleSandboxPlugin:
### What our Daytona plugin supplies
-For the hosted API, the Daytona sandbox plugin registers the offer for its `daytona` mode from the package set of the Daytona sandbox snapshot it boots, the lock its snapshot image is built from: the snapshot's distributions and versions, and the marker environment of its base image. The runner, where the check runs, must resolve the same snapshot name the worker's sandboxes boot from, since the worker is the process that creates them; a runner that cannot resolve it answers `None` and refuses every method declaring requirements.
+For the hosted API, the Daytona sandbox plugin states the package set of the Daytona sandbox snapshot it boots, the lock its snapshot image is built from: the snapshot's distributions and versions, and the marker environment of its base image. The runner, where the check runs, must resolve the same snapshot name the worker's sandboxes boot from, since the worker is the process that creates them; a runner that cannot resolve it answers `None` and refuses every method declaring requirements. A method needing packages the base snapshot lacks runs in an environment the plugin builds for its requirements when a PipeFunc call provisions its box, and since the factory receives the requirements, the plugin can state that environment when the method loads, or refuse the requirements with its resolver's explanation.
## Errors
| Error | Domain | When |
| --- | --- | --- |
| [`MethodRequirementsFileError`](../errors/method-requirements-file-error.md) | `input` | The file holds a line the standard refuses, naming each, or is not UTF-8 text. |
-| [`MethodRequirementsNotMetError`](../errors/method-requirements-not-met-error.md) | `input` | The environment does not meet a requirement, naming each and why. |
+| [`MethodRequirementsNotMetError`](../errors/method-requirements-not-met-error.md) | `input` | The environment does not meet a requirement, naming each and why, or the deployment refuses the requirements with its own reason, no environment it can build holding them. |
| [`HostLibraryRequirementsError`](../errors/host-library-requirements-error.md) | `config` | A method of a host's own library directories, loaded beside a caller's bundles, is refused for its requirements. |
| [`PipeFuncPackageSetUnstatedError`](../errors/pipe-func-package-set-unstated-error.md) | `config` | A sandbox-hosted deployment states no package set for its execution mode. |
diff --git a/pipelex/core/method_trees/requirements_file.py b/pipelex/core/method_trees/requirements_file.py
new file mode 100644
index 000000000..91e05e249
--- /dev/null
+++ b/pipelex/core/method_trees/requirements_file.py
@@ -0,0 +1,94 @@
+"""Reading a method's `requirements.txt` as the MTHDS standard does: its requirements, and the lines it refuses.
+
+The MTHDS standard (`mthds/docs/spec/method-tree.md`, "Python Dependencies: `requirements.txt`") lets a method declare
+the Python packages its own Python needs in a `requirements.txt` at the root of its method tree: each line is blank, a
+comment whose first character other than a space or a tab is `#`, or exactly one PEP 508 requirement, markers
+included. A requirement naming a URL, an option of any kind (a line starting with `-`), a comment after a requirement
+and a line continued with a backslash are refused, naming the line. Only the root's file declares anything.
+
+This module is the file's grammar alone, with no notion of where a method runs: checking the requirements against the
+environment a method's PipeFunc steps run in is `pipelex/libraries/method_requirements.py`, and a plugin stating the
+packages of its sandbox receives the requirements as read here (`pipelex/plugins/pipe_func_package_offer_registry.py`).
+"""
+
+import re
+from typing import NamedTuple
+
+from packaging.requirements import InvalidRequirement, Requirement
+
+REQUIREMENTS_FILE_NAME = "requirements.txt"
+
+_UTF8_BOM = ""
+
+# A comment inside a line starts at a `#` that opens the line or follows whitespace, as pip reads one, so the fragment of
+# a URL, such as `#sha256=…`, is not one.
+_INLINE_COMMENT = re.compile(r"(^|\s+)#.*$")
+
+
+class RequirementLine(NamedTuple):
+ """A requirement of a `requirements.txt`, with its 1-based line number and its text as written, stripped."""
+
+ line_number: int
+ text: str
+ requirement: Requirement
+
+
+class RefusedRequirementsLine(NamedTuple):
+ """A line of a `requirements.txt` the standard refuses, with why."""
+
+ line_number: int
+ text: str
+ reason: str
+
+
+class ParsedRequirements(NamedTuple):
+ """What a `requirements.txt` declares: its requirements, and the lines it holds that the standard refuses."""
+
+ requirements: list[RequirementLine]
+ refused_lines: list[RefusedRequirementsLine]
+
+
+def _refusal_reason(*, stripped: str) -> tuple[Requirement | None, str | None]:
+ """The requirement a non-blank, non-comment line declares, or why the standard refuses the line."""
+ if stripped.endswith("\\"):
+ return None, "a line continued with a backslash is refused; write each requirement on a line of its own"
+ if stripped.startswith("-"):
+ return None, "an option line, such as -r, -e, -c or --index-url, is refused; write each package as a PEP 508 requirement"
+ # The requirement is judged without the comment after it, so a comment that mentions a URL is not read as one.
+ remainder = _INLINE_COMMENT.sub("", stripped).rstrip()
+ try:
+ requirement = Requirement(remainder)
+ except InvalidRequirement as exc:
+ if "://" in remainder:
+ return None, "a requirement naming a URL is refused; name the package and the versions it accepts instead"
+ parser_message = str(exc).splitlines()[0] if str(exc) else "it does not parse"
+ return None, f"the line is not a PEP 508 requirement: {parser_message}"
+ if requirement.url is not None:
+ return None, "a requirement naming a URL is refused; name the package and the versions it accepts instead"
+ if remainder != stripped:
+ return None, "a comment after a requirement is refused; write the comment on a line of its own"
+ return requirement, None
+
+
+def parse_requirements_text(*, text: str) -> ParsedRequirements:
+ """Read the lines of a `requirements.txt` as the MTHDS standard does, keeping every refused line rather than the first.
+
+ A leading byte order mark is skipped, and a line may end with a carriage return before its line feed.
+ """
+ requirements: list[RequirementLine] = []
+ refused_lines: list[RefusedRequirementsLine] = []
+ text = text.removeprefix(_UTF8_BOM)
+ for line_index, raw_line in enumerate(text.split("\n")):
+ line_number = line_index + 1
+ line = raw_line.removesuffix("\r")
+ if not line.strip():
+ continue
+ if line.lstrip(" \t").startswith("#"):
+ continue
+ stripped = line.strip()
+ requirement, reason = _refusal_reason(stripped=stripped)
+ if requirement is not None:
+ requirements.append(RequirementLine(line_number=line_number, text=stripped, requirement=requirement))
+ elif reason is not None:
+ refused_lines.append(RefusedRequirementsLine(line_number=line_number, text=stripped, reason=reason))
+ return ParsedRequirements(requirements=requirements, refused_lines=refused_lines)
diff --git a/pipelex/libraries/exceptions.py b/pipelex/libraries/exceptions.py
index edd040b45..cd026b6c2 100644
--- a/pipelex/libraries/exceptions.py
+++ b/pipelex/libraries/exceptions.py
@@ -153,8 +153,9 @@ class MethodRequirementsNotMetError(MethodRequirementsError):
"""Raised when the Python environment a method's PipeFunc steps run in does not satisfy its `requirements.txt`.
Locally that environment is the interpreter Pipelex runs in, which Pipelex never installs anything into; on a
- sandbox-hosted deployment it is the package set the deployment states for its PipeFunc sandbox, which is the whole
- offer. The message names each requirement not met and why.
+ sandbox-hosted deployment it is the package set the deployment states its PipeFunc sandbox offers for the method's
+ requirements, which is the whole offer. The message names each requirement not met and why, or, when the deployment
+ refuses the requirements because no environment it can build holds them, carries the deployment's own reason.
"""
diff --git a/pipelex/libraries/method_requirements.py b/pipelex/libraries/method_requirements.py
index 6131b38b7..cb4d6426b 100644
--- a/pipelex/libraries/method_requirements.py
+++ b/pipelex/libraries/method_requirements.py
@@ -1,10 +1,8 @@
"""Checking a method's `requirements.txt` against the Python environment its PipeFunc steps run in, when the method loads.
The MTHDS standard (`mthds/docs/spec/method-tree.md`, "Python Dependencies: `requirements.txt`") lets a method declare
-the Python packages its own Python needs in a `requirements.txt` at the root of its method tree: each line is blank, a
-comment whose first character other than a space or a tab is `#`, or exactly one PEP 508 requirement, markers
-included. A requirement naming a URL, an option of any kind (a line starting with `-`), a comment after a requirement
-and a line continued with a backslash are refused, naming the line. Only the root's file declares anything.
+the Python packages its own Python needs in a `requirements.txt` at the root of its method tree, whose lines are read
+as `pipelex/core/method_trees/requirements_file.py` reads them: a line the standard refuses is named with why.
Pipelex refuses a method whose requirements are not met when it loads, naming each package it cannot satisfy, and
never installs anything:
@@ -13,7 +11,10 @@ never installs anything:
with no Python has nothing that could import a package. The check runs before any of the tree's Python is imported
or captured, and before any of its pipes is built.
- The environment is the one the PipeFunc steps run in (`pipelex/libraries/package_environments.py`): this interpreter
- in the `direct` execution mode, and in a sandbox-hosted mode the package set the deployment states for its sandbox.
+ in the `direct` execution mode, and in a sandbox-hosted mode the packages the deployment states its sandbox offers
+ for the tree's requirements, asked for each tree, so trees judged together each get the offer stated for their own.
+ A deployment that cannot hold a tree's requirements in any environment refuses them with its own reason, which the
+ method's refusal carries.
- A requirement whose marker is false in that environment is not needed there and is not checked. One is met when the
environment holds a distribution of its name at a version its specifier contains, pre-releases included, and, for
each extra it names, when what the extra needs is met in turn.
@@ -26,7 +27,7 @@ directory of its own names no path on the host at all (`ensure_served_method_req
"""
import re
-from collections.abc import Iterable, Sequence
+from collections.abc import Iterable, Mapping, Sequence
from pathlib import Path
from typing import NamedTuple
@@ -37,15 +38,19 @@ from packaging.utils import canonicalize_name
from packaging.version import InvalidVersion, Version
from pipelex.core.exceptions import PipelexBundleBlueprintValidationErrorData
+from pipelex.core.method_trees.requirements_file import (
+ REQUIREMENTS_FILE_NAME,
+ RefusedRequirementsLine,
+ RequirementLine,
+ parse_requirements_text,
+)
from pipelex.libraries.exceptions import MethodRequirementsFileError, MethodRequirementsNotMetError
from pipelex.libraries.method_trees import list_library_dir_trees, listing_identity, reporting_host_library_tree_faults
from pipelex.libraries.package_environments import PythonPackageEnvironment, resolve_pipe_func_package_environment
+from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageRefusal
-REQUIREMENTS_FILE_NAME = "requirements.txt"
PYTHON_SUFFIX = ".py"
-_UTF8_BOM = "\ufeff"
-
# The variables a PEP 508 marker can read. `extra` is not the environment's to state: a line of a requirements file
# names no extra of its own, so it reads as empty, as pip reads it.
_MARKER_VARIABLES = frozenset(
@@ -66,35 +71,9 @@ _MARKER_VARIABLES = frozenset(
}
)
_QUOTED_MARKER_VALUE = re.compile(r"\"[^\"]*\"|'[^']*'")
-# A comment inside a line starts at a `#` that opens the line or follows whitespace, as pip reads one, so the fragment of
-# a URL, such as `#sha256=…`, is not one.
-_INLINE_COMMENT = re.compile(r"(^|\s+)#.*$")
_MARKER_WORD = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")
-class RequirementLine(NamedTuple):
- """A requirement of a `requirements.txt`, with its 1-based line number and its text as written, stripped."""
-
- line_number: int
- text: str
- requirement: Requirement
-
-
-class RefusedRequirementsLine(NamedTuple):
- """A line of a `requirements.txt` the standard refuses, with why."""
-
- line_number: int
- text: str
- reason: str
-
-
-class ParsedRequirements(NamedTuple):
- """What a `requirements.txt` declares: its requirements, and the lines it holds that the standard refuses."""
-
- requirements: list[RequirementLine]
- refused_lines: list[RefusedRequirementsLine]
-
-
class UnmetRequirement(NamedTuple):
"""A requirement the environment does not meet, with why."""
@@ -102,60 +81,14 @@ class UnmetRequirement(NamedTuple):
reason: str
-def _refusal_reason(*, stripped: str) -> tuple[Requirement | None, str | None]:
- """The requirement a non-blank, non-comment line declares, or why the standard refuses the line."""
- if stripped.endswith("\\"):
- return None, "a line continued with a backslash is refused; write each requirement on a line of its own"
- if stripped.startswith("-"):
- return None, "an option line, such as -r, -e, -c or --index-url, is refused; write each package as a PEP 508 requirement"
- # The requirement is judged without the comment after it, so a comment that mentions a URL is not read as one.
- remainder = _INLINE_COMMENT.sub("", stripped).rstrip()
- try:
- requirement = Requirement(remainder)
- except InvalidRequirement as exc:
- if "://" in remainder:
- return None, "a requirement naming a URL is refused; name the package and the versions it accepts instead"
- parser_message = str(exc).splitlines()[0] if str(exc) else "it does not parse"
- return None, f"the line is not a PEP 508 requirement: {parser_message}"
- if requirement.url is not None:
- return None, "a requirement naming a URL is refused; name the package and the versions it accepts instead"
- if remainder != stripped:
- return None, "a comment after a requirement is refused; write the comment on a line of its own"
- return requirement, None
-
-
-def parse_requirements_text(*, text: str) -> ParsedRequirements:
- """Read the lines of a `requirements.txt` as the MTHDS standard does, keeping every refused line rather than the first.
-
- A leading byte order mark is skipped, and a line may end with a carriage return before its line feed.
- """
- requirements: list[RequirementLine] = []
- refused_lines: list[RefusedRequirementsLine] = []
- text = text.removeprefix(_UTF8_BOM)
- for line_index, raw_line in enumerate(text.split("\n")):
- line_number = line_index + 1
- line = raw_line.removesuffix("\r")
- if not line.strip():
- continue
- if line.lstrip(" \t").startswith("#"):
- continue
- stripped = line.strip()
- requirement, reason = _refusal_reason(stripped=stripped)
- if requirement is not None:
- requirements.append(RequirementLine(line_number=line_number, text=stripped, requirement=requirement))
- elif reason is not None:
- refused_lines.append(RefusedRequirementsLine(line_number=line_number, text=stripped, reason=reason))
- return ParsedRequirements(requirements=requirements, refused_lines=refused_lines)
-
-
def marker_variables_of(*, marker: Marker) -> set[str]:
"""The PEP 508 variables a marker reads, its quoted values set aside so a value spelled like a variable is never one."""
unquoted = _QUOTED_MARKER_VALUE.sub(" ", str(marker))
return {word for word in _MARKER_WORD.findall(unquoted) if word in _MARKER_VARIABLES or word == "extra"}
-def _evaluate_marker(*, marker: Marker, environment: PythonPackageEnvironment, extra: str) -> bool:
- """A marker's value in the environment, reading `extra` as given.
+def _evaluate_marker(*, marker: Marker, marker_environment: Mapping[str, str], extra: str) -> bool:
+ """A marker's value in the marker environment, reading `extra` as given.
Raises:
UndefinedComparison: The marker compares values that are not versions.
@@ -163,7 +96,7 @@ def _evaluate_marker(*, marker: Marker, environment: PythonPackageEnvironment, e
InvalidVersion: The marker compares, as versions, a value that is not one; releases of `packaging` before 22
raise it where later ones compare the values as strings.
"""
- return marker.evaluate(environment={**environment.marker_environment, "extra": extra})
+ return marker.evaluate(environment={**marker_environment, "extra": extra})
def _unmet_reason(
@@ -220,8 +153,8 @@ def _unmet_extra_reason(
if dependency.marker is None:
continue
try:
- needed_by_extra = _evaluate_marker(marker=dependency.marker, environment=environment, extra=extra)
- needed_anyway = _evaluate_marker(marker=dependency.marker, environment=environment, extra="")
+ needed_by_extra = _evaluate_marker(marker=dependency.marker, marker_environment=environment.marker_environment, extra=extra)
+ needed_anyway = _evaluate_marker(marker=dependency.marker, marker_environment=environment.marker_environment, extra="")
except (UndefinedComparison, UndefinedEnvironmentName, InvalidVersion):
continue
if not needed_by_extra or needed_anyway:
@@ -242,6 +175,33 @@ def unmet_requirements(*, requirements: Iterable[RequirementLine], environment:
return unmet
+def requirements_needed_in(*, requirements: Iterable[RequirementLine], marker_environment: Mapping[str, str]) -> list[RequirementLine]:
+ """The requirements a place with this PEP 508 marker environment needs, in the order given: all but those whose marker is false there.
+
+ The check skips a requirement whose marker is false in the environment it judges against, as not needed there, so
+ a package-offer factory refusing a method's requirements rests its refusal on these alone, read in the marker
+ environment of the place it would run the method in, never on a requirement the place does not need. A requirement
+ whose marker reads a variable the marker environment does not state, or cannot be evaluated, is kept: the check
+ refuses it there rather than skipping it.
+ """
+ needed: list[RequirementLine] = []
+ for requirement_line in requirements:
+ marker = requirement_line.requirement.marker
+ if marker is None or not _is_marker_false(marker=marker, marker_environment=marker_environment):
+ needed.append(requirement_line)
+ return needed
+
+
+def _is_marker_false(*, marker: Marker, marker_environment: Mapping[str, str]) -> bool:
+ """Whether a marker is judged false in the marker environment, as the check judges it: one it cannot judge is not."""
+ if marker_variables_of(marker=marker) - set(marker_environment) - {"extra"}:
+ return False
+ try:
+ return not _evaluate_marker(marker=marker, marker_environment=marker_environment, extra="")
+ except (UndefinedComparison, UndefinedEnvironmentName, InvalidVersion):
+ return False
+
+
def _requirement_line_unmet_reason(*, requirement_line: RequirementLine, environment: PythonPackageEnvironment) -> str | None:
marker = requirement_line.requirement.marker
if marker is not None:
@@ -249,7 +209,7 @@ def _requirement_line_unmet_reason(*, requirement_line: RequirementLine, environ
if unstated:
return f"its marker reads {', '.join(unstated)}, which {environment.description} does not state"
try:
- if not _evaluate_marker(marker=marker, environment=environment, extra=""):
+ if not _evaluate_marker(marker=marker, marker_environment=environment.marker_environment, extra=""):
return None
except (UndefinedComparison, UndefinedEnvironmentName, InvalidVersion) as exc:
return f"its marker cannot be evaluated: {exc}"
@@ -350,19 +310,37 @@ def _raise_unmet(*, judged: JudgedMethod, unmet: list[UnmetRequirement], environ
raise MethodRequirementsNotMetError(msg, items=items)
+def _refused_by_deployment_error(*, judged: JudgedMethod, refusal: PipeFuncPackageRefusal) -> MethodRequirementsNotMetError:
+ """The refusal of a method whose requirements the deployment cannot hold in any environment, carrying its reason.
+
+ The deployment's reason is about the requirements together, such as a conflict its resolver found among them, so the
+ one item it rides on names the file and no line.
+ """
+ msg = (
+ f"{judged.method_name} needs Python packages that {refusal.environment} cannot offer, "
+ f"as its {REQUIREMENTS_FILE_NAME} declares: {refusal.reason}"
+ )
+ item = PipelexBundleBlueprintValidationErrorData(
+ source=judged.requirements_source,
+ message=f"The requirements are refused by {refusal.environment}: {refusal.reason}",
+ )
+ return MethodRequirementsNotMetError(msg, items=[item])
+
+
def _ensure_judged_methods_met(*, judged_methods: Iterable[JudgedMethod], names_host_paths: bool) -> None:
"""Refuse the first of the methods whose `requirements.txt` the standard refuses or its PipeFunc environment does not meet.
- Each tree is judged once however often it is given, in the order given. The environment is resolved only once a tree
- declares a requirement to judge, so methods declaring none never read it.
+ Each tree is judged once however often it is given, in the order given, against the environment stated for its own
+ requirements: in a sandbox-hosted mode, the plugin providing the mode is asked for each tree declaring requirements,
+ so trees judged together never share an environment stated for another's. Methods declaring none never read it.
Raises:
MethodRequirementsFileError: A tree's `requirements.txt` holds a line the standard refuses, or cannot be read.
- MethodRequirementsNotMetError: The environment does not meet a tree's requirements.
+ MethodRequirementsNotMetError: The environment does not meet a tree's requirements, or the deployment refuses
+ them with its own reason, no environment of its holding them.
PipeFuncPackageSetUnstatedError: PipeFunc steps run out of this process and the deployment states no package
set for them.
"""
- environment: PythonPackageEnvironment | None = None
judged_trees: set[tuple[str, str, bool]] = set()
for judged in judged_methods:
tree_identity = listing_identity(tree=judged.tree)
@@ -376,8 +354,9 @@ def _ensure_judged_methods_met(*, judged_methods: Iterable[JudgedMethod], names_
_raise_refused_lines(judged=judged, refused_lines=parsed.refused_lines)
if not parsed.requirements:
continue
- if environment is None:
- environment = resolve_pipe_func_package_environment(names_host_paths=names_host_paths)
+ environment = resolve_pipe_func_package_environment(requirements=parsed.requirements, names_host_paths=names_host_paths)
+ if isinstance(environment, PipeFuncPackageRefusal):
+ raise _refused_by_deployment_error(judged=judged, refusal=environment)
unmet = unmet_requirements(requirements=parsed.requirements, environment=environment)
if unmet:
_raise_unmet(judged=judged, unmet=unmet, environment=environment)
@@ -395,7 +374,8 @@ def ensure_method_requirements_met(*, trees: Iterable[MethodTree], names_host_pa
Raises:
MethodRequirementsFileError: A tree's `requirements.txt` holds a line the standard refuses, or cannot be read.
- MethodRequirementsNotMetError: The environment does not meet a tree's requirements.
+ MethodRequirementsNotMetError: The environment does not meet a tree's requirements, or the deployment refuses
+ them with its own reason.
PipeFuncPackageSetUnstatedError: PipeFunc steps run out of this process and the deployment states no package
set for them.
"""
@@ -413,7 +393,8 @@ def ensure_package_requirements_met(*, tree: MethodTree, package_address: str) -
Raises:
MethodRequirementsFileError: The package's `requirements.txt` holds a line the standard refuses, or cannot be read.
- MethodRequirementsNotMetError: The environment does not meet the package's requirements.
+ MethodRequirementsNotMetError: The environment does not meet the package's requirements, or the deployment
+ refuses them with its own reason.
PipeFuncPackageSetUnstatedError: PipeFunc steps run out of this process and the deployment states no package
set for them.
"""
@@ -437,7 +418,8 @@ def ensure_served_method_requirements_met(*, library_dirs: Sequence[Path], limit
Raises:
HostLibraryTreeError: A tree of the directories could not be listed.
MethodRequirementsFileError: A tree's `requirements.txt` holds a line the standard refuses, or cannot be read.
- MethodRequirementsNotMetError: The environment does not meet a tree's requirements.
+ MethodRequirementsNotMetError: The environment does not meet a tree's requirements, or the deployment refuses
+ them with its own reason.
PipeFuncPackageSetUnstatedError: PipeFunc steps run out of this process and the deployment states no package
set for them.
"""
diff --git a/pipelex/libraries/package_environments.py b/pipelex/libraries/package_environments.py
index d6cb320b8..09383e642 100644
--- a/pipelex/libraries/package_environments.py
+++ b/pipelex/libraries/package_environments.py
@@ -5,9 +5,11 @@ Where a PipeFunc runs decides what its requirements are checked against:
- In the `direct` execution mode it runs in this interpreter, so the check reads the distributions installed here and
this interpreter's own marker environment, and can follow a requirement's extras through their metadata.
- In any other mode it runs in a sandbox this process never sees, such as the Daytona sandbox, whose packages are the
- whole offer. The plugin providing the mode states them (`pipelex/plugins/pipe_func_package_offer_registry.py`), and
- the check reads that statement alone: a marker variable it does not state cannot be judged, and an extra's needs,
- which it does not state, cannot be followed. A mode with no statement fails closed.
+ whole offer. The plugin providing the mode states them for the requirements being judged
+ (`pipelex/plugins/pipe_func_package_offer_registry.py`), a fixed package set or an environment it builds for them,
+ and the check reads that statement alone: a marker variable it does not state cannot be judged, and an extra's needs,
+ which it does not state, cannot be followed. The plugin may instead refuse the requirements with its own reason, when
+ no environment it can build holds them. A mode with no statement fails closed.
How an environment names itself in a refusal depends on who reads it: this interpreter is named by its path on the
host, and its advice gives the exact install command, only where the refusal may name paths on the host.
@@ -15,7 +17,7 @@ host, and its advice gives the exact install command, only where the refusal may
import shlex
import sys
-from collections.abc import Mapping
+from collections.abc import Mapping, Sequence
from importlib.metadata import PackageNotFoundError, distribution
from typing import Protocol
@@ -24,9 +26,10 @@ from packaging.utils import canonicalize_name
from typing_extensions import override
from pipelex.config import get_config, get_pipe_func_execution_mode, is_pipe_func_sandbox_hosted
+from pipelex.core.method_trees.requirements_file import RequirementLine
from pipelex.interpreter_hub import get_pipe_func_package_offer_registry
from pipelex.libraries.exceptions import PipeFuncPackageSetUnstatedError
-from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOffer
+from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOffer, PipeFuncPackageOfferRequest, PipeFuncPackageRefusal
class PythonPackageEnvironment(Protocol):
@@ -140,10 +143,17 @@ class OfferedPackageEnvironment(PythonPackageEnvironment):
)
-def resolve_pipe_func_package_environment(*, names_host_paths: bool) -> PythonPackageEnvironment:
- """The environment this process's PipeFunc steps run in, as its execution mode decides.
+def resolve_pipe_func_package_environment(
+ *, requirements: Sequence[RequirementLine], names_host_paths: bool
+) -> PythonPackageEnvironment | PipeFuncPackageRefusal:
+ """The environment this process's PipeFunc steps would run a method declaring `requirements` in, as its execution mode decides.
+
+ In the `direct` mode it is this interpreter, whatever the requirements. In any other mode the plugin providing the
+ mode is asked for the offer meeting these requirements, and may refuse them instead, when no environment it can
+ run the method in holds them: its refusal is returned as it came, for the check to refuse the method with its reason.
Args:
+ requirements: Every requirement the method's `requirements.txt` declares, in the file's order.
names_host_paths: Whether a refusal naming the environment may name paths on this host, such as this
interpreter's.
@@ -157,7 +167,9 @@ def resolve_pipe_func_package_environment(*, names_host_paths: bool) -> PythonPa
offer_factory = get_pipe_func_package_offer_registry().get_optional(mode=execution_mode)
if offer_factory is None:
raise PipeFuncPackageSetUnstatedError(execution_mode=execution_mode, offer_registered=False)
- offer = offer_factory(get_config().interpreter.pipe_func)
- if offer is None:
+ answer = offer_factory(PipeFuncPackageOfferRequest(pipe_func_config=get_config().interpreter.pipe_func, requirements=tuple(requirements)))
+ if answer is None:
raise PipeFuncPackageSetUnstatedError(execution_mode=execution_mode, offer_registered=True)
- return OfferedPackageEnvironment(offer=offer)
+ if isinstance(answer, PipeFuncPackageRefusal):
+ return answer
+ return OfferedPackageEnvironment(offer=answer)
diff --git a/pipelex/plugins/contract.py b/pipelex/plugins/contract.py
index 392294e02..1cd12bc8d 100644
--- a/pipelex/plugins/contract.py
+++ b/pipelex/plugins/contract.py
@@ -81,6 +81,11 @@ if TYPE_CHECKING:
# reads under the run's read scope (``BundleValidatorProtocol``). A validator written against v7 would be called with a
# keyword it does not accept on its first validation; the bump turns that into a discovery-time mismatch naming the
# plugin.
+#
+# The PipeFunc package-offer factory changed under v8 too, before any release carried v8: a
+# ``PipeFuncPackageOfferFactoryFn`` receives a ``PipeFuncPackageOfferRequest``, the pipe_func config and the requirements
+# of the one method being judged, where it received the config alone, and may answer a ``PipeFuncPackageRefusal``
+# beside an offer or ``None``. A factory written against v7 would read the request as the config it expected.
PLUGIN_API_VERSION: int = 8
diff --git a/pipelex/plugins/pipe_func_package_offer_registry.py b/pipelex/plugins/pipe_func_package_offer_registry.py
index 1071a7010..991d39db4 100644
--- a/pipelex/plugins/pipe_func_package_offer_registry.py
+++ b/pipelex/plugins/pipe_func_package_offer_registry.py
@@ -7,15 +7,20 @@ runs in a sandbox this process never sees, such as the Daytona sandbox, whose pa
cannot read them, so the plugin that contributes the mode's executor states them here, with
``PluginRegistrar.add_pipe_func_package_offer``, keyed by the same execution mode token.
+The factory is asked for one method's requirements at a time, so a mode whose sandbox is not one fixed package set, such
+as one building an environment for each method, states the environment it would run that method in, or refuses the
+requirements with its own reason when no environment it can build holds them.
+
A mode for which no plugin states a package set, or whose factory answers ``None``, meets no requirement at all: the
check fails closed rather than letting a method load whose Python would then fail inside the sandbox.
"""
from collections.abc import Callable, Mapping
-from typing import TypeAlias
+from typing import NamedTuple, TypeAlias
from pydantic import BaseModel, ConfigDict, Field
+from pipelex.core.method_trees.requirements_file import RequirementLine
from pipelex.system.configuration.pipe_func_config import PipeFuncConfig
@@ -41,10 +46,57 @@ class PipeFuncPackageOffer(BaseModel):
)
-# A plugin's factory for the package offer of one PipeFunc execution mode: the pipe_func config in, the offer out, or
-# ``None`` when the deployment cannot state it. Called each time a method declaring requirements loads, never at
-# registration, so a factory may read its own configuration and should cache what it computes.
-PipeFuncPackageOfferFactoryFn: TypeAlias = Callable[[PipeFuncConfig], PipeFuncPackageOffer | None]
+class PipeFuncPackageRefusal(BaseModel):
+ """An execution mode's answer that no environment it can run a PipeFunc step in holds the requirements it was asked for.
+
+ The method declaring them is refused when it loads, as an input fault of the method's, so a refusal rests only on
+ requirements whose markers hold in the place the mode would run the method in: one whose marker is false there is
+ not needed there. The error carries the reason, so the reason names no path on the host, since it reaches the
+ method's author, a server's caller included.
+ """
+
+ model_config = ConfigDict(frozen=True, extra="forbid")
+
+ environment: str = Field(
+ description="The place the packages would be offered, as the refusal names it to the method's author, such as 'the Daytona sandbox'."
+ )
+ reason: str = Field(
+ description=(
+ "Why no environment of the place holds the requirements, in the plugin's own words, such as a resolver's "
+ "explanation of the conflict it found."
+ )
+ )
+
+
+class PipeFuncPackageOfferRequest(NamedTuple):
+ """What a package-offer factory is asked: the packages it offers for one method's requirements.
+
+ ``requirements`` are every requirement the method's `requirements.txt` declares, in the file's order, markers
+ included and none of them evaluated yet, as the MTHDS standard reads them. Pipelex cannot evaluate a marker before
+ it knows the environment the mode would run the method in, so the markers are the factory's to evaluate, against
+ the marker environment of the place it would offer: a requirement whose marker is false there is not needed there,
+ and the check skips it, so the factory never builds for it and never refuses for it. A refusal rests only on
+ requirements whose markers hold there, which ``requirements_needed_in`` (`pipelex/libraries/method_requirements.py`)
+ picks out as the check would. Pipelex then judges each requirement against the offer.
+ """
+
+ pipe_func_config: PipeFuncConfig
+ requirements: tuple[RequirementLine, ...]
+
+
+# A plugin's factory for the package offer of one PipeFunc execution mode: the request in; out, the offer meeting the
+# requirements it names (the mode's fixed package set, or an environment built for them), a refusal carrying the
+# plugin's reason when no environment of the mode can hold them, or ``None`` when the deployment cannot state its
+# packages at all. The requirements arrive with their markers unevaluated: the factory evaluates them against the marker
+# environment of the place it would offer, and a refusal rests only on those whose markers hold there, never on one the
+# place does not need (``requirements_needed_in``). Called synchronously, once for each method tree declaring
+# requirements each time a check judges it, never at registration, from whatever thread runs the load, which may be
+# running an event loop. It is therefore
+# synchronous, and caching its answers by the requirements is the plugin's job: a factory whose answer takes time to
+# compute, such as one resolving the requirements with a resolver, bounds the work with a timeout of its own and caches
+# by the requirements' texts, so only a process's first check of a requirement set pays for it. A fault of its own,
+# such as a resolver it cannot reach, is raised as an error of the plugin's, never answered as a refusal.
+PipeFuncPackageOfferFactoryFn: TypeAlias = Callable[[PipeFuncPackageOfferRequest], PipeFuncPackageOffer | PipeFuncPackageRefusal | None]
class PipeFuncPackageOfferRegistry:
diff --git a/pipelex/plugins/registrar.py b/pipelex/plugins/registrar.py
index b6198bca6..be22e2761 100644
--- a/pipelex/plugins/registrar.py
+++ b/pipelex/plugins/registrar.py
@@ -457,9 +457,12 @@ class PluginRegistrar:
A method's ``requirements.txt`` is checked when it loads against the packages its PipeFunc steps will run
with. In ``direct`` mode those are this interpreter's own; in any other mode they are the sandbox's, which this
process cannot read, so the plugin contributing that mode's executor states them here (e.g. our Daytona plugin,
- from the lock of the Daytona sandbox snapshot). ``factory`` is called with the pipe_func config each time a
- method declaring requirements loads, never here, and answers ``None`` when it cannot state the offer; a mode
- with no offer, or a ``None`` answer, meets no requirement. Fail-loud on a duplicate mode, naming both plugins.
+ from the lock of the Daytona sandbox snapshot). ``factory`` is called with a ``PipeFuncPackageOfferRequest``,
+ the pipe_func config and one method's requirements, each time a method declaring requirements loads, never
+ here. It answers the offer meeting them, such as an environment built for them, a ``PipeFuncPackageRefusal``
+ carrying its reason when no environment of the mode holds them, or ``None`` when it cannot state the offer; a
+ mode with no offer, or a ``None`` answer, meets no requirement. The call is synchronous, so a factory caches its
+ answers by the requirements. Fail-loud on a duplicate mode, naming both plugins.
"""
self._require_interpreter_layer(capability=f"pipe_func package offer {mode}")
self._add(
diff --git a/tests/integration/pipelex/libraries/test_method_requirements_load.py b/tests/integration/pipelex/libraries/test_method_requirements_load.py
index 160c84161..1f651c13a 100644
--- a/tests/integration/pipelex/libraries/test_method_requirements_load.py
+++ b/tests/integration/pipelex/libraries/test_method_requirements_load.py
@@ -10,8 +10,13 @@ from pipelex.interpreter_hub import get_library_manager, get_required_pipe, scop
from pipelex.libraries.exceptions import MethodRequirementsNotMetError, PipeFuncPackageSetUnstatedError
from pipelex.mthds_parsing.parser import MthdsParser
from pipelex.pipe_operators.func.pipe_func import PipeFunc
-from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOffer, PipeFuncPackageOfferFactoryFn, PipeFuncPackageOfferRegistry
-from pipelex.system.configuration.pipe_func_config import PipeFuncConfig
+from pipelex.plugins.pipe_func_package_offer_registry import (
+ PipeFuncPackageOffer,
+ PipeFuncPackageOfferFactoryFn,
+ PipeFuncPackageOfferRegistry,
+ PipeFuncPackageOfferRequest,
+ PipeFuncPackageRefusal,
+)
from tests.integration.pipelex.libraries.installed_packages import isolate_installed_methods, vendor_probe
from tests.integration.pipelex.libraries.test_data import ProbePackageTestData
@@ -65,14 +70,15 @@ def _method(*, directory: Path, requirements: str, sentinel: Path) -> Path:
return directory
-def _offering(mocker: MockerFixture, *, offer: PipeFuncPackageOffer | None) -> None:
- """Register what the `sandbox` execution mode's plugin would state, or nothing at all when `offer` is None."""
+def _offering(mocker: MockerFixture, *, factory: PipeFuncPackageOfferFactoryFn | None) -> None:
+ """Register the factory of the `sandbox` execution mode's plugin, or no factory at all when `factory` is None."""
+ offers: dict[str, PipeFuncPackageOfferFactoryFn] = {} if factory is None else {"sandbox": factory}
+ mocker.patch("pipelex.libraries.package_environments.get_pipe_func_package_offer_registry", return_value=PipeFuncPackageOfferRegistry(offers))
- def stated_offer(_config: PipeFuncConfig) -> PipeFuncPackageOffer | None:
- return offer
- offers: dict[str, PipeFuncPackageOfferFactoryFn] = {} if offer is None else {"sandbox": stated_offer}
- mocker.patch("pipelex.libraries.package_environments.get_pipe_func_package_offer_registry", return_value=PipeFuncPackageOfferRegistry(offers))
+def _stated_offer(_request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
+ """A sandbox of one fixed package set, whatever the requirements."""
+ return _OFFER
class TestMethodRequirementsLoad:
@@ -161,7 +167,7 @@ class TestMethodRequirementsLoad:
@pytest.mark.usefixtures("sandbox_hosted_mode")
def test_on_a_sandbox_hosted_deployment_the_stated_package_set_is_the_whole_offer(self, mocker: MockerFixture, tmp_path: Path) -> None:
- _offering(mocker, offer=_OFFER)
+ _offering(mocker, factory=_stated_offer)
sentinel = tmp_path / "funcs-module-ran"
refused_dir = _method(directory=tmp_path / "refused", requirements="pandas>=2\npandas>=3\n", sentinel=sentinel)
accepted_dir = _method(directory=tmp_path / "accepted", requirements="pandas>=2\nopenpyxl==3.1.5\n", sentinel=sentinel)
@@ -194,7 +200,7 @@ class TestMethodRequirementsLoad:
def test_a_sandbox_hosted_deployment_stating_no_package_set_refuses_a_method_with_requirements(
self, mocker: MockerFixture, tmp_path: Path
) -> None:
- _offering(mocker, offer=None)
+ _offering(mocker, factory=None)
library_dir = _method(directory=tmp_path / "method", requirements="pandas\n", sentinel=tmp_path / "funcs-module-ran")
library_manager = get_library_manager()
library_id, _ = library_manager.open_library()
@@ -203,3 +209,62 @@ class TestMethodRequirementsLoad:
library_manager.load_libraries(library_id=library_id, library_dirs=[library_dir])
finally:
library_manager.teardown(library_id=library_id)
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_a_sandbox_mode_building_an_environment_per_method_states_each_trees_own(self, mocker: MockerFixture, tmp_path: Path) -> None:
+ """Two methods loaded together each load against the environment stated for their own requirements, neither in the base."""
+ sentinel = tmp_path / "funcs-module-ran"
+ polars_dir = _method(directory=tmp_path / "polars", requirements="polars>=1\n", sentinel=sentinel)
+ excel_dir = _method(directory=tmp_path / "excel", requirements="pandas>=3\nxlsxwriter\n", sentinel=sentinel)
+ (excel_dir / "main.mthds").write_text(_MTHDS.replace('domain = "requirements_load"', 'domain = "requirements_excel"'), encoding="utf-8")
+ asked: list[list[str]] = []
+
+ def environment_for(request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
+ requirement_texts = [line.text for line in request.requirements]
+ asked.append(requirement_texts)
+ added = {"polars": "1.2.0"} if requirement_texts == ["polars>=1"] else {"pandas": "3.0.1", "xlsxwriter": "3.2.0"}
+ return PipeFuncPackageOffer(
+ environment=f"the Daytona sandbox environment for {', '.join(requirement_texts)}",
+ distributions={**_OFFER.distributions, **added},
+ marker_environment=_OFFER.marker_environment,
+ )
+
+ _offering(mocker, factory=environment_for)
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ try:
+ with scoped_current_library(library_id=library_id):
+ pipes = library_manager.load_libraries(library_id=library_id, library_dirs=[polars_dir, excel_dir])
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert sorted(pipe.domain_code for pipe in pipes) == ["requirements_excel", "requirements_load"]
+ # A load judges a tree at each of its checks, so the factory is asked again for the same requirements, which is
+ # why a plugin caches by them; each request carries one tree's requirements, never another's.
+ assert {tuple(requirement_texts) for requirement_texts in asked} == {("pandas>=3", "xlsxwriter"), ("polars>=1",)}
+ assert not sentinel.exists()
+
+ @pytest.mark.usefixtures("sandbox_hosted_mode")
+ def test_a_requirement_no_environment_can_hold_refuses_the_method_with_the_plugins_reason(self, mocker: MockerFixture, tmp_path: Path) -> None:
+ reason = "No solution found when resolving dependencies: pandas<3 conflicts with the base snapshot's pandas==2.2.3."
+
+ def refuse(_request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
+ return PipeFuncPackageRefusal(environment="the Daytona sandbox", reason=reason)
+
+ _offering(mocker, factory=refuse)
+ sentinel = tmp_path / "funcs-module-ran"
+ library_dir = _method(directory=tmp_path / "method", requirements="pandas<2\n", sentinel=sentinel)
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ try:
+ with scoped_current_library(library_id=library_id), pytest.raises(MethodRequirementsNotMetError) as exc_info:
+ library_manager.load_libraries(library_id=library_id, library_dirs=[library_dir])
+ finally:
+ library_manager.teardown(library_id=library_id)
+
+ assert exc_info.value.message == (
+ f"The method rooted at '{library_dir.resolve()}' needs Python packages that the Daytona sandbox cannot offer, "
+ f"as its requirements.txt declares: {reason}"
+ )
+ assert [(item.source, item.line) for item in exc_info.value.blueprint_validation_errors] == [("requirements.txt", None)]
+ assert not sentinel.exists()
diff --git a/tests/unit/pipelex/libraries/conftest.py b/tests/unit/pipelex/libraries/conftest.py
new file mode 100644
index 000000000..3457fd10c
--- /dev/null
+++ b/tests/unit/pipelex/libraries/conftest.py
@@ -0,0 +1,28 @@
+from collections.abc import Callable, Mapping
+from types import SimpleNamespace
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOfferFactoryFn, PipeFuncPackageOfferRegistry
+from pipelex.system.configuration.pipe_func_config import PipeFuncConfig
+
+_PACKAGE_ENVIRONMENTS = "pipelex.libraries.package_environments"
+
+
+def sandbox_offering(mocker: MockerFixture) -> Callable[..., PipeFuncConfig]:
+ """Run PipeFunc steps in the `daytona` mode with the package-offer factories given to `sandbox_offering(offers=...)` registered.
+
+ The call returns the mode's pipe_func config, the one each factory receives in its request.
+ """
+
+ def offering(*, offers: Mapping[str, PipeFuncPackageOfferFactoryFn]) -> PipeFuncConfig:
+ pipe_func_config = PipeFuncConfig(execution_mode="daytona")
+ mocker.patch(f"{_PACKAGE_ENVIRONMENTS}.is_pipe_func_sandbox_hosted", return_value=True)
+ mocker.patch(f"{_PACKAGE_ENVIRONMENTS}.get_pipe_func_execution_mode", return_value="daytona")
+ mocker.patch(f"{_PACKAGE_ENVIRONMENTS}.get_pipe_func_package_offer_registry", return_value=PipeFuncPackageOfferRegistry(offers))
+ mocker.patch(f"{_PACKAGE_ENVIRONMENTS}.get_config", return_value=SimpleNamespace(interpreter=SimpleNamespace(pipe_func=pipe_func_config)))
+ return pipe_func_config
+
+ return offering
diff --git a/tests/unit/pipelex/libraries/test_data.py b/tests/unit/pipelex/libraries/test_data.py
index 4e87a96c2..1eda280c4 100644
--- a/tests/unit/pipelex/libraries/test_data.py
+++ b/tests/unit/pipelex/libraries/test_data.py
@@ -102,6 +102,16 @@ class MethodRequirementsTestData:
]
SANDBOX_ENVIRONMENT = "the Daytona sandbox snapshot 'pipelex-base-test'"
+ # A Linux CPython 3.12, the marker environment of every box of the Daytona sandbox.
+ SANDBOX_MARKER_ENVIRONMENT: ClassVar[dict[str, str]] = {
+ "implementation_name": "cpython",
+ "os_name": "posix",
+ "platform_machine": "x86_64",
+ "platform_python_implementation": "CPython",
+ "platform_system": "Linux",
+ "python_version": "3.12",
+ "sys_platform": "linux",
+ }
@classmethod
def sandbox_offer(cls) -> PipeFuncPackageOffer:
@@ -109,13 +119,14 @@ class MethodRequirementsTestData:
return PipeFuncPackageOffer(
environment=cls.SANDBOX_ENVIRONMENT,
distributions={"pandas": "2.2.3", "openpyxl": "3.1.5", "numpy": "2.1.0rc1", "Zope.Interface": "7.0"},
- marker_environment={
- "implementation_name": "cpython",
- "os_name": "posix",
- "platform_machine": "x86_64",
- "platform_python_implementation": "CPython",
- "platform_system": "Linux",
- "python_version": "3.12",
- "sys_platform": "linux",
- },
+ marker_environment=dict(cls.SANDBOX_MARKER_ENVIRONMENT),
+ )
+
+ @classmethod
+ def environment_offer(cls, *, snapshot_name: str, distributions: dict[str, str]) -> PipeFuncPackageOffer:
+ """An environment a sandbox mode builds for one method's requirements, as it would state it: its own snapshot and packages."""
+ return PipeFuncPackageOffer(
+ environment=f"the Daytona sandbox environment '{snapshot_name}'",
+ distributions=distributions,
+ marker_environment=dict(cls.SANDBOX_MARKER_ENVIRONMENT),
)
diff --git a/tests/unit/pipelex/libraries/test_method_requirements_check.py b/tests/unit/pipelex/libraries/test_method_requirements_check.py
index a7e07618f..08c978e74 100644
--- a/tests/unit/pipelex/libraries/test_method_requirements_check.py
+++ b/tests/unit/pipelex/libraries/test_method_requirements_check.py
@@ -1,5 +1,6 @@
import shlex
import sys
+from collections.abc import Callable
from pathlib import Path
import pytest
@@ -17,11 +18,14 @@ from pipelex.libraries.method_requirements import (
ensure_method_requirements_met,
ensure_package_requirements_met,
ensure_served_method_requirements_met,
+ requirements_needed_in,
)
from pipelex.libraries.method_trees import reporting_host_library_tree_faults
from pipelex.libraries.package_environments import OfferedPackageEnvironment
from pipelex.pipeline.exceptions import ValidateBundleError
from pipelex.pipeline.validate_bundle_translation import translate_to_validate_bundle_error
+from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOffer, PipeFuncPackageOfferRequest, PipeFuncPackageRefusal
+from pipelex.system.configuration.pipe_func_config import PipeFuncConfig
from tests.unit.pipelex.libraries.test_data import MethodRequirementsTestData
_MODULE = "pipelex.libraries.method_requirements"
@@ -130,17 +134,114 @@ class TestMethodRequirementsCheck:
assert "line 2 'pandas>=3' (pandas 2.2.3 is there); line 3 'polars' (polars is missing)" in message
assert [(item.source, item.line) for item in exc_info.value.blueprint_validation_errors] == [("requirements.txt", 2), ("requirements.txt", 3)]
- def test_the_environment_is_resolved_once_and_each_tree_judged_once(self, mocker: MockerFixture, tmp_path: Path) -> None:
+ def test_each_tree_is_judged_once_against_the_environment_resolved_for_its_own_requirements(self, mocker: MockerFixture, tmp_path: Path) -> None:
resolve = mocker.patch(
f"{_MODULE}.resolve_pipe_func_package_environment",
return_value=OfferedPackageEnvironment(offer=MethodRequirementsTestData.sandbox_offer()),
)
first = list_method_tree(_method(directory=tmp_path / "first", requirements="pandas\n"))
- second = list_method_tree(_method(directory=tmp_path / "second", requirements="openpyxl\n"))
+ second = list_method_tree(_method(directory=tmp_path / "second", requirements="# spreadsheets\nopenpyxl\n"))
ensure_method_requirements_met(trees=[first, second, first], names_host_paths=True)
- resolve.assert_called_once_with(names_host_paths=True)
+ resolved_for = [
+ ([(line.line_number, line.text) for line in call.kwargs["requirements"]], call.kwargs["names_host_paths"])
+ for call in resolve.call_args_list
+ ]
+ assert resolved_for == [([(1, "pandas")], True), ([(2, "openpyxl")], True)]
+
+ def test_trees_judged_in_one_check_each_get_the_offer_stated_for_their_own_requirements(
+ self, tmp_path: Path, sandbox_offering: Callable[..., PipeFuncConfig]
+ ) -> None:
+ """A sandbox mode building an environment per method states one for each tree, and each tree is judged against its own."""
+ offers_by_requirements: dict[tuple[str, ...], PipeFuncPackageOffer] = {
+ ("polars>=1",): MethodRequirementsTestData.environment_offer(snapshot_name="pipelex-env-polars", distributions={"polars": "1.2.0"}),
+ ("pandas>=3",): MethodRequirementsTestData.environment_offer(snapshot_name="pipelex-env-pandas", distributions={"pandas": "3.0.1"}),
+ ("pandas>=4",): MethodRequirementsTestData.environment_offer(snapshot_name="pipelex-env-stale", distributions={"pandas": "3.0.1"}),
+ }
+ asked: list[tuple[str, ...]] = []
+
+ def offer_for(request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
+ requirement_texts = tuple(line.text for line in request.requirements)
+ asked.append(requirement_texts)
+ return offers_by_requirements[requirement_texts]
+
+ sandbox_offering(offers={"daytona": offer_for})
+ polars_tree = list_method_tree(_method(directory=tmp_path / "polars", requirements="polars>=1\n"))
+ pandas_tree = list_method_tree(_method(directory=tmp_path / "pandas", requirements="pandas>=3\n"))
+ stale_tree = list_method_tree(_method(directory=tmp_path / "stale", requirements="pandas>=4\n"))
+
+ ensure_method_requirements_met(trees=[polars_tree, pandas_tree], names_host_paths=False)
+ assert asked == [("polars>=1",), ("pandas>=3",)]
+
+ with pytest.raises(MethodRequirementsNotMetError) as exc_info:
+ ensure_method_requirements_met(trees=[pandas_tree, stale_tree], names_host_paths=False)
+
+ message = exc_info.value.message
+ assert message.startswith("The method needs Python packages that the Daytona sandbox environment 'pipelex-env-stale' does not have")
+ assert "line 1 'pandas>=4' (pandas 3.0.1 is there)" in message
+ assert "pipelex-env-pandas" not in message
+ assert [(item.source, item.line) for item in exc_info.value.blueprint_validation_errors] == [("requirements.txt", 1)]
+
+ def test_a_factory_refusing_only_what_its_sandbox_needs_lets_a_false_marker_requirement_load(
+ self, tmp_path: Path, sandbox_offering: Callable[..., PipeFuncConfig]
+ ) -> None:
+ """The factory sees every requirement, markers unevaluated, and rests a refusal only on those its sandbox needs."""
+ base = MethodRequirementsTestData.sandbox_offer()
+ asked: list[list[str]] = []
+
+ def offer_or_refuse(request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
+ asked.append([line.text for line in request.requirements])
+ needed = requirements_needed_in(requirements=request.requirements, marker_environment=base.marker_environment)
+ beyond = [line.text for line in needed if line.requirement.name not in base.distributions]
+ if beyond:
+ return PipeFuncPackageRefusal(environment="the Daytona sandbox", reason=f"no environment holds {', '.join(beyond)}")
+ return base
+
+ sandbox_offering(offers={"daytona": offer_or_refuse})
+ windows_only = list_method_tree(
+ _method(directory=tmp_path / "windows-only", requirements="pandas>=2\npywin32>=306; sys_platform == 'win32'\n")
+ )
+ everywhere = list_method_tree(_method(directory=tmp_path / "everywhere", requirements="pandas>=2\npywin32>=306\n"))
+
+ ensure_method_requirements_met(trees=[windows_only], names_host_paths=False)
+ assert asked == [["pandas>=2", "pywin32>=306; sys_platform == 'win32'"]]
+
+ with pytest.raises(MethodRequirementsNotMetError) as exc_info:
+ ensure_method_requirements_met(trees=[everywhere], names_host_paths=False)
+
+ assert exc_info.value.message == (
+ "The method needs Python packages that the Daytona sandbox cannot offer, "
+ "as its requirements.txt declares: no environment holds pywin32>=306"
+ )
+
+ def test_a_deployment_refusing_the_requirements_refuses_the_method_with_its_reason(
+ self, tmp_path: Path, sandbox_offering: Callable[..., PipeFuncConfig]
+ ) -> None:
+ """No environment of the mode can hold them: the method is refused as its own fault, carrying the plugin's words."""
+ reason = "uv explains: Because pandas<3 conflicts with the base snapshot's pandas==3.0.1, the requirements are unsatisfiable."
+
+ def refuse(_request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
+ return PipeFuncPackageRefusal(environment="the Daytona sandbox", reason=reason)
+
+ sandbox_offering(offers={"daytona": refuse})
+ method_dir = _method(directory=tmp_path / "request-files", requirements="pandas<3\n")
+
+ with pytest.raises(MethodRequirementsNotMetError) as refusal_info:
+ ensure_served_method_requirements_met(library_dirs=[method_dir])
+ with pytest.raises(ValidateBundleError) as verdict_info, translate_to_validate_bundle_error():
+ ensure_served_method_requirements_met(library_dirs=[method_dir])
+
+ refusal = refusal_info.value.to_error_report()
+ assert refusal.error_domain == ErrorDomain.INPUT
+ message = refusal.to_dict(disclosure_mode=DisclosureMode.STRICT)["message"]
+ assert message == f"The method needs Python packages that the Daytona sandbox cannot offer, as its requirements.txt declares: {reason}"
+ verdict = verdict_info.value.to_error_report()
+ assert verdict.validation_errors is not None
+ (item,) = verdict.validation_errors
+ assert (item.source, item.line) == ("requirements.txt", None)
+ assert item.message == f"The requirements are refused by the Daytona sandbox: {reason}"
+ assert str(tmp_path) not in str(refusal.to_dict(disclosure_mode=DisclosureMode.STRICT))
def test_a_served_method_checks_the_packages_nested_in_its_directory(self, tmp_path: Path) -> None:
_write(directory=tmp_path, tree_path="top.mthds", content="domain = 'requirements_top'\n")
diff --git a/tests/unit/pipelex/libraries/test_method_requirements_judgment.py b/tests/unit/pipelex/libraries/test_method_requirements_judgment.py
index 226f90b20..15b3fbbe6 100644
--- a/tests/unit/pipelex/libraries/test_method_requirements_judgment.py
+++ b/tests/unit/pipelex/libraries/test_method_requirements_judgment.py
@@ -7,7 +7,8 @@ from packaging.markers import Marker
from packaging.version import InvalidVersion
from pytest_mock import MockerFixture
-from pipelex.libraries.method_requirements import marker_variables_of, parse_requirements_text, unmet_requirements
+from pipelex.core.method_trees.requirements_file import parse_requirements_text
+from pipelex.libraries.method_requirements import marker_variables_of, requirements_needed_in, unmet_requirements
from pipelex.libraries.package_environments import InterpreterPackageEnvironment, OfferedPackageEnvironment
from tests.unit.pipelex.libraries.test_data import MethodRequirementsTestData
@@ -62,6 +63,29 @@ class TestMethodRequirementsJudgment:
(reason,) = reasons
assert expected_reason in reason, topic
+ @pytest.mark.parametrize(
+ ("topic", "line", "is_needed"),
+ [
+ ("no marker", "pandas>=2", True),
+ ("marker true there", 'polars; sys_platform == "linux"', True),
+ ("marker false there", 'pywin32; sys_platform == "win32"', False),
+ ("a line names no extra", 'polars; extra == "fast"', False),
+ ("marker variable not stated", 'pandas; platform_release >= "6"', True),
+ ("marker that cannot be evaluated", 'pandas; python_version ~= "abc"', True),
+ ],
+ )
+ def test_a_place_needs_every_requirement_its_marker_environment_does_not_rule_out(self, topic: str, line: str, is_needed: bool) -> None:
+ """What a factory may refuse for: the check skips exactly the requirements dropped here, and refuses the ones it cannot judge."""
+ parsed = parse_requirements_text(text=f"# first\n{line}\n")
+
+ needed = requirements_needed_in(requirements=parsed.requirements, marker_environment=MethodRequirementsTestData.SANDBOX_MARKER_ENVIRONMENT)
+
+ assert needed == (parsed.requirements if is_needed else []), topic
+ environment = OfferedPackageEnvironment(
+ offer=MethodRequirementsTestData.environment_offer(snapshot_name="pipelex-env-none", distributions={})
+ )
+ assert bool(unmet_requirements(requirements=parsed.requirements, environment=environment)) is is_needed, topic
+
def test_a_marker_value_spelled_like_a_variable_is_not_read_as_one(self) -> None:
"""Only the marker's variables are checked against what an environment states, never its quoted values."""
parsed = parse_requirements_text(text='pandas; os_name == "python_full_version"\n')
diff --git a/tests/unit/pipelex/libraries/test_method_requirements_parsing.py b/tests/unit/pipelex/libraries/test_method_requirements_parsing.py
index 68ab9480f..f94eb70ec 100644
--- a/tests/unit/pipelex/libraries/test_method_requirements_parsing.py
+++ b/tests/unit/pipelex/libraries/test_method_requirements_parsing.py
@@ -1,6 +1,6 @@
import pytest
-from pipelex.libraries.method_requirements import parse_requirements_text
+from pipelex.core.method_trees.requirements_file import parse_requirements_text
from tests.unit.pipelex.libraries.test_data import MethodRequirementsTestData
diff --git a/tests/unit/pipelex/libraries/test_package_environments.py b/tests/unit/pipelex/libraries/test_package_environments.py
index e7112fdf9..4fc61b1f8 100644
--- a/tests/unit/pipelex/libraries/test_package_environments.py
+++ b/tests/unit/pipelex/libraries/test_package_environments.py
@@ -1,50 +1,66 @@
-from types import SimpleNamespace
+from collections.abc import Callable
import pytest
from pytest_mock import MockerFixture
from pipelex.base_exceptions import ErrorDomain
+from pipelex.core.method_trees.requirements_file import RequirementLine, parse_requirements_text
from pipelex.libraries.exceptions import PipeFuncPackageSetUnstatedError
from pipelex.libraries.package_environments import InterpreterPackageEnvironment, OfferedPackageEnvironment, resolve_pipe_func_package_environment
-from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOffer, PipeFuncPackageOfferFactoryFn, PipeFuncPackageOfferRegistry
+from pipelex.plugins.pipe_func_package_offer_registry import (
+ PipeFuncPackageOffer,
+ PipeFuncPackageOfferFactoryFn,
+ PipeFuncPackageOfferRequest,
+ PipeFuncPackageRefusal,
+)
+from pipelex.system.configuration.pipe_func_config import PipeFuncConfig
from tests.unit.pipelex.libraries.test_data import MethodRequirementsTestData
_MODULE = "pipelex.libraries.package_environments"
-def _sandbox_hosted(mocker: MockerFixture, *, offers: dict[str, PipeFuncPackageOfferFactoryFn]) -> object:
- """Make this process run PipeFunc steps in the `daytona` mode, with the given offers registered; return its pipe_func config."""
- pipe_func_config = object()
- mocker.patch(f"{_MODULE}.is_pipe_func_sandbox_hosted", return_value=True)
- mocker.patch(f"{_MODULE}.get_pipe_func_execution_mode", return_value="daytona")
- mocker.patch(f"{_MODULE}.get_pipe_func_package_offer_registry", return_value=PipeFuncPackageOfferRegistry(offers))
- mocker.patch(f"{_MODULE}.get_config", return_value=SimpleNamespace(interpreter=SimpleNamespace(pipe_func=pipe_func_config)))
- return pipe_func_config
+def _requirements(text: str) -> list[RequirementLine]:
+ return parse_requirements_text(text=text).requirements
class TestPackageEnvironments:
- def test_in_direct_mode_the_environment_is_this_interpreter(self, mocker: MockerFixture) -> None:
+ def test_in_direct_mode_the_environment_is_this_interpreter_whatever_the_requirements(self, mocker: MockerFixture) -> None:
registry = mocker.patch(f"{_MODULE}.get_pipe_func_package_offer_registry")
mocker.patch(f"{_MODULE}.is_pipe_func_sandbox_hosted", return_value=False)
- environment = resolve_pipe_func_package_environment(names_host_paths=True)
+ environment = resolve_pipe_func_package_environment(requirements=_requirements("polars>=1\n"), names_host_paths=True)
assert isinstance(environment, InterpreterPackageEnvironment)
registry.assert_not_called()
- def test_in_a_sandbox_mode_the_environment_is_the_offer_its_plugin_states(self, mocker: MockerFixture) -> None:
+ def test_in_a_sandbox_mode_the_factory_is_given_the_requirements_and_its_offer_is_the_environment(
+ self, mocker: MockerFixture, sandbox_offering: Callable[..., PipeFuncConfig]
+ ) -> None:
factory = mocker.Mock(return_value=MethodRequirementsTestData.sandbox_offer())
- pipe_func_config = _sandbox_hosted(mocker, offers={"daytona": factory})
+ pipe_func_config = sandbox_offering(offers={"daytona": factory})
+ requirements = _requirements("pandas>=2\n# spreadsheets\nopenpyxl; sys_platform == 'linux'\n")
- environment = resolve_pipe_func_package_environment(names_host_paths=True)
+ environment = resolve_pipe_func_package_environment(requirements=requirements, names_host_paths=True)
assert isinstance(environment, OfferedPackageEnvironment)
- factory.assert_called_once_with(pipe_func_config)
+ factory.assert_called_once_with(PipeFuncPackageOfferRequest(pipe_func_config=pipe_func_config, requirements=tuple(requirements)))
+ (request,) = factory.call_args.args
+ assert [(line.line_number, line.text) for line in request.requirements] == [(1, "pandas>=2"), (3, "openpyxl; sys_platform == 'linux'")]
assert environment.description == MethodRequirementsTestData.SANDBOX_ENVIRONMENT
assert environment.version_of(distribution_name="zope-interface") == "7.0"
assert environment.version_of(distribution_name="polars") is None
assert environment.dependencies_of(distribution_name="pandas") is None
+ def test_a_refusal_the_factory_answers_is_returned_as_it_came(
+ self, mocker: MockerFixture, sandbox_offering: Callable[..., PipeFuncConfig]
+ ) -> None:
+ refusal = PipeFuncPackageRefusal(environment="the Daytona sandbox", reason="pandas<3 conflicts with the base snapshot's pandas 3.0.1")
+ sandbox_offering(offers={"daytona": mocker.Mock(return_value=refusal)})
+
+ answer = resolve_pipe_func_package_environment(requirements=_requirements("pandas<3\n"), names_host_paths=True)
+
+ assert answer is refusal
+
@pytest.mark.parametrize(
("topic", "offer_registered", "remedy"),
[
@@ -53,18 +69,18 @@ class TestPackageEnvironments:
],
)
def test_a_sandbox_mode_without_a_stated_package_set_fails_closed_naming_its_remedy(
- self, mocker: MockerFixture, topic: str, offer_registered: bool, remedy: str
+ self, sandbox_offering: Callable[..., PipeFuncConfig], topic: str, offer_registered: bool, remedy: str
) -> None:
"""No requirement can be judged, so a method declaring one is refused, as the deployment's fault rather than the method's."""
- def unstated(_config: object) -> PipeFuncPackageOffer | None:
+ def unstated(_request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
return None
offers: dict[str, PipeFuncPackageOfferFactoryFn] = {"daytona": unstated} if offer_registered else {"elsewhere": unstated}
- _sandbox_hosted(mocker, offers=offers)
+ sandbox_offering(offers=offers)
with pytest.raises(PipeFuncPackageSetUnstatedError) as exc_info:
- resolve_pipe_func_package_environment(names_host_paths=True)
+ resolve_pipe_func_package_environment(requirements=_requirements("pandas\n"), names_host_paths=True)
assert exc_info.value.execution_mode == "daytona"
assert exc_info.value.offer_registered is offer_registered, topic
diff --git a/tests/unit/pipelex/plugins/test_pipe_func_package_offer_registry.py b/tests/unit/pipelex/plugins/test_pipe_func_package_offer_registry.py
index 794eb5b5f..ffd66bcc9 100644
--- a/tests/unit/pipelex/plugins/test_pipe_func_package_offer_registry.py
+++ b/tests/unit/pipelex/plugins/test_pipe_func_package_offer_registry.py
@@ -7,12 +7,16 @@ import pytest
from pipelex.plugins.contract import PLUGIN_API_VERSION
from pipelex.plugins.exceptions import DuplicatePipeFuncPackageOfferError
-from pipelex.plugins.pipe_func_package_offer_registry import PipeFuncPackageOffer, PipeFuncPackageOfferRegistry
+from pipelex.plugins.pipe_func_package_offer_registry import (
+ PipeFuncPackageOffer,
+ PipeFuncPackageOfferRegistry,
+ PipeFuncPackageOfferRequest,
+ PipeFuncPackageRefusal,
+)
from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar
if TYPE_CHECKING:
from pipelex.system.configuration.configs import PipelexConfig
- from pipelex.system.configuration.pipe_func_config import PipeFuncConfig
_OFFER = PipeFuncPackageOffer(
environment="the Daytona sandbox snapshot 'pipelex-base-test'",
@@ -25,7 +29,7 @@ def _make_registrar() -> PluginRegistrar:
return PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(runtime=SimpleNamespace(plugins=SimpleNamespace(disabled=[])))))
-def _offer_factory(_config: PipeFuncConfig) -> PipeFuncPackageOffer | None:
+def _offer_factory(_request: PipeFuncPackageOfferRequest) -> PipeFuncPackageOffer | PipeFuncPackageRefusal | None:
return _OFFER
@@ -59,3 +63,10 @@ class TestPipeFuncPackageOfferRegistry:
"""A plugin's statement is closed plain data, so a misspelled field fails at the plugin rather than reading as unstated."""
with pytest.raises(ValueError, match="extra"):
PipeFuncPackageOffer.model_validate({**_OFFER.model_dump(), "marker_environments": {}})
+
+ def test_a_refusal_states_where_and_why_and_nothing_else(self) -> None:
+ refusal = PipeFuncPackageRefusal(environment="the Daytona sandbox", reason="no wheel of odfpy for the box")
+
+ assert refusal.model_dump() == {"environment": "the Daytona sandbox", "reason": "no wheel of odfpy for the box"}
+ with pytest.raises(ValueError, match="extra"):
+ PipeFuncPackageRefusal.model_validate({**refusal.model_dump(), "requirements": ["odfpy"]})