Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Inter-step-concept-check · L-261004-4593a9 (#1351)

Louis Choqueld6e61fd0bad1
diff --git a/.test_durations b/.test_durations
index d80935b69..5f290a007 100644
--- a/.test_durations
+++ b/.test_durations
@@ -1062,7 +1062,7 @@
"tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_list_output_bug.py::TestPipeSequenceListOutputBug::test_pipe_llm_list_output_produces_list_content_in_sequence": 8.8437,
"tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_list_output_bug.py::TestPipeSequenceListOutputBug::test_standalone_pipe_llm_with_list_output": 4.9491,
"tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_multiple_steps.py::TestPipeSequenceMultipleSteps::test_pipe_sequence_multiple_sub_pipes": 2.0117,
- "tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_output_multiplicity.py::TestPipeSequenceOutputMultiplicity::test_batch_over_with_nb_output_fixed_multiplicity": 0.267,
+ "tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_output_multiplicity.py::TestPipeSequenceOutputMultiplicity::test_batch_over_with_nb_output_yields_a_variable_list": 0.267,
"tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_output_multiplicity.py::TestPipeSequenceOutputMultiplicity::test_batch_over_with_no_multiplicity_fails": 0.0716,
"tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_output_multiplicity.py::TestPipeSequenceOutputMultiplicity::test_batch_over_with_variable_multiplicity_passes": 0.0644,
"tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_output_multiplicity.py::TestPipeSequenceOutputMultiplicity::test_fixed_multiplicity_compatible_with_variable_multiplicity": 0.0829,
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c36becd50..af7bc8281 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,6 +18,7 @@
- **An input name is a plain name (Breaking)**: every key of a pipe's `inputs`, on every operator, controller and signature, and a PipeBatch's `input_list_name` must match `[a-z][a-z0-9_]*`, so a dotted name such as `"invoice.total" = "Number"` or a malformed one such as `InvoiceTotal` is refused when the bundle is parsed, as the new `PipeValidationErrorType` value `invalid_input_name`, which also replaces PipeJudge's own refusal that reached callers with no `error_type`. Its message names the remedies: declare the root with its whole concept (`invoice = "Invoice"`) and read the field in the template (`$invoice.total`), which every operator handles by the concept the path reaches, so `@page.page_view` with `page = "Page"` is still attached as an image, or have the calling sequence bind the field with a binding step (`{ from = "invoice.total", result = "total_amount" }`). The generated MTHDS JSON Schema refuses the same names, through `patternProperties` on every `inputs` table and a `pattern` on `input_list_name`, and the corpus vocabulary records `error.invalid_input_name` with `fails_at = "schema"`.
- **A sequence's steps include binding steps (Breaking)**: `PipeSequenceBlueprint.steps` holds `BindingStepBlueprint` beside `SubPipeBlueprint`, and `PipeSequence.sequential_sub_pipes` holds `BindingStep` beside `SubPipe`, so code reading `pipe` or `pipe_code` off every step reads the new `pipe_steps` property instead. A PipeParallel's `branches` keep the pipe-step shape.
- **Names a nested controller stores (Breaking)**: a name a nested sequence, a condition's outcomes or an `add_each_output` parallel always stores in the caller's memory counts as stored by the step calling it, so it is no input of the caller, and a sequence that declared such a name only to pass validation is now refused as `extraneous_input_variable`: remove the declaration. A nested sequence or condition skipped for an absent plain input now records an absence for every name it always stores, so a later step finds a value or an absence there, never nothing.
+- **Every sequence step is checked against what it reads (Breaking)**: validation follows a PipeSequence step by step and refuses, as `input_stuff_spec_mismatch`, a pipe step whose pipe reads a name as a concept or a multiplicity the value stored there does not have, whether the sequence declared it or an earlier step stored it, so a step reading `cv_pages = "Page"` where an extraction stored `Page[]` now fails before anything runs: declare `Page[]`. A declared input every step reading it accepts validates even when it differs from what the last one reads, so a sequence whose first step reads `note` as `Markdown` and a later one as `Text` declares `note = "Markdown"`, which the sequence's inputs check used to refuse for not being the last step's `Text`, and that declaration is what the sequence needs from its caller, so a step calling it with a `Text` under `note` is refused. A step calling a nested controller is checked against the inputs the controller declares. A value a pipe step stores as `Anything` or `Dynamic`, as a condition whose outcomes produce different concepts declares its output, is assumed to be the concept a later step reads, its multiplicity still checked, while a sequence's own input declared `Anything` and a binding's value are checked as any other, also where a condition whose outcomes do not all store the name may leave them in place. A batched last step yields `X[]` whatever `nb_output` it asks for, so a sequence ending with one declares a variable list, where a fixed count or a single output used to pass. `PipeLibraryAbstract` and `ConceptLibraryAbstract` gain the abstract property `state_token`, which every change to a library replaces, so a sequence builds its typed flow once per state of the libraries it resolves in.
- **A dotted `batch_over` is a binding followed by a batch (Breaking)**: a sequence step's `batch_over = "catalog.pages"` binds the list at that path under a private name, `_bound_catalog_pages`, by every rule of a binding step, then batches over it, so a list root such as `catalogs.pages` batches over the pages of every catalog, the sequence needs the root as its own concept, and the execution graph draws the binding feeding the batch. A path deriving a single value is refused before the run as `input_stuff_spec_mismatch`, as a batch over a value that is not a list is, and a path outside the path grammar, or a dotted `batch_over` on a PipeParallel branch, as `binding_step_invalid`, which the MTHDS JSON Schema refuses too. The file-format check before the run follows each item of the bound list back to where it sits, as it does after an explicit binding step, so a document at `cases[1].transcripts[0]` that the batched pipe's model cannot read is refused before the batch starts. The `_bound_` prefix is reserved for these private names, since a nested sequence binds in its caller's working memory and would otherwise overwrite a caller's value stored under one: a pipe step's or a PipeParallel branch's `result`, `batch_as` or plain `batch_over`, or a PipeBatch's `input_item_name`, starting with `_bound_` is refused as `invalid_input_name`, which the MTHDS JSON Schema refuses too.
### Fixed
diff --git a/api/postman/sample-bundles/batch_candidate_screening.mthds b/api/postman/sample-bundles/batch_candidate_screening.mthds
index afe1a0157..750ee58a8 100644
--- a/api/postman/sample-bundles/batch_candidate_screening.mthds
+++ b/api/postman/sample-bundles/batch_candidate_screening.mthds
@@ -66,7 +66,7 @@ type = "PipeLLM"
description = """
Parses and summarizes the job requirements from the extracted job offer content, identifying required skills, responsibilities, qualifications, and nice-to-haves
"""
-inputs = { job_offer_pages = "Page" }
+inputs = { job_offer_pages = "Page[]" }
output = "JobRequirements"
model = "$writing-factual"
system_prompt = """
@@ -102,7 +102,7 @@ type = "PipeLLM"
description = """
Parses and summarizes the candidate's professional profile from the extracted CV content, identifying skills, experience, education, and achievements
"""
-inputs = { cv_pages = "Page" }
+inputs = { cv_pages = "Page[]" }
output = "CandidateProfile"
model = "$writing-factual"
system_prompt = """
diff --git a/docs/building-methods/pipes/pipe-controllers/PipeSequence.md b/docs/building-methods/pipes/pipe-controllers/PipeSequence.md
index 5f87dfaa4..db1d93e15 100644
--- a/docs/building-methods/pipes/pipe-controllers/PipeSequence.md
+++ b/docs/building-methods/pipes/pipe-controllers/PipeSequence.md
@@ -55,7 +55,31 @@ A binding step is a table with exactly these two keys:
| `result` | string | The name the bound value is stored under, a plain snake_case input name. | Yes |
!!! important "Output Concept Matching"
- The output concept of the `PipeSequence` has to match the output of its last step: the output of the last pipe, or the concept a binding step derives when the sequence ends with one.
+ The output concept of the `PipeSequence` has to match the output of its last step: the output of the last pipe, or the concept a binding step derives when the sequence ends with one. A batched last step yields the variable list of its branches' results, `X[]`, whatever `nb_output` it asks for, so the sequence declares a variable list.
+
+## What each step reads
+
+Validation follows the values through the sequence, one step after the other, and checks every pipe step against what it reads. At each step, a name in working memory holds the concept and multiplicity of the value last stored under it:
+
+- the sequence's declared `inputs`, to begin with;
+- a pipe step's `result`, its pipe's output, made a list by `nb_output` or `multiple_output`, and always a variable list `X[]` for a batched step, whatever `nb_output` it asks for;
+- what a nested `PipeSequence` or a `PipeCondition`'s outcome stores in the caller's memory, and the branch results of a `PipeParallel` with `add_each_output`;
+- what a binding step binds (see [What a binding binds](#what-a-binding-binds)).
+
+A pipe step is checked against the `inputs` its pipe declares. For a nested `PipeSequence`, `PipeCondition`, `PipeParallel` or `PipeBatch`, that declaration is its contract, which its own validation holds its steps, outcomes or branches to, whatever the last of them reads. A pipe step whose pipe reads a name as a concept or a multiplicity the name does not hold is refused with `input_stuff_spec_mismatch`, and the message names the step, the name and what stored it:
+
+```text
+In pipe 'process_cv', step 2 (pipe 'analyze_one_cv') reads 'cv_pages' as 'Page', but step 1 (pipe 'extract_one_cv') stores it as 'Page[]'. Declare the input as 'Page[]' in pipe 'analyze_one_cv', or make step 1 (pipe 'extract_one_cv') store a 'Page' under 'cv_pages'.
+```
+
+- The concept must be compatible with the read, as a sequence's output must be with its last step's: the same concept or a concept refining it, for instance, and any concept when the step reads `Anything` or `Dynamic`.
+- The multiplicity must match: a list is not a single value, and a single value is not a list. A fixed count such as `Color[5]` satisfies a step reading a variable list `Color[]`, and no other count.
+- A step's `batch_over` must name a list, whose items its pipe reads as the batch item's concept.
+- A name whose values may have different concepts, as when the outcomes of a `PipeCondition` store it under different concepts, must satisfy the read with each of them, including the value it held before when an outcome stores nothing under it.
+- A name a pipe that does not resolve at validation stored, such as a pipe of a dependency not loaded yet, is assumed to hold what the step reads.
+- A value a pipe step stores as `Anything` or `Dynamic`, as the result of a `PipeCondition` whose outcomes produce different concepts, has a concept the run alone knows, so it is assumed to be the concept the step reads, and the items of a list stored as `Anything[]` the concept a batched pipe reads. Its multiplicity is still checked: a single `Anything` read as a list, or batched over, is refused. The sequence's own input declared as `Anything`, and a binding's value of `Anything`, are checked as any other value, also where a `PipeCondition` whose outcomes do not all store the name may leave them in place, and a binding cannot walk a path through a value stored as `Anything`, which has no structure (`binding_path_unresolved`).
+
+Every step reading a declared input is checked this way, not only the last one, so a declared input must satisfy each step that reads it, and one that does is valid even when the last step reads it as something else. When a first step reads `note` as `Markdown` and a later one reads it as `Text`, the sequence declares `note = "Markdown"`, which both accept, and declaring `note = "Text"` is refused at the first step; when a first step reads `pages` as `Page[3]` and a later one as `Page[]`, it declares `pages = "Page[3]"`. That declaration is then what the sequence needs from whatever calls it: a step calling it with a `Text` under `note`, or a `Page[]` under `pages`, is refused. A declared input no step reads is still refused with `extraneous_input_variable`, and a name a step reads from the caller that the sequence does not declare with `missing_input_variable`.
## Binding steps
@@ -132,7 +156,7 @@ The bound value is a deep copy taken when the step runs: changing the root after
### What validation checks
-- A step that reads a bound name is checked against the derived concept and multiplicity: reading `total_amount` as `Text` when the binding derives a `Number` is refused with `input_stuff_spec_mismatch`, and `batch_over` a bound name requires a list.
+- A step that reads a bound name is checked against the derived concept and multiplicity, as every step is checked against what it reads (see [What each step reads](#what-each-step-reads)): reading `total_amount` as `Text` when the binding derives a `Number` is refused with `input_stuff_spec_mismatch`, and `batch_over` a bound name requires a list.
- A sequence ending with a binding step is checked as one ending with a pipe: its output concept with `inadequate_output_concept`, its multiplicity with `inadequate_output_multiplicity`, and a maybe-absent result with `optional_not_handled`.
- A step asking a sequence that ends with a binding step for a count of outputs, with `nb_output` or `multiple_output`, is refused with `inadequate_output_multiplicity` when the binding does not bind that count: a binding binds what its path derives, whatever count its caller asks for. A batched step asks its branches for nothing, so it is not checked.
- A root that is neither an input of the sequence nor stored by an earlier step on every run is refused with `missing_input_variable`.
diff --git a/docs/building-methods/pipes/understanding-multiplicity.md b/docs/building-methods/pipes/understanding-multiplicity.md
index bf162f837..67c133590 100644
--- a/docs/building-methods/pipes/understanding-multiplicity.md
+++ b/docs/building-methods/pipes/understanding-multiplicity.md
@@ -202,13 +202,13 @@ Create a single unified summary that captures the key points across all document
"""
```
-!!! info "When You Actually Need `[]` Notation"
- The `[]` notation is only required for **advanced use cases**:
-
- 1. **Batching over items**: When using `batch_over` in a `PipeSequence` to process each item separately
- 2. **Looping in templates**: When you need to iterate over items using Jinja2 syntax (`{% for item in items %}`) in `PipeLLM`, `PipeCompose`, or `PipeCondition` prompts
-
- For most cases where you simply pass multiple items to a pipe that processes them all together, you don't need to declare the input with `[]`. The pipe will receive the list and process it as a whole.
+!!! info "Declare the list a pipe reads"
+ A pipe that reads several items declares its input with `[]`, or `[N]` for a fixed count, even when it processes them all together in one prompt. Inside a `PipeSequence`, validation checks every step against what the steps before it stored (see [What each step reads](pipe-controllers/PipeSequence.md#what-each-step-reads)): a step whose pipe reads a single `Page` where an extraction stored `Page[]` is refused with `input_stuff_spec_mismatch`, and so is a step reading a list where a single value was stored.
+
+ The list notation is also what lets you:
+
+ 1. **Batch over the items**: `batch_over` in a `PipeSequence` runs a pipe once per item of a list
+ 2. **Loop in templates**: iterate over the items using Jinja2 syntax (`{% for item in items %}`) in `PipeLLM`, `PipeCompose`, or `PipeCondition` prompts
**3. Fixed count (bracket notation `[N]`)**
diff --git a/docs/features/validation-dry-run.md b/docs/features/validation-dry-run.md
index eeb9cc635..1c96ebdde 100644
--- a/docs/features/validation-dry-run.md
+++ b/docs/features/validation-dry-run.md
@@ -18,12 +18,13 @@ Check pipeline syntax, structure, and compatibility without execution:
- **Syntax validation** — Catch MTHDS language errors via plxt linting
- **Structure validation** — Resolve pipe and concept references in the loaded library, refuse concepts whose structures form a cycle or whose fields name a concept that cannot be found, and verify each pipe's declared inputs against the concepts that operator accepts — plus a controller's declared output against what it actually produces (for a `PipeSequence`, its last step's output concept and multiplicity)
- **Input validation** — Ensure required inputs are provided and correctly typed, and that every variable a step reads is bound by the time that step runs
+- **What each sequence step reads** — Follow each `PipeSequence` step by step and check every step against the concept and multiplicity of each name it reads, whichever declared input or earlier step stored it (`input_stuff_spec_mismatch`): a step reading a single `Page` where an extraction stored `Page[]` is refused before anything runs
- **Declared inputs are read** — For `PipeLLM`, `PipeCompose`, `PipeSearch` and `PipeImgGen`, every variable a template reads must be declared in `inputs` (`missing_input_variable`), and every declared input must be read by one of the pipe's templates (`extraneous_input_variable`): the prompt and system prompt of a `PipeLLM`, the template or the construct of a `PipeCompose`, the prompt of a `PipeSearch`, the prompt and negative prompt of a `PipeImgGen`. An unread input is refused on the pipe that declares it, before any controller above it is asked to supply it. A `PipeJudge` is the exception on that second rule: every input it declares is material for its model, so only the variables its `question` reads must be declared. A name the template sets with `{% set %}` is its own, not an input, from that statement on: after an `{% if %}` it stays set only when every branch sets it, the `{% else %}` included, so a name only some branches set is read from the input of that name on the other paths; and a loop, a macro or a block keeps what it sets to its own body.
A pipe that references a sub-pipe from a package that isn't loaded is reported as **skipped** rather than validated, so a passing run is not proof that every cross-package dependency resolves.
!!! note "Steps read from working memory, not from the previous step"
- A `PipeSequence` step resolves its inputs by name from working memory. They may come from the pipeline's inputs or from any earlier step — not necessarily the one immediately before it. Validation therefore checks that each name is bound when the step runs, rather than matching one step's output concept to the next step's declared input concept. There is no output-to-input chaining contract between consecutive steps to verify.
+ A `PipeSequence` step resolves its inputs by name from working memory. They may come from the pipeline's inputs or from any earlier step — not necessarily the one immediately before it. Validation therefore follows each name through the sequence rather than matching one step's output to the next step's input: it checks that each name is bound when the step runs, and that the value last stored under it has the concept and multiplicity the step reads. A refined concept satisfies a step reading its parent, a batched step always stores a list, a step calling a nested controller is held to the inputs it declares, and a name a pipe that does not resolve at validation stored, or a value a pipe step stores as `Anything`, such as the result of a condition whose outcomes produce different concepts, is assumed to hold the concept the step reads. See [PipeSequence](../building-methods/pipes/pipe-controllers/PipeSequence.md#what-each-step-reads).
## Dry Run Mode
diff --git a/docs/get-started/cv-batch-screening.md b/docs/get-started/cv-batch-screening.md
index 7f7626dde..b1cf377b7 100644
--- a/docs/get-started/cv-batch-screening.md
+++ b/docs/get-started/cv-batch-screening.md
@@ -93,7 +93,7 @@ type = "PipeLLM"
description = """
Parses and summarizes the job requirements from the extracted job offer content, identifying required skills, responsibilities, qualifications, and nice-to-haves
"""
-inputs = { job_offer_pages = "Page" }
+inputs = { job_offer_pages = "Page[]" }
output = "JobRequirements"
model = "$writing-factual"
system_prompt = """
@@ -128,7 +128,7 @@ type = "PipeLLM"
description = """
Parses and summarizes the candidate's professional profile from the extracted CV content, identifying skills, experience, education, and achievements
"""
-inputs = { cv_pages = "Page" }
+inputs = { cv_pages = "Page[]" }
output = "CandidateProfile"
model = "$writing-factual"
system_prompt = """
diff --git a/pipelex/libraries/concept/concept_library.py b/pipelex/libraries/concept/concept_library.py
index 71487d35f..5f0eece4e 100644
--- a/pipelex/libraries/concept/concept_library.py
+++ b/pipelex/libraries/concept/concept_library.py
@@ -14,6 +14,7 @@ from pipelex.core.qualified_ref import QualifiedRef
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.libraries.concept.concept_library_abstract import ConceptLibraryAbstract
from pipelex.libraries.concept.exceptions import ConceptLibraryError
+from pipelex.libraries.library_state import next_library_state_token
from pipelex.runtime_hub import get_class_registry
from pipelex.tools.typing.class_utils import are_structure_classes_compatible
@@ -26,6 +27,16 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
@override
def model_post_init(self, _context: Any) -> None:
self._concept_resolver: Callable[[str], Concept | None] | None = None
+ # Replaced on every change to what the library holds, so a value derived from the concepts it resolves knows it is stale.
+ self._state_token: int = next_library_state_token()
+
+ @property
+ @override
+ def state_token(self) -> int:
+ return self._state_token
+
+ def _mark_changed(self) -> None:
+ self._state_token = next_library_state_token()
def set_concept_resolver(self, resolver: Callable[[str], Concept | None]) -> None:
"""Set a resolver callback for cross-package concept lookups.
@@ -34,6 +45,7 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
resolver: A callable that takes a concept ref and returns the Concept or None
"""
self._concept_resolver = resolver
+ self._mark_changed()
@model_validator(mode="after")
def validation_static(self):
@@ -50,6 +62,7 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
@override
def teardown(self):
self.root = {}
+ self._mark_changed()
@override
def reset(self):
@@ -86,6 +99,7 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
msg = f"Concept '{concept.concept_ref}' already exists in the library"
raise ConceptLibraryError(msg)
self.root[concept.concept_ref] = concept
+ self._mark_changed()
@override
def add_concepts(self, concepts: list[Concept]):
@@ -97,6 +111,7 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
for concept_ref in concept_refs:
if concept_ref in self.root:
del self.root[concept_ref]
+ self._mark_changed()
@override
def is_compatible(self, *, tested_concept: Concept, wanted_concept: Concept, strict: bool = False) -> bool:
@@ -344,6 +359,7 @@ class ConceptLibrary(RootModel[ConceptLibraryRoot], ConceptLibraryAbstract):
msg = f"Dependency concept '{key}' already exists in the library"
raise ConceptLibraryError(msg)
self.root[key] = concept
+ self._mark_changed()
def is_concept_exists(self, concept_ref: str) -> bool:
return concept_ref in self.root
diff --git a/pipelex/libraries/concept/concept_library_abstract.py b/pipelex/libraries/concept/concept_library_abstract.py
index 55bf71a78..56a0ea7be 100644
--- a/pipelex/libraries/concept/concept_library_abstract.py
+++ b/pipelex/libraries/concept/concept_library_abstract.py
@@ -12,6 +12,14 @@ class ConceptLibraryAbstract(ConceptProviderAbstract):
interpreter layer.
"""
+ @property
+ @abstractmethod
+ def state_token(self) -> int:
+ """The token of what the library holds now, which every change to it replaces (`library_state`).
+
+ A value derived from the concepts the library resolves records it, and is stale once it differs.
+ """
+
@abstractmethod
def add_new_concept(self, concept: Concept) -> None:
pass
diff --git a/pipelex/libraries/library_state.py b/pipelex/libraries/library_state.py
new file mode 100644
index 000000000..b533aa151
--- /dev/null
+++ b/pipelex/libraries/library_state.py
@@ -0,0 +1,16 @@
+"""State tokens: a number naming one state of a library's contents, never reused.
+
+A pipe or concept library takes a fresh token when it is created and again on every change to what it holds, so a value
+derived from a library, such as the typed flow a PipeSequence builds from the pipes and concepts it resolves, can record
+the tokens it was derived under and know it is stale once either differs. Tokens come from one process-wide counter, so
+two libraries, or a library torn down and another created in its place, never share one.
+"""
+
+from itertools import count
+
+_STATE_TOKENS = count(start=1)
+
+
+def next_library_state_token() -> int:
+ """A token no library state has had before in this process."""
+ return next(_STATE_TOKENS)
diff --git a/pipelex/libraries/pipe/pipe_library.py b/pipelex/libraries/pipe/pipe_library.py
index 478697e96..2280bd317 100644
--- a/pipelex/libraries/pipe/pipe_library.py
+++ b/pipelex/libraries/pipe/pipe_library.py
@@ -6,6 +6,7 @@ from typing_extensions import override
from pipelex import pretty_print
from pipelex.core.qualified_ref import QualifiedRef
+from pipelex.libraries.library_state import next_library_state_token
from pipelex.libraries.pipe.exceptions import EntryPipeAmbiguousError, EntryPipeNotFoundError, PipeLibraryError, PipeNotFoundError
from pipelex.libraries.pipe.pipe_library_abstract import PipeLibraryAbstract
from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
@@ -24,6 +25,16 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
# The aliased keys of dependency pipes the package meant to load but that failed to build, with the reason, so a
# reference to one is refused naming that failure rather than as a pipe the package does not have.
_unbuilt_dependency_reasons: dict[str, str] = PrivateAttr(default_factory=dict)
+ # Replaced on every change to what the library holds, so a value derived from the pipes it resolves knows it is stale.
+ _state_token: int = PrivateAttr(default_factory=next_library_state_token)
+
+ @property
+ @override
+ def state_token(self) -> int:
+ return self._state_token
+
+ def _mark_changed(self) -> None:
+ self._state_token = next_library_state_token()
@override
def setup(self):
@@ -35,6 +46,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
self._private_dependency_keys = set()
self._withheld_dependency_keys = set()
self._unbuilt_dependency_reasons = {}
+ self._mark_changed()
@override
def reset(self):
@@ -58,6 +70,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
)
raise PipeLibraryError(msg)
self.root[pipe.pipe_ref] = pipe
+ self._mark_changed()
@override
def add_pipes(self, pipes: list[PipeAbstract]):
@@ -189,6 +202,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
self.root[key] = pipe
if not is_exported:
self._private_dependency_keys.add(key)
+ self._mark_changed()
def is_private_dependency_pipe(self, *, pipe_key: str) -> bool:
"""Whether `pipe_key`, an aliased key `alias->domain.code`, holds a dependency pipe its package does not export."""
@@ -197,6 +211,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
def add_withheld_dependency_pipes(self, *, alias: str, pipe_refs: set[str]) -> None:
"""Record the `domain.code` of the pipes a dependency declares but does not load, being private and unreached."""
self._withheld_dependency_keys.update(f"{alias}->{pipe_ref}" for pipe_ref in pipe_refs)
+ self._mark_changed()
def is_withheld_dependency_pipe(self, *, pipe_code: str) -> bool:
"""Whether a cross-package reference names a pipe its package declares but withholds, as private and unreached.
@@ -208,6 +223,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
def add_unbuilt_dependency_pipe(self, *, alias: str, pipe_ref: str, reason: str) -> None:
"""Record a dependency pipe, by its `domain.code`, that failed to build, and why."""
self._unbuilt_dependency_reasons[f"{alias}->{pipe_ref}"] = reason
+ self._mark_changed()
def unbuilt_dependency_pipe_reason(self, *, pipe_code: str) -> str | None:
"""Why the dependency pipe a cross-package reference names failed to build, or `None` when none it names failed.
@@ -267,6 +283,7 @@ class PipeLibrary(RootModel[PipeLibraryRoot], PipeLibraryAbstract):
if pipe_ref in self.root:
del self.root[pipe_ref]
self._private_dependency_keys.discard(pipe_ref)
+ self._mark_changed()
@override
def pretty_list_pipes(self) -> None:
diff --git a/pipelex/libraries/pipe/pipe_library_abstract.py b/pipelex/libraries/pipe/pipe_library_abstract.py
index d3decdbde..a1c7f767e 100644
--- a/pipelex/libraries/pipe/pipe_library_abstract.py
+++ b/pipelex/libraries/pipe/pipe_library_abstract.py
@@ -32,6 +32,14 @@ class PipeLibraryAbstract(ABC):
def get_required_pipe(self, pipe_code: str) -> PipeAbstract:
"""Resolve an in-body pipe reference, raising when it is not known."""
+ @property
+ @abstractmethod
+ def state_token(self) -> int:
+ """The token of what the library holds now, which every change to it replaces (`library_state`).
+
+ A value derived from the pipes the library resolves records it, and is stale once it differs.
+ """
+
@abstractmethod
def setup(self) -> None:
pass
diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.py
index aa9d3cc8c..4aa1a78d6 100644
--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py
+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py
@@ -1,7 +1,7 @@
import sys
from typing import TYPE_CHECKING, Any, Literal
-from pydantic import field_validator
+from pydantic import PrivateAttr, field_validator
from typing_extensions import override
from pipelex.core.concepts.native.concept_native import NativeConceptCode
@@ -16,7 +16,7 @@ from pipelex.core.pipes.variable_multiplicity import PresenceMarker, VariableMul
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff import Stuff
-from pipelex.interpreter_hub import get_concept_library, get_native_concept, get_optional_pipe, get_required_pipe
+from pipelex.interpreter_hub import get_concept_library, get_native_concept, get_optional_pipe, get_pipe_library, get_required_pipe
from pipelex.pipe_controllers.absence_taint import (
ForceConsumptionInfo,
LiftableStepInfo,
@@ -33,6 +33,7 @@ from pipelex.pipe_controllers.sequence.exceptions import PipeSequenceValueError
from pipelex.pipe_controllers.sequence.sequence_typed_flow import (
DerivedBinding,
FlowSlot,
+ SequenceFlowMemo,
SequenceStep,
SequenceTypedFlow,
authored_step_number,
@@ -41,7 +42,7 @@ from pipelex.pipe_controllers.sequence.sequence_typed_flow import (
step_memory_writes,
)
from pipelex.pipe_controllers.sub_pipe import SubPipe
-from pipelex.pipe_machinery.memory_writes import MemoryWrite, SlotTaint, taint_after_write
+from pipelex.pipe_machinery.memory_writes import MemoryWrite, SlotTaint, is_same_value_spec, taint_after_write
from pipelex.pipe_machinery.validation import is_valid_input_name
from pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunParams, output_multiplicity_to_apply
from pipelex.system.job_metadata import JobMetadata
@@ -50,12 +51,17 @@ from pipelex.validation_error_types import PipeValidationErrorType
if TYPE_CHECKING:
from pipelex.core.concepts.concept import Concept
from pipelex.libraries.library_crate import LibraryCrate
+ from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
class PipeSequence(PipeController):
type: Literal["PipeSequence"] = "PipeSequence"
# The steps in order: pipe steps, which run a pipe, and binding steps, which bind a value already in working memory.
sequential_sub_pipes: list[SequenceStep]
+ # The typed flow and what the sequence stores in its caller's memory, kept for the state of the libraries they were
+ # derived in (`SequenceFlowMemo`). An empty memo until the sequence derives them, never `None`: pydantic compares private
+ # attributes, and every memo equals every other, so what the sequence derived never changes what it equals.
+ _flow_memo: SequenceFlowMemo = PrivateAttr(default_factory=SequenceFlowMemo)
@override
def required_variables(self) -> set[str]:
@@ -84,11 +90,16 @@ class PipeSequence(PipeController):
@property
def has_binding_step(self) -> bool:
- """Whether a step binds a value: only binding steps read the typed flow, so a sequence without one never builds it."""
+ """Whether a step binds a value, which only a run of a sequence with one reads from the typed flow.
+
+ Every sequence builds its flow, to check each step against it and to say what it stores in its caller's memory
+ (`memory_writes`), once per state of the libraries it resolves in (`SequenceFlowMemo`): a run reads the flow kept
+ since validation, and a sequence without a binding step never reads it while it runs.
+ """
return any(isinstance(step, BindingStep) for step in self.sequential_sub_pipes)
def build_typed_flow(self) -> SequenceTypedFlow:
- """The typed flow of the steps, deriving what each binding step binds.
+ """The typed flow of the steps, deriving what each binding step binds, built once per state of the current libraries.
Raises:
PipeValidationError: ``BINDING_PATH_UNRESOLVED`` when a binding's path cannot be walked.
@@ -96,20 +107,61 @@ class PipeSequence(PipeController):
return self._build_typed_flow(visited_pipes=set())
def _build_typed_flow(self, *, visited_pipes: set[str]) -> SequenceTypedFlow:
- return build_sequence_typed_flow(
+ """The typed flow, from the memo when a walk that visited none of the pipes the sequence reaches already built it."""
+ memo = self._current_flow_memo()
+ is_kept = self._is_independent_of_walk(memo=memo, visited_pipes=visited_pipes)
+ if is_kept and memo.typed_flow is not None:
+ return memo.typed_flow
+ typed_flow = build_sequence_typed_flow(
steps=self.sequential_sub_pipes,
declared_inputs=self.inputs,
sequence_code=self.code,
domain_code=self.domain_code,
visited_pipes=visited_pipes | {self.visit_key},
)
+ if is_kept:
+ memo.typed_flow = typed_flow
+ return typed_flow
+
+ def _current_flow_memo(self) -> SequenceFlowMemo:
+ """The memo of what this sequence derived from the current pipe and concept libraries, a fresh one once either changed."""
+ library_state = (get_pipe_library().state_token, get_concept_library().state_token)
+ memo = self._flow_memo
+ if not memo.is_valid_for(owner=self, library_state=library_state):
+ memo = SequenceFlowMemo(owner=self, library_state=library_state)
+ self._flow_memo = memo
+ return memo
+
+ def _is_independent_of_walk(self, *, memo: SequenceFlowMemo, visited_pipes: set[str]) -> bool:
+ """Whether what the sequence derives inside a walk that visited `visited_pipes` is what it derives in any walk.
+
+ It is unless the walk visited a pipe the sequence reaches, which only a cycle does: the sequence's own walk then
+ stops at that pipe, which it would otherwise enter.
+ """
+ if memo.reachable_visit_keys is None:
+ memo.reachable_visit_keys = self._reachable_visit_keys()
+ return memo.reachable_visit_keys.isdisjoint(visited_pipes)
+
+ def _reachable_visit_keys(self) -> frozenset[str]:
+ """The `visit_key` of every pipe the sequence reaches through its steps' pipes, at any depth, itself excepted."""
+ reached_keys: set[str] = {self.visit_key}
+ pending_pipes: list[PipeAbstract] = [self]
+ while pending_pipes:
+ pending_pipe = pending_pipes.pop()
+ for dependency_code in pending_pipe.pipe_dependencies():
+ dependency = get_optional_pipe(pipe_code=dependency_code)
+ if dependency is None or dependency.visit_key in reached_keys:
+ continue
+ reached_keys.add(dependency.visit_key)
+ pending_pipes.append(dependency)
+ return frozenset(reached_keys - {self.visit_key})
@override
def memory_writes(self, *, visited_pipes: set[str] | None = None) -> dict[str, MemoryWrite]:
"""What the steps store, as the flow stands after the last one: a sequence runs its steps on its caller's memory.
Each name the steps store carries the spec the typed flow gives it, the absence the taint walk leaves on it, and
- whether some step stores it on every run.
+ whether some step stores it on every run. Built once per state of the current libraries, as the typed flow is.
Raises:
PipeValidationError: ``BINDING_PATH_UNRESOLVED`` when a binding's path cannot be walked, as validating the
@@ -119,9 +171,13 @@ class PipeSequence(PipeController):
visited_pipes = set()
if self.visit_key in visited_pipes:
return {}
+ memo = self._current_flow_memo()
+ is_kept = self._is_independent_of_walk(memo=memo, visited_pipes=visited_pipes)
+ if is_kept and memo.memory_writes is not None:
+ return dict(memo.memory_writes)
typed_flow = self._build_typed_flow(visited_pipes=visited_pipes)
taint_analysis = self._analyze_taint(visited_pipes=visited_pipes | {self.visit_key}, typed_flow=typed_flow)
- return {
+ written = {
name: MemoryWrite(
stuff_spec=slot.stuff_spec,
disagreement=slot.disagreement,
@@ -130,6 +186,9 @@ class PipeSequence(PipeController):
)
for name, slot in typed_flow.written_slots.items()
}
+ if is_kept:
+ memo.memory_writes = written
+ return dict(written)
def _walk_needed_inputs(self, *, visited_pipes: set[str]) -> tuple[InputStuffSpecs, set[str]]:
"""The inputs the steps need from the sequence's caller, and, among them, the roots of binding steps.
@@ -246,52 +305,100 @@ class PipeSequence(PipeController):
@override
def validate_inputs_with_library(self):
- """Derive every binding step through the typed flow, then check the steps reading a binding's result against it.
+ """Check every pipe step against the typed flow: each name its pipe reads must hold the concept and multiplicity it reads.
- A step asking a sequence ending with a binding for a count of outputs is checked against what that binding binds first.
+ The flow derives every binding step on the way. A step asking a sequence ending with a binding for a count of outputs
+ is checked against what that binding binds first.
"""
self._refuse_unhonoured_output_counts()
- if not self.has_binding_step:
- return
typed_flow = self.build_typed_flow()
for step_index, step in enumerate(self.sequential_sub_pipes):
if not isinstance(step, SubPipe):
continue
- binding_slots = typed_flow.binding_slots_by_pipe_step.get(step_index)
- if binding_slots:
- self._check_step_reading_bindings(step_index=step_index, step=step, binding_slots=binding_slots)
+ refusal = self._step_input_refusal(step_index=step_index, step=step, slots=typed_flow.slots_by_pipe_step.get(step_index, {}))
+ if refusal is not None:
+ raise refusal
+
+ def _accepted_declared_names(self, *, visited_pipes: set[str]) -> frozenset[str]:
+ """The declared inputs whose declared spec every pipe step reading the caller's value under them accepts, as the check
+ of each step against the typed flow (`validate_inputs_with_library`) judges it, once per state of the current libraries.
- def _check_step_reading_bindings(self, *, step_index: int, step: SubPipe, binding_slots: dict[str, FlowSlot]) -> None:
- """Refuse a pipe step reading, among the bindings' results it can see, one whose spec its pipe does not need.
+ That check is the authority on whether a declared input fits the steps reading it, so a declaration it accepts is what
+ the sequence needs (`needed_inputs`): a first step reading `note` as `Markdown` and a second reading it as `Text` are
+ both satisfied by a `Markdown`, which is not the last step's need.
Raises:
- PipeValidationError: ``INPUT_STUFF_SPEC_MISMATCH`` naming the step and the binding.
+ PipeValidationError: ``BINDING_PATH_UNRESOLVED`` when a binding's path cannot be walked, as validating the sequence
+ itself reports.
+ """
+ memo = self._current_flow_memo()
+ is_kept = self._is_independent_of_walk(memo=memo, visited_pipes=visited_pipes)
+ if is_kept and memo.accepted_declared_names is not None:
+ return memo.accepted_declared_names
+ typed_flow = self._build_typed_flow(visited_pipes=visited_pipes)
+ refused_names: set[str] = set()
+ for step_index, step in enumerate(self.sequential_sub_pipes):
+ if not isinstance(step, SubPipe):
+ continue
+ for variable_name, slot in typed_flow.slots_by_pipe_step.get(step_index, {}).items():
+ if variable_name in refused_names or variable_name not in self.inputs.root or slot.producer_step_index is not None:
+ # Already refused, not a declared input, or a value an earlier step stored there rather than the caller's.
+ continue
+ if self._step_input_refusal(step_index=step_index, step=step, slots={variable_name: slot}) is not None:
+ refused_names.add(variable_name)
+ accepted_names = frozenset(name for name in self.inputs.root if name not in refused_names)
+ if is_kept:
+ memo.accepted_declared_names = accepted_names
+ return accepted_names
+
+ def _step_input_refusal(self, *, step_index: int, step: SubPipe, slots: dict[str, FlowSlot]) -> PipeValidationError | None:
+ """The refusal of a pipe step whose pipe declares an input, among the slots given, as a spec the slot does not hold,
+ `None` when every one of them holds what the pipe declares.
+
+ A step is checked against the inputs its pipe declares, its contract. An operator needs exactly what it declares. A
+ controller's declaration is what its own validation holds its steps, outcomes or branches to, while what it needs
+ (`needed_inputs`) keeps one need per name, which for a nested sequence is its declaration only where every step reading
+ the name accepts it. Reading the declaration never walks the pipe's own steps, so the sequence's `needed_inputs` can ask
+ this check which of its declarations it accepts.
+
+ A slot the flow cannot type, a value a pipe that does not resolve stored, is assumed to deliver, and so is the concept of
+ a value a pipe step stored as `Anything` or `Dynamic`. A slot whose possible values have different specs is checked
+ against each of them, each read as what stored it. The refusal is returned rather than raised, so that the same check
+ answers which declared inputs are accepted (`_accepted_declared_names`).
+
+ Returns:
+ ``INPUT_STUFF_SPEC_MISMATCH`` naming the step, the name and what stored it, or `None`.
"""
step_pipe = get_optional_pipe(pipe_code=step.pipe_code)
if step_pipe is None:
- return
- step_needs = step_pipe.needed_inputs()
+ return None
+ step_contract = step_pipe.inputs
batch_item_name: str | None = None
if step.batch_params is not None:
batch_item_name = step.batch_params.input_item_stuff_name
- list_slot = binding_slots.get(step.batch_params.input_list_stuff_name)
+ list_slot = slots.get(step.batch_params.input_list_stuff_name)
if list_slot is not None:
- self._check_batched_binding(
+ list_refusal = self._batched_list_refusal(
step_index=step_index,
step_pipe_code=step_pipe.code,
batch_params=step.batch_params,
slot=list_slot,
- item_need=step_needs.root.get(batch_item_name),
+ item_need=step_contract.root.get(batch_item_name),
)
- for input_name, needed_spec in step_needs.items:
+ if list_refusal is not None:
+ return list_refusal
+ for input_name, needed_spec in step_contract.items:
if input_name == batch_item_name:
continue
- slot = binding_slots.get(input_name)
- if slot is None or slot.stuff_spec is None:
+ slot = slots.get(input_name)
+ if slot is None:
continue
- self._check_binding_consumer(
+ read_refusal = self._slot_read_refusal(
step_index=step_index, step_pipe_code=step_pipe.code, variable_name=input_name, slot=slot, needed_spec=needed_spec
)
+ if read_refusal is not None:
+ return read_refusal
+ return None
def final_binding_spec(self) -> StuffSpec | None:
"""The spec the binding step ending the sequence binds, `None` when the last step runs a pipe or the flow cannot type the binding's root.
@@ -355,101 +462,205 @@ class PipeSequence(PipeController):
required_concept_codes=[step_pipe.output.concept.concept_ref],
)
- def _check_binding_consumer(self, *, step_index: int, step_pipe_code: str, variable_name: str, slot: FlowSlot, needed_spec: StuffSpec) -> None:
- """Refuse a step reading a binding's result as a concept or multiplicity the binding does not derive."""
- bound_spec = slot.stuff_spec
- if bound_spec is None or slot.binding_step_index is None:
- return
- if needed_spec.concept.code in {NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING}:
- return
- binding_step = self.sequential_sub_pipes[slot.binding_step_index]
- binding_label = (
- binding_step.label if isinstance(binding_step, BindingStep) else f"step {self._step_number(step_index=slot.binding_step_index)}"
- )
- is_concept_compatible = get_concept_library().is_compatible(tested_concept=bound_spec.concept, wanted_concept=needed_spec.concept)
- is_multiplicity_ok = is_multiplicity_compatible(source_multiplicity=bound_spec.multiplicity, target_multiplicity=needed_spec.multiplicity)
- if is_concept_compatible and is_multiplicity_ok:
- return
- bound_ref = StuffSpec(concept=bound_spec.concept, multiplicity=bound_spec.multiplicity).to_bundle_representation(
- relative_to_domain=self.domain_code
- )
- needed_ref = StuffSpec(concept=needed_spec.concept, multiplicity=needed_spec.multiplicity).to_bundle_representation(
+ def _render_spec(self, *, stuff_spec: StuffSpec) -> str:
+ """A spec's concept and multiplicity as an author writes them in this sequence's domain, its presence aside."""
+ return StuffSpec(concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity).to_bundle_representation(
relative_to_domain=self.domain_code
)
- msg = (
- f"In pipe '{self.code}', step {self._step_number(step_index=step_index)} (pipe '{step_pipe_code}') reads '{variable_name}' as "
- f"'{needed_ref}', but the {binding_label} binds it as '{bound_ref}'. Declare the input as '{bound_ref}' in pipe '{step_pipe_code}', "
- f"or bind a path that reaches a '{needed_ref}'."
- )
- raise PipeValidationError(
+
+ def _producer_label(self, *, producer_step_index: int) -> str:
+ """How a message names the pipe step that stored a value, e.g. "step 2 (pipe 'weigh_parcel')"."""
+ producer_step = self.sequential_sub_pipes[producer_step_index]
+ if isinstance(producer_step, BindingStep):
+ return f"the {producer_step.label}"
+ producer_pipe = get_optional_pipe(pipe_code=producer_step.pipe_code)
+ producer_code = producer_pipe.code if producer_pipe is not None else producer_step.pipe_code
+ return f"step {self._step_number(step_index=producer_step_index)} (pipe '{producer_code}')"
+
+ def _slot_read_refusal(
+ self, *, step_index: int, step_pipe_code: str, variable_name: str, slot: FlowSlot, needed_spec: StuffSpec
+ ) -> PipeValidationError | None:
+ """The refusal of a step reading a name as a concept or multiplicity the value the flow carries there does not have,
+ `None` when the value satisfies the read.
+
+ A refined concept satisfies its parent, and a fixed count a variable list; a flexible need (`Dynamic`, `Anything`)
+ takes any value, and a value a pipe step stored as `Anything` or `Dynamic` has a concept known only when it runs, so
+ its concept is assumed to satisfy the read while its multiplicity is still checked. Among the values a disagreement
+ lists, only those a pipe step stored are read that way: the value a step may leave in place, when the caller declared it
+ or a binding stored it, is checked as any other. The message names what stored the value: a binding step, an earlier
+ pipe step, or the sequence's own declared input.
+
+ Returns:
+ ``INPUT_STUFF_SPEC_MISMATCH`` naming the step, the name and what stored it, or `None`.
+ """
+ if needed_spec.concept.code in {NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING}:
+ return None
+ step_label = f"step {self._step_number(step_index=step_index)} (pipe '{step_pipe_code}')"
+ needed_ref = self._render_spec(stuff_spec=needed_spec)
+ provided_spec = slot.stuff_spec
+ if provided_spec is None:
+ if slot.disagreement is None:
+ # A value a pipe that does not resolve at validation stored: nothing types it, so it is assumed to deliver.
+ return None
+ # Each value a disagreement lists is read as what stored it: a condition's outcome is a pipe store, while the value
+ # the name held before may be the caller's declared input or a binding's, which is checked as any other.
+ unreadable_specs = [
+ stored_spec.stuff_spec
+ for stored_spec in slot.disagreement.stored_specs
+ if not self._is_readable_as(
+ provided_spec=stored_spec.stuff_spec, needed_spec=needed_spec, is_stored_by_pipe_step=stored_spec.is_stored_by_pipe_step
+ )
+ ]
+ if not unreadable_specs:
+ return None
+ msg = (
+ f"In pipe '{self.code}', {step_label} reads '{variable_name}' as '{needed_ref}', but the values '{variable_name}' may hold "
+ f"have different specs: {slot.disagreement.describe(relative_to_domain=self.domain_code)}. Store '{variable_name}' as a "
+ f"'{needed_ref}' wherever it is stored, or declare the input of pipe '{step_pipe_code}' as a concept every one of them satisfies."
+ )
+ return PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,
+ domain_code=self.domain_code,
+ pipe_code=self.code,
+ variable_names=[variable_name],
+ provided_concept_code=unreadable_specs[0].concept.concept_ref,
+ required_concept_codes=[needed_spec.concept.concept_ref],
+ )
+ if self._is_readable_as(provided_spec=provided_spec, needed_spec=needed_spec, is_stored_by_pipe_step=slot.is_stored_by_pipe_step):
+ return None
+ provided_ref = self._render_spec(stuff_spec=provided_spec)
+ declare_remedy = f"Declare the input as '{provided_ref}' in pipe '{step_pipe_code}'"
+ if slot.binding_step_index is not None:
+ stored_phrase = f"{self._producer_label(producer_step_index=slot.binding_step_index)} binds it"
+ remedy = f"{declare_remedy}, or bind a path that reaches a '{needed_ref}'."
+ elif slot.producer_step_index is not None:
+ producer_label = self._producer_label(producer_step_index=slot.producer_step_index)
+ stored_phrase = f"{producer_label} stores it"
+ remedy = f"{declare_remedy}, or make {producer_label} store a '{needed_ref}' under '{variable_name}'."
+ else:
+ stored_phrase = "the sequence declares it"
+ remedy = f"{declare_remedy}, or declare '{variable_name}' as '{needed_ref}' in the inputs of pipe '{self.code}'."
+ msg = f"In pipe '{self.code}', {step_label} reads '{variable_name}' as '{needed_ref}', but {stored_phrase} as '{provided_ref}'. {remedy}"
+ return PipeValidationError(
message=msg,
error_type=PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,
domain_code=self.domain_code,
pipe_code=self.code,
variable_names=[variable_name],
- provided_concept_code=bound_spec.concept.concept_ref,
+ provided_concept_code=provided_spec.concept.concept_ref,
required_concept_codes=[needed_spec.concept.concept_ref],
)
- def _check_batched_binding(
+ @classmethod
+ def _is_readable_as(cls, *, provided_spec: StuffSpec, needed_spec: StuffSpec, is_stored_by_pipe_step: bool) -> bool:
+ """Whether a value of `provided_spec` satisfies a read as `needed_spec`: a compatible concept and multiplicity, the
+ concept of a value a pipe step stored being assumed to satisfy any read when it is known only when it runs.
+ """
+ if not is_multiplicity_compatible(source_multiplicity=provided_spec.multiplicity, target_multiplicity=needed_spec.multiplicity):
+ return False
+ if is_stored_by_pipe_step and cls._is_concept_known_only_at_run(stuff_spec=provided_spec):
+ return True
+ return get_concept_library().is_compatible(tested_concept=provided_spec.concept, wanted_concept=needed_spec.concept)
+
+ @staticmethod
+ def _is_concept_known_only_at_run(*, stuff_spec: StuffSpec) -> bool:
+ """Whether a pipe's output spec leaves the concept of the value it stores to the run: `Anything` or `Dynamic`, which a pipe
+ declares when what it produces varies, as a condition whose outcomes produce different concepts must declare `Anything`.
+
+ Only a pipe step's store is read this way. The run stores the concept the pipe actually produced, which nothing knows
+ before it, so the step reading it is assumed to get what it reads, as from a pipe that does not resolve at validation.
+ The sequence's own declared input is a contract its caller is held to, and a binding over a field holding `Anything`
+ binds a value whose concept is `Anything`, so both are checked as any other value, also where a later step may leave
+ them in place (`FlowSlot.is_stored_by_pipe_step`, `StoredSpec.is_stored_by_pipe_step`). The multiplicity, which the
+ pipe's declaration and the step's batch or count set, is checked whatever the concept.
+ """
+ return stuff_spec.concept.code in {NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING}
+
+ def _batched_list_refusal(
self, *, step_index: int, step_pipe_code: str, batch_params: BatchParams, slot: FlowSlot, item_need: StuffSpec | None
- ) -> None:
- """Refuse a step batching over a binding's result that is not a list, or whose items its pipe reads as another concept.
+ ) -> PipeValidationError | None:
+ """The refusal of a step batching over a value that is not a list, or whose items its pipe reads as another concept,
+ `None` when the value is a list of what the pipe reads.
A dotted `batch_over` is named by the path its author wrote, never by the private name the sequence bound it under.
-
- Raises:
- PipeValidationError: ``INPUT_STUFF_SPEC_MISMATCH``, as a batch over a value that is not a list is refused, naming
- the step and what the binding binds.
+ A value the flow cannot type is assumed to deliver; one whose possible values have different specs is checked
+ against each of them, each read as what stored it. A list a pipe step stored as `Anything[]` or `Dynamic[]` holds
+ items whose concept is known only when it runs, so they are assumed to be what the pipe reads, while a single value
+ is still no list; a list the caller declared or a binding stored as `Anything[]` is checked as any other.
+
+ Returns:
+ ``INPUT_STUFF_SPEC_MISMATCH``, as a batch over a value that is not a list is refused, naming the step and what
+ stored the list, or `None`.
"""
- bound_spec = slot.stuff_spec
- if bound_spec is None or slot.binding_step_index is None:
- return
- binding_step = self.sequential_sub_pipes[slot.binding_step_index]
- if not isinstance(binding_step, BindingStep):
- return
- if binding_step.is_dotted_batch_over:
- batched_phrase = f"the dotted path '{binding_step.from_path}', which derives"
- else:
- batched_phrase = f"'{batch_params.input_list_stuff_name}', which the {binding_step.label} binds as"
+ list_name = batch_params.input_list_stuff_name
+ item_name = batch_params.input_item_stuff_name
step_label = f"step {self._step_number(step_index=step_index)} (pipe '{step_pipe_code}')"
- bound_ref = StuffSpec(concept=bound_spec.concept, multiplicity=bound_spec.multiplicity).to_bundle_representation(
- relative_to_domain=self.domain_code
- )
- if not bound_spec.is_multiple():
+ binding_step = self.sequential_sub_pipes[slot.binding_step_index] if slot.binding_step_index is not None else None
+ is_bound_list = isinstance(binding_step, BindingStep)
+ # Each value the list may be, with whether a pipe step stored it: the slot's own provenance for a typed slot, and each
+ # value's for the values a disagreement lists, where the value the name held before may be the caller's or a binding's.
+ stored_values: list[tuple[StuffSpec, bool]]
+ batched_phrase: str
+ list_label = list_name
+ if slot.stuff_spec is not None:
+ stored_values = [(slot.stuff_spec, slot.is_stored_by_pipe_step)]
+ if isinstance(binding_step, BindingStep):
+ if binding_step.is_dotted_batch_over:
+ batched_phrase = f"the dotted path '{binding_step.from_path}', which derives"
+ list_label = binding_step.from_path
+ else:
+ batched_phrase = f"'{list_name}', which the {binding_step.label} binds as"
+ elif slot.producer_step_index is not None:
+ batched_phrase = f"'{list_name}', which {self._producer_label(producer_step_index=slot.producer_step_index)} stores as"
+ else:
+ batched_phrase = f"'{list_name}', which the sequence declares as"
+ elif slot.disagreement is not None:
+ stored_values = [(stored_spec.stuff_spec, stored_spec.is_stored_by_pipe_step) for stored_spec in slot.disagreement.stored_specs]
+ batched_phrase = f"'{list_name}', which may hold"
+ else:
+ # A list a pipe that does not resolve at validation stored: nothing types it, so it is assumed to deliver.
+ return None
+ for stored_spec, is_stored_by_pipe_step in stored_values:
+ stored_ref = self._render_spec(stuff_spec=stored_spec)
+ if not stored_spec.is_multiple():
+ list_remedy = "a path that reaches a list, through a list root or a list field" if is_bound_list else "a name holding a list"
+ msg = (
+ f"In pipe '{self.code}', {step_label} batches over {batched_phrase} a single '{stored_ref}', not a list: a batch runs its "
+ f"pipe once per item of a list. Batch over {list_remedy}, or run the step on the value itself, without `batch_over`."
+ )
+ return PipeValidationError(
+ message=msg,
+ error_type=PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,
+ domain_code=self.domain_code,
+ pipe_code=self.code,
+ variable_names=[list_label],
+ provided_concept_code=stored_spec.concept.concept_ref,
+ )
+ if item_need is None or item_need.concept.code in {NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING}:
+ continue
+ if is_stored_by_pipe_step and self._is_concept_known_only_at_run(stuff_spec=stored_spec):
+ continue
+ if get_concept_library().is_compatible(tested_concept=stored_spec.concept, wanted_concept=item_need.concept):
+ continue
+ item_ref = StuffSpec(concept=stored_spec.concept).to_bundle_representation(relative_to_domain=self.domain_code)
+ needed_ref = StuffSpec(concept=item_need.concept).to_bundle_representation(relative_to_domain=self.domain_code)
+ list_remedy = "a path that reaches a list" if is_bound_list else "a list"
msg = (
- f"In pipe '{self.code}', {step_label} batches over {batched_phrase} a single '{bound_ref}', not a list: a batch runs its "
- "pipe once per item of a list. Batch over a path that reaches a list, through a list root or a list field, or run the step "
- "on the value itself, without `batch_over`."
+ f"In pipe '{self.code}', {step_label} batches over {batched_phrase} '{stored_ref}', but its pipe reads each item, "
+ f"'{item_name}', as '{needed_ref}'. Declare '{item_name}' as '{item_ref}' in pipe '{step_pipe_code}', or batch over "
+ f"{list_remedy} of '{needed_ref}'."
)
- raise PipeValidationError(
+ return PipeValidationError(
message=msg,
error_type=PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,
domain_code=self.domain_code,
pipe_code=self.code,
- variable_names=[binding_step.from_path if binding_step.is_dotted_batch_over else batch_params.input_list_stuff_name],
- provided_concept_code=bound_spec.concept.concept_ref,
+ variable_names=[item_name],
+ provided_concept_code=stored_spec.concept.concept_ref,
+ required_concept_codes=[item_need.concept.concept_ref],
)
- if item_need is None or item_need.concept.code in {NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING}:
- return
- if get_concept_library().is_compatible(tested_concept=bound_spec.concept, wanted_concept=item_need.concept):
- return
- item_ref = StuffSpec(concept=bound_spec.concept).to_bundle_representation(relative_to_domain=self.domain_code)
- needed_ref = StuffSpec(concept=item_need.concept).to_bundle_representation(relative_to_domain=self.domain_code)
- msg = (
- f"In pipe '{self.code}', {step_label} batches over {batched_phrase} '{bound_ref}', but its pipe reads each item, "
- f"'{batch_params.input_item_stuff_name}', as '{needed_ref}'. Declare '{batch_params.input_item_stuff_name}' as '{item_ref}' in "
- f"pipe '{step_pipe_code}', or batch over a path that reaches a list of '{needed_ref}'."
- )
- raise PipeValidationError(
- message=msg,
- error_type=PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,
- domain_code=self.domain_code,
- pipe_code=self.code,
- variable_names=[batch_params.input_item_stuff_name],
- provided_concept_code=bound_spec.concept.concept_ref,
- required_concept_codes=[item_need.concept.concept_ref],
- )
+ return None
@override
def validate_output_static(self):
@@ -489,25 +700,12 @@ class PipeSequence(PipeController):
last_step_pipe = get_required_pipe(pipe_code=last_step_pipe_code)
last_step_concept = last_step_pipe.output.concept
last_step_label = f"'{last_step_pipe.code}'"
- is_last_step_output_optional = last_step_pipe.output.presence.is_optional
-
- # The step's effective output, resolved exactly the way the run path resolves it (declared
- # multiplicity + the step-level override), so this static promise cannot rule a count
- # differently from the execution it describes. Reading the two apart — an override tested for
- # truthiness, a declaration tested for `is None` — is what let a `nb_output = 1` step read as
- # plural here while the runtime made it single.
- multiplicity_resolution = output_multiplicity_to_apply(
- base_multiplicity=last_step_pipe.output.multiplicity,
- override_multiplicity=last_step.output_multiplicity,
- )
- # A declaration spells a singular slot with no suffix at all; `resolved_multiplicity` spells a
- # forced single as `False`, which is override vocabulary and renders as nothing an author writes.
- if not multiplicity_resolution.is_multiple_outputs_enabled:
- effective_last_step_output_multiplicity = None
- elif multiplicity_resolution.specific_output_count is not None:
- effective_last_step_output_multiplicity = multiplicity_resolution.specific_output_count
- else:
- effective_last_step_output_multiplicity = True
+ # What the step stores under its result, resolved exactly the way the run path and the typed flow resolve it
+ # (`SubPipe.result_spec`): the pipe's declared multiplicity with the step's `nb_output` or `multiple_output`
+ # override, or, for a batched step, the variable list of its branches' results, whatever count it asks for.
+ last_step_result_spec = last_step.result_spec(step_pipe=last_step_pipe)
+ is_last_step_output_optional = last_step_result_spec.presence.is_optional
+ effective_last_step_output_multiplicity = last_step_result_spec.multiplicity
is_plural_last_step_output = is_multiple_multiplicity(multiplicity=effective_last_step_output_multiplicity)
taint_analysis = self.analyze_taint()
@@ -780,6 +978,16 @@ class PipeSequence(PipeController):
@override
def needed_inputs(self, *, visited_pipes: set[str] | None = None) -> InputStuffSpecs:
+ """What the sequence needs from its caller: every name a step reads before an earlier step always stores it.
+
+ The walk of the steps keeps one need per name, the last step's reading it. Where the sequence declares the name and
+ every step reading the caller's value under it accepts the declaration (`_accepted_declared_names`), the declaration is
+ the need, its concept and multiplicity, so whatever calls the sequence, another sequence, a condition or a parallel, is
+ held to what every step accepts rather than to what the last one reads: a sequence declaring `note = "Markdown"` whose
+ last step reads a `Text` needs a `Markdown`. A declaration some step refuses leaves the last step's need, which the
+ generic check of the inputs then reports with the fix it offers. The presence stays the steps', as every reader of it
+ takes the declared marker first.
+ """
if visited_pipes is None:
visited_pipes = set()
@@ -787,8 +995,26 @@ class PipeSequence(PipeController):
if self.visit_key in visited_pipes:
return InputStuffSpecsFactory.make_empty()
- needed_inputs, _ = self._walk_needed_inputs(visited_pipes=visited_pipes)
- return needed_inputs
+ walked_needs, _ = self._walk_needed_inputs(visited_pipes=visited_pipes)
+ differing_names = [
+ input_name
+ for input_name, walked_spec in walked_needs.items
+ if input_name in self.inputs.root and not is_same_value_spec(first_spec=self.inputs.root[input_name], second_spec=walked_spec)
+ ]
+ if not differing_names:
+ return walked_needs
+ accepted_names = self._accepted_declared_names(visited_pipes=visited_pipes)
+ for input_name in differing_names:
+ if input_name not in accepted_names:
+ continue
+ declared_spec = self.inputs.root[input_name]
+ walked_needs.add_stuff_spec(
+ variable_name=input_name,
+ concept=declared_spec.concept,
+ multiplicity=declared_spec.multiplicity,
+ presence=walked_needs.root[input_name].presence,
+ )
+ return walked_needs
@override
def pipe_dependencies(self) -> set[str]:
@@ -979,21 +1205,23 @@ class PipeSequence(PipeController):
BindingStepRunError: Naming the step, the binding and both specs.
"""
runtime_slots: dict[str, FlowSlot] = {}
- for name, slot in typed_flow.binding_slots_by_pipe_step.get(step_index, {}).items():
+ for name, slot in typed_flow.slots_by_pipe_step.get(step_index, {}).items():
if slot.binding_step_index is None or slot.binding_step_index not in runtime_bindings:
continue
runtime_spec = runtime_bindings[slot.binding_step_index].stuff_spec
- runtime_slots[name] = FlowSlot(stuff_spec=runtime_spec, binding_step_index=slot.binding_step_index)
+ runtime_slots[name] = FlowSlot(
+ stuff_spec=runtime_spec, binding_step_index=slot.binding_step_index, producer_step_index=slot.binding_step_index
+ )
if not runtime_slots:
return
- try:
- self._check_step_reading_bindings(step_index=step_index, step=step, binding_slots=runtime_slots)
- except PipeValidationError as exc:
- msg = (
- f"{exc} The binding was derived when it ran, from the value its root held, since a pipe that did not resolve at "
- "validation stored the root."
- )
- raise BindingStepRunError(msg) from exc
+ refusal = self._step_input_refusal(step_index=step_index, step=step, slots=runtime_slots)
+ if refusal is None:
+ return
+ msg = (
+ f"{refusal} The binding was derived when it ran, from the value its root held, since a pipe that did not resolve at "
+ "validation stored the root."
+ )
+ raise BindingStepRunError(msg) from refusal
@override
async def _dry_run_controller_pipe(
diff --git a/pipelex/pipe_controllers/sequence/sequence_typed_flow.py b/pipelex/pipe_controllers/sequence/sequence_typed_flow.py
index ad6748073..0af585a59 100644
--- a/pipelex/pipe_controllers/sequence/sequence_typed_flow.py
+++ b/pipelex/pipe_controllers/sequence/sequence_typed_flow.py
@@ -18,14 +18,22 @@ run derives it from the value it holds, checking it there against what reads it.
`step_memory_writes` is the one place a step's stores beside its result are computed: the sequence's needed
inputs and its absence-taint walk read them from it too, so the three analyses agree on every name.
-It is used in three places, and only these: a binding types its root from the flow, a step reading a
-binding's result is checked against the spec the binding derives, and a binding ending the sequence is
-checked against the sequence's output. Extending the check to every step's inputs is a separate change.
+The flow serves three checks: a binding types its root from it, every pipe step is checked against the spec
+the flow carries for each name its pipe declares, whichever declared input, pipe step or binding put it there,
+and a binding ending the sequence is checked against the sequence's output.
+
+A pipe step's result is typed by the output its pipe declares. A pipe declaring `Anything` or `Dynamic`, as a
+condition whose outcomes produce different concepts must, stores a value whose concept is known only when it
+runs, so a step reading it is assumed to get the concept it reads, as from a pipe that does not resolve. The
+flow keeps what stored each value (`FlowSlot.is_stored_by_pipe_step`), and so does each value a disagreement
+lists (`StoredSpec.is_stored_by_pipe_step`), so a declared input or a binding's value a step may leave in
+place is checked as any other, whatever spec it holds.
"""
-from typing import NamedTuple
+from typing import Any, NamedTuple, Self
-from pydantic import BaseModel, ConfigDict
+from pydantic import BaseModel, ConfigDict, Field
+from typing_extensions import override
from pipelex.core.concepts.concept import Concept
from pipelex.core.pipes.exceptions import PipeValidationError
@@ -39,7 +47,7 @@ from pipelex.pipe_controllers.binding.binding_derivation import BindingDerivatio
from pipelex.pipe_controllers.binding.binding_step import BindingStep
from pipelex.pipe_controllers.binding.exceptions import BindingPathUnresolvedError
from pipelex.pipe_controllers.sub_pipe import SubPipe
-from pipelex.pipe_machinery.memory_writes import MemoryWrite, SpecDisagreement, StoredSpec, is_same_value_spec
+from pipelex.pipe_machinery.memory_writes import MemoryWrite, SpecDisagreement, is_same_value_spec, possible_values
from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
from pipelex.validation_error_types import PipeValidationErrorType
@@ -48,7 +56,7 @@ SequenceStep = SubPipe | BindingStep
class FlowSlot(BaseModel):
- """What a name holds at a point of the flow, and which binding step stored it, if one did."""
+ """What a name holds at a point of the flow, which step stored it, and which binding step, if one did."""
model_config = ConfigDict(frozen=True)
@@ -58,6 +66,20 @@ class FlowSlot(BaseModel):
binding_step_index: int | None = None
# Set only when `stuff_spec` is `None` because the values the name may hold have different specs, seen before the run.
disagreement: SpecDisagreement | None = None
+ # The index of the step that stored the value, as its result or as a name its pipe stores besides, `None` for a value
+ # the sequence's caller passed in, one of its declared inputs.
+ producer_step_index: int | None = None
+
+ @property
+ def is_stored_by_pipe_step(self) -> bool:
+ """Whether a pipe step stored the value, as its result or as a name its pipe stores besides, rather than the sequence's
+ caller under a declared input or a binding step.
+
+ The concept of a value a pipe step stored as `Anything` or `Dynamic` is known only when it runs, so a step reading it is
+ assumed to get the concept it reads. A declared input is a contract the caller is held to, and a binding derives its
+ concept from a path, so a value either one put there is checked as any other.
+ """
+ return self.producer_step_index is not None and self.binding_step_index is None
class SequenceTypedFlow(BaseModel):
@@ -69,8 +91,8 @@ class SequenceTypedFlow(BaseModel):
binding_derivations: dict[int, BindingDerivation]
# By step index, the derived spec of each binding step whose root the flow types.
binding_specs: dict[int, StuffSpec]
- # By pipe-step index, the slots a binding step stored that are still visible when the step runs.
- binding_slots_by_pipe_step: dict[int, dict[str, FlowSlot]]
+ # By pipe-step index, every slot visible when the step runs: the declared inputs and what earlier steps stored.
+ slots_by_pipe_step: dict[int, dict[str, FlowSlot]]
# The flow after the last step.
final_slots: dict[str, FlowSlot]
# The slots the steps stored, as they stand after the last step: what the sequence leaves in the memory it runs on.
@@ -81,6 +103,73 @@ class SequenceTypedFlow(BaseModel):
memory_writes_by_pipe_step: dict[int, dict[str, MemoryWrite]]
+class SequenceFlowMemo(BaseModel):
+ """What one PipeSequence derived from one state of the libraries its steps resolve in, kept so it is built once.
+
+ The typed flow and what the sequence stores in its caller's memory depend only on the sequence's own steps and on the
+ pipes and concepts they resolve to in the current library, so the memo records the state tokens of the current pipe
+ and concept libraries (`library_state`) and is discarded once either differs: a library that loads or removes a pipe or
+ a concept, or another library where a pipe code resolves differently, never reads a flow built before.
+
+ A walk that cut a cycle short, having already visited a pipe the sequence reaches, builds a flow that depends on what it
+ visited, so only the flows built by a walk that visited none of `reachable_visit_keys` are kept: those are the same
+ whatever else the walk visited.
+
+ The memo rests on one assumption: a pipe is never changed in place after it is built, and a changed pipe enters a library
+ as a new pipe, which replaces the library's state token.
+
+ A memo is a cache, never part of what its sequence is. Pydantic compares the private attributes of two models, where a
+ sequence keeps its memo, so every memo compares equal to every other: two sequences of one definition are equal whatever
+ each has derived, and comparing them never walks from a memo to its owner, whose memo it would compare again without end.
+ A sequence holds a memo from its creation, an empty one that is valid for no sequence, so that no comparison weighs a memo
+ against none. A copy of a memo, by `copy`, `deepcopy` or pickling, is empty.
+ """
+
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ # The sequence that built the memo, `None` for an empty memo, compared by identity only. A shallow copy of the sequence
+ # shares the memo object, and must not read it.
+ owner: Any = Field(default=None, repr=False)
+ # The state tokens of the pipe library and of the concept library the memo was built in, `(0, 0)` for an empty memo: no
+ # library state has the token 0.
+ library_state: tuple[int, int] = (0, 0)
+ # The `visit_key` of every pipe the sequence reaches through its steps, at any depth, itself excepted.
+ reachable_visit_keys: frozenset[str] | None = None
+ typed_flow: SequenceTypedFlow | None = None
+ memory_writes: dict[str, MemoryWrite] | None = None
+ # The declared inputs the check of each step accepts for every pipe step reading the caller's value under them.
+ accepted_declared_names: frozenset[str] | None = None
+
+ def is_valid_for(self, *, owner: object, library_state: tuple[int, int]) -> bool:
+ return self.owner is owner and self.library_state == library_state
+
+ @override
+ def __eq__(self, other: object) -> bool:
+ """Every memo equals every other, so that what a sequence derived never takes part in what it equals."""
+ if isinstance(other, SequenceFlowMemo):
+ return True
+ return NotImplemented
+
+ # A memo is never hashed, as the sequence holding it, a mutable model, is not. Python's own idiom for an unhashable class
+ # beside its `__eq__`, which mypy reads as an override of `object.__hash__`.
+ __hash__ = None # type: ignore[assignment]
+
+ @override
+ def __copy__(self) -> Self:
+ """A copy is empty: what the memo derived belongs to the sequence that built it."""
+ return self.__class__()
+
+ @override
+ def __deepcopy__(self, memo: dict[int, Any] | None = None) -> Self:
+ """A copy is empty, and never copies the sequence that owns the memo, which would copy the memo again."""
+ return self.__class__()
+
+ @override
+ def __reduce__(self) -> tuple[type[Self], tuple[()]]:
+ """A pickled memo comes back empty, as a copy does: the sequence unpickled beside it never built what it derived."""
+ return (self.__class__, ())
+
+
class DerivedBinding(NamedTuple):
"""What a binding step binds: the derivation of its path, and the spec of the value it stores."""
@@ -169,32 +258,39 @@ def step_memory_writes(*, step: SubPipe, step_pipe: PipeAbstract, visited_pipes:
return step_pipe.memory_writes(visited_pipes=visited_pipes)
-def slot_after_write(*, prior_slot: FlowSlot | None, memory_write: MemoryWrite, step_label: str) -> FlowSlot:
+def slot_after_write(*, prior_slot: FlowSlot | None, memory_write: MemoryWrite, step_index: int, step_label: str) -> FlowSlot:
"""What a name holds once a step stored it: the stored spec, or, when the step may leave the name as it was, the spec
the stored value and the value already there agree on, untyped when they do not.
- A name left untyped records why when the two specs are known and differ, or when either side carries a disagreement
- already; a side nothing could type, a pipe that does not resolve having stored it, leaves it untyped with no disagreement.
- `step_label` names the step for that record, e.g. "step 2 (pipe 'swap_record')".
+ A name left untyped records why when both sides are seen before the run, each typed or carrying a disagreement already:
+ the disagreement lists every value of both sides, so that a step reading the name is checked against each. A side nothing
+ could type, a pipe that does not resolve having stored it, leaves the name untyped with only the other side's disagreement,
+ if it carries one. The value held before keeps what stored it, so a declared input or a binding's value the step may leave
+ in place is still read as one (`StoredSpec.is_stored_by_pipe_step`). `step_index` is the storing step's, and `step_label`
+ names it for that record, e.g. "step 2 (pipe 'swap_record')".
"""
if memory_write.is_always_written or prior_slot is None:
- return FlowSlot(stuff_spec=memory_write.stuff_spec, disagreement=memory_write.disagreement)
+ return FlowSlot(stuff_spec=memory_write.stuff_spec, disagreement=memory_write.disagreement, producer_step_index=step_index)
if is_same_value_spec(first_spec=prior_slot.stuff_spec, second_spec=memory_write.stuff_spec):
return prior_slot
+ prior_values = possible_values(
+ stored_by=f"the value it held before {step_label}, which the step leaves when it stores nothing,",
+ stuff_spec=prior_slot.stuff_spec,
+ disagreement=prior_slot.disagreement,
+ is_stored_by_pipe_step=prior_slot.is_stored_by_pipe_step,
+ )
+ written_values = possible_values(
+ stored_by=f"{step_label}, when it stores a value,",
+ stuff_spec=memory_write.stuff_spec,
+ disagreement=memory_write.disagreement,
+ is_stored_by_pipe_step=True,
+ )
disagreement: SpecDisagreement | None
- if prior_slot.stuff_spec is not None and memory_write.stuff_spec is not None:
- disagreement = SpecDisagreement(
- stored_specs=(
- StoredSpec(
- stored_by=f"the value it held before {step_label}, which the step leaves when it stores nothing,",
- stuff_spec=prior_slot.stuff_spec,
- ),
- StoredSpec(stored_by=f"{step_label}, when it stores a value,", stuff_spec=memory_write.stuff_spec),
- )
- )
- else:
+ if prior_values is None or written_values is None:
disagreement = prior_slot.disagreement or memory_write.disagreement
- return FlowSlot(stuff_spec=None, disagreement=disagreement)
+ else:
+ disagreement = SpecDisagreement(stored_specs=prior_values + written_values)
+ return FlowSlot(stuff_spec=None, disagreement=disagreement, producer_step_index=step_index)
def untyped_root_error(*, sequence_code: str, domain_code: str, binding_step: BindingStep, disagreement: SpecDisagreement) -> PipeValidationError:
@@ -236,7 +332,7 @@ def build_sequence_typed_flow(
always_written_names: set[str] = set()
binding_derivations: dict[int, BindingDerivation] = {}
binding_specs: dict[int, StuffSpec] = {}
- binding_slots_by_pipe_step: dict[int, dict[str, FlowSlot]] = {}
+ slots_by_pipe_step: dict[int, dict[str, FlowSlot]] = {}
memory_writes_by_pipe_step: dict[int, dict[str, MemoryWrite]] = {}
for step_index, step in enumerate(steps):
@@ -250,23 +346,23 @@ def build_sequence_typed_flow(
# A root nothing types: an undeclared one, refused as a missing input of the sequence; or a value a pipe that
# does not resolve stored, assumed to deliver as the rest of the sequence's checks assume, which the run
# derives from the value it holds and checks against what reads it.
- slots[step.output_name] = FlowSlot(stuff_spec=None, binding_step_index=step_index)
+ slots[step.output_name] = FlowSlot(stuff_spec=None, binding_step_index=step_index, producer_step_index=step_index)
continue
derived_binding = derive_binding_spec(
binding_step=step, root_spec=root_slot.stuff_spec, sequence_code=sequence_code, domain_code=domain_code
)
binding_derivations[step_index] = derived_binding.derivation
binding_specs[step_index] = derived_binding.stuff_spec
- slots[step.output_name] = FlowSlot(stuff_spec=derived_binding.stuff_spec, binding_step_index=step_index)
+ slots[step.output_name] = FlowSlot(stuff_spec=derived_binding.stuff_spec, binding_step_index=step_index, producer_step_index=step_index)
continue
- binding_slots_by_pipe_step[step_index] = {name: slot for name, slot in slots.items() if slot.binding_step_index is not None}
+ slots_by_pipe_step[step_index] = dict(slots)
step_pipe = get_optional_pipe(pipe_code=step.pipe_code)
if step_pipe is None:
if step.output_name:
written_names.add(step.output_name)
always_written_names.add(step.output_name)
- slots[step.output_name] = FlowSlot(stuff_spec=None)
+ slots[step.output_name] = FlowSlot(stuff_spec=None, producer_step_index=step_index)
continue
step_writes = step_memory_writes(step=step, step_pipe=step_pipe, visited_pipes=visited_pipes)
memory_writes_by_pipe_step[step_index] = step_writes
@@ -275,16 +371,18 @@ def build_sequence_typed_flow(
written_names.add(written_name)
if memory_write.is_always_written:
always_written_names.add(written_name)
- slots[written_name] = slot_after_write(prior_slot=slots.get(written_name), memory_write=memory_write, step_label=step_label)
+ slots[written_name] = slot_after_write(
+ prior_slot=slots.get(written_name), memory_write=memory_write, step_index=step_index, step_label=step_label
+ )
if step.output_name:
written_names.add(step.output_name)
always_written_names.add(step.output_name)
- slots[step.output_name] = FlowSlot(stuff_spec=step.result_spec(step_pipe=step_pipe))
+ slots[step.output_name] = FlowSlot(stuff_spec=step.result_spec(step_pipe=step_pipe), producer_step_index=step_index)
return SequenceTypedFlow(
binding_derivations=binding_derivations,
binding_specs=binding_specs,
- binding_slots_by_pipe_step=binding_slots_by_pipe_step,
+ slots_by_pipe_step=slots_by_pipe_step,
final_slots=slots,
written_slots={name: slot for name, slot in slots.items() if name in written_names},
always_written_names=frozenset(always_written_names),
diff --git a/pipelex/pipe_machinery/memory_writes.py b/pipelex/pipe_machinery/memory_writes.py
index 2d4713510..6c53483b7 100644
--- a/pipelex/pipe_machinery/memory_writes.py
+++ b/pipelex/pipe_machinery/memory_writes.py
@@ -42,6 +42,11 @@ class StoredSpec(BaseModel):
# What stores the value, written to precede "as 'X'", e.g. "outcome 'stash_parcel' of pipe 'stash_by_mode'".
stored_by: str
stuff_spec: StuffSpec
+ # Whether a pipe step's run stored the value, so that a step reading it takes the concept of a value stored as `Anything`
+ # or `Dynamic` to be the one it reads, which only the run knows. A condition's outcomes are pipe stores. The value a name
+ # held before a step that may leave it is not one when the sequence's caller passed it under a declared input or a binding
+ # step stored it (`FlowSlot.is_stored_by_pipe_step`): a step reading it is held to its spec, as to any other declaration.
+ is_stored_by_pipe_step: bool = True
class SpecDisagreement(BaseModel):
@@ -82,6 +87,20 @@ class MemoryWrite(BaseModel):
is_always_written: bool = True
+def possible_values(
+ *, stored_by: str, stuff_spec: StuffSpec | None, disagreement: SpecDisagreement | None, is_stored_by_pipe_step: bool
+) -> tuple[StoredSpec, ...] | None:
+ """The values one side may leave under a name, as a disagreement lists them: the one value of a typed side, which `stored_by`
+ names, or each value the side's own disagreement lists, with what stored it. `None` for a side nothing could type before
+ the run, a pipe that does not resolve having stored it.
+ """
+ if stuff_spec is not None:
+ return (StoredSpec(stored_by=stored_by, stuff_spec=stuff_spec, is_stored_by_pipe_step=is_stored_by_pipe_step),)
+ if disagreement is not None:
+ return disagreement.stored_specs
+ return None
+
+
def is_same_value_spec(*, first_spec: StuffSpec | None, second_spec: StuffSpec | None) -> bool:
"""Whether two specs type the same value: the same concept and multiplicity. Presence aside, which an absence carries."""
if first_spec is None or second_spec is None:
@@ -102,9 +121,10 @@ def merge_alternative_writes(*, alternatives: list[AlternativeWrites]) -> dict[s
A name is always written only if every alternative always writes it. It may hold an absence if any alternative may leave
one. It keeps a spec only if every alternative storing it stores the same value spec; an alternative that does not store
it leaves the caller's value, which the caller merges with what the name held before (`is_always_written` is then false).
- A name left untyped records why when it is seen before the run (`SpecDisagreement`): the alternatives typing it store it
- under different specs, or one of them stores it with a disagreement of its own. Otherwise an alternative that could not
- type it, a pipe that does not resolve having stored it, leaves it untyped with no disagreement.
+ A name left untyped records why when it is seen before the run (`SpecDisagreement`), listing every value an alternative
+ may leave: the alternatives typing it store it under different specs, or one of them stores it with a disagreement of its
+ own. Otherwise an alternative that could not type it, a pipe that does not resolve having stored it, leaves it untyped
+ with no disagreement.
"""
merged_writes: dict[str, MemoryWrite] = {}
for alternative in alternatives:
@@ -128,18 +148,20 @@ def merge_alternative_writes(*, alternatives: list[AlternativeWrites]) -> dict[s
def _alternatives_disagreement(*, labeled_writes: list[tuple[str, MemoryWrite]]) -> SpecDisagreement | None:
"""Why the alternatives storing a name leave it untyped, when it is seen before the run: the specs they store disagree, or
- one of them stores the name with a disagreement of its own. `None` when an alternative could not type the name at all.
+ one of them stores the name with a disagreement of its own. The disagreement lists every value an alternative may leave,
+ those its own disagreement lists included, so that a step reading the name is checked against each. An alternative nothing
+ could type, a pipe that does not resolve having stored the name, lists none. `None` when the values listed agree.
"""
- typed_specs: list[StoredSpec] = []
+ stored_specs: list[StoredSpec] = []
for label, name_write in labeled_writes:
- if name_write.stuff_spec is not None:
- typed_specs.append(StoredSpec(stored_by=label, stuff_spec=name_write.stuff_spec))
- first_typed_spec = typed_specs[0].stuff_spec if typed_specs else None
- if any(not is_same_value_spec(first_spec=first_typed_spec, second_spec=typed_spec.stuff_spec) for typed_spec in typed_specs):
- return SpecDisagreement(stored_specs=tuple(typed_specs))
- for _, name_write in labeled_writes:
- if name_write.disagreement is not None:
- return name_write.disagreement
+ alternative_values = possible_values(
+ stored_by=label, stuff_spec=name_write.stuff_spec, disagreement=name_write.disagreement, is_stored_by_pipe_step=True
+ )
+ if alternative_values is not None:
+ stored_specs.extend(alternative_values)
+ first_spec = stored_specs[0].stuff_spec if stored_specs else None
+ if any(not is_same_value_spec(first_spec=first_spec, second_spec=stored_spec.stuff_spec) for stored_spec in stored_specs):
+ return SpecDisagreement(stored_specs=tuple(stored_specs))
return None
diff --git a/tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening.mthds b/tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening.mthds
index 44b3f468d..5b2c8a4b8 100644
--- a/tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening.mthds
+++ b/tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening.mthds
@@ -65,7 +65,7 @@ type = "PipeLLM"
description = """
Parses and summarizes the job requirements from the extracted job offer content, identifying required skills, responsibilities, qualifications, and nice-to-haves
"""
-inputs = { job_offer_pages = "Page" }
+inputs = { job_offer_pages = "Page[]" }
output = "JobRequirements"
model = "$writing-factual"
system_prompt = """
@@ -100,7 +100,7 @@ type = "PipeLLM"
description = """
Parses and summarizes the candidate's professional profile from the extracted CV content, identifying skills, experience, education, and achievements
"""
-inputs = { cv_pages = "Page" }
+inputs = { cv_pages = "Page[]" }
output = "CandidateProfile"
model = "$writing-factual"
system_prompt = """
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/cv_batch.mthds b/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/cv_batch.mthds
index a4b96b23b..317ef96c4 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/cv_batch.mthds
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/cv_batch.mthds
@@ -55,7 +55,7 @@ type = "PipeLLM"
description = """
Parses and summarizes the job requirements from the extracted job offer content, identifying required skills, responsibilities, qualifications, and nice-to-haves
"""
-inputs = { job_offer_pages = "Page" }
+inputs = { job_offer_pages = "Page[]" }
output = "JobRequirements"
model = "$writing-factual"
system_prompt = """
@@ -90,7 +90,7 @@ type = "PipeLLM"
description = """
Parses and summarizes the candidate's professional profile from the extracted CV content, identifying skills, experience, education, and achievements
"""
-inputs = { cv_pages = "Page" }
+inputs = { cv_pages = "Page[]" }
output = "CandidateProfile"
model = "$writing-factual"
system_prompt = """
diff --git a/tests/e2e/pipelex/pipes/pipe_operators/pipe_compose/cv_job_match.mthds b/tests/e2e/pipelex/pipes/pipe_operators/pipe_compose/cv_job_match.mthds
index 158b1e19e..3a4ad2ed0 100644
--- a/tests/e2e/pipelex/pipes/pipe_operators/pipe_compose/cv_job_match.mthds
+++ b/tests/e2e/pipelex/pipes/pipe_operators/pipe_compose/cv_job_match.mthds
@@ -52,7 +52,7 @@ model = "@default-text-from-pdf"
[pipe.analyze_documents]
type = "PipeParallel"
description = "Analyzes both the CV and job offer documents concurrently to extract structured information"
-inputs = { cv_pages = "Page", job_offer_pages = "Page" }
+inputs = { cv_pages = "Page[]", job_offer_pages = "Page[]" }
output = "Composite"
branches = [
{ pipe = "analyze_cv", result = "cv_analysis" },
@@ -65,7 +65,7 @@ type = "PipeLLM"
description = """
Analyzes the CV to extract key information including skills, experience, education, previous roles, and notable achievements
"""
-inputs = { cv_pages = "Page" }
+inputs = { cv_pages = "Page[]" }
output = "CVAnalysis"
model = "$writing-factual"
system_prompt = """
@@ -82,7 +82,7 @@ type = "PipeLLM"
description = """
Analyzes the job offer to extract requirements including required skills, preferred skills, responsibilities, qualifications, and experience level
"""
-inputs = { job_offer_pages = "Page" }
+inputs = { job_offer_pages = "Page[]" }
output = "JobRequirements"
model = "$writing-factual"
system_prompt = """
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_2.mthds b/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_2.mthds
index 686ab86ec..d064c1a0c 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_2.mthds
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_2.mthds
@@ -49,7 +49,7 @@ Provide sentiment (positive/negative/neutral) and confidence score.
[pipe.aggregate_review_results]
type = "PipeLLM"
description = "Aggregate all review analyses into final rating"
-inputs = { sentiment_analyses = "SentimentAnalysis" }
+inputs = { sentiment_analyses = "SentimentAnalysis[]" }
output = "ProductRating"
model = "$testing-text"
prompt = """
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_3.mthds b/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_3.mthds
index d0a460aa0..d3433493a 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_3.mthds
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_3.mthds
@@ -79,7 +79,7 @@ Provide a comprehensive assessment.
[pipe.evaluate_all_ideas]
type = "PipeLLM"
description = "Evaluate and rank all analyzed ideas"
-inputs = { detailed_analyses = "IdeaAnalysis" }
+inputs = { detailed_analyses = "IdeaAnalysis[]" }
output = "IdeaEvaluation"
model = "$testing-text"
prompt = """
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_typed_flow.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_typed_flow.py
index fb7a77b00..ffa6eccfd 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_typed_flow.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_typed_flow.py
@@ -13,7 +13,6 @@ from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.text_content import TextContent
from pipelex.interpreter_hub import get_library_manager
from pipelex.mthds_parsing.parser import MthdsParser
-from pipelex.pipe_controllers.sequence import pipe_sequence as sequence_module
from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
from pipelex.pipeline.exceptions import ValidateBundleError
from pipelex.pipeline.runner import PipelexMTHDSProtocol
@@ -405,28 +404,16 @@ class TestBindingTypedFlow:
assert flow.binding_specs[2].concept.concept_ref == "native.Number"
assert flow.binding_specs[3].concept.concept_ref == "native.Text"
assert flow.binding_specs[3].multiplicity is True
- assert sorted(flow.binding_slots_by_pipe_step[1]) == ["amount"]
- assert sorted(flow.binding_slots_by_pipe_step[4]) == ["amount", "labels", "weight"]
+ binding_slot_names_by_pipe_step = {
+ step_index: sorted(name for name, slot in slots.items() if slot.binding_step_index is not None)
+ for step_index, slots in flow.slots_by_pipe_step.items()
+ }
+ assert binding_slot_names_by_pipe_step == {1: ["amount"], 4: ["amount", "labels", "weight"]}
+ assert sorted(flow.slots_by_pipe_step[4]) == ["amount", "labels", "record", "weight"]
+ assert flow.slots_by_pipe_step[4]["record"].producer_step_index == 1
assert flow.final_slots["record"].stuff_spec is not None
assert flow.final_slots["record"].stuff_spec.concept.concept_ref == "depot_records.Parcel"
- def test_a_sequence_without_a_binding_step_never_builds_its_flow(self, load_empty_library: Callable[[], str], mocker: MockerFixture) -> None:
- """Only binding steps read the typed flow, so validating a sequence of pipe steps alone never builds it."""
- flow_builder_spy = mocker.spy(sequence_module, "build_sequence_typed_flow")
- mthds_content = _REBOUND_ROOT_BUNDLE.replace(
- """ { from = "record.total", result = "amount" },
- { pipe = "weigh_parcel", result = "record" },
- { from = "record.weight", result = "weight" },
- { from = "record.labels", result = "labels" },
- { pipe = "write_line", result = "line" },""",
- """ { pipe = "weigh_parcel", result = "parcel" },
- { pipe = "write_line", result = "line" },""",
- ).replace('inputs = { record = "Invoice" }', 'inputs = { amount = "Number", weight = "Number" }')
-
- _load_sequence(mthds_content=mthds_content, library_id=load_empty_library(), pipe_code="read_records")
-
- assert flow_builder_spy.call_count == 0
-
def test_a_binding_walking_the_replaced_concept_is_refused(self, load_empty_library: Callable[[], str]) -> None:
"""The library refuses the sequence when it loads: the root was replaced by a Parcel, which has no `total`."""
mthds_content = _REBOUND_ROOT_BUNDLE.replace('{ from = "record.weight", result = "weight" }', '{ from = "record.total", result = "weight" }')
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_inter_step_concept_check.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_inter_step_concept_check.py
new file mode 100644
index 000000000..ca190ce1c
--- /dev/null
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_inter_step_concept_check.py
@@ -0,0 +1,934 @@
+from collections.abc import Callable
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.core.pipes.exceptions import PipeValidationError
+from pipelex.interpreter_hub import get_library_manager, get_required_pipe
+from pipelex.mthds_parsing.parser import MthdsParser
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.validation_error_types import PipeValidationErrorType
+from tests.integration.pipelex.pipes.controller.pipe_sequence.unresolved_pipes import hide_pipe_from_sequence_analyses
+
+# The pipes every case's sequence reads from: each case appends one sequence, `flow`, whose steps hand values between them.
+_DEPOT_PIPES = """domain = "depot_steps"
+description = "The steps of a depot's sequences, each reading what an earlier step stored"
+
+[concept.Parcel]
+description = "A parcel received at the depot"
+
+[concept.Parcel.structure]
+weight = { type = "number", description = "The weight, in kilograms", required = true }
+
+[concept.FragileParcel]
+description = "A parcel the depot handles with care"
+refines = "Parcel"
+
+[concept.Invoice]
+description = "An invoice received at the depot"
+
+[concept.Invoice.structure]
+total = { type = "number", description = "The amount due, in euros", required = true }
+
+[pipe.weigh_parcel]
+type = "PipeCompose"
+description = "Writes out a parcel of a given weight"
+inputs = { amount = "Number" }
+output = "Parcel"
+
+[pipe.weigh_parcel.construct]
+weight = { from = "amount.number" }
+
+[pipe.pack_fragile_parcel]
+type = "PipeCompose"
+description = "Writes out a fragile parcel of a given weight"
+inputs = { amount = "Number" }
+output = "FragileParcel"
+
+[pipe.pack_fragile_parcel.construct]
+weight = { from = "amount.number" }
+
+[pipe.make_invoice]
+type = "PipeCompose"
+description = "Writes out an invoice for an amount"
+inputs = { amount = "Number" }
+output = "Invoice"
+
+[pipe.make_invoice.construct]
+total = { from = "amount.number" }
+
+[pipe.label_parcel]
+type = "PipeCompose"
+description = "Writes the label of a parcel"
+inputs = { parcel = "Parcel" }
+output = "Text"
+template = "Parcel of $parcel.weight kilograms"
+
+[pipe.render_note]
+type = "PipeCompose"
+description = "Renders a note written in Markdown"
+inputs = { note = "Markdown" }
+output = "Text"
+template = "Rendered: $note"
+
+[pipe.archive_note]
+type = "PipeCompose"
+description = "Archives a note"
+inputs = { note = "Text" }
+output = "Text"
+template = "Archived: $note"
+
+[pipe.list_three_parcels]
+type = "PipeCompose"
+description = "Writes the manifest of a crate of three parcels"
+inputs = { parcels = "Parcel[3]" }
+output = "Text"
+template = "Crate: $parcels"
+
+[pipe.label_record]
+type = "PipeCompose"
+description = "Writes the label of the parcel on record"
+inputs = { record = "Parcel" }
+output = "Text"
+template = "Parcel of $record.weight kilograms"
+
+[pipe.list_parcels]
+type = "PipeCompose"
+description = "Writes the manifest of several parcels"
+inputs = { parcels = "Parcel[]" }
+output = "Text"
+template = "Manifest: $parcels"
+
+[pipe.settle_invoice]
+type = "PipeCompose"
+description = "Writes the settlement of the invoice on record"
+inputs = { record = "Invoice" }
+output = "Text"
+template = "Settled: $record.total euros"
+
+[pipe.describe_amount]
+type = "PipeCompose"
+description = "Writes out an amount given in words"
+inputs = { amount = "Text" }
+output = "Text"
+template = "Amount: $amount"
+
+[pipe.note_anything]
+type = "PipeCompose"
+description = "Writes a note about whatever is on record"
+inputs = { record = "Anything" }
+output = "Text"
+template = "On record: $record"
+
+[pipe.swap_for_parcel]
+type = "PipeSequence"
+description = "Stores a parcel under the record's name"
+inputs = { amount = "Number" }
+output = "Parcel"
+steps = [
+ { pipe = "weigh_parcel", result = "record" },
+]
+
+[pipe.swap_for_invoice]
+type = "PipeSequence"
+description = "Stores an invoice under the record's name"
+inputs = { amount = "Number" }
+output = "Invoice"
+steps = [
+ { pipe = "make_invoice", result = "record" },
+]
+
+[pipe.swap_record]
+type = "PipeCondition"
+description = "Stores a parcel or an invoice under the record's name, as the mode says"
+inputs = { amount = "Number", mode = "Text" }
+output = "Anything"
+expression = "mode"
+default_outcome = "fail"
+
+[pipe.swap_record.outcomes]
+parcel = "swap_for_parcel"
+invoice = "swap_for_invoice"
+
+[pipe.label_swapped]
+type = "PipeCompose"
+description = "Writes the label of the parcel a swap stored"
+inputs = { swapped = "Parcel" }
+output = "Text"
+template = "Parcel of $swapped.weight kilograms"
+
+[pipe.weigh_anything]
+type = "PipeSequence"
+description = "Weighs a parcel, declaring its output as whatever it is"
+inputs = { amount = "Number" }
+output = "Anything"
+steps = [
+ { pipe = "weigh_parcel", result = "weighed" },
+]
+
+[pipe.render_then_archive]
+type = "PipeSequence"
+description = "Renders a note written in Markdown, then archives it as a text"
+inputs = { note = "Markdown" }
+output = "Text"
+steps = [
+ { pipe = "render_note", result = "rendered" },
+ { pipe = "archive_note", result = "archived" },
+]
+
+[pipe.crate_then_list]
+type = "PipeSequence"
+description = "Writes the manifest of a crate of three parcels, then lists them"
+inputs = { parcels = "Parcel[3]" }
+output = "Text"
+steps = [
+ { pipe = "list_three_parcels", result = "crate" },
+ { pipe = "list_parcels", result = "manifest" },
+]
+
+[pipe.label_given_parcel]
+type = "PipeSequence"
+description = "Writes the label of a parcel it is given"
+inputs = { parcel = "Parcel" }
+output = "Text"
+steps = [
+ { pipe = "label_parcel", result = "given_label" },
+]
+
+[pipe.weigh_record_then_note]
+type = "PipeSequence"
+description = "Weighs a parcel as heavy as the parcel on record, then notes the record, whatever it is"
+inputs = { record = "Parcel" }
+output = "Text"
+steps = [
+ { from = "record.weight", result = "amount" },
+ { pipe = "weigh_parcel", result = "weighed" },
+ { pipe = "note_anything", result = "noted" },
+]
+
+[pipe.archive_record]
+type = "PipeCompose"
+description = "Archives the record, read as a text"
+inputs = { record = "Text" }
+output = "Text"
+template = "Archived: $record"
+
+[pipe.store_note_as_record]
+type = "PipeSequence"
+description = "Stores the archived note under the record's name"
+inputs = { note = "Text" }
+output = "Text"
+steps = [
+ { pipe = "archive_note", result = "record" },
+]
+
+[pipe.maybe_store_note]
+type = "PipeCondition"
+description = "Stores the archived note under the record's name, or leaves the record as it is, as the mode says"
+inputs = { mode = "Text", note = "Text" }
+output = "Text?"
+expression = "mode"
+default_outcome = "fail"
+
+[pipe.maybe_store_note.outcomes]
+store = "store_note_as_record"
+keep = "continue"
+
+[pipe.summarize_records]
+type = "PipeCompose"
+description = "Writes a summary of the records, read as texts"
+inputs = { records = "Text[]" }
+output = "Text"
+template = "Records: $records"
+
+[pipe.store_notes_as_records]
+type = "PipeSequence"
+description = "Stores the archived notes under the records' name, then summarizes them"
+inputs = { notes = "Text[]" }
+output = "Text"
+steps = [
+ { pipe = "archive_note", batch_over = "notes", batch_as = "note", result = "records" },
+ { pipe = "summarize_records", result = "summary" },
+]
+
+[pipe.maybe_store_notes]
+type = "PipeCondition"
+description = "Stores the archived notes under the records' name, or leaves the records as they are, as the mode says"
+inputs = { mode = "Text", notes = "Text[]" }
+output = "Text?"
+expression = "mode"
+default_outcome = "fail"
+
+[pipe.maybe_store_notes.outcomes]
+store = "store_notes_as_records"
+keep = "continue"
+
+[pipe.weigh_anything_as_record]
+type = "PipeSequence"
+description = "Stores a parcel, declared as whatever it is, under the record's name"
+inputs = { amount = "Number" }
+output = "Anything"
+steps = [
+ { pipe = "weigh_anything", result = "record" },
+]
+
+[pipe.store_anything_or_note]
+type = "PipeCondition"
+description = "Stores a parcel declared as whatever it is, or the archived note, under the record's name, as the mode says"
+inputs = { amount = "Number", note = "Text", mode = "Text" }
+output = "Anything"
+expression = "mode"
+default_outcome = "fail"
+
+[pipe.store_anything_or_note.outcomes]
+weigh = "weigh_anything_as_record"
+note = "store_note_as_record"
+
+[pipe.maybe_store_anything_or_note]
+type = "PipeCondition"
+description = "Stores a parcel declared as whatever it is, or the archived note, under the record's name, or leaves the record as it is"
+inputs = { amount = "Number", note = "Text", mode = "Text" }
+output = "Anything?"
+expression = "mode"
+default_outcome = "fail"
+
+[pipe.maybe_store_anything_or_note.outcomes]
+weigh = "weigh_anything_as_record"
+note = "store_note_as_record"
+keep = "continue"
+
+[pipe.store_invoice_or_either]
+type = "PipeCondition"
+description = "Stores an invoice under the record's name, or a parcel declared as whatever it is, or the archived note"
+inputs = { amount = "Number", note = "Text", mode = "Text" }
+output = "Anything"
+expression = "mode"
+default_outcome = "fail"
+
+[pipe.store_invoice_or_either.outcomes]
+invoice = "swap_for_invoice"
+either = "store_anything_or_note"
+"""
+
+
+def _flow(*, inputs: str, output: str, steps: list[str]) -> str:
+ """The sequence `flow`, declaring `inputs` (a TOML inline table's content) and running `steps` in order."""
+ step_lines = "".join(f" {step},\n" for step in steps)
+ return (
+ "\n[pipe.flow]\n"
+ 'type = "PipeSequence"\n'
+ 'description = "Hands values from step to step"\n'
+ f"inputs = {{ {inputs} }}\n"
+ f'output = "{output}"\n'
+ f"steps = [\n{step_lines}]\n"
+ )
+
+
+def _load_flow(*, flow: str, library_id: str) -> PipeSequence:
+ """Load the depot's pipes and `flow` into the library, which validates every pipe, and return `flow`."""
+ blueprint = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=_DEPOT_PIPES + flow, mthds_source="depot.mthds")
+ pipes = get_library_manager().load_from_blueprints(library_id=library_id, blueprints=[blueprint])
+ sequences = [pipe for pipe in pipes if isinstance(pipe, PipeSequence) and pipe.code == "flow"]
+ assert len(sequences) == 1
+ return sequences[0]
+
+
+class TestInterStepConceptCheck:
+ @pytest.mark.parametrize(
+ ("flow", "message"),
+ [
+ pytest.param(
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "weigh_parcel", result = "record" }', '{ pipe = "settle_invoice", result = "note" }'],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'settle_invoice') reads 'record' as 'Invoice', but step 1 (pipe 'weigh_parcel') stores it "
+ "as 'Parcel'. Declare the input as 'Parcel' in pipe 'settle_invoice', or make step 1 (pipe 'weigh_parcel') store a "
+ "'Invoice' under 'record'."
+ ),
+ id="a-concept-another-step-stores",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "weigh_parcel", result = "parcels" }', '{ pipe = "list_parcels", result = "manifest" }'],
+ ),
+ "In pipe 'flow', step 2 (pipe 'list_parcels') reads 'parcels' as 'Parcel[]', but step 1 (pipe 'weigh_parcel') stores it as 'Parcel'.",
+ id="a-list-read-where-a-single-value-is-stored",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amounts = "Number[]"',
+ output="Text",
+ steps=[
+ '{ pipe = "weigh_parcel", batch_over = "amounts", batch_as = "amount", result = "parcel" }',
+ '{ pipe = "label_parcel", result = "label" }',
+ ],
+ ),
+ "In pipe 'flow', step 2 (pipe 'label_parcel') reads 'parcel' as 'Parcel', but step 1 (pipe 'weigh_parcel') stores it as 'Parcel[]'.",
+ id="a-single-value-read-where-a-batch-stores-a-list",
+ ),
+ pytest.param(
+ # The last step reading `amount` sets what the sequence's inputs are checked against, so only the check of each
+ # step against the flow sees the first one.
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=[
+ '{ pipe = "describe_amount", result = "note" }',
+ '{ pipe = "weigh_parcel", result = "parcel" }',
+ '{ pipe = "label_parcel", result = "label" }',
+ ],
+ ),
+ (
+ "In pipe 'flow', step 1 (pipe 'describe_amount') reads 'amount' as 'Text', but the sequence declares it as 'Number'. "
+ "Declare the input as 'Number' in pipe 'describe_amount', or declare 'amount' as 'Text' in the inputs of pipe 'flow'."
+ ),
+ id="a-declared-input-read-as-another-concept",
+ ),
+ pytest.param(
+ # A declared input the last step reading it accepts, but an earlier one reads as a concept refining it.
+ _flow(
+ inputs='note = "Text"',
+ output="Text",
+ steps=['{ pipe = "render_note", result = "rendered" }', '{ pipe = "archive_note", result = "archived" }'],
+ ),
+ (
+ "In pipe 'flow', step 1 (pipe 'render_note') reads 'note' as 'Markdown', but the sequence declares it as 'Text'. "
+ "Declare the input as 'Text' in pipe 'render_note', or declare 'note' as 'Markdown' in the inputs of pipe 'flow'."
+ ),
+ id="a-declared-input-an-earlier-step-reads-as-a-refinement",
+ ),
+ pytest.param(
+ # A declared list the last step reading it accepts, but an earlier one reads as a fixed count.
+ _flow(
+ inputs='parcels = "Parcel[]"',
+ output="Text",
+ steps=['{ pipe = "list_three_parcels", result = "crate" }', '{ pipe = "list_parcels", result = "manifest" }'],
+ ),
+ (
+ "In pipe 'flow', step 1 (pipe 'list_three_parcels') reads 'parcels' as 'Parcel[3]', but the sequence declares it as "
+ "'Parcel[]'. Declare the input as 'Parcel[]' in pipe 'list_three_parcels', or declare 'parcels' as 'Parcel[3]' in the "
+ "inputs of pipe 'flow'."
+ ),
+ id="a-declared-list-an-earlier-step-reads-as-a-fixed-count",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amount = "Number"',
+ output="Text[]",
+ steps=[
+ '{ pipe = "weigh_parcel", result = "parcels" }',
+ '{ pipe = "label_parcel", batch_over = "parcels", batch_as = "parcel", result = "labels" }',
+ ],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'label_parcel') batches over 'parcels', which step 1 (pipe 'weigh_parcel') stores as a "
+ "single 'Parcel', not a list: a batch runs its pipe once per item of a list. Batch over a name holding a list, or run "
+ "the step on the value itself, without `batch_over`."
+ ),
+ id="a-batch-over-a-single-value",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amounts = "Number[]"',
+ output="Text[]",
+ steps=[
+ '{ pipe = "weigh_parcel", batch_over = "amounts", batch_as = "amount", result = "parcels" }',
+ '{ pipe = "settle_invoice", batch_over = "parcels", batch_as = "record", result = "settlements" }',
+ ],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'settle_invoice') batches over 'parcels', which step 1 (pipe 'weigh_parcel') stores "
+ "as 'Parcel[]', but its pipe reads each item, 'record', as 'Invoice'. Declare 'record' as 'Parcel' in pipe "
+ "'settle_invoice', or batch over a list of 'Invoice'."
+ ),
+ id="a-batch-whose-items-are-read-as-another-concept",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amount = "Number", mode = "Text"',
+ output="Text",
+ steps=['{ pipe = "swap_record", result = "swapped" }', '{ pipe = "label_record", result = "label" }'],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'label_record') reads 'record' as 'Parcel', but the values 'record' may hold have different "
+ "specs: outcome 'swap_for_invoice' of pipe 'swap_record' as 'Invoice'; outcome 'swap_for_parcel' of pipe 'swap_record' "
+ "as 'Parcel'."
+ ),
+ id="a-name-the-outcomes-of-a-condition-store-under-different-concepts",
+ ),
+ pytest.param(
+ # The nested sequence's first step reads `note` as `Markdown`, its last as `Text`: its declaration, not its
+ # last reader, is what the calling step is checked against.
+ _flow(
+ inputs='note = "Text"',
+ output="Text",
+ steps=['{ pipe = "render_then_archive", result = "done" }', '{ pipe = "archive_note", result = "archived_again" }'],
+ ),
+ (
+ "In pipe 'flow', step 1 (pipe 'render_then_archive') reads 'note' as 'Markdown', but the sequence declares it as 'Text'. "
+ "Declare the input as 'Text' in pipe 'render_then_archive', or declare 'note' as 'Markdown' in the inputs of pipe 'flow'."
+ ),
+ id="a-nested-sequence-declaring-a-refinement-its-last-step-reads-as-the-parent",
+ ),
+ pytest.param(
+ # The sequence's only reader of `note` is the nested sequence, so its inputs check names the nested declaration.
+ _flow(
+ inputs='note = "Text"',
+ output="Text",
+ steps=['{ pipe = "render_then_archive", result = "done" }'],
+ ),
+ "In pipe 'flow', input 'note' is declared as 'Text' but its step needs 'Markdown'. Update the input to 'Markdown'.",
+ id="a-nested-sequence-called-with-the-parent-of-what-it-declares",
+ ),
+ pytest.param(
+ _flow(
+ inputs='parcels = "Parcel[]"',
+ output="Text",
+ steps=['{ pipe = "crate_then_list", result = "done" }'],
+ ),
+ "In pipe 'flow', input 'parcels' is declared as 'Parcel[]' but its step needs 'Parcel[3]'. Update the input to 'Parcel[3]'.",
+ id="a-nested-sequence-declaring-a-fixed-count-its-last-step-reads-as-a-variable-list",
+ ),
+ pytest.param(
+ # The nested sequence binds `record.weight` from the `Parcel` it declares, then reads `record` as `Anything`.
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "make_invoice", result = "record" }', '{ pipe = "weigh_record_then_note", result = "done" }'],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'weigh_record_then_note') reads 'record' as 'Parcel', but step 1 (pipe 'make_invoice') "
+ "stores it as 'Invoice'."
+ ),
+ id="a-nested-sequence-binding-over-what-it-declares-given-another-concept",
+ ),
+ pytest.param(
+ # A value stored as `Anything` is not known before the run, but its multiplicity is.
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "weigh_anything", result = "parcels" }', '{ pipe = "list_parcels", result = "manifest" }'],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'list_parcels') reads 'parcels' as 'Parcel[]', but step 1 (pipe 'weigh_anything') stores "
+ "it as 'Anything'."
+ ),
+ id="a-single-anything-read-as-a-list",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amount = "Number"',
+ output="Text[]",
+ steps=[
+ '{ pipe = "weigh_anything", result = "parcels" }',
+ '{ pipe = "label_parcel", batch_over = "parcels", batch_as = "parcel", result = "labels" }',
+ ],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'label_parcel') batches over 'parcels', which step 1 (pipe 'weigh_anything') stores as a "
+ "single 'Anything', not a list"
+ ),
+ id="a-batch-over-a-single-anything",
+ ),
+ pytest.param(
+ # The sequence's own declaration is known: declaring `Anything` where a step reads a `Parcel` is refused.
+ _flow(
+ inputs='record = "Anything"',
+ output="Text",
+ steps=['{ pipe = "label_record", result = "label" }'],
+ ),
+ "In pipe 'flow', input 'record' is declared as 'Anything' but its step needs 'Parcel'. Update the input to 'Parcel'.",
+ id="a-declared-anything-read-as-a-concept",
+ ),
+ pytest.param(
+ # Regression: the condition may leave the caller's `Anything` in place, which is a declaration, not a pipe
+ # step's store, so it is checked even among the values the name may hold.
+ _flow(
+ inputs='record = "Anything", mode = "Text", note = "Text"',
+ output="Text",
+ steps=['{ pipe = "maybe_store_note", result = "stored" }', '{ pipe = "archive_record", result = "archived" }'],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'archive_record') reads 'record' as 'Text', but the values 'record' may hold have different "
+ "specs: the value it held before step 1 (pipe 'maybe_store_note'), which the step leaves when it stores nothing, as "
+ "'Anything'; step 1 (pipe 'maybe_store_note'), when it stores a value, as 'Text'."
+ ),
+ id="a-declared-anything-a-condition-may-leave-read-as-a-concept",
+ ),
+ pytest.param(
+ # Regression: as above, for a list batched over.
+ _flow(
+ inputs='records = "Anything[]", mode = "Text", notes = "Text[]"',
+ output="Text[]",
+ steps=[
+ '{ pipe = "maybe_store_notes", result = "stored" }',
+ '{ pipe = "archive_record", batch_over = "records", batch_as = "record", result = "archived" }',
+ ],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'archive_record') batches over 'records', which may hold 'Anything[]', but its pipe reads "
+ "each item, 'record', as 'Text'."
+ ),
+ id="a-declared-anything-list-a-condition-may-leave-batched-over-as-a-concept",
+ ),
+ pytest.param(
+ # A binding's value is no pipe step's store either: binding a declared `Anything` binds an `Anything`.
+ _flow(
+ inputs='given = "Anything", mode = "Text", note = "Text"',
+ output="Text",
+ steps=[
+ '{ from = "given", result = "record" }',
+ '{ pipe = "maybe_store_note", result = "stored" }',
+ '{ pipe = "archive_record", result = "archived" }',
+ ],
+ ),
+ (
+ "In pipe 'flow', step 3 (pipe 'archive_record') reads 'record' as 'Text', but the values 'record' may hold have different "
+ "specs: the value it held before step 2 (pipe 'maybe_store_note'), which the step leaves when it stores nothing, as "
+ "'Anything'; step 2 (pipe 'maybe_store_note'), when it stores a value, as 'Text'."
+ ),
+ id="a-bound-anything-a-condition-may-leave-read-as-a-concept",
+ ),
+ pytest.param(
+ # Regression: the outcomes that store `record` disagree among themselves, and the caller's `Anything`, which the
+ # `continue` outcome leaves, is listed beside them rather than dropped.
+ _flow(
+ inputs='record = "Anything", amount = "Number", mode = "Text", note = "Text"',
+ output="Text",
+ steps=['{ pipe = "maybe_store_anything_or_note", result = "stored" }', '{ pipe = "archive_record", result = "archived" }'],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'archive_record') reads 'record' as 'Text', but the values 'record' may hold have different "
+ "specs: the value it held before step 1 (pipe 'maybe_store_anything_or_note'), which the step leaves when it stores "
+ "nothing, as 'Anything'; outcome 'store_note_as_record' of pipe 'maybe_store_anything_or_note' as 'Text'; outcome "
+ "'weigh_anything_as_record' of pipe 'maybe_store_anything_or_note' as 'Anything'."
+ ),
+ id="a-declared-anything-a-condition-may-leave-beside-outcomes-that-disagree",
+ ),
+ pytest.param(
+ # Regression: one outcome stores an `Invoice`, the other a value whose own outcomes disagree, and the `Invoice` is
+ # listed beside their values rather than dropped.
+ _flow(
+ inputs='amount = "Number", note = "Text", mode = "Text"',
+ output="Text",
+ steps=['{ pipe = "store_invoice_or_either", result = "stored" }', '{ pipe = "archive_record", result = "archived" }'],
+ ),
+ (
+ "In pipe 'flow', step 2 (pipe 'archive_record') reads 'record' as 'Text', but the values 'record' may hold have different "
+ "specs: outcome 'store_note_as_record' of pipe 'store_anything_or_note' as 'Text'; outcome 'weigh_anything_as_record' of "
+ "pipe 'store_anything_or_note' as 'Anything'; outcome 'swap_for_invoice' of pipe 'store_invoice_or_either' as 'Invoice'."
+ ),
+ id="an-outcome-storing-another-concept-beside-one-whose-values-disagree",
+ ),
+ ],
+ )
+ def test_a_step_reading_what_the_flow_does_not_carry_is_refused(self, load_empty_library: Callable[[], str], flow: str, message: str) -> None:
+ """A step whose pipe reads a name as a concept or a multiplicity the flow does not carry there is refused at validation."""
+ with pytest.raises(PipeValidationError) as exc_info:
+ _load_flow(flow=flow, library_id=load_empty_library())
+
+ assert exc_info.value.error_type == PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH
+ assert exc_info.value.pipe_code == "flow"
+ assert message in str(exc_info.value)
+
+ @pytest.mark.parametrize(
+ "flow",
+ [
+ pytest.param(
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "pack_fragile_parcel", result = "parcel" }', '{ pipe = "label_parcel", result = "label" }'],
+ ),
+ id="a-refinement-read-as-its-parent",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amounts = "Number[]"',
+ output="Text",
+ steps=[
+ '{ pipe = "weigh_parcel", batch_over = "amounts", batch_as = "amount", result = "parcels" }',
+ '{ pipe = "list_parcels", result = "manifest" }',
+ ],
+ ),
+ id="a-batched-result-read-as-a-list",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amounts = "Number[]"',
+ output="Text",
+ steps=[
+ '{ pipe = "weigh_parcel", batch_over = "amounts", batch_as = "amount", result = "parcels", nb_output = 1 }',
+ '{ pipe = "list_parcels", result = "manifest" }',
+ ],
+ ),
+ id="a-batched-result-asking-for-one-output-read-as-a-list",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amounts = "Number[]"',
+ output="Text",
+ steps=[
+ '{ pipe = "weigh_parcel", batch_over = "amounts", batch_as = "amount", result = "parcels" }',
+ '{ pipe = "pack_fragile_parcel", batch_over = "amounts", batch_as = "amount", result = "fragile_parcels" }',
+ '{ pipe = "label_parcel", batch_over = "fragile_parcels", batch_as = "parcel", result = "labels" }',
+ '{ pipe = "list_parcels", result = "manifest" }',
+ ],
+ ),
+ id="a-batch-over-a-list-of-refinements",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "weigh_parcel", result = "record" }', '{ pipe = "note_anything", result = "note" }'],
+ ),
+ id="a-name-read-as-anything",
+ ),
+ pytest.param(
+ _flow(
+ inputs='record = "Invoice", amount = "Number"',
+ output="Text",
+ steps=[
+ '{ pipe = "settle_invoice", result = "settlement" }',
+ '{ pipe = "weigh_parcel", result = "record" }',
+ '{ pipe = "label_record", result = "label" }',
+ ],
+ ),
+ id="a-declared-input-replaced-by-what-a-step-stores",
+ ),
+ pytest.param(
+ # The last step reading `note` reads a `Text`, so only the check of each step against the flow accepts a
+ # declaration other than the last reader's need.
+ _flow(
+ inputs='note = "Markdown"',
+ output="Text",
+ steps=['{ pipe = "render_note", result = "rendered" }', '{ pipe = "archive_note", result = "archived" }'],
+ ),
+ id="a-declared-input-read-as-a-refinement-then-as-its-parent",
+ ),
+ pytest.param(
+ _flow(
+ inputs='parcels = "Parcel[3]"',
+ output="Text",
+ steps=['{ pipe = "list_three_parcels", result = "crate" }', '{ pipe = "list_parcels", result = "manifest" }'],
+ ),
+ id="a-declared-fixed-count-read-as-it-then-as-a-variable-list",
+ ),
+ pytest.param(
+ _flow(
+ inputs='note = "Markdown"',
+ output="Text",
+ steps=['{ pipe = "render_then_archive", result = "done" }'],
+ ),
+ id="a-nested-sequence-called-with-what-it-declares",
+ ),
+ pytest.param(
+ _flow(
+ inputs='parcels = "Parcel[3]"',
+ output="Text",
+ steps=['{ pipe = "crate_then_list", result = "done" }'],
+ ),
+ id="a-nested-sequence-called-with-the-fixed-count-it-declares",
+ ),
+ pytest.param(
+ _flow(
+ inputs='parcel = "FragileParcel"',
+ output="Text",
+ steps=['{ pipe = "label_given_parcel", result = "done" }'],
+ ),
+ id="a-nested-sequence-declaring-a-parent-given-a-declared-refinement",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "pack_fragile_parcel", result = "parcel" }', '{ pipe = "label_given_parcel", result = "done" }'],
+ ),
+ id="a-nested-sequence-declaring-a-parent-given-a-stored-refinement",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "weigh_parcel", result = "record" }', '{ pipe = "weigh_record_then_note", result = "done" }'],
+ ),
+ id="a-nested-sequence-binding-over-what-it-declares-given-it",
+ ),
+ pytest.param(
+ # A condition whose outcomes produce different concepts declares `Anything`: what it stores is not known
+ # before the run, so a step reading it as a concept is assumed to get it, as from a pipe that does not resolve.
+ _flow(
+ inputs='amount = "Number", mode = "Text"',
+ output="Text",
+ steps=['{ pipe = "swap_record", result = "swapped" }', '{ pipe = "label_swapped", result = "label" }'],
+ ),
+ id="a-condition-result-declared-anything-read-as-a-concept",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "weigh_anything", result = "parcel" }', '{ pipe = "label_parcel", result = "label" }'],
+ ),
+ id="a-nested-sequence-result-declared-anything-read-as-a-concept",
+ ),
+ pytest.param(
+ _flow(
+ inputs='amounts = "Number[]"',
+ output="Text[]",
+ steps=[
+ '{ pipe = "weigh_anything", batch_over = "amounts", batch_as = "amount", result = "parcels" }',
+ '{ pipe = "label_parcel", batch_over = "parcels", batch_as = "parcel", result = "labels" }',
+ ],
+ ),
+ id="a-batch-over-a-list-of-anything-whose-items-are-read-as-a-concept",
+ ),
+ pytest.param(
+ # The outcomes store `record` as `Anything` and as `Text`: both are pipe stores, so a step reading it as `Text`
+ # is assumed to get one.
+ _flow(
+ inputs='amount = "Number", note = "Text", mode = "Text"',
+ output="Text",
+ steps=['{ pipe = "store_anything_or_note", result = "stored" }', '{ pipe = "archive_record", result = "archived" }'],
+ ),
+ id="outcomes-storing-anything-and-a-concept-read-as-the-concept",
+ ),
+ pytest.param(
+ # The condition may leave the `Anything` an earlier pipe step stored, which is a pipe step's store too.
+ _flow(
+ inputs='amount = "Number", mode = "Text", note = "Text"',
+ output="Text",
+ steps=[
+ '{ pipe = "weigh_anything", result = "record" }',
+ '{ pipe = "maybe_store_note", result = "stored" }',
+ '{ pipe = "archive_record", result = "archived" }',
+ ],
+ ),
+ id="a-stored-anything-a-condition-may-leave-read-as-a-concept",
+ ),
+ ],
+ )
+ def test_a_step_reading_what_the_flow_carries_validates(self, load_empty_library: Callable[[], str], flow: str) -> None:
+ """A refined concept satisfies a step reading its parent, a batched step always stores a list, a step calling a nested
+ sequence is checked against what the nested sequence declares, and a value a step stores as `Anything` satisfies any concept.
+ """
+ sequence = _load_flow(flow=flow, library_id=load_empty_library())
+
+ assert sequence.code == "flow"
+
+ @pytest.mark.parametrize(
+ ("flow", "error_type", "message", "expected_inputs"),
+ [
+ pytest.param(
+ _flow(
+ inputs='note = "Markdown", remark = "Text"',
+ output="Text",
+ steps=['{ pipe = "render_note", result = "rendered" }', '{ pipe = "archive_note", result = "archived" }'],
+ ),
+ PipeValidationErrorType.EXTRANEOUS_INPUT_VARIABLE,
+ "Extraneous input 'remark' found in the inputs of pipe flow",
+ {"note": "Markdown"},
+ id="a-declared-input-no-step-reads",
+ ),
+ pytest.param(
+ _flow(
+ inputs='note = "Markdown"',
+ output="Text",
+ steps=[
+ '{ pipe = "render_note", result = "rendered" }',
+ '{ pipe = "archive_note", result = "archived" }',
+ '{ pipe = "label_record", result = "record_label" }',
+ ],
+ ),
+ PipeValidationErrorType.MISSING_INPUT_VARIABLE,
+ "Required variable 'record' is not in the inputs of pipe 'flow'.",
+ {"note": "Markdown", "record": "Parcel"},
+ id="a-needed-input-not-declared",
+ ),
+ ],
+ )
+ def test_the_declared_names_are_checked_beside_a_declaration_every_step_accepts(
+ self,
+ load_empty_library: Callable[[], str],
+ flow: str,
+ error_type: PipeValidationErrorType,
+ message: str,
+ expected_inputs: dict[str, str],
+ ) -> None:
+ """A declaration every step reading it accepts leaves the sequence's inputs checked for missing and extraneous names, and
+ the inputs a fix would write keep that declaration rather than the need of the last step reading it.
+ """
+ with pytest.raises(PipeValidationError) as exc_info:
+ _load_flow(flow=flow, library_id=load_empty_library())
+
+ assert exc_info.value.error_type == error_type
+ assert exc_info.value.pipe_code == "flow"
+ assert message in str(exc_info.value)
+ assert exc_info.value.expected_inputs == expected_inputs
+
+ def test_a_name_an_unresolved_pipe_stores_is_assumed_to_deliver(self, load_empty_library: Callable[[], str], mocker: MockerFixture) -> None:
+ """A pipe that does not resolve at validation, as a dependency not loaded yet, gives the flow nothing to check a reader
+ against, so the step reading what it stores is assumed to get what it reads, as every other check of the sequence assumes.
+ """
+ hide_pipe_from_sequence_analyses(mocker=mocker, pipe_code="weigh_parcel")
+ flow = _flow(
+ inputs='amount = "Number"',
+ output="Text",
+ steps=['{ pipe = "weigh_parcel", result = "record" }', '{ pipe = "settle_invoice", result = "note" }'],
+ )
+
+ sequence = _load_flow(flow=flow, library_id=load_empty_library())
+
+ record_slot = sequence.build_typed_flow().slots_by_pipe_step[1]["record"]
+ assert record_slot.stuff_spec is None
+ assert record_slot.disagreement is None
+ assert record_slot.producer_step_index == 0
+
+ def test_a_binding_over_a_value_stored_as_anything_is_refused(self, load_empty_library: Callable[[], str]) -> None:
+ """A value stored as `Anything` satisfies a step reading it as a concept, but a binding cannot walk a path through it."""
+ flow = _flow(
+ inputs='amount = "Number", mode = "Text"',
+ output="Number",
+ steps=['{ pipe = "swap_record", result = "swapped" }', '{ from = "swapped.weight", result = "weight" }'],
+ )
+
+ with pytest.raises(PipeValidationError) as exc_info:
+ _load_flow(flow=flow, library_id=load_empty_library())
+
+ assert exc_info.value.error_type == PipeValidationErrorType.BINDING_PATH_UNRESOLVED
+ assert exc_info.value.pipe_code == "flow"
+ assert (
+ "In pipe 'flow', the binding step { from = \"swapped.weight\", result = \"weight\" } cannot be derived. Cannot bind 'swapped.weight'"
+ in str(exc_info.value)
+ )
+ assert "which is structureless by definition, so the segment 'weight' has no structure to walk" in str(exc_info.value)
+
+ @pytest.mark.parametrize(
+ ("pipe_code", "variable_name", "expected_spec"),
+ [
+ pytest.param("render_then_archive", "note", "Markdown", id="a-refinement-its-last-step-reads-as-the-parent"),
+ pytest.param("crate_then_list", "parcels", "Parcel[3]", id="a-fixed-count-its-last-step-reads-as-a-variable-list"),
+ pytest.param("weigh_record_then_note", "record", "Parcel", id="a-binding-root-its-last-step-reads-as-anything"),
+ ],
+ )
+ def test_a_sequence_needs_what_it_declares_when_every_step_accepts_it(
+ self, load_empty_library: Callable[[], str], pipe_code: str, variable_name: str, expected_spec: str
+ ) -> None:
+ """What a sequence needs from its caller under a declared input every step accepts is its declaration, not the need of the
+ last step reading it, so every caller, a sequence, a condition or a parallel above it, is held to the declaration.
+ """
+ _load_flow(
+ flow=_flow(inputs='amount = "Number"', output="Parcel", steps=['{ pipe = "weigh_parcel", result = "parcel" }']),
+ library_id=load_empty_library(),
+ )
+ nested_sequence = get_required_pipe(pipe_code=f"depot_steps.{pipe_code}")
+
+ needed_spec = nested_sequence.needed_inputs().root[variable_name]
+
+ assert needed_spec.to_bundle_representation(relative_to_domain="depot_steps") == expected_spec
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_output_multiplicity.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_output_multiplicity.py
index e29c4906e..fdf2e3b68 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_output_multiplicity.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_output_multiplicity.py
@@ -12,6 +12,7 @@ from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint
from pipelex.pipe_machinery.pipe_factory import PipeFactory
from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+from pipelex.validation_error_types import PipeValidationErrorType
class TestPipeSequenceOutputMultiplicity:
@@ -453,19 +454,25 @@ class TestPipeSequenceOutputMultiplicity:
error_message = str(exc_info.value).lower()
assert "multiplicity" in error_message, f"Error should mention multiplicity mismatch, got: {exc_info.value}"
- def test_batch_over_with_nb_output_fixed_multiplicity(
+ @pytest.mark.parametrize(
+ ("sequence_output", "nb_output", "is_refused"),
+ [
+ pytest.param("Text[6]", 6, True, id="a-fixed-count-output-is-refused"),
+ pytest.param("Text", 1, True, id="a-single-output-is-refused"),
+ pytest.param("Text[]", 6, False, id="a-variable-list-output-validates"),
+ ],
+ )
+ def test_batch_over_with_nb_output_yields_a_variable_list(
self,
load_empty_library: Callable[[], str],
+ sequence_output: str,
+ nb_output: int,
+ is_refused: bool,
):
- """Test that when last step has batch_over/batch_as AND nb_output, multiplicity uses nb_output.
+ """A batched last step yields the variable list of its branches' results, whatever `nb_output` it asks for.
- When a step has batch_over, batch_as, AND nb_output=3, the effective output multiplicity
- should be 3, not True.
-
- Creates:
- - A PipeLLM with output="Text" (single output)
- - A PipeSequence with output="Text[3]" where last step has batch_over/batch_as and nb_output=3
- The validation should pass because nb_output=3 overrides to give multiplicity=3.
+ The run stores one result per item of the batched list, so the sequence's output must be a variable list: a fixed
+ count, or the single form `nb_output = 1` spells, is refused as the typed flow reads the step (`SubPipe.result_spec`).
"""
load_empty_library()
@@ -490,14 +497,14 @@ class TestPipeSequenceOutputMultiplicity:
sequence_blueprint = PipeSequenceBlueprint(
description="Sequence that batches with fixed output count",
inputs={"cvs": "Document[]"}, # Multiple PDFs input
- output="Text[6]", # Fixed multiplicity - should match nb_output
+ output=sequence_output,
steps=[
SubPipeBlueprint(
pipe=f"{domain}.process_cv_fixed",
result="match_analyses",
batch_over="cvs",
batch_as="cv_pdf",
- nb_output=6, # Fixed output count
+ nb_output=nb_output,
),
],
)
@@ -508,8 +515,14 @@ class TestPipeSequenceOutputMultiplicity:
)
get_pipe_library().add_new_pipe(sequence_pipe)
- # Validation should pass - nb_output=3 gives multiplicity=3
- sequence_pipe.validate_with_libraries()
+ if not is_refused:
+ sequence_pipe.validate_with_libraries()
+ return
+ with pytest.raises(PipeValidationError) as exc_info:
+ sequence_pipe.validate_with_libraries()
+
+ assert exc_info.value.error_type == PipeValidationErrorType.INADEQUATE_OUTPUT_MULTIPLICITY
+ assert f"declares its output as '{sequence_output}', but its last step 'process_cv_fixed' yields 'Text[]'" in str(exc_info.value)
def test_nb_output_only_fixed_multiplicity(
self,
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_sequence_flow_memo.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_sequence_flow_memo.py
new file mode 100644
index 000000000..773585316
--- /dev/null
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_sequence_flow_memo.py
@@ -0,0 +1,222 @@
+import copy
+import pickle # ruff: ignore[suspicious-pickle-import] - the test pickles a sequence it built itself
+from collections.abc import Callable
+from typing import TYPE_CHECKING
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.core.stuffs.number_content import NumberContent
+from pipelex.interpreter_hub import get_library_manager, get_pipe_library, scoped_current_library
+from pipelex.mthds_parsing.parser import MthdsParser
+from pipelex.pipe_controllers.sequence import pipe_sequence as sequence_module
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
+from pipelex.pipeline.runner import PipelexMTHDSProtocol
+from pipelex.system.pipe_run_mode import PipeRunMode
+
+if TYPE_CHECKING:
+ from mthds.protocol.pipeline_inputs import PipelineInputs
+
+# A sequence reading each parcel's weight through a binding step, run once per parcel by the sequence that calls it.
+_WEIGHING_BUNDLE = """domain = "depot_weighing"
+description = "Reading the weight of every parcel received at the depot"
+main_pipe = "read_all_weights"
+
+[concept.Parcel]
+description = "A parcel received at the depot"
+
+[concept.Parcel.structure]
+weight = { type = "number", description = "The weight, in kilograms", required = true }
+
+[pipe.weigh_parcel]
+type = "PipeCompose"
+description = "Writes out a parcel of a given weight"
+inputs = { amount = "Number" }
+output = "Parcel"
+
+[pipe.weigh_parcel.construct]
+weight = { from = "amount.number" }
+
+[pipe.read_weight]
+type = "PipeSequence"
+description = "Weighs a parcel, then binds its weight"
+inputs = { amount = "Number" }
+output = "Number"
+steps = [
+ { pipe = "weigh_parcel", result = "parcel" },
+ { from = "parcel.weight", result = "weight" },
+]
+
+[pipe.read_all_weights]
+type = "PipeSequence"
+description = "Reads the weight of a parcel for each amount"
+inputs = { amounts = "Number[]" }
+output = "Number[]"
+steps = [
+ { pipe = "read_weight", batch_over = "amounts", batch_as = "amount", result = "weights" },
+]
+
+[pipe.read_one_weight]
+type = "PipeSequence"
+description = "Reads the weight of one parcel through the sequence that reads it"
+inputs = { amount = "Number" }
+output = "Number"
+steps = [
+ { pipe = "read_weight", result = "weight" },
+]
+"""
+
+# A pipe of the same code storing another concept, loaded into a second library beside the first library's sequence.
+_OTHER_WEIGHING_BUNDLE = """domain = "depot_weighing"
+description = "A depot where weighing a parcel writes out its invoice"
+
+[concept.Invoice]
+description = "An invoice for a parcel received at the depot"
+
+[concept.Invoice.structure]
+weight = { type = "number", description = "The weight billed, in kilograms", required = true }
+
+[pipe.weigh_parcel]
+type = "PipeCompose"
+description = "Writes out the invoice of a parcel of a given weight"
+inputs = { amount = "Number" }
+output = "Invoice"
+
+[pipe.weigh_parcel.construct]
+weight = { from = "amount.number" }
+"""
+
+
+def _load_pipes(*, mthds_content: str, library_id: str) -> dict[str, PipeAbstract]:
+ blueprint = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=mthds_content, mthds_source="weighing.mthds")
+ pipes = get_library_manager().load_from_blueprints(library_id=library_id, blueprints=[blueprint])
+ return {pipe.code: pipe for pipe in pipes}
+
+
+def _sequence(*, pipes: dict[str, PipeAbstract], pipe_code: str) -> PipeSequence:
+ sequence = pipes[pipe_code]
+ assert isinstance(sequence, PipeSequence)
+ return sequence
+
+
+def _shallow_model_copy(sequence: PipeSequence) -> PipeSequence:
+ return sequence.model_copy()
+
+
+def _deep_model_copy(sequence: PipeSequence) -> PipeSequence:
+ return sequence.model_copy(deep=True)
+
+
+def _pickled_copy(sequence: PipeSequence) -> PipeSequence:
+ copied = pickle.loads(pickle.dumps(sequence)) # ruff: ignore[suspicious-pickle-usage] - a sequence this test pickled itself
+ assert isinstance(copied, PipeSequence)
+ return copied
+
+
+class TestSequenceFlowMemo:
+ def test_a_flow_is_built_once_per_state_of_the_library(self, load_empty_library: Callable[[], str], mocker: MockerFixture) -> None:
+ """Validating every sequence builds each flow once, and the walks that follow read it, until the library changes."""
+ flow_builder_spy = mocker.spy(sequence_module, "build_sequence_typed_flow")
+ pipes = _load_pipes(mthds_content=_WEIGHING_BUNDLE, library_id=load_empty_library())
+ read_weight = _sequence(pipes=pipes, pipe_code="read_weight")
+ read_one_weight = _sequence(pipes=pipes, pipe_code="read_one_weight")
+
+ assert flow_builder_spy.call_count == 3
+
+ read_weight.build_typed_flow()
+ read_one_weight.needed_inputs()
+ read_weight.memory_writes(visited_pipes={read_one_weight.visit_key})
+ read_one_weight.analyze_taint()
+
+ assert flow_builder_spy.call_count == 3
+
+ get_pipe_library().remove_pipes_by_refs(pipe_refs=[pipes["read_all_weights"].pipe_ref])
+ flow = read_weight.build_typed_flow()
+
+ assert flow_builder_spy.call_count == 4
+ assert flow.binding_specs[1].concept.concept_ref == "native.Number"
+
+ @pytest.mark.asyncio(loop_scope="class")
+ async def test_a_run_reads_the_flow_kept_since_validation(self, mocker: MockerFixture) -> None:
+ """A sequence run once per item of a batch derives its binding from the flow its library validated, built once."""
+ flow_builder_spy = mocker.spy(sequence_module, "build_sequence_typed_flow")
+ inputs: PipelineInputs = {"amounts": {"concept": "native.Number", "content": [{"number": 2.5}, {"number": 4}, {"number": 7}]}}
+
+ response = await PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.LIVE).execute(mthds_contents=[_WEIGHING_BUNDLE], inputs=inputs)
+
+ assert [item.number for item in response.pipe_output.main_stuff_as_items(item_type=NumberContent)] == [2.5, 4, 7]
+ assert flow_builder_spy.call_count == 3
+
+ def test_a_flow_never_crosses_into_a_library_where_its_pipes_resolve_differently(self, load_empty_library: Callable[[], str]) -> None:
+ """The same sequence, held by two libraries whose pipe of one code stores different concepts, types its flow in each."""
+ first_library_id = load_empty_library()
+ read_weight = _sequence(pipes=_load_pipes(mthds_content=_WEIGHING_BUNDLE, library_id=first_library_id), pipe_code="read_weight")
+ parcel_spec = read_weight.build_typed_flow().final_slots["parcel"].stuff_spec
+ assert parcel_spec is not None
+ assert parcel_spec.concept.concept_ref == "depot_weighing.Parcel"
+
+ library_manager = get_library_manager()
+ second_library_id, second_library = library_manager.open_library()
+ try:
+ _load_pipes(mthds_content=_OTHER_WEIGHING_BUNDLE, library_id=second_library_id)
+ second_library.pipe_library.add_new_pipe(read_weight)
+ with scoped_current_library(library_id=second_library_id):
+ invoice_spec = read_weight.build_typed_flow().final_slots["parcel"].stuff_spec
+ finally:
+ library_manager.teardown(library_id=second_library_id)
+
+ assert invoice_spec is not None
+ assert invoice_spec.concept.concept_ref == "depot_weighing.Invoice"
+ parcel_spec_again = read_weight.build_typed_flow().final_slots["parcel"].stuff_spec
+ assert parcel_spec_again is not None
+ assert parcel_spec_again.concept.concept_ref == "depot_weighing.Parcel"
+
+ @pytest.mark.parametrize(
+ "make_copy",
+ [
+ pytest.param(_shallow_model_copy, id="a-shallow-copy"),
+ pytest.param(_deep_model_copy, id="a-deep-copy"),
+ pytest.param(copy.deepcopy, id="a-copy-module-deep-copy"),
+ pytest.param(_pickled_copy, id="a-pickled-copy"),
+ ],
+ )
+ def test_a_copy_of_a_sequence_builds_its_own_flow(
+ self, load_empty_library: Callable[[], str], mocker: MockerFixture, make_copy: Callable[[PipeSequence], PipeSequence]
+ ) -> None:
+ """A copy may change the steps it was copied with, so it never reads the flow its original built."""
+ read_weight = _sequence(pipes=_load_pipes(mthds_content=_WEIGHING_BUNDLE, library_id=load_empty_library()), pipe_code="read_weight")
+ read_weight.build_typed_flow()
+ flow_builder_spy = mocker.spy(sequence_module, "build_sequence_typed_flow")
+
+ copied = make_copy(read_weight)
+ copied.build_typed_flow()
+ copied.build_typed_flow()
+ read_weight.build_typed_flow()
+
+ assert flow_builder_spy.call_count == 1
+
+ def test_a_memo_never_changes_what_a_sequence_equals(self, load_empty_library: Callable[[], str]) -> None:
+ """Two sequences of one definition are equal whatever each has derived, and a memo never makes a comparison recurse."""
+ first_pipes = _load_pipes(mthds_content=_WEIGHING_BUNDLE, library_id=load_empty_library())
+ first_read_weight = _sequence(pipes=first_pipes, pipe_code="read_weight")
+ first_read_weight.build_typed_flow()
+ never_validated = copy.deepcopy(first_read_weight)
+
+ library_manager = get_library_manager()
+ second_library_id, _ = library_manager.open_library()
+ try:
+ second_pipes = _load_pipes(mthds_content=_WEIGHING_BUNDLE, library_id=second_library_id)
+ second_read_weight = _sequence(pipes=second_pipes, pipe_code="read_weight")
+ with scoped_current_library(library_id=second_library_id):
+ second_read_weight.build_typed_flow()
+ finally:
+ library_manager.teardown(library_id=second_library_id)
+
+ assert first_read_weight is not second_read_weight
+ assert first_read_weight == second_read_weight
+ assert second_read_weight == first_read_weight
+ assert first_read_weight == never_validated
+ assert never_validated == second_read_weight
+ assert first_read_weight != _sequence(pipes=first_pipes, pipe_code="read_one_weight")
+ assert second_read_weight != _sequence(pipes=first_pipes, pipe_code="read_all_weights")
diff --git a/tests/integration/pipelex/pipes/pipelines/multiplicity.mthds b/tests/integration/pipelex/pipes/pipelines/multiplicity.mthds
index 87288cc30..eec8230ae 100644
--- a/tests/integration/pipelex/pipes/pipelines/multiplicity.mthds
+++ b/tests/integration/pipelex/pipes/pipelines/multiplicity.mthds
@@ -50,14 +50,27 @@ Imagine a fantasy scene including the following products of nature:
Keep it short and concise, just one sentence.
"""
+[pipe.imagine_fantasy_scene_including_several_products_of_nature]
+type = "PipeLLM"
+description = "Imagine a fantasy scene including several products of nature"
+inputs = { products_of_nature = "ProductOfNature[]" }
+output = "Text"
+prompt = """
+Imagine a fantasy scene including the following products of nature:
+
+@products_of_nature
+
+Keep it short and concise, just one sentence.
+"""
+
[pipe.imagine_nature_scene_of_original_power_rangers_colors]
type = "PipeSequence"
description = "Imagine nature scenes of Power Rangers colors"
output = "Text"
steps = [
- { pipe = "original_power_ranger_colors", result = "color" },
- { pipe = "imagine_nature_product", result = "product_of_nature" },
- { pipe = "imagine_fantasy_scene_including_products_of_nature" },
+ { pipe = "original_power_ranger_colors", result = "colors" },
+ { pipe = "imagine_nature_product", batch_over = "colors", batch_as = "color", result = "products_of_nature" },
+ { pipe = "imagine_fantasy_scene_including_several_products_of_nature" },
]
[pipe.imagine_nature_scene_of_alltime_power_rangers_colors]