Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Batch-over-binding · L-261004-9951e1 (#1349)
Louis Choquela23950bcb9f5
diff --git a/.pipelex/plxt.toml b/.pipelex/plxt.tomlindex d8c625617..afc14b3a8 100644--- a/.pipelex/plxt.toml+++ b/.pipelex/plxt.toml@@ -52,6 +52,8 @@ exclude = [# entry-validation gate runs the real validation engine over every entry in CI and is red both when# an invalid entry fails differently and when it accidentally validates."**/mthds_corpus/entries/invalid_binding_step_batch_fields/*.mthds",+ "**/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_branch/*.mthds",+ "**/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_malformed/*.mthds","**/mthds_corpus/entries/invalid_binding_step_malformed_path/*.mthds","**/mthds_corpus/entries/invalid_binding_step_parallel_branch/*.mthds","**/mthds_corpus/entries/invalid_binding_step_pipe_and_from/*.mthds",diff --git a/.test_durations b/.test_durationsindex 9bcc07bfd..d80935b69 100644--- a/.test_durations+++ b/.test_durations@@ -993,7 +993,6 @@"tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_compaction.py::TestPipeBatchCompaction::test_absent_branch_results_are_dropped": 3.534,"tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_compaction.py::TestPipeBatchCompaction::test_all_branches_absent_yields_empty_list": 1.656,"tests/integration/pipelex/pipes/controller/pipe_batch/test_pipe_batch_simple.py::TestPipeBatchSimple::test_simple_batch_processing": 4.431,- "tests/integration/pipelex/pipes/controller/pipe_batch_dotted_path/test_dotted_batch.py::TestDottedBatchOver::test_dotted_batch_resolves_nested_list": 4.7059,"tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_complex.py::TestPipeConditionComplex::test_business_finance_routing": 6.7838,"tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_complex.py::TestPipeConditionComplex::test_complex_pipeline_dry_run_missing_inputs": 0.5977,"tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_complex.py::TestPipeConditionComplex::test_complex_pipeline_dry_run_success": 6.3922,diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d56cfed1c..c36becd50 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,7 +4,7 @@### Added-- **The binding step**: a PipeSequence step `{ from = "invoice.total", result = "total_amount" }` binds a deep copy of the value at a path in working memory to a new name, as a fresh stuff whose concept is derived from the declared structures the path walks (`Number` for a `number` field, `Image` for `page.page_view`, `X[]` for a path crossing a list, which is mapped and flattened), and a later step reads it under that plain name. A path reaching a field that holds nothing records a `DECLARED_ABSENT` absence and an absent root skips the binding, a step reading the result is checked against the derived concept, a step asking a sequence that ends with a binding for a count of outputs the binding does not bind is refused as `inadequate_output_multiplicity`, a binding derived as a single value that finds a list at run time fails rather than keep one item, and the execution graph draws the binding as a node of the new `NodeKind` `binding`. A malformed step is refused as the new `PipeValidationErrorType` value `binding_step_invalid`, which the MTHDS JSON Schema refuses too, through a `oneOf` of the two step shapes with the path grammar on `from` and the plain-name pattern on `result`, and a path the structures cannot walk, into a concept declared with a description alone included, as `binding_path_unresolved`. The typed flow types a batched step's result as `X[]` and a dependency package's concept by the package's own definitions, and a path may end on a field holding `native.Anything`, whose value is stored as an `Anything` input is. What a nested sequence, a condition's outcome or an `add_each_output` parallel stores in the caller's memory is read alike by the caller's typed flow, needed inputs and absence analysis, for a pipe step reading it as for a binding: a name it always stores types the root, carries its absence and is no input of the caller, so a sequence that declared such a name only to pass validation now has an extraneous input, a name only some outcomes store is still an input, and a nested sequence or condition skipped for an absent plain input records an absence for every name it always stores. A binding whose root may hold values of different concepts, as when the outcomes of a condition store it under different concepts, is refused before the run as `binding_path_unresolved`, the message naming each outcome and the concept it stores, while one whose root a pipe unresolved at validation stores is derived when it runs and checked then against the sequence's output and the input of the step reading it, a mismatch being a run error; the MTHDS Test Corpus covers both refusals and the `feature.binding_step` tag, and its offline-tier entries now run with their `inputs.json` in the test suite.+- **The binding step**: a PipeSequence step `{ from = "invoice.total", result = "total_amount" }` binds a deep copy of the value at a path in working memory to a new name, as a fresh stuff whose concept is derived from the declared structures the path walks, a path crossing a list binding one flattened `X[]`, and a later step reads it under that plain name. The sequence checks a step reading a binding, or a binding ending it, against the derived concept and multiplicity, a path reaching nothing records an absence under the optionality model, and the execution graph draws each binding as a node of the new `NodeKind` `binding`. A malformed step is refused as the new `PipeValidationErrorType` value `binding_step_invalid`, which the MTHDS JSON Schema refuses too, and a path the declared structures cannot walk as `binding_path_unresolved`; the MTHDS Test Corpus covers both under the `feature.binding_step` tag, and its offline-tier entries now run with their `inputs.json` in the test suite.- **`pipelex fix bundle` deletes a dotted input key beside its root**: the new rule `delete-redundant-dotted-input` removes a key such as `"page.page_view" = "Image"` from a pipe's `inputs` when the same table declares its root `page`, which already supplies every field a template reads through it. The fix is safe, and applied, only when the root's declared concept holds the field the key names and the root keeps the presence marker and multiplicity the table gave it: a root declared with a concept that lacks the field, such as `data = "Number"` beside `"data.text" = "Text"`, or a key that set the markers otherwise, such as `"data.text" = "Text!"` declared after `data = "Text?"`, gets an unsafe fix that `pipelex fix bundle` never applies, and both the fix and the refusal name what differs, with the edit that settles it on the root (here, replace `?` with `!` on `data`). A lone dotted key gets no automatic fix, since nothing states its root's concept, and its refusal names the two remedies instead.### Changed@@ -17,9 +17,12 @@- **`pipelex show backends` lists every backend with its credential variables**: a backend whose key is unset is listed instead of hidden, a new column names the variables each backend reads its credentials from, and an endpoint read from a variable is shown as not resolved.- **An input name is a plain name (Breaking)**: every key of a pipe's `inputs`, on every operator, controller and signature, and a PipeBatch's `input_list_name` must match `[a-z][a-z0-9_]*`, so a dotted name such as `"invoice.total" = "Number"` or a malformed one such as `InvoiceTotal` is refused when the bundle is parsed, as the new `PipeValidationErrorType` value `invalid_input_name`, which also replaces PipeJudge's own refusal that reached callers with no `error_type`. Its message names the remedies: declare the root with its whole concept (`invoice = "Invoice"`) and read the field in the template (`$invoice.total`), which every operator handles by the concept the path reaches, so `@page.page_view` with `page = "Page"` is still attached as an image, or have the calling sequence bind the field with a binding step (`{ from = "invoice.total", result = "total_amount" }`). The generated MTHDS JSON Schema refuses the same names, through `patternProperties` on every `inputs` table and a `pattern` on `input_list_name`, and the corpus vocabulary records `error.invalid_input_name` with `fails_at = "schema"`.- **A sequence's steps include binding steps (Breaking)**: `PipeSequenceBlueprint.steps` holds `BindingStepBlueprint` beside `SubPipeBlueprint`, and `PipeSequence.sequential_sub_pipes` holds `BindingStep` beside `SubPipe`, so code reading `pipe` or `pipe_code` off every step reads the new `pipe_steps` property instead. A PipeParallel's `branches` keep the pipe-step shape.+- **Names a nested controller stores (Breaking)**: a name a nested sequence, a condition's outcomes or an `add_each_output` parallel always stores in the caller's memory counts as stored by the step calling it, so it is no input of the caller, and a sequence that declared such a name only to pass validation is now refused as `extraneous_input_variable`: remove the declaration. A nested sequence or condition skipped for an absent plain input now records an absence for every name it always stores, so a later step finds a value or an absence there, never nothing.+- **A dotted `batch_over` is a binding followed by a batch (Breaking)**: a sequence step's `batch_over = "catalog.pages"` binds the list at that path under a private name, `_bound_catalog_pages`, by every rule of a binding step, then batches over it, so a list root such as `catalogs.pages` batches over the pages of every catalog, the sequence needs the root as its own concept, and the execution graph draws the binding feeding the batch. A path deriving a single value is refused before the run as `input_stuff_spec_mismatch`, as a batch over a value that is not a list is, and a path outside the path grammar, or a dotted `batch_over` on a PipeParallel branch, as `binding_step_invalid`, which the MTHDS JSON Schema refuses too. The file-format check before the run follows each item of the bound list back to where it sits, as it does after an explicit binding step, so a document at `cases[1].transcripts[0]` that the batched pipe's model cannot read is refused before the batch starts. The `_bound_` prefix is reserved for these private names, since a nested sequence binds in its caller's working memory and would otherwise overwrite a caller's value stored under one: a pipe step's or a PipeParallel branch's `result`, `batch_as` or plain `batch_over`, or a PipeBatch's `input_item_name`, starting with `_bound_` is refused as `invalid_input_name`, which the MTHDS JSON Schema refuses too.### Fixed+- **A batched step's other inputs**: a sequence now needs every input a batched step's pipe reads besides its item even when an earlier step stored the list, where it needed them only for a list the sequence received, so declaring one was refused as `extraneous_input_variable` and leaving it out failed the run inside the batch.- **A PipeBatch branch gets its own copy of its item**: each branch's item is taken from the branch's own copy of the batched list, so a branch that rewrites its item in place, as a `PipeFunc` can, no longer changes the batch's input list.- **A dry-run condition's output is one stuff every outcome produces**: in the graph of a dry run, every outcome of a `PipeCondition` now carries the condition's output digest, so the step reading the condition's result is wired to all of them rather than to the outcome whose pipe code sorts last, and when the outcomes write or declare different outputs, the condition's declared output types that stuff on every node outside the outcomes: the condition, the controllers whose output it is, such as a sequence ending on the condition, and the steps reading it, while each outcome keeps the concept it writes. Each outcome but the last now runs on a copy of the memory the condition received, so an outcome reading the condition's slot reads the value from before the condition rather than a sibling's output, and the default outcome runs last, so the steps after the condition read its value.- **Validation and dry runs no longer depend on which keys the machine holds**: a boot without inference (`Pipelex.make(needs_inference=False)`, used by `pipelex validate`, `pipelex run --dry-run`, `pipelex show` and the agent CLI's validation) now loads every enabled backend with its models and constraints and resolves none of their credentials, where it dropped every backend whose key was missing. A method using a search preset or pinning a bare handle such as `model = "gpt-4o-mini"` now validates on a machine or CI runner with no key, and such a boot no longer reads the backends' keys or requests a Vertex AI token.diff --git a/docs/building-methods/concepts/native-concepts.md b/docs/building-methods/concepts/native-concepts.mdindex 487a5c474..bbaf511bb 100644--- a/docs/building-methods/concepts/native-concepts.md+++ b/docs/building-methods/concepts/native-concepts.md@@ -205,6 +205,8 @@ The run is then refused, with an input error naming the input, when:- an `Image` input holds a file that is not an image, whatever reads it;- a document input is certain to reach a step whose model does not read its format: for example, Word transcripts batched into a `PipeExtract` whose extract model reads only PDF. The error names each input, its format, the step, the model and the formats that model reads.+The check follows each input through the steps of the method: the items of a batch, the fields of a structured input, and what a binding step or a dotted `batch_over` binds, so a Word transcript at `cases[1].transcripts[0]`, batched through `batch_over = "cases.transcripts"` over a list of cases, is refused before the run. A list a binding gathers from a field that is not itself a list, such as `cases.attachment` over a list of cases, is not followed, and a step that reads its items refuses a file when it gets to it.+A step reached only through a `PipeCondition`, or a step that may be skipped because an optional input is absent, may never run, so the run is not refused on its account. If it does run, it refuses the file itself, with the same kind of input error. The same holds for files a run produces along the way.### NumberContentdiff --git a/docs/building-methods/pipes/pipe-controllers/PipeBatch.md b/docs/building-methods/pipes/pipe-controllers/PipeBatch.mdindex 2c3efb199..d3c86a056 100644--- a/docs/building-methods/pipes/pipe-controllers/PipeBatch.md+++ b/docs/building-methods/pipes/pipe-controllers/PipeBatch.md@@ -49,8 +49,8 @@ This is the batch arm of the "route or skip" pattern: batch over items with a co| `inputs` | dictionary | The input concept(s) for the batch operation, as a dictionary mapping input names to concept codes. | Yes || `output` | string | The output concept produced by the batch operation. | Yes || `branch_pipe_code` | string | The name of the single pipe to execute for each item in the input list. | Yes |-| `input_list_name` | string | The name of the input list to iterate over. Must match one of the keys in `inputs`, so it is a plain input name, a `snake_case` identifier matching `[a-z][a-z0-9_]*`, and never a path into a field: a dotted name such as `catalog.pages` is refused as `invalid_input_name`. Typically a plural noun (e.g. `articles`). | Yes |-| `input_item_name` | string | The name that an individual item from the list will have inside its execution branch — this is how the branch pipe finds its input. Must differ from `input_list_name` and from every key in `inputs`. Typically the singular form of the list name (e.g. `article`). | Yes |+| `input_list_name` | string | The name of the input list to iterate over. Must match one of the keys in `inputs`, so it is a plain input name, a `snake_case` identifier matching `[a-z][a-z0-9_]*`, and never a path into a field: a dotted name such as `catalog.pages` is refused as `invalid_input_name`. To map a pipe over a list held in a field, have the calling sequence bind it to the batch's list with a binding step, or run the branch pipe in a sequence step whose dotted `batch_over` binds the list and batches over it (see [Batching over a field](PipeSequence.md#batching-over-a-field)). Typically a plural noun (e.g. `articles`). | Yes |+| `input_item_name` | string | The name that an individual item from the list will have inside its execution branch — this is how the branch pipe finds its input. Must differ from `input_list_name` and from every key in `inputs`, and never starts with `_bound_`, a prefix reserved for the bound list of a dotted `batch_over` (see [Batching over a field](PipeSequence.md#batching-over-a-field)), which is refused as `invalid_input_name`. Typically the singular form of the list name (e.g. `article`). | Yes |### Example: Summarizing a list of articlesdiff --git a/docs/building-methods/pipes/pipe-controllers/PipeParallel.md b/docs/building-methods/pipes/pipe-controllers/PipeParallel.mdindex ab8af5966..daec6878e 100644--- a/docs/building-methods/pipes/pipe-controllers/PipeParallel.md+++ b/docs/building-methods/pipes/pipe-controllers/PipeParallel.md@@ -64,7 +64,7 @@ Each entry in the `branches` array is a table with the following keys:| `pipe` | string | The name of the pipe to execute for this branch. | Yes || `result` | string | The name for this branch's output. Must be unique within the `PipeParallel` definition. | Yes |-A branch is always a pipe step. A [binding step](PipeSequence.md#binding-steps), `{ from = "invoice.total", result = "total_amount" }`, is refused in `branches` with `binding_step_invalid`: the branches run at once, and a binding orders a value before the steps that read it, so bind the value in the calling sequence, in a step before the `PipeParallel`, and have the branch read the bound name.+A branch is always a pipe step. A [binding step](PipeSequence.md#binding-steps), `{ from = "invoice.total", result = "total_amount" }`, is refused in `branches` with `binding_step_invalid`: the branches run at once, and a binding orders a value before the steps that read it, so bind the value in the calling sequence, in a step before the `PipeParallel`, and have the branch read the bound name. A branch may carry `batch_over` and `batch_as`, but its `batch_over` is a plain name: a dotted one, such as `catalog.pages`, binds before it batches (see [Batching over a field](PipeSequence.md#batching-over-a-field)), so it is refused with `binding_step_invalid` too, and the calling sequence binds the list before the `PipeParallel`.### Example: Extracting different details from a textdiff --git a/docs/building-methods/pipes/pipe-controllers/PipeSequence.md b/docs/building-methods/pipes/pipe-controllers/PipeSequence.mdindex ff28c2873..5f87dfaa4 100644--- a/docs/building-methods/pipes/pipe-controllers/PipeSequence.md+++ b/docs/building-methods/pipes/pipe-controllers/PipeSequence.md@@ -42,8 +42,8 @@ A pipe step is a table with the following keys:| `result` | string | The name to give to this step's output in the working memory. When omitted, the output is stored only in the unnamed `main_stuff` slot (the default output), so later steps can pick it up as their implicit input but cannot reference it by a dedicated name. | No || `nb_output` | integer | Request a fixed number of outputs from this step's pipe. Cannot be combined with `multiple_output`. | No || `multiple_output` | boolean | Request a variable number of outputs from this step's pipe (the model decides how many). Cannot be combined with `nb_output`. | No |-| `batch_over` | string | The name of a list in the working memory to batch this step over, running the pipe once per item. Must be provided together with `batch_as`. See [Understanding Multiplicity](../understanding-multiplicity.md). | No |-| `batch_as` | string | The name each item takes in the working memory during a `batch_over` run. Must differ from `batch_over` (e.g. `batch_over = "items"`, `batch_as = "item"`). | No |+| `batch_over` | string | The list in the working memory to batch this step over, running the pipe once per item: a name, or a dotted path to a list held in a field, such as `catalog.pages`, which is a binding followed by a batch (see [Batching over a field](#batching-over-a-field)). Must be provided together with `batch_as`. See [Understanding Multiplicity](../understanding-multiplicity.md). | No |+| `batch_as` | string | The name each item takes in the working memory during a `batch_over` run. Must differ from `batch_over` (e.g. `batch_over = "items"`, `batch_as = "item"`). Like `result` and `batch_over`, it never starts with `_bound_`, a prefix reserved for the bound list of a dotted `batch_over`. | No |#### Binding steps@@ -136,10 +136,45 @@ The bound value is a deep copy taken when the step runs: changing the root after- A sequence ending with a binding step is checked as one ending with a pipe: its output concept with `inadequate_output_concept`, its multiplicity with `inadequate_output_multiplicity`, and a maybe-absent result with `optional_not_handled`.- A step asking a sequence that ends with a binding step for a count of outputs, with `nb_output` or `multiple_output`, is refused with `inadequate_output_multiplicity` when the binding does not bind that count: a binding binds what its path derives, whatever count its caller asks for. A batched step asks its branches for nothing, so it is not checked.- A root that is neither an input of the sequence nor stored by an earlier step on every run is refused with `missing_input_variable`.-- A malformed step is refused with `binding_step_invalid`: `pipe` beside `from`, a binding without `result`, a binding carrying `nb_output`, `multiple_output`, `batch_over` or `batch_as`, a `from` or a `result` outside its grammar. The schema refuses each of these too.+- A malformed step is refused with `binding_step_invalid`: `pipe` beside `from`, a binding without `result`, a binding carrying `nb_output`, `multiple_output`, `batch_over` or `batch_as`, a `from` or a `result` outside its grammar, and a dotted `batch_over` outside the path grammar or on a `PipeParallel` branch. The schema refuses each of these too.A binding step lives in a `PipeSequence`'s `steps` only. A [`PipeParallel`](PipeParallel.md) branch is always a pipe step, since its branches run at once and a binding orders a value before the steps that read it: bind in the sequence before the parallel.+### Batching over a field++A pipe step's `batch_over` may be a dotted path to a list held in a field. The step is then a binding followed by a batch: the path is bound under a private name, by every rule of a binding step's `from`, and the step batches over the bound list.++```toml+[pipe.index_catalog]+type = "PipeSequence"+description = "Writes one index line per page of a catalog"+inputs = { catalog = "Catalog" }+output = "Text[]"+steps = [+ { pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "index_lines" },+]+```++This step runs exactly as the two steps below, apart from the name the list is bound under:++```toml+steps = [+ { from = "catalog.pages", result = "pages" },+ { pipe = "write_index_line", batch_over = "pages", batch_as = "page", result = "index_lines" },+]+```++So a dotted `batch_over` shares everything a binding does:++- The root is typed by the latest step that stored it, or by the sequence's `inputs`, and the sequence needs it as its own concept: `catalog` as a `Catalog`, never as the item's `CatalogPage`.+- Lists map and flatten, so `batch_over = "catalogs.pages"` over `Catalog[]` runs one branch per page of every catalog, and an absent root binds an empty list, which runs no branch.+- The path must reach a list, through a list root, a list field along the way, or a list field it ends on. A path deriving a single value, such as `catalog.season`, is refused before the run with `input_stuff_spec_mismatch`, as a batch over a value that is not a list is, and so is a list whose items the pipe reads as another concept.+- A path the structures cannot walk is refused with `binding_path_unresolved`, and a root that is neither an input of the sequence nor stored by an earlier step with `missing_input_variable`, each message naming the `batch_over` as written.+- A path outside the path grammar, such as `catalog..pages`, is refused with `binding_step_invalid`, which the schema catches too.+- In the execution graph, the binding is a node of kind `binding`, fed by the root's producer, and it feeds the batch.++The private name starts with an underscore, `_bound_catalog_pages` for `catalog.pages`, so no template or input can read it: it shows only in the working memory the run leaves and in the graph. The `_bound_` prefix is reserved for these names. A nested `PipeSequence` binds in the working memory of the sequence calling it, so a name of the caller taking the prefix could be overwritten by the list a sequence it calls binds: a pipe step's `result`, `batch_as` or plain `batch_over`, on a sequence step or a `PipeParallel` branch, and a [`PipeBatch`](PipeBatch.md)'s `input_item_name` that starts with `_bound_` are refused with `invalid_input_name`, which the schema catches too, and the message asks for another name. Only a sequence's steps bind, so a [`PipeParallel`](PipeParallel.md) branch carries a plain `batch_over` only, and a dotted one there is refused with `binding_step_invalid`: bind the list in a step before the `PipeParallel`, and batch the branch over the bound name.+### ExampleLet's imagine a pipeline that first extracts text from an image, then summarizes that text, and finally translates the summary into French.diff --git a/docs/building-methods/pipes/understanding-multiplicity.md b/docs/building-methods/pipes/understanding-multiplicity.mdindex 3abd1dae9..bf162f837 100644--- a/docs/building-methods/pipes/understanding-multiplicity.md+++ b/docs/building-methods/pipes/understanding-multiplicity.md@@ -268,6 +268,8 @@ steps = []```+When the list is held in a field of a bigger value, such as the scans of an `InvoiceBatch`, `batch_over` takes the dotted path, `batch_over = "invoice_batch.scans"`: the sequence binds the list at that path, then batches over it (see [Batching over a field](pipe-controllers/PipeSequence.md#batching-over-a-field)).+### Use Case 2: Generate Fixed Alternatives for TestingCreate exactly 3 subject line variations for A/B testing:diff --git a/pipelex/core/memory/working_memory.py b/pipelex/core/memory/working_memory.pyindex a0fbccc27..71f75d171 100644--- a/pipelex/core/memory/working_memory.py+++ b/pipelex/core/memory/working_memory.py@@ -35,6 +35,12 @@ from pipelex.tools.misc.string_utils import find_private_path_segmentMAIN_STUFF_NAME = "main_stuff"BATCH_ITEM_STUFF_NAME = "BATCH_ITEM"+# The prefix of the private name a PipeSequence binds a dotted `batch_over`'s list under: underscore-led, so no template or+# input reads it, and never a param's, so working memory holds it like any stuff. The prefix is the runtime's alone: a nested+# sequence binds in its caller's working memory, so validation refuses any name an author writes taking it+# (`check_name_is_not_reserved`), and a caller's value is never overwritten by a list a sequence it calls binds. This one+# constant spells the prefix for the name `make_private_binding_name` generates, the params exemption below, and that refusal.+PRIVATE_BINDING_NAME_PREFIX = "_bound_"PRETTY_PRINT_MAX_LENGTH = 1000TEST_DUMMY_NAME = "dummy_result"@@ -42,6 +48,11 @@ StuffDict = dict[str, Stuff]StuffArtefactDict = dict[str, StuffArtefact]+def _is_reserved_for_params(*, name: str) -> bool:+ """Whether a stuff name takes the underscore that marks params, which a private binding name carries by design."""+ return name.startswith("_") and name != BATCH_ITEM_STUFF_NAME and not name.startswith(PRIVATE_BINDING_NAME_PREFIX)++class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):root: StuffDict = Field(default_factory=dict)aliases: dict[str, str] = Field(default_factory=dict)@@ -54,14 +65,14 @@ class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):@model_validator(mode="after")def validate_stuff_names(self) -> Self:for key, stuff in self.root.items():- if key.startswith("_") and key != BATCH_ITEM_STUFF_NAME:+ if _is_reserved_for_params(name=key):log.warning(f"Stuff key '{key}' starts with '_', which is reserved for params")if not stuff.stuff_name:self.root[key].stuff_name = keyelif key not in {MAIN_STUFF_NAME, stuff.stuff_name}:log.warning(f"Stuff name '{stuff.stuff_name}' does not match the key '{key}'")- elif stuff.stuff_name.startswith("_") and stuff.stuff_name != BATCH_ITEM_STUFF_NAME:+ elif _is_reserved_for_params(name=stuff.stuff_name):log.warning(f"Stuff name '{stuff.stuff_name}' starts with '_', which is reserved for params")return selfdiff --git a/pipelex/language/mthds_schema_generator.py b/pipelex/language/mthds_schema_generator.pyindex 34c75dcfa..4e6f4cb5c 100644--- a/pipelex/language/mthds_schema_generator.py+++ b/pipelex/language/mthds_schema_generator.py@@ -12,7 +12,14 @@ from typing import TYPE_CHECKING, Any, cast, get_argsfrom pipelex.mthds_parsing.pipelex_bundle_blueprint import PipeBlueprintUnion, PipelexBundleBlueprintfrom pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint-from pipelex.pipe_machinery.validation import INPUT_NAME_PATTERN+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint+from pipelex.pipe_machinery.validation import (+ INPUT_NAME_PATTERN,+ PARALLEL_BRANCH_BATCH_OVER_PATTERN,+ RESERVED_NAME_PATTERN,+ SEQUENCE_STEP_BATCH_OVER_PATTERN,+)from pipelex.pipe_signature.pipe_signature_blueprint import PipeSignatureBlueprintfrom pipelex.tools.misc.package_utils import get_package_version@@ -37,6 +44,14 @@ _PIPE_DEFINITION_NAMES: frozenset[str] = frozenset(member.__name__ for member in# is rejected (extra property under the arm's `additionalProperties: false`)._SIGNATURE_DEFINITION_NAME = PipeSignatureBlueprint.__name__+# The definition a PipeParallel branch takes: `SubPipeBlueprint`'s, with a `batch_over` that is never dotted+# (`_constrain_batch_over`). No Python class carries the name, since steps and branches share one blueprint.+_PARALLEL_BRANCH_DEFINITION_NAME = "ParallelBranchBlueprint"++# The fields of a pipe step, and of a PipeParallel branch, naming a slot of working memory: the runtime refuses each when it+# takes the prefix reserved for the bound list of a dotted `batch_over` (`_reserve_private_binding_names`).+_PIPE_STEP_NAME_FIELDS: tuple[str, ...] = ("result", "batch_as", "batch_over")+def generate_mthds_schema() -> dict[str, Any]:"""Generate a Taplo-compatible JSON Schema for .mthds files.@@ -59,6 +74,9 @@ def generate_mthds_schema() -> dict[str, Any]:schema = _convert_to_draft4(schema)schema = _patch_construct_schema(schema)schema = _constrain_input_names(schema)+ # Before `_constrain_batch_over`, so the branch definition it copies from the pipe step's carries the reservation too.+ schema = _reserve_private_binding_names(schema)+ schema = _constrain_batch_over(schema)return _add_taplo_metadata(schema)@@ -291,6 +309,83 @@ def _constrain_input_names(schema: dict[str, Any]) -> dict[str, Any]:return schema+def _reserve_private_binding_names(schema: dict[str, Any]) -> dict[str, Any]:+ """Refuse the names the runtime reserves for the bound list of a dotted `batch_over`, as the runtime does.++ A pipe step's `result`, `batch_as` and plain `batch_over`, and a PipeBatch's `input_item_name`, never take the `_bound_`+ prefix: the runtime refuses each as `invalid_input_name` when the bundle is parsed, which keeps that error type's+ `fails_at = "schema"` true. Each is a Draft-4 `not` holding a `pattern` on the field's string arm: a negative lookahead in+ the pattern itself would say the same, but not every validator's regex engine has one, while every Draft-4 validator+ applies `not`, plxt's included. The other names an author writes into working memory, input names and a binding step's+ `result`, already follow the plain-name grammar, which no underscore-led name matches.+ """+ schema = copy.deepcopy(schema)+ definitions = schema.get("definitions", {})+ pipe_step_properties = definitions.get(SubPipeBlueprint.__name__, {}).get("properties", {})+ for field_name in _PIPE_STEP_NAME_FIELDS:+ _refuse_reserved_names(field_schema=pipe_step_properties.get(field_name))+ _refuse_reserved_names(field_schema=definitions.get(PipeBatchBlueprint.__name__, {}).get("properties", {}).get("input_item_name"))+ return schema+++def _refuse_reserved_names(*, field_schema: dict[str, Any] | None) -> None:+ if field_schema is None:+ return+ for arm in field_schema.get("anyOf", [field_schema]):+ if arm.get("type") == "string":+ arm["not"] = {"pattern": RESERVED_NAME_PATTERN}+++def _constrain_batch_over(schema: dict[str, Any]) -> dict[str, Any]:+ """Give `batch_over` its grammar: a dotted path on a PipeSequence step, and a name with no dot on a PipeParallel branch.++ A dotted `batch_over` binds the list at its path before batching over it, so on a sequence's pipe step it follows the+ binding path grammar, and a PipeParallel branch, which never binds, carries none: the runtime refuses both faults as+ `binding_step_invalid`, which this makes the schema refuse first. Both steps and branches parse into `SubPipeBlueprint`,+ so the branch takes a copy of its definition, `ParallelBranchBlueprint`, carrying the stricter `pattern`. Each is a+ `pattern` on the string arm of the field, which a Draft-4 validator such as plxt's applies.+ """+ schema = copy.deepcopy(schema)+ definitions = schema.get("definitions", {})+ pipe_step_schema = definitions.get(SubPipeBlueprint.__name__)+ branches_schema = definitions.get(PipeParallelBlueprint.__name__, {}).get("properties", {}).get("branches")+ if pipe_step_schema is None or branches_schema is None:+ return schema++ branch_schema = copy.deepcopy(pipe_step_schema)+ branch_schema["title"] = _PARALLEL_BRANCH_DEFINITION_NAME+ _set_batch_over_grammar(+ step_schema=pipe_step_schema,+ pattern=SEQUENCE_STEP_BATCH_OVER_PATTERN,+ description=(+ "The list in working memory to batch this step over, running the pipe once per item. A dotted path such as "+ "`catalog.pages` is a binding followed by a batch: the path is bound under a private name, by the rules of a binding "+ "step's `from`, and the step batches over the bound list, which must be a list."+ ),+ )+ _set_batch_over_grammar(+ step_schema=branch_schema,+ pattern=PARALLEL_BRANCH_BATCH_OVER_PATTERN,+ description=(+ "The list in working memory to batch this branch over, running the pipe once per item. A name with no dot: a branch "+ "never binds, so a list held in a field is bound by the calling sequence before the PipeParallel step."+ ),+ )+ definitions[_PARALLEL_BRANCH_DEFINITION_NAME] = branch_schema+ branches_schema["items"] = {"$ref": f"#/definitions/{_PARALLEL_BRANCH_DEFINITION_NAME}"}+ return schema+++def _set_batch_over_grammar(*, step_schema: dict[str, Any], pattern: str, description: str) -> None:+ batch_over_schema = step_schema.get("properties", {}).get("batch_over")+ if batch_over_schema is None:+ return+ batch_over_schema["description"] = description+ for arm in batch_over_schema.get("anyOf", [batch_over_schema]):+ if arm.get("type") == "string":+ arm["pattern"] = pattern++def _build_construct_field_schema() -> dict[str, Any]:"""Build a JSON Schema for a construct field value as written in MTHDS files.diff --git a/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py b/pipelex/pipe_controllers/batch/pipe_batch_blueprint.pyindex f761bf08d..52e1bc491 100644--- a/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py+++ b/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py@@ -4,7 +4,7 @@ from typing_extensions import overridefrom pipelex.core.pipes.exceptions import PipeValidationErrorfrom pipelex.pipe_machinery.pipe_blueprint import PipeBlueprint-from pipelex.pipe_machinery.validation import check_input_list_name+from pipelex.pipe_machinery.validation import check_input_list_name, check_name_is_not_reservedfrom pipelex.validation_error_types import PipeValidationErrorType@@ -25,7 +25,7 @@ class PipeBatchBlueprint(PipeBlueprint):def validate_inputs(self):# The list is one of the batch's own inputs, so its name is a plain input name: a list held in a# field is declared under a plain name and handed to the batch by the calling sequence.- check_input_list_name(input_list_name=self.input_list_name)+ check_input_list_name(input_list_name=self.input_list_name, branch_pipe_code=self.branch_pipe_code, input_item_name=self.input_item_name)# The PipeBatch will iterate over a list and pass each item to the branch pipe,# so we must have the list's name as part of the batch inputs# and conversely, we must not have the item's name as part of the batch inputs@@ -35,6 +35,9 @@ class PipeBatchBlueprint(PipeBlueprint):if not self.input_item_name:msg = "Empty input item name is not allowed"raise ValueError(msg)+ # The item is written into each branch's working memory under its name, so the name never takes the prefix the runtime+ # reserves for the bound list of a dotted `batch_over`. The list's name needs no such check: it is a plain input name.+ check_name_is_not_reserved(name=self.input_item_name, field_label="The PipeBatch's `input_item_name`")# Specialization: catch the most common mistake (item name == list name) with a targeted messageif self.input_item_name == self.input_list_name:msg = (diff --git a/pipelex/pipe_controllers/binding/binding_derivation.py b/pipelex/pipe_controllers/binding/binding_derivation.pyindex 2878a5f5d..370902af5 100644--- a/pipelex/pipe_controllers/binding/binding_derivation.py+++ b/pipelex/pipe_controllers/binding/binding_derivation.py@@ -188,6 +188,8 @@ class BindingDerivation(BaseModel):# How the runtime stores the value it reaches: a concept's content is copied whole, a plain value wrapped.leaf_kind: BindingValueKindsegments: tuple[BindingSegment, ...] = ()+ # The multiplicity of the root the path starts from, as the sequence knows it at the binding step.+ root_multiplicity: VariableMultiplicity | None = None@propertydef root_name(self) -> str:@@ -201,6 +203,40 @@ class BindingDerivation(BaseModel):def is_plural(self) -> bool:return is_multiple_multiplicity(multiplicity=self.multiplicity)+ def provenance_path(self, *, item_segment: str) -> tuple[str, ...] | None:+ """Where, below its root, the value the binding binds comes from, `item_segment` standing for any item of a list crossed.++ A list sits at its own path and its items at that path followed by the item segment, so what the binding binds has+ such a path when it is the value at one path of its root, or the items of a list field reached across items:++ - a single value, or one list field of a single root, is at its fields' path: `invoice.lines` is `("lines",)`;+ - a bare name is its root's own value, a list included: `()`;+ - a list gathered across items whose last field is itself a list holds the items of that field, wherever they sit:+ the item segment follows a list root, and every field before the last that crosses a list, so `cases.transcripts`+ over a list root is `("[]", "transcripts")` and `case.folders.transcripts` is `("folders", "[]", "transcripts")`.++ A list gathered from a field that is not itself a list, `cases.attachment` over a list root, holds values that are+ items of no list at one path, so it has no such path: `None`.++ Args:+ item_segment: The segment the caller's paths use for any item of a list.++ Returns:+ The segments below the root, or `None` for a list gathered from a field that is not a list.+ """+ if self.is_bare_name:+ return ()+ *leading_segments, last_segment = self.segments+ if self.is_plural and not last_segment.crosses_list:+ return None+ provenance: list[str] = [item_segment] if is_multiple_multiplicity(multiplicity=self.root_multiplicity) else []+ for segment in leading_segments:+ provenance.append(segment.name)+ if segment.crosses_list:+ provenance.append(item_segment)+ provenance.append(last_segment.name)+ return tuple(provenance)+def _quoted(*, names: list[str]) -> str:return ", ".join(f"'{name}'" for name in names)@@ -236,6 +272,7 @@ def derive_binding(*, path: str, root: BindingRoot, resolver: ConceptWalkResolvemultiplicity=root.multiplicity,may_find_nothing=False,leaf_kind=BindingValueKind.CONCEPT,+ root_multiplicity=root.multiplicity,)crosses_any_list = is_multiple_multiplicity(multiplicity=root.multiplicity)@@ -349,4 +386,5 @@ def derive_binding(*, path: str, root: BindingRoot, resolver: ConceptWalkResolvefirst_optional_path=first_optional_path if not crosses_any_list else None,leaf_kind=leaf_kind,segments=tuple(segments),+ root_multiplicity=root.multiplicity,)diff --git a/pipelex/pipe_controllers/binding/binding_step.py b/pipelex/pipe_controllers/binding/binding_step.pyindex 6f07e1ff2..8648b26c0 100644--- a/pipelex/pipe_controllers/binding/binding_step.py+++ b/pipelex/pipe_controllers/binding/binding_step.py@@ -20,7 +20,7 @@ from pydantic import BaseModel, ConfigDictfrom pipelex import logfrom pipelex.core.concepts.concept import Conceptfrom pipelex.core.memory.absence import AbsenceKind, AbsenceRecord-from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.memory.working_memory import PRIVATE_BINDING_NAME_PREFIX, WorkingMemoryfrom pipelex.core.pipes.inputs.exceptions import PipeRunInputsErrorfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.stuff import Stuff@@ -52,6 +52,24 @@ class BindingOutcome(BaseModel):return self.absence is not None and self.absence.kind.is_skipped+def make_private_binding_name(*, path: str, taken_names: set[str]) -> str:+ """The private name a sequence binds a dotted `batch_over`'s list under: underscore-led, so no template or input reads it.++ It spells the path, `_bound_catalog_pages` for `catalog.pages`, and takes a numbered suffix when a name of the sequence+ already holds that spelling, as when two of its steps batch over the same dotted path. It is generated once the bundle is+ validated, and never checked against the plain-name grammar of a binding step's `result`, which governs the steps an author+ writes. No name an author writes takes the prefix, which validation refuses, so a private name never takes the name of a+ value the sequence's caller holds in the working memory they share.+ """+ base_name = f"{PRIVATE_BINDING_NAME_PREFIX}{path.replace('.', '_')}"+ private_name = base_name+ suffix = 2+ while private_name in taken_names:+ private_name = f"{base_name}_{suffix}"+ suffix += 1+ return private_name++class BindingStep(BaseModel):"""A PipeSequence step binding the value at `from_path` to `output_name`, as the runtime holds it."""@@ -59,6 +77,9 @@ class BindingStep(BaseModel):from_path: stroutput_name: str+ # Whether the sequence wrote this step itself, rewriting a pipe step's dotted `batch_over` into a binding of its path under+ # a private name, followed by the same step batching over that name. Messages then name the `batch_over` the author wrote.+ is_dotted_batch_over: bool = False@propertydef root_name(self) -> str:@@ -69,6 +90,13 @@ class BindingStep(BaseModel):"""The step as MTHDS writes it, for messages."""return f'{{ from = "{self.from_path}", result = "{self.output_name}" }}'+ @property+ def label(self) -> str:+ """How a message names the step, after "the": the binding step as written, or the dotted `batch_over` the sequence rewrote."""+ if self.is_dotted_batch_over:+ return f'dotted `batch_over = "{self.from_path}"`'+ return f"binding step {self.as_written}"+def bind(self,*,@@ -107,10 +135,7 @@ class BindingStep(BaseModel):record = AbsenceRecord(variable_name=self.output_name,kind=AbsenceKind.DECLARED_ABSENT,- reason=(- f"the binding step {self.as_written} of pipe '{calling_pipe_code}' bound nothing, "- f"because '{bound_content.empty_path}' holds nothing"- ),+ reason=(f"the {self.label} of pipe '{calling_pipe_code}' bound nothing, because '{bound_content.empty_path}' holds nothing"),)working_memory.record_new_main_absence(record)log.verbose(f"Binding '{self.from_path}' found nothing at '{bound_content.empty_path}': '{self.output_name}' is recorded absent")@@ -147,7 +172,7 @@ class BindingStep(BaseModel):root_absence = working_memory.get_optional_absence(self.root_name)if root_absence is None:msg = (- f"The binding step {self.as_written} of pipe '{calling_pipe_code}' reads '{self.root_name}', which is not in working "+ f"The {self.label} of pipe '{calling_pipe_code}' reads '{self.root_name}', which is not in working "f"memory and has no recorded absence. Valid keys are: {working_memory.list_keys()}")raise PipeRunInputsError(diff --git a/pipelex/pipe_controllers/binding/binding_step_blueprint.py b/pipelex/pipe_controllers/binding/binding_step_blueprint.pyindex 328d6441a..1fb0e64aa 100644--- a/pipelex/pipe_controllers/binding/binding_step_blueprint.py+++ b/pipelex/pipe_controllers/binding/binding_step_blueprint.py@@ -16,7 +16,8 @@ BINDING_FROM_KEY = "from"# The fields of a pipe step that a binding step never carries, `result` aside, which both shapes share.PIPE_STEP_ONLY_FIELDS: tuple[str, ...] = ("pipe", "nb_output", "multiple_output", "batch_over", "batch_as")-_PATH_GRAMMAR = (+# How a message states the path grammar, after "a path is": the grammar of a binding step's `from` and of a dotted `batch_over`.+PATH_GRAMMAR_DESCRIPTION = ("a name in working memory followed by zero or more field names, separated by single dots, each a letter followed by letters, "f"digits and underscores (`{FIELD_PATH_SEGMENT_REGEX}`), with no subscript, expression, whitespace or underscore-led segment")@@ -52,7 +53,8 @@ def check_binding_step_shape(*, raw_step: Mapping[str, Any], step_label: str) ->quoted_fields = ", ".join(f"`{field_name}`" for field_name in pipe_step_fields)msg = (f"{step_label} is a binding step (it carries `from`), which carries only `from` and `result`, so it cannot carry {quoted_fields}. "- "To batch over a bound list, bind it first, then batch over the bound name in the next pipe step."+ f'To batch over the list at `{from_path}`, write it as a pipe step\'s `batch_over = "{from_path}"`, which binds it and batches over '+ "the bound list, or bind it first, then batch over the bound name in the next pipe step.")raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.BINDING_STEP_INVALID)if "result" not in raw_step:@@ -86,7 +88,7 @@ class BindingStepBlueprint(BaseModel):@classmethoddef validate_from_path(cls, from_path: str) -> str:if not is_field_path(path=from_path):- msg = f"The binding step's `from` '{from_path}' is not a path: a path is {_PATH_GRAMMAR}."+ msg = f"The binding step's `from` '{from_path}' is not a path: a path is {PATH_GRAMMAR_DESCRIPTION}."raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.BINDING_STEP_INVALID, variable_names=[from_path])return from_pathdiff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.pyindex 36795b967..88b7c4752 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py@@ -41,7 +41,6 @@ from pipelex.pipe_machinery.memory_writes import MemoryWrite, SlotTaintfrom pipelex.pipe_machinery.pipe_abstract import CompanionSlotfrom pipelex.pipe_run.pipe_run_params import PipeRunParams, output_multiplicity_to_applyfrom pipelex.system.job_metadata import JobMetadata-from pipelex.tools.misc.string_utils import get_root_from_dotted_pathfrom pipelex.validation_error_types import PipeValidationErrorTypeif TYPE_CHECKING:@@ -129,9 +128,9 @@ class PipeParallel(PipeController):f"in this Parallel Pipe '{self.code}' input requirements: {pipe_needed_inputs}")raise PipeValidationError(message=msg) from exc- input_list_root = get_root_from_dotted_path(sub_pipe.batch_params.input_list_stuff_name)+ # A branch batches over a plain name only: a dotted `batch_over` is refused on a branch, since it binds.needed_inputs.add_stuff_spec(- variable_name=input_list_root,+ variable_name=sub_pipe.batch_params.input_list_stuff_name,concept=stuff_spec.concept,multiplicity=True,)diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.pyindex e07ce1dde..2123bb87d 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py@@ -7,7 +7,7 @@ from pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.pipes.exceptions import PipeValidationErrorfrom pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.pipe_controllers.binding.binding_step_blueprint import BINDING_FROM_KEY, is_binding_step_dict, raw_step_mapping-from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint, is_dotted_batch_overfrom pipelex.pipe_machinery.pipe_blueprint import PipeBlueprintfrom pipelex.validation_error_types import PipeValidationErrorType@@ -27,25 +27,41 @@ class PipeParallelBlueprint(PipeBlueprint):@field_validator("branches", mode="before")@classmethoddef refuse_binding_branches(cls, branches: Any) -> Any:- """Refuse a branch written as a binding step: a branch is always a pipe step.+ """Refuse a branch that binds: a branch written as a binding step, or a branch whose `batch_over` is a dotted path.A binding orders a value before the steps that read it, and branches run concurrently, so a binding- among them would only be a binding before the parallel, written in the wrong place.+ among them would only be a binding before the parallel, written in the wrong place. A dotted+ `batch_over` is a binding followed by a batch, so a branch carries only a plain one."""if not isinstance(branches, list):# Not a list at all: pydantic's own `list_type` error names the field and what it holds.return branchesbranch_list = cast("list[Any]", branches)for branch_index, branch in enumerate(branch_list):- raw_branch = raw_step_mapping(raw_step=branch)- if raw_branch is not None and is_binding_step_dict(raw_step=raw_branch):- from_path = raw_branch.get(BINDING_FROM_KEY)+ batch_over: Any+ if isinstance(branch, SubPipeBlueprint):+ batch_over = branch.batch_over+ else:+ raw_branch = raw_step_mapping(raw_step=branch)+ if raw_branch is None:+ continue+ if is_binding_step_dict(raw_step=raw_branch):+ from_path = raw_branch.get(BINDING_FROM_KEY)+ msg = (+ f"Branch {branch_index + 1} of the parallel is a binding step (it carries `from`), but a PipeParallel branch is always "+ f"a pipe step: bind the value in a step of the calling PipeSequence, before the PipeParallel step "+ f'(`{{ from = "{from_path}", result = "<name>" }}`), and have the branch read that name.'+ )+ raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.BINDING_STEP_INVALID)+ batch_over = raw_branch.get("batch_over")+ if isinstance(batch_over, str) and is_dotted_batch_over(batch_over=batch_over):msg = (- f"Branch {branch_index + 1} of the parallel is a binding step (it carries `from`), but a PipeParallel branch is always "- f"a pipe step: bind the value in a step of the calling PipeSequence, before the PipeParallel step "- f'(`{{ from = "{from_path}", result = "<name>" }}`), and have the branch read that name.'+ f"Branch {branch_index + 1} of the parallel batches over the dotted path '{batch_over}', but a dotted `batch_over` binds "+ "the list at its path before batching over it, and a PipeParallel branch never binds: bind the list in a step of the "+ f'calling PipeSequence, before the PipeParallel step (`{{ from = "{batch_over}", result = "<name>" }}`), and have the '+ 'branch batch over that name (`batch_over = "<name>"`).')- raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.BINDING_STEP_INVALID)+ raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.BINDING_STEP_INVALID, variable_names=[batch_over])return branch_list@overridediff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.pyindex ca478691b..aa9d3cc8c 100644--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py@@ -35,6 +35,7 @@ from pipelex.pipe_controllers.sequence.sequence_typed_flow import (FlowSlot,SequenceStep,SequenceTypedFlow,+ authored_step_number,build_sequence_typed_flow,derive_binding_spec,step_memory_writes,@@ -42,9 +43,8 @@ from pipelex.pipe_controllers.sequence.sequence_typed_flow import (from pipelex.pipe_controllers.sub_pipe import SubPipefrom pipelex.pipe_machinery.memory_writes import MemoryWrite, SlotTaint, taint_after_writefrom pipelex.pipe_machinery.validation import is_valid_input_name-from pipelex.pipe_run.pipe_run_params import PipeRunParams, output_multiplicity_to_apply+from pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunParams, output_multiplicity_to_applyfrom pipelex.system.job_metadata import JobMetadata-from pipelex.tools.misc.string_utils import get_root_from_dotted_pathfrom pipelex.validation_error_types import PipeValidationErrorTypeif TYPE_CHECKING:@@ -78,6 +78,10 @@ class PipeSequence(PipeController):"""The steps that run a pipe, in order; a binding step runs none."""return [step for step in self.sequential_sub_pipes if isinstance(step, SubPipe)]+ def _step_number(self, *, step_index: int) -> int:+ """The number messages give a step: its place among the steps the author wrote, counting from 1."""+ return authored_step_number(steps=self.sequential_sub_pipes, step_index=step_index)+@propertydef has_binding_step(self) -> bool:"""Whether a step binds a value: only binding steps read the typed flow, so a sequence without one never builds it."""@@ -172,33 +176,27 @@ class PipeSequence(PipeController):# Use the centralized recursion detectionsub_pipe_needed_inputs = sub_pipe.needed_inputs(visited_pipes=visited_pipes_with_current)- if sequential_sub_pipe.batch_params:- input_list_root = get_root_from_dotted_path(sequential_sub_pipe.batch_params.input_list_stuff_name)- if input_list_root not in generated_outputs:- try:- stuff_spec = sub_pipe_needed_inputs.get_required_stuff_spec(- variable_name=sequential_sub_pipe.batch_params.input_item_stuff_name- )- except InputStuffSpecNotFoundError as exc:- msg = (- f"Batch input item named '{sequential_sub_pipe.batch_params.input_item_stuff_name}' is not "- f"in this PipeSequence '{self.code}' input requirements: {sub_pipe_needed_inputs.format_for_display()}"- )- raise PipeSequenceValueError(msg) from exc- is_dotted_path = "." in sequential_sub_pipe.batch_params.input_list_stuff_name- needed_inputs.add_stuff_spec(- variable_name=input_list_root,- concept=stuff_spec.concept,- multiplicity=True if not is_dotted_path else None,+ if batch_params := sequential_sub_pipe.batch_params:+ try:+ item_stuff_spec = sub_pipe_needed_inputs.get_required_stuff_spec(variable_name=batch_params.input_item_stuff_name)+ except InputStuffSpecNotFoundError as exc:+ msg = (+ f"Batch input item named '{batch_params.input_item_stuff_name}' is not "+ f"in this PipeSequence '{self.code}' input requirements: {sub_pipe_needed_inputs.format_for_display()}")- for input_name, stuff_spec in sub_pipe_needed_inputs.items:- if input_name != sequential_sub_pipe.batch_params.input_item_stuff_name and input_name not in generated_outputs:- needed_inputs.add_stuff_spec(- variable_name=input_name,- concept=stuff_spec.concept,- multiplicity=stuff_spec.multiplicity,- presence=stuff_spec.presence,- )+ raise PipeSequenceValueError(msg) from exc+ # The list is needed only when no earlier step stores it, as the binding of a dotted `batch_over` always does,+ # while the batched pipe's other inputs are needed whichever step stored the list.+ if batch_params.input_list_stuff_name not in generated_outputs:+ needed_inputs.add_stuff_spec(variable_name=batch_params.input_list_stuff_name, concept=item_stuff_spec.concept, multiplicity=True)+ for input_name, stuff_spec in sub_pipe_needed_inputs.items:+ if input_name != batch_params.input_item_stuff_name and input_name not in generated_outputs:+ needed_inputs.add_stuff_spec(+ variable_name=input_name,+ concept=stuff_spec.concept,+ multiplicity=stuff_spec.multiplicity,+ presence=stuff_spec.presence,+ )else:for input_name, stuff_spec in sub_pipe_needed_inputs.items:if input_name not in generated_outputs:@@ -235,7 +233,7 @@ class PipeSequence(PipeController):f"store a value under '{variable_name}' in an earlier step, whose `result` names it, or bind from a plain name.")msg = (- f"In pipe '{self.code}', the binding step {step.as_written} reads '{variable_name}', which is neither an input of the "+ f"In pipe '{self.code}', the {step.label} reads '{variable_name}', which is neither an input of the "f"sequence nor always stored by an earlier step. {remedy}")raise PipeValidationError(@@ -277,14 +275,13 @@ class PipeSequence(PipeController):if step.batch_params is not None:batch_item_name = step.batch_params.input_item_stuff_namelist_slot = binding_slots.get(step.batch_params.input_list_stuff_name)- item_need = step_needs.root.get(batch_item_name)- if list_slot is not None and list_slot.stuff_spec is not None and item_need is not None:- self._check_binding_consumer(+ if list_slot is not None:+ self._check_batched_binding(step_index=step_index,step_pipe_code=step_pipe.code,- variable_name=step.batch_params.input_list_stuff_name,+ batch_params=step.batch_params,slot=list_slot,- needed_spec=StuffSpec(concept=item_need.concept, multiplicity=True),+ item_need=step_needs.root.get(batch_item_name),)for input_name, needed_spec in step_needs.items:if input_name == batch_item_name:@@ -341,11 +338,12 @@ class PipeSequence(PipeController):if is_multiplicity_compatible(source_multiplicity=bound_spec.multiplicity, target_multiplicity=requested_multiplicity):continuefinal_binding = step_pipe.sequential_sub_pipes[-1]- binding_label = final_binding.as_written if isinstance(final_binding, BindingStep) else "a binding step"+ binding_label = final_binding.label if isinstance(final_binding, BindingStep) else "a binding step"bound_ref = bound_spec.to_bundle_representation(relative_to_domain=self.domain_code)msg = (- f"In pipe '{self.code}', step {step_index + 1} asks pipe '{step_pipe.code}' for {requested_label}, but '{step_pipe.code}' "- f"ends with the binding step {binding_label}, which binds '{bound_ref}': a binding binds what its path derives, whatever "+ f"In pipe '{self.code}', step {self._step_number(step_index=step_index)} asks pipe '{step_pipe.code}' for {requested_label}, "+ f"but '{step_pipe.code}' "+ f"ends with the {binding_label}, which binds '{bound_ref}': a binding binds what its path derives, whatever "f"count its caller asks for. Remove the step's `nb_output` or `multiple_output`, or ask for what the binding binds.")raise PipeValidationError(@@ -365,7 +363,9 @@ class PipeSequence(PipeController):if needed_spec.concept.code in {NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING}:returnbinding_step = self.sequential_sub_pipes[slot.binding_step_index]- binding_label = binding_step.as_written if isinstance(binding_step, BindingStep) else f"step {slot.binding_step_index + 1}"+ binding_label = (+ binding_step.label if isinstance(binding_step, BindingStep) else f"step {self._step_number(step_index=slot.binding_step_index)}"+ )is_concept_compatible = get_concept_library().is_compatible(tested_concept=bound_spec.concept, wanted_concept=needed_spec.concept)is_multiplicity_ok = is_multiplicity_compatible(source_multiplicity=bound_spec.multiplicity, target_multiplicity=needed_spec.multiplicity)if is_concept_compatible and is_multiplicity_ok:@@ -377,9 +377,9 @@ class PipeSequence(PipeController):relative_to_domain=self.domain_code)msg = (- f"In pipe '{self.code}', step {step_index + 1} (pipe '{step_pipe_code}') reads '{variable_name}' as '{needed_ref}', but the binding "- f"step {binding_label} binds it as '{bound_ref}'. Declare the input as '{bound_ref}' in pipe '{step_pipe_code}', or bind a path "- f"that reaches a '{needed_ref}'."+ f"In pipe '{self.code}', step {self._step_number(step_index=step_index)} (pipe '{step_pipe_code}') reads '{variable_name}' as "+ f"'{needed_ref}', but the {binding_label} binds it as '{bound_ref}'. Declare the input as '{bound_ref}' in pipe '{step_pipe_code}', "+ f"or bind a path that reaches a '{needed_ref}'.")raise PipeValidationError(message=msg,@@ -391,6 +391,66 @@ class PipeSequence(PipeController):required_concept_codes=[needed_spec.concept.concept_ref],)+ def _check_batched_binding(+ self, *, step_index: int, step_pipe_code: str, batch_params: BatchParams, slot: FlowSlot, item_need: StuffSpec | None+ ) -> None:+ """Refuse a step batching over a binding's result that is not a list, or whose items its pipe reads as another concept.++ A dotted `batch_over` is named by the path its author wrote, never by the private name the sequence bound it under.++ Raises:+ PipeValidationError: ``INPUT_STUFF_SPEC_MISMATCH``, as a batch over a value that is not a list is refused, naming+ the step and what the binding binds.+ """+ bound_spec = slot.stuff_spec+ if bound_spec is None or slot.binding_step_index is None:+ return+ binding_step = self.sequential_sub_pipes[slot.binding_step_index]+ if not isinstance(binding_step, BindingStep):+ return+ if binding_step.is_dotted_batch_over:+ batched_phrase = f"the dotted path '{binding_step.from_path}', which derives"+ else:+ batched_phrase = f"'{batch_params.input_list_stuff_name}', which the {binding_step.label} binds as"+ step_label = f"step {self._step_number(step_index=step_index)} (pipe '{step_pipe_code}')"+ bound_ref = StuffSpec(concept=bound_spec.concept, multiplicity=bound_spec.multiplicity).to_bundle_representation(+ relative_to_domain=self.domain_code+ )+ if not bound_spec.is_multiple():+ msg = (+ f"In pipe '{self.code}', {step_label} batches over {batched_phrase} a single '{bound_ref}', not a list: a batch runs its "+ "pipe once per item of a list. Batch over a path that reaches a list, through a list root or a list field, or run the step "+ "on the value itself, without `batch_over`."+ )+ raise PipeValidationError(+ message=msg,+ error_type=PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,+ domain_code=self.domain_code,+ pipe_code=self.code,+ variable_names=[binding_step.from_path if binding_step.is_dotted_batch_over else batch_params.input_list_stuff_name],+ provided_concept_code=bound_spec.concept.concept_ref,+ )+ if item_need is None or item_need.concept.code in {NativeConceptCode.DYNAMIC, NativeConceptCode.ANYTHING}:+ return+ if get_concept_library().is_compatible(tested_concept=bound_spec.concept, wanted_concept=item_need.concept):+ return+ item_ref = StuffSpec(concept=bound_spec.concept).to_bundle_representation(relative_to_domain=self.domain_code)+ needed_ref = StuffSpec(concept=item_need.concept).to_bundle_representation(relative_to_domain=self.domain_code)+ msg = (+ f"In pipe '{self.code}', {step_label} batches over {batched_phrase} '{bound_ref}', but its pipe reads each item, "+ f"'{batch_params.input_item_stuff_name}', as '{needed_ref}'. Declare '{batch_params.input_item_stuff_name}' as '{item_ref}' in "+ f"pipe '{step_pipe_code}', or batch over a path that reaches a list of '{needed_ref}'."+ )+ raise PipeValidationError(+ message=msg,+ error_type=PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,+ domain_code=self.domain_code,+ pipe_code=self.code,+ variable_names=[batch_params.input_item_stuff_name],+ provided_concept_code=bound_spec.concept.concept_ref,+ required_concept_codes=[item_need.concept.concept_ref],+ )+@overridedef validate_output_static(self):pass@@ -417,7 +477,7 @@ class PipeSequence(PipeController):self._refuse_escaping_absence(taint_analysis=self._analyze_taint(visited_pipes=None, typed_flow=typed_flow))returnlast_step_concept = binding_spec.concept- last_step_label = f"the binding step {last_step.as_written}"+ last_step_label = f"the {last_step.label}"# A binding's own maybe-absence is the taint pass's to report, as its result's taint.is_last_step_output_optional = Falseeffective_last_step_output_multiplicity = binding_spec.multiplicity@@ -708,15 +768,12 @@ class PipeSequence(PipeController):origin_slot_name=root_taint.origin_slot_name,chain=(*root_taint.chain,- (- f"binding step {binding_step.as_written} may be skipped when '{binding_step.root_name}' is absent"- f" → slot '{binding_step.output_name}'"- ),+ (f"{binding_step.label} may be skipped when '{binding_step.root_name}' is absent → slot '{binding_step.output_name}'"),),)if derivation is not None and derivation.may_find_nothing:return SlotTaint(- source=f"binding step {binding_step.as_written}, whose path may find nothing at '{derivation.first_optional_path}'",+ source=f"{binding_step.label}, whose path may find nothing at '{derivation.first_optional_path}'",origin_slot_name=binding_step.output_name,)return None@@ -905,7 +962,7 @@ class PipeSequence(PipeController):).to_bundle_representation(relative_to_domain=self.domain_code)declared_ref = self.output.to_bundle_representation(relative_to_domain=self.domain_code)msg = (- f"In pipe '{self.code}', the binding step {binding_step.as_written} ends the sequence, whose output is declared "+ f"In pipe '{self.code}', the {binding_step.label} ends the sequence, whose output is declared "f"'{declared_ref}', but it binds '{bound_ref}'. A pipe that did not resolve at validation stored '{binding_step.root_name}', ""so the binding was derived when it ran, from the value it found. Declare the sequence's output as what the binding binds, ""or bind a path that reaches the declared output."diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py b/pipelex/pipe_controllers/sequence/pipe_sequence_factory.pyindex 2d79aa2e4..73274db6c 100644--- a/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py+++ b/pipelex/pipe_controllers/sequence/pipe_sequence_factory.py@@ -4,17 +4,34 @@ from typing_extensions import overridefrom pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecsfrom pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec-from pipelex.pipe_controllers.binding.binding_step import BindingStep+from pipelex.pipe_controllers.binding.binding_step import BindingStep, make_private_binding_namefrom pipelex.pipe_controllers.binding.binding_step_blueprint import BindingStepBlueprintfrom pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequencefrom pipelex.pipe_controllers.sequence.pipe_sequence_blueprint import PipeSequenceBlueprint+from pipelex.pipe_controllers.sub_pipe_blueprint import is_dotted_batch_overfrom pipelex.pipe_controllers.sub_pipe_factory import SubPipeFactoryfrom pipelex.pipe_machinery.pipe_factory import PipeFactoryProtocol+from pipelex.tools.misc.string_utils import get_root_from_dotted_pathif TYPE_CHECKING:from pipelex.pipe_controllers.sequence.sequence_typed_flow import SequenceStep+def _names_in_sequence(*, inputs: InputStuffSpecs, blueprint: PipeSequenceBlueprint) -> set[str]:+ """Every name the sequence's inputs and steps write or read, which a private name must not take."""+ names = set(inputs.root)+ for step in blueprint.steps:+ if isinstance(step, BindingStepBlueprint):+ names.update({step.root_name, step.result})+ continue+ for name in (step.result, step.batch_as):+ if name is not None:+ names.add(name)+ if step.batch_over is not None:+ names.add(get_root_from_dotted_path(step.batch_over))+ return names++class PipeSequenceFactory(PipeFactoryProtocol[PipeSequenceBlueprint, PipeSequence]):@classmethod@override@@ -30,13 +47,27 @@ class PipeSequenceFactory(PipeFactoryProtocol[PipeSequenceBlueprint, PipeSequencoutput: StuffSpec,blueprint: PipeSequenceBlueprint,) -> PipeSequence:+ """Build the runtime sequence, writing each pipe step's dotted `batch_over` as a binding followed by a batch.++ `{ pipe = "describe_page", batch_over = "catalog.pages", batch_as = "page" }` becomes a binding step of `catalog.pages`+ under a private name, followed by the same step batching over that name, so the dotted path is typed, walked, copied,+ recorded absent and drawn exactly as a binding step's `from` is.+ """sequential_sub_pipes: list[SequenceStep] = []+ taken_names = _names_in_sequence(inputs=inputs, blueprint=blueprint)for step in blueprint.steps:if isinstance(step, BindingStepBlueprint):sequential_sub_pipes.append(BindingStep(from_path=step.from_path, output_name=step.result))- else:- sequential_sub_pipes.append(SubPipeFactory.make_from_blueprint(blueprint=step))+ continue+ pipe_step = step+ if step.batch_over is not None and is_dotted_batch_over(batch_over=step.batch_over):+ private_name = make_private_binding_name(path=step.batch_over, taken_names=taken_names)+ taken_names.add(private_name)+ sequential_sub_pipes.append(BindingStep(from_path=step.batch_over, output_name=private_name, is_dotted_batch_over=True))+ # A copy, never validated again: the private name is the runtime's own, outside the names an author writes.+ pipe_step = step.model_copy(update={"batch_over": private_name})+ sequential_sub_pipes.append(SubPipeFactory.make_from_blueprint(blueprint=pipe_step))return PipeSequence(domain_code=domain_code,diff --git a/pipelex/pipe_controllers/sequence/sequence_typed_flow.py b/pipelex/pipe_controllers/sequence/sequence_typed_flow.pyindex b62d5253e..ad6748073 100644--- a/pipelex/pipe_controllers/sequence/sequence_typed_flow.py+++ b/pipelex/pipe_controllers/sequence/sequence_typed_flow.py@@ -88,6 +88,16 @@ class DerivedBinding(NamedTuple):stuff_spec: StuffSpec+def authored_step_number(*, steps: list[SequenceStep], step_index: int) -> int:+ """The number of a step, counting from 1, as its author wrote the sequence.++ The binding a sequence writes for a dotted `batch_over` is no step of the author's: it shares the number of the pipe+ step it was written for, which follows it.+ """+ rewritten_before = sum(1 for step in steps[:step_index] if isinstance(step, BindingStep) and step.is_dotted_batch_over)+ return step_index + 1 - rewritten_before++def resolve_library_concept(*, concept_library: ConceptLibraryAbstract, concept_ref: str) -> Concept | None:"""The library's concept for a ref, looked up directly, then among a dependency's aliased keys when one alone matches."""try:@@ -102,7 +112,7 @@ def resolve_library_concept(*, concept_library: ConceptLibraryAbstract, concept_def binding_path_unresolved_error(*, sequence_code: str, domain_code: str, binding_step: BindingStep, exc: BindingPathUnresolvedError) -> PipeValidationError:- msg = f"In pipe '{sequence_code}', the binding step {binding_step.as_written} cannot be derived. {exc}"+ msg = f"In pipe '{sequence_code}', the {binding_step.label} cannot be derived. {exc}"return PipeValidationError(message=msg,error_type=PipeValidationErrorType.BINDING_PATH_UNRESOLVED,@@ -260,7 +270,7 @@ def build_sequence_typed_flow(continuestep_writes = step_memory_writes(step=step, step_pipe=step_pipe, visited_pipes=visited_pipes)memory_writes_by_pipe_step[step_index] = step_writes- step_label = f"step {step_index + 1} (pipe '{step_pipe.code}')"+ step_label = f"step {authored_step_number(steps=steps, step_index=step_index)} (pipe '{step_pipe.code}')"for written_name, memory_write in step_writes.items():written_names.add(written_name)if memory_write.is_always_written:diff --git a/pipelex/pipe_controllers/sub_pipe.py b/pipelex/pipe_controllers/sub_pipe.pyindex 2aed0bb91..a6b844db4 100644--- a/pipelex/pipe_controllers/sub_pipe.py+++ b/pipelex/pipe_controllers/sub_pipe.py@@ -1,23 +1,20 @@-from typing import TYPE_CHECKING, cast+from typing import TYPE_CHECKING-from pydantic import BaseModel+from pydantic import BaseModel, field_validatorfrom pipelex import logfrom pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundErrorfrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.pipes.inputs.exceptions import InputStuffSpecNotFoundError, PipeRunInputsError+from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecsfrom pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpecfrom pipelex.core.pipes.variable_multiplicity import PresenceMarker, VariableMultiplicityfrom pipelex.core.stuffs.list_content import ListContent-from pipelex.core.stuffs.stuff_content import StuffContent-from pipelex.core.stuffs.stuff_factory import StuffFactoryfrom pipelex.interpreter_hub import get_pipe_router, get_required_pipefrom pipelex.pipe_controllers.batch.pipe_batch import PipeBatch-from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprintfrom pipelex.pipe_controllers.condition.pipe_condition import PipeConditionfrom pipelex.pipe_machinery.pipe_abstract import PipeAbstract-from pipelex.pipe_machinery.pipe_factory import PipeFactoryfrom pipelex.pipe_run.pipe_job_factory import PipeJobFactoryfrom pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunParams, output_multiplicity_to_applyfrom pipelex.system.job_metadata import JobMetadata@@ -33,6 +30,20 @@ class SubPipe(BaseModel):output_multiplicity: VariableMultiplicity | None = Nonebatch_params: BatchParams | None = None+ @field_validator("batch_params", mode="after")+ @classmethod+ def refuse_dotted_input_list(cls, batch_params: BatchParams | None) -> BatchParams | None:+ """Refuse a batch over a dotted path: the sequence holding the step rewrites a dotted `batch_over` into a binding step+ followed by a batch over the bound name, so a step only ever batches over a name in working memory.+ """+ if batch_params is not None and "." in batch_params.input_list_stuff_name:+ msg = (+ f"A step batches over the dotted path '{batch_params.input_list_stuff_name}': a step batches over a name in working memory, "+ "and the PipeSequence holding it binds a dotted `batch_over` to a name of its own first."+ )+ raise ValueError(msg)+ return batch_params+def result_spec(self, *, step_pipe: PipeAbstract) -> StuffSpec:"""The spec of what the step stores under its result, `step_pipe` being the pipe it runs, resolved as the run path resolves it.@@ -79,86 +90,6 @@ class SubPipe(BaseModel):if batch_params := self.batch_params:sub_pipe_run_params.batch_params = batch_params- # Pre-resolve dotted paths (e.g. "search_result.sources") before PipeBatch- synthetic_flat_name: str | None = None- input_list_stuff_name = batch_params.input_list_stuff_name- if "." in input_list_stuff_name:- try:- # wanted_type=None: the nested attribute may be a plain list (not ListContent), so skip type checking here- resolved = working_memory.get_typed_object_or_attribute(name=input_list_stuff_name, wanted_type=None)- except WorkingMemoryStuffNotFoundError as exc:- msg = (- f"Input list stuff named '{input_list_stuff_name}' required by sub_pipe '{self.pipe_code}' "- f"of pipe '{calling_pipe_code}' not found in working memory: {exc}"- )- raise PipeRunInputsError(- message=msg,- run_mode=sub_pipe_run_params.run_mode,- pipe_code=self.pipe_code,- variable_name=input_list_stuff_name,- concept_code=None,- ) from exc-- list_content: ListContent[StuffContent]- if isinstance(resolved, ListContent):- list_content = cast("ListContent[StuffContent]", resolved)- elif isinstance(resolved, list):- list_content = ListContent[StuffContent](items=cast("list[StuffContent]", resolved))- else:- msg = (- f"Dotted path '{input_list_stuff_name}' resolved to {type(resolved).__name__}, "- f"expected ListContent or list for batch_over in sub_pipe '{self.pipe_code}' "- f"of pipe '{calling_pipe_code}'"- )- raise PipeRunInputsError(- message=msg,- run_mode=sub_pipe_run_params.run_mode,- pipe_code=self.pipe_code,- variable_name=input_list_stuff_name,- concept_code=None,- )-- # Inject resolved list under a synthetic flat name and update batch_params- synthetic_flat_name = input_list_stuff_name.replace(".", "__")- if working_memory.is_stuff_exists(name=synthetic_flat_name):- msg = (- f"Cannot use synthetic name '{synthetic_flat_name}' for dotted-path batch resolution "- f"in sub_pipe '{self.pipe_code}' of pipe '{calling_pipe_code}': "- f"a stuff with that name already exists in working memory"- )- raise PipeRunInputsError(- message=msg,- run_mode=sub_pipe_run_params.run_mode,- pipe_code=self.pipe_code,- variable_name=synthetic_flat_name,- concept_code=None,- )- try:- item_stuff_spec = sub_pipe.inputs.get_required_stuff_spec(variable_name=batch_params.input_item_stuff_name)- except InputStuffSpecNotFoundError as exc:- msg = (- f"Batch input item named '{batch_params.input_item_stuff_name}' from '{calling_pipe_code}' is not "- f"in SubPipe '{self.pipe_code}' input stuff specs: {sub_pipe.inputs}"- )- raise PipeRunInputsError(- message=msg,- run_mode=sub_pipe_run_params.run_mode,- pipe_code=self.pipe_code,- variable_name=batch_params.input_item_stuff_name,- concept_code=None,- ) from exc- synthetic_stuff = StuffFactory.make_stuff(- concept=item_stuff_spec.concept,- content=list_content,- name=synthetic_flat_name,- )- working_memory.add_new_stuff(name=synthetic_flat_name, stuff=synthetic_stuff)- batch_params = BatchParams(- input_list_stuff_name=synthetic_flat_name,- input_item_stuff_name=batch_params.input_item_stuff_name,- )- sub_pipe_run_params.batch_params = batch_params-try:working_memory.get_typed_object_or_attribute(name=batch_params.input_list_stuff_name, wanted_type=ListContent)except WorkingMemoryStuffNotFoundError as exc:@@ -188,45 +119,30 @@ class SubPipe(BaseModel):variable_name=batch_params.input_item_stuff_name,concept_code=None,) from exc- pipe_batch_blueprint = PipeBatchBlueprint(+ # The batch is built as the runtime pipe, not from a blueprint: its list is a name of the calling sequence's memory,+ # a private one for a dotted `batch_over` the sequence rewrote, or any name a step stores under, never an input+ # name an author declares, so the input-name grammar a blueprint enforces does not apply to it.+ pipe_batch = PipeBatch(+ domain_code=sub_pipe.domain_code,+ # Derived from the resolved pipe's LOCAL code, not from `self.pipe_code` — that is a qualified ref+ # (`domain.foo`), and suffixing it would name `domain.foo_batch` while the domain is added a second time.+ code=f"{sub_pipe.code}_batch",description=f"Batch processing for {self.pipe_code}",+ inputs=InputStuffSpecs(root={batch_params.input_list_stuff_name: StuffSpec(concept=item_stuff_spec.concept)}),+ output=StuffSpec(concept=sub_pipe.output.concept),branch_pipe_code=self.pipe_code,- # The full concept_ref, not the bare code: the factory resolves a bare code in- # `domain_code` below (the sub-pipe's own domain), so a sub-pipe whose output concept- # lives in another domain would resolve to the wrong concept — or to nothing.- output=sub_pipe.output.concept.concept_ref,- input_list_name=batch_params.input_list_stuff_name,- input_item_name=batch_params.input_item_stuff_name,- inputs={- batch_params.input_list_stuff_name: item_stuff_spec.concept.concept_ref,- },+ batch_params=batch_params,)-- # Derived from the resolved pipe's LOCAL code, not from `self.pipe_code` — that is a- # qualified ref (`domain.foo`), and suffixing it would name `domain.foo_batch` while- # `domain_code` below adds the domain a second time.- pipe_batch_adhoc_pipe_code = f"{sub_pipe.code}_batch"- pipe_batch = PipeFactory[PipeBatch].make_from_blueprint(- domain_code=sub_pipe.domain_code,- pipe_code=pipe_batch_adhoc_pipe_code,- blueprint=pipe_batch_blueprint,- concept_codes_from_the_same_domain=[concept.code for concept in sub_pipe.concept_dependencies],+ pipe_output = await get_pipe_router().run(+ pipe_job=PipeJobFactory.make_pipe_job(+ pipe=pipe_batch,+ job_metadata=job_metadata,+ working_memory=working_memory,+ pipe_run_params=sub_pipe_run_params,+ output_name=self.output_name,+ library_crate=library_crate,+ ),)- try:- pipe_output = await get_pipe_router().run(- pipe_job=PipeJobFactory.make_pipe_job(- pipe=pipe_batch,- job_metadata=job_metadata,- working_memory=working_memory,- pipe_run_params=sub_pipe_run_params,- output_name=self.output_name,- library_crate=library_crate,- ),- )- finally:- # Clean up synthetic stuff injected for dotted-path resolution- if synthetic_flat_name:- working_memory.remove_stuff(name=synthetic_flat_name)# Case 2: Condition processingelif isinstance(sub_pipe, PipeCondition):pipe_output = await get_pipe_router().run(diff --git a/pipelex/pipe_controllers/sub_pipe_blueprint.py b/pipelex/pipe_controllers/sub_pipe_blueprint.pyindex 61014283f..d55039ff5 100644--- a/pipelex/pipe_controllers/sub_pipe_blueprint.py+++ b/pipelex/pipe_controllers/sub_pipe_blueprint.py@@ -3,10 +3,18 @@ from typing import Selffrom pydantic import BaseModel, ConfigDict, model_validatorfrom pipelex.core.pipes.exceptions import PipeValidationError+from pipelex.pipe_controllers.binding.binding_step_blueprint import PATH_GRAMMAR_DESCRIPTION+from pipelex.pipe_machinery.validation import check_name_is_not_reserved+from pipelex.tools.misc.string_utils import is_field_pathfrom pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_listfrom pipelex.validation_error_types import PipeValidationErrorType+def is_dotted_batch_over(*, batch_over: str) -> bool:+ """Whether a `batch_over` is a dotted path, which a PipeSequence binds before batching over the bound list."""+ return "." in batch_over++class SubPipeBlueprint(BaseModel):model_config = ConfigDict(extra="forbid")@@ -47,3 +55,29 @@ class SubPipeBlueprint(BaseModel):)return self++ @model_validator(mode="after")+ def validate_dotted_batch_over(self) -> Self:+ """Refuse a dotted `batch_over` outside the path grammar: it binds the list at that path, as a binding step's `from` does."""+ if self.batch_over is None or not is_dotted_batch_over(batch_over=self.batch_over) or is_field_path(path=self.batch_over):+ return self+ msg = (+ f"In pipe '{self.pipe}': the dotted `batch_over` '{self.batch_over}' is not a path. A dotted `batch_over` binds the list "+ f"at its path, as a binding step's `from` does, and a path is {PATH_GRAMMAR_DESCRIPTION}."+ )+ raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.BINDING_STEP_INVALID, variable_names=[self.batch_over])++ @model_validator(mode="after")+ def validate_reserved_names(self) -> Self:+ """Refuse a name taking the prefix the runtime reserves for the bound list of a dotted `batch_over`, on a sequence step and a+ parallel branch alike.++ A step stores its `result` and hands each item to its pipe under `batch_as`, both in working memory, and batches over a+ plain `batch_over` by name. A nested sequence binds in its caller's working memory, so a name of the caller taking the+ prefix could be overwritten by a list the nested sequence binds, and read back as another value.+ """+ named_fields = (("result", self.result), ("batch_as", self.batch_as), ("batch_over", self.batch_over))+ for field_name, name in named_fields:+ if name is not None:+ check_name_is_not_reserved(name=name, field_label=f"The `{field_name}` of the step running pipe '{self.pipe}'")+ return selfdiff --git a/pipelex/pipe_machinery/validation.py b/pipelex/pipe_machinery/validation.pyindex 6fadef812..7bdc5904d 100644--- a/pipelex/pipe_machinery/validation.py+++ b/pipelex/pipe_machinery/validation.py@@ -3,6 +3,7 @@ from collections.abc import Mappingfrom typing import NamedTuplefrom pipelex.base_exceptions import PipelexUnexpectedError+from pipelex.core.memory.working_memory import PRIVATE_BINDING_NAME_PREFIXfrom pipelex.core.pipes.exceptions import PipeValidationError, PipeVariableMultiplicityErrorfrom pipelex.core.pipes.variable_multiplicity import (PresenceMarker,@@ -17,6 +18,7 @@ from pipelex.core.pipes.variable_multiplicity import ()from pipelex.tools.misc.string_utils import (FIELD_PATH_PATTERN,+ FIELD_PATH_SEGMENT_REGEX,SNAKE_CASE_IDENTIFIER_REGEX,SNAKE_CASE_PATTERN,get_root_from_dotted_path,@@ -40,6 +42,22 @@ BINDING_RESULT_PATTERN = INPUT_NAME_PATTERN# whitespace and underscore-led segments are outside it.BINDING_PATH_PATTERN = FIELD_PATH_PATTERN+# The grammar of a PipeSequence pipe step's `batch_over`: a name with no dot, which names the list as a step or the+# sequence's inputs store it, or a dotted path following the binding path grammar, which the sequence binds before+# batching over the bound list. A dotted `batch_over` outside the path grammar is `binding_step_invalid`.+SEQUENCE_STEP_BATCH_OVER_PATTERN = rf"^(?:[^.]*|{FIELD_PATH_SEGMENT_REGEX}(?:\.{FIELD_PATH_SEGMENT_REGEX})+)$"++# The grammar of a PipeParallel branch's `batch_over`: a name with no dot. A dotted `batch_over` binds, and only a+# sequence's steps bind, so a branch carrying one is `binding_step_invalid`.+PARALLEL_BRANCH_BATCH_OVER_PATTERN = r"^[^.]*$"++# The names the runtime reserves: those taking the prefix a PipeSequence binds a dotted `batch_over`'s list under. A nested+# sequence binds in its caller's working memory, so a caller's name taking the prefix could be overwritten by that list.+# `check_name_is_not_reserved` refuses it on every name an author writes into working memory or batches over by name, the+# ones no other grammar already keeps underscore-free, and the MTHDS JSON Schema generator writes this pattern under `not`+# on the same fields, so a structural check refuses exactly what the runtime refuses.+RESERVED_NAME_PATTERN = f"^{re.escape(PRIVATE_BINDING_NAME_PREFIX)}"+# The marker of an input declaration as `_input_marker` writes it after the concept: an optional multiplicity# suffix (`[]` or `[N]`), then an optional presence symbol (`?` or `!`). Group 1 is the bracket content, group 2# the presence symbol.@@ -426,11 +444,13 @@ def validate_input_names(*, input_specs: Mapping[str, str]) -> None:raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.INVALID_INPUT_NAME, variable_names=[first_invalid_name])-def check_input_list_name(*, input_list_name: str) -> None:+def check_input_list_name(*, input_list_name: str, branch_pipe_code: str, input_item_name: str) -> None:"""Refuse a PipeBatch's `input_list_name` that is not a plain input name.It names one of the batch's own inputs, so it follows the input-name grammar: a list held in a field- of a larger value is declared by the batch under a plain name, and handed to it by the calling sequence.+ of a larger value is declared by the batch under a plain name, and handed to it by the calling sequence,+ which binds the field to that name, or runs the branch pipe in a step whose dotted `batch_over` binds the+ list and batches over it, in place of the PipeBatch.Raises:PipeValidationError: ``INVALID_INPUT_NAME`` naming the list.@@ -443,12 +463,41 @@ def check_input_list_name(*, input_list_name: str) -> None:f"`input_list_name` '{input_list_name}' is not a plain input name: a PipeBatch maps over a list it declares as an input "f'of its own, under a plain name. Declare the list itself (`{plain_name} = "<Concept>[]"`, with 'f'`input_list_name = "{plain_name}"`), and have the calling sequence bind the field to that name with a binding step '- f"({_binding_step(dotted_path=input_list_name)})."+ f"({_binding_step(dotted_path=input_list_name)}), or have the calling sequence run '{branch_pipe_code}' in a step that "+ f'batches over the field itself, which binds the list and batches over it, in place of the PipeBatch (`{{ pipe = "{branch_pipe_code}", '+ f'batch_over = "{input_list_name}", batch_as = "{input_item_name}" }}`).')else:msg = f"`input_list_name` '{input_list_name}' is not a valid input name: {_PLAIN_NAME_RULE}."raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.INVALID_INPUT_NAME, variable_names=[input_list_name])+def check_name_is_not_reserved(*, name: str, field_label: str) -> None:+ """Refuse a name an author writes that takes the prefix the runtime reserves for the bound list of a dotted `batch_over`.++ A PipeSequence binds a dotted `batch_over`'s list under a private name taking the prefix, `_bound_catalog_pages` for+ `catalog.pages`, and a nested sequence binds in the working memory of the sequence calling it. A caller's name taking+ the prefix could then be overwritten by the list a sequence it calls binds, so the prefix is the runtime's alone: a+ name an author stores a value under, hands an item to a pipe under, or batches over by name never takes it.++ Args:+ name: The name as written.+ field_label: The field holding it, for the message, such as "The `result` of the step running pipe 'describe_page'".++ Raises:+ PipeValidationError: ``INVALID_INPUT_NAME`` naming the name.+ """+ if not name.startswith(PRIVATE_BINDING_NAME_PREFIX):+ return+ unreserved_name = name.removeprefix(PRIVATE_BINDING_NAME_PREFIX)+ suggestion = f", such as '{unreserved_name}'" if is_snake_case(unreserved_name) else ""+ msg = (+ f"{field_label}, '{name}', takes the `{PRIVATE_BINDING_NAME_PREFIX}` prefix, which is reserved for the bound list of a dotted "+ f"`batch_over`: a PipeSequence binds that list under a `{PRIVATE_BINDING_NAME_PREFIX}` name in the working memory it shares "+ f"with the sequence calling it, so only the runtime writes or reads such a name. Choose another name{suggestion}."+ )+ raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.INVALID_INPUT_NAME, variable_names=[name])++def is_pipe_code_valid(pipe_code: str) -> bool:return is_snake_case(pipe_code)diff --git a/pipelex/pipeline/file_input_consumers.py b/pipelex/pipeline/file_input_consumers.pyindex f348c96c8..44ae0fa64 100644--- a/pipelex/pipeline/file_input_consumers.py+++ b/pipelex/pipeline/file_input_consumers.py@@ -9,8 +9,13 @@ resolving sub-pipes through the hub:that step. A step overwrites its result's name, and also whatever a nested sequence or conditionoutcome writes, since those run on the caller's memory; after a step that may write a name thewalk cannot know, nothing more is followed. A step's batch parameters map the list slot to its- item slot. A binding step that binds a single value makes its result stand for the entry path it- reads, a renamed copy for its root's own path; one gathering a list stops its result being followed.+ item slot. A binding step makes its result stand for where the value it binds comes from: a single+ value, or the list one field of a single root holds, for the entry path it reads, a renamed copy+ for its root's own path, and a list gathered across items whose last field is itself a list for+ every item of that field, `cases[].transcripts` for `cases.transcripts` over a list root. A list+ gathered from a field that is not a list, `cases.attachment`, has no path a frame can say, so its+ result stops being followed. A dotted `batch_over` is such a binding followed by a batch over its+ result.- **Parallel.** Every branch is visited.- **Batch.** The list slot maps to the item slot, and the branch pipe is visited.- **Condition.** Every outcome is visited, and whatever is found below it is conditional.@@ -329,14 +334,22 @@ def _visit_sequence(step_frame = dict(frame)for step_index, sub_pipe in enumerate(sequence.sequential_sub_pipes):if isinstance(sub_pipe, BindingStep):- # A binding stores a copy of the value at its path: a single value stands for the entry path it reads,- # while a list gathered across items has no one entry path, so it stops being followed, as an unknown would.+ # A binding stores a copy of the value at its path, and its result stands for where that value comes from, built+ # on the path its root stands for: a single value, a renamed copy or one list field of a single root is at the+ # entry path it reads, and a list whose last field is itself a list, gathered across the items of a list root or+ # of a list crossed before it, holds the items of that field, `("cases", "[]", "transcripts")` for+ # `cases.transcripts` over a list root. A batch over the result, a dotted `batch_over` included, then maps its+ # item to that path followed by the item segment, which covers a file at `cases[1].transcripts[0]`. A list+ # gathered from a field that is not a list, `cases.attachment`, holds values sitting at `cases[].attachment`+ # rather than at the items of one path, which a frame cannot say, since it maps a slot to the value at one path+ # and a batch's item to that path's items, so its result stops being followed.derivation = typed_flow.binding_derivations.get(step_index) if typed_flow is not None else None- bound_path = _tracked_path(frame=step_frame, variable_path=sub_pipe.from_path) if derivation and not derivation.is_plural else None- if bound_path is None:+ root_path = step_frame.get(sub_pipe.root_name)+ provenance_path = derivation.provenance_path(item_segment=LIST_ITEM_SEGMENT) if derivation is not None else None+ if root_path is None or provenance_path is None:step_frame.pop(sub_pipe.output_name, None)else:- step_frame[sub_pipe.output_name] = bound_path+ step_frame[sub_pipe.output_name] = (*root_path, *provenance_path)continuestep_pipe = _visit_sub_pipe(sub_pipe=sub_pipe, frame=step_frame, is_conditional=is_conditional, liftable_refs=liftable_refs, visiting=visiting, consumers=consumersdiff --git a/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalog_pages/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalog_pages/bundle.mthdsnew file mode 100644index 000000000..0fa466521--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalog_pages/bundle.mthds@@ -0,0 +1,32 @@+domain = "nursery_catalog"+description = "Writing one index line for every page of a plant nursery's catalog"+main_pipe = "index_catalog"++[concept.CatalogPage]+description = "A page of a printed catalog"++[concept.CatalogPage.structure]+title = { type = "text", description = "The title printed at the top of the page", required = true }++[concept.Catalog]+description = "A printed catalog of a plant nursery"++[concept.Catalog.structure]+season = { type = "text", description = "The season the catalog covers", required = true }+pages = { type = "list", item_type = "concept", item_concept_ref = "CatalogPage", description = "The pages of the catalog", required = true }++[pipe.index_catalog]+type = "PipeSequence"+description = "Writes one index line per page of the catalog, batching over the pages its field holds"+inputs = { catalog = "Catalog" }+output = "Text[]"+steps = [+ { pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "index_lines" },+]++[pipe.write_index_line]+type = "PipeCompose"+description = "Writes the index line for one page"+inputs = { page = "CatalogPage" }+output = "Text"+template = "Page: {{ page.title }}"diff --git a/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalog_pages/entry.toml b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalog_pages/entry.tomlnew file mode 100644index 000000000..3abbf3532--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalog_pages/entry.toml@@ -0,0 +1,11 @@+name = "feature_binding_step_batch_over_catalog_pages"+description = "One index line written for each page of a catalog, batching over the pages its field holds with a dotted batch_over"++validity = "valid"+tier = "offline"+granularity = "focused"++covers = [+ "feature.binding_step",+ "controller.batch",+]diff --git a/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalog_pages/inputs.json b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalog_pages/inputs.jsonnew file mode 100644index 000000000..57a1fc72d--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalog_pages/inputs.json@@ -0,0 +1,16 @@+{+ "catalog": {+ "concept": "nursery_catalog.Catalog",+ "content": {+ "season": "spring",+ "pages": [+ {+ "title": "Climbing roses"+ },+ {+ "title": "Fruit trees"+ }+ ]+ }+ }+}diff --git a/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalogs/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalogs/bundle.mthdsnew file mode 100644index 000000000..72a69d828--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalogs/bundle.mthds@@ -0,0 +1,32 @@+domain = "furniture_catalogs"+description = "Writing one index line for every page of a furniture maker's catalogs"+main_pipe = "index_catalogs"++[concept.CatalogPage]+description = "A page of a printed catalog"++[concept.CatalogPage.structure]+title = { type = "text", description = "The title printed at the top of the page", required = true }++[concept.Catalog]+description = "A printed catalog of a furniture maker"++[concept.Catalog.structure]+season = { type = "text", description = "The season the catalog covers", required = true }+pages = { type = "list", item_type = "concept", item_concept_ref = "CatalogPage", description = "The pages of the catalog", required = true }++[pipe.index_catalogs]+type = "PipeSequence"+description = "Writes one index line per page of every catalog, batching over the pages of all of them"+inputs = { catalogs = "Catalog[]" }+output = "Text[]"+steps = [+ { pipe = "write_index_line", batch_over = "catalogs.pages", batch_as = "page", result = "index_lines" },+]++[pipe.write_index_line]+type = "PipeCompose"+description = "Writes the index line for one page"+inputs = { page = "CatalogPage" }+output = "Text"+template = "Page: {{ page.title }}"diff --git a/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalogs/entry.toml b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalogs/entry.tomlnew file mode 100644index 000000000..771bca0ed--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalogs/entry.toml@@ -0,0 +1,11 @@+name = "feature_binding_step_batch_over_catalogs"+description = "One index line written for each page of several catalogs, a dotted batch_over over a list root batching over the flattened pages"++validity = "valid"+tier = "offline"+granularity = "focused"++covers = [+ "feature.binding_step",+ "controller.batch",+]diff --git a/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalogs/inputs.json b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalogs/inputs.jsonnew file mode 100644index 000000000..72df7e7f4--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_catalogs/inputs.json@@ -0,0 +1,26 @@+{+ "catalogs": {+ "concept": "furniture_catalogs.Catalog",+ "content": [+ {+ "season": "spring",+ "pages": [+ {+ "title": "Garden chairs"+ },+ {+ "title": "Parasols"+ }+ ]+ },+ {+ "season": "autumn",+ "pages": [+ {+ "title": "Bookcases"+ }+ ]+ }+ ]+ }+}diff --git a/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_equivalence/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_equivalence/bundle.mthdsnew file mode 100644index 000000000..e50a0deb9--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_equivalence/bundle.mthds@@ -0,0 +1,34 @@+domain = "order_pricing"+description = "Pricing every line of an order in the customer's currency, twice, to compare the two spellings of a batch over a field"+main_pipe = "price_order"++[concept.OrderLine]+description = "One line of an order"++[concept.OrderLine.structure]+article = { type = "text", description = "The article ordered", required = true }+amount = { type = "number", description = "What the line costs", required = true }++[concept.Order]+description = "An order placed by a customer"++[concept.Order.structure]+lines = { type = "list", item_type = "concept", item_concept_ref = "OrderLine", description = "The lines of the order", required = true }++[pipe.price_order]+type = "PipeSequence"+description = "Prices every line of an order with a dotted batch_over, then again with a binding followed by a batch over the bound name"+inputs = { order = "Order", currency = "Text" }+output = "Text[]"+steps = [+ { pipe = "price_line", batch_over = "order.lines", batch_as = "line", result = "priced_by_path" },+ { from = "order.lines", result = "lines" },+ { pipe = "price_line", batch_over = "lines", batch_as = "line", result = "priced_by_binding" },+]++[pipe.price_line]+type = "PipeCompose"+description = "Writes the price of one line in the customer's currency"+inputs = { line = "OrderLine", currency = "Text" }+output = "Text"+template = "{{ line.article }}: {{ line.amount }} {{ currency.text }}"diff --git a/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_equivalence/entry.toml b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_equivalence/entry.tomlnew file mode 100644index 000000000..7cb5009b5--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_equivalence/entry.toml@@ -0,0 +1,11 @@+name = "feature_binding_step_batch_over_equivalence"+description = "The lines of an order priced through a dotted batch_over and through a binding then a batch, both reading the customer's currency beside each line"++validity = "valid"+tier = "offline"+granularity = "focused"++covers = [+ "feature.binding_step",+ "controller.batch",+]diff --git a/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_equivalence/inputs.json b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_equivalence/inputs.jsonnew file mode 100644index 000000000..dfce78c88--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/feature_binding_step_batch_over_equivalence/inputs.json@@ -0,0 +1,18 @@+{+ "order": {+ "concept": "order_pricing.Order",+ "content": {+ "lines": [+ {+ "article": "Linen tea towel",+ "amount": 12.5+ },+ {+ "article": "Enamel mug",+ "amount": 9+ }+ ]+ }+ },+ "currency": "euros"+}diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_branch/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_branch/bundle.mthdsnew file mode 100644index 000000000..d66dbfdf5--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_branch/bundle.mthds@@ -0,0 +1,42 @@+domain = "catalog_printing"+description = "Preparing a catalog for the printer"+main_pipe = "prepare_catalog"++[concept.CatalogPage]+description = "A page of a printed catalog"++[concept.CatalogPage.structure]+title = { type = "text", description = "The title printed at the top of the page", required = true }++[concept.Catalog]+description = "A printed catalog"++[concept.Catalog.structure]+season = { type = "text", description = "The season the catalog covers", required = true }+pages = { type = "list", item_type = "concept", item_concept_ref = "CatalogPage", description = "The pages of the catalog", required = true }++[pipe.write_index_line]+type = "PipeCompose"+description = "Writes the index line for one page"+inputs = { page = "CatalogPage" }+output = "Text"+template = "Page: {{ page.title }}"++[pipe.write_cover_note]+type = "PipeCompose"+description = "Writes the note printed on the cover"+inputs = { catalog = "Catalog" }+output = "Text"+template = "Catalog for {{ catalog.season }}"++[pipe.prepare_catalog]+type = "PipeParallel"+description = "Writes the index lines and the cover note at the same time"+inputs = { catalog = "Catalog" }+output = "Composite"+add_each_output = true+# Wrong on purpose: a dotted batch_over binds, and a branch never binds; bind the pages in a sequence before the parallel.+branches = [+ { pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "index_lines" },+ { pipe = "write_cover_note", result = "cover_note" },+]diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_branch/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_branch/entry.tomlnew file mode 100644index 000000000..fcd21eed4--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_branch/entry.toml@@ -0,0 +1,12 @@+name = "invalid_binding_step_dotted_batch_over_branch"+description = "The pages of a catalog batched over by a parallel branch through a dotted batch_over, which only a sequence step may carry"++validity = "invalid"+tier = "static"+granularity = "focused"++covers = [+ "error.binding_step_invalid",+]++expected_error = "binding_step_invalid"diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_malformed/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_malformed/bundle.mthdsnew file mode 100644index 000000000..0cf34db2c--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_malformed/bundle.mthds@@ -0,0 +1,33 @@+domain = "catalog_layout"+description = "Writing the index of a printed catalog"+main_pipe = "index_catalog"++[concept.CatalogPage]+description = "A page of a printed catalog"++[concept.CatalogPage.structure]+title = { type = "text", description = "The title printed at the top of the page", required = true }++[concept.Catalog]+description = "A printed catalog"++[concept.Catalog.structure]+season = { type = "text", description = "The season the catalog covers", required = true }+pages = { type = "list", item_type = "concept", item_concept_ref = "CatalogPage", description = "The pages of the catalog", required = true }++[pipe.write_index_line]+type = "PipeCompose"+description = "Writes the index line for one page"+inputs = { page = "CatalogPage" }+output = "Text"+template = "Page: {{ page.title }}"++[pipe.index_catalog]+type = "PipeSequence"+description = "Writes one index line per page of the catalog"+inputs = { catalog = "Catalog" }+output = "Text[]"+# Wrong on purpose: a dotted batch_over is a path, field names separated by single dots, so `..` is not part of one.+steps = [+ { pipe = "write_index_line", batch_over = "catalog..pages", batch_as = "page", result = "index_lines" },+]diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_malformed/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_malformed/entry.tomlnew file mode 100644index 000000000..4575bf908--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_binding_step_dotted_batch_over_malformed/entry.toml@@ -0,0 +1,12 @@+name = "invalid_binding_step_dotted_batch_over_malformed"+description = "The pages of a catalog batched over through a dotted batch_over with an empty segment, which a path cannot hold"++validity = "invalid"+tier = "static"+granularity = "focused"++covers = [+ "error.binding_step_invalid",+]++expected_error = "binding_step_invalid"diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_dotted_batch_over/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_dotted_batch_over/bundle.mthdsnew file mode 100644index 000000000..c4abeda0a--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_dotted_batch_over/bundle.mthds@@ -0,0 +1,27 @@+domain = "supplier_labels"+description = "Writing the shelf labels for a supplier's invoice"+main_pipe = "label_invoice"++[concept.Invoice]+description = "An invoice received from a supplier"++[concept.Invoice.structure]+supplier_name = { type = "text", description = "The name of the supplier", required = true }+total = { type = "number", description = "The amount due, in euros", required = true }++[pipe.write_label]+type = "PipeCompose"+description = "Writes a shelf label for a supplier"+inputs = { supplier = "Text" }+output = "Text"+template = "Supplied by $supplier"++[pipe.label_invoice]+type = "PipeSequence"+description = "Writes a shelf label for the supplier of an invoice"+inputs = { invoice = "Invoice" }+output = "Text[]"+# Wrong on purpose: a dotted batch_over must reach a list, and the supplier's name is a single text.+steps = [+ { pipe = "write_label", batch_over = "invoice.supplier_name", batch_as = "supplier", result = "labels" },+]diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_dotted_batch_over/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_dotted_batch_over/entry.tomlnew file mode 100644index 000000000..092a4347a--- /dev/null+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch_dotted_batch_over/entry.toml@@ -0,0 +1,12 @@+name = "invalid_input_stuff_spec_mismatch_dotted_batch_over"+description = "A batch over the supplier's name of an invoice through a dotted batch_over, whose path reaches a single text rather than a list"++validity = "invalid"+tier = "static"+granularity = "focused"++covers = [+ "error.input_stuff_spec_mismatch",+]++expected_error = "input_stuff_spec_mismatch"diff --git a/pipelex/validation_error_types.py b/pipelex/validation_error_types.pyindex 915d7eaad..50677e7ee 100644--- a/pipelex/validation_error_types.py+++ b/pipelex/validation_error_types.py@@ -49,9 +49,11 @@ class PipeValidationErrorType(StrEnum):LLM_OUTPUT_CANNOT_BE_IMAGE = "llm_output_cannot_be_image"INVALID_PIPE_CODE_SYNTAX = "invalid_pipe_code_syntax"# An input name that is not a plain snake_case identifier, a dotted name (`invoice.total`) included,- # on any pipe's `inputs` or as a PipeBatch's `input_list_name`. Detected at blueprint parse time; a- # dotted name whose root the same `inputs` table declares carries the enrichment the fix planner- # turns into deleting the key.+ # on any pipe's `inputs` or as a PipeBatch's `input_list_name`; or a name taking the `_bound_` prefix+ # the runtime reserves for the bound list of a dotted `batch_over`, as a pipe step's or a parallel+ # branch's `result`, `batch_as` or plain `batch_over`, or a PipeBatch's `input_item_name`. Detected at+ # blueprint parse time; a dotted name whose root the same `inputs` table declares carries the+ # enrichment the fix planner turns into deleting the key.INVALID_INPUT_NAME = "invalid_input_name"UNKNOWN_PIPE_TYPE = "unknown_pipe_type"# A `[pipe.x]` section declared no `type` yet carries fields beyond the signature contractdiff --git a/subject_grants.toml b/subject_grants.tomlindex 56294bbee..11dc7df6f 100644--- a/subject_grants.toml+++ b/subject_grants.toml@@ -3552,6 +3552,10 @@ rationale = "Factory from source: builds the table from the template mapping; siparam = "schema"rationale = "Schema-pipeline stage transforming the schema dict; single operand."+["pipelex/language/mthds_schema_generator.py::_constrain_batch_over"]+param = "schema"+rationale = "One post-processing pass of the schema pipeline, called as a verb on the schema it rewrites, like its sibling passes."+["pipelex/language/mthds_schema_generator.py::_constrain_input_names"]param = "schema"rationale = "Schema-pipeline stage transforming the schema dict; single operand."@@ -3584,6 +3588,10 @@ rationale = "Verb-object: mutates the schema object's required list; field_namesparam = "schema"rationale = "Schema-pipeline stage transforming the schema dict; single operand."+["pipelex/language/mthds_schema_generator.py::_reserve_private_binding_names"]+param = "schema"+rationale = "One post-processing pass of the schema pipeline, called as a verb on the schema it rewrites, like its sibling passes."+["pipelex/language/mthds_schema_generator.py::_walk_schema"]param = "node"rationale = "Verb-object: walks the schema node tree; the visitor stays keyword."diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/test_pipe_batch_graph.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/test_pipe_batch_graph.pyindex 07f883299..e4be936ba 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/test_pipe_batch_graph.py+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_batch/test_pipe_batch_graph.py@@ -7,7 +7,10 @@ import pytestfrom pipelex import log, pretty_printfrom pipelex.config import get_config+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.memory.working_memory import PRIVATE_BINDING_NAME_PREFIXfrom pipelex.core.stuffs.document_content import DocumentContent+from pipelex.core.stuffs.list_content import ListContentfrom pipelex.graph.graph_factory import generate_graph_outputsfrom pipelex.graph.graphspec import EdgeKind, GraphSpec, NodeSpecfrom pipelex.pipeline.runner import PipelexMTHDSProtocol@@ -303,11 +306,11 @@ class TestPipeBatchGraph:log.info("Structural validation passed: BATCH_AGGREGATE edges correctly target PipeBatch")async def test_article_briefing_dotted_batch_over(self, pipe_run_mode: PipeRunMode):- """Test that the article_briefing pipeline with dotted-path batch_over runs end-to-end.+ """The article_briefing pipeline batches over `search_result.sources`, a list field of the search result, end to end.- This exercises batch_over="search_result.sources" where sources is a nested- attribute of the SearchResult stuff, validating the dotted-path resolution- implemented in SubPipe.run_pipe().+ A dotted `batch_over` is a binding followed by a batch: the sequence binds the list at that path under the private+ name `_bound_search_result_sources`, then batches the fetch over it, and the bound list stays in the run's working+ memory beside the search result it was read from."""runner = PipelexMTHDSProtocol(library_dirs=["tests/e2e/pipelex/pipes/pipe_controller/pipe_batch"],@@ -323,13 +326,14 @@ class TestPipeBatchGraph:assert pipe_output.working_memory is not Noneassert pipe_output.main_stuff is not None- # Verify the synthetic flat name was cleaned up after dotted-path batch processing+ # The binding the dotted `batch_over` was rewritten into holds the sources as a list of documentsfinal_memory = pipe_output.working_memory- assert final_memory.get_optional_stuff("search_result__sources") is None, (- "Synthetic flat name 'search_result__sources' should be cleaned up after batch processing"- )+ bound_sources = final_memory.get_optional_stuff(f"{PRIVATE_BINDING_NAME_PREFIX}search_result_sources")+ assert bound_sources is not None, "The bound list '_bound_search_result_sources' should be held in the run's working memory"+ assert isinstance(bound_sources.content, ListContent), f"The bound sources should be a list, got {type(bound_sources.content).__name__}"+ assert bound_sources.concept.concept_ref == NativeConceptCode.DOCUMENT.concept_ref- # Verify the original search_result remains in working memory+ # The search result the sources were read from stays in working memorysearch_result = final_memory.get_optional_stuff("search_result")assert search_result is not None, "search_result should remain in working memory after batch processing"diff --git a/tests/integration/pipelex/pipeline/test_file_input_consumers.py b/tests/integration/pipelex/pipeline/test_file_input_consumers.pyindex 55aa94ebd..11f7f57d4 100644--- a/tests/integration/pipelex/pipeline/test_file_input_consumers.py+++ b/tests/integration/pipelex/pipeline/test_file_input_consumers.py@@ -195,6 +195,90 @@ steps = [{ pipe = "extract_pdf", batch_over = "transcripts", batch_as = "transcrassert _summary(consumers["transcripts"]) == [("extract_pdf", "transcripts[]", False)]assert "transcript" not in consumers+ @pytest.mark.parametrize(+ ("root_name", "root_concept", "steps", "expected_summary"),+ [+ pytest.param(+ "case",+ "Case",+ ['{ pipe = "extract_pdf", batch_over = "case.transcripts", batch_as = "transcript", result = "pages" }'],+ [("extract_pdf", "case.transcripts[]", False)],+ id="one-list-field-of-a-single-root",+ ),+ pytest.param(+ "cases",+ "Case[]",+ ['{ pipe = "extract_pdf", batch_over = "cases.transcripts", batch_as = "transcript", result = "pages" }'],+ [("extract_pdf", "cases[].transcripts[]", False)],+ id="a-list-field-of-every-item-of-a-list-root",+ ),+ pytest.param(+ "cases",+ "Case[]",+ [+ '{ from = "cases.transcripts", result = "case_transcripts" }',+ '{ pipe = "extract_pdf", batch_over = "case_transcripts", batch_as = "transcript", result = "pages" }',+ ],+ [("extract_pdf", "cases[].transcripts[]", False)],+ id="an-explicit-binding-then-a-batch",+ ),+ pytest.param(+ "case",+ "Case",+ ['{ pipe = "extract_pdf", batch_over = "case.folders.transcripts", batch_as = "transcript", result = "pages" }'],+ [("extract_pdf", "case.folders[].transcripts[]", False)],+ id="a-list-field-of-every-item-of-a-list-field",+ ),+ pytest.param(+ "cases",+ "Case[]",+ ['{ pipe = "extract_pdf", batch_over = "cases.attachment", batch_as = "transcript", result = "pages" }'],+ [],+ id="a-single-field-gathered-across-items",+ ),+ ],+ )+ def test_a_bound_list_maps_its_items_to_where_they_come_from(+ self,+ load_empty_library: Callable[[], str],+ root_name: str,+ root_concept: str,+ steps: list[str],+ expected_summary: list[tuple[str, str, bool]],+ ):+ """A dotted `batch_over` binds its path, then batches over the bound list, as an explicit binding then a batch does.++ A list whose last field is itself a list holds the items of that field, wherever the path crossed a list before it, so+ the batch maps its item to every item of that field. A list gathered from a single field of every item has no path the+ walk can give its items, so it is not followed.+ """+ pipes = f"""+[concept.Folder]+description = "A folder of a case file"++[concept.Folder.structure]+transcripts = {{ type = "list", item_type = "concept", item_concept_ref = "native.Document", description = "Its transcripts", required = true }}++[concept.Case]+description = "A case file"++[concept.Case.structure]+transcripts = {{ type = "list", item_type = "concept", item_concept_ref = "native.Document", description = "Its transcripts", required = true }}+attachment = {{ type = "concept", concept_ref = "native.Document", description = "Its attachment", required = true }}+folders = {{ type = "list", item_type = "concept", item_concept_ref = "Folder", description = "Its folders", required = true }}++[pipe.main]+type = "PipeSequence"+description = "Extract every transcript of the case files"+inputs = {{ {root_name} = "{root_concept}" }}+output = "Page[]"+steps = [{", ".join(steps)}]+"""+ consumers = _consumers(load_empty_library=load_empty_library, domain="fic_bound_list", pipes=pipes)++ assert _summary(consumers.get(root_name, [])) == expected_summary+ assert set(consumers) <= {root_name}+def test_a_standalone_pipe_batch_maps_the_list_slot_to_its_item_slot(self, load_empty_library: Callable[[], str]):pipes = """[pipe.main]diff --git a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_consumer_check.py b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_consumer_check.pyindex cdca530de..49189b356 100644--- a/tests/integration/pipelex/pipeline/test_pipeline_run_setup_consumer_check.py+++ b/tests/integration/pipelex/pipeline/test_pipeline_run_setup_consumer_check.py@@ -64,6 +64,41 @@ output = "Page[]"model = "docling-extract-text""""+_CASES_HEADER = """+domain = "consumer_check_cases"+description = "Extract the transcripts of every case file"++[concept.Case]+description = "A case file"++[concept.Case.structure]+transcripts = { type = "list", item_type = "concept", item_concept_ref = "native.Document", description = "Its transcripts", required = true }++[pipe.extract_transcript]+type = "PipeExtract"+description = "Extract one transcript"+inputs = { transcript = "Document" }+output = "Page[]"+model = "pypdfium2-extract-pdf"++[pipe.extract_case_transcripts]+type = "PipeSequence"+description = "Extract every transcript of every case"+inputs = { cases = "Case[]" }+output = "Page[]"+"""++_CASES_DOTTED_MTHDS = (+ _CASES_HEADER + 'steps = [{ pipe = "extract_transcript", batch_over = "cases.transcripts", batch_as = "transcript", result = "pages" }]\n'+)++_CASES_EXPLICIT_MTHDS = (+ _CASES_HEADER + "steps = [\n"+ ' { from = "cases.transcripts", result = "case_transcripts" },\n'+ ' { pipe = "extract_transcript", batch_over = "case_transcripts", batch_as = "transcript", result = "pages" },\n'+ "]\n"+)+_CONDITIONAL_MTHDS = """domain = "consumer_check_conditional"description = "Extract a transcript in one mode only"@@ -168,6 +203,46 @@ class TestPipelineRunSetupConsumerCheck:assert "transcripts[1]" in str(exc_info.value)assert "transcripts[0]" not in str(exc_info.value)+ @pytest.mark.parametrize(+ "mthds_content",+ [+ pytest.param(_CASES_DOTTED_MTHDS, id="a-dotted-batch-over"),+ pytest.param(_CASES_EXPLICIT_MTHDS, id="an-explicit-binding-then-a-batch"),+ ],+ )+ async def test_a_word_transcript_of_one_case_among_pdf_transcripts_is_refused_before_the_run(self, mthds_content: str):+ """The transcripts of every case are bound into one list, then batched: the walk follows each item back to its case."""+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)+ pdf_url = _data_url(mime_type="application/pdf", raw_bytes=PDF_BYTES)+ docx_url = _data_url(mime_type="application/octet-stream", raw_bytes=DOCX_BYTES)++ with pytest.raises(PipelineInputFormatUnsupportedError) as exc_info:+ await pipeline_run_setup(+ storage_scope="test/scope",+ read_scope=None,+ user_id="test-user",+ execution_config=execution_config,+ mthds_contents=[mthds_content],+ pipe_code="extract_case_transcripts",+ inputs={+ "cases": {+ "concept": "consumer_check_cases.Case",+ "content": [+ {"transcripts": [{"url": pdf_url}]},+ {"transcripts": [{"url": docx_url}, {"url": pdf_url}]},+ ],+ }+ },+ )++ message = str(exc_info.value)+ assert (+ "Input 'cases[1].transcripts[0]' is a Word document (.docx): pipe 'extract_transcript' extracts it "+ "with model 'pypdfium2-extract-pdf', which reads PDF." in message+ )+ assert "cases[0].transcripts[0]" not in message+ assert "cases[1].transcripts[1]" not in message+async def test_pdf_transcripts_pass_setup(self):execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False)pdf_url = _data_url(mime_type="application/pdf", raw_bytes=PDF_BYTES)diff --git a/tests/integration/pipelex/pipes/controller/pipe_batch_dotted_path/dotted_batch.mthds b/tests/integration/pipelex/pipes/controller/pipe_batch_dotted_path/dotted_batch.mthdsdeleted file mode 100644index 489210784..000000000--- a/tests/integration/pipelex/pipes/controller/pipe_batch_dotted_path/dotted_batch.mthds+++ /dev/null@@ -1,11 +0,0 @@-domain = "dotted_batch_test"-description = "Branch pipe for dotted-path batch_over integration test"-main_pipe = "process_document"--[pipe.process_document]-type = "PipeLLM"-description = "Summarize a single document"-inputs = { document = "Document" }-output = "Text"-model = "$testing-text"-prompt = "Summarize this document: $document"diff --git a/tests/integration/pipelex/pipes/controller/pipe_batch_dotted_path/test_dotted_batch.py b/tests/integration/pipelex/pipes/controller/pipe_batch_dotted_path/test_dotted_batch.pydeleted file mode 100644index 1763d5247..000000000--- a/tests/integration/pipelex/pipes/controller/pipe_batch_dotted_path/test_dotted_batch.py+++ /dev/null@@ -1,107 +0,0 @@-"""Integration test for dotted-path batch_over resolution in SubPipe.run_pipe()."""--from pathlib import Path-from typing import Callable--import pytest--from pipelex import log, pretty_print-from pipelex.core.concepts.concept_factory import ConceptFactory-from pipelex.core.concepts.native.concept_native import NativeConceptCode-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.stuffs.document_content import DocumentContent-from pipelex.core.stuffs.search_result_content import SearchResultContent-from pipelex.core.stuffs.stuff_factory import StuffFactory-from pipelex.core.stuffs.text_content import TextContent-from pipelex.interpreter_hub import get_required_entry_pipe-from pipelex.pipe_controllers.sub_pipe import SubPipe-from pipelex.pipe_run.pipe_run_params import BatchParams-from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory-from pipelex.system.job_metadata import JobMetadata-from pipelex.system.pipe_run_mode import PipeRunMode--[email protected]_runnable[email protected](loop_scope="class")-class TestDottedBatchOver:- async def test_dotted_batch_resolves_nested_list(- self,- job_metadata: JobMetadata,- pipe_run_mode: PipeRunMode,- load_test_library: Callable[[list[Path]], None],- ):- """Test that SubPipe.run_pipe() resolves a dotted-path batch_over and iterates over the nested list.-- Hand-crafts working memory with a SearchResultContent containing DocumentContent sources,- then runs SubPipe with batch_params pointing to "search_result.sources". Verifies:- - The dotted path is resolved to the nested list- - PipeBatch executes one branch per list item- - The synthetic flat name is cleaned up afterward- """- load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_batch_dotted_path")])-- # Build a SearchResultContent with 3 DocumentContent sources- search_content = SearchResultContent(- answer="Mock search answer about testing",- sources=[- DocumentContent(url="https://example.com/doc1", title="Document 1"),- DocumentContent(url="https://example.com/doc2", title="Document 2"),- DocumentContent(url="https://example.com/doc3", title="Document 3"),- ],- )-- # Create Stuff wrapping the SearchResultContent- search_result_stuff = StuffFactory.make_stuff(- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.SEARCH_RESULT),- content=search_content,- name="search_result",- )- working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff=search_result_stuff)-- # Build SubPipe with dotted-path batch_params. `pipe_code` on a SubPipe is an in-body ref, so- # it carries the domain — this is what the crate qualification pass produces for a built pipe.- sub_pipe = SubPipe(- pipe_code="dotted_batch_test.process_document",- output_name="processed_docs",- batch_params=BatchParams(- input_list_stuff_name="search_result.sources",- input_item_stuff_name="document",- ),- )-- # Get the branch pipe to verify it exists- branch_pipe = get_required_entry_pipe(pipe_code="process_document")- log.info(f"Branch pipe: {branch_pipe.code}, inputs: {branch_pipe.inputs}")-- # Run SubPipe- pipe_output = await sub_pipe.run_pipe(- calling_pipe_code="test_sequence",- working_memory=working_memory,- job_metadata=job_metadata,- sub_pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),- )-- # Verify output- assert pipe_output is not None- assert pipe_output.working_memory is not None- assert pipe_output.main_stuff is not None-- pretty_print(pipe_output, title="SubPipe dotted batch output")-- # Verify synthetic flat name was cleaned up- final_memory = pipe_output.working_memory- assert final_memory.get_optional_stuff("search_result__sources") is None, (- "Synthetic flat name 'search_result__sources' should be cleaned up after batch processing"- )-- # Original search_result should still be in working memory- original = final_memory.get_optional_stuff("search_result")- assert original is not None, "search_result should remain in working memory"-- # Output should be a list with one processed item per source document- output_list = pipe_output.main_stuff_as_list(item_type=TextContent)- assert len(output_list.items) == 3, f"Expected 3 processed items (one per source), got {len(output_list.items)}"-- log.info(f"Dotted-path batch resolved and processed {len(output_list.items)} items successfully")diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_dotted_batch_over.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_dotted_batch_over.pynew file mode 100644index 000000000..5546694be--- /dev/null+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_dotted_batch_over.py@@ -0,0 +1,575 @@+from collections import Counter+from collections.abc import Callable+from typing import TYPE_CHECKING, Any++import pytest+from pydantic import ValidationError+from pytest_mock import MockerFixture++from pipelex.config import get_config+from pipelex.core.memory.working_memory import PRIVATE_BINDING_NAME_PREFIX, WorkingMemory+from pipelex.core.pipes.exceptions import PipeValidationError+from pipelex.core.stuffs.text_content import TextContent+from pipelex.graph.graph_tracer import GraphTracer+from pipelex.graph.graphspec import EdgeKind, GraphSpec, NodeKind, NodeSpec+from pipelex.interpreter_hub import get_library_manager+from pipelex.mthds_parsing.exceptions import MthdsParserError+from pipelex.mthds_parsing.parser import MthdsParser+from pipelex.pipe_controllers.batch.pipe_batch import PipeBatch+from pipelex.pipe_controllers.binding.binding_step import BindingStep+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence+from pipelex.pipe_controllers.sub_pipe import SubPipe+from pipelex.pipe_machinery.pipe_abstract import PipeAbstract+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_job_factory import PipeJobFactory+from pipelex.pipe_run.pipe_run_params import BatchParams+from pipelex.pipeline.pipeline_response import PipelexRunResultExecute+from pipelex.pipeline.runner import PipelexMTHDSProtocol+from pipelex.runtime_hub import scoped_event_log+from pipelex.system.pipe_run_mode import PipeRunMode+from pipelex.tracing.in_memory_event_log import InMemoryEventLog+from pipelex.validation_error_types import PipeValidationErrorType++if TYPE_CHECKING:+ from mthds.protocol.pipeline_inputs import PipelineInputs++_CATALOG_HEADER = """domain = "catalog_index"+description = "Writing one index line for every page of a printed catalog"+main_pipe = "index_pages"++[concept.CatalogPage]+description = "A page of a printed catalog"++[concept.CatalogPage.structure]+title = { type = "text", description = "The title printed on the page", required = true }++[concept.Catalog]+description = "A printed catalog"++[concept.Catalog.structure]+season = { type = "text", description = "The season the catalog covers", required = true }+pages = { type = "list", item_type = "concept", item_concept_ref = "CatalogPage", description = "The pages of the catalog", required = true }++[pipe.write_index_line]+type = "PipeCompose"+description = "Writes the index line of one page"+inputs = { page = "CatalogPage" }+output = "Text"+template = "Page: {{ page.title }}"++[pipe.make_catalog]+type = "PipeCompose"+description = "Writes out a catalog of a season from its pages"+inputs = { season = "Text", pages = "CatalogPage[]" }+output = "Catalog"++[pipe.make_catalog.construct]+season = { from = "season.text" }+pages = { from = "pages" }+"""++_SPRING_CATALOG = {"season": "spring", "pages": [{"title": "Garden chairs"}, {"title": "Parasols"}]}+_SUMMER_CATALOG = {"season": "summer", "pages": [{"title": "Deckchairs"}]}+++def _catalog_bundle(*, inputs: str, steps: list[str], output: str = "Text[]") -> str:+ """The catalog bundle with an `index_pages` sequence of these inputs, steps and output."""+ return (+ f"{_CATALOG_HEADER}\n"+ "[pipe.index_pages]\n"+ 'type = "PipeSequence"\n'+ 'description = "Writes one index line per page"\n'+ f"inputs = {inputs}\n"+ f'output = "{output}"\n'+ "steps = [\n" + "".join(f" {step},\n" for step in steps) + "]\n"+ )+++_SINGLE_ROOT_BUNDLE = _catalog_bundle(+ inputs='{ catalog = "Catalog" }',+ steps=['{ pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "index_lines" }'],+)+_LIST_ROOT_BUNDLE = _catalog_bundle(+ inputs='{ catalogs = "Catalog[]" }',+ steps=['{ pipe = "write_index_line", batch_over = "catalogs.pages", batch_as = "page", result = "index_lines" }'],+)+# The root's producer is a step of the sequence, so the graph has an edge from it to the binding.+_PRODUCED_ROOT_SUGAR_BUNDLE = _catalog_bundle(+ inputs='{ season = "Text", pages = "CatalogPage[]" }',+ steps=[+ '{ pipe = "make_catalog", result = "catalog" }',+ '{ pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "index_lines" }',+ ],+)+_PRODUCED_ROOT_EXPLICIT_BUNDLE = _catalog_bundle(+ inputs='{ season = "Text", pages = "CatalogPage[]" }',+ steps=[+ '{ pipe = "make_catalog", result = "catalog" }',+ '{ from = "catalog.pages", result = "catalog_pages" }',+ '{ pipe = "write_index_line", batch_over = "catalog_pages", batch_as = "page", result = "index_lines" }',+ ],+)+_PRODUCED_ROOT_INPUTS: "PipelineInputs" = {+ "season": "spring",+ "pages": {"concept": "catalog_index.CatalogPage", "content": _SPRING_CATALOG["pages"]},+}++_ORDER_BUNDLE_HEADER = """domain = "order_pricing"+description = "Pricing every line of an order in the customer's currency"+main_pipe = "price_order"++[concept.OrderLine]+description = "One line of an order"++[concept.OrderLine.structure]+article = { type = "text", description = "The article ordered", required = true }+amount = { type = "number", description = "What the line costs", required = true }++[concept.Order]+description = "An order placed by a customer"++[concept.Order.structure]+lines = { type = "list", item_type = "concept", item_concept_ref = "OrderLine", description = "The lines of the order", required = true }++[pipe.price_line]+type = "PipeCompose"+description = "Writes the price of one line in a currency"+inputs = { line = "OrderLine", currency = "Text" }+output = "Text"+template = "{{ line.article }}: {{ line.amount }} {{ currency.text }}"++[pipe.price_order]+type = "PipeSequence"+description = "Prices every line of an order"+inputs = { order = "Order", currency = "Text" }+output = "Text[]"+"""+++def _nested_binding_bundle(*, featured_lines_name: str) -> str:+ """A sequence storing the index lines of the featured pages under a name, then calling `index_pages`, which binds+ `catalog.pages` in the same working memory, then copying the featured lines it batches over by that name.+ """+ return _SINGLE_ROOT_BUNDLE.replace('main_pipe = "index_pages"', 'main_pipe = "index_featured"') + (+ "\n[pipe.copy_line]\n"+ 'type = "PipeCompose"\n'+ 'description = "Copies a line"\n'+ 'inputs = { line = "Text" }\n'+ 'output = "Text"\n'+ 'template = "Copy of $line"\n'+ "\n[pipe.index_featured]\n"+ 'type = "PipeSequence"\n'+ 'description = "Indexes the featured pages and the catalog, then copies the featured lines"\n'+ 'inputs = { catalog = "Catalog", featured = "Catalog" }\n'+ 'output = "Text[]"\n'+ "steps = [\n"+ f' {{ pipe = "write_index_line", batch_over = "featured.pages", batch_as = "page", result = "{featured_lines_name}" }},\n'+ ' { pipe = "index_pages", result = "index_lines" },\n'+ f' {{ pipe = "copy_line", batch_over = "{featured_lines_name}", batch_as = "line", result = "featured_lines" }},\n'+ "]\n"+ )+++def _load_sequence(*, mthds_content: str, library_id: str, pipe_code: str) -> PipeSequence:+ blueprint = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=mthds_content, mthds_source="flow.mthds")+ pipes = get_library_manager().load_from_blueprints(library_id=library_id, blueprints=[blueprint])+ sequences = [pipe for pipe in pipes if isinstance(pipe, PipeSequence) and pipe.code == pipe_code]+ assert len(sequences) == 1+ return sequences[0]+++async def _index_lines(*, mthds_content: str, inputs: "PipelineInputs") -> PipelexRunResultExecute:+ return await PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.LIVE).execute(mthds_contents=[mthds_content], inputs=inputs)+++def _node(graph: GraphSpec, *, pipe_code: str) -> NodeSpec:+ nodes = [node for node in graph.nodes if node.pipe_code == pipe_code]+ assert len(nodes) == 1, f"expected one node for '{pipe_code}', got {[(node.kind, node.pipe_code) for node in graph.nodes]}"+ return nodes[0]+++def _data_edges(graph: GraphSpec) -> set[tuple[str, str]]:+ return {(edge.source, edge.target) for edge in graph.edges if edge.kind == EdgeKind.DATA}+++def _graph_shape(graph: GraphSpec) -> tuple[Counter[tuple[str, str | None, str | None]], Counter[tuple[str, str | None, str | None]]]:+ """The nodes by kind, type and pipe code, and the edges by kind and the pipe codes they join: the graph without its names."""+ pipe_codes = {node.node_id: node.pipe_code for node in graph.nodes}+ nodes = Counter((str(node.kind), node.pipe_type, node.pipe_code) for node in graph.nodes)+ edges = Counter((str(edge.kind), pipe_codes.get(edge.source), pipe_codes.get(edge.target)) for edge in graph.edges)+ return nodes, edges+++async def _run_with_graph(*, mthds_content: str, inputs: "PipelineInputs") -> tuple[PipelexRunResultExecute, GraphSpec]:+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=True)+ runner = PipelexMTHDSProtocol(execution_config=execution_config, pipe_run_mode=PipeRunMode.LIVE)+ with scoped_event_log(InMemoryEventLog()):+ response = await runner.execute(mthds_contents=[mthds_content], inputs=inputs)+ graph = response.pipe_output.graph_spec+ assert graph is not None+ return response, graph++[email protected](loop_scope="class")+class TestDottedBatchOver:+ async def test_b1_a_single_root_runs_one_branch_per_page(self) -> None:+ response = await _index_lines(+ mthds_content=_SINGLE_ROOT_BUNDLE, inputs={"catalog": {"concept": "catalog_index.Catalog", "content": _SPRING_CATALOG}}+ )++ lines = [item.text for item in response.pipe_output.main_stuff_as_items(item_type=TextContent)]+ assert lines == ["Page: Garden chairs", "Page: Parasols"]++ async def test_b2_a_list_root_runs_over_the_flattened_pages(self) -> None:+ """Regression: a dotted `batch_over` over a list root raised, since the path was read without crossing the list."""+ response = await _index_lines(+ mthds_content=_LIST_ROOT_BUNDLE,+ inputs={"catalogs": {"concept": "catalog_index.Catalog", "content": [_SPRING_CATALOG, _SUMMER_CATALOG]}},+ )++ lines = [item.text for item in response.pipe_output.main_stuff_as_items(item_type=TextContent)]+ assert lines == ["Page: Garden chairs", "Page: Parasols", "Page: Deckchairs"]++ @pytest.mark.parametrize(+ ("mthds_content", "root_name", "root_multiplicity"),+ [+ pytest.param(_SINGLE_ROOT_BUNDLE, "catalog", None, id="a-single-root"),+ pytest.param(_LIST_ROOT_BUNDLE, "catalogs", True, id="a-list-root"),+ ],+ )+ async def test_b3_the_root_is_needed_as_its_own_concept(+ self, load_empty_library: Callable[[], str], mthds_content: str, root_name: str, root_multiplicity: bool | None+ ) -> None:+ """Regression: the sequence needed a dotted root as the batch item's concept, `CatalogPage`, with no multiplicity."""+ sequence = _load_sequence(mthds_content=mthds_content, library_id=load_empty_library(), pipe_code="index_pages")++ needed_inputs = sequence.needed_inputs()++ assert needed_inputs.variables == [root_name]+ root_need = needed_inputs.root[root_name]+ assert root_need.concept.concept_ref == "catalog_index.Catalog"+ assert root_need.multiplicity == root_multiplicity++ async def test_the_step_is_a_binding_under_a_private_name_followed_by_a_batch(self, load_empty_library: Callable[[], str]) -> None:+ sequence = _load_sequence(mthds_content=_SINGLE_ROOT_BUNDLE, library_id=load_empty_library(), pipe_code="index_pages")++ binding_step, pipe_step = sequence.sequential_sub_pipes+ assert isinstance(binding_step, BindingStep)+ assert binding_step.from_path == "catalog.pages"+ assert binding_step.is_dotted_batch_over is True+ assert binding_step.output_name == f"{PRIVATE_BINDING_NAME_PREFIX}catalog_pages"+ assert isinstance(pipe_step, SubPipe)+ assert pipe_step.batch_params is not None+ assert pipe_step.batch_params.input_list_stuff_name == binding_step.output_name+ assert pipe_step.batch_params.input_item_stuff_name == "page"+ assert sequence.build_typed_flow().binding_specs[0].multiplicity is True++ async def test_a_step_never_batches_over_a_dotted_path_itself(self) -> None:+ """Only the sequence's rewrite reads a dotted path, as a binding: a step built to batch over one is refused."""+ with pytest.raises(ValidationError) as exc_info:+ SubPipe(+ pipe_code="catalog_index.write_index_line",+ output_name="index_lines",+ batch_params=BatchParams(input_list_stuff_name="catalog.pages", input_item_stuff_name="page"),+ )++ assert "A step batches over the dotted path 'catalog.pages'" in str(exc_info.value)++ async def test_b4_a_binding_node_fed_by_the_root_producer_feeds_the_batch(self, mocker: MockerFixture) -> None:+ teardown_spy = mocker.spy(GraphTracer, "teardown")++ response, assembled_graph = await _run_with_graph(mthds_content=_PRODUCED_ROOT_SUGAR_BUNDLE, inputs=_PRODUCED_ROOT_INPUTS)++ assert [item.text for item in response.pipe_output.main_stuff_as_items(item_type=TextContent)] == ["Page: Garden chairs", "Page: Parasols"]+ in_process_graphs = [graph for graph in teardown_spy.spy_return_list if isinstance(graph, GraphSpec)]+ assert len(in_process_graphs) == 1+ for graph in (assembled_graph, in_process_graphs[0]):+ binding_node = _node(graph, pipe_code="catalog.pages")+ assert binding_node.kind == NodeKind.BINDING+ assert [io_spec.name for io_spec in binding_node.node_io.inputs] == ["catalog"]+ assert [io_spec.name for io_spec in binding_node.node_io.outputs] == [f"{PRIVATE_BINDING_NAME_PREFIX}catalog_pages"]+ producer_node = _node(graph, pipe_code="make_catalog")+ batch_node = _node(graph, pipe_code="write_index_line_batch")+ data_edges = _data_edges(graph)+ assert (producer_node.node_id, binding_node.node_id) in data_edges+ assert (binding_node.node_id, batch_node.node_id) in data_edges+ assert (producer_node.node_id, batch_node.node_id) not in data_edges++ async def test_b5_working_memory_never_holds_a_synthetic_name(self, mocker: MockerFixture) -> None:+ """Regression: the dotted path was resolved into a stuff named `catalog__pages`, of the item's concept, which no step produced."""+ batch_memory_keys: list[list[str]] = []+ make_pipe_job = PipeJobFactory.make_pipe_job++ def record_batch_memory(pipe: PipeAbstract, **kwargs: Any) -> PipeJob:+ # The keys as the batch starts, before the run adds anything to the memory or takes anything away.+ working_memory = kwargs.get("working_memory")+ if isinstance(pipe, PipeBatch) and isinstance(working_memory, WorkingMemory):+ batch_memory_keys.append(working_memory.list_keys())+ return make_pipe_job(pipe, **kwargs)++ mocker.patch.object(PipeJobFactory, "make_pipe_job", side_effect=record_batch_memory)++ response = await _index_lines(+ mthds_content=_SINGLE_ROOT_BUNDLE, inputs={"catalog": {"concept": "catalog_index.Catalog", "content": _SPRING_CATALOG}}+ )++ assert len(batch_memory_keys) == 1+ private_name = f"{PRIVATE_BINDING_NAME_PREFIX}catalog_pages"+ assert private_name in batch_memory_keys[0]+ final_keys = response.pipe_output.working_memory.list_keys()+ for keys in (batch_memory_keys[0], final_keys):+ assert "catalog__pages" not in keys+ assert not [key for key in keys if "__" in key]+ bound_pages = response.pipe_output.working_memory.get_stuff(private_name)+ assert bound_pages.concept.concept_ref == "catalog_index.CatalogPage"++ async def test_b6_the_sugar_and_the_explicit_binding_give_the_same_outputs_and_graph(self) -> None:+ sugar_response, sugar_graph = await _run_with_graph(mthds_content=_PRODUCED_ROOT_SUGAR_BUNDLE, inputs=_PRODUCED_ROOT_INPUTS)+ explicit_response, explicit_graph = await _run_with_graph(mthds_content=_PRODUCED_ROOT_EXPLICIT_BUNDLE, inputs=_PRODUCED_ROOT_INPUTS)++ sugar_lines = [item.text for item in sugar_response.pipe_output.main_stuff_as_items(item_type=TextContent)]+ explicit_lines = [item.text for item in explicit_response.pipe_output.main_stuff_as_items(item_type=TextContent)]+ assert sugar_lines == explicit_lines == ["Page: Garden chairs", "Page: Parasols"]+ sugar_memory = sugar_response.pipe_output.working_memory+ explicit_memory = explicit_response.pipe_output.working_memory+ sugar_pages = sugar_memory.get_stuff(f"{PRIVATE_BINDING_NAME_PREFIX}catalog_pages")+ explicit_pages = explicit_memory.get_stuff("catalog_pages")+ assert sugar_pages.concept.concept_ref == explicit_pages.concept.concept_ref == "catalog_index.CatalogPage"+ assert sugar_pages.content.model_dump() == explicit_pages.content.model_dump()+ assert _graph_shape(sugar_graph) == _graph_shape(explicit_graph)++ @pytest.mark.parametrize(+ "steps",+ [+ pytest.param(+ ['{ pipe = "price_line", batch_over = "order.lines", batch_as = "line", result = "priced_lines" }'],+ id="the-dotted-sugar",+ ),+ pytest.param(+ [+ '{ from = "order.lines", result = "lines" }',+ '{ pipe = "price_line", batch_over = "lines", batch_as = "line", result = "priced_lines" }',+ ],+ id="an-explicit-binding-then-a-batch",+ ),+ ],+ )+ async def test_a_batched_pipe_keeps_its_other_inputs_after_a_binding(self, load_empty_library: Callable[[], str], steps: list[str]) -> None:+ """Regression: a batched pipe's inputs besides its item were needed only when no earlier step stored the list, so after a+ binding the sequence never needed `currency`, and declaring it was refused as extraneous.+ """+ mthds_content = _ORDER_BUNDLE_HEADER + "steps = [\n" + "".join(f" {step},\n" for step in steps) + "]\n"+ sequence = _load_sequence(mthds_content=mthds_content, library_id=load_empty_library(), pipe_code="price_order")++ assert sorted(sequence.needed_inputs().variables) == ["currency", "order"]++ response = await PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.LIVE).execute(+ mthds_contents=[mthds_content],+ inputs={+ "order": {+ "concept": "order_pricing.Order",+ "content": {"lines": [{"article": "Tea towel", "amount": 12.5}, {"article": "Mug", "amount": 9}]},+ },+ "currency": "euros",+ },+ )+ lines = [item.text for item in response.pipe_output.main_stuff_as_items(item_type=TextContent)]+ assert lines == ["Tea towel: 12.5 euros", "Mug: 9.0 euros"]++ async def test_a_batched_pipe_keeps_its_other_inputs_after_a_pipe_step_stores_its_list(self, load_empty_library: Callable[[], str]) -> None:+ """Regression: the same fault without any binding, a pipe step storing the list, so the sequence never needed `shop`."""+ mthds_content = (+ _ORDER_BUNDLE_HEADER.replace(+ 'inputs = { order = "Order", currency = "Text" }', 'inputs = { lines = "OrderLine[]", currency = "Text", shop = "Text" }'+ )+ + "steps = [\n"+ + ' { pipe = "price_line", batch_over = "lines", batch_as = "line", result = "priced_lines" },\n'+ + ' { pipe = "sign_line", batch_over = "priced_lines", batch_as = "priced_line", result = "signed_lines" },\n'+ + "]\n\n"+ + '[pipe.sign_line]\ntype = "PipeCompose"\ndescription = "Signs a priced line with the shop name"\n'+ + 'inputs = { priced_line = "Text", shop = "Text" }\noutput = "Text"\ntemplate = "$priced_line, from $shop"\n'+ )+ sequence = _load_sequence(mthds_content=mthds_content, library_id=load_empty_library(), pipe_code="price_order")++ assert sorted(sequence.needed_inputs().variables) == ["currency", "lines", "shop"]++ response = await PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.LIVE).execute(+ mthds_contents=[mthds_content],+ inputs={+ "lines": {"concept": "order_pricing.OrderLine", "content": [{"article": "Mug", "amount": 9}]},+ "currency": "euros",+ "shop": "Ty Coz",+ },+ )+ lines = [item.text for item in response.pipe_output.main_stuff_as_items(item_type=TextContent)]+ assert lines == ["Mug: 9.0 euros, from Ty Coz"]++ async def test_an_absent_optional_root_batches_over_no_page(self) -> None:+ """An absent root lifts the binding, which binds an empty list, as a list result is never absent: the batch runs no branch."""+ mthds_content = _catalog_bundle(+ inputs='{ catalog = "Catalog?" }',+ steps=['{ pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "index_lines" }'],+ )++ response = await _index_lines(mthds_content=mthds_content, inputs={})++ assert response.pipe_output.main_stuff_as_items(item_type=TextContent) == []++ async def test_a_batch_over_a_result_that_is_not_a_plain_name_runs(self) -> None:+ """Regression: the batch a step runs was built from a blueprint, which refused any list name but a plain input name, so a+ batch over a result named `IndexLines`, which a pipe step may store under, failed when it ran.+ """+ mthds_content = _catalog_bundle(+ inputs='{ catalog = "Catalog" }',+ steps=[+ '{ pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "IndexLines" }',+ '{ pipe = "copy_line", batch_over = "IndexLines", batch_as = "line", result = "copied_lines" }',+ ],+ ).replace(+ "[pipe.make_catalog]",+ '[pipe.copy_line]\ntype = "PipeCompose"\ndescription = "Copies a line"\ninputs = { line = "Text" }\noutput = "Text"\n'+ 'template = "Copy of $line"\n\n[pipe.make_catalog]',+ )++ response = await _index_lines(+ mthds_content=mthds_content, inputs={"catalog": {"concept": "catalog_index.Catalog", "content": _SPRING_CATALOG}}+ )++ lines = [item.text for item in response.pipe_output.main_stuff_as_items(item_type=TextContent)]+ assert lines == ["Copy of Page: Garden chairs", "Copy of Page: Parasols"]++ async def test_two_dotted_batch_overs_of_one_path_take_two_private_names(self, load_empty_library: Callable[[], str]) -> None:+ mthds_content = _catalog_bundle(+ inputs='{ catalog = "Catalog" }',+ steps=[+ '{ pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "first_lines" }',+ '{ pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "index_lines" }',+ ],+ )+ sequence = _load_sequence(mthds_content=mthds_content, library_id=load_empty_library(), pipe_code="index_pages")++ private_names = [step.output_name for step in sequence.sequential_sub_pipes if isinstance(step, BindingStep)]+ private_name = f"{PRIVATE_BINDING_NAME_PREFIX}catalog_pages"+ assert private_names == [private_name, f"{private_name}_2"]++ async def test_a_caller_never_stores_under_the_name_a_nested_sequence_binds(self) -> None:+ """Regression: a nested sequence binds in its caller's working memory, so a caller's step storing its `result` under+ `_bound_catalog_pages` had it overwritten by the pages a nested `batch_over = "catalog.pages"` binds, and a later step+ batching over that name read the catalog's pages in place of the featured lines. The `_bound_` prefix is the runtime's.+ """+ reserved_name = f"{PRIVATE_BINDING_NAME_PREFIX}catalog_pages"+ with pytest.raises(MthdsParserError) as exc_info:+ MthdsParser.make_pipelex_bundle_blueprint(+ mthds_content=_nested_binding_bundle(featured_lines_name=reserved_name),+ mthds_source="flow.mthds",+ )++ errors = exc_info.value.validation_errors+ assert {error.error_type for error in errors} == {PipeValidationErrorType.INVALID_INPUT_NAME}+ assert {error.pipe_code for error in errors} == {"index_featured"}+ result_refusals = [error for error in errors if "The `result` of the step running pipe 'write_index_line'" in error.message]+ assert len(result_refusals) == 1+ assert result_refusals[0].variable_names == [reserved_name]+ assert (+ f"'{reserved_name}', takes the `_bound_` prefix, which is reserved for the bound list of a dotted `batch_over`"+ in result_refusals[0].message+ )+ assert "Choose another name, such as 'catalog_pages'." in result_refusals[0].message++ async def test_a_caller_name_outside_the_prefix_keeps_its_value_across_a_nested_binding(self) -> None:+ response = await _index_lines(+ mthds_content=_nested_binding_bundle(featured_lines_name="featured_index_lines"),+ inputs={+ "catalog": {"concept": "catalog_index.Catalog", "content": _SPRING_CATALOG},+ "featured": {"concept": "catalog_index.Catalog", "content": _SUMMER_CATALOG},+ },+ )++ featured_lines = [item.text for item in response.pipe_output.main_stuff_as_items(item_type=TextContent)]+ assert featured_lines == ["Copy of Page: Deckchairs"]+ working_memory = response.pipe_output.working_memory+ index_lines = working_memory.get_stuff_as_list("index_lines", item_type=TextContent)+ assert [item.text for item in index_lines.items] == ["Page: Garden chairs", "Page: Parasols"]+ nested_bound_pages = working_memory.get_stuff(f"{PRIVATE_BINDING_NAME_PREFIX}catalog_pages")+ assert nested_bound_pages.concept.concept_ref == "catalog_index.CatalogPage"++ @pytest.mark.parametrize(+ ("inputs", "steps", "error_type", "message_fragments"),+ [+ pytest.param(+ '{ catalog = "Catalog" }',+ ['{ pipe = "write_index_line", batch_over = "catalog.season", batch_as = "page", result = "index_lines" }'],+ PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,+ ["step 1 (pipe 'write_index_line') batches over the dotted path 'catalog.season', which derives a single 'Text', not a list"],+ id="a-path-deriving-a-single-value",+ ),+ pytest.param(+ '{ catalog = "Catalog" }',+ [+ '{ from = "catalog.season", result = "season" }',+ '{ pipe = "write_index_line", batch_over = "season", batch_as = "page", result = "index_lines" }',+ ],+ PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,+ [+ (+ "step 2 (pipe 'write_index_line') batches over 'season', which the binding step "+ '{ from = "catalog.season", result = "season" } binds as a single \'Text\', not a list'+ )+ ],+ id="an-explicit-binding-of-a-single-value",+ ),+ pytest.param(+ '{ catalog = "Catalog" }',+ [+ '{ from = "catalog.season", result = "season" }',+ '{ pipe = "write_title", batch_over = "catalog.pages", batch_as = "page", result = "index_lines" }',+ ],+ PipeValidationErrorType.INPUT_STUFF_SPEC_MISMATCH,+ [+ (+ "step 2 (pipe 'write_title') batches over the dotted path 'catalog.pages', which derives 'CatalogPage[]', but its pipe "+ "reads each item, 'page', as 'Number'. Declare 'page' as 'CatalogPage' in pipe 'write_title'"+ )+ ],+ id="items-read-as-another-concept",+ ),+ pytest.param(+ '{ catalog = "Catalog" }',+ ['{ pipe = "write_index_line", batch_over = "catalog.pagez", batch_as = "page", result = "index_lines" }'],+ PipeValidationErrorType.BINDING_PATH_UNRESOLVED,+ ['the dotted `batch_over = "catalog.pagez"` cannot be derived', "has no field 'pagez'. Its fields are: 'season', 'pages'."],+ id="a-path-the-structures-cannot-walk",+ ),+ pytest.param(+ '{ season = "Text" }',+ ['{ pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "index_lines" }'],+ PipeValidationErrorType.MISSING_INPUT_VARIABLE,+ [+ "the dotted `batch_over = \"catalog.pages\"` reads 'catalog', which is neither an input of the sequence",+ "Declare 'catalog' in the sequence's `inputs`",+ ],+ id="a-root-nowhere",+ ),+ ],+ )+ async def test_a_dotted_batch_over_is_refused_as_a_binding_is(+ self,+ load_empty_library: Callable[[], str],+ inputs: str,+ steps: list[str],+ error_type: PipeValidationErrorType,+ message_fragments: list[str],+ ) -> None:+ mthds_content = _catalog_bundle(inputs=inputs, steps=steps).replace(+ "[pipe.make_catalog]",+ '[pipe.write_title]\ntype = "PipeCompose"\ndescription = "Writes a title"\ninputs = { page = "Number" }\noutput = "Text"\n'+ 'template = "Number $page"\n\n[pipe.make_catalog]',+ )++ with pytest.raises(PipeValidationError) as exc_info:+ _load_sequence(mthds_content=mthds_content, library_id=load_empty_library(), pipe_code="index_pages")++ assert exc_info.value.error_type == error_type+ message = str(exc_info.value)+ for fragment in message_fragments:+ assert fragment in message, f"'{fragment}' not in: {message}"diff --git a/tests/integration/pipelex/test_extras/test_data.py b/tests/integration/pipelex/test_extras/test_data.pyindex 487dbc785..a5ce97c30 100644--- a/tests/integration/pipelex/test_extras/test_data.py+++ b/tests/integration/pipelex/test_extras/test_data.py@@ -1,7 +1,7 @@from typing import Any, ClassVar, NamedTuplefrom pipelex.core.memory.absence import AbsenceKind-from pipelex.core.memory.working_memory import MAIN_STUFF_NAME+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, PRIVATE_BINDING_NAME_PREFIXclass Bound(NamedTuple):@@ -125,6 +125,39 @@ class OfflineRunExpectations:content={"items": [{"text": "Photograph: Garden chairs on a lawn"}, {"text": "Photograph: Two parasols"}]},),},+ # A dotted `batch_over` binds its path under a private name, then batches over it: the bound list is kept there.+ "feature_binding_step_batch_over_catalog_pages": {+ f"{PRIVATE_BINDING_NAME_PREFIX}catalog_pages": Bound(+ concept_ref="nursery_catalog.CatalogPage", content={"items": [{"title": "Climbing roses"}, {"title": "Fruit trees"}]}+ ),+ "index_lines": Bound(concept_ref="native.Text", content={"items": [{"text": "Page: Climbing roses"}, {"text": "Page: Fruit trees"}]}),+ },+ "feature_binding_step_batch_over_catalogs": {+ f"{PRIVATE_BINDING_NAME_PREFIX}catalogs_pages": Bound(+ concept_ref="furniture_catalogs.CatalogPage",+ content={"items": [{"title": "Garden chairs"}, {"title": "Parasols"}, {"title": "Bookcases"}]},+ ),+ "index_lines": Bound(+ concept_ref="native.Text",+ content={"items": [{"text": "Page: Garden chairs"}, {"text": "Page: Parasols"}, {"text": "Page: Bookcases"}]},+ ),+ },+ "feature_binding_step_batch_over_equivalence": {+ f"{PRIVATE_BINDING_NAME_PREFIX}order_lines": Bound(+ concept_ref="order_pricing.OrderLine",+ content={"items": [{"article": "Linen tea towel", "amount": 12.5}, {"article": "Enamel mug", "amount": 9.0}]},+ ),+ "lines": Bound(+ concept_ref="order_pricing.OrderLine",+ content={"items": [{"article": "Linen tea towel", "amount": 12.5}, {"article": "Enamel mug", "amount": 9.0}]},+ ),+ "priced_by_path": Bound(+ concept_ref="native.Text", content={"items": [{"text": "Linen tea towel: 12.5 euros"}, {"text": "Enamel mug: 9.0 euros"}]}+ ),+ "priced_by_binding": Bound(+ concept_ref="native.Text", content={"items": [{"text": "Linen tea towel: 12.5 euros"}, {"text": "Enamel mug: 9.0 euros"}]}+ ),+ },"feature_binding_step_parallel_root": {"price_tag": Bound(concept_ref="market_stall.PriceTag", content={"produce": "Plougastel strawberries", "price": 3.2}),"price": Bound(concept_ref="native.Number", content={"number": 3.2}),diff --git a/tests/unit/pipelex/language/test_mthds_schema.py b/tests/unit/pipelex/language/test_mthds_schema.pyindex bca7be9e2..f06100366 100644--- a/tests/unit/pipelex/language/test_mthds_schema.py+++ b/tests/unit/pipelex/language/test_mthds_schema.py@@ -395,7 +395,77 @@ class TestMthdsSchemaGeneration:table = {**_minimal_pipe_table("PipeParallel"), "branches": [{"from": "invoice.total", "result": "total"}]}assert validator.is_valid(table) is Falsebranch_items = schema["definitions"]["PipeParallelBlueprint"]["properties"]["branches"]["items"]- assert branch_items == {"$ref": "#/definitions/SubPipeBlueprint"}+ assert branch_items == {"$ref": "#/definitions/ParallelBranchBlueprint"}+ branch_schema = dict(schema["definitions"]["ParallelBranchBlueprint"])+ pipe_step_schema = dict(schema["definitions"]["SubPipeBlueprint"])+ for step_schema in (branch_schema, pipe_step_schema):+ step_schema.pop("title")+ step_schema["properties"] = {name: field for name, field in step_schema["properties"].items() if name != "batch_over"}+ assert branch_schema == pipe_step_schema++ @pytest.mark.parametrize(+ ("batch_over", "is_valid_on_a_step", "is_valid_on_a_branch"),+ [+ pytest.param("pages", True, True, id="a-plain-name"),+ pytest.param("PagesOfTheCatalog", True, True, id="a-name-a-pipe-step-may-store-under"),+ pytest.param("catalog.pages", True, False, id="a-dotted-path"),+ pytest.param("catalogs.pages.page_view", True, False, id="a-deep-dotted-path"),+ pytest.param("catalog..pages", False, False, id="an-empty-segment"),+ pytest.param("catalog.pages[0]", False, False, id="a-subscript"),+ pytest.param("catalog._pages", False, False, id="an-underscore-led-segment"),+ pytest.param(".pages", False, False, id="a-leading-dot"),+ ],+ )+ def test_batch_over_is_a_path_on_a_sequence_step_and_a_plain_name_on_a_parallel_branch(+ self, schema: dict[str, Any], batch_over: str, is_valid_on_a_step: bool, is_valid_on_a_branch: bool+ ) -> None:+ """A dotted `batch_over` binds before it batches: it follows the path grammar on a sequence step, and a branch never binds.++ Both refusals are `binding_step_invalid`, which the runtime raises when the bundle is parsed and the schema refuses first.+ """+ validator = _pipe_union_oneof_validator(schema)+ pipe_step = {"pipe": "describe_page", "batch_over": batch_over, "batch_as": "page", "result": "descriptions"}+ sequence_table = {**_minimal_pipe_table("PipeSequence"), "steps": [pipe_step]}+ parallel_table = {**_minimal_pipe_table("PipeParallel"), "branches": [pipe_step]}+ assert validator.is_valid(sequence_table) is is_valid_on_a_step, f"sequence step batch_over {batch_over!r}"+ assert validator.is_valid(parallel_table) is is_valid_on_a_branch, f"parallel branch batch_over {batch_over!r}"++ @pytest.mark.parametrize(+ ("name", "should_validate"),+ [+ pytest.param("catalog_pages", True, id="a-plain-name"),+ pytest.param("bound_pages", True, id="the-prefix-without-its-underscore"),+ pytest.param("_draft", True, id="another-underscore-led-name"),+ pytest.param("_bound_catalog_pages", False, id="the-reserved-prefix"),+ pytest.param("_bound_", False, id="the-reserved-prefix-alone"),+ ],+ )+ @pytest.mark.parametrize("field_name", ["result", "batch_as", "batch_over"])+ def test_a_step_and_a_branch_refuse_the_reserved_prefix(self, schema: dict[str, Any], field_name: str, name: str, should_validate: bool) -> None:+ """A pipe step's `result`, `batch_as` and plain `batch_over` never take `_bound_`, the runtime's prefix for a dotted `batch_over`'s list.++ The runtime refuses them as `invalid_input_name` when the bundle is parsed, and the schema refuses them first, through a+ Draft-4 `not` on the string arm, on a PipeSequence step and on a PipeParallel branch alike.+ """+ validator = _pipe_union_oneof_validator(schema)+ pipe_step: dict[str, Any] = {"pipe": "describe_page", "batch_over": "pages", "batch_as": "page", "result": "descriptions", field_name: name}+ sequence_table = {**_minimal_pipe_table("PipeSequence"), "steps": [pipe_step]}+ parallel_table = {**_minimal_pipe_table("PipeParallel"), "branches": [pipe_step]}+ assert validator.is_valid(sequence_table) is should_validate, f"sequence step {field_name} {name!r}"+ assert validator.is_valid(parallel_table) is should_validate, f"parallel branch {field_name} {name!r}"++ @pytest.mark.parametrize(+ ("input_item_name", "should_validate"),+ [+ pytest.param("item", True, id="a-plain-name"),+ pytest.param("_draft_item", True, id="another-underscore-led-name"),+ pytest.param("_bound_item", False, id="the-reserved-prefix"),+ ],+ )+ def test_batch_input_item_name_refuses_the_reserved_prefix(self, schema: dict[str, Any], input_item_name: str, should_validate: bool) -> None:+ validator = _pipe_union_oneof_validator(schema)+ table = {**_minimal_pipe_table("PipeBatch"), "input_item_name": input_item_name}+ assert validator.is_valid(table) is should_validatedef test_minimal_table_coverage_matches_schema_pipe_kinds(self, schema: dict[str, Any]) -> None:"""Guard: the test's per-kind table map covers exactly the *concrete* pipe kinds in the schema.diff --git a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.py b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.pyindex b120cf13b..b1f0edd61 100644--- a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.py+++ b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.py@@ -84,3 +84,38 @@ class TestPipeBatchValidation:input_list_name="items",input_item_name="",)++ @pytest.mark.parametrize(+ ("input_list_name", "input_item_name", "reserved_name", "message_fragment"),+ [+ pytest.param(+ "items",+ "_bound_item",+ "_bound_item",+ "The PipeBatch's `input_item_name`, '_bound_item', takes the `_bound_` prefix, which is reserved for the bound list of a "+ "dotted `batch_over`",+ id="input-item-name",+ ),+ pytest.param("_bound_items", "item", "_bound_items", "is not a valid input name", id="input-list-name"),+ ],+ )+ def test_rejects_a_name_taking_the_reserved_prefix(+ self, input_list_name: str, input_item_name: str, reserved_name: str, message_fragment: str+ ) -> None:+ """The batch writes its item into the branch's memory under its own name, so the runtime's `_bound_` prefix is refused there too."""+ with pytest.raises(ValidationError) as exc_info:+ PipeBatchBlueprint(+ description="Process each item",+ inputs={"items": "Item[]"},+ output="Result[]",+ branch_pipe_code="process_item",+ input_list_name=input_list_name,+ input_item_name=input_item_name,+ )+ refusals = [raw_error.get("ctx", {}).get("error") for raw_error in exc_info.value.errors()]+ reserved_name_refusals = [+ refusal for refusal in refusals if isinstance(refusal, PipeValidationError) and refusal.variable_names == [reserved_name]+ ]+ assert len(reserved_name_refusals) == 1+ assert reserved_name_refusals[0].error_type == PipeValidationErrorType.INVALID_INPUT_NAME+ assert message_fragment in str(reserved_name_refusals[0])diff --git a/tests/unit/pipelex/pipe_controllers/binding/test_binding_derivation.py b/tests/unit/pipelex/pipe_controllers/binding/test_binding_derivation.pyindex 73807bf5e..d638dff87 100644--- a/tests/unit/pipelex/pipe_controllers/binding/test_binding_derivation.py+++ b/tests/unit/pipelex/pipe_controllers/binding/test_binding_derivation.py@@ -4,6 +4,9 @@ from pipelex.core.pipes.variable_multiplicity import VariableMultiplicityfrom pipelex.pipe_controllers.binding.binding_derivation import BindingRoot, BindingValueKind, derive_bindingfrom tests.unit.pipelex.pipe_controllers.binding.test_data import INVOICE, RESOLVER, make_root+# The segment a caller of `provenance_path` chooses to stand for any item of a list.+_ITEM = "[]"+class TestBindingDerivationTable:@pytest.mark.parametrize(@@ -91,6 +94,34 @@ class TestBindingDerivationTable:assert derivation.may_find_nothing is may_find_nothingassert derivation.first_optional_path == first_optional_path+ @pytest.mark.parametrize(+ ("path", "root", "provenance_path"),+ [+ # A single value sits at its path's fields+ ("invoice.total", INVOICE, ("total",)),+ ("invoice.supplier.address.city", INVOICE, ("supplier", "address", "city")),+ # A bare name is its root's own value, a list root's included+ ("invoice", INVOICE, ()),+ ("pages", make_root("native.Page", multiplicity=True), ()),+ # One list field of a single root is the list at that field's path+ ("invoice.lines", INVOICE, ("lines",)),+ # A list gathered across items whose last field is a list: the item segment follows a list root and every list crossed before+ ("shipments.parcels", make_root("billing.Shipment", multiplicity=True), (_ITEM, "parcels")),+ ("manifest.shipments.parcels", make_root("billing.Manifest"), ("shipments", _ITEM, "parcels")),+ ("manifests.shipments.parcels", make_root("billing.Manifest", multiplicity=3), (_ITEM, "shipments", _ITEM, "parcels")),+ # A list gathered from a field that is not itself a list holds values that are items of no list at one path+ ("pages.page_view", make_root("native.Page", multiplicity=True), None),+ ("invoice.lines.amount", INVOICE, None),+ ("invoice.pages.page_view", INVOICE, None),+ ],+ )+ def test_provenance_path_places_the_item_segment_after_every_list_crossed(+ self, path: str, root: BindingRoot, provenance_path: tuple[str, ...] | None+ ):+ derivation = derive_binding(path=path, root=root, resolver=RESOLVER)++ assert derivation.provenance_path(item_segment=_ITEM) == provenance_path+def test_segments_record_the_lists_they_cross(self):derivation = derive_binding(path="invoice.lines.amount", root=INVOICE, resolver=RESOLVER)diff --git a/tests/unit/pipelex/pipe_controllers/binding/test_data.py b/tests/unit/pipelex/pipe_controllers/binding/test_data.pyindex 80bcfc3ec..ab75c4b56 100644--- a/tests/unit/pipelex/pipe_controllers/binding/test_data.py+++ b/tests/unit/pipelex/pipe_controllers/binding/test_data.py@@ -64,6 +64,10 @@ CONCEPTS: dict[str, ConceptBlueprint | str] = {structure={"parcels": make_field(field_type=FieldType.LIST, item_type="concept", item_concept_ref="Parcel", required=True)},),"billing.Parcel": ConceptBlueprint(description="A parcel", structure={"weight": make_field(field_type=FieldType.NUMBER, required=True)}),+ "billing.Manifest": ConceptBlueprint(+ description="A manifest of shipments",+ structure={"shipments": make_field(field_type=FieldType.LIST, item_type="concept", item_concept_ref="Shipment", required=True)},+ ),"billing.Crate": ConceptBlueprint(description="A crate holding anything",structure={diff --git a/tests/unit/pipelex/pipe_controllers/sequence/test_sequence_step_union.py b/tests/unit/pipelex/pipe_controllers/sequence/test_sequence_step_union.pyindex c087d89c5..233ff8322 100644--- a/tests/unit/pipelex/pipe_controllers/sequence/test_sequence_step_union.py+++ b/tests/unit/pipelex/pipe_controllers/sequence/test_sequence_step_union.py@@ -3,6 +3,7 @@ from typing import Anyimport pytestfrom pydantic import ValidationError+from pipelex.core.pipes.exceptions import PipeValidationErrorfrom pipelex.mthds_parsing.exceptions import MthdsParserErrorfrom pipelex.mthds_parsing.parser import MthdsParserfrom pipelex.pipe_controllers.binding.binding_step_blueprint import BindingStepBlueprint@@ -167,6 +168,139 @@ branches = [assert errors[0].pipe_code == "acknowledge_in_parallel"assert "bind the value in a step of the calling PipeSequence, before the PipeParallel step" in errors[0].message+ def test_a_dotted_batch_over_parses_on_a_pipe_step(self) -> None:+ """The blueprint keeps the path as written: the sequence rewrites it into a binding and a batch when it loads."""+ sequence = _parse_sequence(+ steps_toml=' { pipe = "write_receipt", batch_over = "invoice.lines", batch_as = "total_amount", result = "receipts" },',+ )++ assert sequence.steps == [SubPipeBlueprint(pipe="write_receipt", batch_over="invoice.lines", batch_as="total_amount", result="receipts")]++ @pytest.mark.parametrize(+ "batch_over",+ [+ pytest.param("invoice..lines", id="an-empty-segment"),+ pytest.param("invoice.lines[0]", id="a-subscript"),+ pytest.param("invoice._lines", id="an-underscore-led-segment"),+ pytest.param("invoice.lines ", id="whitespace"),+ ],+ )+ def test_a_dotted_batch_over_outside_the_path_grammar_is_refused_as_binding_step_invalid(self, batch_over: str) -> None:+ step_toml = f' {{ pipe = "write_receipt", batch_over = "{batch_over}", batch_as = "total_amount", result = "receipts" }},'+ with pytest.raises(MthdsParserError) as exc_info:+ MthdsParser.make_pipelex_bundle_blueprint(mthds_content=_sequence_bundle(steps_toml=step_toml), mthds_source="main.mthds")++ errors = exc_info.value.validation_errors+ assert [error.error_type for error in errors] == [PipeValidationErrorType.BINDING_STEP_INVALID]+ assert errors[0].pipe_code == "acknowledge_invoice"+ assert f"the dotted `batch_over` '{batch_over}' is not a path" in errors[0].message++ @pytest.mark.parametrize(+ "batch_over",+ [+ pytest.param("invoice.lines", id="a-path"),+ pytest.param("invoice..lines", id="a-malformed-path"),+ ],+ )+ def test_a_dotted_batch_over_in_a_parallel_branch_is_refused(self, batch_over: str) -> None:+ """A dotted `batch_over` binds, and a branch never binds: the calling sequence binds the list before the parallel."""+ mthds_content = f"""{_BUNDLE_HEADER}steps = [{{ pipe = "acknowledge_in_parallel", result = "receipt" }}]++[pipe.acknowledge_in_parallel]+type = "PipeParallel"+description = "Writes receipts in parallel"+inputs = {{ invoice = "Invoice" }}+output = "Text"+add_each_output = true+branches = [+ {{ pipe = "write_receipt", batch_over = "{batch_over}", batch_as = "total_amount", result = "receipts" }},+]+"""+ with pytest.raises(MthdsParserError) as exc_info:+ MthdsParser.make_pipelex_bundle_blueprint(mthds_content=mthds_content, mthds_source="main.mthds")++ errors = exc_info.value.validation_errors+ assert [error.error_type for error in errors] == [PipeValidationErrorType.BINDING_STEP_INVALID]+ assert errors[0].pipe_code == "acknowledge_in_parallel"+ assert f"Branch 1 of the parallel batches over the dotted path '{batch_over}'" in errors[0].message+ assert f'`{{ from = "{batch_over}", result = "<name>" }}`' in errors[0].message++ def test_a_parallel_branch_built_with_a_dotted_batch_over_is_refused(self) -> None:+ """A branch built in Python, not parsed from a table, is refused the same way."""+ with pytest.raises(ValidationError) as exc_info:+ PipeParallelBlueprint(+ description="Writes receipts in parallel",+ output="Composite",+ branches=[SubPipeBlueprint(pipe="write_receipt", batch_over="invoice.lines", batch_as="line", result="receipts")],+ )++ refusal = exc_info.value.errors()[0].get("ctx", {}).get("error")+ assert isinstance(refusal, PipeValidationError)+ assert refusal.error_type == PipeValidationErrorType.BINDING_STEP_INVALID++ @pytest.mark.parametrize(+ ("step_toml", "field_name", "reserved_name"),+ [+ pytest.param('{ pipe = "write_receipt", result = "_bound_invoice_lines" }', "result", "_bound_invoice_lines", id="result"),+ pytest.param(+ '{ pipe = "write_receipt", batch_over = "lines", batch_as = "_bound_line", result = "receipts" }',+ "batch_as",+ "_bound_line",+ id="batch-as",+ ),+ pytest.param(+ '{ pipe = "write_receipt", batch_over = "_bound_invoice_lines", batch_as = "total_amount", result = "receipts" }',+ "batch_over",+ "_bound_invoice_lines",+ id="plain-batch-over",+ ),+ ],+ )+ @pytest.mark.parametrize("controller", ["sequence", "parallel"])+ def test_a_name_taking_the_reserved_prefix_is_refused_on_a_step_and_a_branch(+ self, controller: str, step_toml: str, field_name: str, reserved_name: str+ ) -> None:+ """A nested sequence binds a dotted `batch_over` in its caller's working memory, under a `_bound_` name no author may write."""+ pipe_code: str+ mthds_content: str+ if controller == "sequence":+ pipe_code = "acknowledge_invoice"+ mthds_content = _sequence_bundle(steps_toml=f" {step_toml},")+ else:+ pipe_code = "acknowledge_in_parallel"+ mthds_content = f"""{_BUNDLE_HEADER}steps = [{{ pipe = "acknowledge_in_parallel", result = "receipt" }}]++[pipe.acknowledge_in_parallel]+type = "PipeParallel"+description = "Writes receipts in parallel"+inputs = {{ invoice = "Invoice" }}+output = "Text"+add_each_output = true+branches = [+ {step_toml},+]+"""+ with pytest.raises(MthdsParserError) as exc_info:+ MthdsParser.make_pipelex_bundle_blueprint(mthds_content=mthds_content, mthds_source="main.mthds")++ errors = exc_info.value.validation_errors+ assert [error.error_type for error in errors] == [PipeValidationErrorType.INVALID_INPUT_NAME]+ assert errors[0].pipe_code == pipe_code+ assert errors[0].variable_names == [reserved_name]+ assert f"The `{field_name}` of the step running pipe 'write_receipt', '{reserved_name}', takes the `_bound_` prefix" in errors[0].message+ assert "which is reserved for the bound list of a dotted `batch_over`" in errors[0].message++ def test_a_plain_batch_over_in_a_parallel_branch_parses(self) -> None:+ blueprint = PipeParallelBlueprint.model_validate(+ {+ "description": "Writes receipts in parallel",+ "output": "Composite",+ "branches": [{"pipe": "write_receipt", "batch_over": "lines", "batch_as": "line", "result": "receipts"}],+ }+ )++ assert blueprint.branches == [SubPipeBlueprint(pipe="write_receipt", batch_over="lines", batch_as="line", result="receipts")]+def test_a_parallel_branch_keeps_the_pipe_step_shape(self) -> None:assert PipeParallelBlueprint.model_fields["branches"].annotation == list[SubPipeBlueprint]diff --git a/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py b/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.pyindex 7449ae34e..1a73e059a 100644--- a/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py+++ b/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py@@ -1,7 +1,9 @@import pytestfrom pydantic import ValidationError+from pipelex.core.pipes.exceptions import PipeValidationErrorfrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint+from pipelex.validation_error_types import PipeValidationErrorTypeclass TestSubPipeBlueprint:@@ -65,3 +67,46 @@ class TestSubPipeBlueprint:batch_over="items",batch_as="items",)++ @pytest.mark.parametrize(+ ("step_fields", "field_name", "reserved_name"),+ [+ pytest.param({"result": "_bound_catalog_pages"}, "result", "_bound_catalog_pages", id="result"),+ pytest.param({"batch_over": "catalog_pages", "batch_as": "_bound_page"}, "batch_as", "_bound_page", id="batch-as"),+ pytest.param({"batch_over": "_bound_catalog_pages", "batch_as": "page"}, "batch_over", "_bound_catalog_pages", id="plain-batch-over"),+ ],+ )+ def test_a_name_taking_the_reserved_prefix_is_refused_as_invalid_input_name(+ self, step_fields: dict[str, str], field_name: str, reserved_name: str+ ) -> None:+ """The `_bound_` prefix is the runtime's, for the list a dotted `batch_over` binds in a caller's working memory."""+ with pytest.raises(ValidationError) as exc_info:+ SubPipeBlueprint.model_validate({"pipe": "describe_page", **step_fields})++ refusal = exc_info.value.errors()[0].get("ctx", {}).get("error")+ assert isinstance(refusal, PipeValidationError)+ assert refusal.error_type == PipeValidationErrorType.INVALID_INPUT_NAME+ assert refusal.variable_names == [reserved_name]+ message = str(refusal)+ assert f"The `{field_name}` of the step running pipe 'describe_page', '{reserved_name}', takes the `_bound_` prefix" in message+ assert "reserved for the bound list of a dotted `batch_over`" in message+ assert "Choose another name" in message++ def test_the_refusal_suggests_the_name_without_the_prefix(self) -> None:+ with pytest.raises(ValidationError) as exc_info:+ SubPipeBlueprint(pipe="describe_page", result="_bound_catalog_pages")++ assert "Choose another name, such as 'catalog_pages'." in str(exc_info.value)++ @pytest.mark.parametrize(+ "name",+ [+ pytest.param("bound_pages", id="the-prefix-without-its-underscore"),+ pytest.param("_draft", id="another-underscore-led-name"),+ pytest.param("catalog_bound_pages", id="the-prefix-inside-the-name"),+ ],+ )+ def test_a_name_outside_the_reserved_prefix_is_accepted(self, name: str) -> None:+ blueprint = SubPipeBlueprint(pipe="describe_page", result=name, batch_over="pages", batch_as=name)++ assert blueprint.result == blueprint.batch_as == namediff --git a/tests/unit/pipelex/pipe_machinery/test_input_name_refusal.py b/tests/unit/pipelex/pipe_machinery/test_input_name_refusal.pyindex 448704c24..d9595b75e 100644--- a/tests/unit/pipelex/pipe_machinery/test_input_name_refusal.py+++ b/tests/unit/pipelex/pipe_machinery/test_input_name_refusal.py@@ -118,6 +118,7 @@ class TestInputNameRefusal:assert error.variable_names == ["catalog.pages"]assert error.redundant_input_name is Noneassert '{ from = "catalog.pages", result = "pages" }' in error.message+ assert '`{ pipe = "total_page", batch_over = "catalog.pages", batch_as = "page" }`' in error.messagedef test_a_dotted_batch_list_declared_as_an_input_is_refused(self) -> None:"""The other spelling of a dotted batch list: the list declared under its dotted name is refused by the shared input-name check."""diff --git a/tests/unit/pipelex/pipe_machinery/test_validation.py b/tests/unit/pipelex/pipe_machinery/test_validation.pyindex e045499e2..76cb24f00 100644--- a/tests/unit/pipelex/pipe_machinery/test_validation.py+++ b/tests/unit/pipelex/pipe_machinery/test_validation.py@@ -218,12 +218,14 @@ class TestInputNameValidation:assert "`[a-z][a-z0-9_]*`" in str(error)def test_a_plain_list_name_passes(self):- check_input_list_name(input_list_name="pages")+ check_input_list_name(input_list_name="pages", branch_pipe_code="describe_page", input_item_name="page")def test_a_dotted_list_name_is_refused(self):- """A PipeBatch's list is one of its own inputs, so a dotted list name is refused, naming the binding step."""+ """A PipeBatch's list is one of its own inputs, so a dotted list name is refused, naming both ways the calling sequence+ hands the field over: a binding step to the declared list, or a step whose dotted `batch_over` binds it and batches over it.+ """with pytest.raises(PipeValidationError) as exc_info:- check_input_list_name(input_list_name="catalog.pages")+ check_input_list_name(input_list_name="catalog.pages", branch_pipe_code="describe_page", input_item_name="page")error = exc_info.valueassert error.error_type == PipeValidationErrorType.INVALID_INPUT_NAME@@ -232,10 +234,11 @@ class TestInputNameValidation:assert "`input_list_name` 'catalog.pages' is not a plain input name" in messageassert '`input_list_name = "pages"`' in messageassert '{ from = "catalog.pages", result = "pages" }' in message+ assert '`{ pipe = "describe_page", batch_over = "catalog.pages", batch_as = "page" }`' in messagedef test_a_malformed_list_name_is_refused(self):with pytest.raises(PipeValidationError) as exc_info:- check_input_list_name(input_list_name="CatalogPages")+ check_input_list_name(input_list_name="CatalogPages", branch_pipe_code="describe_page", input_item_name="page")assert exc_info.value.error_type == PipeValidationErrorType.INVALID_INPUT_NAMEassert "is not a valid input name" in str(exc_info.value)