Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feature/Plain-stored-names · L-261009-677091 (#1387)

Louis Choquel7c4adc2cb042
diff --git a/.pipelex/plxt.toml b/.pipelex/plxt.toml
index afc14b3a8..17b6e5609 100644
--- a/.pipelex/plxt.toml
+++ b/.pipelex/plxt.toml
@@ -59,8 +59,13 @@ exclude = [
"**/mthds_corpus/entries/invalid_binding_step_pipe_and_from/*.mthds",
"**/mthds_corpus/entries/invalid_binding_step_result_not_plain/*.mthds",
"**/mthds_corpus/entries/invalid_binding_step_without_result/*.mthds",
+ "**/mthds_corpus/entries/invalid_input_name_batch_item_name_not_plain/*.mthds",
+ "**/mthds_corpus/entries/invalid_input_name_branch_batch_as_reserved/*.mthds",
+ "**/mthds_corpus/entries/invalid_input_name_branch_result_not_plain/*.mthds",
"**/mthds_corpus/entries/invalid_input_name_lone_dotted/*.mthds",
"**/mthds_corpus/entries/invalid_input_name_redundant_dotted/*.mthds",
+ "**/mthds_corpus/entries/invalid_input_name_step_batch_as_dotted/*.mthds",
+ "**/mthds_corpus/entries/invalid_input_name_step_result_not_plain/*.mthds",
"**/mthds_corpus/entries/invalid_missing_pipe_type/*.mthds",
"**/mthds_corpus/entries/invalid_unknown_pipe_type/*.mthds",
] # Glob patterns for files to ignore.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index abf595e03..7fc730cb2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@
### Changed
+- **Stored names are plain input names (Breaking)**: every name under which a step stores a value in working memory, a `PipeSequence` step's or a `PipeParallel` branch's `result`, the `batch_as` of either, and a `PipeBatch`'s `input_item_name`, must now be a plain input name matching `[a-z][a-z0-9_]*`, as a binding step's `result` already had to, and a name in any other form, such as `Pages`, `doc.pages` or `_pages`, is refused as `invalid_input_name` wherever a bundle loads, and by the MTHDS JSON Schema, which now carries the input-name pattern on these fields. Rename such a name to a plain one, such as `pages`, along with every input, binding path and `batch_over` that reads it; a plain `batch_over` is still refused only when it starts with the reserved `_bound_`.
- **Hosted-run guidance in messages and help**: the next step of a refused Pipelex API key names `pipelex login` and the working directory's `.env` rather than a shell export, which a `.env` file setting the key overrides, and `pipelex login` names `[run] execution = "hosted"` rather than `pipelex init` to make hosted runs the default. A CLI run missing a provider key, and the `pipelex doctor` report, offer the hosted Pipelex API as a way out; a failed `pipelex-agent` boot points at `pipelex-agent migrate` and `pipelex-agent init` rather than the interactive `pipelex init config`; and `FormerReleaseConfigError` carries its remedy, `pipelex migrate`, as its `user_action`. The `run` help of both CLIs names hosted runs, catalog ids, local method directories and TOML or inline inputs, and `pipelex-agent --help` describes the program rather than printing an internal note.
## [v0.79.1] - 2026-10-09
diff --git a/docs/building-methods/pipes/pipe-controllers/PipeBatch.md b/docs/building-methods/pipes/pipe-controllers/PipeBatch.md
index d3c86a056..ae0f68e3e 100644
--- a/docs/building-methods/pipes/pipe-controllers/PipeBatch.md
+++ b/docs/building-methods/pipes/pipe-controllers/PipeBatch.md
@@ -50,7 +50,7 @@ This is the batch arm of the "route or skip" pattern: batch over items with a co
| `output` | string | The output concept produced by the batch operation. | Yes |
| `branch_pipe_code` | string | The name of the single pipe to execute for each item in the input list. | Yes |
| `input_list_name` | string | The name of the input list to iterate over. Must match one of the keys in `inputs`, so it is a plain input name, a `snake_case` identifier matching `[a-z][a-z0-9_]*`, and never a path into a field: a dotted name such as `catalog.pages` is refused as `invalid_input_name`. To map a pipe over a list held in a field, have the calling sequence bind it to the batch's list with a binding step, or run the branch pipe in a sequence step whose dotted `batch_over` binds the list and batches over it (see [Batching over a field](PipeSequence.md#batching-over-a-field)). Typically a plural noun (e.g. `articles`). | Yes |
-| `input_item_name` | string | The name that an individual item from the list will have inside its execution branch — this is how the branch pipe finds its input. Must differ from `input_list_name` and from every key in `inputs`, and never starts with `_bound_`, a prefix reserved for the bound list of a dotted `batch_over` (see [Batching over a field](PipeSequence.md#batching-over-a-field)), which is refused as `invalid_input_name`. Typically the singular form of the list name (e.g. `article`). | Yes |
+| `input_item_name` | string | The name that an individual item from the list will have inside its execution branch — this is how the branch pipe finds its input. It is a [stored name](PipeSequence.md#stored-names), so a plain input name like `input_list_name`, a `snake_case` identifier matching `[a-z][a-z0-9_]*`: a name in any other form, such as `Article`, `issue.article` or `_bound_article`, is refused as `invalid_input_name`. Must differ from `input_list_name` and from every key in `inputs`. Typically the singular form of the list name (e.g. `article`). | Yes |
### Example: Summarizing a list of articles
diff --git a/docs/building-methods/pipes/pipe-controllers/PipeParallel.md b/docs/building-methods/pipes/pipe-controllers/PipeParallel.md
index daec6878e..1ea409563 100644
--- a/docs/building-methods/pipes/pipe-controllers/PipeParallel.md
+++ b/docs/building-methods/pipes/pipe-controllers/PipeParallel.md
@@ -62,7 +62,9 @@ Each entry in the `branches` array is a table with the following keys:
| Key | Type | Description | Required |
| -------- | ------ | ---------------------------------------------------------------------------------------- | -------- |
| `pipe` | string | The name of the pipe to execute for this branch. | Yes |
-| `result` | string | The name for this branch's output. Must be unique within the `PipeParallel` definition. | Yes |
+| `result` | string | The name for this branch's output, a plain `snake_case` input name such as `sentiment`. Must be unique within the `PipeParallel` definition. | Yes |
+
+A branch's `result` and `batch_as` are [stored names](PipeSequence.md#stored-names), as on a sequence step: the branch stores a value under each for a pipe to read through an input, so each is a plain input name matching `[a-z][a-z0-9_]*`, and one in any other form, such as `Sentiment` or `text.sentiment`, is refused with `invalid_input_name`. Since the combined output holds each branch result under its `result` name, the fields of a structured output concept that the branches fill have plain names too.
A branch is always a pipe step. A [binding step](PipeSequence.md#binding-steps), `{ from = "invoice.total", result = "total_amount" }`, is refused in `branches` with `binding_step_invalid`: the branches run at once, and a binding orders a value before the steps that read it, so bind the value in the calling sequence, in a step before the `PipeParallel`, and have the branch read the bound name. A branch may carry `batch_over` and `batch_as`, but its `batch_over` is a plain name: a dotted one, such as `catalog.pages`, binds before it batches (see [Batching over a field](PipeSequence.md#batching-over-a-field)), so it is refused with `binding_step_invalid` too, and the calling sequence binds the list before the `PipeParallel`.
diff --git a/docs/building-methods/pipes/pipe-controllers/PipeSequence.md b/docs/building-methods/pipes/pipe-controllers/PipeSequence.md
index db1d93e15..c37b2aeea 100644
--- a/docs/building-methods/pipes/pipe-controllers/PipeSequence.md
+++ b/docs/building-methods/pipes/pipe-controllers/PipeSequence.md
@@ -39,11 +39,17 @@ A pipe step is a table with the following keys:
| Key | Type | Description | Required |
| -------- | ------ | ------------------------------------------------------------------ | -------- |
| `pipe` | string | The name of the pipe to execute for this step. | Yes |
-| `result` | string | The name to give to this step's output in the working memory. When omitted, the output is stored only in the unnamed `main_stuff` slot (the default output), so later steps can pick it up as their implicit input but cannot reference it by a dedicated name. | No |
+| `result` | string | The name to give to this step's output in the working memory, a plain `snake_case` input name such as `pages` (see [Stored names](#stored-names)). When omitted, the output is stored only in the unnamed `main_stuff` slot (the default output), so later steps can pick it up as their implicit input but cannot reference it by a dedicated name. | No |
| `nb_output` | integer | Request a fixed number of outputs from this step's pipe. Cannot be combined with `multiple_output`. | No |
| `multiple_output` | boolean | Request a variable number of outputs from this step's pipe (the model decides how many). Cannot be combined with `nb_output`. | No |
| `batch_over` | string | The list in the working memory to batch this step over, running the pipe once per item: a name, or a dotted path to a list held in a field, such as `catalog.pages`, which is a binding followed by a batch (see [Batching over a field](#batching-over-a-field)). Must be provided together with `batch_as`. See [Understanding Multiplicity](../understanding-multiplicity.md). | No |
-| `batch_as` | string | The name each item takes in the working memory during a `batch_over` run. Must differ from `batch_over` (e.g. `batch_over = "items"`, `batch_as = "item"`). Like `result` and `batch_over`, it never starts with `_bound_`, a prefix reserved for the bound list of a dotted `batch_over`. | No |
+| `batch_as` | string | The name each item takes in the working memory during a `batch_over` run, under which the step's pipe reads it through an input: a plain `snake_case` input name, like `result` (see [Stored names](#stored-names)). Must differ from `batch_over` (e.g. `batch_over = "items"`, `batch_as = "item"`). | No |
+
+#### Stored names
+
+A step's `result` and `batch_as` are **stored names**: the step stores a value in the working memory under each, for a later pipe to read through an input. An input name is always a plain `snake_case` identifier matching `[a-z][a-z0-9_]*`, so a stored name takes the same form, and every value a step stores can be read. A `result` or a `batch_as` in any other form, such as `Pages`, a dotted `doc.pages` or an underscore-led `_pages`, is refused with `invalid_input_name`, which the schema catches too, and the message suggests a plain name. Rename it, such as to `pages`, along with every input, binding path and `batch_over` that reads it. The same rule holds a [`PipeParallel`](PipeParallel.md) branch's `result` and `batch_as`, a [`PipeBatch`](PipeBatch.md)'s `input_item_name` and a binding step's `result`, which a binding step that breaks it reports as `binding_step_invalid`.
+
+A plain `batch_over` reads a name rather than storing one, so it is not held to this form, but it never starts with `_bound_`, a prefix reserved for the list a dotted `batch_over` binds (see [Batching over a field](#batching-over-a-field)), and one that does is refused with `invalid_input_name` too.
#### Binding steps
@@ -197,7 +203,7 @@ So a dotted `batch_over` shares everything a binding does:
- A path outside the path grammar, such as `catalog..pages`, is refused with `binding_step_invalid`, which the schema catches too.
- In the execution graph, the binding is a node of kind `binding`, fed by the root's producer, and it feeds the batch.
-The private name starts with an underscore, `_bound_catalog_pages` for `catalog.pages`, so no template or input can read it: it shows only in the working memory the run leaves and in the graph. The `_bound_` prefix is reserved for these names. A nested `PipeSequence` binds in the working memory of the sequence calling it, so a name of the caller taking the prefix could be overwritten by the list a sequence it calls binds: a pipe step's `result`, `batch_as` or plain `batch_over`, on a sequence step or a `PipeParallel` branch, and a [`PipeBatch`](PipeBatch.md)'s `input_item_name` that starts with `_bound_` are refused with `invalid_input_name`, which the schema catches too, and the message asks for another name. Only a sequence's steps bind, so a [`PipeParallel`](PipeParallel.md) branch carries a plain `batch_over` only, and a dotted one there is refused with `binding_step_invalid`: bind the list in a step before the `PipeParallel`, and batch the branch over the bound name.
+The private name starts with an underscore, `_bound_catalog_pages` for `catalog.pages`, so no template or input can read it: it shows only in the working memory the run leaves and in the graph. The `_bound_` prefix is reserved for these names. A nested `PipeSequence` binds in the working memory of the sequence calling it, so a name of the caller taking the prefix could be overwritten by the list a sequence it calls binds. No name a step stores a value under can take it, since a [stored name](#stored-names) is a plain input name, which never starts with an underscore, and a plain `batch_over` that starts with `_bound_`, on a sequence step or a `PipeParallel` branch, is refused with `invalid_input_name`, which the schema catches too, and the message asks for another name. Only a sequence's steps bind, so a [`PipeParallel`](PipeParallel.md) branch carries a plain `batch_over` only, and a dotted one there is refused with `binding_step_invalid`: bind the list in a step before the `PipeParallel`, and batch the branch over the bound name.
### Example
diff --git a/docs/building-methods/pipes/working-memory.md b/docs/building-methods/pipes/working-memory.md
index 414e40905..668309710 100644
--- a/docs/building-methods/pipes/working-memory.md
+++ b/docs/building-methods/pipes/working-memory.md
@@ -35,6 +35,8 @@ How does data get from `generate_tagline` to `extract_keywords_from_tagline`? Th
This mechanism allows you to chain pipes together, creating a flow of information through your pipeline.
+Because a later pipe reads a stored value through an input, every name a value is stored under is a plain input name, like `tagline`: a `result`, a `batch_as` or a `PipeBatch`'s `input_item_name` written in any other form, such as `Tagline` or `marketing.tagline`, is refused with `invalid_input_name` (see [Stored names](pipe-controllers/PipeSequence.md#stored-names)).
+
## Working Memory Lifecycle
* **Creation**: The Working Memory is created at the start of a pipeline run.
diff --git a/pipelex/core/memory/working_memory.py b/pipelex/core/memory/working_memory.py
index 71f75d171..30c881379 100644
--- a/pipelex/core/memory/working_memory.py
+++ b/pipelex/core/memory/working_memory.py
@@ -37,9 +37,10 @@ MAIN_STUFF_NAME = "main_stuff"
BATCH_ITEM_STUFF_NAME = "BATCH_ITEM"
# The prefix of the private name a PipeSequence binds a dotted `batch_over`'s list under: underscore-led, so no template or
# input reads it, and never a param's, so working memory holds it like any stuff. The prefix is the runtime's alone: a nested
-# sequence binds in its caller's working memory, so validation refuses any name an author writes taking it
-# (`check_name_is_not_reserved`), and a caller's value is never overwritten by a list a sequence it calls binds. This one
-# constant spells the prefix for the name `make_private_binding_name` generates, the params exemption below, and that refusal.
+# sequence binds in its caller's working memory, so validation refuses any name an author writes taking it, a stored name by
+# holding it to the plain input-name form (`check_stored_name`) and a plain `batch_over` by name (`check_name_is_not_reserved`),
+# and a caller's value is never overwritten by a list a sequence it calls binds. This one constant spells the prefix for the
+# name `make_private_binding_name` generates, the params exemption below, and those refusals.
PRIVATE_BINDING_NAME_PREFIX = "_bound_"
PRETTY_PRINT_MAX_LENGTH = 1000
TEST_DUMMY_NAME = "dummy_result"
diff --git a/pipelex/language/mthds_schema_generator.py b/pipelex/language/mthds_schema_generator.py
index 4e6f4cb5c..5aba25154 100644
--- a/pipelex/language/mthds_schema_generator.py
+++ b/pipelex/language/mthds_schema_generator.py
@@ -19,6 +19,7 @@ from pipelex.pipe_machinery.validation import (
PARALLEL_BRANCH_BATCH_OVER_PATTERN,
RESERVED_NAME_PATTERN,
SEQUENCE_STEP_BATCH_OVER_PATTERN,
+ STORED_NAME_PATTERN,
)
from pipelex.pipe_signature.pipe_signature_blueprint import PipeSignatureBlueprint
from pipelex.tools.misc.package_utils import get_package_version
@@ -48,9 +49,9 @@ _SIGNATURE_DEFINITION_NAME = PipeSignatureBlueprint.__name__
# (`_constrain_batch_over`). No Python class carries the name, since steps and branches share one blueprint.
_PARALLEL_BRANCH_DEFINITION_NAME = "ParallelBranchBlueprint"
-# The fields of a pipe step, and of a PipeParallel branch, naming a slot of working memory: the runtime refuses each when it
-# takes the prefix reserved for the bound list of a dotted `batch_over` (`_reserve_private_binding_names`).
-_PIPE_STEP_NAME_FIELDS: tuple[str, ...] = ("result", "batch_as", "batch_over")
+# The fields of a pipe step, and of a PipeParallel branch, naming a slot of working memory the step stores a value under: each
+# is a stored name, held to the input-name grammar (`_constrain_stored_names`).
+_PIPE_STEP_STORED_NAME_FIELDS: tuple[str, ...] = ("result", "batch_as")
def generate_mthds_schema() -> dict[str, Any]:
@@ -74,8 +75,8 @@ def generate_mthds_schema() -> dict[str, Any]:
schema = _convert_to_draft4(schema)
schema = _patch_construct_schema(schema)
schema = _constrain_input_names(schema)
- # Before `_constrain_batch_over`, so the branch definition it copies from the pipe step's carries the reservation too.
- schema = _reserve_private_binding_names(schema)
+ # Before `_constrain_batch_over`, so the branch definition it copies from the pipe step's carries these constraints too.
+ schema = _constrain_stored_names(schema)
schema = _constrain_batch_over(schema)
return _add_taplo_metadata(schema)
@@ -309,31 +310,37 @@ def _constrain_input_names(schema: dict[str, Any]) -> dict[str, Any]:
return schema
-def _reserve_private_binding_names(schema: dict[str, Any]) -> dict[str, Any]:
- """Refuse the names the runtime reserves for the bound list of a dotted `batch_over`, as the runtime does.
+def _constrain_stored_names(schema: dict[str, Any]) -> dict[str, Any]:
+ """Hold every stored name to the input-name grammar, and keep a plain `batch_over` off the reserved prefix, as the runtime does.
- A pipe step's `result`, `batch_as` and plain `batch_over`, and a PipeBatch's `input_item_name`, never take the `_bound_`
- prefix: the runtime refuses each as `invalid_input_name` when the bundle is parsed, which keeps that error type's
- `fails_at = "schema"` true. Each is a Draft-4 `not` holding a `pattern` on the field's string arm: a negative lookahead in
- the pattern itself would say the same, but not every validator's regex engine has one, while every Draft-4 validator
- applies `not`, plxt's included. The other names an author writes into working memory, input names and a binding step's
- `result`, already follow the plain-name grammar, which no underscore-led name matches.
+ A pipe step's `result` and `batch_as`, the same fields on a PipeParallel branch, and a PipeBatch's `input_item_name` are
+ stored names: a step stores a value under each for a pipe to read through an input, so each takes the plain input-name
+ grammar (`STORED_NAME_PATTERN`), as a binding step's `result` does through its own field. The runtime refuses any other
+ form as `invalid_input_name` when the bundle is parsed, which keeps that error type's `fails_at = "schema"` true, and no
+ name of that grammar takes the `_bound_` prefix the runtime reserves for the bound list of a dotted `batch_over`, being
+ never underscore-led. A plain `batch_over` reads a name rather than storing one, so no such grammar holds it, and it is
+ kept off the prefix by a Draft-4 `not` holding a `pattern` on its string arm: a negative lookahead in the pattern itself
+ would say the same, but not every validator's regex engine has one, while every Draft-4 validator applies `not`, plxt's
+ included. `_constrain_batch_over` gives it its path grammar besides.
"""
schema = copy.deepcopy(schema)
definitions = schema.get("definitions", {})
pipe_step_properties = definitions.get(SubPipeBlueprint.__name__, {}).get("properties", {})
- for field_name in _PIPE_STEP_NAME_FIELDS:
- _refuse_reserved_names(field_schema=pipe_step_properties.get(field_name))
- _refuse_reserved_names(field_schema=definitions.get(PipeBatchBlueprint.__name__, {}).get("properties", {}).get("input_item_name"))
+ for field_name in _PIPE_STEP_STORED_NAME_FIELDS:
+ _set_string_arm_constraint(field_schema=pipe_step_properties.get(field_name), constraint={"pattern": STORED_NAME_PATTERN})
+ _set_string_arm_constraint(field_schema=pipe_step_properties.get("batch_over"), constraint={"not": {"pattern": RESERVED_NAME_PATTERN}})
+ batch_properties = definitions.get(PipeBatchBlueprint.__name__, {}).get("properties", {})
+ _set_string_arm_constraint(field_schema=batch_properties.get("input_item_name"), constraint={"pattern": STORED_NAME_PATTERN})
return schema
-def _refuse_reserved_names(*, field_schema: dict[str, Any] | None) -> None:
+def _set_string_arm_constraint(*, field_schema: dict[str, Any] | None, constraint: dict[str, Any]) -> None:
+ """Add a constraint to every string arm of a field, the field itself standing for its one arm when it has no `anyOf`."""
if field_schema is None:
return
for arm in field_schema.get("anyOf", [field_schema]):
if arm.get("type") == "string":
- arm["not"] = {"pattern": RESERVED_NAME_PATTERN}
+ arm.update(copy.deepcopy(constraint))
def _constrain_batch_over(schema: dict[str, Any]) -> dict[str, Any]:
diff --git a/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py b/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py
index 52e1bc491..f5e6c9dd4 100644
--- a/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py
+++ b/pipelex/pipe_controllers/batch/pipe_batch_blueprint.py
@@ -4,7 +4,7 @@ from typing_extensions import override
from pipelex.core.pipes.exceptions import PipeValidationError
from pipelex.pipe_machinery.pipe_blueprint import PipeBlueprint
-from pipelex.pipe_machinery.validation import check_input_list_name, check_name_is_not_reserved
+from pipelex.pipe_machinery.validation import check_input_list_name, check_stored_name
from pipelex.validation_error_types import PipeValidationErrorType
@@ -35,9 +35,9 @@ class PipeBatchBlueprint(PipeBlueprint):
if not self.input_item_name:
msg = "Empty input item name is not allowed"
raise ValueError(msg)
- # The item is written into each branch's working memory under its name, so the name never takes the prefix the runtime
- # reserves for the bound list of a dotted `batch_over`. The list's name needs no such check: it is a plain input name.
- check_name_is_not_reserved(name=self.input_item_name, field_label="The PipeBatch's `input_item_name`")
+ # The item is stored in each branch's working memory under its name, for the branch pipe to read through an input of
+ # that name, so the name is a stored name and takes the plain input-name form.
+ check_stored_name(name=self.input_item_name, field_label="The PipeBatch's `input_item_name`")
# Specialization: catch the most common mistake (item name == list name) with a targeted message
if self.input_item_name == self.input_list_name:
msg = (
diff --git a/pipelex/pipe_controllers/binding/binding_step_blueprint.py b/pipelex/pipe_controllers/binding/binding_step_blueprint.py
index f447b33d6..25b91ae9f 100644
--- a/pipelex/pipe_controllers/binding/binding_step_blueprint.py
+++ b/pipelex/pipe_controllers/binding/binding_step_blueprint.py
@@ -4,7 +4,7 @@ from typing import Any, cast
from pydantic import BaseModel, ConfigDict, Field, SerializerFunctionWrapHandler, field_validator, model_serializer
from pipelex.core.pipes.exceptions import PipeValidationError
-from pipelex.pipe_machinery.validation import BINDING_PATH_PATTERN, BINDING_RESULT_PATTERN
+from pipelex.pipe_machinery.validation import BINDING_PATH_PATTERN, STORED_NAME_PATTERN
from pipelex.tools.misc.string_utils import FIELD_PATH_SEGMENT_REGEX, SNAKE_CASE_IDENTIFIER_REGEX, is_field_path, is_snake_case
from pipelex.validation_error_types import PipeValidationErrorType
@@ -81,7 +81,7 @@ class BindingStepBlueprint(BaseModel):
)
result: str = Field(
description="The name under which the bound value is stored in working memory: a plain input name.",
- json_schema_extra={"pattern": BINDING_RESULT_PATTERN},
+ json_schema_extra={"pattern": STORED_NAME_PATTERN},
)
@field_validator("from_path", mode="after")
diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.py
index 4aa1a78d6..06a23bb81 100644
--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py
+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py
@@ -43,7 +43,7 @@ from pipelex.pipe_controllers.sequence.sequence_typed_flow import (
)
from pipelex.pipe_controllers.sub_pipe import SubPipe
from pipelex.pipe_machinery.memory_writes import MemoryWrite, SlotTaint, is_same_value_spec, taint_after_write
-from pipelex.pipe_machinery.validation import is_valid_input_name
+from pipelex.pipe_machinery.validation import is_valid_input_name, suggest_plain_name
from pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunParams, output_multiplicity_to_apply
from pipelex.system.job_metadata import JobMetadata
from pipelex.validation_error_types import PipeValidationErrorType
@@ -275,7 +275,8 @@ class PipeSequence(PipeController):
def refuse_undeclared_needed_input(self, *, variable_name: str) -> None:
"""Refuse a binding's root that the sequence neither declares nor always stores, asking for the concept its path walks.
- A root that is not a plain input name cannot be declared, so the refusal asks for a step storing it instead.
+ A root that is not a plain input name can be neither declared nor stored by a step, since every stored name is a plain
+ input name too, so the refusal asks to bind from a plain name instead.
"""
_, binding_root_needs = self._walk_needed_inputs(visited_pipes=set())
if variable_name not in binding_root_needs:
@@ -287,9 +288,12 @@ class PipeSequence(PipeController):
f"Declare '{variable_name}' in the sequence's `inputs`, with the concept whose structure holds the path '{step.from_path}'."
)
else:
+ plain_name = suggest_plain_name(name=variable_name)
+ suggestion = f", such as '{plain_name}'" if plain_name is not None else ""
remedy = (
- f"'{variable_name}' cannot be an input of the sequence, since an input name is a plain snake_case identifier: "
- f"store a value under '{variable_name}' in an earlier step, whose `result` names it, or bind from a plain name."
+ f"'{variable_name}' is not a plain snake_case identifier, so it can be neither an input of the sequence nor a name "
+ f"a step stores a value under: bind from a plain name{suggestion}, declared in the sequence's `inputs` or stored by "
+ "an earlier step."
)
msg = (
f"In pipe '{self.code}', the {step.label} reads '{variable_name}', which is neither an input of the "
diff --git a/pipelex/pipe_controllers/sub_pipe_blueprint.py b/pipelex/pipe_controllers/sub_pipe_blueprint.py
index d55039ff5..f5b7e934d 100644
--- a/pipelex/pipe_controllers/sub_pipe_blueprint.py
+++ b/pipelex/pipe_controllers/sub_pipe_blueprint.py
@@ -4,7 +4,7 @@ from pydantic import BaseModel, ConfigDict, model_validator
from pipelex.core.pipes.exceptions import PipeValidationError
from pipelex.pipe_controllers.binding.binding_step_blueprint import PATH_GRAMMAR_DESCRIPTION
-from pipelex.pipe_machinery.validation import check_name_is_not_reserved
+from pipelex.pipe_machinery.validation import check_name_is_not_reserved, check_stored_name
from pipelex.tools.misc.string_utils import is_field_path
from pipelex.tools.typing.validation_utils import has_more_than_one_among_attributes_from_list
from pipelex.validation_error_types import PipeValidationErrorType
@@ -68,16 +68,19 @@ class SubPipeBlueprint(BaseModel):
raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.BINDING_STEP_INVALID, variable_names=[self.batch_over])
@model_validator(mode="after")
- def validate_reserved_names(self) -> Self:
- """Refuse a name taking the prefix the runtime reserves for the bound list of a dotted `batch_over`, on a sequence step and a
- parallel branch alike.
+ def validate_stored_names(self) -> Self:
+ """Hold the names the step stores values under to the input-name form, and keep a plain `batch_over` off the reserved
+ prefix, on a sequence step and a parallel branch alike.
- A step stores its `result` and hands each item to its pipe under `batch_as`, both in working memory, and batches over a
- plain `batch_over` by name. A nested sequence binds in its caller's working memory, so a name of the caller taking the
- prefix could be overwritten by a list the nested sequence binds, and read back as another value.
+ A step stores its `result` and hands each item to its pipe under `batch_as`, both in working memory, for a pipe to read
+ through an input, so each is a stored name and takes the plain input-name form. A plain `batch_over` reads a name rather
+ than storing one, and only stays off the prefix the runtime reserves for the bound list of a dotted `batch_over`: a nested
+ sequence binds in its caller's working memory, so a step batching over a name taking the prefix could read a list a
+ sequence it calls binds. A dotted `batch_over` follows the path grammar, whose segments are never underscore-led.
"""
- named_fields = (("result", self.result), ("batch_as", self.batch_as), ("batch_over", self.batch_over))
- for field_name, name in named_fields:
+ for field_name, name in (("result", self.result), ("batch_as", self.batch_as)):
if name is not None:
- check_name_is_not_reserved(name=name, field_label=f"The `{field_name}` of the step running pipe '{self.pipe}'")
+ check_stored_name(name=name, field_label=f"The `{field_name}` of the step running pipe '{self.pipe}'")
+ if self.batch_over is not None:
+ check_name_is_not_reserved(name=self.batch_over, field_label=f"The `batch_over` of the step running pipe '{self.pipe}'")
return self
diff --git a/pipelex/pipe_machinery/validation.py b/pipelex/pipe_machinery/validation.py
index 7bdc5904d..8ff069793 100644
--- a/pipelex/pipe_machinery/validation.py
+++ b/pipelex/pipe_machinery/validation.py
@@ -21,6 +21,7 @@ from pipelex.tools.misc.string_utils import (
FIELD_PATH_SEGMENT_REGEX,
SNAKE_CASE_IDENTIFIER_REGEX,
SNAKE_CASE_PATTERN,
+ camel_to_snake_case,
get_root_from_dotted_path,
is_snake_case,
)
@@ -33,9 +34,13 @@ from pipelex.validation_error_types import PipeValidationErrorType
# snake_case grammar of `string_utils`, the one source every name grammar here is composed from.
INPUT_NAME_PATTERN = SNAKE_CASE_PATTERN
-# The grammar of a binding step's `result`: a plain input name, since it exists to be read by a later
-# step's input or as a later binding's root, both of which are plain names.
-BINDING_RESULT_PATTERN = INPUT_NAME_PATTERN
+# The grammar of a stored name, a name under which a step stores a value in working memory: a pipe step's or
+# a PipeParallel branch's `result`, the `batch_as` of either, a PipeBatch's `input_item_name`, and a binding
+# step's `result`. A later pipe reads a stored value through an input, or a later binding through its root,
+# both plain names, so a stored name is a plain input name and every value a step stores can be read. The
+# MTHDS JSON Schema generator writes this pattern on each of these fields, so a structural check refuses
+# exactly what `check_stored_name` and the binding step's own validator refuse.
+STORED_NAME_PATTERN = INPUT_NAME_PATTERN
# The grammar of a binding step's `from`: a root name followed by zero or more field names, separated by
# single dots, each segment a letter followed by letters, digits and underscores. Subscripts, expressions,
@@ -52,10 +57,11 @@ SEQUENCE_STEP_BATCH_OVER_PATTERN = rf"^(?:[^.]*|{FIELD_PATH_SEGMENT_REGEX}(?:\.{
PARALLEL_BRANCH_BATCH_OVER_PATTERN = r"^[^.]*$"
# The names the runtime reserves: those taking the prefix a PipeSequence binds a dotted `batch_over`'s list under. A nested
-# sequence binds in its caller's working memory, so a caller's name taking the prefix could be overwritten by that list.
-# `check_name_is_not_reserved` refuses it on every name an author writes into working memory or batches over by name, the
-# ones no other grammar already keeps underscore-free, and the MTHDS JSON Schema generator writes this pattern under `not`
-# on the same fields, so a structural check refuses exactly what the runtime refuses.
+# sequence binds in its caller's working memory, so a caller's name taking the prefix could be overwritten by that list. No
+# stored name or input name can take it, being plain and so never underscore-led, and `check_name_is_not_reserved` refuses
+# it on the one name an author writes that no such grammar holds, a plain `batch_over`, which reads a name rather than
+# storing one. The MTHDS JSON Schema generator writes this pattern under `not` on that field, so a structural check refuses
+# exactly what the runtime refuses.
RESERVED_NAME_PATTERN = f"^{re.escape(PRIVATE_BINDING_NAME_PREFIX)}"
# The marker of an input declaration as `_input_marker` writes it after the concept: an optional multiplicity
@@ -472,17 +478,70 @@ def check_input_list_name(*, input_list_name: str, branch_pipe_code: str, input_
raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.INVALID_INPUT_NAME, variable_names=[input_list_name])
+def suggest_plain_name(*, name: str) -> str | None:
+ """A plain input name close to a name that is not one, for a refusal to offer, or `None` when none comes out of it.
+
+ The reserved prefix is dropped, dots and hyphens become underscores and PascalCase or camelCase becomes snake_case, so
+ `Pages` gives `pages`, `catalog.pages` gives `catalog_pages` and `_bound_catalog_pages` gives `catalog_pages`.
+ """
+ unprefixed_name = name.removeprefix(PRIVATE_BINDING_NAME_PREFIX).replace(".", "_").replace("-", "_")
+ candidate = re.sub(r"_+", "_", camel_to_snake_case(name=unprefixed_name)).strip("_")
+ if candidate == name or not is_valid_input_name(candidate):
+ return None
+ return candidate
+
+
+def check_stored_name(*, name: str, field_label: str) -> None:
+ """Refuse a stored name that is not a plain input name.
+
+ A stored name is a name under which a step stores a value in working memory: a pipe step's or a PipeParallel branch's
+ `result`, the `batch_as` of either, and a PipeBatch's `input_item_name`. A later pipe reads a stored value through an
+ input, and an input name is a plain snake_case identifier, so a value stored under any other name could never be read.
+ No plain name takes the prefix the runtime reserves for the bound list of a dotted `batch_over`, being never
+ underscore-led, so this refuses such a name too, and the message says why the prefix is not the author's. A binding
+ step's `result` is a stored name held to the same grammar, which its own blueprint refuses as `binding_step_invalid`.
+
+ Args:
+ name: The name as written.
+ field_label: The field holding it, for the message, such as "The `result` of the step running pipe 'describe_page'".
+
+ Raises:
+ PipeValidationError: ``INVALID_INPUT_NAME`` naming the name.
+ """
+ if is_valid_input_name(name):
+ return
+ stored_name_rule = "a value is stored under it in working memory for a pipe to read through an input, so it must be a plain input name"
+ reason: str
+ if name.startswith(PRIVATE_BINDING_NAME_PREFIX):
+ reason = (
+ f"{stored_name_rule}, and the `{PRIVATE_BINDING_NAME_PREFIX}` prefix it starts with is reserved for the bound list of a "
+ "dotted `batch_over`, which only the runtime writes or reads."
+ )
+ elif "." in name:
+ reason = f"{stored_name_rule}, which names one whole value and cannot reach into a field with a dot."
+ else:
+ reason = f"{stored_name_rule}, and {_PLAIN_NAME_RULE}."
+ plain_name = suggest_plain_name(name=name)
+ suggestion = f", such as '{plain_name}'" if plain_name is not None else ""
+ msg = (
+ f"{field_label}, '{name}', is not a plain input name: {reason} Rename it to a plain name{suggestion}, and rename every input, "
+ "binding path and `batch_over` reading it to match."
+ )
+ raise PipeValidationError(message=msg, error_type=PipeValidationErrorType.INVALID_INPUT_NAME, variable_names=[name])
+
+
def check_name_is_not_reserved(*, name: str, field_label: str) -> None:
- """Refuse a name an author writes that takes the prefix the runtime reserves for the bound list of a dotted `batch_over`.
+ """Refuse a plain `batch_over` taking the prefix the runtime reserves for the bound list of a dotted `batch_over`.
A PipeSequence binds a dotted `batch_over`'s list under a private name taking the prefix, `_bound_catalog_pages` for
- `catalog.pages`, and a nested sequence binds in the working memory of the sequence calling it. A caller's name taking
- the prefix could then be overwritten by the list a sequence it calls binds, so the prefix is the runtime's alone: a
- name an author stores a value under, hands an item to a pipe under, or batches over by name never takes it.
+ `catalog.pages`, and a nested sequence binds in the working memory of the sequence calling it. A step batching over such
+ a name by hand would read a list a sequence it calls binds, so the prefix is the runtime's alone. A stored name never
+ takes it, being a plain input name (`check_stored_name`), but a plain `batch_over` reads a name rather than storing
+ one, so no other grammar keeps it off the prefix.
Args:
name: The name as written.
- field_label: The field holding it, for the message, such as "The `result` of the step running pipe 'describe_page'".
+ field_label: The field holding it, for the message, such as "The `batch_over` of the step running pipe 'describe_page'".
Raises:
PipeValidationError: ``INVALID_INPUT_NAME`` naming the name.
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_batch_item_name_not_plain/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_batch_item_name_not_plain/bundle.mthds
new file mode 100644
index 000000000..ecd8cb211
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_batch_item_name_not_plain/bundle.mthds
@@ -0,0 +1,28 @@
+domain = "cheese_counter"
+description = "Writing the price tag of every cheese on a shop counter"
+main_pipe = "tag_cheeses"
+
+[concept.Cheese]
+description = "A cheese sold by weight at the counter"
+
+[concept.Cheese.structure]
+variety = { type = "text", description = "The kind of cheese", required = true }
+price_per_kilo = { type = "number", description = "The price of one kilogram, in euros", required = true }
+
+[pipe.tag_cheeses]
+type = "PipeBatch"
+description = "Writes the price tag of each cheese in turn"
+inputs = { cheeses = "Cheese[]" }
+output = "Text[]"
+branch_pipe_code = "write_price_tag"
+input_list_name = "cheeses"
+# Wrong on purpose: each cheese is stored under `input_item_name` for the branch pipe to read through an input, so the name
+# is a plain snake_case one. Renaming it to `cheese`, the input the branch pipe reads, is the whole fix.
+input_item_name = "Cheese"
+
+[pipe.write_price_tag]
+type = "PipeCompose"
+description = "Writes the price tag of one cheese"
+inputs = { cheese = "Cheese" }
+output = "Text"
+template = "{{ cheese.variety }}: {{ cheese.price_per_kilo }} euros per kilo"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_batch_item_name_not_plain/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_batch_item_name_not_plain/entry.toml
new file mode 100644
index 000000000..10c31d5bb
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_batch_item_name_not_plain/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_input_name_batch_item_name_not_plain"
+description = "The price tags of the cheeses on a shop counter written by a batch that hands each cheese to its branch pipe under a name that is not a plain input name"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.invalid_input_name",
+]
+
+expected_error = "invalid_input_name"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_batch_as_reserved/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_batch_as_reserved/bundle.mthds
new file mode 100644
index 000000000..0f4abaf08
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_batch_as_reserved/bundle.mthds
@@ -0,0 +1,38 @@
+domain = "weather_station"
+description = "Writing the daily log of a weather station"
+main_pipe = "write_daily_log"
+
+[concept.Reading]
+description = "One reading taken by a weather station"
+
+[concept.Reading.structure]
+hour = { type = "text", description = "The hour the reading was taken", required = true }
+temperature = { type = "number", description = "The temperature, in degrees Celsius", required = true }
+
+[pipe.write_reading_line]
+type = "PipeCompose"
+description = "Writes the line of the daily log for one reading"
+inputs = { reading = "Reading" }
+output = "Text"
+template = "{{ reading.hour }}: {{ reading.temperature }} degrees"
+
+[pipe.write_log_heading]
+type = "PipeCompose"
+description = "Writes the heading of the daily log"
+inputs = { station_name = "Text" }
+output = "Text"
+template = "Daily log of $station_name"
+
+[pipe.write_daily_log]
+type = "PipeParallel"
+description = "Writes the heading and the line of every reading at the same time"
+inputs = { readings = "Reading[]", station_name = "Text" }
+output = "Composite"
+add_each_output = true
+# Wrong on purpose: each reading is stored under `batch_as` for the line pipe to read through an input, so the name is a plain
+# snake_case one, which never takes the `_bound_` prefix the runtime keeps for itself. Renaming it to `reading`, the input the
+# line pipe reads, is the whole fix.
+branches = [
+ { pipe = "write_reading_line", batch_over = "readings", batch_as = "_bound_reading", result = "reading_lines" },
+ { pipe = "write_log_heading", result = "heading" },
+]
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_batch_as_reserved/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_batch_as_reserved/entry.toml
new file mode 100644
index 000000000..f2f7b73b5
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_batch_as_reserved/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_input_name_branch_batch_as_reserved"
+description = "The daily log of a weather station written in parallel, one branch handing each reading to its pipe under a name taking the reserved prefix"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.invalid_input_name",
+]
+
+expected_error = "invalid_input_name"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_result_not_plain/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_result_not_plain/bundle.mthds
new file mode 100644
index 000000000..c1db68d7d
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_result_not_plain/bundle.mthds
@@ -0,0 +1,38 @@
+domain = "weeknight_recipes"
+description = "Getting ready to cook a recipe"
+main_pipe = "prepare_recipe"
+
+[concept.Recipe]
+description = "A recipe from a family cookbook"
+
+[concept.Recipe.structure]
+title = { type = "text", description = "The name of the dish", required = true }
+ingredients = { type = "text", description = "The ingredients, as the cookbook lists them", required = true }
+method = { type = "text", description = "How to cook the dish", required = true }
+
+[pipe.write_shopping_list]
+type = "PipeCompose"
+description = "Writes the shopping list of a recipe"
+inputs = { recipe = "Recipe" }
+output = "Text"
+template = "To buy for {{ recipe.title }}: {{ recipe.ingredients }}"
+
+[pipe.write_cooking_card]
+type = "PipeCompose"
+description = "Writes the card pinned above the stove"
+inputs = { recipe = "Recipe" }
+output = "Text"
+template = "{{ recipe.title }}: {{ recipe.method }}"
+
+[pipe.prepare_recipe]
+type = "PipeParallel"
+description = "Writes the shopping list and the cooking card at the same time"
+inputs = { recipe = "Recipe" }
+output = "Composite"
+add_each_output = true
+# Wrong on purpose: a branch's result is stored for a later pipe to read through an input, so the name is a plain
+# snake_case one. Renaming it to `shopping_list` is the whole fix.
+branches = [
+ { pipe = "write_shopping_list", result = "ShoppingList" },
+ { pipe = "write_cooking_card", result = "cooking_card" },
+]
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_result_not_plain/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_result_not_plain/entry.toml
new file mode 100644
index 000000000..1b7f03728
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_branch_result_not_plain/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_input_name_branch_result_not_plain"
+description = "The shopping list and the cooking card of a recipe written in parallel, one branch storing its list under a name that is not a plain input name"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.invalid_input_name",
+]
+
+expected_error = "invalid_input_name"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_batch_as_dotted/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_batch_as_dotted/bundle.mthds
new file mode 100644
index 000000000..0d162f338
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_batch_as_dotted/bundle.mthds
@@ -0,0 +1,28 @@
+domain = "parcel_dispatch"
+description = "Writing the shipping label of every parcel in a delivery round"
+main_pipe = "label_round"
+
+[concept.Parcel]
+description = "A parcel waiting to be delivered"
+
+[concept.Parcel.structure]
+recipient = { type = "text", description = "Who the parcel is for", required = true }
+address = { type = "text", description = "Where the parcel is delivered", required = true }
+
+[pipe.write_label]
+type = "PipeCompose"
+description = "Writes the shipping label of one parcel"
+inputs = { parcel = "Parcel" }
+output = "Text"
+template = "{{ parcel.recipient }}, {{ parcel.address }}"
+
+[pipe.label_round]
+type = "PipeSequence"
+description = "Writes the label of each parcel in the round"
+inputs = { parcels = "Parcel[]" }
+output = "Text[]"
+# Wrong on purpose: each parcel is stored under `batch_as` for the label pipe to read through an input, so the name is never
+# dotted. Renaming it to `parcel`, the input the label pipe reads, is the whole fix.
+steps = [
+ { pipe = "write_label", batch_over = "parcels", batch_as = "round.parcel", result = "labels" },
+]
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_batch_as_dotted/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_batch_as_dotted/entry.toml
new file mode 100644
index 000000000..a4583916d
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_batch_as_dotted/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_input_name_step_batch_as_dotted"
+description = "The shipping labels of a delivery round written by a sequence step that hands each parcel to the label pipe under a dotted name"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.invalid_input_name",
+]
+
+expected_error = "invalid_input_name"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_result_not_plain/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_result_not_plain/bundle.mthds
new file mode 100644
index 000000000..82c5337e3
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_result_not_plain/bundle.mthds
@@ -0,0 +1,36 @@
+domain = "rail_timetables"
+description = "Announcing the next departures from a station"
+main_pipe = "announce_departures"
+
+[concept.Departure]
+description = "One train leaving the station"
+
+[concept.Departure.structure]
+destination = { type = "text", description = "Where the train goes", required = true }
+platform = { type = "text", description = "The platform it leaves from", required = true }
+
+[pipe.list_departures]
+type = "PipeLLM"
+description = "Lists the departures shown on a timetable board"
+inputs = { board = "Text" }
+output = "Departure[]"
+prompt = "List the departures shown on this timetable board: $board"
+
+[pipe.write_announcement]
+type = "PipeCompose"
+description = "Writes the announcement of one departure"
+inputs = { departure = "Departure" }
+output = "Text"
+template = "The train to {{ departure.destination }} leaves from platform {{ departure.platform }}."
+
+[pipe.announce_departures]
+type = "PipeSequence"
+description = "Lists the departures, then writes the announcement of each"
+inputs = { board = "Text" }
+output = "Text[]"
+# Wrong on purpose: a step stores its result for a later pipe to read through an input, so the name is a plain
+# snake_case one. Nothing reads it here, so renaming it to `announcements` is the whole fix.
+steps = [
+ { pipe = "list_departures", result = "departures" },
+ { pipe = "write_announcement", batch_over = "departures", batch_as = "departure", result = "Announcements" },
+]
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_result_not_plain/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_result_not_plain/entry.toml
new file mode 100644
index 000000000..0eabc1964
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_name_step_result_not_plain/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_input_name_step_result_not_plain"
+description = "The departures from a station announced by a sequence whose last step stores its announcements under a name that is not a plain input name"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.invalid_input_name",
+]
+
+expected_error = "invalid_input_name"
diff --git a/pipelex/validation_error_types.py b/pipelex/validation_error_types.py
index 7a0c6cff3..320d7f0a8 100644
--- a/pipelex/validation_error_types.py
+++ b/pipelex/validation_error_types.py
@@ -55,11 +55,13 @@ class PipeValidationErrorType(StrEnum):
LLM_SETTING_REFUSED_BY_MODEL = "llm_setting_refused_by_model"
INVALID_PIPE_CODE_SYNTAX = "invalid_pipe_code_syntax"
# An input name that is not a plain snake_case identifier, a dotted name (`invoice.total`) included,
- # on any pipe's `inputs` or as a PipeBatch's `input_list_name`; or a name taking the `_bound_` prefix
- # the runtime reserves for the bound list of a dotted `batch_over`, as a pipe step's or a parallel
- # branch's `result`, `batch_as` or plain `batch_over`, or a PipeBatch's `input_item_name`. Detected at
- # blueprint parse time; a dotted name whose root the same `inputs` table declares carries the
- # enrichment the fix planner turns into deleting the key.
+ # on any pipe's `inputs` or as a PipeBatch's `input_list_name`; a stored name that is not one either,
+ # in any form (`Pages`, `doc.pages`, `_bound_pages`), as a pipe step's or a parallel branch's `result`
+ # or `batch_as`, or a PipeBatch's `input_item_name`; or a plain `batch_over` taking the `_bound_`
+ # prefix the runtime reserves for the bound list of a dotted `batch_over`, the one check a plain
+ # `batch_over` gets. A binding step's `result` is a stored name too, refused as BINDING_STEP_INVALID.
+ # Detected at blueprint parse time; a dotted input name whose root the same `inputs` table declares
+ # carries the enrichment the fix planner turns into deleting the key.
INVALID_INPUT_NAME = "invalid_input_name"
UNKNOWN_PIPE_TYPE = "unknown_pipe_type"
# A `[pipe.x]` section declared no `type` yet carries fields beyond the signature contract
diff --git a/subject_grants.toml b/subject_grants.toml
index da6ed3a72..0043e596d 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -3556,6 +3556,10 @@ rationale = "One post-processing pass of the schema pipeline, called as a verb o
param = "schema"
rationale = "Schema-pipeline stage transforming the schema dict; single operand."
+["pipelex/language/mthds_schema_generator.py::_constrain_stored_names"]
+param = "schema"
+rationale = "One post-processing pass of the schema pipeline, called as a verb on the schema it rewrites, like its sibling passes."
+
["pipelex/language/mthds_schema_generator.py::_convert_to_draft4"]
param = "schema"
rationale = "Schema-pipeline stage transforming the schema dict; single operand."
@@ -3584,10 +3588,6 @@ rationale = "Verb-object: mutates the schema object's required list; field_names
param = "schema"
rationale = "Schema-pipeline stage transforming the schema dict; single operand."
-["pipelex/language/mthds_schema_generator.py::_reserve_private_binding_names"]
-param = "schema"
-rationale = "One post-processing pass of the schema pipeline, called as a verb on the schema it rewrites, like its sibling passes."
-
["pipelex/language/mthds_schema_generator.py::_walk_schema"]
param = "node"
rationale = "Verb-object: walks the schema node tree; the visitor stays keyword."
diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_add_each.mthds b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_add_each.mthds
index 60c2fe80d..a141073ff 100644
--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_add_each.mthds
+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_add_each.mthds
@@ -16,7 +16,7 @@ description = "Run parallel summaries then consume one downstream"
inputs = { input_text = "Text" }
output = "Text"
steps = [
- { pipe = "parallel_summarize", result = "..." },
+ { pipe = "parallel_summarize", result = "summaries" },
{ pipe = "combine_summaries" },
]
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_refusal_remedy.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_refusal_remedy.py
index e3d2c2caa..0d1695824 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_refusal_remedy.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_binding_refusal_remedy.py
@@ -48,8 +48,8 @@ output = "Text"
template = "View: {{{{ page_view.caption }}}}"
"""
-# A root that is not a plain name: no input can bear it, so only an earlier pipe step's `result`, which the standard leaves
-# unrestricted, can store a value under it.
+# A root that is not a plain name: no input can bear it, and no step can store a value under it either, since a pipe step's
+# `result` is a stored name, held to the plain input-name form like an input name.
_UPPERCASE_ROOT_BUNDLE = """domain = "billing"
description = "Reading the total of an invoice stored under a name that is not a plain name"
main_pipe = "read_total"
@@ -117,17 +117,21 @@ class TestBindingRefusalRemedy:
assert result.pipe_output.main_stuff.as_text.text == "View: Garden chairs on a lawn"
@pytest.mark.asyncio(loop_scope="class")
- async def test_a_root_that_is_not_a_plain_name_reads_an_earlier_steps_result(self) -> None:
- """The path grammar lets the root be any segment, so it can read a pipe step's `result` that is not a plain name."""
- result = await PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.LIVE).execute(
- mthds_contents=[_UPPERCASE_ROOT_BUNDLE], inputs={"amount": {"concept": "native.Number", "content": {"number": 120}}}
- )
+ async def test_a_pipe_step_storing_under_a_name_that_is_not_plain_is_refused(self) -> None:
+ """A pipe step's `result` is a stored name, so it cannot store the value a root that is not a plain name would read."""
+ with pytest.raises(ValidateBundleError) as exc_info:
+ await validate_bundle(mthds_contents=[_UPPERCASE_ROOT_BUNDLE])
- assert result.pipe_output.main_stuff_as_number.number == 120
+ validation_errors = exc_info.value.to_error_report().validation_errors or []
+ assert [error.error_type for error in validation_errors] == [PipeValidationErrorType.INVALID_INPUT_NAME]
+ assert validation_errors[0].variable_names == ["Invoice"]
+ message = validation_errors[0].message or ""
+ assert "The `result` of the step running pipe 'make_invoice', 'Invoice', is not a plain input name" in message
+ assert "Rename it to a plain name, such as 'invoice'," in message
@pytest.mark.asyncio(loop_scope="class")
async def test_a_missing_root_that_is_not_a_plain_name_is_never_asked_for_as_an_input(self) -> None:
- """Regression: the refusal of a root nothing stores never tells the author to declare an input no input name can match."""
+ """Regression: the refusal of a root nothing stores never asks to declare it, or to store a value under it, when it is not plain."""
mthds_content = _UPPERCASE_ROOT_BUNDLE.replace(' { pipe = "make_invoice", result = "Invoice" },\n', "")
with pytest.raises(ValidateBundleError) as exc_info:
@@ -137,5 +141,6 @@ class TestBindingRefusalRemedy:
assert [error.error_type for error in validation_errors] == [PipeValidationErrorType.MISSING_INPUT_VARIABLE]
message = validation_errors[0].message or ""
assert "Declare 'Invoice'" not in message
- assert "'Invoice' cannot be an input of the sequence" in message
- assert "store a value under 'Invoice' in an earlier step" in message
+ assert "store a value under 'Invoice'" not in message
+ assert "it can be neither an input of the sequence nor a name a step stores a value under" in message
+ assert "bind from a plain name, such as 'invoice'," in message
diff --git a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_dotted_batch_over.py b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_dotted_batch_over.py
index 5546694be..656cc8087 100644
--- a/tests/integration/pipelex/pipes/controller/pipe_sequence/test_dotted_batch_over.py
+++ b/tests/integration/pipelex/pipes/controller/pipe_sequence/test_dotted_batch_over.py
@@ -415,29 +415,6 @@ class TestDottedBatchOver:
assert response.pipe_output.main_stuff_as_items(item_type=TextContent) == []
- async def test_a_batch_over_a_result_that_is_not_a_plain_name_runs(self) -> None:
- """Regression: the batch a step runs was built from a blueprint, which refused any list name but a plain input name, so a
- batch over a result named `IndexLines`, which a pipe step may store under, failed when it ran.
- """
- mthds_content = _catalog_bundle(
- inputs='{ catalog = "Catalog" }',
- steps=[
- '{ pipe = "write_index_line", batch_over = "catalog.pages", batch_as = "page", result = "IndexLines" }',
- '{ pipe = "copy_line", batch_over = "IndexLines", batch_as = "line", result = "copied_lines" }',
- ],
- ).replace(
- "[pipe.make_catalog]",
- '[pipe.copy_line]\ntype = "PipeCompose"\ndescription = "Copies a line"\ninputs = { line = "Text" }\noutput = "Text"\n'
- 'template = "Copy of $line"\n\n[pipe.make_catalog]',
- )
-
- response = await _index_lines(
- mthds_content=mthds_content, inputs={"catalog": {"concept": "catalog_index.Catalog", "content": _SPRING_CATALOG}}
- )
-
- lines = [item.text for item in response.pipe_output.main_stuff_as_items(item_type=TextContent)]
- assert lines == ["Copy of Page: Garden chairs", "Copy of Page: Parasols"]
-
async def test_two_dotted_batch_overs_of_one_path_take_two_private_names(self, load_empty_library: Callable[[], str]) -> None:
mthds_content = _catalog_bundle(
inputs='{ catalog = "Catalog" }',
@@ -470,11 +447,9 @@ class TestDottedBatchOver:
result_refusals = [error for error in errors if "The `result` of the step running pipe 'write_index_line'" in error.message]
assert len(result_refusals) == 1
assert result_refusals[0].variable_names == [reserved_name]
- assert (
- f"'{reserved_name}', takes the `_bound_` prefix, which is reserved for the bound list of a dotted `batch_over`"
- in result_refusals[0].message
- )
- assert "Choose another name, such as 'catalog_pages'." in result_refusals[0].message
+ assert f"'{reserved_name}', is not a plain input name" in result_refusals[0].message
+ assert "the `_bound_` prefix it starts with is reserved for the bound list of a dotted `batch_over`" in result_refusals[0].message
+ assert "Rename it to a plain name, such as 'catalog_pages'," in result_refusals[0].message
async def test_a_caller_name_outside_the_prefix_keeps_its_value_across_a_nested_binding(self) -> None:
response = await _index_lines(
diff --git a/tests/unit/pipelex/language/test_mthds_schema.py b/tests/unit/pipelex/language/test_mthds_schema.py
index f06100366..bc2c85de4 100644
--- a/tests/unit/pipelex/language/test_mthds_schema.py
+++ b/tests/unit/pipelex/language/test_mthds_schema.py
@@ -407,7 +407,7 @@ class TestMthdsSchemaGeneration:
("batch_over", "is_valid_on_a_step", "is_valid_on_a_branch"),
[
pytest.param("pages", True, True, id="a-plain-name"),
- pytest.param("PagesOfTheCatalog", True, True, id="a-name-a-pipe-step-may-store-under"),
+ pytest.param("PagesOfTheCatalog", True, True, id="a-name-that-is-not-snake-case"),
pytest.param("catalog.pages", True, False, id="a-dotted-path"),
pytest.param("catalogs.pages.page_view", True, False, id="a-deep-dotted-path"),
pytest.param("catalog..pages", False, False, id="an-empty-segment"),
@@ -435,17 +435,23 @@ class TestMthdsSchemaGeneration:
[
pytest.param("catalog_pages", True, id="a-plain-name"),
pytest.param("bound_pages", True, id="the-prefix-without-its-underscore"),
- pytest.param("_draft", True, id="another-underscore-led-name"),
+ pytest.param("page2", True, id="a-digit-after-the-first-letter"),
+ pytest.param("Pages", False, id="pascal-case"),
+ pytest.param("catalog.pages", False, id="dotted"),
+ pytest.param("_draft", False, id="underscore-led"),
+ pytest.param("2nd_pages", False, id="digit-led"),
pytest.param("_bound_catalog_pages", False, id="the-reserved-prefix"),
pytest.param("_bound_", False, id="the-reserved-prefix-alone"),
],
)
- @pytest.mark.parametrize("field_name", ["result", "batch_as", "batch_over"])
- def test_a_step_and_a_branch_refuse_the_reserved_prefix(self, schema: dict[str, Any], field_name: str, name: str, should_validate: bool) -> None:
- """A pipe step's `result`, `batch_as` and plain `batch_over` never take `_bound_`, the runtime's prefix for a dotted `batch_over`'s list.
+ @pytest.mark.parametrize("field_name", ["result", "batch_as"])
+ def test_a_step_and_a_branch_hold_their_stored_names_to_the_plain_name_grammar(
+ self, schema: dict[str, Any], field_name: str, name: str, should_validate: bool
+ ) -> None:
+ """A pipe step's `result` and `batch_as` are stored names, plain input names on a PipeSequence step and a PipeParallel branch alike.
- The runtime refuses them as `invalid_input_name` when the bundle is parsed, and the schema refuses them first, through a
- Draft-4 `not` on the string arm, on a PipeSequence step and on a PipeParallel branch alike.
+ The runtime refuses any other form as `invalid_input_name` when the bundle is parsed, and the schema refuses it first,
+ through the input-name `pattern` on the string arm, which no underscore-led name, the reserved prefix's included, matches.
"""
validator = _pipe_union_oneof_validator(schema)
pipe_step: dict[str, Any] = {"pipe": "describe_page", "batch_over": "pages", "batch_as": "page", "result": "descriptions", field_name: name}
@@ -454,19 +460,68 @@ class TestMthdsSchemaGeneration:
assert validator.is_valid(sequence_table) is should_validate, f"sequence step {field_name} {name!r}"
assert validator.is_valid(parallel_table) is should_validate, f"parallel branch {field_name} {name!r}"
+ @pytest.mark.parametrize(
+ ("batch_over", "should_validate"),
+ [
+ pytest.param("catalog_pages", True, id="a-plain-name"),
+ pytest.param("bound_pages", True, id="the-prefix-without-its-underscore"),
+ pytest.param("_draft", True, id="another-underscore-led-name"),
+ pytest.param("_bound_catalog_pages", False, id="the-reserved-prefix"),
+ pytest.param("_bound_", False, id="the-reserved-prefix-alone"),
+ ],
+ )
+ def test_a_step_and_a_branch_keep_a_plain_batch_over_off_the_reserved_prefix(
+ self, schema: dict[str, Any], batch_over: str, should_validate: bool
+ ) -> None:
+ """A plain `batch_over` reads a name rather than storing one, and never `_bound_`, the runtime's prefix for a dotted `batch_over`'s list.
+
+ The runtime refuses it as `invalid_input_name` when the bundle is parsed, and the schema refuses it first, through a
+ Draft-4 `not` on the string arm, on a PipeSequence step and on a PipeParallel branch alike.
+ """
+ validator = _pipe_union_oneof_validator(schema)
+ pipe_step: dict[str, Any] = {"pipe": "describe_page", "batch_over": batch_over, "batch_as": "page", "result": "descriptions"}
+ sequence_table = {**_minimal_pipe_table("PipeSequence"), "steps": [pipe_step]}
+ parallel_table = {**_minimal_pipe_table("PipeParallel"), "branches": [pipe_step]}
+ assert validator.is_valid(sequence_table) is should_validate, f"sequence step batch_over {batch_over!r}"
+ assert validator.is_valid(parallel_table) is should_validate, f"parallel branch batch_over {batch_over!r}"
+
@pytest.mark.parametrize(
("input_item_name", "should_validate"),
[
pytest.param("item", True, id="a-plain-name"),
- pytest.param("_draft_item", True, id="another-underscore-led-name"),
+ pytest.param("item2", True, id="a-digit-after-the-first-letter"),
+ pytest.param("Item", False, id="pascal-case"),
+ pytest.param("catalog.item", False, id="dotted"),
+ pytest.param("_draft_item", False, id="underscore-led"),
pytest.param("_bound_item", False, id="the-reserved-prefix"),
],
)
- def test_batch_input_item_name_refuses_the_reserved_prefix(self, schema: dict[str, Any], input_item_name: str, should_validate: bool) -> None:
+ def test_batch_input_item_name_follows_the_plain_name_grammar(self, schema: dict[str, Any], input_item_name: str, should_validate: bool) -> None:
+ """A PipeBatch's `input_item_name` is a stored name, the branch pipe reading each item through an input of that name."""
validator = _pipe_union_oneof_validator(schema)
table = {**_minimal_pipe_table("PipeBatch"), "input_item_name": input_item_name}
assert validator.is_valid(table) is should_validate
+ @pytest.mark.parametrize(
+ "pattern_path",
+ [
+ pytest.param(("SubPipeBlueprint", "result"), id="step-result"),
+ pytest.param(("SubPipeBlueprint", "batch_as"), id="step-batch-as"),
+ pytest.param(("ParallelBranchBlueprint", "result"), id="branch-result"),
+ pytest.param(("ParallelBranchBlueprint", "batch_as"), id="branch-batch-as"),
+ pytest.param(("PipeBatchBlueprint", "input_item_name"), id="batch-input-item-name"),
+ pytest.param(("BindingStepBlueprint", "result"), id="binding-step-result"),
+ ],
+ )
+ def test_every_stored_name_carries_the_input_name_pattern(self, schema: dict[str, Any], pattern_path: tuple[str, str]) -> None:
+ """Each stored name's string arm carries the same input-name pattern, and nothing else, as a structural consumer reads it."""
+ definition_name, field_name = pattern_path
+ field_schema = schema["definitions"][definition_name]["properties"][field_name]
+ string_arms = [arm for arm in field_schema.get("anyOf", [field_schema]) if arm.get("type") == "string"]
+ assert len(string_arms) == 1
+ assert string_arms[0]["pattern"] == "^[a-z][a-z0-9_]*$"
+ assert "not" not in string_arms[0]
+
def test_minimal_table_coverage_matches_schema_pipe_kinds(self, schema: dict[str, Any]) -> None:
"""Guard: the test's per-kind table map covers exactly the *concrete* pipe kinds in the schema.
diff --git a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.py b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.py
index b1f0edd61..4e40aa440 100644
--- a/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.py
+++ b/tests/unit/pipelex/pipe_controllers/batch/test_pipe_batch_validation.py
@@ -86,36 +86,69 @@ class TestPipeBatchValidation:
)
@pytest.mark.parametrize(
- ("input_list_name", "input_item_name", "reserved_name", "message_fragment"),
+ "input_item_name",
[
- pytest.param(
- "items",
- "_bound_item",
- "_bound_item",
- "The PipeBatch's `input_item_name`, '_bound_item', takes the `_bound_` prefix, which is reserved for the bound list of a "
- "dotted `batch_over`",
- id="input-item-name",
- ),
- pytest.param("_bound_items", "item", "_bound_items", "is not a valid input name", id="input-list-name"),
+ pytest.param("Item", id="pascal-case"),
+ pytest.param("catalog.item", id="dotted"),
+ pytest.param("_bound_item", id="the-reserved-prefix"),
+ pytest.param("_item", id="underscore-led"),
+ pytest.param("2nd_item", id="digit-led"),
],
)
- def test_rejects_a_name_taking_the_reserved_prefix(
- self, input_list_name: str, input_item_name: str, reserved_name: str, message_fragment: str
- ) -> None:
- """The batch writes its item into the branch's memory under its own name, so the runtime's `_bound_` prefix is refused there too."""
+ def test_rejects_an_input_item_name_that_is_not_a_plain_input_name(self, input_item_name: str) -> None:
+ """The batch stores each item under `input_item_name` for the branch pipe to read through an input, so it is a plain name."""
with pytest.raises(ValidationError) as exc_info:
PipeBatchBlueprint(
description="Process each item",
inputs={"items": "Item[]"},
output="Result[]",
branch_pipe_code="process_item",
- input_list_name=input_list_name,
+ input_list_name="items",
input_item_name=input_item_name,
)
refusals = [raw_error.get("ctx", {}).get("error") for raw_error in exc_info.value.errors()]
- reserved_name_refusals = [
- refusal for refusal in refusals if isinstance(refusal, PipeValidationError) and refusal.variable_names == [reserved_name]
+ assert len(refusals) == 1
+ refusal = refusals[0]
+ assert isinstance(refusal, PipeValidationError)
+ assert refusal.error_type == PipeValidationErrorType.INVALID_INPUT_NAME
+ assert refusal.variable_names == [input_item_name]
+ assert f"The PipeBatch's `input_item_name`, '{input_item_name}', is not a plain input name" in str(refusal)
+
+ @pytest.mark.parametrize(
+ "input_item_name",
+ [
+ pytest.param("item", id="a-plain-name"),
+ pytest.param("bound_item", id="the-prefix-without-its-underscore"),
+ pytest.param("item2", id="a-digit-after-the-first-letter"),
+ ],
+ )
+ def test_accepts_a_plain_input_item_name(self, input_item_name: str) -> None:
+ blueprint = PipeBatchBlueprint(
+ description="Process each item",
+ inputs={"items": "Item[]"},
+ output="Result[]",
+ branch_pipe_code="process_item",
+ input_list_name="items",
+ input_item_name=input_item_name,
+ )
+
+ assert blueprint.input_item_name == input_item_name
+
+ def test_rejects_an_input_list_name_taking_the_reserved_prefix(self) -> None:
+ """The list's name is a plain input name, which no underscore-led name is."""
+ with pytest.raises(ValidationError) as exc_info:
+ PipeBatchBlueprint(
+ description="Process each item",
+ inputs={"items": "Item[]"},
+ output="Result[]",
+ branch_pipe_code="process_item",
+ input_list_name="_bound_items",
+ input_item_name="item",
+ )
+ refusals = [raw_error.get("ctx", {}).get("error") for raw_error in exc_info.value.errors()]
+ list_name_refusals = [
+ refusal for refusal in refusals if isinstance(refusal, PipeValidationError) and refusal.variable_names == ["_bound_items"]
]
- assert len(reserved_name_refusals) == 1
- assert reserved_name_refusals[0].error_type == PipeValidationErrorType.INVALID_INPUT_NAME
- assert message_fragment in str(reserved_name_refusals[0])
+ assert len(list_name_refusals) == 1
+ assert list_name_refusals[0].error_type == PipeValidationErrorType.INVALID_INPUT_NAME
+ assert "is not a valid input name" in str(list_name_refusals[0])
diff --git a/tests/unit/pipelex/pipe_controllers/sequence/test_sequence_step_union.py b/tests/unit/pipelex/pipe_controllers/sequence/test_sequence_step_union.py
index 233ff8322..06aeb5cc0 100644
--- a/tests/unit/pipelex/pipe_controllers/sequence/test_sequence_step_union.py
+++ b/tests/unit/pipelex/pipe_controllers/sequence/test_sequence_step_union.py
@@ -239,28 +239,31 @@ branches = [
assert refusal.error_type == PipeValidationErrorType.BINDING_STEP_INVALID
@pytest.mark.parametrize(
- ("step_toml", "field_name", "reserved_name"),
+ ("field_name", "refused_name", "message_fragment"),
[
- pytest.param('{ pipe = "write_receipt", result = "_bound_invoice_lines" }', "result", "_bound_invoice_lines", id="result"),
- pytest.param(
- '{ pipe = "write_receipt", batch_over = "lines", batch_as = "_bound_line", result = "receipts" }',
- "batch_as",
- "_bound_line",
- id="batch-as",
- ),
- pytest.param(
- '{ pipe = "write_receipt", batch_over = "_bound_invoice_lines", batch_as = "total_amount", result = "receipts" }',
- "batch_over",
- "_bound_invoice_lines",
- id="plain-batch-over",
- ),
+ pytest.param("result", "Receipts", "is not a plain input name", id="result-pascal-case"),
+ pytest.param("result", "invoice.receipts", "is not a plain input name", id="result-dotted"),
+ pytest.param("result", "_bound_invoice_lines", "is not a plain input name", id="result-reserved-prefix"),
+ pytest.param("batch_as", "TotalAmount", "is not a plain input name", id="batch-as-pascal-case"),
+ pytest.param("batch_as", "line.total", "is not a plain input name", id="batch-as-dotted"),
+ pytest.param("batch_as", "_bound_line", "is not a plain input name", id="batch-as-reserved-prefix"),
+ pytest.param("batch_over", "_bound_invoice_lines", "takes the `_bound_` prefix", id="plain-batch-over-reserved-prefix"),
],
)
@pytest.mark.parametrize("controller", ["sequence", "parallel"])
- def test_a_name_taking_the_reserved_prefix_is_refused_on_a_step_and_a_branch(
- self, controller: str, step_toml: str, field_name: str, reserved_name: str
+ def test_a_stored_name_that_is_not_plain_or_a_reserved_batch_over_is_refused_on_a_step_and_a_branch(
+ self, controller: str, field_name: str, refused_name: str, message_fragment: str
) -> None:
- """A nested sequence binds a dotted `batch_over` in its caller's working memory, under a `_bound_` name no author may write."""
+ """A step's `result` and `batch_as` are stored names, plain input names on a sequence step and a parallel branch alike.
+
+ A plain `batch_over` reads a name rather than storing one, and only stays off the `_bound_` prefix, under which a nested
+ sequence binds a dotted `batch_over` in its caller's working memory.
+ """
+ step_fields = {"result": "receipts", "batch_over": "lines", "batch_as": "total_amount", field_name: refused_name}
+ step_toml = (
+ f'{{ pipe = "write_receipt", batch_over = "{step_fields["batch_over"]}", batch_as = "{step_fields["batch_as"]}", '
+ f'result = "{step_fields["result"]}" }}'
+ )
pipe_code: str
mthds_content: str
if controller == "sequence":
@@ -286,9 +289,22 @@ branches = [
errors = exc_info.value.validation_errors
assert [error.error_type for error in errors] == [PipeValidationErrorType.INVALID_INPUT_NAME]
assert errors[0].pipe_code == pipe_code
- assert errors[0].variable_names == [reserved_name]
- assert f"The `{field_name}` of the step running pipe 'write_receipt', '{reserved_name}', takes the `_bound_` prefix" in errors[0].message
- assert "which is reserved for the bound list of a dotted `batch_over`" in errors[0].message
+ assert errors[0].variable_names == [refused_name]
+ assert f"The `{field_name}` of the step running pipe 'write_receipt', '{refused_name}', {message_fragment}" in errors[0].message
+
+ @pytest.mark.parametrize("controller", ["sequence", "parallel"])
+ def test_plain_stored_names_parse_on_a_step_and_a_branch(self, controller: str) -> None:
+ step = SubPipeBlueprint(pipe="write_receipt", batch_over="lines", batch_as="total_amount", result="receipts")
+ if controller == "sequence":
+ sequence = _parse_sequence(
+ steps_toml=' { pipe = "write_receipt", batch_over = "lines", batch_as = "total_amount", result = "receipts" },'
+ )
+ assert sequence.steps == [step]
+ else:
+ parallel = PipeParallelBlueprint.model_validate(
+ {"description": "Writes receipts in parallel", "output": "Composite", "branches": [step.model_dump(exclude_none=True)]}
+ )
+ assert parallel.branches == [step]
def test_a_plain_batch_over_in_a_parallel_branch_parses(self) -> None:
blueprint = PipeParallelBlueprint.model_validate(
diff --git a/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py b/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py
index 1a73e059a..520fd21ef 100644
--- a/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py
+++ b/tests/unit/pipelex/pipe_controllers/test_sub_pipe_blueprint.py
@@ -69,44 +69,89 @@ class TestSubPipeBlueprint:
)
@pytest.mark.parametrize(
- ("step_fields", "field_name", "reserved_name"),
+ "stored_name",
[
- pytest.param({"result": "_bound_catalog_pages"}, "result", "_bound_catalog_pages", id="result"),
- pytest.param({"batch_over": "catalog_pages", "batch_as": "_bound_page"}, "batch_as", "_bound_page", id="batch-as"),
- pytest.param({"batch_over": "_bound_catalog_pages", "batch_as": "page"}, "batch_over", "_bound_catalog_pages", id="plain-batch-over"),
+ pytest.param("Pages", id="pascal-case"),
+ pytest.param("catalog.pages", id="dotted"),
+ pytest.param("_bound_catalog_pages", id="the-reserved-prefix"),
+ pytest.param("_draft", id="underscore-led"),
+ pytest.param("2nd_pages", id="digit-led"),
+ pytest.param("page-views", id="hyphenated"),
+ pytest.param("...", id="dots-only"),
],
)
- def test_a_name_taking_the_reserved_prefix_is_refused_as_invalid_input_name(
- self, step_fields: dict[str, str], field_name: str, reserved_name: str
- ) -> None:
- """The `_bound_` prefix is the runtime's, for the list a dotted `batch_over` binds in a caller's working memory."""
+ @pytest.mark.parametrize("field_name", ["result", "batch_as"])
+ def test_a_stored_name_that_is_not_a_plain_input_name_is_refused_as_invalid_input_name(self, field_name: str, stored_name: str) -> None:
+ """A step stores its `result`, and each item under `batch_as`, for a later pipe to read through an input, so both are plain names."""
+ step_fields: dict[str, str] = {"result": "descriptions", "batch_over": "pages", "batch_as": "page", field_name: stored_name}
with pytest.raises(ValidationError) as exc_info:
SubPipeBlueprint.model_validate({"pipe": "describe_page", **step_fields})
refusal = exc_info.value.errors()[0].get("ctx", {}).get("error")
assert isinstance(refusal, PipeValidationError)
assert refusal.error_type == PipeValidationErrorType.INVALID_INPUT_NAME
- assert refusal.variable_names == [reserved_name]
- message = str(refusal)
- assert f"The `{field_name}` of the step running pipe 'describe_page', '{reserved_name}', takes the `_bound_` prefix" in message
- assert "reserved for the bound list of a dotted `batch_over`" in message
- assert "Choose another name" in message
+ assert refusal.variable_names == [stored_name]
+ assert f"The `{field_name}` of the step running pipe 'describe_page', '{stored_name}', is not a plain input name" in str(refusal)
+
+ @pytest.mark.parametrize(
+ ("stored_name", "message_fragment"),
+ [
+ pytest.param("Pages", "Rename it to a plain name, such as 'pages',", id="pascal-case"),
+ pytest.param("PagesOfTheCatalog", "Rename it to a plain name, such as 'pages_of_the_catalog',", id="a-longer-pascal-case-name"),
+ pytest.param("catalog.pages", "cannot reach into a field with a dot. Rename it to a plain name, such as 'catalog_pages',", id="dotted"),
+ pytest.param(
+ "_bound_catalog_pages",
+ "the `_bound_` prefix it starts with is reserved for the bound list of a dotted `batch_over`",
+ id="the-reserved-prefix",
+ ),
+ pytest.param("_bound_catalog_pages", "Rename it to a plain name, such as 'catalog_pages',", id="the-reserved-prefix-suggestion"),
+ pytest.param("page-views", "Rename it to a plain name, such as 'page_views',", id="hyphenated"),
+ pytest.param("...", "Rename it to a plain name, and", id="no-suggestion"),
+ ],
+ )
+ def test_the_refusal_explains_the_form_and_suggests_a_plain_name(self, stored_name: str, message_fragment: str) -> None:
+ with pytest.raises(ValidationError) as exc_info:
+ SubPipeBlueprint(pipe="describe_page", result=stored_name)
- def test_the_refusal_suggests_the_name_without_the_prefix(self) -> None:
+ assert message_fragment in str(exc_info.value)
+
+ def test_a_plain_batch_over_taking_the_reserved_prefix_is_refused_as_invalid_input_name(self) -> None:
+ """A plain `batch_over` reads a name rather than storing one, but never a name the runtime binds under its reserved prefix."""
with pytest.raises(ValidationError) as exc_info:
- SubPipeBlueprint(pipe="describe_page", result="_bound_catalog_pages")
+ SubPipeBlueprint(pipe="describe_page", batch_over="_bound_catalog_pages", batch_as="page")
- assert "Choose another name, such as 'catalog_pages'." in str(exc_info.value)
+ refusal = exc_info.value.errors()[0].get("ctx", {}).get("error")
+ assert isinstance(refusal, PipeValidationError)
+ assert refusal.error_type == PipeValidationErrorType.INVALID_INPUT_NAME
+ assert refusal.variable_names == ["_bound_catalog_pages"]
+ message = str(refusal)
+ assert "The `batch_over` of the step running pipe 'describe_page', '_bound_catalog_pages', takes the `_bound_` prefix" in message
+ assert "reserved for the bound list of a dotted `batch_over`" in message
+ assert "Choose another name, such as 'catalog_pages'." in message
@pytest.mark.parametrize(
- "name",
+ "stored_name",
[
+ pytest.param("pages", id="a-plain-name"),
pytest.param("bound_pages", id="the-prefix-without-its-underscore"),
- pytest.param("_draft", id="another-underscore-led-name"),
pytest.param("catalog_bound_pages", id="the-prefix-inside-the-name"),
+ pytest.param("page2", id="a-digit-after-the-first-letter"),
+ ],
+ )
+ def test_a_plain_stored_name_is_accepted(self, stored_name: str) -> None:
+ blueprint = SubPipeBlueprint(pipe="describe_page", result=stored_name, batch_over="catalog_pages", batch_as=stored_name)
+
+ assert blueprint.result == blueprint.batch_as == stored_name
+
+ @pytest.mark.parametrize(
+ "batch_over",
+ [
+ pytest.param("PagesOfTheCatalog", id="pascal-case"),
+ pytest.param("_draft_pages", id="underscore-led-outside-the-reserved-prefix"),
],
)
- def test_a_name_outside_the_reserved_prefix_is_accepted(self, name: str) -> None:
- blueprint = SubPipeBlueprint(pipe="describe_page", result=name, batch_over="pages", batch_as=name)
+ def test_a_plain_batch_over_is_held_only_to_the_reserved_prefix(self, batch_over: str) -> None:
+ """A plain `batch_over` reads a name rather than storing one, so the standard holds it only off the reserved prefix."""
+ blueprint = SubPipeBlueprint(pipe="describe_page", batch_over=batch_over, batch_as="page")
- assert blueprint.result == blueprint.batch_as == name
+ assert blueprint.batch_over == batch_over