Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

The error.* axis: an invalid-method coverage surface for the MTHDS Test Corpus (#1139)

Louis Choquela234f28387e7
diff --git a/.pipelex/plxt.toml b/.pipelex/plxt.toml
index a843f8e7e..7a53ce931 100644
--- a/.pipelex/plxt.toml
+++ b/.pipelex/plxt.toml
@@ -24,6 +24,14 @@ exclude = [
"node_modules/**",
".git/**",
"*.lock",
+ # The MTHDS Test Corpus's deliberately invalid entries — the one .mthds population this repo does
+ # not lint, because each one exists to be rejected. Their fault is sometimes structural (a typeless
+ # pipe section, an unknown pipe `type`), which the schema catches as the error the entry was written
+ # to trigger. What guards them instead is strictly stronger: the corpus's entry-validation gate runs
+ # the real validation engine over every entry in CI and is red both when an invalid entry fails
+ # differently and when it accidentally validates. Excluded rather than schema-exempted because a
+ # per-rule `exclude` does not lift the schema, only narrows which rule applies.
+ "**/mthds_corpus/entries/invalid_*/*.mthds",
] # Glob patterns for files to ignore.
# These are evaluated relative to the config file location.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1e1269c6a..fb67478e5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,8 +4,14 @@
### Added
+- **The corpus's invalid axis: an `error.*` tag namespace and the entries that cover it.** The MTHDS Test Corpus covered the valid language surface; the faults it can report were untagged and unexercised. `vocabulary.toml` now carries an `error.*` namespace generated from `VALIDATION_ERROR_TYPES`, and the corpus carries a focused invalid entry for every tag in it that a `.mthds` bundle can actually produce — each one authored to trigger exactly one error, and gated on doing so. The namespace needed no rule of its own despite its registry mixing two spellings: the existing normalization leaves an already-snake_case stage code untouched and turns `DryRunError` into `error.dry_run_error` exactly as it turns `YesNo` into `native.yes_no`. Five codes are excluded, each with the measurement behind it rather than an argument from the call graph — `circular_dependency_error` and `unknown_concept` are unreachable through bundle validation (the first is raised only by the builder's spec-to-blueprint conversion, the second is shadowed by the concept-reference resolution that reports the same fault as `unresolved_concept`), `optional_force_redundant` is advisory-only and rides the `warnings` array, and the two `unknown_*` fallbacks name the absence of a diagnosis rather than a language fault.
+- **The exhaustivity gate now requires an invalid entry to agree with itself.** Such an entry states its fault twice — as `expected_error`, the wire string, and as an `error.*` tag in `covers`, the normalized form — and nothing checked that the two matched, so an entry could advertise coverage of a fault it does not produce. The check joins through the vocabulary's own `code` field rather than re-normalizing, so it carries no second copy of the normalization rule. The same claim is gated from the other side too: a valid entry may not carry an `error.*` tag, since its bundle produces no diagnostic and the exhaustivity arm would otherwise count a claim nothing honours.
- **The `error_type` values a validation verdict can carry are now an enumerable, closed registry**: a `.mthds` bundle that fails validation reports each fault under an `error_type`, and until now there was no way to ask the runtime which faults exist. The values were reachable only by reading raise sites, and the only curated list of them lived outside this repo, in a hand-maintained conformance suite that could silently fall behind. `pipelex/validation_error_types.py` now holds the vocabulary in full and enumerates it as `VALIDATION_ERROR_TYPES`, so a consumer building coverage over the language surface — a test corpus, a client mapping faults onto its own UI — reads the registry instead of collecting strings from whichever diagnostics it happened to have seen. It is a **union of the enums the runtime already raises**, never a second list beside them: the two stage vocabularies plus a new `ValidationResidualErrorType` naming the one residual channel that had no enum of its own and rode an inline string. A member added to any of them is in the registry the moment it is declared. `ValidationErrorItem.error_type` is typed against their union, which is what makes the registry closed rather than merely documented — an unregistered string can no longer be constructed or parsed onto an item — and which publishes the vocabulary into the OpenAPI schema served for `/validate`, where the field was previously an open `string`. **No wire value changed**, including the dry-run residual's `DryRunError`: it keeps the exception-class spelling it has always had, because normalizing it would break every consumer that pins the string and would buy nothing an enumeration does not already give. Membership means a value is reachable on the wire, not that it is worth exercising — the advisory-only warning type and the two `unknown_*` fallbacks are in the registry and a coverage consumer excludes them on its own side, with a reason, rather than pruning the runtime's truth.
+### Fixed
+
+- **The same bundle produced different diagnostics depending on how it reached the validator**, and the library-directory route — the one the CLI and every local tool take — was the lossy one. Validating in-memory contents (what the HTTP `/validate` layer does) reported a pipe or concept fault as a categorized item with its `error_type` and locators; validating the identical bundle from a file resolved against a library directory reported one untyped residual carrying only a message. Two arms in `LibraryManager._load_mthds_files_into_library` were destroying the structured data on the way out: a pydantic `ValidationError` was rewrapped as a bare `LibraryError`, which routes to the error cascade's structured-forwarding arm rather than its categorizing arm, so nothing ever categorized it; and a `ConceptLibraryError` was re-raised message-only, dropping the per-reference items it now carries. In practice `unresolved_concept`, `optional_input_unguarded` and `optional_output_required` were simply unreachable from a file on disk. Both arms now preserve what they were discarding, and a new parity test validates the same bundle through both entry shapes and requires the two to agree.
+
### Changed
- **Two test fixtures were naming faults that do not exist**, and typing `error_type` against the registry is what surfaced them: one item claimed `MISSING_CONCEPT` — a code no enum has ever defined — and others used member names (`INADEQUATE_OUTPUT_CONCEPT`) where the wire carries values (`inadequate_output_concept`). Every one of them was accepted silently while the field was an open string, so the tests asserted on renderings of a fault the runtime cannot produce. They now name registry members.
diff --git a/docs/contribute/mthds-test-corpus.md b/docs/contribute/mthds-test-corpus.md
index 378ef864b..337d8bc15 100644
--- a/docs/contribute/mthds-test-corpus.md
+++ b/docs/contribute/mthds-test-corpus.md
@@ -64,13 +64,17 @@ An **invalid** entry additionally declares the exact wire `error_type` it must p
.venv/bin/pipelex-agent validate bundle <path> -L <entry_dir>/ --format json --error-format json
```
+Two rules shape an invalid entry beyond that. Its **`covers` list is its `error.*` tag and nothing else** — it is tempting to also tag the pipe kind that carries the fault, but the exhaustivity gate reads `covers` as the claim that a tag has a focused entry, and a deliberately broken bundle must never be what satisfies that claim for a working feature. And the two ways it names its fault have to agree: `expected_error` carries the wire string, `covers` carries the normalized tag, and the exhaustivity gate requires the entry to cover the tag whose `code` is its `expected_error`, so an entry cannot advertise coverage it does not produce. The rule has a second half, gated alongside it: a **valid** entry may not carry an `error.*` tag at all, because its bundle produces no diagnostic and the claim would go unhonoured in exactly the same way.
+
!!! warning "The `.mthds` editor hook fights invalid entries"
The `pipelex` plugin's `PostToolUse` hook lints, formats and **blocks on an invalid verdict** for every `Write`/`Edit` of a `.mthds` file, and has no skip mechanism. Author valid entries with the editing tools to collect the free lint; write an invalid entry's bundle through a shell heredoc, which the hook's matcher does not intercept.
+ An invalid entry's bundle is outside `make lint` and `make format` as well — `.pipelex/plxt.toml` excludes the whole `invalid_*` population, for the reason recorded beside that glob. So nothing formats it for you: match the layout of the entry next door by hand. Its `entry.toml` is still linted normally.
+
## The tag vocabulary
-Tags are namespaced and the vocabulary is closed: an entry covering a tag the vocabulary does not declare fails the gate. `native.*` is generated from `NativeConceptCode`. `operator.*` and `controller.*` are generated from `PipeType` — one registry, split by its own `category` property, so a pipe kind added tomorrow lands in the right namespace with nothing to update here. Their local names drop the `Pipe` prefix every code carries, since the namespace already says which half it is: `PipeLLM` becomes `operator.llm`, `PipeSequence` becomes `controller.sequence`. `error.*` arrives with its registry.
+Tags are namespaced and the vocabulary is closed: an entry covering a tag the vocabulary does not declare fails the gate. `native.*` is generated from `NativeConceptCode`. `operator.*` and `controller.*` are generated from `PipeType` — one registry, split by its own `category` property, so a pipe kind added tomorrow lands in the right namespace with nothing to update here. Their local names drop the `Pipe` prefix every code carries, since the namespace already says which half it is: `PipeLLM` becomes `operator.llm`, `PipeSequence` becomes `controller.sequence`. `error.*` is generated from `VALIDATION_ERROR_TYPES`, the closed registry of `error_type` values a validation diagnostic can carry — and it needs no rule of its own even though its registry mixes two spellings: the normalization steps leave an already-snake_case stage code (`missing_input_variable`) untouched and turn the dry-run residual `DryRunError` into `error.dry_run_error`, exactly as they turn `YesNo` into `native.yes_no`.
`feature.*` is the one hand-maintained namespace, because a language feature is a human-named cluster of behaviour rather than a registry entry — some span several blueprint fields (optionals show up as `?` in a concept spec), some none at all (a multi-file library is a property of the entry *directory*), and there is no feature registry in `pipelex` to walk. A tag there carries a `description` where a generated one carries `code`, since that sentence is the only place its meaning can live. **Add a feature tag in the same commit that lands its first covering entry, never ahead of it** — declaring one reds the exhaustivity gate until an entry covers it.
@@ -86,12 +90,16 @@ An **exclusion** marks a registry code that no standalone focused entry could me
No native concept is excluded — every one of them turned out to support a real entry. `PipeFunc` is excluded: it names a Python function that the runtime resolves against its function registry *at validation time*, so an entry declaring one fails with `Function '<name>' not found in registry` in every consumer that has not registered that exact function — and a cross-language consumer cannot register a Python function at all. Exercise `PipeFunc` in a consumer's own fixtures, where the function exists.
+`error.*` carries the rest of the exclusions, and the distinction they turn on is worth stating: **membership in the registry means a value is reachable on the wire, never that a focused entry can produce it.** A corpus entry is a `.mthds` bundle validated through `validate bundle`, so a fault the validate path cannot reach is excluded here rather than pruned from the runtime's own truth. Two are unreachable by construction — `circular_dependency_error` is raised only by the pipe sorter, which runs on the builder's spec-to-blueprint conversion and never on a parsed `.mthds` file, and `unknown_concept` fires in the pipe factory under exactly the condition the earlier concept-reference resolution has already rejected as `unresolved_concept`. One has no shape an entry could take: `optional_force_redundant` rides the validation report's `warnings` array and never makes a verdict invalid, and the manifest models no warnings axis. The two `unknown_*` fallbacks are excluded because an entry pinning one would be pinning the absence of a diagnosis, and would have to be rewritten the moment the fault got a code of its own.
+
+**Measure an exclusion, do not reason your way to one.** Every `error.*` exclusion reason above was established by writing the bundle that ought to trigger it and reading what the runtime actually said — which is how the first two turned out to be shadowed rather than merely awkward, and how the two the registry made look unreachable (`missing_pipe_type`, `native_concept_redeclaration`) turned out to have entries after all.
+
## The gates, and what a red one means
| Gate | Where | Red means |
|---|---|---|
| Vocabulary drift | `tests/unit/pipelex/test_extras/test_mthds_corpus_vocabulary.py` | A registry changed and the committed vocabulary was not regenerated. Run `make generate-corpus-vocabulary`. |
-| Exhaustivity | `tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py` | A vocabulary tag has no `focused` entry covering it — write the entry, or record an exclusion with a reason. Also fires when an entry covers a tag that is not in the vocabulary. |
+| Exhaustivity | `tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py` | A vocabulary tag has no `focused` entry covering it — write the entry, or record an exclusion with a reason. Also fires when an entry covers a tag that is not in the vocabulary, when an invalid entry's `covers` is not exactly the tag its `expected_error` names, and when a valid entry covers an `error.*` tag it cannot produce. |
| Manifest | `tests/unit/pipelex/test_extras/test_mthds_corpus_manifest.py` | The strict `entry.toml` model rejected something. |
| Layout and filters | `tests/unit/pipelex/test_extras/test_mthds_corpus_loader.py` | An entry's name does not match its directory, its bundle does not resolve, or the loader's filter semantics changed. |
| Entry validation | `tests/integration/pipelex/test_extras/test_mthds_corpus_entries.py` | A valid entry stopped validating, or an invalid one stopped failing with exactly its declared error. Runs over every entry regardless of tier, so an `inference`-tier entry that broke is caught without spending a token. |
diff --git a/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py b/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py
index 89aa954bf..c71b56787 100644
--- a/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py
+++ b/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py
@@ -21,6 +21,7 @@ from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.pipe_machinery.pipe_blueprint import PipeCategory, PipeType
from pipelex.runtime_hub import get_console
from pipelex.test_extras.mthds_corpus.vocabulary import vocabulary_path
+from pipelex.validation_error_types import VALIDATION_ERROR_TYPES, PipeFactoryErrorType, PipeValidationErrorType, ValidationErrorType
_ACRONYM_BOUNDARY_RE = re.compile(r"(?<=[A-Z0-9])(?=[A-Z][a-z])")
_WORD_BOUNDARY_RE = re.compile(r"(?<=[a-z0-9])(?=[A-Z])")
@@ -28,9 +29,9 @@ _WORD_BOUNDARY_RE = re.compile(r"(?<=[a-z0-9])(?=[A-Z])")
_HEADER = """# The MTHDS Test Corpus tag vocabulary — the closed set an entry's `covers` draws from.
#
# GENERATED by `pipelex-dev generate-corpus-vocabulary`. Do not edit this file: `native.*`,
-# `operator.*` and `controller.*` are derived from the runtime's own registries, `feature.*` is
-# hand-maintained in the generator because no registry describes a language feature, and every
-# exclusion reason is declared in the generator's source alongside them.
+# `operator.*`, `controller.*` and `error.*` are derived from the runtime's own registries,
+# `feature.*` is hand-maintained in the generator because no registry describes a language feature,
+# and every exclusion reason is declared in the generator's source alongside them.
#
# This file ships in the pipelex wheel, so it names a document its readers can reach:
# https://docs.pipelex.com/contribute/mthds-test-corpus/ — which in turn points at the
@@ -106,6 +107,55 @@ _PIPE_TYPE_EXCLUSIONS: dict[PipeType, str] = {
}
+# `error.*` is generated from `VALIDATION_ERROR_TYPES`, the closed registry of `error_type` values a
+# validation diagnostic can carry. It is the one generated namespace whose registry is already partly
+# snake_case: the stage enums spell their codes `missing_input_variable`, the dry-run residual spells
+# itself `DryRunError`. `normalize_registry_code` absorbs both with no namespace-local rule — the
+# three normalization steps leave an already-snake_case code untouched — and `code` records each
+# registry spelling verbatim, exactly as `native.yes_no` records `code = "YesNo"`.
+#
+# Membership in the registry means a value is *reachable on the wire*, never that a focused entry can
+# produce it. A corpus entry is a `.mthds` bundle validated through `validate bundle`, so a type the
+# validate path cannot reach is excluded here with the measurement that established it — the runtime's
+# truth is not pruned to match the corpus's reach. Keyed on the enums for the same reason the other
+# exclusion maps are: a code that leaves the registry breaks this module at import instead of leaving
+# a dead exclusion behind in the generated file.
+_ERROR_TYPE_EXCLUSIONS: dict[ValidationErrorType, str] = {
+ PipeValidationErrorType.CIRCULAR_DEPENDENCY_ERROR: (
+ "Raised only from the pipe sorter, and the pipe sorter runs only on `PipelexBundleSpec.to_blueprint()` "
+ "— the builder's spec-to-blueprint conversion, which is how an AI-authored spec becomes a bundle. A "
+ "`.mthds` file parses straight into a blueprint and never passes through it, so no corpus entry can "
+ "produce this. Measured rather than read off the call graph: a self-referencing PipeSequence fails the "
+ "dry run with `DryRunError`, and a mutually recursive PipeCondition pair exhausts the interpreter's "
+ "stack — neither reports a circular dependency."
+ ),
+ PipeValidationErrorType.OPTIONAL_FORCE_REDUNDANT: (
+ "Advisory-only: it rides the validation report's `warnings` array and never makes a verdict invalid. An "
+ "entry contract that models one `expected_error` on an invalid entry has nowhere to put a warning on a "
+ "valid one, so there is no shape a focused entry could take. Revisit if the manifest ever grows a "
+ "warnings axis."
+ ),
+ PipeValidationErrorType.UNKNOWN_VALIDATION_ERROR: (
+ "The generic fallback for a validation failure that matched no specific code. Reaching it on purpose "
+ "would mean authoring a bundle that breaks in a way the categorizer does not recognize — which is a "
+ "state to fix in the runtime, not to pin in a corpus entry that would then have to be rewritten the "
+ "moment the fault gets a code of its own."
+ ),
+ PipeFactoryErrorType.UNKNOWN_CONCEPT: (
+ "Shadowed: it fires in the pipe factory when a pipe's output names a concept that is neither native, "
+ "nor declared in the pipe's own domain, nor domain-qualified — exactly the condition the earlier "
+ "concept-reference resolution already rejects, and it reports the same fault as `unresolved_concept`. "
+ "Measured across unqualified, cross-domain and qualified-but-missing output refs: every one of them "
+ "comes back as `unresolved_concept`, so the factory's own check is unreachable from a `.mthds` file."
+ ),
+ PipeFactoryErrorType.UNKNOWN_FACTORY_ERROR: (
+ "The generic fallback for a pipe-construction failure that matched no specific code, and excluded for "
+ "the same reason as its validation counterpart: an entry pinning it would be pinning the absence of a "
+ "diagnosis rather than a language fault."
+ ),
+}
+
+
def normalize_registry_code(*, code: str) -> str:
"""Turn a CamelCase registry code into a snake_case tag local name.
@@ -141,6 +191,17 @@ def _render_feature_namespace() -> str:
return "\n\n".join(blocks)
+def _render_error_namespace() -> str:
+ blocks: list[str] = []
+ for error_type in VALIDATION_ERROR_TYPES:
+ lines = [f"[error.{normalize_registry_code(code=error_type.value)}]", f"code = {_toml_string(value=error_type.value)}"]
+ exclusion_reason = _ERROR_TYPE_EXCLUSIONS.get(error_type)
+ if exclusion_reason is not None:
+ lines.append(f"excluded = {_toml_string(value=exclusion_reason)}")
+ blocks.append("\n".join(lines))
+ return "\n\n".join(blocks)
+
+
def _pipe_type_block(*, pipe_type: PipeType, namespace: str) -> str:
local_name = normalize_registry_code(code=pipe_type.value.removeprefix(_PIPE_TYPE_PREFIX))
lines = [f"[{namespace}.{local_name}]", f"code = {_toml_string(value=pipe_type.value)}"]
@@ -176,10 +237,12 @@ def generate_corpus_vocabulary_content() -> str:
Each generated namespace follows its registry's declaration order, and the two pipe namespaces
are emitted one after the other rather than interleaved — the file is read by humans choosing an
entry's `covers`, and grouping should not depend on where a future pipe kind is inserted into
- the enum. `feature.*` has no registry to follow, so it renders alphabetically; it goes last
- because it is the one namespace a reader cannot predict from a registry.
+ the enum. `error.*` follows `VALIDATION_ERROR_TYPES`, which is itself ordered by contributing enum
+ and then by declaration order, so a new fault shows up as one added block rather than a reshuffle.
+ `feature.*` has no registry to follow, so it renders alphabetically; it goes last because it is
+ the one namespace a reader cannot predict from a registry.
"""
- namespaces = [_render_native_namespace(), *_render_pipe_type_namespaces(), _render_feature_namespace()]
+ namespaces = [_render_native_namespace(), *_render_pipe_type_namespaces(), _render_error_namespace(), _render_feature_namespace()]
return f"{_HEADER}\n" + "\n\n".join(namespaces) + "\n"
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index 9368d238a..5b2ec850d 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -44,6 +44,7 @@ from pipelex.libraries.library_utils import (
from pipelex.libraries.pipe.exceptions import PipeLibraryError
from pipelex.libraries.visibility_utils import check_visibility_for_blueprints, make_visibility_checker
from pipelex.mthds_parsing.exceptions import MthdsParserError
+from pipelex.mthds_parsing.handle_pipe_errors import categorize_pipe_validation_error
from pipelex.mthds_parsing.parser import MthdsParser
from pipelex.mthds_parsing.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.pipe_machinery.pipe_abstract import PipeAbstract
@@ -773,18 +774,36 @@ class LibraryManager(LibraryManagerAbstract):
try:
return self.load_from_blueprints(library_id=library_id, blueprints=blueprints)
except ValidationError as validation_error:
- validation_error_msg = report_validation_error(validation_error=validation_error).message
- msg = f"Could not load blueprints from {[str(pth) for pth in valid_mthds_paths]} because of: {validation_error_msg}"
- raise LibraryError(
- message=msg,
+ # Categorize before wrapping. `translate_to_validate_bundle_error` has its own
+ # `except ValidationError` arm that categorizes, but it never sees this error: wrapping it
+ # in a bare `LibraryError` here routed it to the `except LibraryError` arm instead, which
+ # forwards structured data and has none to forward — so every pipe/concept error a
+ # pydantic validator raises during the merge arrived on the wire as one untyped residual.
+ # That is the whole reason `optional_input_unguarded` and `optional_output_required` were
+ # reachable through `resolve_crate_from_contents` but not through a library directory.
+ raise LibraryLoadingError(
+ message=(
+ f"Could not load blueprints from {[str(pth) for pth in valid_mthds_paths]} because of: "
+ f"{report_validation_error(validation_error=validation_error).message}"
+ ),
+ pipe_concept_validation_errors=categorize_pipe_validation_error(validation_error=validation_error),
) from validation_error
except (ConceptLibraryError, PipeLibraryError) as ref_error:
# Merge-time reference errors (undeclared cross-file concept refs, signature/concrete
- # contract mismatches, duplicate refs) carry no structured validation payload — unlike
- # LibraryLoadingError, which must propagate intact so its aggregated errors survive.
- # Add the batch's file list as context, then let the outer
- # translate_to_validate_bundle_error LibraryError arm shape it into a ValidateBundleError.
+ # contract mismatches, duplicate refs). Add the batch's file list as context, then let the
+ # outer translate_to_validate_bundle_error LibraryError arm shape it into a
+ # ValidateBundleError. Re-raising rather than mutating keeps the message immutable, and
+ # carrying the payload across is what keeps the re-raise from being lossy:
+ # `ConceptLibraryError` is a `LibraryLoadingError` and reaches here holding the per-reference
+ # items that make an unresolved concept a categorized `unresolved_concept` rather than a
+ # bare residual. Reconstructing it message-only silently dropped them.
msg = f"Could not load blueprints from {[str(pth) for pth in valid_mthds_paths]} because of: {ref_error}"
+ if isinstance(ref_error, LibraryLoadingError):
+ raise type(ref_error)(
+ msg,
+ blueprint_validation_errors=ref_error.blueprint_validation_errors,
+ pipe_concept_validation_errors=ref_error.pipe_concept_validation_errors,
+ ) from ref_error
raise type(ref_error)(msg) from ref_error
def _warn_if_mthds_version_unsatisfied(
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_batch_item_name_collision/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_batch_item_name_collision/bundle.mthds
new file mode 100644
index 000000000..33b75d76c
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_batch_item_name_collision/bundle.mthds
@@ -0,0 +1,25 @@
+domain = "crate_labels"
+description = "Write a label for every crate in a shipment"
+main_pipe = "label_crates"
+
+[concept]
+Crate = "One crate in a shipment"
+CrateLabel = "The label to stick on a crate"
+
+[pipe.label_crates]
+type = "PipeBatch"
+description = "Write a label for each crate in turn"
+inputs = { crates = "Crate[]" }
+output = "CrateLabel[]"
+branch_pipe_code = "label_one_crate"
+input_list_name = "crates"
+input_item_name = "crates"
+
+# The branch pipe is well-formed and its input matches `input_item_name`, so the collision above
+# is the only fault whatever order validation runs in.
+[pipe.label_one_crate]
+type = "PipeLLM"
+description = "Write the label for one crate"
+inputs = { crates = "Crate" }
+output = "CrateLabel"
+prompt = "Write the label for this crate: $crates"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_batch_item_name_collision/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_batch_item_name_collision/entry.toml
new file mode 100644
index 000000000..014a8ed70
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_batch_item_name_collision/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_batch_item_name_collision"
+description = "A label written for every crate in a shipment, where the crate list and the single crate are given one and the same name"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.batch_item_name_collision",
+]
+
+expected_error = "batch_item_name_collision"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_dry_run_residual/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_dry_run_residual/bundle.mthds
new file mode 100644
index 000000000..ba35e7b6e
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_dry_run_residual/bundle.mthds
@@ -0,0 +1,30 @@
+domain = "parcel_routing"
+description = "Send a parcel down the express lane or the standard lane"
+main_pipe = "route_parcel"
+
+[concept]
+Parcel = "A parcel waiting at the sorting bench"
+
+# The rule renders to nothing at all, so the round has no lane to take and fails on the way through.
+[pipe.route_parcel]
+type = "PipeCondition"
+description = "Choose the lane a parcel goes down"
+inputs = { parcel = "Parcel" }
+output = "Parcel"
+expression = "''"
+outcomes = { express = "send_express", standard = "send_standard" }
+default_outcome = "send_standard"
+
+[pipe.send_express]
+type = "PipeLLM"
+description = "Note the parcel onto the express lane"
+inputs = { parcel = "Parcel" }
+output = "Parcel"
+prompt = "Note this parcel onto the express lane: $parcel"
+
+[pipe.send_standard]
+type = "PipeLLM"
+description = "Note the parcel onto the standard lane"
+inputs = { parcel = "Parcel" }
+output = "Parcel"
+prompt = "Note this parcel onto the standard lane: $parcel"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_dry_run_residual/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_dry_run_residual/entry.toml
new file mode 100644
index 000000000..6af7d7f7c
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_dry_run_residual/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_dry_run_residual"
+description = "A parcel routed on a rule that decides nothing, so the round reaches the sorting bench with no lane to take"
+
+validity = "invalid"
+tier = "dry"
+granularity = "focused"
+
+covers = [
+ "error.dry_run_error",
+]
+
+expected_error = "DryRunError"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_extraneous_input_variable/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_extraneous_input_variable/bundle.mthds
new file mode 100644
index 000000000..a6cc258c8
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_extraneous_input_variable/bundle.mthds
@@ -0,0 +1,10 @@
+domain = "delay_notices"
+description = "Write the notice announcing that a service is running late"
+main_pipe = "write_delay_notice"
+
+[pipe.write_delay_notice]
+type = "PipeLLM"
+description = "Write the delay notice for a service"
+inputs = { service = "Text", platform = "Text" }
+output = "Text"
+prompt = "Write the notice announcing that the $service is running late."
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_extraneous_input_variable/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_extraneous_input_variable/entry.toml
new file mode 100644
index 000000000..d01c608c6
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_extraneous_input_variable/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_extraneous_input_variable"
+description = "A delay notice that asks for the platform number and then never mentions it"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.extraneous_input_variable",
+]
+
+expected_error = "extraneous_input_variable"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_concept/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_concept/bundle.mthds
new file mode 100644
index 000000000..1eee3ef5e
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_concept/bundle.mthds
@@ -0,0 +1,10 @@
+domain = "garden_signs"
+description = "Paint the sign that stands at the end of a flower bed"
+main_pipe = "paint_garden_sign"
+
+[pipe.paint_garden_sign]
+type = "PipeImgGen"
+description = "Paint the sign for a flower bed"
+inputs = { bed_note = "Text" }
+output = "Text"
+prompt = "A painted wooden garden sign for $bed_note"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_concept/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_concept/entry.toml
new file mode 100644
index 000000000..c641f4cc3
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_concept/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_inadequate_output_concept"
+description = "A garden sign that is painted, and then asked for back as written words rather than as a picture"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.inadequate_output_concept",
+]
+
+expected_error = "inadequate_output_concept"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_multiplicity/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_multiplicity/bundle.mthds
new file mode 100644
index 000000000..9fb3f99fc
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_multiplicity/bundle.mthds
@@ -0,0 +1,17 @@
+domain = "menu_cards"
+description = "Write the menu card from the kitchen's note of what it has in"
+main_pipe = "write_menu_card"
+
+[pipe.write_menu_card]
+type = "PipeSequence"
+description = "Write the menu card, promising two of them"
+inputs = { kitchen_note = "Text" }
+output = "Text[2]"
+steps = [{ pipe = "write_card", result = "card" }]
+
+[pipe.write_card]
+type = "PipeLLM"
+description = "Write one menu card"
+inputs = { kitchen_note = "Text" }
+output = "Text"
+prompt = "Write the menu card from this kitchen note: $kitchen_note"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_multiplicity/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_multiplicity/entry.toml
new file mode 100644
index 000000000..48767f565
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_inadequate_output_multiplicity/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_inadequate_output_multiplicity"
+description = "A menu card promised in two copies and written in one"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.inadequate_output_multiplicity",
+]
+
+expected_error = "inadequate_output_multiplicity"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch/bundle.mthds
new file mode 100644
index 000000000..88410e69d
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch/bundle.mthds
@@ -0,0 +1,13 @@
+domain = "opening_hours"
+description = "Set out a shop's opening hours as a filled-in table"
+main_pipe = "set_out_opening_hours"
+
+[concept.OpeningHours]
+description = "The days a shop is open and the hours it keeps"
+structure = { weekday = "text", hours = "text" }
+
+[pipe.set_out_opening_hours]
+type = "PipeStructure"
+description = "Set out the opening hours as a table"
+inputs = { shop_front = "Image" }
+output = "OpeningHours"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch/entry.toml
new file mode 100644
index 000000000..7fa5cc00d
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_input_stuff_spec_mismatch/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_input_stuff_spec_mismatch"
+description = "A shop's opening hours set out from a photograph of the shop front, handed to a step that reads only writing"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.input_stuff_spec_mismatch",
+]
+
+expected_error = "input_stuff_spec_mismatch"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_llm_output_cannot_be_image/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_llm_output_cannot_be_image/bundle.mthds
new file mode 100644
index 000000000..1d8b96d9b
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_llm_output_cannot_be_image/bundle.mthds
@@ -0,0 +1,10 @@
+domain = "parcel_labels"
+description = "Say in writing what is printed on a parcel label"
+main_pipe = "describe_parcel_label"
+
+[pipe.describe_parcel_label]
+type = "PipeLLM"
+description = "Say what a parcel label reads"
+inputs = { label_note = "Text" }
+output = "Image"
+prompt = "Say what this parcel label reads: $label_note"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_llm_output_cannot_be_image/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_llm_output_cannot_be_image/entry.toml
new file mode 100644
index 000000000..8837b3b18
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_llm_output_cannot_be_image/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_llm_output_cannot_be_image"
+description = "A parcel label read out in writing, with a photograph expected back instead of the words"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.llm_output_cannot_be_image",
+]
+
+expected_error = "llm_output_cannot_be_image"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_missing_input_variable/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_missing_input_variable/bundle.mthds
new file mode 100644
index 000000000..d8bf61054
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_missing_input_variable/bundle.mthds
@@ -0,0 +1,10 @@
+domain = "berth_notes"
+description = "Write the note telling a vessel where and when to moor"
+main_pipe = "write_berth_note"
+
+[pipe.write_berth_note]
+type = "PipeLLM"
+description = "Write the berth note for a vessel"
+inputs = { vessel = "Text" }
+output = "Text"
+prompt = "Write the berth note for $vessel arriving on the $tide tide."
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_missing_input_variable/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_missing_input_variable/entry.toml
new file mode 100644
index 000000000..8d124fbeb
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_missing_input_variable/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_missing_input_variable"
+description = "A berth note whose wording reaches for a tide the note never asks anyone for"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.missing_input_variable",
+]
+
+expected_error = "missing_input_variable"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_missing_pipe_type/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_missing_pipe_type/bundle.mthds
new file mode 100644
index 000000000..18c4a2610
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_missing_pipe_type/bundle.mthds
@@ -0,0 +1,9 @@
+domain = "postal_rounds"
+description = "Write the round sheet a postal walker carries out with them"
+main_pipe = "write_round_sheet"
+
+[pipe.write_round_sheet]
+description = "Write the round sheet for a walker"
+inputs = { walk_note = "Text" }
+output = "Text"
+prompt = "Write the round sheet from this walk note: $walk_note"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_missing_pipe_type/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_missing_pipe_type/entry.toml
new file mode 100644
index 000000000..5440e7ac4
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_missing_pipe_type/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_missing_pipe_type"
+description = "A postal round sheet written by a step that says what it does and never says what kind of step it is"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.missing_pipe_type",
+]
+
+expected_error = "missing_pipe_type"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_native_concept_redeclaration/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_native_concept_redeclaration/bundle.mthds
new file mode 100644
index 000000000..d2d9a8936
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_native_concept_redeclaration/bundle.mthds
@@ -0,0 +1,13 @@
+domain = "allotment_plots"
+description = "Write the notice that goes up on an allotment gate"
+main_pipe = "write_gate_notice"
+
+[concept]
+Text = "A line of writing on an allotment notice"
+
+[pipe.write_gate_notice]
+type = "PipeLLM"
+description = "Write the notice for the allotment gate"
+inputs = { plot_note = "Text" }
+output = "Text"
+prompt = "Write the gate notice from this plot note: $plot_note"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_native_concept_redeclaration/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_native_concept_redeclaration/entry.toml
new file mode 100644
index 000000000..e599749f4
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_native_concept_redeclaration/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_native_concept_redeclaration"
+description = "An allotment gate notice whose bundle gives its own meaning to a name the language already reserves"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.native_concept_redeclaration",
+]
+
+expected_error = "native_concept_redeclaration"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_branch_required_field/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_branch_required_field/bundle.mthds
new file mode 100644
index 000000000..a611a5ac5
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_branch_required_field/bundle.mthds
@@ -0,0 +1,41 @@
+domain = "weather_reports"
+description = "Put the morning weather report together from the rain-gauge readings and, when it comes in, the coastal station's word"
+main_pipe = "write_weather_report"
+
+[concept]
+GaugeReading = "The readings taken off the rain gauge"
+CoastalWord = "What the coastal station sent over, when it sent anything"
+
+[concept.WeatherReport]
+description = "The morning weather report"
+
+[concept.WeatherReport.structure]
+inland = { type = "text", description = "What the gauge readings say", required = true }
+coastal = { type = "text", description = "What the coastal station says", required = true }
+
+# `coastal_word` may not arrive, the branch that reads it takes it plainly, and the report
+# insists on a coastal line all the same.
+[pipe.write_weather_report]
+type = "PipeParallel"
+description = "Write both halves of the report at once"
+inputs = { gauge = "GaugeReading", coastal_word = "CoastalWord?" }
+output = "WeatherReport"
+add_each_output = false
+branches = [
+ { pipe = "read_gauge", result = "inland" },
+ { pipe = "read_coastal_word", result = "coastal" },
+]
+
+[pipe.read_gauge]
+type = "PipeLLM"
+description = "Write up what the gauge readings say"
+inputs = { gauge = "GaugeReading" }
+output = "Text"
+prompt = "Write up what these gauge readings say: $gauge"
+
+[pipe.read_coastal_word]
+type = "PipeLLM"
+description = "Write up what the coastal station sent"
+inputs = { coastal_word = "CoastalWord" }
+output = "Text"
+prompt = "Write up what the coastal station sent: $coastal_word"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_branch_required_field/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_branch_required_field/entry.toml
new file mode 100644
index 000000000..15bddc606
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_branch_required_field/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_optional_branch_required_field"
+description = "A weather report whose coastal line is insisted upon, and whose coastal station may send nothing"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.optional_branch_required_field",
+]
+
+expected_error = "optional_branch_required_field"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_input_unguarded/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_input_unguarded/bundle.mthds
new file mode 100644
index 000000000..79828cc16
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_input_unguarded/bundle.mthds
@@ -0,0 +1,18 @@
+domain = "library_loans"
+description = "Write the reminder that a borrowed book is due back, naming the shelf it belongs on when the desk noted one"
+main_pipe = "write_loan_reminder"
+
+[concept]
+BookTitle = "The title of a borrowed book"
+Shelf = "The shelf a book came off, when the desk noted one"
+Reminder = "The reminder sent to a borrower"
+
+[pipe.write_loan_reminder]
+type = "PipeLLM"
+description = "Write the reminder for a borrowed book"
+inputs = { book_title = "BookTitle", shelf = "Shelf?" }
+output = "Reminder"
+prompt = """
+Write a reminder that $book_title is due back.
+Say which shelf it belongs on: $shelf.
+"""
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_input_unguarded/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_input_unguarded/entry.toml
new file mode 100644
index 000000000..4422936ca
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_input_unguarded/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_optional_input_unguarded"
+description = "A library loan reminder that names the shelf a book belongs on without allowing for the desk having noted none"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.optional_input_unguarded",
+]
+
+expected_error = "optional_input_unguarded"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_marker_invalid/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_marker_invalid/bundle.mthds
new file mode 100644
index 000000000..13115e0ea
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_marker_invalid/bundle.mthds
@@ -0,0 +1,14 @@
+domain = "meeting_minutes"
+description = "Boil the notes taken at a meeting down into its minutes"
+main_pipe = "write_minutes"
+
+[concept]
+MeetingNote = "One note taken during a meeting"
+Minutes = "The minutes of a meeting"
+
+[pipe.write_minutes]
+type = "PipeLLM"
+description = "Write the minutes from the notes taken"
+inputs = { notes = "MeetingNote[]?" }
+output = "Minutes"
+prompt = "Write the minutes from these notes: $notes"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_marker_invalid/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_marker_invalid/entry.toml
new file mode 100644
index 000000000..44cafdf2c
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_marker_invalid/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_optional_marker_invalid"
+description = "A set of meeting minutes drawn from notes that are marked both as several and as possibly absent"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.optional_marker_invalid",
+]
+
+expected_error = "optional_marker_invalid"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_not_handled/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_not_handled/bundle.mthds
new file mode 100644
index 000000000..a1bdbdedf
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_not_handled/bundle.mthds
@@ -0,0 +1,21 @@
+domain = "bakery_orders"
+description = "Write the docket that goes to the bakery bench, from the flavour a customer named when they named one"
+main_pipe = "write_order_docket"
+
+[concept]
+Flavour = "The flavour a customer named, when they named one"
+Docket = "The docket that goes to the bakery bench"
+
+[pipe.write_order_docket]
+type = "PipeSequence"
+description = "Write the bakery docket"
+inputs = { flavour = "Flavour?" }
+output = "Docket"
+steps = [{ pipe = "draft_docket", result = "docket" }]
+
+[pipe.draft_docket]
+type = "PipeLLM"
+description = "Draft the docket for the bench"
+inputs = { flavour = "Flavour" }
+output = "Docket"
+prompt = "Write the bakery docket for a $flavour order."
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_not_handled/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_not_handled/entry.toml
new file mode 100644
index 000000000..bd80feaf7
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_not_handled/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_optional_not_handled"
+description = "A bakery docket promised outright, drawn from a flavour the customer may never have named"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.optional_not_handled",
+]
+
+expected_error = "optional_not_handled"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_output_required/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_output_required/bundle.mthds
new file mode 100644
index 000000000..6ae9bd6a3
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_output_required/bundle.mthds
@@ -0,0 +1,25 @@
+domain = "campsite_bookings"
+description = "Answer a campsite booking once the warden has approved it, and let it lie when they have not"
+main_pipe = "answer_booking"
+
+[concept]
+Verdict = "What the warden decided about a booking"
+Answer = "The answer sent back to the camper"
+
+[pipe.answer_booking]
+type = "PipeCondition"
+description = "Answer the booking, or let it lie"
+inputs = { verdict = "Verdict" }
+output = "Answer"
+expression_template = "{{ verdict.text }}"
+default_outcome = "continue"
+
+[pipe.answer_booking.outcomes]
+approved = "write_answer"
+
+[pipe.write_answer]
+type = "PipeLLM"
+description = "Write the answer that goes back to the camper"
+inputs = { verdict = "Verdict" }
+output = "Answer"
+prompt = "Write the answer to the camper for: $verdict"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_optional_output_required/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_output_required/entry.toml
new file mode 100644
index 000000000..c9d3fcfb0
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_optional_output_required/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_optional_output_required"
+description = "A campsite booking answered when the warden approves and left to lie when they do not, while the answer is promised either way"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.optional_output_required",
+]
+
+expected_error = "optional_output_required"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_pipe_code_syntax/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_pipe_code_syntax/bundle.mthds
new file mode 100644
index 000000000..f565cea22
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_pipe_code_syntax/bundle.mthds
@@ -0,0 +1,14 @@
+domain = "laundry_tickets"
+description = "Write the ticket that goes with a bag of laundry left at the counter"
+main_pipe = "Not A Valid Step Name!"
+
+[concept]
+LaundryBag = "A bag of laundry left at the counter"
+Ticket = "The ticket that goes with a bag of laundry"
+
+[pipe.write_laundry_ticket]
+type = "PipeLLM"
+description = "Write the ticket for a bag of laundry"
+inputs = { laundry_bag = "LaundryBag" }
+output = "Ticket"
+prompt = "Write the ticket for this bag: $laundry_bag"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_pipe_code_syntax/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_pipe_code_syntax/entry.toml
new file mode 100644
index 000000000..78dc293d1
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_pipe_code_syntax/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_pipe_code_syntax"
+description = "A laundry ticket whose method names its first step in words no step can be called"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.invalid_pipe_code_syntax",
+]
+
+expected_error = "invalid_pipe_code_syntax"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_unknown_pipe_type/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_unknown_pipe_type/bundle.mthds
new file mode 100644
index 000000000..d6710425d
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_unknown_pipe_type/bundle.mthds
@@ -0,0 +1,10 @@
+domain = "hardware_catalog"
+description = "Write the catalog line for an item on the hardware shelf"
+main_pipe = "write_catalog_line"
+
+[pipe.write_catalog_line]
+type = "PipeWondrous"
+description = "Write the catalog line for a shelf item"
+inputs = { shelf_item = "Text" }
+output = "Text"
+prompt = "Write the catalog line for $shelf_item"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_unknown_pipe_type/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_unknown_pipe_type/entry.toml
new file mode 100644
index 000000000..e8c5c553d
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_unknown_pipe_type/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_unknown_pipe_type"
+description = "A hardware catalog line written by a kind of step the language does not have"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.unknown_pipe_type",
+]
+
+expected_error = "unknown_pipe_type"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept/bundle.mthds
new file mode 100644
index 000000000..e3aa015cd
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept/bundle.mthds
@@ -0,0 +1,10 @@
+domain = "orchard_yields"
+description = "Count the crates of fruit a row of trees gave this season"
+main_pipe = "count_orchard_crates"
+
+[pipe.count_orchard_crates]
+type = "PipeLLM"
+description = "Count the crates a row of trees gave"
+inputs = { picking_note = "Text" }
+output = "CrateTally"
+prompt = "Count the crates recorded on this picking note: $picking_note"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept/entry.toml
new file mode 100644
index 000000000..0ef64147f
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_unresolved_concept"
+description = "A count of orchard crates handed back as a tally nobody in the method has described"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.unresolved_concept",
+]
+
+expected_error = "unresolved_concept"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept_owned/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept_owned/bundle.mthds
new file mode 100644
index 000000000..3e0d8bc08
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept_owned/bundle.mthds
@@ -0,0 +1,13 @@
+domain = "bike_repairs"
+description = "Write up the repair a bicycle needs, in the shop's own words"
+main_pipe = "write_repair_note"
+
+[concept]
+RepairNote = { description = "The shop's write-up of a repair", refines = "WorkshopEntry" }
+
+[pipe.write_repair_note]
+type = "PipeLLM"
+description = "Write up the repair a bicycle needs"
+inputs = { fault_report = "Text" }
+output = "RepairNote"
+prompt = "Write up the repair this bicycle needs: $fault_report"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept_owned/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept_owned/entry.toml
new file mode 100644
index 000000000..ce51ca658
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_concept_owned/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_unresolved_concept_owned"
+description = "A bicycle repair note described as a narrower kind of a workshop entry that the method never describes"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.unresolved_concept",
+]
+
+expected_error = "unresolved_concept"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_pipe_dependency/bundle.mthds b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_pipe_dependency/bundle.mthds
new file mode 100644
index 000000000..a067ece0e
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_pipe_dependency/bundle.mthds
@@ -0,0 +1,20 @@
+domain = "ferry_crossings"
+description = "Draw up the sailing notice for a ferry crossing in two passes"
+main_pipe = "write_sailing_notice"
+
+[pipe.write_sailing_notice]
+type = "PipeSequence"
+description = "Draw up the sailing notice"
+inputs = { crossing = "Text" }
+output = "Text"
+steps = [
+ { pipe = "draft_notice", result = "draft" },
+ { pipe = "check_against_tide_table", result = "notice" },
+]
+
+[pipe.draft_notice]
+type = "PipeLLM"
+description = "Draft the sailing notice"
+inputs = { crossing = "Text" }
+output = "Text"
+prompt = "Draft the sailing notice for this crossing: $crossing"
diff --git a/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_pipe_dependency/entry.toml b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_pipe_dependency/entry.toml
new file mode 100644
index 000000000..1ba36051e
--- /dev/null
+++ b/pipelex/test_extras/mthds_corpus/entries/invalid_unresolved_pipe_dependency/entry.toml
@@ -0,0 +1,12 @@
+name = "invalid_unresolved_pipe_dependency"
+description = "A ferry sailing notice drafted and then checked by a second step the method never writes down"
+
+validity = "invalid"
+tier = "static"
+granularity = "focused"
+
+covers = [
+ "error.unresolved_pipe_dependency",
+]
+
+expected_error = "unresolved_pipe_dependency"
diff --git a/pipelex/test_extras/mthds_corpus/vocabulary.toml b/pipelex/test_extras/mthds_corpus/vocabulary.toml
index 3915b8b45..71bb04858 100644
--- a/pipelex/test_extras/mthds_corpus/vocabulary.toml
+++ b/pipelex/test_extras/mthds_corpus/vocabulary.toml
@@ -1,9 +1,9 @@
# The MTHDS Test Corpus tag vocabulary — the closed set an entry's `covers` draws from.
#
# GENERATED by `pipelex-dev generate-corpus-vocabulary`. Do not edit this file: `native.*`,
-# `operator.*` and `controller.*` are derived from the runtime's own registries, `feature.*` is
-# hand-maintained in the generator because no registry describes a language feature, and every
-# exclusion reason is declared in the generator's source alongside them.
+# `operator.*`, `controller.*` and `error.*` are derived from the runtime's own registries,
+# `feature.*` is hand-maintained in the generator because no registry describes a language feature,
+# and every exclusion reason is declared in the generator's source alongside them.
#
# This file ships in the pipelex wheel, so it names a document its readers can reach:
# https://docs.pipelex.com/contribute/mthds-test-corpus/ — which in turn points at the
@@ -88,6 +88,83 @@ code = "PipeParallel"
[controller.sequence]
code = "PipeSequence"
+[error.missing_input_variable]
+code = "missing_input_variable"
+
+[error.extraneous_input_variable]
+code = "extraneous_input_variable"
+
+[error.input_stuff_spec_mismatch]
+code = "input_stuff_spec_mismatch"
+
+[error.inadequate_output_concept]
+code = "inadequate_output_concept"
+
+[error.inadequate_output_multiplicity]
+code = "inadequate_output_multiplicity"
+
+[error.circular_dependency_error]
+code = "circular_dependency_error"
+excluded = "Raised only from the pipe sorter, and the pipe sorter runs only on `PipelexBundleSpec.to_blueprint()` — the builder's spec-to-blueprint conversion, which is how an AI-authored spec becomes a bundle. A `.mthds` file parses straight into a blueprint and never passes through it, so no corpus entry can produce this. Measured rather than read off the call graph: a self-referencing PipeSequence fails the dry run with `DryRunError`, and a mutually recursive PipeCondition pair exhausts the interpreter's stack — neither reports a circular dependency."
+
+[error.llm_output_cannot_be_image]
+code = "llm_output_cannot_be_image"
+
+[error.invalid_pipe_code_syntax]
+code = "invalid_pipe_code_syntax"
+
+[error.unknown_pipe_type]
+code = "unknown_pipe_type"
+
+[error.missing_pipe_type]
+code = "missing_pipe_type"
+
+[error.batch_item_name_collision]
+code = "batch_item_name_collision"
+
+[error.optional_marker_invalid]
+code = "optional_marker_invalid"
+
+[error.optional_not_handled]
+code = "optional_not_handled"
+
+[error.optional_output_required]
+code = "optional_output_required"
+
+[error.optional_input_unguarded]
+code = "optional_input_unguarded"
+
+[error.optional_branch_required_field]
+code = "optional_branch_required_field"
+
+[error.optional_force_redundant]
+code = "optional_force_redundant"
+excluded = "Advisory-only: it rides the validation report's `warnings` array and never makes a verdict invalid. An entry contract that models one `expected_error` on an invalid entry has nowhere to put a warning on a valid one, so there is no shape a focused entry could take. Revisit if the manifest ever grows a warnings axis."
+
+[error.native_concept_redeclaration]
+code = "native_concept_redeclaration"
+
+[error.unresolved_concept]
+code = "unresolved_concept"
+
+[error.unresolved_pipe_dependency]
+code = "unresolved_pipe_dependency"
+
+[error.unknown_validation_error]
+code = "unknown_validation_error"
+excluded = "The generic fallback for a validation failure that matched no specific code. Reaching it on purpose would mean authoring a bundle that breaks in a way the categorizer does not recognize — which is a state to fix in the runtime, not to pin in a corpus entry that would then have to be rewritten the moment the fault gets a code of its own."
+
+[error.unknown_concept]
+code = "unknown_concept"
+excluded = "Shadowed: it fires in the pipe factory when a pipe's output names a concept that is neither native, nor declared in the pipe's own domain, nor domain-qualified — exactly the condition the earlier concept-reference resolution already rejects, and it reports the same fault as `unresolved_concept`. Measured across unqualified, cross-domain and qualified-but-missing output refs: every one of them comes back as `unresolved_concept`, so the factory's own check is unreachable from a `.mthds` file."
+
+[error.unknown_factory_error]
+code = "unknown_factory_error"
+excluded = "The generic fallback for a pipe-construction failure that matched no specific code, and excluded for the same reason as its validation counterpart: an entry pinning it would be pinning the absence of a diagnosis rather than a language fault."
+
+[error.dry_run_error]
+code = "DryRunError"
+
[feature.multi_file_library]
description = "A method split across several `.mthds` files in one directory and merged additively behind a `bundle.mthds` entry point, so a pipe can be forward-declared as a signature in one file and satisfied by a concrete definition in another."
diff --git a/tests/e2e/pipelex/cli/test_validate_cmd.py b/tests/e2e/pipelex/cli/test_validate_cmd.py
index ec0331527..5e55ec007 100644
--- a/tests/e2e/pipelex/cli/test_validate_cmd.py
+++ b/tests/e2e/pipelex/cli/test_validate_cmd.py
@@ -1,11 +1,34 @@
+"""Smoke-test for the `validate --all` CLI path: a directory of bundles validates and dry-runs.
+
+The directory is assembled from the corpus's **valid** entries rather than pointed at a tree on
+disk. That used to be `pipelex/`, the package directory, chosen to avoid the conflicting fixtures
+under `tests/` — but the package's only `.mthds` files are the corpus's, so the test has been
+sweeping the corpus for a while without saying so, and the moment the corpus grew deliberately
+invalid entries it started failing on bundles that are *supposed* to fail.
+
+Selecting by `validity` states the intent the path expression only implied, and keeps the test
+correct as the corpus grows in either direction. The per-entry guarantee — that a valid entry
+validates and an invalid one fails with exactly its declared error — is not this test's job; it
+belongs to the corpus entry-validation gate, which is stronger. What this covers is the CLI's
+whole-directory path: many bundles merged into one library, then dry-run together.
+"""
+
+import shutil
from pathlib import Path
from pipelex.cli.commands.validate._validate_core import do_validate_all_libraries_and_dry_run # noqa: PLC2701
+from pipelex.test_extras.mthds_corpus.loader import iter_entries
+from pipelex.test_extras.mthds_corpus.manifest import EntryValidity
class TestValidateCommand:
- def test_validate_all(self):
- # Scan from the pipelex package directory, not CWD, to avoid picking up
- # test fixture .mthds files that conflict with each other (e.g. phase1
- # hierarchical domain fixtures with cross-domain refs but no METHODS.toml)
- do_validate_all_libraries_and_dry_run(library_dirs=[Path(__file__).resolve().parents[4] / "pipelex"])
+ def test_validate_all(self, tmp_path: Path):
+ copied = 0
+ for entry in iter_entries(validity=EntryValidity.VALID):
+ shutil.copytree(entry.directory, tmp_path / entry.name)
+ copied += 1
+ # An empty directory validates vacuously, so the assembly is pinned rather than assumed:
+ # a selection that silently returned nothing would leave this test green over no bundles.
+ assert copied, "The corpus yielded no valid entries, so this test would validate an empty directory"
+
+ do_validate_all_libraries_and_dry_run(library_dirs=[tmp_path])
diff --git a/tests/integration/pipelex/pipeline/test_validate_bundle_entry_shape_parity.py b/tests/integration/pipelex/pipeline/test_validate_bundle_entry_shape_parity.py
new file mode 100644
index 000000000..741d2dca2
--- /dev/null
+++ b/tests/integration/pipelex/pipeline/test_validate_bundle_entry_shape_parity.py
@@ -0,0 +1,128 @@
+"""Pin: the same bundle produces the same diagnostics whichever way it enters ``validate_bundle``.
+
+``validate_bundle`` has two entry shapes — in-memory ``mthds_contents`` (what the HTTP
+``/validate`` layer uses) and a ``mthds_file_path`` resolved against ``library_dirs`` (what the
+CLI, and therefore the MTHDS Test Corpus entry-validation gate, uses). They are meant to be the
+same check. They were not: the library-directory path ran through
+``LibraryManager._load_mthds_files_into_library``, whose two exception arms both destroyed
+structured error data on the way out, so a pipe/concept fault that the contents path reported as
+a categorized item with an ``error_type`` came back as one untyped residual.
+
+The gap was invisible because every structured-error test used the contents shape. This module
+asserts the property directly instead: for each bundle, validate it both ways and require the
+observed ``(category, error_type)`` list to match. It is the check that would have caught the
+divergence, and the one that keeps a future wrapper from re-introducing it.
+
+The bundles here are the two faults the loss actually reached — an unresolved concept reference
+(raised as a ``ConceptLibraryError`` carrying per-reference items) and an optionals violation
+(raised as a pydantic ``ValidationError`` during the merge) — plus one already-parity bundle, so
+a fix that made the two shapes agree by making *both* untyped would still fail.
+"""
+
+from collections.abc import Callable
+from pathlib import Path
+
+import pytest
+
+from pipelex.base_exceptions import ValidationErrorCategory
+from pipelex.pipeline.exceptions import ValidateBundleError
+from pipelex.pipeline.validate_bundle import validate_bundle
+from pipelex.validation_error_types import PipeValidationErrorType
+
+_UNRESOLVED_CONCEPT_MTHDS = """
+domain = "parity_unresolved_concept"
+main_pipe = "tally_crates"
+
+[pipe.tally_crates]
+type = "PipeLLM"
+description = "Count the crates listed on a manifest."
+inputs = { manifest = "Text" }
+output = "CrateTally"
+prompt = "Count the crates listed on this manifest: $manifest"
+"""
+
+_OPTIONAL_INPUT_UNGUARDED_MTHDS = """
+domain = "parity_optional_unguarded"
+main_pipe = "write_delay_notice"
+
+[pipe.write_delay_notice]
+type = "PipeLLM"
+description = "Write a delay notice that reaches for an optional platform without guarding it."
+inputs = { service = "Text", platform = "Text?" }
+output = "Text"
+prompt = "The $service is delayed. It will leave from platform $platform."
+"""
+
+_MISSING_INPUT_VARIABLE_MTHDS = """
+domain = "parity_missing_input"
+main_pipe = "write_delay_notice"
+
+[pipe.write_delay_notice]
+type = "PipeLLM"
+description = "Write a delay notice whose prompt reaches for an input it never declares."
+inputs = { service = "Text" }
+output = "Text"
+prompt = "The $service is delayed and will leave from platform $platform."
+"""
+
+
+async def _observed_from_contents(mthds_contents: str) -> list[tuple[ValidationErrorCategory, str | None]]:
+ with pytest.raises(ValidateBundleError) as raised:
+ await validate_bundle(mthds_contents=[mthds_contents])
+ return _observed(raised.value)
+
+
+async def _observed_from_library_dir(mthds_contents: str, *, directory: Path) -> list[tuple[ValidationErrorCategory, str | None]]:
+ bundle_path = directory / "bundle.mthds"
+ bundle_path.write_text(mthds_contents, encoding="utf-8")
+ with pytest.raises(ValidateBundleError) as raised:
+ await validate_bundle(mthds_file_path=bundle_path, library_dirs=[directory])
+ return _observed(raised.value)
+
+
+def _observed(error: ValidateBundleError) -> list[tuple[ValidationErrorCategory, str | None]]:
+ items = error.to_error_report().validation_errors or []
+ return [(item.category, item.error_type) for item in items]
+
+
[email protected](loop_scope="class")
+class TestValidateBundleEntryShapeParity:
+ @pytest.mark.parametrize(
+ ("mthds_contents", "expected"),
+ [
+ pytest.param(
+ _UNRESOLVED_CONCEPT_MTHDS,
+ [(ValidationErrorCategory.PIPE_VALIDATION, PipeValidationErrorType.UNRESOLVED_CONCEPT)],
+ id="unresolved_concept",
+ ),
+ pytest.param(
+ _OPTIONAL_INPUT_UNGUARDED_MTHDS,
+ [(ValidationErrorCategory.PIPE_VALIDATION, PipeValidationErrorType.OPTIONAL_INPUT_UNGUARDED)],
+ id="optional_input_unguarded",
+ ),
+ pytest.param(
+ _MISSING_INPUT_VARIABLE_MTHDS,
+ [(ValidationErrorCategory.BLUEPRINT_VALIDATION, PipeValidationErrorType.MISSING_INPUT_VARIABLE)],
+ id="missing_input_variable",
+ ),
+ ],
+ )
+ async def test_both_entry_shapes_report_the_same_categorized_errors(
+ self,
+ load_empty_library: Callable[[], str],
+ tmp_path: Path,
+ mthds_contents: str,
+ expected: list[tuple[ValidationErrorCategory, str | None]],
+ ) -> None:
+ """Both shapes report the expected categorized items — and therefore each other's.
+
+ Asserting each against the explicit expectation rather than only against one another is
+ what stops the parity from being satisfiable by both shapes degrading together.
+ """
+ load_empty_library()
+ from_contents = await _observed_from_contents(mthds_contents)
+ assert from_contents == expected, f"contents shape produced {from_contents!r}"
+
+ load_empty_library()
+ from_library_dir = await _observed_from_library_dir(mthds_contents, directory=tmp_path)
+ assert from_library_dir == expected, f"library-directory shape produced {from_library_dir!r}"
diff --git a/tests/integration/pipelex/test_extras/test_mthds_corpus_entries.py b/tests/integration/pipelex/test_extras/test_mthds_corpus_entries.py
index d2d34394c..0017d78c7 100644
--- a/tests/integration/pipelex/test_extras/test_mthds_corpus_entries.py
+++ b/tests/integration/pipelex/test_extras/test_mthds_corpus_entries.py
@@ -52,28 +52,29 @@ class TestMthdsCorpusEntries:
f"Corpus entry '{entry.name}' declares expected_error={entry.manifest.expected_error!r} but produced {observed!r}"
)
- async def test_the_invalid_arm_bites_on_a_deliberately_broken_bundle(self, tmp_path: Path) -> None:
- """No corpus entry is invalid yet, so the invalid arm is proven here instead.
-
- The `error.*` namespace an invalid entry's `covers` would draw on does not exist until
- the error-type registry lands, and an invalid entry tagged with a native it does not
- exist for would be a lie in the manifest. The machinery the arm above depends on — that
- a broken bundle raises, and that the raised error projects the exact wire `error_type`
- an `expected_error` is compared against — is exercised against a bundle built for the
- purpose, so the arm is not dead code waiting on Phase 3.
+ async def test_the_invalid_arm_bites_when_a_declared_error_stops_being_produced(self, tmp_path: Path) -> None:
+ """The arm above is red when an invalid entry stops meaning what it claims — shown, not assumed.
+
+ Every corpus entry passes that arm, which is exactly what makes its failure mode
+ unobservable in the suite: nothing in a green run demonstrates that a *wrong* declaration
+ would be caught. So the arm's comparison is run here in both directions over a bundle whose
+ single fault is known — equal to the declaration a correct entry would carry, unequal to the
+ one a drifted entry would. It also pins the machinery the arm rests on: that a broken bundle
+ raises, and that the raised error projects the exact wire `error_type` an `expected_error`
+ is compared against.
"""
bundle_path = tmp_path / "bundle.mthds"
bundle_path.write_text(
- """domain = "harbour_moorings"
-description = "A berth note whose prompt reaches for a variable the pipe never declares"
-main_pipe = "write_berth_note"
+ """domain = "harbour_moorings"
+description = "Write the note telling a vessel where and when to moor"
+main_pipe = "write_berth_note"
[pipe.write_berth_note]
-type = "PipeLLM"
-description = "Write a berth note"
-inputs = { vessel = "Text" }
-output = "Text"
-prompt = "Write a berth note for $vessel arriving on $tide."
+type = "PipeLLM"
+description = "Write the berth note for a vessel"
+inputs = { vessel = "Text" }
+output = "Text"
+prompt = "Write the berth note for $vessel arriving on the $tide tide."
""",
encoding="utf-8",
)
@@ -81,5 +82,9 @@ prompt = "Write a berth note for $vessel arriving on $tide."
with pytest.raises(ValidateBundleError) as raised:
await validate_bundle(mthds_file_path=bundle_path, library_dirs=[tmp_path])
+ declared_by_a_correct_entry = "missing_input_variable"
+ declared_by_a_drifted_entry = "unknown_pipe_type"
+
observed = [item.error_type for item in raised.value.to_error_report().validation_errors or []]
- assert observed == ["missing_input_variable"]
+ assert observed == [declared_by_a_correct_entry]
+ assert observed != [declared_by_a_drifted_entry]
diff --git a/tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py b/tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py
index 46ebebc78..c8856ef68 100644
--- a/tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py
+++ b/tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py
@@ -6,10 +6,17 @@ Contract: ``docs/specs/mthds-test-corpus.md`` (workspace root), section "Gates"
The moment a runtime registry grows a code, regeneration adds its tag and this gate stays red
until a focused entry covers it. It is why "the `Time` native concept has no fixture anywhere"
cannot happen again silently.
+
+The agreement check is the third arm and belongs here rather than with the entry-validation
+gate, because it compares two *declarations* and needs no runtime: an invalid entry states its
+fault twice in one file — once as ``expected_error``, the wire string the runtime emits, and once
+as an ``error.*`` tag in ``covers``, the normalized form the vocabulary declares. Nothing else
+checks that the two agree, so without it an entry could claim coverage of a tag it does not
+produce and the exhaustivity arm above would count that claim.
"""
from pipelex.test_extras.mthds_corpus.loader import iter_entries
-from pipelex.test_extras.mthds_corpus.manifest import EntryGranularity
+from pipelex.test_extras.mthds_corpus.manifest import EntryGranularity, EntryValidity
from pipelex.test_extras.mthds_corpus.vocabulary import load_vocabulary
@@ -29,3 +36,65 @@ class TestMthdsCorpusExhaustivity:
if tag not in declared:
unknown.add(f"{entry.name}: {tag}")
assert not unknown, f"Corpus entries cover tags that are not in the vocabulary: {', '.join(sorted(unknown))}"
+
+ def test_every_invalid_entry_covers_the_tag_its_expected_error_names(self) -> None:
+ """The two spellings of one fault must agree, and the fault is all the entry covers.
+
+ Equality rather than membership, because the arm above counts every focused entry's
+ ``covers`` regardless of validity — it has to, since the ``error.*`` tags are covered by
+ nothing else. A broken bundle that also tagged the pipe kind carrying its fault would
+ therefore satisfy that tag, and the working feature would silently lose its own fixture.
+
+ The join runs through the vocabulary's own ``code`` field rather than by re-normalizing
+ ``expected_error`` here: ``code`` records the registry spelling verbatim, so looking the
+ tag up by it keeps this gate from carrying a second copy of the normalization rule that
+ could drift from the generator's.
+ """
+ tag_by_error_code = {
+ tag.code: f"{namespace}.{local_name}"
+ for namespace, tags_in_namespace in load_vocabulary().namespaces.items()
+ if namespace == "error"
+ for local_name, tag in tags_in_namespace.items()
+ if tag.code is not None
+ }
+ disagreements: list[str] = []
+ for entry in iter_entries(validity=EntryValidity.INVALID):
+ expected_error = entry.manifest.expected_error
+ assert expected_error, f"Corpus entry '{entry.name}' is invalid, so the manifest guarantees an expected_error"
+ owed_tag = tag_by_error_code.get(expected_error)
+ if owed_tag is None:
+ disagreements.append(f"{entry.name}: expected_error '{expected_error}' is in no error.* tag's `code`")
+ elif entry.manifest.covers != [owed_tag]:
+ disagreements.append(f"{entry.name}: expected_error '{expected_error}' means `covers` must be exactly ['{owed_tag}']")
+ assert not disagreements, (
+ "Invalid entries disagree with themselves: "
+ + "; ".join(sorted(disagreements))
+ + ". An invalid entry's `covers` is its error.* tag and nothing else: the arm above reads `covers` as the "
+ "claim that a tag has a focused entry, and a deliberately broken bundle must never be what satisfies that "
+ "claim for a working feature."
+ )
+
+ def test_no_valid_entry_claims_an_error_tag(self) -> None:
+ """The other half of the same hole: a bundle that validates cannot cover a fault.
+
+ The arm above stops an invalid entry from claiming a tag it does not produce. Nothing
+ stopped a *valid* one from doing it, and the first arm counts every focused entry's
+ ``covers`` regardless of validity — so a valid entry carrying an ``error.*`` tag would
+ satisfy that tag's exhaustivity claim while its bundle, by definition, produces no error
+ at all. The fault would then have no entry exercising it and the gate would stay green.
+
+ An ``error.*`` tag is a claim about a diagnostic, and only an entry that declares an
+ ``expected_error`` can honour one.
+ """
+ error_namespace_prefix = "error."
+ offenders = [
+ f"{entry.name}: {tag}"
+ for entry in iter_entries(validity=EntryValidity.VALID)
+ for tag in entry.manifest.covers
+ if tag.startswith(error_namespace_prefix)
+ ]
+ assert not offenders, (
+ "Valid entries cover error.* tags they cannot produce: "
+ + "; ".join(sorted(offenders))
+ + ". Only an invalid entry, which declares the `expected_error` it must fail with, can cover an error.* tag."
+ )
diff --git a/wip/pr-1139-review-notes.md b/wip/pr-1139-review-notes.md
new file mode 100644
index 000000000..a224d8158
--- /dev/null
+++ b/wip/pr-1139-review-notes.md
@@ -0,0 +1,45 @@
+# PR #1139 review notes — the pytest-standards rules keep getting re-raised
+
+PR: <https://github.com/Pipelex/pipelex/pull/1139> (`feature/MthdsTestCorpus-p3`, "the `error.*` axis: an invalid-method coverage surface for the MTHDS Test Corpus")
+
+Nothing on this PR was deferred. Four of the five review threads were false positives and one was a real gap that got fixed on the branch. This note exists for the pattern behind three of those four, which is now on its third consecutive appearance and is not a question a PR-review pass should keep re-answering.
+
+## The pattern
+
+Three of the five threads on this PR — all from Codex — cited two rules from the same source file and asked for the same two changes that were asked for on #1069 and, in a sibling form, on #1138:
+
+- **`tests/integration/pipelex/pipeline/test_validate_bundle_entry_shape_parity.py:32`** — move the three inline MTHDS payloads into a `test_data.py`, citing `AGENTS.md:375`.
+- **`test_validate_bundle_entry_shape_parity.py:1`** and **`tests/e2e/pipelex/cli/test_validate_cmd.py:1`** — move the module-level docstring onto the test method, citing `AGENTS.md:438`.
+
+Both rules are real text, quoted accurately. Both trace to one source, `pipelex/kit/agent_rules/pytest_standards.md` — line 17 for the test-data rule and line 80 for the docstring rule — from which the repo-root `AGENTS.md` and `CLAUDE.md` are generated. That file is **shipped to users** as part of the package kit, which is the same dogfooding shape [`pr-1138-review-notes.md`](pr-1138-review-notes.md) recorded for the one-`TestClass`-per-module rule in the very same document.
+
+## What the tree actually does — measured on this branch
+
+**The `test_data.py` rule, as applied to MTHDS payloads.** Ninety test modules inline an MTHDS bundle literal. **Zero** of them keep it in a `test_data.py`. Not "few" — none. In the flagged file's own directory, thirty-two of the thirty-three modules inline one. Of the fifty-two `test_data.py` files in the tree, the only one holding anything MTHDS-shaped is `tests/unit/pipelex/graph/test_data.py`, and it holds GraphSpec dicts, imported by two sibling modules. That reveals the norm the tree actually follows, and it is a coherent one: **`test_data.py` is where data shared across modules goes.** The three payloads on this PR are `_`-private, each consumed exactly once, in the `pytest.param` list sixty lines below its own definition.
+
+**The docstring rule.** Six hundred and ninety-one of one thousand one hundred and thirty test modules — sixty-one percent — open with a module-level docstring, counted with `ast.get_docstring` on the module node. By directory: thirty-two of thirty-three in `tests/integration/pipelex/pipeline/`, nine of thirteen in `tests/e2e/pipelex/cli/`, four of four in `tests/unit/pipelex/test_extras/`. The other half of the same sentence fares no better: five hundred and fifty-three of one thousand two hundred and sixteen `Test*` classes carry a class docstring. Nothing enforces any of it — `pyproject.toml` selects `ALL` but ignores `D100` and `D101`, and no rule anywhere forbids a module docstring.
+
+[`pr-1069-review-notes.md`](pr-1069-review-notes.md) measured five hundred and seventy-nine of one thousand and eighteen for the same rule. The proportion has not moved; the tree has simply grown.
+
+## Why the three threads were closed rather than acted on
+
+Because acting on them would have moved this PR *away* from the code around it. The new integration module is a deliberate mirror of `tests/integration/pipelex/pipeline/test_validate_bundle_structured_errors.py`, which pre-exists on `dev` and is structurally identical — module docstring, then `_`-private inline MTHDS constants, then one parametrized test class. In `tests/e2e/pipelex/cli/`, the immediate neighbour `test_fix_bundle_cmd.py` opens with a module docstring *and* inlines a bundle literal, both patterns already shipped side by side. And `tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py` carried a module docstring before this PR touched it, naming the governing spec section — the module docstring is the established house style for this whole feature area.
+
+Content matters too, and the reviewer misread it. The rule targets "this test tests X" boilerplate migrating up to file level. Neither flagged docstring is that: one records why `_load_mthds_files_into_library` used to destroy structured error data and why a third already-parity bundle sits in the parametrize list; the other records why the e2e test now assembles its directory from corpus entries selected by `EntryValidity.VALID` instead of pointing at a path, and why the old path expression became wrong once the corpus grew deliberately invalid entries. Deleting the second one invites someone to "simplify" the `copytree` loop straight back into the bug this PR just fixed. The parity module, meanwhile, already satisfies the rule's positive half — its concise purpose is on the method.
+
+## The open question, unchanged since #1069
+
+Two coherent resolutions, and choosing between them is a judgment call about how much we want to spend on test-file shape:
+
+1. **Make the rules match the practice.** Narrow the test-data rule to data *shared across modules*, which is the norm the tree already follows and which the one real `test_data.py` in the graph package exemplifies. Soften the docstring prohibition to what it was probably reaching for — no per-test purpose docstrings migrating up to file or class level — while allowing a module docstring that explains why the module exists. Both edits go in `pipelex/kit/agent_rules/pytest_standards.md`, the generated `AGENTS.md` and `CLAUDE.md` follow, and our users stop receiving rules we do not hold ourselves to.
+2. **Keep the rules and enforce them.** That means a guard in the `check-keyword-only` mould plus a sweep across six hundred and ninety-one modules and ninety inline payloads, which is its own piece of work.
+
+What tips the balance now is the recurring cost. `AGENTS.md` is the Codex-facing file and carries both rules; `CLAUDE.md` carries neither. So every Codex review of every new test module will keep raising these same findings, and someone will keep spending a review pass measuring the tree to close them. Three PRs in, that is no longer a one-off.
+
+Nothing is broken and no gate is failing, so there is no urgency — only accumulation. See also the sibling one-`TestClass`-per-module question in [`pr-1138-review-notes.md`](pr-1138-review-notes.md), which lives in the same source document and wants deciding at the same time.
+
+## For the record — what was fixed rather than deferred
+
+The one real finding on this PR (Codex, `tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py`) was that the new agreement gate tested membership where the contract landed in the same PR says equality. It is fixed on the branch, and the mutation that proves it bites is described in the thread reply.
+
+The greptile thread on `.pipelex/plxt.toml` was a false positive, but it had one genuine residue: the invalid entries are outside `make lint` and `make format` and the contributor docs never said so. `docs/contribute/mthds-test-corpus.md` now says it.