Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Say which layer catches a fault first, and lint the entries that answer runtime (#1143)

Louis Choquelb0f27622fe82
diff --git a/.pipelex/plxt.toml b/.pipelex/plxt.toml
index 7a53ce931..2b8425c74 100644
--- a/.pipelex/plxt.toml
+++ b/.pipelex/plxt.toml
@@ -24,14 +24,24 @@ exclude = [
"node_modules/**",
".git/**",
"*.lock",
- # The MTHDS Test Corpus's deliberately invalid entries — the one .mthds population this repo does
- # not lint, because each one exists to be rejected. Their fault is sometimes structural (a typeless
- # pipe section, an unknown pipe `type`), which the schema catches as the error the entry was written
- # to trigger. What guards them instead is strictly stronger: the corpus's entry-validation gate runs
- # the real validation engine over every entry in CI and is red both when an invalid entry fails
- # differently and when it accidentally validates. Excluded rather than schema-exempted because a
- # per-rule `exclude` does not lift the schema, only narrows which rule applies.
- "**/mthds_corpus/entries/invalid_*/*.mthds",
+ # The MTHDS Test Corpus's schema-fault entries, and ONLY those. Every other deliberately invalid
+ # entry is linted like any other .mthds file in this repo: its fault needs the runtime to notice,
+ # so the schema has nothing to say about it and a clean lint is the correct outcome. The two below
+ # are the exceptions — their fault is structural, so the schema rejects them as the very error they
+ # were written to trigger, and linting them would report a defect that is the entry's whole point.
+ #
+ # This list is not hand-maintained: it is gated against the corpus vocabulary's `fails_at` signal by
+ # tests/unit/pipelex/test_extras/test_mthds_corpus_plxt_exclusions.py, which fails if the two
+ # disagree. That is also what makes pipelex the signal's first consumer, and it closes the loop in
+ # the direction that matters — an error type mis-declared `runtime` leaves its entry linted, and
+ # `make plxt-lint` goes red naming it.
+ #
+ # Excluded rather than schema-exempted because a per-rule `exclude` does not lift the schema, only
+ # narrows which rule applies. What guards these two instead is strictly stronger: the corpus's
+ # entry-validation gate runs the real validation engine over every entry in CI and is red both when
+ # an invalid entry fails differently and when it accidentally validates.
+ "**/mthds_corpus/entries/invalid_missing_pipe_type/*.mthds",
+ "**/mthds_corpus/entries/invalid_unknown_pipe_type/*.mthds",
] # Glob patterns for files to ignore.
# These are evaluated relative to the config file location.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 77aa3662f..c192ddd2c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,25 @@
# Changelog
+## [Unreleased]
+
+### Added
+
+- **Every `error.*` tag in the MTHDS Test Corpus vocabulary now says which layer of checking catches its fault first.** A non-excluded tag carries `fails_at = "schema"` when a pass over the raw document's shape already rejects a bundle carrying the fault — a pipe section with no `type` key, a `type` outside the closed set of pipe kinds — and `fails_at = "runtime"` when the document has to be interpreted to notice, which covers every other fault the corpus exercises. The consumer rule is one sentence: **a structural sweep expects a diagnostic on an entry exactly when its tag says `schema`, and silence on every other entry.**
+
+ This closes a hole that had every consumer re-deriving the runtime's own knowledge. A JSON-Schema sweep, an editor diagnostic, or a linter run over the corpus previously had to hardcode which faults it believed were structural — a second, downstream reading of this repo's validation-error registry, guaranteed to drift from it. The corpus being swept now carries the answer, and it ships in the wheel, so a consumer that vendors `vocabulary.toml` gets it with no code of its own.
+
+ The signal lives on the vocabulary tag rather than on each `entry.toml`, because it is a property of the *error type*, not of the entry: per-entry restatement would let two entries covering one fault disagree, with nothing to catch it. A `schema` fault is still rejected by the runtime — the field names where a fault is caught first, not who may catch it — which is why a schema-fault entry declares the same `expected_error` as any other invalid entry. The value `schema` is also `plxt`'s own `error[schema]` diagnostic category, so a consumer branching on the field and a human reading a linter's output use one word for one thing.
+
+- **The exhaustivity gate now requires every `error.*` tag that is owed an entry to declare a `fails_at`**, so a new validation error type cannot land carrying a fault a structural consumer has no way to sweep for. Excluded tags carry no value, deliberately: an excluded tag has no entry, so there is nothing to have measured on.
+
+### Changed
+
+- **The corpus's deliberately invalid entries are linted again — all but the two whose fault the schema is supposed to reject.** `.pipelex/plxt.toml` excluded the whole `invalid_*` population behind one blanket glob, because some of those entries carry a structural fault and the config had no way to say which. `fails_at` is that way, so the exclusion now names exactly the schema-fault entries and every other invalid bundle is linted like any ordinary `.mthds` file in the tree.
+
+ That also makes this repo the first consumer of its own signal, and it is what keeps the signal *measured* rather than merely asserted: the exclusion list is gated against the vocabulary by `test_mthds_corpus_plxt_exclusions.py`, so declaring a fault `runtime` when the schema in fact rejects it leaves its entry linted and turns `make plxt-lint` red naming it. The config is gated rather than generated because `plxt` is a static binary reading a static file, with nothing to hook a generation step onto.
+
+- **Documentation**: `docs/contribute/mthds-test-corpus.md` gains a `fails_at` section covering the consumer rule, why the field sits on the tag, and how the linter config closes the measurement loop. Its invalid-entry authoring warning is corrected to match the narrowed exclusion: only a schema-fault bundle has to be laid out by hand now, since `make lint` and `make format` cover every other invalid entry.
+
## [v0.50.0] - 2026-08-20
### Changed
diff --git a/docs/contribute/mthds-test-corpus.md b/docs/contribute/mthds-test-corpus.md
index 337d8bc15..bb55c04a3 100644
--- a/docs/contribute/mthds-test-corpus.md
+++ b/docs/contribute/mthds-test-corpus.md
@@ -70,7 +70,7 @@ Two rules shape an invalid entry beyond that. Its **`covers` list is its `error.
The `pipelex` plugin's `PostToolUse` hook lints, formats and **blocks on an invalid verdict** for every `Write`/`Edit` of a `.mthds` file, and has no skip mechanism. Author valid entries with the editing tools to collect the free lint; write an invalid entry's bundle through a shell heredoc, which the hook's matcher does not intercept.
- An invalid entry's bundle is outside `make lint` and `make format` as well — `.pipelex/plxt.toml` excludes the whole `invalid_*` population, for the reason recorded beside that glob. So nothing formats it for you: match the layout of the entry next door by hand. Its `entry.toml` is still linted normally.
+ `make lint` and `make format` cover an invalid entry's bundle like any other `.mthds` file in the tree, with one carve-out: `.pipelex/plxt.toml` excludes the corpus's **schema-fault** entries — those whose tag says `fails_at = "schema"` — because a linter reporting them would be reporting the very defect the entry exists to carry, and that list is gated against the vocabulary rather than hand-maintained (see [`fails_at`](#fails_at-which-layer-catches-a-fault-first)). Nothing formats those, so match the layout of the entry next door by hand; every other invalid bundle is formatted for you. Its `entry.toml` is linted normally either way.
## The tag vocabulary
@@ -94,12 +94,25 @@ No native concept is excluded — every one of them turned out to support a real
**Measure an exclusion, do not reason your way to one.** Every `error.*` exclusion reason above was established by writing the bundle that ought to trigger it and reading what the runtime actually said — which is how the first two turned out to be shadowed rather than merely awkward, and how the two the registry made look unreachable (`missing_pipe_type`, `native_concept_redeclaration`) turned out to have entries after all.
+### `fails_at` — which layer catches a fault first
+
+An `error.*` tag that is *not* excluded carries one more field: `fails_at`, either `schema` or `runtime`. It names the earliest layer of checking that rejects a bundle carrying the fault. `schema` means a pass over the raw document's shape already catches it — a section missing a required key, a value outside a closed set — so a JSON-Schema validator, an editor diagnostic, or `plxt lint` reports it without ever interpreting the document. `runtime` means the document has to be *interpreted* to notice: an unresolved reference, an input the flow never supplies, an output whose concept does not fit.
+
+**The consumer rule is one sentence: a structural sweep expects a diagnostic on an entry exactly when its tag says `fails_at = "schema"`, and expects silence on every other entry.** That is what the field exists for. Before it, a consumer running a schema checker over the corpus had to hardcode which faults it thought were structural — a second, downstream reading of `pipelex`'s own registry, guaranteed to drift from it. Now the corpus being swept carries the answer.
+
+A `schema` fault is still rejected by the runtime; the field names where a fault is caught *first*, not who is allowed to catch it. That is why a schema-fault entry declares the same `expected_error` as any other invalid entry — the runtime's diagnostic is what the entry pins. The two spellings line up on purpose, too: `schema` is also `plxt`'s own `error[schema]` diagnostic category, so a consumer branching on the field and a human reading a linter's output use one word for one thing.
+
+Only non-excluded tags carry it, for the same reason exclusion reasons are measured: an excluded tag has no entry, so there is nothing to have measured on, and a value invented for one would be an argument dressed as a measurement.
+
+**And it stays measured, because this repo consumes it first.** `.pipelex/plxt.toml` excludes the corpus's schema-fault entries from `plxt lint` — they *should* produce a diagnostic, which is the entry's whole point — and lints every other invalid entry like any ordinary `.mthds` file. That exclusion list is not hand-maintained: `tests/unit/pipelex/test_extras/test_mthds_corpus_plxt_exclusions.py` fails when it disagrees with the vocabulary. The config is gated rather than generated because `plxt` is a static binary reading a static file, with nothing to hook a generation step onto — but the loop still closes in the direction that matters. Declare a fault `runtime` when the schema in fact rejects it, and its entry stays linted, so `make plxt-lint` goes red naming it.
+
## The gates, and what a red one means
| Gate | Where | Red means |
|---|---|---|
| Vocabulary drift | `tests/unit/pipelex/test_extras/test_mthds_corpus_vocabulary.py` | A registry changed and the committed vocabulary was not regenerated. Run `make generate-corpus-vocabulary`. |
-| Exhaustivity | `tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py` | A vocabulary tag has no `focused` entry covering it — write the entry, or record an exclusion with a reason. Also fires when an entry covers a tag that is not in the vocabulary, when an invalid entry's `covers` is not exactly the tag its `expected_error` names, and when a valid entry covers an `error.*` tag it cannot produce. |
+| Exhaustivity | `tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py` | A vocabulary tag has no `focused` entry covering it — write the entry, or record an exclusion with a reason. Also fires when an entry covers a tag that is not in the vocabulary, when an invalid entry's `covers` is not exactly the tag its `expected_error` names, when a valid entry covers an `error.*` tag it cannot produce, and when a required `error.*` tag declares no `fails_at`. |
+| Linter exclusions | `tests/unit/pipelex/test_extras/test_mthds_corpus_plxt_exclusions.py` | `.pipelex/plxt.toml`'s corpus exclusions disagree with the vocabulary's `fails_at` signal — an entry whose fault the schema rejects must be excluded, every other invalid entry must be linted. |
| Manifest | `tests/unit/pipelex/test_extras/test_mthds_corpus_manifest.py` | The strict `entry.toml` model rejected something. |
| Layout and filters | `tests/unit/pipelex/test_extras/test_mthds_corpus_loader.py` | An entry's name does not match its directory, its bundle does not resolve, or the loader's filter semantics changed. |
| Entry validation | `tests/integration/pipelex/test_extras/test_mthds_corpus_entries.py` | A valid entry stopped validating, or an invalid one stopped failing with exactly its declared error. Runs over every entry regardless of tier, so an `inference`-tier entry that broke is caught without spending a token. |
diff --git a/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py b/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py
index c71b56787..915688767 100644
--- a/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py
+++ b/pipelex/cli/dev_cli/commands/generate_corpus_vocabulary_cmd.py
@@ -20,7 +20,7 @@ from rich.panel import Panel
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.pipe_machinery.pipe_blueprint import PipeCategory, PipeType
from pipelex.runtime_hub import get_console
-from pipelex.test_extras.mthds_corpus.vocabulary import vocabulary_path
+from pipelex.test_extras.mthds_corpus.vocabulary import ValidationLayer, vocabulary_path
from pipelex.validation_error_types import VALIDATION_ERROR_TYPES, PipeFactoryErrorType, PipeValidationErrorType, ValidationErrorType
_ACRONYM_BOUNDARY_RE = re.compile(r"(?<=[A-Z0-9])(?=[A-Z][a-z])")
@@ -33,6 +33,11 @@ _HEADER = """# The MTHDS Test Corpus tag vocabulary — the closed set an entry'
# `feature.*` is hand-maintained in the generator because no registry describes a language feature,
# and every exclusion reason is declared in the generator's source alongside them.
#
+# An `error.*` tag that is not excluded also carries `fails_at`, the earliest layer of checking that
+# rejects a bundle carrying the fault: `schema` when a check of the document's shape alone already
+# catches it, `runtime` when the document has to be interpreted to notice. A structural consumer
+# expects a diagnostic on exactly the `schema` entries and none on the others.
+#
# This file ships in the pipelex wheel, so it names a document its readers can reach:
# https://docs.pipelex.com/contribute/mthds-test-corpus/ — which in turn points at the
# cross-repo contract, `docs/specs/mthds-test-corpus.md` in the Pipelex workspace repo.
@@ -156,6 +161,41 @@ _ERROR_TYPE_EXCLUSIONS: dict[ValidationErrorType, str] = {
}
+# `fails_at` says, for a fault a corpus entry can produce, which layer of checking rejects it FIRST.
+# The two layers are the two ways a `.mthds` document gets read: a JSON-Schema pass over the raw
+# document, which sees only shape, and the runtime, which interprets what the shape means. A schema
+# check catches a section missing a required key or holding a value outside a closed set; everything
+# else — an unresolved reference, an input the flow never supplies, an output whose concept does not
+# fit — needs interpretation and belongs to the runtime.
+#
+# The set below is the `schema` half, and it is short by nature: only a fault visible in the shape
+# alone can land in it. Every other non-excluded error type is `runtime`, so this set is the whole
+# declaration and there is no second list to keep in step with it. An excluded type gets no
+# `fails_at` at all — it has no entry, so there is nothing to have measured, and a value invented for
+# one would be an argument dressed as a measurement.
+#
+# **This is measured, not argued, and CI is what keeps it measured.** The list was established by
+# linting every invalid entry against the generated MTHDS JSON Schema: exactly the members below
+# produce a diagnostic, every other invalid entry passes. That measurement is then held in place by
+# `.pipelex/plxt.toml`, whose corpus exclusions are gated against this set — so the invalid entries
+# NOT listed here are linted on every `make plxt-lint` run. Declare a fault `runtime` when the schema
+# in fact rejects it and the lint goes red naming the entry; the loop closes without anyone
+# remembering to re-measure.
+#
+# Keyed on the enums for the same reason the exclusion maps are: a code that leaves the registry
+# breaks this module at import instead of leaving a stale value behind in the generated file.
+_ERROR_TYPE_SCHEMA_FAULTS: frozenset[ValidationErrorType] = frozenset(
+ {
+ # A `[pipe.x]` section carrying implementation fields but no `type` key, which the schema
+ # requires on any pipe that is not a bare signature.
+ PipeValidationErrorType.MISSING_PIPE_TYPE,
+ # A `type` naming a pipe kind that does not exist, which the schema's closed enum of pipe
+ # kinds rejects on sight.
+ PipeValidationErrorType.UNKNOWN_PIPE_TYPE,
+ }
+)
+
+
def normalize_registry_code(*, code: str) -> str:
"""Turn a CamelCase registry code into a snake_case tag local name.
@@ -192,12 +232,30 @@ def _render_feature_namespace() -> str:
def _render_error_namespace() -> str:
+ """The `error.*` namespace: every registry code, each either excluded or carrying a `fails_at`.
+
+ The two are mutually exclusive by construction, and a type in both maps is a contradiction rather
+ than a precedence question — an exclusion says no entry exists to measure, while a `fails_at`
+ claims a measurement was taken on one. Raising routes it through the command's own failure path
+ instead of quietly emitting whichever branch happens to come first.
+ """
+ contradictions = sorted(error_type.value for error_type in _ERROR_TYPE_SCHEMA_FAULTS if error_type in _ERROR_TYPE_EXCLUSIONS)
+ if contradictions:
+ msg = (
+ f"Error types are declared both excluded and schema faults: {', '.join(contradictions)}. "
+ "An excluded type has no corpus entry, so no schema measurement exists for it."
+ )
+ raise ValueError(msg)
+
blocks: list[str] = []
for error_type in VALIDATION_ERROR_TYPES:
lines = [f"[error.{normalize_registry_code(code=error_type.value)}]", f"code = {_toml_string(value=error_type.value)}"]
exclusion_reason = _ERROR_TYPE_EXCLUSIONS.get(error_type)
if exclusion_reason is not None:
lines.append(f"excluded = {_toml_string(value=exclusion_reason)}")
+ else:
+ fails_at = ValidationLayer.SCHEMA if error_type in _ERROR_TYPE_SCHEMA_FAULTS else ValidationLayer.RUNTIME
+ lines.append(f"fails_at = {_toml_string(value=fails_at)}")
blocks.append("\n".join(lines))
return "\n\n".join(blocks)
diff --git a/pipelex/test_extras/mthds_corpus/vocabulary.py b/pipelex/test_extras/mthds_corpus/vocabulary.py
index 2800ae191..e17e5a75a 100644
--- a/pipelex/test_extras/mthds_corpus/vocabulary.py
+++ b/pipelex/test_extras/mthds_corpus/vocabulary.py
@@ -6,6 +6,7 @@ committed, so a consumer reading it needs neither the generator nor a registry w
module is the reader; the generator lives in the dev CLI, which the wheel excludes.
"""
+from enum import StrEnum
from pathlib import Path
from pydantic import BaseModel, ConfigDict
@@ -15,6 +16,38 @@ from pipelex.tools.misc.toml_utils import load_toml_from_path
VOCABULARY_FILE_NAME = "vocabulary.toml"
+ERROR_NAMESPACE = "error"
+
+
+class ValidationLayer(StrEnum):
+ """The layer of checking a fault is caught at — the value of a tag's ``fails_at``.
+
+ The two layers are ordered, and ``fails_at`` names the **earliest** one that rejects a bundle
+ carrying the fault. ``SCHEMA`` is what a JSON-Schema check over the raw document already
+ catches, because the fault is structural: a section that is missing a required key, or one
+ whose value is outside a closed set. ``RUNTIME`` is everything a document has to be
+ *interpreted* to notice — an unresolved reference, an input the flow never provides, an
+ output whose concept does not fit.
+
+ ``SCHEMA`` faults are rejected by the runtime too. The layers name where a fault is caught
+ first, not who is allowed to catch it, which is why a schema-fault entry still declares the
+ ``expected_error`` the runtime reports for it. The spelling matches plxt's own
+ ``error[schema]`` diagnostic category, so a consumer branching on this field and a human
+ reading a linter's output are using one word for one thing.
+ """
+
+ SCHEMA = "schema"
+ RUNTIME = "runtime"
+
+ @property
+ def is_schema(self) -> bool:
+ """True for the layer a check of the document's shape alone already rejects."""
+ match self:
+ case ValidationLayer.SCHEMA:
+ return True
+ case ValidationLayer.RUNTIME:
+ return False
+
class VocabularyTag(BaseModel):
"""One tag in the vocabulary.
@@ -26,11 +59,19 @@ class VocabularyTag(BaseModel):
can live. ``excluded`` carries the reason a tag is not required to have a focused entry,
and is present only on excluded tags.
- All three fields are optional here, and no validator enforces the ``code``-or-``description``
- split, deliberately. This model only *reads*: the generator is the single writer, and the
- drift gate regenerates the committed file in memory and compares it byte for byte, so a tag
- carrying neither cannot reach a reader through the channel this model serves. A validator
- would guard a state the pipeline cannot produce.
+ ``fails_at`` is the ``error.*`` namespace's own field, and it is the answer to a question
+ every structural consumer was otherwise guessing at: does a check that only reads the
+ document's *shape* reject this fault, or does it take the runtime to notice? A consumer
+ sweeping the corpus with a schema validator expects a diagnostic on exactly the entries
+ whose tag says :attr:`ValidationLayer.SCHEMA`, and expects silence on the rest — so the
+ signal turns a sweep that had to hardcode a list of known-structural faults into one that
+ reads the corpus it is sweeping.
+
+ Every field is optional here, and no validator enforces which combinations are legal,
+ deliberately. This model only *reads*: the generator is the single writer, and the drift
+ gate regenerates the committed file in memory and compares it byte for byte, so a tag
+ carrying a nonsensical combination cannot reach a reader through the channel this model
+ serves. A validator would guard a state the pipeline cannot produce.
"""
model_config = ConfigDict(extra="forbid", frozen=True)
@@ -38,6 +79,7 @@ class VocabularyTag(BaseModel):
code: str | None = None
description: str | None = None
excluded: str | None = None
+ fails_at: ValidationLayer | None = None
class CorpusVocabulary(BaseModel):
@@ -55,6 +97,23 @@ class CorpusVocabulary(BaseModel):
declared.update(f"{namespace}.{local_name}" for local_name in tags_in_namespace)
return frozenset(declared)
+ @property
+ def schema_fault_tags(self) -> frozenset[str]:
+ """The tags whose fault a check of the document's shape alone already rejects.
+
+ This is the consumer rule in one call: a structural sweep — a JSON-Schema pass, an editor
+ diagnostic, a linter run over the corpus — expects a diagnostic on an entry covering a tag
+ in this set, and expects none on any other entry. Reading it off the shipped vocabulary is
+ what keeps a consumer from re-deriving the runtime's own knowledge downstream, whether by
+ hardcoding a list of structural faults or by pattern-matching tag names.
+ """
+ return frozenset(
+ f"{namespace}.{local_name}"
+ for namespace, tags_in_namespace in self.namespaces.items()
+ for local_name, tag in tags_in_namespace.items()
+ if tag.fails_at is not None and tag.fails_at.is_schema
+ )
+
@property
def required_tags(self) -> frozenset[str]:
"""The tags the exhaustivity gate requires a focused entry for — every tag that is not excluded."""
diff --git a/pipelex/test_extras/mthds_corpus/vocabulary.toml b/pipelex/test_extras/mthds_corpus/vocabulary.toml
index 71bb04858..89ac417b6 100644
--- a/pipelex/test_extras/mthds_corpus/vocabulary.toml
+++ b/pipelex/test_extras/mthds_corpus/vocabulary.toml
@@ -5,6 +5,11 @@
# `feature.*` is hand-maintained in the generator because no registry describes a language feature,
# and every exclusion reason is declared in the generator's source alongside them.
#
+# An `error.*` tag that is not excluded also carries `fails_at`, the earliest layer of checking that
+# rejects a bundle carrying the fault: `schema` when a check of the document's shape alone already
+# catches it, `runtime` when the document has to be interpreted to notice. A structural consumer
+# expects a diagnostic on exactly the `schema` entries and none on the others.
+#
# This file ships in the pipelex wheel, so it names a document its readers can reach:
# https://docs.pipelex.com/contribute/mthds-test-corpus/ — which in turn points at the
# cross-repo contract, `docs/specs/mthds-test-corpus.md` in the Pipelex workspace repo.
@@ -90,18 +95,23 @@ code = "PipeSequence"
[error.missing_input_variable]
code = "missing_input_variable"
+fails_at = "runtime"
[error.extraneous_input_variable]
code = "extraneous_input_variable"
+fails_at = "runtime"
[error.input_stuff_spec_mismatch]
code = "input_stuff_spec_mismatch"
+fails_at = "runtime"
[error.inadequate_output_concept]
code = "inadequate_output_concept"
+fails_at = "runtime"
[error.inadequate_output_multiplicity]
code = "inadequate_output_multiplicity"
+fails_at = "runtime"
[error.circular_dependency_error]
code = "circular_dependency_error"
@@ -109,33 +119,43 @@ excluded = "Raised only from the pipe sorter, and the pipe sorter runs only on `
[error.llm_output_cannot_be_image]
code = "llm_output_cannot_be_image"
+fails_at = "runtime"
[error.invalid_pipe_code_syntax]
code = "invalid_pipe_code_syntax"
+fails_at = "runtime"
[error.unknown_pipe_type]
code = "unknown_pipe_type"
+fails_at = "schema"
[error.missing_pipe_type]
code = "missing_pipe_type"
+fails_at = "schema"
[error.batch_item_name_collision]
code = "batch_item_name_collision"
+fails_at = "runtime"
[error.optional_marker_invalid]
code = "optional_marker_invalid"
+fails_at = "runtime"
[error.optional_not_handled]
code = "optional_not_handled"
+fails_at = "runtime"
[error.optional_output_required]
code = "optional_output_required"
+fails_at = "runtime"
[error.optional_input_unguarded]
code = "optional_input_unguarded"
+fails_at = "runtime"
[error.optional_branch_required_field]
code = "optional_branch_required_field"
+fails_at = "runtime"
[error.optional_force_redundant]
code = "optional_force_redundant"
@@ -143,12 +163,15 @@ excluded = "Advisory-only: it rides the validation report's `warnings` array and
[error.native_concept_redeclaration]
code = "native_concept_redeclaration"
+fails_at = "runtime"
[error.unresolved_concept]
code = "unresolved_concept"
+fails_at = "runtime"
[error.unresolved_pipe_dependency]
code = "unresolved_pipe_dependency"
+fails_at = "runtime"
[error.unknown_validation_error]
code = "unknown_validation_error"
@@ -164,6 +187,7 @@ excluded = "The generic fallback for a pipe-construction failure that matched no
[error.dry_run_error]
code = "DryRunError"
+fails_at = "runtime"
[feature.multi_file_library]
description = "A method split across several `.mthds` files in one directory and merged additively behind a `bundle.mthds` entry point, so a pipe can be forward-declared as a signature in one file and satisfied by a concrete definition in another."
diff --git a/tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py b/tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py
index c8856ef68..17341093c 100644
--- a/tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py
+++ b/tests/unit/pipelex/test_extras/test_mthds_corpus_exhaustivity.py
@@ -13,11 +13,18 @@ fault twice in one file — once as ``expected_error``, the wire string the runt
as an ``error.*`` tag in ``covers``, the normalized form the vocabulary declares. Nothing else
checks that the two agree, so without it an entry could claim coverage of a tag it does not
produce and the exhaustivity arm above would count that claim.
+
+The ``fails_at`` arm extends exhaustivity from tags to what a tag has to *say*. A consumer sweeping the
+corpus with a structural checker branches on ``fails_at``, so a tag that is owed an entry and
+declares no layer leaves that consumer with nothing to branch on — the same hole as a tag with no
+entry, one level down. Keeping it here rather than with the generator's own drift gate is
+deliberate: this reads the loaded vocabulary, which is what a consumer sees, so the requirement
+survives a change in how the file comes to be written.
"""
from pipelex.test_extras.mthds_corpus.loader import iter_entries
from pipelex.test_extras.mthds_corpus.manifest import EntryGranularity, EntryValidity
-from pipelex.test_extras.mthds_corpus.vocabulary import load_vocabulary
+from pipelex.test_extras.mthds_corpus.vocabulary import ERROR_NAMESPACE, load_vocabulary
class TestMthdsCorpusExhaustivity:
@@ -53,7 +60,7 @@ class TestMthdsCorpusExhaustivity:
tag_by_error_code = {
tag.code: f"{namespace}.{local_name}"
for namespace, tags_in_namespace in load_vocabulary().namespaces.items()
- if namespace == "error"
+ if namespace == ERROR_NAMESPACE
for local_name, tag in tags_in_namespace.items()
if tag.code is not None
}
@@ -74,6 +81,28 @@ class TestMthdsCorpusExhaustivity:
"claim for a working feature."
)
+ def test_every_required_error_tag_declares_the_layer_it_fails_at(self) -> None:
+ """A fault a consumer is owed an entry for must also say which layer catches it first.
+
+ Required rather than universal: an excluded ``error.*`` tag has no entry, so no measurement
+ of it exists and inventing one would be an argument dressed as a measurement. The other
+ namespaces name language features, not faults, and have no layer to declare at all.
+ """
+ vocabulary = load_vocabulary()
+ error_prefix = f"{ERROR_NAMESPACE}."
+ silent = sorted(
+ f"{ERROR_NAMESPACE}.{local_name}"
+ for local_name, tag in vocabulary.namespaces[ERROR_NAMESPACE].items()
+ if tag.fails_at is None and f"{error_prefix}{local_name}" in vocabulary.required_tags
+ )
+ assert not silent, (
+ "Required error.* tags declare no `fails_at`: "
+ + ", ".join(silent)
+ + ". A structural consumer expects a diagnostic on exactly the `schema` faults and none on the "
+ "`runtime` ones, so a fault that names no layer is one it cannot sweep for. Declare it in the "
+ "vocabulary generator's schema-fault set and regenerate with `make generate-corpus-vocabulary`."
+ )
+
def test_no_valid_entry_claims_an_error_tag(self) -> None:
"""The other half of the same hole: a bundle that validates cannot cover a fault.
@@ -86,7 +115,7 @@ class TestMthdsCorpusExhaustivity:
An ``error.*`` tag is a claim about a diagnostic, and only an entry that declares an
``expected_error`` can honour one.
"""
- error_namespace_prefix = "error."
+ error_namespace_prefix = f"{ERROR_NAMESPACE}."
offenders = [
f"{entry.name}: {tag}"
for entry in iter_entries(validity=EntryValidity.VALID)
diff --git a/tests/unit/pipelex/test_extras/test_mthds_corpus_plxt_exclusions.py b/tests/unit/pipelex/test_extras/test_mthds_corpus_plxt_exclusions.py
new file mode 100644
index 000000000..027cee3c1
--- /dev/null
+++ b/tests/unit/pipelex/test_extras/test_mthds_corpus_plxt_exclusions.py
@@ -0,0 +1,57 @@
+"""This repo's own linter config, gated against the corpus's ``fails_at`` signal.
+
+Contract: ``docs/specs/mthds-test-corpus.md`` (workspace root), section "Tag vocabulary".
+
+`pipelex` is the first consumer of its own signal, and `.pipelex/plxt.toml` is where it consumes
+it. Every deliberately invalid corpus entry used to be excluded from `plxt lint` by one blanket
+glob, because *some* of them carry a structural fault the schema rejects and the config had no way
+to say which. `fails_at` is that way: the entries whose tag says `schema` stay excluded, and every
+other invalid entry is linted like any ordinary `.mthds` file in the tree.
+
+The config is gated rather than generated because `plxt` is a static binary reading a static file —
+there is nothing to hook a generation step onto. What this test buys is the same guarantee in the
+direction that matters: a fault mis-declared `runtime` leaves its entry linted, and `make plxt-lint`
+goes red naming it, which is what keeps the signal measured instead of merely asserted.
+"""
+
+from pathlib import Path
+from typing import Any
+
+from pipelex.test_extras.mthds_corpus.loader import iter_entries
+from pipelex.test_extras.mthds_corpus.manifest import EntryValidity
+from pipelex.test_extras.mthds_corpus.vocabulary import load_vocabulary
+from pipelex.tools.misc.toml_utils import load_toml_from_path
+
+_REPO_ROOT = Path(__file__).resolve().parents[4]
+_PLXT_CONFIG_PATH = _REPO_ROOT / ".pipelex" / "plxt.toml"
+
+_CORPUS_GLOB_MARKER = "mthds_corpus"
+
+
+def _corpus_exclusion_globs() -> set[str]:
+ """The corpus-related entries of the config's top-level `exclude` list.
+
+ Narrowed by the marker rather than compared whole: the rest of that list is ordinary build-output
+ noise (`.venv/**`, `target/**`) that has nothing to do with the corpus and should not have to be
+ restated here to keep this gate green.
+ """
+ config: dict[str, Any] = load_toml_from_path(_PLXT_CONFIG_PATH)
+ excluded: list[str] = config["exclude"]
+ return {glob for glob in excluded if _CORPUS_GLOB_MARKER in glob}
+
+
+class TestMthdsCorpusPlxtExclusions:
+ def test_the_excluded_entries_are_exactly_the_schema_fault_entries(self) -> None:
+ schema_fault_tags = load_vocabulary().schema_fault_tags
+ assert schema_fault_tags, "The vocabulary declares no schema fault, so this gate would pass vacuously"
+
+ expected = {
+ f"**/mthds_corpus/entries/{entry.name}/*.mthds"
+ for entry in iter_entries(validity=EntryValidity.INVALID)
+ if not schema_fault_tags.isdisjoint(entry.manifest.covers)
+ }
+ assert _corpus_exclusion_globs() == expected, (
+ "`.pipelex/plxt.toml`'s corpus exclusions disagree with the vocabulary's `fails_at` signal. "
+ "An entry whose tag says `schema` must be excluded — the schema rejects it as the very error "
+ "it exists to trigger — and every other invalid entry must be linted like any other .mthds file."
+ )