Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature: generate the shared projection fixture corpus (#1168)
Louis Choquel12d78aef34b2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c226d8e35..2f3756476 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,8 +4,13 @@### Added+- **`pipelex-dev generate-projection-corpus`**: The generator of the shared fixture corpus that pins the two client-side inputs-template projections — the TypeScript one in `mthds` and its Python twin — against each other. Each client SDK now renders a pipe's fill-in template from the input-form descriptor it already has instead of asking a server for one, so the two renderings have to agree exactly, TOML `# concept:` comment lines included. The command pairs each pipe's descriptor with the expected template in both shapes (compact and explicit) and both formats (JSON and TOML), and records where that expectation deliberately differs from this engine's own inputs-template renderer: the expectation is authored from a reference projection that walks the descriptor, where the engine reflects the runtime content classes. Each class of difference is declared rather than discovered — an undeclared one fails the run, and so does a declared one that no longer occurs, so an engine fix retires its entry deliberately. A new corpus bundle covers what the existing probe bundles did not: a text field merely named `url` beside a real file position, optional file fields nested inside a structure, and both a fixed `[N]` and a variable `[]` slot over a structured concept. See [Projection Fixture Corpus](contribute/generate-projection-corpus.md).- **Include a method by address — fetch-on-miss**: a bundle referencing another method by address (`github.com/...->domain.pipe`) that misses the installed methods now fetches the package by reference — honoring an `@<tag>` pin, through the same grammar, bounds, and tags-only rules as a direct fetch, with the same hosted-parity warning for Python structure classes locally and a hard refusal on sandbox-hosted deployments — installs it into `~/.mthds/methods/` (atomic staging write, `.git/` stripped, fetch provenance recorded beside the manifest in a `.provenance.json` sidecar: address, tag, and the resolved commit SHA), and loads it so resolution proceeds. Once installed, the method is reused and the network is never touched again — including for a reference pinning a different tag, which warns and uses the installed copy. The former silent-miss warning in the library manager is now a real diagnostic: a miss that cannot be bridged (fetch disabled via the new `[interpreter.methods] fetch_on_miss` config or the `PIPELEX_METHODS_FETCH_ON_MISS` environment variable, an unfetchable address, or a failed fetch) raises an error naming the address and the remedy, never a silent pass. The install target is guarded by identity: two packages sharing a bare directory name but coming from different addresses collide with a loud error naming both addresses (never a silent load of the wrong package, never an overwrite), a concurrent install of the same package is tolerated (each attempt stages under a unique dot-prefixed directory, and losing the final rename race to a matching occupant uses the winner's copy), and a fetched package containing a symlink is refused outright, since a link's target would bypass the content scans. Installed-method lookup by full address is now case-insensitive, matching GitHub's owner/repository semantics and the fetch path's manifest-identity matching, and discovery skips dot-prefixed directories. See [Packages → Include by Address](building-methods/packages.md) and [Methods Configuration](configuration/config-practical/methods-config.md).+### Changed++- **`kajson` version**: The pin moves to `kajson==0.7.1`, the release making `ClassRegistry.register_classes_dict({})` a no-op instead of raising `IndexError` — its logging branch for the single-class case indexed into an empty collection. This engine seeds a fresh per-run or per-library registry from a snapshot of the process-global one, so a snapshot that happens to be empty used to crash the seeding rather than register nothing; both seeding sites (distributed-run rehydration and the scoped-registry test fixture) are now safe unconditionally. The release changes nothing in the encoder or decoder, so the serialized wire format is untouched in both directions. Everyone downstream inherits that exact version.+### Fixed- **`--help` works again on every `method` subcommand**: `pipelex validate method`, `pipelex run method`, the three `pipelex build ... method` commands and their agent-CLI twins all crashed with a rich `MarkupError` traceback instead of printing their help. The method-address example in each command's `NAME` argument help — `github.com/owner/repo[/name][@tag]` — reaches rich's markup parser through typer, which read `[/name]` as a closing tag with no opening one. The brackets are escaped at each of the sites, so the example renders as written and the help exits zero.diff --git a/CLAUDE.md b/CLAUDE.mdindex a41fd932b..9adab9c88 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -151,6 +151,12 @@.venv/bin/pipelex-dev trace-input-semantics tests/data/input_semantics/probe_bundle.mthds -o /tmp/probe-trace```+ - **`generate-projection-corpus`**: Write the shared inputs-template projection fixture corpus — the descriptors, the expected fill-in templates in both shapes and both formats, and the record of where the expectation deliberately differs from the engine's own renderer. Sole producer of the capture committed byte-identically in `mthds-js/tests/fixtures/protocol/` and `mthds-python/tests/fixtures/protocol/`, which is what pins the TypeScript and Python projections against each other. See `docs/contribute/generate-projection-corpus.md`.++ ```bash+ .venv/bin/pipelex-dev generate-projection-corpus tests/data/input_semantics/*.mthds -o /tmp/projection-corpus+ ```+- **`drift`**: Drift contracts — deterministic review obligations between code and docs, declared in the root `drift.toml` (see `docs/contribute/drift-contracts.md`). When `make drift-check` (part of `make agent-check`, `make check`, and CI) reports an open contract: run `make drift-plan` to see what changed and what to review, actually review the targets and fix what is stale, `git add` the trigger files (the digest reads the git index, not the working tree), then record the review with `make drift-ack CONTRACT=<id> RATIONALE="…"`. The rationale is the on-the-record review decision — write an honest sentence. There is no bypass flag; "reviewed, no doc change needed" is a legitimate rationale.```bashdiff --git a/docs/contribute/generate-projection-corpus.md b/docs/contribute/generate-projection-corpus.mdnew file mode 100644index 000000000..1e1e4767e--- /dev/null+++ b/docs/contribute/generate-projection-corpus.md@@ -0,0 +1,52 @@+# Generating the Projection Fixture Corpus++The MTHDS build routes no longer render a pipe's fill-in inputs template on the server. Each client SDK projects it from the input-form descriptor it already has — once in TypeScript (`mthds`) and once in Python (`mthds-python`) — and the two projections must agree **exactly**, TOML comment lines included, or the JS/Python asymmetry the change set out to remove is simply rebuilt one layer up.++`pipelex-dev generate-projection-corpus` writes the fixture corpus that makes that agreement checkable. It is the **sole producer** of the capture committed byte-identically in `mthds-js/tests/fixtures/protocol/` and `mthds-python/tests/fixtures/protocol/`; `trace-input-semantics` stays what its own page says it is, a debugging tracer.++```bash+.venv/bin/pipelex-dev generate-projection-corpus \+ tests/data/input_semantics/hinted_bundle.mthds \+ tests/data/input_semantics/probe_bundle.mthds \+ tests/data/input_semantics/scaffold_bundle.mthds \+ -o /tmp/projection-corpus+```++Bundle order is part of the capture: it fixes the key order of every emitted map, so pass the bundles exactly as the corpus README in each consumer repo records them. Adding a bundle at the end keeps the existing bytes stable; inserting one in the middle rewrites the whole capture.++## What it writes++- `input_form.json` and `pipe_io_contracts.json` — the descriptor and contract capture, keyed by namespaced `pipe_ref`. Byte for byte what `trace-input-semantics` dumps at hop 5, so migrating the existing committed copies to this command is a no-op diff.+- `inputs_template/<pipe_ref>.<shape>.<format>` — the **expected** template each projection must produce, in both shapes (`compact`, `explicit`) and both formats (`json`, `toml`).+- `inputs_template/manifest.json` — what the corpus covers (bundles, pipes, shapes, formats) plus the declared divergences, each with worked sites a consumer repo can check with no engine present.+- `engine/` — the engine's own renderings of the same pipes. **Not committed.** It is what the divergence record is measured against, not part of the contract.++## Where the expectation comes from++The expected templates are not the engine's output. They come from the reference projection in `pipelex/cli/dev_cli/commands/projection_reference.py`, which walks the **descriptor** — the authored facts a method states — where the engine's renderer (`pipelex/pipe_machinery/rendering/input_renderer.py`) reflects the **runtime content classes**. That is the whole design: the shipped projections have only the descriptor, so the contract has to be authored from the descriptor too.++The reference projection lives in the dev CLI rather than in the runtime because nothing in `pipelex` consumes it. Its only job is to author the corpus bytes.++## The divergence record++Where the two disagree, the difference is **declared, never discovered**. The generator classifies every difference between its own rendering and the engine's, and fails if it meets one it has no declaration for — or if a declared class no longer occurs, so an engine fix retires its entry deliberately instead of leaving the manifest claiming a difference that has gone.++The classes it declares today:++- `optional-field-included` — the engine passes `include_optional=False` at the top of an input's own structure class and not through the recursion, so it hides an optional field at depth one and shows one nested deeper. The projection renders every field the descriptor states, at every depth.+- `file-leaf-not-expanded` — the descriptor states a file-ish node as a leaf whose only fill-in value is a URL; the engine expands the runtime content class and asks whoever fills the template in for a width, a mime type and a caption.+- `fixed-count-honoured` — a `Concept[N]` slot renders `N` elements. The engine emits one whatever the count, and `InputShaper` then rejects that template with `MultiplicityCountMismatchError`, so the scaffold it produced does not run.+- `text-named-url` — the engine picks a placeholder by field **name**, so a text field merely named `url` renders as a URL. The projection reads the descriptor's `kind`.+- `object-native-keeps-envelope` — a native carrying an optional field beside its required one (`native.Date`) renders as an object once the optional field is included, and the shaper's bare-value arm dispatches a native on its scalar kind, so the object form is only re-shapable inside its `{concept, content}` envelope. The projection keeps that envelope; the engine unwraps to a bare scalar, which it can only do because it drops the optional field. A consequence of `optional-field-included`.++Each is a `pipelex` defect filed in the workspace ledger, not a difference of taste; the manifest is what records that the corpus knowingly departs from the engine and why.++## The bundles++`tests/data/input_semantics/` holds the corpus bundles. `probe_bundle.mthds` exercises every construct the language accepts and `hinted_bundle.mthds` the intent hints; `scaffold_bundle.mthds` was added for this corpus and covers what the other two do not — a text field merely named `url` beside a real file position, an optional nested structure, optional `native.Image` and `native.Document` fields inside a structure, and both a fixed `[N]` and a variable `[]` slot over a structured concept.++## When to rerun it++Rerun and re-commit the capture in both consumer repos whenever the descriptor derivation changes, a `kind` is added to the standard, or a bundle changes. The per-repo harness asserts that the set of kinds appearing across the corpus **equals** the closed `FieldKind` vocabulary, so a kind added without a fixture fails by name rather than passing silently.++`tests/unit/pipelex/cli/dev_cli/test_generate_projection_corpus.py` keeps the capture complete and byte-stable, and `test_projection_reference.py` pins the projection's rules one at a time, so a rule that changed fails by name instead of as a wall of differing bytes.diff --git a/docs/under-the-hood/input-form-descriptor.md b/docs/under-the-hood/input-form-descriptor.mdindex 609810699..30e6dbb18 100644--- a/docs/under-the-hood/input-form-descriptor.md+++ b/docs/under-the-hood/input-form-descriptor.md@@ -70,3 +70,7 @@ Inapplicable slots are absent, never JSON `null`: the report's valid arm is dump## Seeing it`pipelex-dev trace-input-semantics` captures the descriptors as `hop5_input_form.json` beside `hop5_pipe_io_contracts.json`, so an authored fact can be checked on both projections at once — see [Tracing Input Semantics](../contribute/trace-input-semantics.md). The assignment table is pinned by `tests/integration/pipelex/pipeline/test_input_form.py` over the committed probe bundle (and the hints behavior over its hinted sibling, `hinted_bundle.mthds`), the emitted wire's null-free shape by `tests/unit/pipelex/pipeline/test_input_form_models.py`, the hint merges and kind-feeding by `tests/unit/pipelex/pipeline/test_input_form_hints.py`, and the escape hatches the library loader keeps unreachable (concept cycles, unregistered classes, a structure class that holds itself) by `tests/unit/pipelex/pipeline/test_input_form_deriver.py`.++## What clients project from it++The descriptor is what a client SDK renders a pipe's fill-in inputs template from — the hosted build routes no longer render one server-side. That projection is written twice, in TypeScript and in Python, and the two are pinned against each other by a shared fixture corpus this engine generates: see [Projection Fixture Corpus](../contribute/generate-projection-corpus.md). The corpus is also where the projection's rules are stated, including the places it is deliberately right where this engine's own inputs-template renderer is not.diff --git a/mkdocs.yml b/mkdocs.ymlindex 64afc887c..e0652d087 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -349,6 +349,7 @@ plugins:- contribute/drift-contracts.md: "Drift Contracts"- contribute/mthds-test-corpus.md: "MTHDS Test Corpus"- contribute/trace-input-semantics.md: "Tracing Input Semantics"+ - contribute/generate-projection-corpus.md: "Projection Fixture Corpus"- migration-ledger.md: "Migration Ledger"- changelog.md: "Changelog"@@ -601,6 +602,7 @@ nav:- Drift Contracts: contribute/drift-contracts.md- MTHDS Test Corpus: contribute/mthds-test-corpus.md- Tracing Input Semantics: contribute/trace-input-semantics.md+ - Projection Fixture Corpus: contribute/generate-projection-corpus.md- Migration Ledger: migration-ledger.md- Code of Conduct: CODE_OF_CONDUCT.md- License: license.mddiff --git a/pipelex/cli/dev_cli/_dev_cli.py b/pipelex/cli/dev_cli/_dev_cli.pyindex ed0a81838..3012fa365 100644--- a/pipelex/cli/dev_cli/_dev_cli.py+++ b/pipelex/cli/dev_cli/_dev_cli.py@@ -24,6 +24,7 @@ from pipelex.cli.dev_cli.commands.generate_corpus_vocabulary_cmd import generatefrom pipelex.cli.dev_cli.commands.generate_error_identity_cmd import generate_error_identity_cmdfrom pipelex.cli.dev_cli.commands.generate_error_pages_cmd import generate_error_pages_cmdfrom pipelex.cli.dev_cli.commands.generate_mthds_schema_cmd import generate_mthds_schema_cmd+from pipelex.cli.dev_cli.commands.generate_projection_corpus_cmd import generate_projection_corpus_cmdfrom pipelex.cli.dev_cli.commands.kit_cmd import kit_appfrom pipelex.cli.dev_cli.commands.preprocess_test_models_cmd import preprocess_test_models_cmdfrom pipelex.cli.dev_cli.commands.refresh_graph_ui_sri_cmd import refresh_graph_ui_sri_cmd@@ -59,6 +60,7 @@ class PipelexDevCLI(TyperGroup):"generate-error-identity","generate-error-pages","generate-mthds-schema",+ "generate-projection-corpus","kit","preprocess-test-models","refresh-graph-ui-sri",@@ -506,6 +508,15 @@ def sync_kit_configs_command(sys.exit(1)[email protected](name="generate-projection-corpus", help="Write the shared inputs-template projection fixture corpus")+def generate_projection_corpus_command(+ bundles: Annotated[list[Path], typer.Argument(help="MTHDS bundle file(s) to load as one batch — order fixes the emitted key order")],+ output_dir: Annotated[Path, typer.Option("--output-dir", "-o", help="Directory receiving the corpus")],+) -> None:+ """Write the descriptors, the expected inputs templates, and the divergence record."""+ generate_projection_corpus_cmd(bundle_paths=bundles, output_dir=output_dir)++@app.command(name="trace-input-semantics", help="Dump one artifact per hop of the input-schema emission chain for a bundle")def trace_input_semantics_command(bundles: Annotated[list[Path], typer.Argument(help="MTHDS bundle file(s) to load as one batch")],diff --git a/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.py b/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.pynew file mode 100644index 000000000..11695a1e8--- /dev/null+++ b/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.py@@ -0,0 +1,396 @@+"""Generate the shared projection fixture corpus the two inputs-template projections are pinned against.++The corpus is committed byte-identically in `mthds-js/tests/fixtures/protocol/` and+`mthds-python/tests/fixtures/protocol/`, beside the descriptor capture already there. It pairs each+pipe's input-form descriptor with the fill-in template both projections must produce from it — the+compact and explicit shapes, as JSON and as TOML — so "equivalent between JS and Python" is a+measured property rather than an aspiration.++What this command writes into the output directory:++- ``input_form.json`` / ``pipe_io_contracts.json`` — the descriptor and contract capture, byte for+ byte what ``trace-input-semantics`` dumps at hop 5. This command is the sole producer of the+ committed copies; the tracer stays a debugging tool.+- ``inputs_template/<pipe_ref>.<shape>.<format>`` — the expected template, from the reference+ projection in ``projection_reference.py``.+- ``inputs_template/manifest.json`` — the pipes covered and the declared divergences from the+ engine's own renderer, each with worked examples a consumer repo can check with no engine present.+- ``engine/`` — the engine's own renderings, for review at capture time. **Not committed**: it is+ what the divergence record is measured against, not part of the contract.++The divergences are declared here, not discovered: a class that stops occurring fails this command+rather than quietly disappearing, so an engine fix retires its entry deliberately.+"""++import asyncio+import json+import shutil+import sys+from pathlib import Path+from typing import Any, cast++from pydantic import BaseModel, ConfigDict, Field++from pipelex.base_exceptions import PipelexError+from pipelex.cli.cli_factory import make_pipelex_for_cli+from pipelex.cli.dev_cli.commands.projection_reference import (+ project_concept_comments,+ project_inputs_template,+)+from pipelex.cli.error_handlers import ErrorContext+from pipelex.interpreter_hub import (+ clear_current_library,+ get_current_library_id_or_none,+ get_library_manager,+ set_current_library,+)+from pipelex.pipe_machinery.pipe_abstract import PipeAbstract+from pipelex.pipe_machinery.rendering.input_renderer import (+ build_inputs_template,+ render_inputs,+ render_inputs_toml,+ serialize_inputs_template_to_toml,+)+from pipelex.pipelex import Pipelex+from pipelex.pipeline.exceptions import ValidateBundleError+from pipelex.pipeline.input_form import PipeInputFormDescriptor, build_input_form+from pipelex.pipeline.pipe_io_contracts import build_pipe_io_contracts+from pipelex.pipeline.validate_bundle import validate_bundle+from pipelex.runtime_hub import get_console+from pipelex.tools.typing.pydantic_utils import empty_list_factory_of++INPUT_FORM_FILE_NAME = "input_form.json"+PIPE_IO_CONTRACTS_FILE_NAME = "pipe_io_contracts.json"+TEMPLATES_DIR_NAME = "inputs_template"+ENGINE_DIR_NAME = "engine"+MANIFEST_FILE_NAME = "manifest.json"+MAX_EXAMPLES_PER_DIVERGENCE = 3+MOCK_URL_PREFIX = "https://mock.invalid/"++COMPACT_SHAPE = "compact"+EXPLICIT_SHAPE = "explicit"++DIVERGENCE_REASONS: dict[str, str] = {+ "optional-field-included": (+ "The engine passes include_optional=False at the top of an input's own structure class and not "+ "through the recursion, so it hides an optional field at depth one and shows one nested deeper. "+ "The projection renders every field the descriptor states, at every depth."+ ),+ "file-leaf-not-expanded": (+ "The descriptor states a file-ish node as a leaf whose only fill-in value is a URL, while the "+ "engine expands the runtime content class and asks whoever fills the template in for a width, a "+ "mime type and a caption."+ ),+ "fixed-count-honoured": (+ "A Concept[N] slot renders N elements. The engine emits one whatever the count, and InputShaper "+ "then rejects that template with MultiplicityCountMismatchError, so the scaffold does not run."+ ),+ "text-named-url": (+ "The engine picks a placeholder by field name, so a text field merely named url or ending in _url "+ "renders as a URL. The projection reads the descriptor's kind instead."+ ),+ "object-native-keeps-envelope": (+ "A native whose pinned definition carries an optional field beside its required one — native.Date — "+ "renders as an object once the optional field is included, and the shaper's bare-value arm dispatches "+ "a native on its scalar kind, so the object form is only re-shapable inside its {concept, content} "+ "envelope. The projection keeps that envelope; the engine unwraps to a bare scalar, which it can only "+ "do because it drops the optional field. A consequence of optional-field-included."+ ),+}+++# The workspace-ledger item tracking the engine fix, where the difference is a defect rather than a+# difference of vantage. Named in the manifest so the corpus records not just that it departs from+# the engine but what would retire the departure. A class with no entry here is deliberate on both+# sides: `file-leaf-not-expanded` is the descriptor's vantage rather than an engine bug, and+# `object-native-keeps-envelope` is a consequence of `optional-field-included` rather than its own.+DIVERGENCE_ITEMS: dict[str, str] = {+ "text-named-url": "L-260830-dc48bf",+ "fixed-count-honoured": "L-260830-f3de29",+ "optional-field-included": "L-260830-d51440",+}+++class DivergenceExample(BaseModel):+ """One worked site of a declared divergence, checkable in a consumer repo with no engine present.++ `expected` is the value the corpus holds at `path` in `<pipe_ref>.<shape>.json`, and `engine` the+ value the engine emits there — `null` when it emits nothing at all, which is unambiguous because a+ projected template never contains a null. A consumer repo checks a class has not lapsed by reading+ its own committed bytes at `path` and finding `expected` rather than `engine`.+ """++ model_config = ConfigDict(extra="forbid")++ pipe_ref: str+ shape: str+ path: str+ engine: Any = None+ expected: Any = None+++class DeclaredDivergence(BaseModel):+ """One class of deliberate difference between the projection and the engine's own renderer.++ `ledger_item` names the workspace-ledger item tracking the engine fix, and is `null` for a class+ that is a difference of vantage rather than a defect — see `DIVERGENCE_ITEMS`.+ """++ model_config = ConfigDict(extra="forbid")++ divergence_id: str+ reason: str+ ledger_item: str | None = None+ occurrences: int+ examples: list[DivergenceExample] = Field(default_factory=empty_list_factory_of(DivergenceExample))+++class CorpusManifest(BaseModel):+ """The corpus's own description of what it covers and where it departs from the engine."""++ model_config = ConfigDict(extra="forbid")++ bundles: list[str] = Field(default_factory=list)+ pipes: list[str] = Field(default_factory=list)+ shapes: list[str] = Field(default_factory=list)+ formats: list[str] = Field(default_factory=list)+ divergences: list[DeclaredDivergence] = Field(default_factory=empty_list_factory_of(DeclaredDivergence))+++def _write_json(*, path: Path, payload: Any) -> None:+ """The capture's byte discipline, shared with the input-semantics tracer."""+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")+++def _write_text(*, path: Path, content: str) -> None:+ """A rendering is written verbatim: the corpus holds exactly what a projection must return."""+ path.parent.mkdir(parents=True, exist_ok=True)+ path.write_text(content, encoding="utf-8")+++class _DivergenceCollector:+ """Walks the engine's template beside the projection's and buckets every difference.++ Every difference must land in a declared class: an unclassified one means the projection changed+ in a way nobody wrote down, and the capture refuses rather than committing bytes no record explains.+ """++ def __init__(self) -> None:+ self.counts: dict[str, int] = {}+ self.examples: dict[str, list[DivergenceExample]] = {}+ self.unclassified: list[str] = []++ def _record(self, *, divergence_id: str, path: list[str], engine: Any, expected: Any) -> None:+ self.counts[divergence_id] = self.counts.get(divergence_id, 0) + 1+ sites = self.examples.setdefault(divergence_id, [])+ if len(sites) < MAX_EXAMPLES_PER_DIVERGENCE:+ sites.append(+ DivergenceExample(+ pipe_ref=path[0],+ shape=path[1],+ path=".".join(path[2:]),+ engine=engine,+ expected=expected,+ )+ )++ def compare(self, *, engine_value: Any, projected_value: Any, path: list[str]) -> None:+ """Bucket every leaf difference between one engine value and its projected counterpart."""+ # Handled before the dict/dict walk below, which iterates the *projected* keys and would+ # therefore find nothing to say about an empty projection. Deliberately not declared in+ # DIVERGENCE_REASONS: no bundle in the corpus reaches it today, so a capture that does must+ # write the declaration rather than inherit one nobody reviewed.+ if isinstance(engine_value, dict) and projected_value == {} and engine_value != {}:+ self._record(divergence_id="unknown-empty-object", path=path, engine=engine_value, expected=projected_value)+ return+ if (+ isinstance(engine_value, dict)+ and isinstance(projected_value, dict)+ and set(cast("dict[str, Any]", projected_value)) == {"url"}+ and len(cast("dict[str, Any]", engine_value)) > 1+ and "url" in cast("dict[str, Any]", engine_value)+ ):+ self._record(divergence_id="file-leaf-not-expanded", path=path, engine=engine_value, expected=projected_value)+ return+ if isinstance(engine_value, dict) and isinstance(projected_value, dict):+ engine_fields = cast("dict[str, Any]", engine_value)+ projected_fields = cast("dict[str, Any]", projected_value)+ for key in projected_fields:+ if key not in engine_fields:+ self._record(divergence_id="optional-field-included", path=[*path, key], engine=None, expected=projected_fields[key])+ continue+ self.compare(engine_value=engine_fields[key], projected_value=projected_fields[key], path=[*path, key])+ return+ if isinstance(engine_value, list) and isinstance(projected_value, list):+ engine_items = cast("list[Any]", engine_value)+ projected_items = cast("list[Any]", projected_value)+ if len(engine_items) == 1 and len(projected_items) > 1:+ self._record(divergence_id="fixed-count-honoured", path=path, engine=engine_items, expected=projected_items)+ for index in range(min(len(engine_items), len(projected_items))):+ self.compare(engine_value=engine_items[index], projected_value=projected_items[index], path=[*path, str(index)])+ return+ if engine_value == projected_value:+ return+ if isinstance(engine_value, str) and engine_value.startswith(MOCK_URL_PREFIX) and isinstance(projected_value, str):+ self._record(divergence_id="text-named-url", path=path, engine=engine_value, expected=projected_value)+ return+ if isinstance(engine_value, dict) or isinstance(projected_value, dict):+ self._record(divergence_id="object-native-keeps-envelope", path=path, engine=engine_value, expected=projected_value)+ return+ self.unclassified.append(f"{'.'.join(path)}: engine={engine_value!r} projected={projected_value!r}")++ def declared(self) -> list[DeclaredDivergence]:+ """The divergence record, refusing a class that is undeclared or has stopped occurring."""+ undeclared = sorted(set(self.counts) - set(DIVERGENCE_REASONS))+ if undeclared:+ msg = f"Undeclared divergence class(es) {undeclared}: add the reason to DIVERGENCE_REASONS or fix the projection."+ raise ValueError(msg)+ lapsed = sorted(set(DIVERGENCE_REASONS) - set(self.counts))+ if lapsed:+ msg = f"Declared divergence class(es) {lapsed} no longer occur — delete the entry, the engine agrees now."+ raise ValueError(msg)+ return [+ DeclaredDivergence(+ divergence_id=divergence_id,+ reason=DIVERGENCE_REASONS[divergence_id],+ ledger_item=DIVERGENCE_ITEMS.get(divergence_id),+ occurrences=self.counts[divergence_id],+ examples=self.examples.get(divergence_id, []),+ )+ for divergence_id in sorted(self.counts)+ ]+++def _render_projection(*, descriptor: PipeInputFormDescriptor, explicit: bool) -> tuple[str, str]:+ """The projected template for one pipe and shape, as the JSON and TOML bytes the corpus holds."""+ template = project_inputs_template(descriptor=descriptor, explicit=explicit)+ json_text = json.dumps(template, indent=2, ensure_ascii=False)+ if explicit:+ toml_text = serialize_inputs_template_to_toml(template)+ else:+ toml_text = serialize_inputs_template_to_toml(template, light=True, concept_comments=project_concept_comments(descriptor=descriptor))+ return json_text, toml_text+++def _capture_pipe(+ *,+ pipe: PipeAbstract,+ descriptor: PipeInputFormDescriptor,+ output_dir: Path,+ collector: _DivergenceCollector,+) -> None:+ """Write one pipe's four projected renderings, the engine's four, and bucket the differences."""+ templates_dir = output_dir / TEMPLATES_DIR_NAME+ engine_dir = output_dir / ENGINE_DIR_NAME+ for explicit in (False, True):+ shape = EXPLICIT_SHAPE if explicit else COMPACT_SHAPE+ json_text, toml_text = _render_projection(descriptor=descriptor, explicit=explicit)+ _write_text(path=templates_dir / f"{pipe.pipe_ref}.{shape}.json", content=json_text)+ _write_text(path=templates_dir / f"{pipe.pipe_ref}.{shape}.toml", content=toml_text)+ _write_text(path=engine_dir / f"{pipe.pipe_ref}.{shape}.json", content=render_inputs(pipe, explicit=explicit))+ _write_text(path=engine_dir / f"{pipe.pipe_ref}.{shape}.toml", content=render_inputs_toml(pipe, explicit=explicit))+ collector.compare(+ engine_value=build_inputs_template(pipe, explicit=explicit),+ projected_value=project_inputs_template(descriptor=descriptor, explicit=explicit),+ path=[pipe.pipe_ref, shape],+ )+++async def generate_projection_corpus(*, bundle_paths: list[Path], output_dir: Path) -> CorpusManifest:+ """Validate the given bundles and write the whole projection fixture corpus.++ Args:+ bundle_paths: The `.mthds` files to load as one batch. Their order fixes the key order of the+ emitted maps, so it is part of the capture: pass them the way the corpus README records.+ output_dir: Directory receiving the corpus (created if missing).++ Returns:+ The manifest, also written to the corpus.++ Raises:+ ValidateBundleError: When a bundle is invalid — the corpus requires valid bundles.+ ValueError: When a difference from the engine is undeclared, or a declared one has lapsed.+ """+ mthds_contents = [bundle_path.read_text(encoding="utf-8") for bundle_path in bundle_paths]+ mthds_sources = [str(bundle_path) for bundle_path in bundle_paths]++ output_dir.mkdir(parents=True, exist_ok=True)+ # Both directories accumulate one file per pipe and shape, so a rerun must replace them wholesale:+ # a leftover rendering from a renamed pipe would read as part of the current corpus.+ for stale_dir_name in (TEMPLATES_DIR_NAME, ENGINE_DIR_NAME):+ stale_dir = output_dir / stale_dir_name+ if stale_dir.is_dir():+ shutil.rmtree(stale_dir)++ prev_library_id = get_current_library_id_or_none()+ validation_library_id: str | None = None+ collector = _DivergenceCollector()+ try:+ result = await validate_bundle(mthds_contents=mthds_contents, mthds_sources=mthds_sources)+ validation_library_id = get_current_library_id_or_none()++ input_form = build_input_form(result.pipes)+ _write_json(+ path=output_dir / INPUT_FORM_FILE_NAME,+ payload={pipe_ref: descriptor.model_dump(mode="json") for pipe_ref, descriptor in input_form.items()},+ )+ io_contracts = build_pipe_io_contracts(result.pipes)+ _write_json(+ path=output_dir / PIPE_IO_CONTRACTS_FILE_NAME,+ payload={pipe_ref: contract.model_dump(mode="json") for pipe_ref, contract in io_contracts.items()},+ )+ for pipe in result.pipes:+ _capture_pipe(pipe=pipe, descriptor=input_form[pipe.pipe_ref], output_dir=output_dir, collector=collector)+ finally:+ if validation_library_id is not None and validation_library_id != prev_library_id:+ if prev_library_id is not None:+ set_current_library(library_id=prev_library_id)+ else:+ clear_current_library()+ get_library_manager().teardown(library_id=validation_library_id)++ if collector.unclassified:+ listed = "\n ".join(collector.unclassified[:20])+ msg = f"The projection differs from the engine at site(s) no declared divergence explains:\n {listed}"+ raise ValueError(msg)++ manifest = CorpusManifest(+ bundles=[bundle_path.name for bundle_path in bundle_paths],+ pipes=sorted(input_form),+ shapes=[COMPACT_SHAPE, EXPLICIT_SHAPE],+ formats=["json", "toml"],+ divergences=collector.declared(),+ )+ _write_json(path=output_dir / TEMPLATES_DIR_NAME / MANIFEST_FILE_NAME, payload=manifest.model_dump(mode="json"))+ return manifest+++def generate_projection_corpus_cmd(*, bundle_paths: list[Path], output_dir: Path) -> None:+ """CLI wrapper: boot Pipelex, write the corpus, report what it covers and where it diverges."""+ console = get_console()+ for bundle_path in bundle_paths:+ if not bundle_path.is_file():+ console.print(f"[red]Bundle file not found: {bundle_path}[/red]")+ sys.exit(2)++ make_pipelex_for_cli(context=ErrorContext.VALIDATION, needs_inference=False, needs_model_specs=True)+ try:+ manifest = asyncio.run(generate_projection_corpus(bundle_paths=bundle_paths, output_dir=output_dir))+ except ValidateBundleError as exc:+ console.print(f"[red]Bundle validation failed — the corpus requires valid bundles:[/red]\n{exc}")+ sys.exit(1)+ except ValueError as exc:+ console.print(f"[red]Divergence record is out of date:[/red]\n{exc}")+ sys.exit(1)+ except PipelexError as exc:+ console.print(f"[red]Corpus generation failed:[/red]\n{exc}")+ sys.exit(1)+ finally:+ Pipelex.teardown_if_needed()++ console.print(f"[green]✓ Projection corpus written to {output_dir}[/green]")+ console.print(f" pipes: {len(manifest.pipes)}, each in {len(manifest.shapes)} shapes × {len(manifest.formats)} formats")+ for divergence in manifest.divergences:+ console.print(f" divergence {divergence.divergence_id}: {divergence.occurrences} site(s)")+ console.print(f" the engine's own renderings are under {ENGINE_DIR_NAME}/ — for review, not for committing")diff --git a/pipelex/cli/dev_cli/commands/projection_reference.py b/pipelex/cli/dev_cli/commands/projection_reference.pynew file mode 100644index 000000000..63329529f--- /dev/null+++ b/pipelex/cli/dev_cli/commands/projection_reference.py@@ -0,0 +1,308 @@+"""The reference projection: an input-form descriptor rendered as a fill-in inputs template.++This module is the **contract** the two shipped projections — the TypeScript helper in the `mthds`+package and its Python twin in `mthds-python` — must reproduce byte for byte. It exists in the dev+CLI, not in the runtime, because nothing in `pipelex` consumes it: its only job is to author the+expected bytes of the shared fixture corpus those two projections are pinned against.++**It walks the descriptor, never the runtime content classes**, and that is the whole difference+from the engine's own renderer (`pipelex/pipe_machinery/rendering/input_renderer.py`). The engine+reflects each input's pydantic class, so its template states what the *runtime* holds; this walks+the authored facts the descriptor states, so its template states what the *method declares*. The+two disagree in three places, deliberately, and the corpus records each one:++- **An optional structure field is rendered.** The engine passes `include_optional=False` at the top+ of an input's own class and not through the recursion, so it hides an optional field at depth one+ and shows one nested deeper. Every field the descriptor states is rendered here, at every depth.+- **A file-ish node is a leaf.** `native.Image` is `kind = "image"` in the descriptor — a leaf whose+ only fill-in value is a URL — while the engine expands `ImageContent` into its nine runtime+ fields, asking whoever fills the template in for a width and a mime type.+- **A fixed-count slot renders its count.** `Concept[N]` gets `N` elements; the engine emits one,+ which `InputShaper._shape_list` then rejects with `MultiplicityCountMismatchError`.++Two further rules are the descriptor's, not the engine's, because the engine has no equivalent:+an `enum` takes its first choice (the engine picks at random, which no committed template can+carry), and an `unknown` node renders as an empty object, the escape hatch's only honest value.++The match over `FieldKind` is exhaustive and carries no default arm on purpose: a kind added to the+standard breaks this generator where the rule for it has to be written, rather than falling through+to a guess.+"""++from typing import Any++from pipelex.core.pipes.variable_multiplicity import PresenceMarker, format_concept_with_multiplicity+from pipelex.pipeline.input_form import (+ FieldKind,+ InputFormField,+ InputFormItem,+ ListItem,+ PipeInputFormDescriptor,+)++MOCK_URL_PREFIX = "https://mock.invalid/"+FILE_CONTENT_KEY = "url"+TIME_FORMAT = "time"++# The single wire key a native scalar's value sits inside. It is a fact about the *payload*, which+# the descriptor deliberately does not carry — so the projection needs this table to build the+# explicit `{concept, content}` envelope. It is the standard's to state, not the runtime's: the+# native content shapes are pinned by `mthds/docs/spec/native-concepts.md`.+TEXT_CONTENT_KEY = "text"+NUMBER_CONTENT_KEY = "number"+BOOLEAN_CONTENT_KEY = "yes_no"+DATE_CONTENT_KEY = "date"+TIME_CONTENT_KEY = "time"++NATIVE_PREFIX = "native."++# The natives the runtime's input shaper cannot build top-down: their light form keeps the whole+# `{concept, content}` envelope, because a bare value at one of these positions is not re-shapable.+# The vocabulary is the standard's closed native set (`mthds/docs/spec/native-concepts.md`), which is+# why a projection may consult it: it reads an identity the descriptor states, never sniffs a shape.+OUT_OF_MATRIX_NATIVES = frozenset(+ {+ "Anything",+ "Composite",+ "Dynamic",+ "Html",+ "JSON",+ "Page",+ "SearchResult",+ "TextAndImages",+ }+)++# `native.Number`'s content is a number union, and the runtime placeholders it as `1` rather than as+# the `0` / `0.0` a plain `type = "number"` structure field takes. The descriptor states `kind =+# "number"` for both, so the native identity is what separates them.+NATIVE_NUMBER = "Number"+NATIVE_NUMBER_PLACEHOLDER = 1+++def native_code(*, node: InputFormItem) -> str | None:+ """The native concept this node's chain names, if any.++ Reads `concept_ref` first, then the `refines` membership list, so a concept refining a native+ resolves the same way the native itself does.+ """+ candidates: list[str] = []+ if node.concept_ref is not None:+ candidates.append(node.concept_ref)+ if node.refines is not None:+ candidates.extend(node.refines)+ for candidate in candidates:+ if candidate.startswith(NATIVE_PREFIX):+ return candidate[len(NATIVE_PREFIX) :]+ return None+++def keeps_envelope(*, node: InputFormItem) -> bool:+ """Whether this slot's light form keeps the ceremonial envelope instead of unwrapping.++ Two ways to earn it, and both mean the same thing — a bare value at this position is not+ re-shapable, so unwrapping would pin a template that does not run. Either the native is one the+ shaper cannot build top-down at all, or the descriptor states it as an object: the shaper's+ bare-value arm dispatches a native on its scalar kind, so it rejects the object outright.+ `native.Date` is the second case — it is a scalar the shaper knows, until the optional `time`+ beside its required `date` makes the rendered form an object.+ """+ code = native_code(node=node)+ if code is None:+ return False+ return code in OUT_OF_MATRIX_NATIVES or node.kind is FieldKind.OBJECT+++def slot_content_key(*, node: InputFormItem) -> str | None:+ """The single wire key a slot-position scalar's value sits inside, or None when it is not one."""+ match node.kind:+ case FieldKind.TEXT | FieldKind.PROSE:+ if node.format == TIME_FORMAT:+ return TIME_CONTENT_KEY+ return TEXT_CONTENT_KEY+ case FieldKind.NUMBER:+ return NUMBER_CONTENT_KEY+ case FieldKind.BOOLEAN:+ return BOOLEAN_CONTENT_KEY+ case FieldKind.DATE:+ return DATE_CONTENT_KEY+ case FieldKind.ENUM:+ return TEXT_CONTENT_KEY+ case FieldKind.IMAGE | FieldKind.DOCUMENT:+ return FILE_CONTENT_KEY+ case FieldKind.OBJECT | FieldKind.LIST | FieldKind.UNKNOWN:+ return None+++def _leaf_placeholder(*, node: InputFormItem, name: str) -> Any:+ """The fill-in value for a node the descriptor states as a leaf.++ `name` is the name of the field the value occupies, which is what the placeholder is built from:+ a structure field's own name when the leaf sits inside a content dict, and the content key when+ it sits at a slot, where the value occupies its native content class's single field.+ """+ match node.kind:+ case FieldKind.TEXT | FieldKind.PROSE:+ if node.format == TIME_FORMAT:+ return "12:00:00"+ return f"{name}_value"+ case FieldKind.DATE:+ if node.datetime:+ return "2026-01-01T12:00:00"+ return "2026-01-01"+ case FieldKind.NUMBER:+ if native_code(node=node) == NATIVE_NUMBER:+ return NATIVE_NUMBER_PLACEHOLDER+ if node.integer:+ return 0+ return 0.0+ case FieldKind.BOOLEAN:+ return False+ case FieldKind.ENUM:+ # The first choice, never a random one: these bytes are committed.+ if node.choices:+ return node.choices[0]+ return f"{name}_value"+ case FieldKind.IMAGE | FieldKind.DOCUMENT:+ return f"{MOCK_URL_PREFIX}{FILE_CONTENT_KEY}"+ case FieldKind.OBJECT | FieldKind.LIST | FieldKind.UNKNOWN:+ msg = f"Not a leaf kind: {node.kind}"+ raise ValueError(msg)+++def _item_repetitions(*, node: ListItem) -> int:+ """How many elements a list renders: its declared count, or one example for a variable list."""+ return node.item_count if node.item_count is not None else 1+++def project_value(*, node: InputFormItem, name: str) -> Any:+ """The value one descriptor node takes inside a content dict.++ A scalar-typed node is its bare placeholder; a concept-typed one (`image`, `document`, `object`)+ is the content dict its concept carries, because that is what sits at the field in the payload.++ One case reads as a scalar and is not: a nested node that names a native concept holds that+ native's own content object, not a bare value — `native.Text` inside a page's text-and-images is+ a `TextContent`, so the payload carries `{"text": ...}` there. The descriptor states the+ difference itself, in whether the node carries a `concept_ref`: an authored `type = "text"`+ structure field carries none and stays bare.+ """+ match node.kind:+ case FieldKind.TEXT | FieldKind.PROSE | FieldKind.DATE | FieldKind.NUMBER | FieldKind.BOOLEAN | FieldKind.ENUM:+ content_key = slot_content_key(node=node) if native_code(node=node) is not None else None+ if content_key is not None:+ return {content_key: _leaf_placeholder(node=node, name=content_key)}+ return _leaf_placeholder(node=node, name=name)+ case FieldKind.IMAGE | FieldKind.DOCUMENT:+ return {FILE_CONTENT_KEY: _leaf_placeholder(node=node, name=FILE_CONTENT_KEY)}+ case FieldKind.OBJECT:+ return {field.name: project_value(node=field, name=field.name) for field in node.fields}+ case FieldKind.LIST:+ item_value = project_value(node=node.item, name=f"{name}_item")+ return [item_value for _ in range(_item_repetitions(node=node))]+ case FieldKind.UNKNOWN:+ return {}+++def _slot_content(*, node: InputFormItem, name: str) -> Any:+ """The `content` half of one slot's envelope — what the concept carries at that position."""+ content_key = slot_content_key(node=node)+ if content_key is not None:+ return {content_key: _leaf_placeholder(node=node, name=content_key)}+ match node.kind:+ case FieldKind.LIST:+ item_content = _slot_content(node=node.item, name=f"{name}_item")+ return [item_content for _ in range(_item_repetitions(node=node))]+ case (+ FieldKind.OBJECT+ | FieldKind.UNKNOWN+ | FieldKind.TEXT+ | FieldKind.PROSE+ | FieldKind.DATE+ | FieldKind.NUMBER+ | FieldKind.BOOLEAN+ | FieldKind.ENUM+ | FieldKind.IMAGE+ | FieldKind.DOCUMENT+ ):+ return project_value(node=node, name=name)+++def _light_value(*, node: InputFormItem, name: str) -> Any:+ """One slot's (or one slot element's) light value: a scalar unwraps, everything else keeps its dict."""+ content_key = slot_content_key(node=node)+ if content_key is not None:+ return _leaf_placeholder(node=node, name=content_key)+ return project_value(node=node, name=name)+++def _compact_slot(*, field: InputFormField) -> Any:+ """One slot in the light shape.++ An out-of-matrix native keeps the whole envelope, exactly as the runtime's own light rendering+ does: the shaper cannot rebuild those values from a bare one, so unwrapping would emit a+ template that no longer runs.+ """+ if keeps_envelope(node=field):+ return {"concept": field.concept_ref, "content": _slot_content(node=field, name=field.name)}+ match field.kind:+ case FieldKind.LIST:+ item_value = _light_value(node=field.item, name=f"{field.name}_item")+ return [item_value for _ in range(_item_repetitions(node=field))]+ case (+ FieldKind.OBJECT+ | FieldKind.UNKNOWN+ | FieldKind.TEXT+ | FieldKind.PROSE+ | FieldKind.DATE+ | FieldKind.NUMBER+ | FieldKind.BOOLEAN+ | FieldKind.ENUM+ | FieldKind.IMAGE+ | FieldKind.DOCUMENT+ ):+ return _light_value(node=field, name=field.name)+++def slot_bundle_representation(*, field: InputFormField) -> str:+ """The `concept: …` comment text a light TOML template carries above each key.++ Rebuilt from the descriptor rather than from the pipe's `StuffSpec`, because that is all the+ shipped projections have: the concept ref, the multiplicity the `list` node states, and the+ presence marker.+ """+ presence = field.presence if field.presence is not None else PresenceMarker.PLAIN+ if isinstance(field, ListItem):+ multiplicity = field.item_count if field.item_count is not None else True+ else:+ multiplicity = None+ concept_ref = field.concept_ref if field.concept_ref is not None else ""+ return f"concept: {format_concept_with_multiplicity(concept_ref, multiplicity=multiplicity, presence=presence)}"+++def project_inputs_template(*, descriptor: PipeInputFormDescriptor, explicit: bool) -> dict[str, Any]:+ """Project one pipe's descriptor into the fill-in inputs template.++ Args:+ descriptor: The pipe's input-form descriptor.+ explicit: When True, keep the ceremonial `{"concept", "content"}` envelope; when False+ (default shape), emit the light form a smart-inputs run accepts directly.++ Returns:+ The template, one entry per declared input slot, in authored order.+ """+ template: dict[str, Any] = {}+ for field in descriptor.fields:+ if explicit:+ template[field.name] = {+ "concept": field.concept_ref,+ "content": _slot_content(node=field, name=field.name),+ }+ else:+ template[field.name] = _compact_slot(field=field)+ return template+++def project_concept_comments(*, descriptor: PipeInputFormDescriptor) -> dict[str, str]:+ """The per-slot `concept: …` comment map a light TOML rendering carries."""+ return {field.name: slot_bundle_representation(field=field) for field in descriptor.fields}diff --git a/pyproject.toml b/pyproject.tomlindex e0c7ee0bc..2ab7ef1a9 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -29,7 +29,7 @@ dependencies = ["instructor>=1.13", # 1.13+ exposes instructor.core; 1.11/1.12 had mypy typing errors"jinja2>=3.1.4","json2html>=1.3.0",- "kajson==0.7.0",+ "kajson==0.7.1","markdown>=3.6","mthds==0.11.1","networkx>=3.4.2",diff --git a/tests/data/input_semantics/scaffold_bundle.mthds b/tests/data/input_semantics/scaffold_bundle.mthdsnew file mode 100644index 000000000..4b53e7ddb--- /dev/null+++ b/tests/data/input_semantics/scaffold_bundle.mthds@@ -0,0 +1,53 @@+domain = "input_semantics_scaffold"+description = "SCAFFOLD_bundle_desc: the cases a descriptor-driven inputs-template projection must get right and the engine's own renderer does not — every authored fact carries a SCAFFOLD_ sentinel so its survival is greppable in every capture"+main_pipe = "scaffold_nesting"++# A structure whose text fields are merely NAMED like URLs. The engine picks a placeholder by+# field name, so it renders these as `https://mock.invalid/<name>` even though the descriptor+# states `kind = "text"` at every one of them. That name heuristic is what misclassified a text+# field as a file position; the projection reads the kind instead.+[concept.Bookmark]+description = "SCAFFOLD_desc_concept_Bookmark: a shelf reference whose text fields are named like links"++[concept.Bookmark.structure]+url = { type = "text", description = "SCAFFOLD_desc_field_url: a required text field merely NAMED url", required = true }+homepage_url = { type = "text", description = "SCAFFOLD_desc_field_homepage_url: an optional text field whose name ENDS in _url" }+label = { type = "text", description = "SCAFFOLD_desc_field_label: ordinary required text beside them", required = true }+shelf_note = { type = "text", description = "SCAFFOLD_desc_field_shelf_note: ordinary optional text beside them" }++# A structure carrying file fields on both sides of the required/optional line. The engine applies+# its optional filter only at the top level of an input's own structure class, so `cover` and+# `attachment` vanish from the template it renders while a nested structure's optional fields survive.+[concept.Dossier]+description = "SCAFFOLD_desc_concept_Dossier: an archive folder holding required and optional attachments"++[concept.Dossier.structure]+title = { type = "text", description = "SCAFFOLD_desc_field_title: required text heading the folder", required = true }+stamp = { type = "concept", concept_ref = "native.Image", description = "SCAFFOLD_desc_field_stamp: a REQUIRED nested image", required = true }+cover = { type = "concept", concept_ref = "native.Image", description = "SCAFFOLD_desc_field_cover: an OPTIONAL nested image" }+attachment = { type = "concept", concept_ref = "native.Document", description = "SCAFFOLD_desc_field_attachment: an OPTIONAL nested document" }+bookmark = { type = "concept", concept_ref = "Bookmark", description = "SCAFFOLD_desc_field_bookmark: the url-named fields one level deeper" }++[pipe.scaffold_nesting]+type = "PipeLLM"+description = "SCAFFOLD_pipe_scaffold_nesting: url-named text fields and optional nested file fields, at two depths"+inputs = { bookmark = "Bookmark", dossier = "Dossier" }+output = "Text"+prompt = """+Consider the bookmark:+@bookmark+And the dossier:+@dossier+"""++[pipe.scaffold_fixed_batch]+type = "PipeLLM"+description = "SCAFFOLD_pipe_scaffold_fixed_batch: a fixed-count list of a structure carrying optional file fields, beside a variable one"+inputs = { dossiers = "Dossier[2]", bookmarks = "Bookmark[]" }+output = "Text"+prompt = """+Exactly two dossiers:+@dossiers+Any number of bookmarks:+@bookmarks+"""diff --git a/tests/unit/pipelex/cli/dev_cli/test_generate_projection_corpus.py b/tests/unit/pipelex/cli/dev_cli/test_generate_projection_corpus.pynew file mode 100644index 000000000..e973cac33--- /dev/null+++ b/tests/unit/pipelex/cli/dev_cli/test_generate_projection_corpus.py@@ -0,0 +1,88 @@+"""The corpus generator produces a complete, stable capture, and every divergence it declares is real.++The generator is the sole producer of the shared projection fixture corpus committed in+`mthds-js/tests/fixtures/protocol/` and `mthds-python/tests/fixtures/protocol/`. Two properties make+that corpus trustworthy, and both are checked here against the corpus bundles themselves:++- **Every declared divergence occurs.** A class declared but no longer produced would leave the+ manifest claiming a difference from the engine that no longer exists, and every consumer repo's+ lapse check reads that manifest. The generator raises on a lapse; this states it as a property.+- **A rerun writes the same bytes.** The corpus is committed, so a capture that varied run to run+ would show up as a diff nobody authored — and the enum placeholder is exactly where the engine's+ own renderer varies.+"""++from __future__ import annotations++import json+from pathlib import Path++import pytest++from pipelex.cli.dev_cli.commands.generate_projection_corpus_cmd import (+ ENGINE_DIR_NAME,+ INPUT_FORM_FILE_NAME,+ MANIFEST_FILE_NAME,+ PIPE_IO_CONTRACTS_FILE_NAME,+ TEMPLATES_DIR_NAME,+ CorpusManifest,+ generate_projection_corpus,+)++CORPUS_BUNDLES = [+ Path("tests/data/input_semantics/hinted_bundle.mthds"),+ Path("tests/data/input_semantics/probe_bundle.mthds"),+ Path("tests/data/input_semantics/scaffold_bundle.mthds"),+]+++def _corpus_files(*, corpus_dir: Path) -> dict[str, str]:+ """Every committed file in the corpus, by path relative to its root — the engine's copies excluded."""+ return {+ str(path.relative_to(corpus_dir)): path.read_text(encoding="utf-8")+ for path in sorted(corpus_dir.rglob("*"))+ if path.is_file() and ENGINE_DIR_NAME not in path.relative_to(corpus_dir).parts+ }++[email protected](loop_scope="class")+class TestGenerateProjectionCorpus:+ async def test_the_corpus_is_complete_and_its_divergences_are_all_real(self, tmp_path: Path) -> None:+ """One capture, checked for the two things a consumer repo cannot check for itself."""+ manifest = await generate_projection_corpus(bundle_paths=CORPUS_BUNDLES, output_dir=tmp_path)++ # Every declared divergence is non-vacuous: it names sites, and each states a value the+ # engine emits and a different one the corpus expects.+ assert manifest.divergences+ for divergence in manifest.divergences:+ assert divergence.examples, f"divergence {divergence.divergence_id} declares no site"+ assert divergence.reason+ for example in divergence.examples:+ assert example.engine != example.expected++ # Every pipe is captured in both shapes and both formats, and nothing is empty.+ template_dir = tmp_path / TEMPLATES_DIR_NAME+ expected_files = {+ f"{pipe_ref}.{shape}.{file_format}" for pipe_ref in manifest.pipes for shape in manifest.shapes for file_format in manifest.formats+ }+ written = {path.name for path in template_dir.iterdir() if path.name != MANIFEST_FILE_NAME}+ assert written == expected_files+ for path in template_dir.iterdir():+ assert path.read_text(encoding="utf-8").strip(), f"{path.name} is empty"++ # The descriptor half is captured beside the templates: a projection needs both.+ assert json.loads((tmp_path / INPUT_FORM_FILE_NAME).read_text(encoding="utf-8")).keys() == set(manifest.pipes)+ assert json.loads((tmp_path / PIPE_IO_CONTRACTS_FILE_NAME).read_text(encoding="utf-8")).keys() == set(manifest.pipes)++ # The manifest round-trips: it is what the consumer repos parse.+ assert CorpusManifest.model_validate_json((template_dir / MANIFEST_FILE_NAME).read_text(encoding="utf-8")) == manifest++ async def test_a_rerun_writes_the_same_bytes(self, tmp_path: Path) -> None:+ """The corpus is committed, so the capture has to be a function of the bundles alone."""+ first_dir = tmp_path / "first"+ second_dir = tmp_path / "second"++ await generate_projection_corpus(bundle_paths=CORPUS_BUNDLES, output_dir=first_dir)+ await generate_projection_corpus(bundle_paths=CORPUS_BUNDLES, output_dir=second_dir)++ assert _corpus_files(corpus_dir=first_dir) == _corpus_files(corpus_dir=second_dir)diff --git a/tests/unit/pipelex/cli/dev_cli/test_projection_reference.py b/tests/unit/pipelex/cli/dev_cli/test_projection_reference.pynew file mode 100644index 000000000..64710b08f--- /dev/null+++ b/tests/unit/pipelex/cli/dev_cli/test_projection_reference.py@@ -0,0 +1,271 @@+"""The reference projection's rules, pinned one by one.++These are the rules the two shipped projections (the `mthds` TypeScript helper and its+`mthds-python` twin) must reproduce, and which the shared fixture corpus states as bytes. They are+asserted here against hand-built descriptors rather than against the corpus, so a rule that changed+fails by name instead of as a wall of differing bytes.++Four of them are deliberate departures from the engine's own renderer+(`pipelex/pipe_machinery/rendering/input_renderer.py`), each recorded in the corpus manifest the+generator writes; the rest is what the engine already does.++Descriptors are built here from plain dictionaries rather than from the model classes, because that+is the shape the shipped projections actually receive — the corpus files are JSON, and a rule that+only holds for a Python-constructed model would not be a rule the corpus can state.+"""++from typing import Any++import pytest++from pipelex.cli.dev_cli.commands.projection_reference import (+ project_concept_comments,+ project_inputs_template,+)+from pipelex.pipeline.input_form import PipeInputFormDescriptor+++def _descriptor(*slots: dict[str, Any]) -> PipeInputFormDescriptor:+ """A descriptor holding the given slots, in order, each stamped with its pipe-slot facts.++ `presence` and `gating` are stated on a top-level field and forbidden below it, so they are+ added here rather than written out at every call site.+ """+ fields = [{**slot, "presence": "plain" if slot["required"] else "optional", "gating": bool(slot["required"])} for slot in slots]+ return PipeInputFormDescriptor.model_validate({"fields": fields})+++def _text(*, name: str, required: bool = True) -> dict[str, Any]:+ """An authored `type = "text"` structure field: no pipe-slot facts, and no `concept_ref`.++ The absence is the point rather than a shortcut. A field authored as a plain type carries no+ concept identity in the descriptor, and that is exactly what separates it from one authored as+ `concept = "native.Text"`, which holds a nested content object instead of a bare value.+ """+ return {"kind": "text", "name": name, "required": required}+++def _native_text(*, name: str, required: bool = True) -> dict[str, Any]:+ """A text field that names `native.Text`, so the value at it is that native's content object."""+ return {"kind": "text", "name": name, "concept_ref": "native.Text", "required": required}+++class TestDeliberateDivergencesFromTheEngine:+ """The four places the projection is right where the engine's renderer is not."""++ def test_a_text_field_merely_named_url_takes_a_text_placeholder(self):+ """The engine picks a placeholder by field name; the descriptor states the kind."""+ bookmark = {+ "kind": "object",+ "name": "bookmark",+ "concept_ref": "shelf.Bookmark",+ "required": True,+ "fields": [_text(name="url")],+ }+ template = project_inputs_template(descriptor=_descriptor(bookmark), explicit=False)+ assert template["bookmark"]["url"] == "url_value"+ assert "mock.invalid" not in template["bookmark"]["url"]++ def test_an_optional_structure_field_is_rendered(self):+ """The engine hides it at depth one, which is how the old template lost a file position."""+ dossier = {+ "kind": "object",+ "name": "dossier",+ "concept_ref": "shelf.Dossier",+ "required": True,+ "fields": [+ _text(name="title"),+ {"kind": "image", "name": "cover", "concept_ref": "native.Image", "required": False},+ ],+ }+ template = project_inputs_template(descriptor=_descriptor(dossier), explicit=False)+ assert set(template["dossier"]) == {"title", "cover"}++ def test_a_fixed_count_slot_renders_its_count(self):+ """`Concept[2]` needs two elements: `InputShaper` rejects a one-element scaffold outright."""+ pair = {+ "kind": "list",+ "name": "pair",+ "concept_ref": "shelf.Bookmark",+ "required": True,+ "item_count": 2,+ "item": {"kind": "object", "required": True, "fields": [_text(name="label")]},+ }+ template = project_inputs_template(descriptor=_descriptor(pair), explicit=False)+ assert len(template["pair"]) == 2+ assert template["pair"][0] == template["pair"][1] == {"label": "label_value"}++ def test_a_file_node_is_a_leaf_carrying_only_its_url(self):+ """The engine expands the runtime content class and asks for a width and a mime type."""+ dossier = {+ "kind": "object",+ "name": "dossier",+ "concept_ref": "shelf.Dossier",+ "required": True,+ "fields": [{"kind": "image", "name": "stamp", "concept_ref": "native.Image", "required": True}],+ }+ template = project_inputs_template(descriptor=_descriptor(dossier), explicit=False)+ assert template["dossier"]["stamp"] == {"url": "https://mock.invalid/url"}+++class TestShapes:+ """Compact and explicit, and what separates them."""++ def test_a_variable_list_renders_one_example_element(self):+ many = {+ "kind": "list",+ "name": "many",+ "concept_ref": "native.Text",+ "required": True,+ "item": {"kind": "text", "required": True, "concept_ref": "native.Text"},+ }+ assert project_inputs_template(descriptor=_descriptor(many), explicit=False)["many"] == ["text_value"]++ def test_a_scalar_slot_unwraps_in_the_compact_shape_and_keeps_its_key_in_the_explicit_one(self):+ descriptor = _descriptor(_native_text(name="note"))+ assert project_inputs_template(descriptor=descriptor, explicit=False)["note"] == "text_value"+ assert project_inputs_template(descriptor=descriptor, explicit=True)["note"] == {+ "concept": "native.Text",+ "content": {"text": "text_value"},+ }++ def test_a_file_slot_unwraps_to_a_bare_url_in_the_compact_shape(self):+ cover = {"kind": "image", "name": "cover", "concept_ref": "native.Image", "required": True}+ assert project_inputs_template(descriptor=_descriptor(cover), explicit=False)["cover"] == "https://mock.invalid/url"++ def test_an_unknown_node_renders_as_an_empty_object(self):+ """The escape hatch's only honest value: the descriptor states nothing to render."""+ opaque = {"kind": "unknown", "name": "opaque", "concept_ref": "shelf.Opaque", "required": True}+ assert project_inputs_template(descriptor=_descriptor(opaque), explicit=False)["opaque"] == {}+++class TestPlaceholdersByKind:+ """Every leaf kind's fill-in value, at a slot and inside a structure."""++ @pytest.mark.parametrize(+ ("node", "expected"),+ [+ ({"kind": "text", "concept_ref": "native.Text"}, "text_value"),+ ({"kind": "text", "concept_ref": "native.Text", "format": "time"}, "12:00:00"),+ ({"kind": "prose", "concept_ref": "native.Text"}, "text_value"),+ ({"kind": "date", "concept_ref": "native.Date", "datetime": False}, "2026-01-01"),+ ({"kind": "date", "concept_ref": "native.Date", "datetime": True}, "2026-01-01T12:00:00"),+ ({"kind": "number", "concept_ref": "native.Number", "integer": False}, 1),+ ({"kind": "number", "integer": True}, 0),+ ({"kind": "number", "integer": False}, 0.0),+ ({"kind": "boolean", "concept_ref": "native.Boolean"}, False),+ ({"kind": "enum", "choices": ["draft", "final"]}, "draft"),+ ({"kind": "image", "concept_ref": "native.Image"}, "https://mock.invalid/url"),+ ({"kind": "document", "concept_ref": "native.Document"}, "https://mock.invalid/url"),+ ],+ )+ def test_a_leaf_slot_takes_its_kind_s_placeholder(self, node: dict[str, Any], expected: Any):+ slot = {**node, "name": "slot", "required": True}+ assert project_inputs_template(descriptor=_descriptor(slot), explicit=False)["slot"] == expected++ def test_an_enum_takes_its_first_choice_never_a_random_one(self):+ """The engine picks at random, which no committed template can carry."""+ slot = {"kind": "enum", "name": "status", "required": True, "choices": ["draft", "review", "final"]}+ descriptor = _descriptor(slot)+ renderings = {project_inputs_template(descriptor=descriptor, explicit=False)["status"] for _ in range(10)}+ assert renderings == {"draft"}+++class TestOutOfMatrixNatives:+ """The natives the runtime's shaper cannot rebuild from a bare value keep their envelope."""++ @pytest.mark.parametrize("native_ref", ["native.Page", "native.Html", "native.TextAndImages"])+ def test_an_out_of_matrix_native_keeps_its_envelope_in_the_compact_shape(self, native_ref: str):+ slot = {+ "kind": "object",+ "name": "payload",+ "concept_ref": native_ref,+ "required": True,+ "fields": [_text(name="inner")],+ }+ projected = project_inputs_template(descriptor=_descriptor(slot), explicit=False)["payload"]+ assert projected == {"concept": native_ref, "content": {"inner": "inner_value"}}++ def test_a_concept_refining_an_out_of_matrix_native_keeps_it_too(self):+ """The identity is read off the whole chain, not off `concept_ref` alone."""+ slot = {+ "kind": "object",+ "name": "payload",+ "concept_ref": "shelf.Rendered",+ "refines": ["native.Html"],+ "required": True,+ "fields": [_text(name="inner")],+ }+ projected = project_inputs_template(descriptor=_descriptor(slot), explicit=False)["payload"]+ assert projected == {"concept": "shelf.Rendered", "content": {"inner": "inner_value"}}++ def test_an_object_shaped_native_keeps_its_envelope_too(self):+ """`native.Date` renders as an object once its optional `time` is included, and the shaper+ dispatches a native's bare value on its scalar kind — so the object form only survives a+ round trip inside its envelope. Unwrapping it would pin a template that does not run.+ """+ date_slot = {+ "kind": "object",+ "name": "date_in",+ "concept_ref": "native.Date",+ "required": True,+ "fields": [+ {"kind": "date", "name": "date", "required": True, "datetime": False},+ {"kind": "text", "name": "time", "required": False, "format": "time"},+ ],+ }+ projected = project_inputs_template(descriptor=_descriptor(date_slot), explicit=False)["date_in"]+ assert projected == {"concept": "native.Date", "content": {"date": "2026-01-01", "time": "12:00:00"}}++ def test_a_nested_field_naming_a_native_holds_that_native_s_content_object(self):+ """A nested `native.Text` is a `TextContent`, so the payload carries `{"text": ...}` there —+ where an authored `type = "text"` field beside it carries the bare value. The descriptor+ states the difference in whether the node has a `concept_ref`, so the projection reads it.+ """+ page = {+ "kind": "object",+ "name": "page",+ "concept_ref": "native.Page",+ "required": True,+ "fields": [_native_text(name="body"), _text(name="caption")],+ }+ content = project_inputs_template(descriptor=_descriptor(page), explicit=False)["page"]["content"]+ assert content == {"body": {"text": "text_value"}, "caption": "caption_value"}++ def test_an_ordinary_structured_concept_unwraps_to_its_content_dict(self):+ slot = {+ "kind": "object",+ "name": "review",+ "concept_ref": "shelf.Review",+ "required": True,+ "fields": [_text(name="inner")],+ }+ assert project_inputs_template(descriptor=_descriptor(slot), explicit=False)["review"] == {"inner": "inner_value"}+++class TestConceptComments:+ """The `# concept: …` line a light TOML rendering carries, rebuilt from the descriptor alone."""++ def test_a_plain_slot_states_its_concept(self):+ assert project_concept_comments(descriptor=_descriptor(_native_text(name="note"))) == {"note": "concept: native.Text"}++ def test_a_fixed_count_slot_states_its_count(self):+ pair = {+ "kind": "list",+ "name": "pair",+ "concept_ref": "shelf.Bookmark",+ "required": True,+ "item_count": 2,+ "item": {"kind": "text", "required": True},+ }+ assert project_concept_comments(descriptor=_descriptor(pair)) == {"pair": "concept: shelf.Bookmark[2]"}++ def test_an_optional_variable_slot_states_both_markers(self):+ many = {+ "kind": "list",+ "name": "many",+ "concept_ref": "shelf.Bookmark",+ "required": False,+ "item": {"kind": "text", "required": True},+ }+ assert project_concept_comments(descriptor=_descriptor(many)) == {"many": "concept: shelf.Bookmark[]?"}diff --git a/uv.lock b/uv.lockindex 5dda15d4c..82a7e516c 100644--- a/uv.lock+++ b/uv.lock@@ -1916,15 +1916,15 @@ wheels = [[[package]]name = "kajson"-version = "0.7.0"+version = "0.7.1"source = { registry = "https://pypi.org/simple" }dependencies = [{ name = "pydantic" },{ name = "tzdata" },]-sdist = { url = "https://files.pythonhosted.org/packages/d2/a4/da91af6dac42516e45a9897ba901d8ec2ee2b13b135caf6ce120867aac2b/kajson-0.7.0.tar.gz", hash = "sha256:1c8f80d157f980c9adb9fda0e77c1cc0078f50547d84ef548a5e19b5a84d8d37", size = 25071, upload-time = "2026-06-10T10:00:56.097Z" }+sdist = { url = "https://files.pythonhosted.org/packages/5e/06/69bacab23fd509b2063489a4e4d3c44c6ce47133606a2d99d0f81cf17a47/kajson-0.7.1.tar.gz", hash = "sha256:725f5c02c065f829d4b4e409553b6016258784a77de2140246a96c851c64f362", size = 25057, upload-time = "2026-08-30T19:39:43.489Z" }wheels = [- { url = "https://files.pythonhosted.org/packages/f0/1c/8251c0dadd1843e020edf4ef162aaa7dd0e98cbd2e1fe33d843807cd7ce9/kajson-0.7.0-py3-none-any.whl", hash = "sha256:d9903327bc7f182ec481ffafd074765347365600592e5f9f67866d5b0ae4f939", size = 31524, upload-time = "2026-06-10T10:00:54.952Z" },+ { url = "https://files.pythonhosted.org/packages/0c/55/24cb6081dc828db12b4fd6c88dd67ed70f628cc8a4c9fc8d1aa7716c81fb/kajson-0.7.1-py3-none-any.whl", hash = "sha256:ba1249bbac58eaa8701a99f321c94edc103809e265506f7e9e270b9ef906126e", size = 31512, upload-time = "2026-08-30T19:39:42.179Z" },][[package]]@@ -3459,7 +3459,7 @@ requires-dist = [{ name = "jinja2", specifier = ">=3.1.4" },{ name = "json2html", specifier = ">=1.3.0" },{ name = "jsonschema", marker = "extra == 'dev'", specifier = ">=4.20.0" },- { name = "kajson", specifier = "==0.7.0" },+ { name = "kajson", specifier = "==0.7.1" },{ name = "linkup-sdk", marker = "extra == 'linkup'", specifier = ">=0.12.0" },{ name = "lxml", marker = "extra == 'docling'", specifier = ">=6.1.0" },{ name = "markdown", specifier = ">=3.6" },