Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix: derive native.JSON's input form from its pinned structure, and cover the open-shaped natives at slot positions (#1176)
Louis Choquelb00310e4f404
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 94ff4e6ab..50ad9a6e2 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -19,6 +19,7 @@### Fixed+- **A `native.JSON` input produced a fill-in template the runtime refused**: the input-form deriver grouped `native.JSON` with the three natives the standard calls structureless — `Dynamic`, `Anything` and `Composite` — so its descriptor node was `unknown`, which every inputs-template projection renders as the empty object. But `JSON` is not structureless: its pinned blueprint carries a required `json_obj`, and `JSONContent` refuses a payload without it, so the template a caller was handed for a `JSON` input could never be filled in and submitted. It now expands through the same pinned-structure arm as `Date`, `Html`, `Page`, `TextAndImages` and `SearchResult`, rendering `{"json_obj": {}}`, which the input shaper takes back. The engine's own `_pinned_structure` already drew the line in exactly this place — it returns a structure for `JSON` and `None` for the other three — so the deriver is now consistent with it, and the empty object stays the honest rendering for the natives that genuinely withhold their shape. The shared projection corpus gained a pipe covering all three of the natives it can reach at a slot position, which is where the defect had been sitting unmeasured. See [Projection Fixture Corpus](contribute/generate-projection-corpus.md).- **A bad pipeline-input url survives STRICT disclosure as caller-facing copy**: `PipelineInputContentError` declared `caller_facing_message = True`, which is the name of the *report field*, not of the class-level flag that sets it (`_authors_caller_facing_message`). The declaration therefore did nothing, and the message a caller who simply omitted a url needs to see — the one listing the accepted schemes — was replaced by the generic internal-error placeholder on any surface serializing in STRICT mode, exactly the redaction the declaration was written to prevent. Repairing the flag also revealed why the class could not carry it as it stood: its other raise site reports an unreadable local path, and that message names the resolved path and the `OSError` subclass that rejected it, which would have made STRICT answer "does this server-side path exist, and may the runner read it?" for any authenticated caller of a deployment leaving `is_upload_local_content_enabled` on. The family is therefore split: `PipelineInputUrlMissingError` carries the blank-url message and is caller-facing, while `PipelineInputContentError` keeps the unreadable-path message and stays redacted. Both are pinned in the class-level-metadata sweep, in both directions.- **The projection corpus's divergence gate sees every engine/projection difference**: `pipelex-dev generate-projection-corpus` states that every difference between the reference projection and the engine's own renderer lands in a declared class, and that an undeclared one fails the command — but the walk missed differences in one direction and absorbed regressions into existing classes in the other, so a projection defect could pass generation green and be committed into `mthds-js` and `mthds-python` as contract bytes. The dict walk iterated only the projected keys, so a field the projection stopped rendering was never compared at all; it is now recorded under an id deliberately absent from the declared reasons, which makes the record refuse rather than pass. The file-leaf arm returned on the expansion without ever comparing the URL both sides carry, where plain recursion would have handed a regressed placeholder to the url-named-text class that absorbs it; the shared keys are now compared strictly. And `fixed-count-honoured` fired on any list longer than the engine's single element, which cannot tell an honoured `Concept[N]` from a wrong count or from a variable `[]` slot wrongly duplicated; it now consults the slot's declared `item_count`, taken from the descriptor. Separately, a pipe declaring no inputs used to abort the whole capture — an empty input form is valid and the projection renders it as `{}`, but the engine's renderer refuses one — and is now captured from the projection alone, with no engine half and nothing compared. Re-measuring the same corpus produces bytes identical to the committed capture, with the same classes at the same counts. See [Projection Fixture Corpus](contribute/generate-projection-corpus.md).- **`--help` works again on every `method` subcommand**: `pipelex validate method`, `pipelex run method`, the three `pipelex build ... method` commands and their agent-CLI twins all crashed with a rich `MarkupError` traceback instead of printing their help. The method-address example in each command's `NAME` argument help — `github.com/owner/repo[/name][@tag]` — reaches rich's markup parser through typer, which read `[/name]` as a closing tag with no opening one. The brackets are escaped at each of the sites, so the example renders as written and the help exits zero.diff --git a/docs/contribute/generate-projection-corpus.md b/docs/contribute/generate-projection-corpus.mdindex ad50cf5b1..be08d3c3d 100644--- a/docs/contribute/generate-projection-corpus.md+++ b/docs/contribute/generate-projection-corpus.md@@ -40,8 +40,9 @@ The classes it declares today:- `fixed-count-honoured` — a `Concept[N]` slot renders `N` elements. The engine emits one whatever the count, and `InputShaper` then rejects that template with `MultiplicityCountMismatchError`, so the scaffold it produced does not run.- `text-named-url` — the engine picks a placeholder by field **name**, so a text field merely named `url` renders as a URL. The projection reads the descriptor's `kind`.- `object-native-keeps-envelope` — a native carrying an optional field beside its required one (`native.Date`) renders as an object once the optional field is included, and the shaper's bare-value arm dispatches a native on its scalar kind, so the object form is only re-shapable inside its `{concept, content}` envelope. The projection keeps that envelope; the engine unwraps to a bare scalar, which it can only do because it drops the optional field. A consequence of `optional-field-included`.+- `unknown-empty-object` — an `unknown` node renders as the empty object, because the descriptor withholds the payload shape at that position and a projection that invented one would stop projecting the descriptor. The engine reflects the runtime content class instead and fills a required dict with a sample key/value pair whoever fills the template in has to delete. Reached by `json_obj` inside `native.JSON`, the corpus's first *required* dict; the optional dict fields elsewhere never reach it, because the engine drops those entirely and that is `optional-field-included`.-Each is a `pipelex` defect filed in the workspace ledger, not a difference of taste; the manifest is what records that the corpus knowingly departs from the engine and why.+Most are `pipelex` defects filed in the workspace ledger rather than differences of taste, and the manifest names the item whose fix would retire each one. Some carry no item, deliberately: `file-leaf-not-expanded` and `unknown-empty-object` are the descriptor's vantage rather than engine bugs, and `object-native-keeps-envelope` is a consequence of `optional-field-included` rather than a defect of its own. The manifest is what records that the corpus knowingly departs from the engine, and why.### What the gate can and cannot separate@@ -68,7 +69,9 @@ The entries declared today are all one descriptor gap — a nested list inside a## The bundles-`tests/data/input_semantics/` holds the corpus bundles. `probe_bundle.mthds` exercises every construct the language accepts and `hinted_bundle.mthds` the intent hints; `scaffold_bundle.mthds` was added for this corpus and covers what the other two do not — a text field merely named `url` beside a real file position, an optional nested structure, optional `native.Image` and `native.Document` fields inside a structure, and both a fixed `[N]` and a variable `[]` slot over a structured concept.+`tests/data/input_semantics/` holds the corpus bundles. `probe_bundle.mthds` exercises every construct the language accepts and `hinted_bundle.mthds` the intent hints; `scaffold_bundle.mthds` was added for this corpus and covers what the other two do not — a text field merely named `url` beside a real file position, an optional nested structure, optional `native.Image` and `native.Document` fields inside a structure, and both a fixed `[N]` and a variable `[]` slot over a structured concept. Its `scaffold_open_natives` pipe holds the natives whose payload shape the descriptor states openly, at slot positions: `native.Dynamic` and `native.Composite`, which the standard calls structureless and whose node is therefore `unknown`, beside `native.JSON`, which is not — its pinned blueprint carries a required `json_obj`, so it expands like any other pinned native and its dict is where `unknown-empty-object` is measured.++`native.Anything` belongs in that pipe and is not there yet: an `Anything` input crashes the engine's contract builder, which resolves a structure class for every input and so asks for the one class the standard says does not exist. Its slot coverage waits on `L-260831-8f7c8c`.## When to rerun itdiff --git a/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.py b/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.pyindex ce2b7045e..c389b0a7f 100644--- a/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.py+++ b/pipelex/cli/dev_cli/commands/generate_projection_corpus_cmd.py@@ -102,6 +102,13 @@ DIVERGENCE_REASONS: dict[str, str] = {"The engine picks a placeholder by field name, so a text field merely named url or ending in _url ""renders as a URL. The projection reads the descriptor's kind instead."),+ "unknown-empty-object": (+ "An unknown node renders as the empty object, because the descriptor withholds the payload shape "+ "at that position and a projection that invented one would stop projecting the descriptor. The "+ "engine reflects the runtime content class instead and fills a required dict with a sample "+ "key/value pair whoever fills the template in has to delete. The empty object is re-shapable: the "+ "round-trip takes it back."+ ),"object-native-keeps-envelope": ("A native whose pinned definition carries an optional field beside its required one — native.Date — ""renders as an object once the optional field is included, and the shaper's bare-value arm dispatches "@@ -115,8 +122,9 @@ DIVERGENCE_REASONS: dict[str, str] = {# The workspace-ledger item tracking the engine fix, where the difference is a defect rather than a# difference of vantage. Named in the manifest so the corpus records not just that it departs from# the engine but what would retire the departure. A class with no entry here is deliberate on both-# sides: `file-leaf-not-expanded` is the descriptor's vantage rather than an engine bug, and-# `object-native-keeps-envelope` is a consequence of `optional-field-included` rather than its own.+# sides: `file-leaf-not-expanded` and `unknown-empty-object` are the descriptor's vantage rather than+# engine bugs, and `object-native-keeps-envelope` is a consequence of `optional-field-included`+# rather than its own.DIVERGENCE_ITEMS: dict[str, str] = {"text-named-url": "L-260830-dc48bf","fixed-count-honoured": "L-260830-f3de29",@@ -217,6 +225,19 @@ def _write_text(*, path: Path, content: str) -> None:path.write_text(content, encoding="utf-8")+def _engine_dict_placeholder(*, path: list[str]) -> dict[str, str]:+ """What the engine renders at a dict-typed field, which is exactly where the descriptor says `unknown`.++ `ConceptRepresentationGenerator._generate_dict_value` returns `{f"{name}_key": f"{name}_value"}`,+ and `generate_field_value` is its only caller — dispatched on `origin is dict`, the same test that+ makes the input-form deriver state the node as `unknown`. A dict inside a list never reaches it+ (`_generate_list_value` routes items through `_generate_basic_value`, which returns no dicts), so+ the placeholder is always keyed by the field's own name — the last segment of the walk's path.+ """+ field_name = path[-1]+ return {f"{field_name}_key": f"{field_name}_value"}++class DivergenceCollector:"""Walks the engine's template beside the projection's and buckets every difference.@@ -289,9 +310,22 @@ class DivergenceCollector:"""Bucket every leaf difference between one engine value and its projected counterpart."""# Handled before the dict/dict walk below, which would report the same thing one key at a# time: a projection rendering nothing where the engine renders an object is one fact, not N.- # Deliberately not declared in DIVERGENCE_REASONS: no bundle in the corpus reaches it today,- # so a capture that does must write the declaration rather than inherit one nobody reviewed.- if isinstance(engine_value, dict) and projected_value == {} and engine_value != {}:+ # Reached by the required dict inside `native.JSON` — `json_obj`, whose descriptor node is+ # `unknown` like every dict field, so the projection renders the empty object while the+ # engine fills it with a sample key. The corpus's optional dict fields never reach here:+ # the engine drops those entirely, which is `optional-field-included`.+ #+ # Recognised by the engine's own placeholder rather than by the empty object alone, because+ # `{}` is what the projection renders for an `object` node with no fields as much as for an+ # `unknown` one: a shape-only arm would file a projection that collapsed a populated object+ # into this declared class and return without ever walking the subtree — the whole-difference+ # swallow the file-leaf and fixed-count arms were repaired for. The placeholder is exact, not+ # a heuristic: `ConceptRepresentationGenerator._generate_dict_value` renders `{name_key:+ # name_value}` and is dispatched from one site only, on `origin is dict`, which is precisely+ # what makes the descriptor node `unknown`. A non-matching value falls through to the dict+ # walk below, where each key the projection dropped is `engine-only-field` and the capture+ # refuses.+ if isinstance(engine_value, dict) and projected_value == {} and engine_value == _engine_dict_placeholder(path=path):self._record(divergence_id="unknown-empty-object", path=path, engine=engine_value, expected=projected_value)returnif (@@ -318,9 +352,9 @@ class DivergenceCollector:projected_fields = cast("dict[str, Any]", projected_value)# Walked before the projected keys, because the walk below iterates those and would# otherwise skip a field the projection stopped rendering entirely — a projection- # regression that reached neither a class nor `unclassified`. Deliberately not declared- # in DIVERGENCE_REASONS, like `unknown-empty-object` above: no bundle in the corpus- # reaches it today, so a capture that does must write the declaration itself.+ # regression that reached neither a class nor `unclassified`. Deliberately absent from+ # DIVERGENCE_REASONS: no bundle in the corpus reaches it today, so a capture that does+ # must write the declaration itself rather than inherit one nobody reviewed.for key in engine_fields:if key not in projected_fields:self._record(divergence_id="engine-only-field", path=[*path, key], engine=engine_fields[key], expected=None)diff --git a/pipelex/pipeline/input_form.py b/pipelex/pipeline/input_form.pyindex 9c04337af..b3abe4aa9 100644--- a/pipelex/pipeline/input_form.py+++ b/pipelex/pipeline/input_form.py@@ -336,6 +336,7 @@ class InputFormDeriver:case (NativeConceptCode.DATE| NativeConceptCode.HTML+ | NativeConceptCode.JSON| NativeConceptCode.PAGE| NativeConceptCode.TEXT_AND_IMAGES| NativeConceptCode.SEARCH_RESULT@@ -349,7 +350,12 @@ class InputFormDeriver:required=True,fields=[self._structure_field(name=field_name, field=field, seen=seen) for field_name, field in pinned_structure.items()],)- case NativeConceptCode.DYNAMIC | NativeConceptCode.ANYTHING | NativeConceptCode.JSON | NativeConceptCode.COMPOSITE:+ case NativeConceptCode.DYNAMIC | NativeConceptCode.ANYTHING | NativeConceptCode.COMPOSITE:+ # The three the standard calls structureless: `_pinned_structure` returns None for+ # exactly these, so there is no authored shape to expand and `unknown` states that+ # honestly. Every other native belongs on an arm above — `native.JSON` sat here+ # until its pinned `json_obj` was noticed, and a template describing it as unknown+ # rendered `{}`, which `JSONContent` then refused.return UnknownField(name=name, concept_ref=node_ref, refines=refines, description=text, required=True)def _class_backed_node(diff --git a/tests/data/input_semantics/scaffold_bundle.mthds b/tests/data/input_semantics/scaffold_bundle.mthdsindex 4b53e7ddb..6a011318e 100644--- a/tests/data/input_semantics/scaffold_bundle.mthds+++ b/tests/data/input_semantics/scaffold_bundle.mthds@@ -51,3 +51,16 @@ Exactly two dossiers:Any number of bookmarks:@bookmarks"""++# The natives whose payload shape the descriptor states openly, at slot positions — the one place+# the corpus never reached. `native.Dynamic` and `native.Composite` are structureless by design, so+# their node is `unknown` and every projection renders the empty object both renderers agree on.+# `native.JSON` is not: its pinned blueprint carries a required `json_obj`, so it expands like any+# other pinned native, and the dict inside it is the corpus's first REQUIRED unknown node — the+# engine invents a sample key there, the projection leaves the object empty for whoever fills it in.+[pipe.scaffold_open_natives]+type = "PipeCompose"+description = "SCAFFOLD_pipe_scaffold_open_natives: a pinned-dict native beside the two structureless ones, each at a slot position"+inputs = { json_in = "JSON", dynamic_in = "Dynamic", composite_in = "Composite" }+output = "Text"+template = "SCAFFOLD_template_open_natives: $json_in $dynamic_in $composite_in"diff --git a/tests/integration/pipelex/pipeline/test_input_form.py b/tests/integration/pipelex/pipeline/test_input_form.pyindex 4ecf6a7f9..69026b2f6 100644--- a/tests/integration/pipelex/pipeline/test_input_form.py+++ b/tests/integration/pipelex/pipeline/test_input_form.py@@ -608,9 +608,14 @@ class TestKindAssignmentTable:search_result = _field_by_name(natives, "search_result_in")assert search_result.kind == FieldKind.OBJECTassert search_result.fields is not None+ json_field = _field_by_name(natives, "json_in")+ assert json_field.kind == FieldKind.OBJECT, "native.JSON has a pinned structure, so it expands like the other pinned natives"+ assert json_field.fields is not None+ assert [field.name for field in json_field.fields] == ["json_obj"], "Its pinned blueprint's single required field"+ # The three the standard calls structureless — `_pinned_structure` returns None for exactly+ # these, so `unknown` is the honest kind and no other native may join them.assert _field_by_name(natives, "dynamic_in").kind == FieldKind.UNKNOWNassert _field_by_name(natives, "anything_in").kind == FieldKind.UNKNOWN- assert _field_by_name(natives, "json_in").kind == FieldKind.UNKNOWNfor field in natives.fields:assert field.concept_ref is not Noneassert field.concept_ref.startswith("native.")diff --git a/tests/unit/pipelex/cli/dev_cli/test_projection_divergence_gate.py b/tests/unit/pipelex/cli/dev_cli/test_projection_divergence_gate.pyindex ecf094385..ae7c51e9e 100644--- a/tests/unit/pipelex/cli/dev_cli/test_projection_divergence_gate.py+++ b/tests/unit/pipelex/cli/dev_cli/test_projection_divergence_gate.py@@ -125,3 +125,33 @@ class TestTheDivergenceGateSeesEveryDifference:assert collector.counts == {"fixed-count-honoured": 1}assert not collector.unclassified++ def test_a_structured_object_collapsed_to_nothing_is_not_absorbed(self) -> None:+ """`unknown-empty-object` means the engine's placeholder for a dict, not any object gone empty."""+ collector = DivergenceCollector()++ collector.compare(+ engine_value={"title": "title_value", "stamp": "stamp_value"},+ projected_value={},+ path=["scaffold.scaffold_nesting", COMPACT_SHAPE, "dossier"],+ )++ assert "unknown-empty-object" not in collector.counts+ # It falls through to the dict walk, where every key the projection stopped rendering is+ # `engine-only-field` — undeclared on purpose, so the whole capture refuses.+ assert collector.counts.get("engine-only-field") == 2+ with pytest.raises(ValueError, match="Undeclared divergence class"):+ collector.declared()++ def test_an_unknown_node_keeps_its_class(self) -> None:+ """The control: the engine's dict placeholder against the empty object the descriptor states."""+ collector = DivergenceCollector()++ collector.compare(+ engine_value={"json_obj_key": "json_obj_value"},+ projected_value={},+ path=["scaffold.scaffold_open_natives", COMPACT_SHAPE, "json_in", "json_obj"],+ )++ assert collector.counts == {"unknown-empty-object": 1}+ assert not collector.unclassified