Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix: reflect class-backed concepts field by field in the input-form descriptor (#1166)

Louis Choquel4def57b50239
diff --git a/.badges/tests.json b/.badges/tests.json
index 79c245966..fdd00b0fa 100644
--- a/.badges/tests.json
+++ b/.badges/tests.json
@@ -1,7 +1,7 @@
{
"schemaVersion": 1,
"label": "tests",
- "message": "11096",
+ "message": "11140",
"color": "blue",
"cacheSeconds": 300
}
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 486c8f353..9c55c3e0e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,7 @@
### Fixed
+- **Class-backed concepts reflect field by field in the input-form descriptor**: a concept declaring `structure = "SomeClass"` used to collapse to `kind: "unknown"` the moment one field annotation could not be mapped, taking every sibling field with it — and a nested non-native pydantic model was reported as an opaque payload rather than walked into. Every `document` and `image` position beneath such a concept disappeared from the descriptor, so a consumer preparing inputs from it passed a local file path at one of those positions through un-uploaded and the run failed on the unresolved path. Reflection is now partial: each annotation is mapped on its own, an unmappable one is `unknown` at that field alone, a nested model is an `object` the reflection walks into, a `list[X]` reports its element node, and a class that holds itself is cut at the revisit instead of recursing. A `RootModel` is reported as its root annotation's node wherever it appears — as a field of a reflected class, and as the registered class backing the concept itself — because the value it accepts is the root value, so the descriptor no longer states an object over a synthetic `root` key that contradicts the `json_schema` derived beside it. The native consistency probe keeps its faithful-or-absent rule — it is compared against a normative pinned blueprint, where a plausible-looking partial answer would be the failure — and the two reflections now share one annotation vocabulary (`pipelex/core/concepts/annotation_shapes.py`) instead of one calling the other. See [Input-Form Descriptor](under-the-hood/input-form-descriptor.md).
- **Generated-image storage key extension follows the actual mime type**: a generated image could be stored under a `.png` key while its bytes and reported `mime_type` were `image/jpeg`, because `GeneratedContentFactory` picked the key's extension from the *requested* image format while `mime_type` came from the provider's *actual* answer — and a provider may return a different format than requested. The factory now resolves the effective mime type once (provider-reported, then base64-extracted, then the requested format, then the `image/jpeg` default) and derives the storage-key extension from that same value, on every path (stored bytes and fetched remote content alike), so the key and the `mime_type` can no longer diverge and anything inferring a type from the extension sees the truth.
## [v0.55.0] - 2026-08-29
diff --git a/docs/under-the-hood/input-form-descriptor.md b/docs/under-the-hood/input-form-descriptor.md
index 85afcefe1..609810699 100644
--- a/docs/under-the-hood/input-form-descriptor.md
+++ b/docs/under-the-hood/input-form-descriptor.md
@@ -28,7 +28,7 @@ The derivation is total. A node it cannot map honestly reports `kind: "unknown"`
Kinds are decided by chain membership and declared types — never by sniffing a schema shape. Per node, in precedence order:
1. A concept with an authored structure *table* anywhere along its refinement chain is an `object`; its fields are the merged structures along the chain, base fields first, a refining concept overriding its parents'. The table decides even when it is empty — the engine backs an empty `[concept.X.structure]` with a field-less structured model, not with `TextContent`, so the descriptor reports `object` with an empty `fields` list rather than falling through to rule 4.
-2. Otherwise, the first `structure = "ClassName"` on the chain decides: a native class name (`TextContent`, `ImageContent`, …) maps by identity to that native's row; any other registered class is reflected field by field into an `object`; an unregistered or unmappable class is `unknown`.
+2. Otherwise, the first `structure = "ClassName"` on the chain decides: a native class name (`TextContent`, `ImageContent`, …) maps by identity to that native's row; any other registered class is reflected field by field into an `object`; a class the registry does not hold — or one that is not a pydantic model — is `unknown`.
3. Otherwise, a chain bottoming at a native concept takes that native's row, keeping the concept's own `concept_ref`, description and `refines`.
4. Otherwise — a description-only or string-described concept — `prose`, with `refines` carrying the authored chain alone (absent when nothing was authored): a producer states only the links it holds, so no `native.Text` link is reconstructed — text-valuedness reaches the wire as the `prose` kind itself.
@@ -49,6 +49,8 @@ The table and rules above ARE the no-hint kind assignment — stated rules, not
Nested structure fields map by their declared type: `text` → `text`; `integer` → `number` with `integer: true`; `number` → `number`; `boolean` → `boolean`; `date` → `date`; `datetime` → `date` with `datetime: true`; `time` → `text` with `format: "time"`; a field with `choices` → `enum` (choices win over `type`, matching the structure generator); `concept` → the concept's node, carrying its namespaced `concept_ref`; `list` → `list` whose `item` comes from `item_type` / `item_concept_ref` (a nested list's inner item is inexpressible and reports `unknown`; the `item` node carries no `name` member — a list's item has no authored name, and the index labels entries); `dict` → `unknown`. The shorthand `field = "description"` form is a required `text`. Nested fields take the blueprint field's description and `required` over the concept's; a scalar flattened at the top level keeps the concept's description.
+A reflected class's fields map by their annotation, and each annotation is mapped **on its own**: a scalar takes its kind from the type, a native content class (`ImageContent`, `DocumentContent`, …) takes that native's row by identity, a `list[X]` is a `list` whose `item` is `X` reflected one layer down, a nested non-native pydantic model is an `object` the reflection walks into, a `RootModel` is its root annotation's node (the value it accepts is the root value itself, never an object over a `root` key), and an annotation with no honest node — a union that is neither `X | None` nor `int | float`, a `dict`, a class already on the path — is `unknown` at **that field**, leaving its siblings stated. Partiality is the point: collapsing the whole payload the moment one annotation could not be mapped hid every `document` and `image` position under the class, and a consumer preparing inputs from the descriptor then passed a local file path through un-uploaded. A class declaring no field is an `object` with an empty `fields` list — a payload that demands nothing, the class-backed twin of an empty authored structure table. A registered class that IS a `RootModel` is read the same way at the top: the concept's node is its root annotation's, carrying the concept's own `concept_ref`, `refines` and description — so a root-valued concept agrees with the `json_schema` derived from that same class beside it, instead of describing an object over a `root` key that `model_validate` would reject. This is deliberately unlike the reflection the native consistency probe performs (`pipelex/codegen/native_expansion.py`), which stays faithful-or-absent because it is compared against a normative pinned blueprint, where a plausible-looking partial answer would be the failure.
+
Constraint slots (`minimum`, `exclusive_minimum`, `min_length`, `pattern`, …) come only from reflected classes: the language cannot author constraint keys (an authored `minimum = 0` is rejected as an unknown structure-field key), whereas a registered class's `Field(gt=0, max_length=8, pattern=...)` is read from the pydantic field metadata and stamped on the matching `number` or `text` node. A pydantic default on a reflected class is an authored fact: `field_info.is_required()` decides the node's `required` and a non-`None` default is reported as its `default_value` — a defaulted field is never required, the same invariant blueprint validation enforces by rejecting the `required = true` + `default_value` pair. (Authored `hints` are a parsed blueprint field with its own leniency rules; see below.)
## Hints
@@ -67,4 +69,4 @@ Inapplicable slots are absent, never JSON `null`: the report's valid arm is dump
## Seeing it
-`pipelex-dev trace-input-semantics` captures the descriptors as `hop5_input_form.json` beside `hop5_pipe_io_contracts.json`, so an authored fact can be checked on both projections at once — see [Tracing Input Semantics](../contribute/trace-input-semantics.md). The assignment table is pinned by `tests/integration/pipelex/pipeline/test_input_form.py` over the committed probe bundle (and the hints behavior over its hinted sibling, `hinted_bundle.mthds`), the emitted wire's null-free shape by `tests/unit/pipelex/pipeline/test_input_form_models.py`, the hint merges and kind-feeding by `tests/unit/pipelex/pipeline/test_input_form_hints.py`, and the escape hatches the library loader keeps unreachable (concept cycles, unregistered classes) by `tests/unit/pipelex/pipeline/test_input_form_deriver.py`.
+`pipelex-dev trace-input-semantics` captures the descriptors as `hop5_input_form.json` beside `hop5_pipe_io_contracts.json`, so an authored fact can be checked on both projections at once — see [Tracing Input Semantics](../contribute/trace-input-semantics.md). The assignment table is pinned by `tests/integration/pipelex/pipeline/test_input_form.py` over the committed probe bundle (and the hints behavior over its hinted sibling, `hinted_bundle.mthds`), the emitted wire's null-free shape by `tests/unit/pipelex/pipeline/test_input_form_models.py`, the hint merges and kind-feeding by `tests/unit/pipelex/pipeline/test_input_form_hints.py`, and the escape hatches the library loader keeps unreachable (concept cycles, unregistered classes, a structure class that holds itself) by `tests/unit/pipelex/pipeline/test_input_form_deriver.py`.
diff --git a/pipelex/codegen/native_expansion.py b/pipelex/codegen/native_expansion.py
index 5bea43011..5b583e97c 100644
--- a/pipelex/codegen/native_expansion.py
+++ b/pipelex/codegen/native_expansion.py
@@ -19,19 +19,22 @@ annotation has no honest blueprint form at all (e.g. a non-Optional union) makes
reflected structure absent.
"""
-import datetime
-from types import UnionType
-from typing import Any, Union, cast, get_args, get_origin
+from typing import Any, cast, get_origin
from pydantic import BaseModel
+from pipelex.core.concepts.annotation_shapes import (
+ is_number_union,
+ is_union,
+ list_item_annotation,
+ native_code_for_content_class,
+ scalar_field_type,
+ strip_optional,
+)
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint, ConceptStructureBlueprintType
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.concepts.native.pinned_blueprints import make_pinned_native_blueprint
-from pipelex.core.stuffs.stuff_content import StuffContent
-
-_NATIVE_CLASS_NAME_TO_CODE: dict[str, NativeConceptCode] = {native_code.structure_class_name: native_code for native_code in NativeConceptCode}
class _UnmappableAnnotationError(Exception):
@@ -58,16 +61,18 @@ def reflect_native_structure(native_code: NativeConceptCode) -> dict[str, Concep
structure_class = native_code.structure_class
if structure_class is None:
return None
- return reflect_structure_class(structure_class=cast("type[BaseModel]", structure_class))
+ return _reflect_structure_class(structure_class=cast("type[BaseModel]", structure_class))
-def reflect_structure_class(*, structure_class: type[BaseModel]) -> dict[str, ConceptStructureBlueprintType] | None:
- """Reflect a registered structure class into blueprint form, faithful-or-absent.
+def _reflect_structure_class(*, structure_class: type[BaseModel]) -> dict[str, ConceptStructureBlueprintType] | None:
+ """Reflect a content class into blueprint form, faithful-or-absent.
- Same rules as native reflection: every field must map unambiguously to a blueprint field, or
- the whole reflected structure is absent (`None`) — never a guessed shape. The input-form
- deriver uses this for class-backed concepts (`structure = "ClassName"`), where the class is
- the only statement of the payload's fields.
+ Every field must map unambiguously to a blueprint field, or the whole reflected structure is
+ absent (`None`) — never a guessed shape, and never a partial one. That severity is what makes
+ this a probe worth trusting: the pinned blueprint it is compared against is the normative form,
+ so a structure reflected from a class that drifted must not look plausible. The input-form
+ deriver reflects a class-backed concept for a different purpose and answers differently — see
+ `InputFormDeriver._reflected_class_fields`.
"""
model_fields = structure_class.model_fields
if not model_fields:
@@ -83,11 +88,11 @@ def reflect_structure_class(*, structure_class: type[BaseModel]) -> dict[str, Co
def _annotation_to_blueprint(annotation: Any, *, description: str) -> ConceptStructureBlueprint:
- inner, required = _strip_optional(annotation)
+ inner, required = strip_optional(annotation=annotation)
- if _is_number_union(inner):
+ if is_number_union(annotation=inner):
return ConceptStructureBlueprint(description=description, type=ConceptStructureBlueprintFieldType.NUMBER, required=required)
- if get_origin(inner) in {Union, UnionType}:
+ if is_union(annotation=inner):
# A union that is not simply Optional[X] nor a number union has no single blueprint shape.
raise _UnmappableAnnotationError
@@ -99,11 +104,11 @@ def _annotation_to_blueprint(annotation: Any, *, description: str) -> ConceptStr
if origin is list:
return _list_blueprint(inner, description=description, required=required)
- scalar_type = _scalar_field_type(inner)
+ scalar_type = scalar_field_type(annotation=inner)
if scalar_type is not None:
return ConceptStructureBlueprint(description=description, type=scalar_type, required=required)
- native_code = _native_code_for_content_class(inner)
+ native_code = native_code_for_content_class(annotation=inner)
if native_code is not None:
return ConceptStructureBlueprint(
description=description, type=ConceptStructureBlueprintFieldType.CONCEPT, concept_ref=native_code.concept_ref, required=required
@@ -121,12 +126,11 @@ def _annotation_to_blueprint(annotation: Any, *, description: str) -> ConceptStr
def _list_blueprint(list_annotation: Any, *, description: str, required: bool) -> ConceptStructureBlueprint:
- args = get_args(list_annotation)
- if not args:
+ item_inner = list_item_annotation(annotation=list_annotation)
+ if item_inner is None:
raise _UnmappableAnnotationError
- item_inner, _ = _strip_optional(args[0])
- native_code = _native_code_for_content_class(item_inner)
+ native_code = native_code_for_content_class(annotation=item_inner)
if native_code is not None:
return ConceptStructureBlueprint(
description=description,
@@ -135,7 +139,7 @@ def _list_blueprint(list_annotation: Any, *, description: str, required: bool) -
item_concept_ref=native_code.concept_ref,
required=required,
)
- scalar_type = _scalar_field_type(item_inner)
+ scalar_type = scalar_field_type(annotation=item_inner)
if scalar_type is not None:
return ConceptStructureBlueprint(
description=description, type=ConceptStructureBlueprintFieldType.LIST, item_type=scalar_type, required=required
@@ -143,47 +147,6 @@ def _list_blueprint(list_annotation: Any, *, description: str, required: bool) -
raise _UnmappableAnnotationError
-def _strip_optional(annotation: Any) -> tuple[Any, bool]:
- """Return (inner, required): peel a single `X | None` into (X, False); otherwise (annotation, True)."""
- if get_origin(annotation) in {Union, UnionType}:
- args = get_args(annotation)
- non_none = [arg for arg in args if arg is not type(None)]
- if len(non_none) == 1 and len(non_none) != len(args):
- return non_none[0], False
- return annotation, True
-
-
-def _is_number_union(annotation: Any) -> bool:
- if get_origin(annotation) not in {Union, UnionType}:
- return False
- return set(get_args(annotation)) == {int, float}
-
-
-def _scalar_field_type(annotation: Any) -> ConceptStructureBlueprintFieldType | None:
- # Order matters: bool is an int subclass, datetime is a date subclass — check the narrower first.
- if annotation is bool:
- return ConceptStructureBlueprintFieldType.BOOLEAN
- if annotation is int:
- return ConceptStructureBlueprintFieldType.INTEGER
- if annotation is float:
- return ConceptStructureBlueprintFieldType.NUMBER
- if annotation is str:
- return ConceptStructureBlueprintFieldType.TEXT
- if annotation is datetime.datetime:
- return ConceptStructureBlueprintFieldType.DATETIME
- if annotation is datetime.date:
- return ConceptStructureBlueprintFieldType.DATE
- if annotation is datetime.time:
- return ConceptStructureBlueprintFieldType.TIME
- return None
-
-
-def _native_code_for_content_class(annotation: Any) -> NativeConceptCode | None:
- if not isinstance(annotation, type) or not issubclass(annotation, StuffContent):
- return None
- return _NATIVE_CLASS_NAME_TO_CODE.get(annotation.__name__)
-
-
def _is_nested_model(annotation: Any) -> bool:
"""A pydantic model that is not itself a native content class — an object on the wire."""
return isinstance(annotation, type) and issubclass(annotation, BaseModel)
diff --git a/pipelex/core/concepts/annotation_shapes.py b/pipelex/core/concepts/annotation_shapes.py
new file mode 100644
index 000000000..5c3282a19
--- /dev/null
+++ b/pipelex/core/concepts/annotation_shapes.py
@@ -0,0 +1,78 @@
+"""How a Python type annotation maps onto the MTHDS shape vocabulary.
+
+Two reflections in this engine read registered pydantic classes and ask an annotation the same
+questions: the native consistency probe (`pipelex/codegen/native_expansion.py`), which reflects a
+native's content class into blueprint form, and the input-form deriver
+(`pipelex/pipeline/input_form.py`), which reflects a class-backed concept into descriptor nodes.
+What they build differs, and so does what they do with an annotation neither can map — the probe
+answers absent, the deriver emits an `unknown` node — but the questions themselves live here, so
+the two reflections cannot come to different answers about the same annotation.
+"""
+
+import datetime
+from types import UnionType
+from typing import Any, Union, get_args, get_origin
+
+from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprintFieldType
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.stuffs.stuff_content import StuffContent
+
+_NATIVE_CLASS_NAME_TO_CODE: dict[str, NativeConceptCode] = {native_code.structure_class_name: native_code for native_code in NativeConceptCode}
+
+
+def strip_optional(*, annotation: Any) -> tuple[Any, bool]:
+ """Return (inner, required): peel a single `X | None` into (X, False); otherwise (annotation, True)."""
+ if get_origin(annotation) in {Union, UnionType}:
+ args = get_args(annotation)
+ non_none = [arg for arg in args if arg is not type(None)]
+ if len(non_none) == 1 and len(non_none) != len(args):
+ return non_none[0], False
+ return annotation, True
+
+
+def is_union(*, annotation: Any) -> bool:
+ """Whether the annotation is a union — asked after `strip_optional`, so `X | None` is already gone."""
+ return get_origin(annotation) in {Union, UnionType}
+
+
+def is_number_union(*, annotation: Any) -> bool:
+ """Whether the annotation is exactly `int | float`, the one union with a single MTHDS shape."""
+ if not is_union(annotation=annotation):
+ return False
+ return set(get_args(annotation)) == {int, float}
+
+
+def scalar_field_type(*, annotation: Any) -> ConceptStructureBlueprintFieldType | None:
+ """The scalar MTHDS field type of the annotation, or `None` when it is not a scalar."""
+ # Order matters: bool is an int subclass, datetime is a date subclass — check the narrower first.
+ if annotation is bool:
+ return ConceptStructureBlueprintFieldType.BOOLEAN
+ if annotation is int:
+ return ConceptStructureBlueprintFieldType.INTEGER
+ if annotation is float:
+ return ConceptStructureBlueprintFieldType.NUMBER
+ if annotation is str:
+ return ConceptStructureBlueprintFieldType.TEXT
+ if annotation is datetime.datetime:
+ return ConceptStructureBlueprintFieldType.DATETIME
+ if annotation is datetime.date:
+ return ConceptStructureBlueprintFieldType.DATE
+ if annotation is datetime.time:
+ return ConceptStructureBlueprintFieldType.TIME
+ return None
+
+
+def native_code_for_content_class(*, annotation: Any) -> NativeConceptCode | None:
+ """The native concept a content class stands for, decided by identity — never by shape."""
+ if not isinstance(annotation, type) or not issubclass(annotation, StuffContent):
+ return None
+ return _NATIVE_CLASS_NAME_TO_CODE.get(annotation.__name__)
+
+
+def list_item_annotation(*, annotation: Any) -> Any:
+ """The element annotation of a list, already peeled of an `X | None` wrapper; `None` when unparameterized."""
+ args = get_args(annotation)
+ if not args:
+ return None
+ inner, _ = strip_optional(annotation=args[0])
+ return inner
diff --git a/pipelex/pipeline/input_form.py b/pipelex/pipeline/input_form.py
index fcd925688..9c04337af 100644
--- a/pipelex/pipeline/input_form.py
+++ b/pipelex/pipeline/input_form.py
@@ -40,7 +40,7 @@ registry, and bundle-defined classes are only reliably current while their libra
"""
from collections.abc import Mapping, Sequence
-from typing import Any
+from typing import Any, get_origin
from annotated_types import Ge, Gt, Le, Lt, MaxLen, MinLen
from mthds.protocol.input_form import (
@@ -78,7 +78,14 @@ from pydantic import BaseModel, TypeAdapter
from pydantic.fields import FieldInfo
from pydantic_core import PydanticUndefined
-from pipelex.codegen.native_expansion import reflect_structure_class
+from pipelex.core.concepts.annotation_shapes import (
+ is_number_union,
+ is_union,
+ list_item_annotation,
+ native_code_for_content_class,
+ scalar_field_type,
+ strip_optional,
+)
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint, ConceptStructureBlueprintType
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
from pipelex.core.concepts.native.concept_native import NativeConceptCode
@@ -355,30 +362,134 @@ class InputFormDeriver:
refines: list[str] | None,
seen: frozenset[str],
) -> InputFormField:
- """A concept whose payload is stated by a registered class: a native class maps by identity, any other is reflected."""
+ """A concept whose payload is stated by a registered class.
+
+ A native class maps by identity, a `RootModel` IS its root value, any other class is reflected.
+ """
native_code = next((code for code in NativeConceptCode if code.structure_class_name == class_name), None)
if native_code is not None:
return self._native_node(name=name, native_code=native_code, concept_ref=concept_ref, description=description, refines=refines, seen=seen)
structure_class = get_class_registry().get_class(name=class_name)
if not (isinstance(structure_class, type) and issubclass(structure_class, BaseModel)):
return _unknown_node(name=name, concept_ref=concept_ref, description=description, refines=refines)
- reflected = reflect_structure_class(structure_class=structure_class)
- if reflected is None:
- if structure_class.model_fields:
- # A field this reflection could not map: the payload's shape is genuinely opaque here.
- return _unknown_node(name=name, concept_ref=concept_ref, description=description, refines=refines)
- # A class declaring no field at all is not opaque — it states a payload that demands
- # nothing, the class-backed twin of an empty authored structure table, and the form says
- # so. (`reflect_structure_class` collapses both into `None` because its other consumer,
- # the native consistency probe, reads `None` as "structureless by design".)
- reflected = {}
- fields = [
+ if structure_class.__pydantic_root_model__:
+ # A `RootModel` IS its root value on the wire at the top of a class-backed concept exactly
+ # as it is one field down (`_reflected_node`): an `object` over the synthetic `root` field
+ # would state a shape `model_validate` rejects, and would contradict the `json_schema`
+ # derived from the same class beside it. The concept's own identity is stamped back on the
+ # node it yields — including on a `list` root, whose `concept_ref` names this concept
+ # rather than an element concept, there being none: a top-level field states the concept
+ # it carries, and a root-valued concept IS the whole value.
+ root_node = self._reflected_node(
+ name=name,
+ annotation=_root_annotation_of(root_model_class=structure_class),
+ seen=seen,
+ classes_seen=frozenset({structure_class}),
+ )
+ return root_node.model_copy(update={"concept_ref": concept_ref, "refines": refines, "description": description or root_node.description})
+ return ObjectField(
+ name=name,
+ concept_ref=concept_ref,
+ refines=refines,
+ description=description,
+ required=True,
+ fields=self._reflected_class_fields(structure_class=structure_class, seen=seen, classes_seen=frozenset()),
+ )
+
+ # ---- Class reflection -------------------------------------------------------------------------
+
+ def _reflected_class_fields(
+ self, *, structure_class: type[BaseModel], seen: frozenset[str], classes_seen: frozenset[type[BaseModel]]
+ ) -> list[InputFormField]:
+ """One node per field the class declares, every annotation mapped on its own.
+
+ Reflection here is **partial**, deliberately unlike the native consistency probe's
+ faithful-or-absent rule (`native_expansion._reflect_structure_class`): an annotation with no
+ honest node makes THAT field `unknown` and leaves its siblings stated. The two answers differ
+ because the purposes do — the probe is compared against a normative pinned form,
+ where a plausible-looking partial answer would be the failure, whereas the descriptor is
+ total by contract. Collapsing the whole payload here hid every `document` and `image`
+ position under a class that one sibling annotation happened to defeat, and a consumer
+ preparing inputs from the descriptor would then pass a local file path through un-uploaded.
+
+ A class declaring no field yields no nodes, and the `object` node built over them says so
+ with an empty `fields` list: a payload that demands nothing — the class-backed twin of an
+ empty authored structure table, not an opaque payload.
+ """
+ classes_seen |= {structure_class}
+ return [
_with_reflected_constraints(
- node=self._structure_field(name=field_name, field=field, seen=seen), field_info=structure_class.model_fields[field_name]
+ node=self._reflected_field(
+ name=field_name,
+ annotation=field_info.annotation,
+ description=field_info.description or field_name,
+ seen=seen,
+ classes_seen=classes_seen,
+ ),
+ field_info=field_info,
)
- for field_name, field in reflected.items()
+ for field_name, field_info in structure_class.model_fields.items()
]
- return ObjectField(name=name, concept_ref=concept_ref, refines=refines, description=description, required=True, fields=fields)
+
+ def _reflected_field(
+ self, *, name: str, annotation: Any, description: str, seen: frozenset[str], classes_seen: frozenset[type[BaseModel]]
+ ) -> InputFormField:
+ """One reflected field: the annotation's node, carrying the field's own description and optionality."""
+ inner, required = strip_optional(annotation=annotation)
+ node = self._reflected_node(name=name, annotation=inner, seen=seen, classes_seen=classes_seen)
+ return node.model_copy(update={"description": description, "required": required})
+
+ def _reflected_node(self, *, name: str, annotation: Any, seen: frozenset[str], classes_seen: frozenset[type[BaseModel]]) -> InputFormField:
+ """The node one annotation maps to, before the field's own facts are stamped on it.
+
+ The shape questions are the blueprint reflection's own (`core/concepts/annotation_shapes.py`),
+ answered in descriptor terms: a native content class is that native's node — routed through
+ `_concept_node`, so the concept cycle guard covers a pinned native's own fields — a nested
+ non-native model is an `object` whose fields are reflected in turn, which is what keeps a
+ file-bearing field one level down visible, a `RootModel` is its root annotation's node
+ because the value it accepts is the root value itself, and an annotation with no honest node
+ is `unknown`.
+ """
+ if is_number_union(annotation=annotation):
+ return NumberField(name=name, required=True, integer=False)
+ if is_union(annotation=annotation):
+ # A union that is neither `X | None` (already peeled) nor a number union has no single node shape.
+ return _unknown_node(name=name)
+ origin = get_origin(annotation)
+ if origin is list:
+ return self._reflected_list_node(name=name, annotation=annotation, seen=seen, classes_seen=classes_seen)
+ if origin is dict:
+ # A mapping with unspecified value types, exactly as a `dict` structure field reports.
+ return _unknown_node(name=name)
+ scalar_type = scalar_field_type(annotation=annotation)
+ if scalar_type is not None:
+ return _scalar_field(field_type=scalar_type, name=name, required=True)
+ native_code = native_code_for_content_class(annotation=annotation)
+ if native_code is not None:
+ return self._concept_node(name=name, concept_ref=native_code.concept_ref, seen=seen)
+ if isinstance(annotation, type) and issubclass(annotation, BaseModel) and annotation not in classes_seen:
+ # A nested model already on the path would recurse forever; the revisit is `unknown`,
+ # the same answer the concept walk gives a concept ref it has already seen.
+ nested_class: type[BaseModel] = annotation
+ deeper: frozenset[type[BaseModel]] = classes_seen | {nested_class}
+ if nested_class.__pydantic_root_model__:
+ # A `RootModel` IS its root value on the wire, so its node is the root annotation's:
+ # an `object` over the synthetic `root` field would state a shape `model_validate` rejects.
+ return self._reflected_node(name=name, annotation=_root_annotation_of(root_model_class=nested_class), seen=seen, classes_seen=deeper)
+ return ObjectField(
+ name=name, required=True, fields=self._reflected_class_fields(structure_class=nested_class, seen=seen, classes_seen=classes_seen)
+ )
+ return _unknown_node(name=name)
+
+ def _reflected_list_node(self, *, name: str, annotation: Any, seen: frozenset[str], classes_seen: frozenset[type[BaseModel]]) -> InputFormField:
+ """A reflected `list[X]`: the element node derived one layer down and carried in `item`."""
+ item_annotation = list_item_annotation(annotation=annotation)
+ item = (
+ _unknown_node(name=name)
+ if item_annotation is None
+ else self._reflected_node(name=name, annotation=item_annotation, seen=seen, classes_seen=classes_seen)
+ )
+ return ListField(name=name, concept_ref=item.concept_ref, refines=item.refines, required=True, item=_as_list_item(node=item))
# ---- Crate walks ------------------------------------------------------------------------------
@@ -528,6 +639,16 @@ def _as_list_item(*, node: InputFormField) -> InputFormItem:
return _LIST_ITEM_ADAPTER.validate_python({slot: value for slot, value in node if slot != "name"})
+def _root_annotation_of(*, root_model_class: type[BaseModel]) -> Any:
+ """The annotation a `RootModel`'s value actually takes: its `root` field's, optionality peeled.
+
+ Callers test `__pydantic_root_model__` — pydantic's own marker — rather than `issubclass(…, RootModel)`,
+ which narrows to an unparameterized generic the type checker then reports as partially unknown.
+ """
+ annotation, _ = strip_optional(annotation=root_model_class.model_fields["root"].annotation)
+ return annotation
+
+
def _with_effective_hints(*, node: InputFormField, hints: dict[str, str] | None) -> InputFormField:
"""Stamp a node's FINAL effective hints; an applicable `intent` word feeds `kind`, never competes.
diff --git a/subject_grants.toml b/subject_grants.toml
index 006f6f1ac..3f561d48a 100644
--- a/subject_grants.toml
+++ b/subject_grants.toml
@@ -1152,26 +1152,10 @@ rationale = "Single-operand conversion of the annotation; description stays keyw
param = "annotation"
rationale = "Predicate on the annotation; single operand."
-["pipelex/codegen/native_expansion.py::_is_number_union"]
-param = "annotation"
-rationale = "Predicate on the annotation; single operand."
-
["pipelex/codegen/native_expansion.py::_list_blueprint"]
param = "list_annotation"
rationale = "Noun-named derivation from the list annotation; the rest stays keyword."
-["pipelex/codegen/native_expansion.py::_native_code_for_content_class"]
-param = "annotation"
-rationale = "Noun-named derivation: inspects the annotation for its native code; single operand."
-
-["pipelex/codegen/native_expansion.py::_scalar_field_type"]
-param = "annotation"
-rationale = "Noun-named derivation: inspects the annotation for its scalar type; single operand."
-
-["pipelex/codegen/native_expansion.py::_strip_optional"]
-param = "annotation"
-rationale = "Single-operand transformation of the annotation."
-
["pipelex/codegen/native_expansion.py::collect_native_refs_from_structure"]
param = "structure"
rationale = "Single-operand analysis over the structure mapping."
diff --git a/tests/integration/pipelex/pipeline/test_input_form.py b/tests/integration/pipelex/pipeline/test_input_form.py
index a52c67dd8..4ecf6a7f9 100644
--- a/tests/integration/pipelex/pipeline/test_input_form.py
+++ b/tests/integration/pipelex/pipeline/test_input_form.py
@@ -27,6 +27,8 @@ from pydantic import Field
from pipelex.core.memory.exceptions import ListWhereSingularError
from pipelex.core.memory.input_shaper import InputShaper, PipelineInputs
+from pipelex.core.stuffs.document_content import DocumentContent
+from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.interpreter_hub import (
@@ -40,7 +42,7 @@ from pipelex.pipeline.input_form import FieldKind, InputFormField, build_input_f
from pipelex.pipeline.pipe_io_contracts import IOMultiplicity, build_pipe_io_contracts
from pipelex.pipeline.validate_bundle import validate_bundle
from pipelex.system.registries.class_registry_access import get_class_registry
-from tests.helpers.input_form import fields_by_name
+from tests.helpers.input_form import as_list, fields_by_name
if TYPE_CHECKING:
from collections.abc import Callable
@@ -484,10 +486,20 @@ class InputFormFieldLessPayload(StructuredContent):
"""A registered structure class declaring no field — a payload that demands nothing."""
-class InputFormUnmappablePayload(StructuredContent):
- """A structure class whose field has no single blueprint shape — reflection must stay absent, never guess."""
+class InputFormAttachment(StructuredContent):
+ """A nested non-native model, carrying a file-bearing field one level below the concept."""
+
+ caption: str = Field(description="PROBE_desc_reflected_caption")
+ picture: ImageContent = Field(description="PROBE_desc_reflected_picture")
+
+
+class InputFormPartlyMappablePayload(StructuredContent):
+ """A structure class one annotation defeats — every other field, the file-bearing ones included, must still be stated."""
payload: str | int = Field(description="PROBE_desc_reflected_payload")
+ label: str = Field(description="PROBE_desc_reflected_label")
+ attachment: InputFormAttachment = Field(description="PROBE_desc_reflected_attachment")
+ scans: list[DocumentContent] = Field(description="PROBE_desc_reflected_scans")
_KIND_TABLE_MTHDS = """
@@ -498,9 +510,9 @@ description = "Bundle pinning the rest of the kind-assignment table"
description = "PROBE_desc_concept_Constrained: class-backed by a hand-written constrained class"
structure = "InputFormConstrainedPayload"
-[concept.Unmappable]
-description = "PROBE_desc_concept_Unmappable: class-backed by a class reflection cannot map"
-structure = "InputFormUnmappablePayload"
+[concept.PartlyMappable]
+description = "PROBE_desc_concept_PartlyMappable: class-backed by a class one annotation defeats"
+structure = "InputFormPartlyMappablePayload"
[concept.FieldLess]
description = "PROBE_desc_concept_FieldLess: class-backed by a class declaring no field"
@@ -538,12 +550,12 @@ template = "$empty_in $refining_in"
[pipe.edges]
type = "PipeLLM"
description = "Fixed-count-of-one multiplicity and class-backed reflection"
-inputs = { one = "Gadget[1]", constrained = "Constrained", unmappable = "Unmappable", field_less = "FieldLess" }
+inputs = { one = "Gadget[1]", constrained = "Constrained", partly_mappable = "PartlyMappable", field_less = "FieldLess" }
output = "Text"
prompt = \"\"\"
@one
@constrained
-@unmappable
+@partly_mappable
@field_less
\"\"\"
"""
@@ -577,7 +589,7 @@ class TestKindAssignmentTable:
try:
registry = get_class_registry()
registry.register_class(InputFormConstrainedPayload)
- registry.register_class(InputFormUnmappablePayload)
+ registry.register_class(InputFormPartlyMappablePayload)
registry.register_class(InputFormFieldLessPayload)
result = await validate_bundle(mthds_contents=[_KIND_TABLE_MTHDS])
return build_input_form(result.pipes)
@@ -736,12 +748,42 @@ class TestKindAssignmentTable:
assert field_less.required is True
assert field_less.gating is True
- async def test_unmappable_class_falls_back_to_unknown(self, load_empty_library: Callable[[], str]) -> None:
- """Reflection is faithful-or-absent: a class with an unmappable field yields `unknown`, with identity kept."""
+ async def test_an_unmappable_annotation_is_unknown_on_that_field_alone(self, load_empty_library: Callable[[], str]) -> None:
+ """Class reflection is partial: the field no annotation maps is `unknown`, its siblings stay stated."""
input_form = await self._derive_kind_table(load_empty_library)
- unmappable = _field_by_name(input_form["input_form_kinds.edges"], "unmappable")
+ partly_mappable = _field_by_name(input_form["input_form_kinds.edges"], "partly_mappable")
+
+ assert partly_mappable.kind == FieldKind.OBJECT, "One defeated annotation no longer collapses the whole payload"
+ assert partly_mappable.concept_ref == "input_form_kinds.PartlyMappable"
+ assert partly_mappable.description is not None
+ assert "PROBE_desc_concept_PartlyMappable" in partly_mappable.description
+ by_name = fields_by_name(partly_mappable)
- assert unmappable.kind == FieldKind.UNKNOWN
- assert unmappable.concept_ref == "input_form_kinds.Unmappable"
- assert unmappable.description is not None
- assert "PROBE_desc_concept_Unmappable" in unmappable.description
+ payload = by_name["payload"]
+ assert payload.kind == FieldKind.UNKNOWN, "A union that is neither optional nor numeric has no honest node"
+ assert payload.description == "PROBE_desc_reflected_payload", "The unmappable field keeps its identity and helper text"
+ assert by_name["label"].kind == FieldKind.TEXT, "The mappable sibling is stated, not lost with it"
+
+ async def test_file_positions_below_a_defeated_sibling_stay_visible(self, load_empty_library: Callable[[], str]) -> None:
+ """What partial reflection is for: a consumer preparing inputs must still see every file position.
+
+ The collapse hid these — an `image` nested in a plain model, a list of `document` — behind the
+ one sibling annotation the reflection could not map, and a local path at such a position went
+ through un-uploaded.
+ """
+ input_form = await self._derive_kind_table(load_empty_library)
+ by_name = fields_by_name(_field_by_name(input_form["input_form_kinds.edges"], "partly_mappable"))
+
+ attachment = by_name["attachment"]
+ assert attachment.kind == FieldKind.OBJECT, "A nested non-native model is walked into, not flattened to an opaque dict"
+ assert attachment.description == "PROBE_desc_reflected_attachment"
+ nested = fields_by_name(attachment)
+ assert nested["caption"].kind == FieldKind.TEXT
+ assert nested["picture"].kind == FieldKind.IMAGE, "The file position one level down is reported"
+ assert nested["picture"].concept_ref == "native.Image", "A content class maps to its native by identity"
+ assert nested["picture"].description == "PROBE_desc_reflected_picture", "The field's own description wins over the native's"
+
+ scans = as_list(by_name["scans"])
+ assert scans.concept_ref == "native.Document", "A list of a content class names the element concept"
+ assert scans.item.kind == FieldKind.DOCUMENT
+ assert not hasattr(scans.item, "name"), "A list item has no authored name"
diff --git a/tests/unit/pipelex/pipeline/test_input_form_deriver.py b/tests/unit/pipelex/pipeline/test_input_form_deriver.py
index 545a25aaf..bcc8888ff 100644
--- a/tests/unit/pipelex/pipeline/test_input_form_deriver.py
+++ b/tests/unit/pipelex/pipeline/test_input_form_deriver.py
@@ -1,16 +1,49 @@
"""Unit-pin the deriver's escape hatches that the library loader keeps out of reach.
-The loader rejects concept cycles (`LibraryLoadingError`) and a `structure = "ClassName"` naming an
-unregistered class before a bundle ever reaches `build_input_form`, so those branches are exercised
-directly on a hand-built qualified crate: the derivation must stay total and finite on any crate.
+The loader rejects concept cycles (`LibraryLoadingError`), a `structure = "ClassName"` naming an
+unregistered class, and a registered class that holds itself (which it reads as a concept cycle),
+before a bundle ever reaches `build_input_form` — so those branches are exercised directly on a
+hand-built qualified crate: the derivation must stay total and finite on any crate.
"""
from __future__ import annotations
+from pydantic import Field, RootModel
+
from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
+from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.pipeline.input_form import FieldKind, InputFormDeriver
-from tests.helpers.input_form import as_object
+from pipelex.system.registries.class_registry_access import get_class_registry
+from tests.helpers.input_form import as_list, as_object, fields_by_name
+
+
+class SelfReferentialPayload(StructuredContent):
+ """A registered structure class that holds itself — reflection must cut the path, not recurse forever."""
+
+ label: str = Field(description="The label")
+ child: "SelfReferentialPayload | None" = Field(default=None, description="The nested twin")
+
+
+class RootBackedTags(RootModel[list[str]]):
+ """A `RootModel` field: the value on the wire IS the root list, never an object over a `root` key."""
+
+
+class RootBackedPayload(StructuredContent):
+ """A registered structure class holding a `RootModel` field."""
+
+ label: str = Field(description="The label")
+ tags: RootBackedTags = Field(description="The tags")
+
+
+# mypy reads the two bases' `model_construct` overloads as incompatible; pydantic builds these classes
+# fine and pyright accepts them, and a class-backed concept over a `RootModel` is what these tests pin.
+class RootBackedSlug(StructuredContent, RootModel[str]): # type: ignore[misc]
+ """A registered structure class that IS a `RootModel` — the concept's whole payload is the root string."""
+
+
+class RootBackedTagList(StructuredContent, RootModel[list[str]]): # type: ignore[misc]
+ """A registered structure class that IS a `RootModel` over a list — the payload is the list itself."""
def _concept_field(*, concept_ref: str) -> ConceptStructureBlueprint:
@@ -72,3 +105,72 @@ class TestInputFormDeriverEscapeHatches:
assert node.kind == FieldKind.UNKNOWN
assert node.refines == ["dep->other.Score"]
assert node.description == "Refines a dependency's score"
+
+ def test_a_self_referential_class_is_cut_at_the_revisit(self) -> None:
+ """Class reflection terminates: the position that revisits the class is `unknown`, its siblings stated."""
+ concepts: dict[str, ConceptBlueprint | str] = {
+ "demo.Recursive": ConceptBlueprint(description="Backed by a class that holds itself", structure="SelfReferentialPayload"),
+ }
+ registry = get_class_registry()
+ registry.register_class(SelfReferentialPayload)
+ try:
+ node = InputFormDeriver(concepts=concepts).derive_concept(name="recursive", concept_ref="demo.Recursive")
+ finally:
+ registry.unregister_class(SelfReferentialPayload)
+
+ assert node.kind == FieldKind.OBJECT
+ by_name = fields_by_name(node)
+ assert by_name["label"].kind == FieldKind.TEXT
+ assert by_name["child"].kind == FieldKind.UNKNOWN
+ assert by_name["child"].required is False, "The self-reference is optional, and cutting the path does not change that"
+
+ def test_a_root_model_field_reports_its_root_annotation(self) -> None:
+ """A `RootModel` is its root value on the wire, so the node is the root's — never an `object` over `root`."""
+ concepts: dict[str, ConceptBlueprint | str] = {
+ "demo.RootBacked": ConceptBlueprint(description="Backed by a class holding a RootModel", structure="RootBackedPayload"),
+ }
+ registry = get_class_registry()
+ registry.register_class(RootBackedPayload)
+ try:
+ node = InputFormDeriver(concepts=concepts).derive_concept(name="root_backed", concept_ref="demo.RootBacked")
+ finally:
+ registry.unregister_class(RootBackedPayload)
+
+ by_name = fields_by_name(node)
+ assert by_name["label"].kind == FieldKind.TEXT
+ tags = as_list(by_name["tags"])
+ assert tags.item.kind == FieldKind.TEXT
+ assert tags.description == "The tags", "The field's own description survives the root reflection"
+
+ def test_a_root_model_backed_concept_reports_its_root_annotation(self) -> None:
+ """A class-backed concept whose class IS a `RootModel` states the root's node, never an `object` over `root`."""
+ concepts: dict[str, ConceptBlueprint | str] = {
+ "demo.Slug": ConceptBlueprint(description="A slug", structure="RootBackedSlug"),
+ }
+ registry = get_class_registry()
+ registry.register_class(RootBackedSlug)
+ try:
+ node = InputFormDeriver(concepts=concepts).derive_concept(name="slug", concept_ref="demo.Slug")
+ finally:
+ registry.unregister_class(RootBackedSlug)
+
+ assert node.kind == FieldKind.TEXT, "The payload `model_validate` accepts is the root string itself"
+ assert node.concept_ref == "demo.Slug", "A top-level field states the concept it carries"
+ assert node.description == "A slug"
+
+ def test_a_root_model_backed_concept_over_a_list_is_a_list_node(self) -> None:
+ """The root annotation decides the kind: a list root is a `list` node carrying the concept's own ref."""
+ concepts: dict[str, ConceptBlueprint | str] = {
+ "demo.TagList": ConceptBlueprint(description="The tags", structure="RootBackedTagList"),
+ }
+ registry = get_class_registry()
+ registry.register_class(RootBackedTagList)
+ try:
+ node = InputFormDeriver(concepts=concepts).derive_concept(name="tag_list", concept_ref="demo.TagList")
+ finally:
+ registry.unregister_class(RootBackedTagList)
+
+ tag_list = as_list(node)
+ assert tag_list.item.kind == FieldKind.TEXT
+ assert tag_list.concept_ref == "demo.TagList", "A root-valued concept IS the whole value, so its ref rides the list node"
+ assert tag_list.description == "The tags"