Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Stop four wrap serializers from erasing their serialization JSON Schema
Louis Choquel4d23517f4b22
diff --git a/AGENTS.md b/AGENTS.mdindex ecee6cb0b..4fe056dae 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -210,6 +210,8 @@- Use Field(default_factory=...) for mutable defaults- Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.+One scoped exception to "every function return must be typed": a pydantic `@model_serializer` / `@field_serializer` return must stay **unannotated** unless the annotation is the exact type the model publishes. Pydantic turns a serializer's return annotation into the model's serialization JSON Schema and prefers it over the schema it would generate, so a `-> dict[str, Any]` publishes the whole model as an opaque `{"type": "object", "additionalProperties": true}` — no properties, no `required`, no closed shape — for every consumer that generates response schemas, which FastAPI always does for its responses. A precise annotation is fine and desirable (`-> str` on a model that serializes to its raw string); a `dict`/`Any` one erases the model, and nothing in a normal test suite notices because validation-mode schemas and `model_dump()` both stay correct.+#### Data-holder shapes (NamedTuple, dataclass, BaseModel)The deciding question is **does this value ever get serialized** — sent across a process or network boundary, persisted to disk or a database, or round-tripped through (de)serialization? If yes, it must be one of the two serialization-safe forms (a `BaseModel` or a pydantic dataclass): a `NamedTuple` or stdlib `@dataclass` loses its type on the way through and silently corrupts the data across the boundary. (A distributed executor like Temporal is the canonical case — its payloads cross the wire between worker processes — but the same applies to any cache, queue, API response, or on-disk record.) Stdlib `@dataclass` is banned outright; pick one of these four forms:diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f6d14b915..6986ee052 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,11 @@# Changelog+## [Unreleased]++### Fixed++- **Blueprint serialization schemas**: `ConceptBlueprint`, `ConceptStructureBlueprint`, `InputSlotBlueprint` and `ConstructBlueprint` publish their real shape again in serialization mode. Each carries a `@model_serializer(mode="wrap")` annotated `-> dict[str, Any]`, and pydantic turns a serializer's return annotation into the model's serialization JSON Schema in preference to the one it would generate — so all four came back as an opaque `{"type": "object", "additionalProperties": true}` with no properties, no `required` and no closed shape. This is the same annotation `mthds` 0.11.1 dropped for the input-form field descriptors. Nothing here noticed, because validation-mode schemas and `model_dump()` were both correct throughout; the damage landed in consumers that generate response-model schemas, which FastAPI always does, so a downstream OpenAPI artifact regains its field-level detail on its next `pipelex` bump. A new sweep test holds every Pipelex model to the rule, and the agent rules now scope "every function return must be typed" so a serializer's return stays unannotated.+## [v0.54.0] - 2026-08-28### Addeddiff --git a/pipelex/core/concepts/concept_blueprint.py b/pipelex/core/concepts/concept_blueprint.pyindex 2ba1c9bd8..707fbbc79 100644--- a/pipelex/core/concepts/concept_blueprint.py+++ b/pipelex/core/concepts/concept_blueprint.py@@ -31,11 +31,17 @@ class ConceptBlueprint(BaseModel):return dict(sorted(hints.items())) if hints else None@model_serializer(mode="wrap")- def serialize_without_absent_hints(self, handler: SerializerFunctionWrapHandler) -> dict[str, Any]:+ def serialize_without_absent_hints(self, handler: SerializerFunctionWrapHandler):"""Absent hints are an absent member — never `null` (spec rule; keeps hint-free cratefingerprints byte-identical to before hints existed). Deliberately NOT a blanket`exclude_none`: the crate spec's general absent-members canonicalization is a separateforward contract whose adoption would migrate every existing fingerprint.++ The return is deliberately unannotated, and stays that way. Pydantic turns a serializer's+ return annotation into the model's serialization JSON Schema and prefers it over the schema+ it would generate, so a `-> dict[str, Any]` here publishes this model as an opaque+ `{"type": "object", "additionalProperties": true}` — no properties, no `required`, no+ closed shape — for every consumer that generates response schemas (FastAPI always does)."""dumped: dict[str, Any] = handler(self)if dumped.get("hints") is None:diff --git a/pipelex/core/concepts/concept_structure_blueprint.py b/pipelex/core/concepts/concept_structure_blueprint.pyindex b2f3f1df0..7638ac1b6 100644--- a/pipelex/core/concepts/concept_structure_blueprint.py+++ b/pipelex/core/concepts/concept_structure_blueprint.py@@ -82,9 +82,12 @@ class ConceptStructureBlueprint(BaseModel):return dict(sorted(hints.items())) if hints else None@model_serializer(mode="wrap")- def serialize_without_absent_hints(self, handler: SerializerFunctionWrapHandler) -> dict[str, Any]:+ def serialize_without_absent_hints(self, handler: SerializerFunctionWrapHandler):"""Absent hints are an absent member — never `null` (spec rule; keeps hint-free cratefingerprints byte-identical to before hints existed).++ The return is deliberately unannotated — see `ConceptBlueprint.serialize_without_absent_hints`:+ an annotation here becomes the model's serialization JSON Schema and erases its shape."""dumped: dict[str, Any] = handler(self)if dumped.get("hints") is None:diff --git a/pipelex/kit/agent_rules/python_standards.md b/pipelex/kit/agent_rules/python_standards.mdindex a173c81c6..12e040f67 100644--- a/pipelex/kit/agent_rules/python_standards.md+++ b/pipelex/kit/agent_rules/python_standards.md@@ -95,6 +95,8 @@- Use Field(default_factory=...) for mutable defaults- Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.+One scoped exception to "every function return must be typed": a pydantic `@model_serializer` / `@field_serializer` return must stay **unannotated** unless the annotation is the exact type the model publishes. Pydantic turns a serializer's return annotation into the model's serialization JSON Schema and prefers it over the schema it would generate, so a `-> dict[str, Any]` publishes the whole model as an opaque `{"type": "object", "additionalProperties": true}` — no properties, no `required`, no closed shape — for every consumer that generates response schemas, which FastAPI always does for its responses. A precise annotation is fine and desirable (`-> str` on a model that serializes to its raw string); a `dict`/`Any` one erases the model, and nothing in a normal test suite notices because validation-mode schemas and `model_dump()` both stay correct.+### Data-holder shapes (NamedTuple, dataclass, BaseModel)The deciding question is **does this value ever get serialized** — sent across a process or network boundary, persisted to disk or a database, or round-tripped through (de)serialization? If yes, it must be one of the two serialization-safe forms (a `BaseModel` or a pydantic dataclass): a `NamedTuple` or stdlib `@dataclass` loses its type on the way through and silently corrupts the data across the boundary. (A distributed executor like Temporal is the canonical case — its payloads cross the wire between worker processes — but the same applies to any cache, queue, API response, or on-disk record.) Stdlib `@dataclass` is banned outright; pick one of these four forms:diff --git a/pipelex/pipe_machinery/pipe_blueprint.py b/pipelex/pipe_machinery/pipe_blueprint.pyindex 8238b8ba9..30483c2ff 100644--- a/pipelex/pipe_machinery/pipe_blueprint.py+++ b/pipelex/pipe_machinery/pipe_blueprint.py@@ -205,10 +205,13 @@ class InputSlotBlueprint(BaseModel):return dict(sorted(hints.items())) if hints else None@model_serializer(mode="wrap")- def serialize_without_absent_hints(self, handler: SerializerFunctionWrapHandler) -> dict[str, Any]:+ def serialize_without_absent_hints(self, handler: SerializerFunctionWrapHandler):"""Absent hints are an absent member — never `null` (spec rule). Unreachable for ahint-free slot in practice (the parse-time collapse leaves it a plain string), kept so anydirectly-constructed instance still serializes canonically.++ The return is deliberately unannotated — see `ConceptBlueprint.serialize_without_absent_hints`:+ an annotation here becomes the model's serialization JSON Schema and erases its shape."""dumped: dict[str, Any] = handler(self)if dumped.get("hints") is None:diff --git a/pipelex/pipe_operators/compose/construct_blueprint.py b/pipelex/pipe_operators/compose/construct_blueprint.pyindex b5b9e643e..b0d63a745 100644--- a/pipelex/pipe_operators/compose/construct_blueprint.py+++ b/pipelex/pipe_operators/compose/construct_blueprint.py@@ -279,11 +279,14 @@ class ConstructBlueprint(BaseModel):return {field_name: field_bp.to_mthds_dict() for field_name, field_bp in self.fields.items()}@model_serializer(mode="wrap")- def serialize_with_context(self, handler: SerializerFunctionWrapHandler, info: SerializationInfo) -> dict[str, Any]:+ def serialize_with_context(self, handler: SerializerFunctionWrapHandler, info: SerializationInfo):"""Serialize with format-aware context.When context contains {"format": "mthds"}, outputs MTHDS-format dict.Otherwise, uses default Pydantic serialization.++ The return is deliberately unannotated — see `ConceptBlueprint.serialize_without_absent_hints`:+ an annotation here becomes the model's serialization JSON Schema and erases its shape."""if info.context and info.context.get("format") == "mthds":return self.to_mthds_dict()diff --git a/tests/unit/pipelex/test_serialization_json_schema.py b/tests/unit/pipelex/test_serialization_json_schema.pynew file mode 100644index 000000000..8fd7dd31e--- /dev/null+++ b/tests/unit/pipelex/test_serialization_json_schema.py@@ -0,0 +1,197 @@+"""No Pipelex model publishes an opaque serialization JSON Schema.++Pydantic derives a model's *serialization* schema from its `@model_serializer`'s return+annotation and prefers it over the schema it would generate. A `-> dict[str, Any]` therefore+replaces the real shape with `{"type": "object", "additionalProperties": true}` — no properties,+no `required`, no closed shape — while validation-mode schemas stay complete and every runtime+`model_dump()` keeps working. Nothing in this repo notices: the suite is green, and no call to+`model_json_schema()` in `pipelex/` asks for serialization mode (the MTHDS schema generator asks+for `mode="validation"` explicitly). The damage lands one repo away, in a consumer that generates+response-model schemas — FastAPI always does — where a whole blueprint reaches clients as an+opaque object and a client generator produces nothing useful from it.++The bug class already recurred: it hit four models here and the input-form field descriptors+upstream in `mthds`. Hence a sweep rather than four hand-written assertions — the next model to+grow a wrap serializer is caught the day it is written, not the next time a consumer regenerates+an artifact.+"""++from __future__ import annotations++import importlib+import pkgutil+from typing import Any, cast++import pytest+from pydantic import BaseModel+from pydantic.errors import PydanticInvalidForJsonSchema++import pipelex+from pipelex.core.concepts.concept_blueprint import ConceptBlueprint+from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint+from pipelex.pipe_machinery.pipe_blueprint import InputSlotBlueprint+from pipelex.pipe_operators.compose.construct_blueprint import ConstructBlueprint++# The models that carry a `@model_serializer(mode="wrap")` and once published nothing because of it.+WRAP_SERIALIZER_MODELS = [+ ConceptBlueprint,+ ConceptStructureBlueprint,+ InputSlotBlueprint,+ ConstructBlueprint,+]++# `FuncRegistry` holds live callables, which have no JSON Schema at all — pydantic refuses to build+# one. Pinned by name rather than skipped silently: a model that newly stops being describable drops+# out of the sweep, and this is what makes that visible instead of quietly shrinking the guard.+MODELS_WITHOUT_A_JSON_SCHEMA = {+ "pipelex.system.registries.func_registry.FuncRegistry",+}++REMEDY = (+ "Drop the return annotation from the model's `@model_serializer` — pydantic turns it into the "+ "model's serialization JSON Schema and prefers it over the generated one."+)+++def _import_every_pipelex_module() -> tuple[int, list[str]]:+ """Import every `pipelex.*` module, returning how many loaded and the failures.++ Failures are returned rather than swallowed: a sweep that quietly skips what it cannot import+ is a guard that has gone quiet exactly where the tree got interesting.+ """+ imported = 0+ failures: list[str] = []+ for module_info in pkgutil.walk_packages(pipelex.__path__, prefix=f"{pipelex.__name__}."):+ try:+ importlib.import_module(module_info.name)+ except Exception as exc:+ failures.append(f"{module_info.name}: {type(exc).__name__}: {exc}")+ else:+ imported += 1+ return imported, failures+++def _discover_pipelex_models() -> list[type[BaseModel]]:+ """Every `BaseModel` subclass declared under `pipelex.`, after loading the whole package."""+ _import_every_pipelex_module()++ seen: set[type[BaseModel]] = set()++ def walk(cls: type[BaseModel]) -> None:+ for sub in cls.__subclasses__():+ if sub not in seen:+ seen.add(sub)+ walk(sub)++ walk(BaseModel)+ models = [cls for cls in seen if cls.__module__.startswith(f"{pipelex.__name__}.")]+ return sorted(models, key=lambda cls: f"{cls.__module__}.{cls.__qualname__}")+++def _resolve_root(schema: dict[str, Any]) -> dict[str, Any]:+ """Follow a top-level `$ref` into `$defs`.++ A self-referential model (`ConstructBlueprint` nests itself) publishes `{"$defs": …, "$ref": …}`+ at the top, so reading `properties` off the raw schema would judge every recursive model opaque.+ """+ ref = schema.get("$ref")+ if not isinstance(ref, str):+ return schema+ definition_name = ref.rsplit("/", maxsplit=1)[-1]+ defs: dict[str, Any] = schema.get("$defs") or {}+ resolved: Any = defs.get(definition_name)+ if isinstance(resolved, dict):+ return cast("dict[str, Any]", resolved)+ return schema+++def _is_opaque_object(schema: dict[str, Any]) -> bool:+ """The exact shape an erasing return annotation produces: an object that says nothing."""+ root = _resolve_root(schema)+ return root.get("type") == "object" and not root.get("properties") and root.get("additionalProperties") is True+++def _erases_its_shape(model: type[BaseModel]) -> bool:+ """The model is opaque when serialized but not when validated — the erasure signature.++ Asymmetry is what makes this a defect rather than a fact. A model that is honestly open —+ `extra="allow"` with no declared fields (`CompositeContent`), or a `RootModel[dict[str, Any]]`+ (`SdkClientRegistry`) — is opaque in *both* modes, and an open dict really is its shape. Testing+ the asymmetry instead of the shape is what lets this sweep run without an allowlist to maintain,+ and an allowlist is the wrong instrument here anyway: it would have to be extended by the very+ person adding the next erasing serializer.+ """+ return _is_opaque_object(model.model_json_schema(mode="serialization")) and not _is_opaque_object(model.model_json_schema(mode="validation"))+++def _sweep_serialization_shapes() -> dict[str, bool]:+ """Every describable Pipelex model, mapped to whether it erases its shape when serialized."""+ verdicts: dict[str, bool] = {}+ for model in _discover_pipelex_models():+ qualified_name = f"{model.__module__}.{model.__qualname__}"+ if qualified_name in MODELS_WITHOUT_A_JSON_SCHEMA:+ continue+ verdicts[qualified_name] = _erases_its_shape(model)+ return verdicts+++def _models_without_a_json_schema() -> set[str]:+ """The models pydantic refuses to describe at all, discovered rather than assumed."""+ undescribable: set[str] = set()+ for model in _discover_pipelex_models():+ try:+ model.model_json_schema(mode="serialization")+ except PydanticInvalidForJsonSchema:+ undescribable.add(f"{model.__module__}.{model.__qualname__}")+ return undescribable+++class TestSerializationJsonSchema:+ def test_discovery_is_not_vacuous(self) -> None:+ """Anti-vacuity: the walk really loaded the package and really found models.++ Deliberately not parametrized — pytest reports `got empty parameter set` and exits 0 on an+ empty list, so the guard would be unreachable exactly when it matters.+ """+ imported, failures = _import_every_pipelex_module()+ assert not failures, "Modules failed to import, so the sweep below is silently narrower than it looks:\n" + "\n".join(failures)+ assert imported, "No pipelex module imported — the sweep would compare two empty sets and pass vacuously."++ models = _discover_pipelex_models()+ assert models, "No pipelex BaseModel discovered — the sweep would pass vacuously."++ discovered_names = {cls.__qualname__ for cls in models}+ assert "ConceptBlueprint" in discovered_names, "The blueprint layer is missing from the sweep — discovery is not reaching the wire surface."++ assert _models_without_a_json_schema() == MODELS_WITHOUT_A_JSON_SCHEMA, (+ "The set of models pydantic cannot describe has moved. A model that newly stops being describable silently "+ "leaves the sweep below, so update MODELS_WITHOUT_A_JSON_SCHEMA deliberately — after checking the model "+ f"really has no wire form.\n expected: {sorted(MODELS_WITHOUT_A_JSON_SCHEMA)}\n actual: {sorted(_models_without_a_json_schema())}"+ )++ def test_no_pipelex_model_publishes_an_opaque_serialization_schema(self) -> None:+ """A model whose serialization schema says nothing is invisible to every client generator."""+ offenders = [name for name, erased in _sweep_serialization_shapes().items() if erased]++ assert not offenders, (+ "These models describe themselves when validated but publish an opaque object when serialized:\n"+ + "\n".join(f" - {name}" for name in offenders)+ + f"\n\n{REMEDY}"+ )++ @pytest.mark.parametrize("model", WRAP_SERIALIZER_MODELS, ids=lambda model: model.__qualname__)+ def test_wrap_serializer_models_keep_their_property_lists(self, model: type[BaseModel]) -> None:+ """Serialization mode describes the same fields as validation mode.++ Sharper than the opaque-shape sweep: it catches a *partial* erasure, where a schema keeps a+ property or two and loses the rest, which the sweep above would wave through.+ """+ serialization_properties = sorted(_resolve_root(model.model_json_schema(mode="serialization")).get("properties", {}))+ validation_properties = sorted(_resolve_root(model.model_json_schema(mode="validation")).get("properties", {}))++ assert serialization_properties, f"{model.__qualname__} publishes no properties in serialization mode. {REMEDY}"+ assert serialization_properties == validation_properties, (+ f"{model.__qualname__} describes different fields depending on the schema mode.\n"+ f" serialization: {serialization_properties}\n"+ f" validation: {validation_properties}\n\n{REMEDY}"+ )