Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feature/Cheap-crate-load · L-260926-c80b97 (#1270)
Louis Choquel09af6679ea1b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b683372c5..83798e20d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,15 @@# Changelog+## [Unreleased]++### Added++- **`load_from_crate` takes `is_crate_prevalidated`**: `LibraryManager.load_from_crate(library_id=…, crate=…, is_crate_prevalidated=True)` skips the library validation that ends a load, for a crate built from a library that the same pipelex version already loaded and validated, such as the crate a distributed run hands its worker; everything else the load does still runs, and the default stays the validating load.++### Changed++- **`are_classes_equivalent` answers without a JSON schema when it can**: a class is equivalent to itself, and two plain models with different field names are not equivalent, both now decided before any schema is generated, which removes most of the cost of validating a library whose pipes output structured concepts. A plain model is one built only from an allow-list of annotations (scalars, `Any`, `Literal`, unions, builtin containers and nested plain models), inert constraints and plain defaults, with no alias and no schema hook or schema-shaping setting, so the properties pydantic publishes for it provably equal its field names; the structure classes generated from a concept's `structure` table and most native content classes are plain. Verdicts are unchanged: every other pair still goes to the full schema comparison, and a schema that cannot be generated still raises.+## [v0.66.0] - 2026-09-25### Highlightsdiff --git a/docs/contribute/hub-layering.md b/docs/contribute/hub-layering.mdindex 535917130..b05fc0df8 100644--- a/docs/contribute/hub-layering.md+++ b/docs/contribute/hub-layering.md@@ -135,7 +135,7 @@ Be precise about what that buys, because it is easy to overstate. The read is stTwo shapes fall out of that, and they are the ones to copy:-- **Compatibility is two tiers, composed by the thing that owns resolution.** `Concept.are_compatible_by_declaration` is pure over the model's fields (plus an injected `concept_resolver`): dynamic short-circuits, ref equality, declared-class-name equality, `refines` chains. `ConceptLibrary.is_compatible` asks that first and only resolves classes when it is inconclusive. A name that should have resolved and didn't now raises `ConceptStructureClassNotFoundError` instead of answering `False` — "unknown" and "incompatible" are different answers, and conflating them was a latent wrong-verdict bug.+- **Compatibility is two tiers, composed by the thing that owns resolution.** `Concept.are_compatible_by_declaration` is pure over the model's fields (plus an injected `concept_resolver`): dynamic short-circuits, ref equality, declared-class-name equality, `refines` chains. `ConceptLibrary.is_compatible` asks that first and only resolves classes when it is inconclusive. A name that should have resolved and didn't now raises `ConceptStructureClassNotFoundError` instead of answering `False` — "unknown" and "incompatible" are different answers, and conflating them was a latent wrong-verdict bug. The class tier's equivalence test (`are_classes_equivalent`) compares JSON schemas, which is the expensive part of loading a library, so it first settles the answers that need no schema: a class is equivalent to itself, and two *plain* models with different field names are not equivalent. A plain model is built only from an allow-list — scalar, `Any`, `Literal`, union and builtin-container annotations, inert constraints such as `gt` or `strict`, defaults made of plain values (a time-zone-aware `datetime` or `time` only in the standard library's `datetime.timezone`), no alias, no schema hook or schema-shaping setting, and nested models that are plain in turn — and for such a model the properties pydantic publishes provably equal its field names. Any other pair, including two plain models with the same field names, goes to the full comparison, so the shortcut only ever answers where the schemas would say `False`, and a schema that cannot be generated still raises. It is an allow-list rather than a list of the ways pydantic can drop a property because those ways keep multiplying (a type alias to `SkipJsonSchema`, a nested model's title generator or typed extras, a default whose own type has a schema hook, a default whose custom time zone raises when asked for its offset), and a deny-list that misses one answers `False` for two classes whose schemas are identical. In practice the structure classes generated from a concept's `structure` table and the native content classes are plain, so library validation's question of whether a pipe's output is an image needs no schema for them; `CompositeContent`, which allows extra fields, is the notable exception.- **Rendering takes the resolved class, not the name.** `Concept.render_concept_representation` receives a `structure_class`; `StuffSpec.render_stuff_spec` takes the `concept_provider`, resolves once, and passes the class down. Interpreter-layer callers (`output_renderer`, `input_renderer`, `pipe_io_contracts`, `builder/runner_code`) supply `get_concept_library()`.The **write side** stays ambient on purpose: `concept_factory` and `structure_generation/generator.py` generate structure classes and register them at library-load time, which is genuinely the registry's business. That those two are the only modules under `pipelex/core/concepts/` touching the registry is pinned as a golden set by `tests/unit/pipelex/core/concepts/test_concept_registry_boundary.py` — nothing else can see it, since both the accessor and its users are kernel-layer.diff --git a/docs/under-the-hood/runtime-bridge-and-transport.md b/docs/under-the-hood/runtime-bridge-and-transport.mdindex 3b3deca1c..106c67b76 100644--- a/docs/under-the-hood/runtime-bridge-and-transport.md+++ b/docs/under-the-hood/runtime-bridge-and-transport.md@@ -77,6 +77,8 @@ Sending a pipe run to a separate worker process introduces three challenges thatThe submitter builds a `LibraryCrate` from its loaded library (`pipelex/libraries/library_crate_factory.py`) and attaches it to the `PipeJob`. On the worker, the job's entry point loads the crate into a scoped library, registering the bundle's dynamic concept classes, so `get_required_pipe()` resolves every pipe at every level of nesting. Because a distributed run can fan out into nested jobs (a controller's child pipes), every nested job carries the same crate; loading is idempotent via the fingerprint, so re-receiving a crate already loaded on a worker is a no-op.+A load ends by validating the whole library (`Library.validate_library`), which checks every pipe against the concepts and pipes it depends on. That check is pure — it raises or returns, and leaves nothing behind that a run later reads — and the crate a worker receives was built from a library the submitter already loaded and validated. So a worker running the same pipelex version as its submitter can call `load_from_crate(library_id=…, crate=…, is_crate_prevalidated=True)`, which skips that final validation and nothing else: fingerprint idempotency, domain and concept loading with the registration of the dynamic classes, the concept-cycle check, pipe construction with each pipe's static validation, and source tracking all still run. The precondition belongs to the host: pass it only where submitter and worker ship on the same pipelex, as a deployment that builds them from one lockfile does. A crate from anywhere else keeps the default, validating load.+---## Deferred hydrationdiff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex 11ff77802..f10b74e9f 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -435,7 +435,7 @@ class LibraryManager(LibraryManagerAbstract):return fingerprint in self._loaded_fingerprints.get(library_id, set())@override- def load_from_crate(self, *, library_id: str, crate: LibraryCrate) -> list[PipeAbstract]:+ def load_from_crate(self, *, library_id: str, crate: LibraryCrate, is_crate_prevalidated: bool = False) -> list[PipeAbstract]:"""Load a LibraryCrate into a live Library.Fingerprint idempotency: if a crate with the same fingerprint was already loaded@@ -449,6 +449,14 @@ class LibraryManager(LibraryManagerAbstract):Args:library_id: The library to load intocrate: The LibraryCrate containing qualified blueprints, domain metadata, and source info+ is_crate_prevalidated: Skip the library validation that ends the load. Precondition: the crate+ was built from a library that this same pipelex version loaded and validated — the runner's+ crate, handed to the worker that executes it. Library validation is a pure check (it writes+ nothing a run later reads), so skipping it changes no behaviour, only the cost of reaching a+ verdict already reached. Everything else still happens: fingerprint idempotency, domain and+ concept loading with class registration, the concept-cycle check, pipe construction with each+ pipe's static validation, and source tracking. Leave it False for any crate whose library was+ not validated by this pipelex version.Returns:List of all pipes that were loaded, or empty list if already loaded@@ -543,7 +551,8 @@ class LibraryManager(LibraryManagerAbstract):library.pipe_library.add_pipes(pipes=all_pipes)- library.validate_library()+ if not is_crate_prevalidated:+ library.validate_library()# Only cache fingerprint after the entire load succeeds — if loading fails# with an exception, subsequent retries must not be skipped.diff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.pyindex 02c3ae598..b83e16bed 100644--- a/pipelex/libraries/library_manager_abstract.py+++ b/pipelex/libraries/library_manager_abstract.py@@ -118,7 +118,7 @@ class LibraryManagerAbstract(ABC):"""@abstractmethod- def load_from_crate(self, *, library_id: str, crate: LibraryCrate) -> list[PipeAbstract]:+ def load_from_crate(self, *, library_id: str, crate: LibraryCrate, is_crate_prevalidated: bool = False) -> list[PipeAbstract]:"""Load a LibraryCrate into a live Library.Note: This method does NOT resolve cross-package address-based dependencies.@@ -127,6 +127,9 @@ class LibraryManagerAbstract(ABC):Args:library_id: The library to load intocrate: The LibraryCrate containing qualified blueprints, domain metadata, and source info+ is_crate_prevalidated: Skip the library validation that ends the load. Precondition: the crate+ was built from a library that this same pipelex version loaded and validated (a runner's+ crate, handed to the worker that executes it). Everything else the load does still happens."""@abstractmethoddiff --git a/pipelex/tools/typing/class_utils.py b/pipelex/tools/typing/class_utils.pyindex 2b3dc948a..d0f21ff3e 100644--- a/pipelex/tools/typing/class_utils.py+++ b/pipelex/tools/typing/class_utils.py@@ -1,15 +1,39 @@+import datetime+import inspectimport types-from typing import TYPE_CHECKING, Annotated, Any, Union, cast, get_args, get_origin--from pydantic import PydanticUndefinedAnnotation, PydanticUserError--if TYPE_CHECKING:- from pydantic.fields import FieldInfo+from typing import Annotated, Any, Literal, Union, cast, get_args, get_origin+from annotated_types import Ge, Gt, Le, Lt, MaxLen, MinLen, MultipleOf+from pydantic import BaseModel, PydanticUndefinedAnnotation, PydanticUserError+from pydantic.fields import FieldInfo+from pydantic.types import Strict+from pydantic_core import PydanticUndefined_NoneType = type(None)_UnionType = getattr(types, "UnionType", None) # Py3.10+: types.UnionType+# The closed vocabulary of a "plain" model, the only kind the class-equivalence shortcut reads (see+# `_is_plain_model`). Every membership test against these tuples is by identity, so an annotation or a value+# that cannot be hashed, or a subclass carrying a pydantic hook of its own, is simply not a member.++# Leaf annotations whose JSON schema pydantic builds itself, with no hook a user could have attached.+_PLAIN_LEAF_ANNOTATIONS: tuple[Any, ...] = (str, int, float, bool, _NoneType, datetime.date, datetime.datetime, datetime.time, Any)+# Containers whose JSON schema is pydantic's own array or object over their item types, bare or subscripted.+_PLAIN_CONTAINER_TYPES: tuple[type[Any], ...] = (list, dict, set, tuple)+_UNION_ORIGINS: tuple[Any, ...] = (Union, types.UnionType)+_PLAIN_LITERAL_VALUE_TYPES: tuple[type[Any], ...] = (str, int, bool)+# Annotation markers that only add constraint keywords to the property they annotate, which is how pydantic+# stores `Field(gt=…, min_length=…, strict=…)` in `FieldInfo.metadata` too. Anything else (a `Field` nested in+# an `Annotated`, `SkipJsonSchema`, `WithJsonSchema`, a validator, a pattern) may rewrite or drop the property.+_INERT_METADATA_TYPES: tuple[type[Any], ...] = (Gt, Ge, Lt, Le, MultipleOf, MinLen, MaxLen, Strict)+# Values a field default may hold. Pydantic encodes a default into the schema through a `TypeAdapter` of the+# default's own type, whatever the field's annotation says, so a default of another type can run that type's+# hooks and drop the property.+_PLAIN_DEFAULT_SCALAR_TYPES: tuple[type[Any], ...] = (_NoneType, str, int, float, bool, datetime.date, datetime.datetime, datetime.time)+_PLAIN_DEFAULT_COLLECTION_TYPES: tuple[type[Any], ...] = (list, tuple, set, frozenset, dict)+# The class-level entry points to a model's JSON schema; a plain model inherits all three from `BaseModel`.+_MODEL_SCHEMA_ENTRY_POINTS = ("model_json_schema", "__get_pydantic_core_schema__", "__get_pydantic_json_schema__")+def normalize_property_for_comparison(prop: dict[str, Any]) -> dict[str, Any]:"""Normalize a property dict by removing description and keeping only structural parts.@@ -66,15 +90,171 @@ def normalize_properties_for_comparison(properties: dict[str, Any]) -> dict[str,return {name: normalize_property_for_comparison(prop) for name, prop in properties.items()}+def _is_any_of(*, candidate: object, options: tuple[object, ...]) -> bool:+ """Whether `candidate` is one of `options`, by identity, so an unhashable candidate is simply not one."""+ return any(candidate is option for option in options)+++def _has_one_of_types(*, value: object, allowed_types: tuple[type[Any], ...]) -> bool:+ """Whether the exact type of `value` is one of `allowed_types`; a subclass, which may carry a pydantic hook, is not."""+ return _is_any_of(candidate=type(value), options=allowed_types)+++def _is_inherited_from_base_model(*, model_class: type[BaseModel], attribute_name: str) -> bool:+ """Whether `model_class` inherits `attribute_name` from `BaseModel` rather than defining its own."""+ for klass in model_class.__mro__:+ if attribute_name in vars(klass):+ return klass is BaseModel+ return False+++def _is_plain_default(*, default_value: object) -> bool:+ """Whether a field default is built only from plain scalars and the builtin collections.++ A collection met twice (a shared or self-containing value) is refused rather than followed.+ """+ pending_values: list[object] = [default_value]+ seen_collection_ids: set[int] = set()+ while pending_values:+ value = pending_values.pop()+ if _has_one_of_types(value=value, allowed_types=_PLAIN_DEFAULT_SCALAR_TYPES):+ # An aware datetime or time is encoded through its tzinfo's `utcoffset()`, which user code can make raise+ # (`PydanticOmit` drops the property); only the stdlib's `datetime.timezone`, which cannot be subclassed, is inert.+ if isinstance(value, (datetime.datetime, datetime.time)) and value.tzinfo is not None and type(value.tzinfo) is not datetime.timezone:+ return False+ continue+ if not _has_one_of_types(value=value, allowed_types=_PLAIN_DEFAULT_COLLECTION_TYPES) or id(value) in seen_collection_ids:+ return False+ seen_collection_ids.add(id(value))+ if isinstance(value, dict):+ for item_key, item_value in cast("dict[object, object]", value).items():+ pending_values += [item_key, item_value]+ else:+ pending_values.extend(cast("list[object] | tuple[object, ...] | set[object] | frozenset[object]", value))+ return True+++def _is_plain_annotation(*, annotation: Any, plain_or_pending_models: set[type[BaseModel]]) -> bool:+ """Whether an annotation is drawn from the plain vocabulary, recursing into its arguments and nested models."""+ origin = get_origin(annotation)+ if origin is None:+ if _is_any_of(candidate=annotation, options=_PLAIN_LEAF_ANNOTATIONS) or _is_any_of(candidate=annotation, options=_PLAIN_CONTAINER_TYPES):+ return True+ if inspect.isclass(annotation) and issubclass(annotation, BaseModel):+ return _is_plain_model(model_class=annotation, plain_or_pending_models=plain_or_pending_models)+ # A `TypeVar`, a `NewType`, a `TypeAliasType`, a forward reference, an enum, a dataclass, any other class.+ return False+ if origin is Literal:+ return all(_has_one_of_types(value=literal_value, allowed_types=_PLAIN_LITERAL_VALUE_TYPES) for literal_value in get_args(annotation))+ if origin is Annotated:+ base_annotation, *metadata = get_args(annotation)+ if not all(_has_one_of_types(value=metadata_item, allowed_types=_INERT_METADATA_TYPES) for metadata_item in metadata):+ return False+ return _is_plain_annotation(annotation=base_annotation, plain_or_pending_models=plain_or_pending_models)+ if _is_any_of(candidate=origin, options=_UNION_ORIGINS) or _is_any_of(candidate=origin, options=_PLAIN_CONTAINER_TYPES):+ return all(+ type_arg is Ellipsis or _is_plain_annotation(annotation=type_arg, plain_or_pending_models=plain_or_pending_models)+ for type_arg in get_args(annotation)+ )+ # A subscripted `TypeAliasType`, a parametrized generic class, `Callable`, the `collections.abc` generics.+ return False+++def _is_plain_field(*, field_info: FieldInfo, plain_or_pending_models: set[type[BaseModel]]) -> bool:+ """Whether a field is published under its own name, whatever pydantic does with its schema."""+ if field_info.alias is not None or field_info.validation_alias is not None or field_info.serialization_alias is not None:+ return False+ if field_info.json_schema_extra is not None or field_info.field_title_generator is not None or field_info.discriminator is not None:+ return False+ if not all(_has_one_of_types(value=metadata_item, allowed_types=_INERT_METADATA_TYPES) for metadata_item in field_info.metadata):+ return False+ if field_info.default is not PydanticUndefined and not _is_plain_default(default_value=field_info.default):+ return False+ return _is_plain_annotation(annotation=field_info.annotation, plain_or_pending_models=plain_or_pending_models)+++def _is_plain_model(*, model_class: type[BaseModel], plain_or_pending_models: set[type[BaseModel]]) -> bool:+ """Whether a model is plain: its validation-mode JSON schema provably publishes one property per field, named after it.++ Pydantic's `GenerateJsonSchema._named_required_fields_schema` (pydantic/json_schema.py) publishes, in the+ default validation mode, every field under its validation alias, except a field whose schema raises+ `PydanticOmit`. A plain model leaves no room for either: it keeps `BaseModel`'s own schema entry points, sets+ no model-level schema customisation (`json_schema_extra` in any form, a title generator, a mode override, an+ alias generator, `extra='allow'` with its typed extras), declares no validator input type, and every field+ has no alias, no field-level schema customisation, only inert constraint markers, a plain default and an+ annotation from a closed vocabulary — the scalars, `Any`, `Literal` of str, int or bool, unions, the+ builtin containers, `Annotated` with inert markers, and nested models that are plain in turn. That+ vocabulary is an allow-list on purpose: anything it does not name is "not known to be plain", never+ "known to omit".++ `plain_or_pending_models` holds the models already found plain and those still being checked, so that a+ model referring to itself terminates; a model under check counts as plain, which is sound because the walk+ stops at its first refusal and every answer that assumed it feeds into the verdict on that same model.+ """+ if model_class in plain_or_pending_models:+ return True+ plain_or_pending_models.add(model_class)+ if model_class.__pydantic_root_model__ or not model_class.__pydantic_complete__:+ return False+ if not all(_is_inherited_from_base_model(model_class=model_class, attribute_name=entry_point) for entry_point in _MODEL_SCHEMA_ENTRY_POINTS):+ return False+ model_config = model_class.model_config+ schema_shaping_settings = (+ model_config.get("json_schema_extra"),+ model_config.get("model_title_generator"),+ model_config.get("field_title_generator"),+ model_config.get("json_schema_mode_override"),+ model_config.get("alias_generator"),+ )+ if any(setting is not None for setting in schema_shaping_settings) or model_config.get("extra") == "allow":+ return False+ for field_validator in model_class.__pydantic_decorators__.field_validators.values():+ if field_validator.info.json_schema_input_type is not PydanticUndefined:+ return False+ return all(+ _is_plain_field(field_info=field_info, plain_or_pending_models=plain_or_pending_models) for field_info in model_class.model_fields.values()+ )+++def _are_property_sets_certainly_different(*, class_1: type[Any], class_2: type[Any]) -> bool:+ """Whether the two classes certainly differ in which properties their JSON schemas publish, decided without generating one.++ It answers only for two plain models (see `_is_plain_model`), whose published property sets provably equal+ their field-name sets, so different field names mean different properties. Where the schema path cannot+ generate a schema, its field-by-field fallback compares those same field names, so both of+ `are_classes_equivalent`'s paths would say `False` too. `False` means "no shortcut", never "equivalent".+ """+ if not (inspect.isclass(class_1) and issubclass(class_1, BaseModel) and inspect.isclass(class_2) and issubclass(class_2, BaseModel)):+ return False+ if set(class_1.model_fields) == set(class_2.model_fields):+ return False+ return _is_plain_model(model_class=class_1, plain_or_pending_models=set()) and _is_plain_model(model_class=class_2, plain_or_pending_models=set())++def are_classes_equivalent(class_1: type[Any], class_2: type[Any]) -> bool:"""Check if two Pydantic classes are structurally equivalent (same fields, types).This compares the structural parts of the JSON schema (properties, required fields, type)and ignores metadata like the class title/name and field descriptions.++ Generating a JSON schema is the expensive part, and two answers never need one. A class is+ equivalent to itself. And two plain models with different field names are not equivalent: a+ plain model is one built only from a closed vocabulary of types, constraints and settings (see+ `_is_plain_model`), for which the set of properties pydantic publishes provably equals the set of+ field names. Any pair that is not two plain models, or two plain models with the same field+ names, goes to the full schema comparison, which decides. Library validation asks the second+ question once per pipe output (is this concept an image?), so it is settled before any schema+ is generated."""+ if class_1 is class_2:+ return True+if not (hasattr(class_1, "model_fields") and hasattr(class_2, "model_fields")):return class_1 == class_2+ if _are_property_sets_certainly_different(class_1=class_1, class_2=class_2):+ return False+# Compare model schemas using Pydantic's built-in capabilitiestry:schema_1: dict[str, Any] = class_1.model_json_schema()diff --git a/tests/integration/pipelex/libraries/test_load_prevalidated_crate.py b/tests/integration/pipelex/libraries/test_load_prevalidated_crate.pynew file mode 100644index 000000000..ca60b17c9--- /dev/null+++ b/tests/integration/pipelex/libraries/test_load_prevalidated_crate.py@@ -0,0 +1,139 @@+import pytest+from kajson.class_registry import ClassRegistry+from kajson.kajson_manager import KajsonManager+from pytest_mock import MockerFixture++from pipelex.config import get_config+from pipelex.interpreter_hub import clear_current_library, get_library_manager, get_required_pipe, set_current_library+from pipelex.libraries.library import Library+from pipelex.libraries.library_crate import LibraryCrate+from pipelex.libraries.library_crate_factory import LibraryCrateFactory+from pipelex.mthds_parsing.parser import MthdsParser+from pipelex.pipeline.execution_seams import prepare_pipe_job+from pipelex.system.pipe_run_mode import PipeRunMode++PREVALIDATED_MTHDS = """\+domain = "prevalidated"+description = "A crate loaded with and without re-validation"+main_pipe = "main"++[concept.Idea]+description = "An idea"++[concept.Idea.structure]+title = { type = "text", description = "The title", required = true }+score = { type = "integer", description = "The score", required = true }++[concept.Verdict]+description = "A verdict on an idea"++[concept.Verdict.structure]+summary = { type = "text", description = "The summary", required = true }+flag = { type = "boolean", description = "The flag", required = true }++[pipe.invent_idea]+type = "PipeLLM"+description = "Invent an idea"+output = "Idea"+model = "$testing-structured"+prompt = "Invent an idea."++[pipe.judge_idea]+type = "PipeLLM"+description = "Judge the idea"+inputs = { idea = "Idea" }+output = "Verdict"+model = "$testing-structured"+prompt = "Judge this idea: $idea"++[pipe.main]+type = "PipeSequence"+description = "Invent an idea, then judge it"+output = "Verdict"+steps = [+ { pipe = "invent_idea", result = "idea" },+ { pipe = "judge_idea", result = "verdict" },+]+"""+++def _open_workflow_library(*, library_id: str) -> None:+ """Open a library the way the Temporal workflow does: fresh, with its own seeded class registry, and current."""+ library = get_library_manager().open_fresh_library(library_id=library_id)+ scoped_registry = ClassRegistry()+ scoped_registry.register_classes_dict(KajsonManager.get_class_registry().get_classes_dict())+ library.set_class_registry(scoped_registry)+ set_current_library(library_id=library_id)+++async def _dry_run_main_pipe(*, library_id: str) -> str:+ """Run the crate's main pipe in dry mode on the given library and return the main output's concept ref."""+ pipe = get_required_pipe(pipe_code="prevalidated.main")+ execution_config = get_config().interpreter.pipeline_execution.with_execution_overrides(generate_graph=False, mock_inputs=True)+ pipe_job = await prepare_pipe_job(+ storage_scope="test/scope",+ user_id="pytest",+ pipe=pipe,+ library_id=library_id,+ execution_config=execution_config,+ pipe_run_mode=PipeRunMode.DRY,+ pipeline_run_id=f"{library_id}-run",+ )+ pipe_output = await pipe.run_pipe(+ job_metadata=pipe_job.job_metadata,+ working_memory=pipe_job.get_working_memory(),+ pipe_run_params=pipe_job.pipe_run_params,+ )+ main_stuff = pipe_output.main_stuff+ assert type(main_stuff.content).__name__ == main_stuff.concept.structure_class_name+ return main_stuff.concept.concept_ref+++def _make_crate() -> LibraryCrate:+ blueprint = MthdsParser.make_pipelex_bundle_blueprint(mthds_content=PREVALIDATED_MTHDS, mthds_source="prevalidated.mthds")+ return LibraryCrateFactory.make_from_blueprints(blueprints=[blueprint])++[email protected](loop_scope="class")+class TestLoadPrevalidatedCrate:+ async def test_prevalidated_load_skips_only_validation(self, mocker: MockerFixture):+ """The same crate loads both ways into equal libraries, each runs its main pipe, and only the trusting load skips validation."""+ library_manager = get_library_manager()+ crate = _make_crate()+ validate_spy = mocker.spy(Library, "validate_library")++ validating_library_id = "prevalidated-crate-validating"+ trusting_library_id = "prevalidated-crate-trusting"+ opened_library_ids: list[str] = []+ try:+ _open_workflow_library(library_id=validating_library_id)+ opened_library_ids.append(validating_library_id)+ validating_pipes = library_manager.load_from_crate(library_id=validating_library_id, crate=crate)+ assert validate_spy.call_count == 1+ validating_library = library_manager.get_library(library_id=validating_library_id)+ assert await _dry_run_main_pipe(library_id=validating_library_id) == "prevalidated.Verdict"++ _open_workflow_library(library_id=trusting_library_id)+ opened_library_ids.append(trusting_library_id)+ trusting_pipes = library_manager.load_from_crate(library_id=trusting_library_id, crate=crate, is_crate_prevalidated=True)+ assert validate_spy.call_count == 1+ trusting_library = library_manager.get_library(library_id=trusting_library_id)+ assert await _dry_run_main_pipe(library_id=trusting_library_id) == "prevalidated.Verdict"++ # Everything but the validation happened: the same pipes, concepts and domains, the crate's+ # dynamic classes registered in the trusting library's own registry, and its fingerprint recorded.+ assert sorted(pipe.pipe_ref for pipe in trusting_pipes) == sorted(pipe.pipe_ref for pipe in validating_pipes)+ assert set(trusting_library.pipe_library.root) == set(validating_library.pipe_library.root)+ assert set(trusting_library.concept_library.root) == set(validating_library.concept_library.root)+ assert set(trusting_library.domain_library.root) == set(validating_library.domain_library.root)+ trusting_registry = trusting_library.get_class_registry()+ assert trusting_registry is not None+ for concept_ref in ("prevalidated.Idea", "prevalidated.Verdict"):+ structure_class_name = trusting_library.concept_library.get_required_concept(concept_ref=concept_ref).structure_class_name+ assert trusting_registry.has_class(name=structure_class_name)+ assert library_manager.is_crate_loaded(library_id=trusting_library_id, fingerprint=crate.fingerprint)+ assert library_manager.load_from_crate(library_id=trusting_library_id, crate=crate, is_crate_prevalidated=True) == []+ finally:+ clear_current_library()+ for library_id in opened_library_ids:+ library_manager.teardown(library_id=library_id)diff --git a/tests/unit/pipelex/tools/typing/test_class_equivalence_shortcuts.py b/tests/unit/pipelex/tools/typing/test_class_equivalence_shortcuts.pynew file mode 100644index 000000000..68e53abad--- /dev/null+++ b/tests/unit/pipelex/tools/typing/test_class_equivalence_shortcuts.py@@ -0,0 +1,211 @@+import itertools++import pydantic.main as pydantic_main+import pytest+from pydantic import BaseModel+from pydantic_core import PydanticOmit+from pytest_mock import MockerFixture++from pipelex.core.concepts.helpers import make_qualified_structure_class_name+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.concepts.structure_generation.generator import StructureGenerator+from pipelex.core.stuffs.composite_content import CompositeContent+from pipelex.core.stuffs.image_content import ImageContent+from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.mermaid_content import MermaidContent+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.core.stuffs.stuff_content import StuffContent+from pipelex.core.stuffs.text_content import TextContent+from pipelex.tools.typing import class_utils+from pipelex.tools.typing.class_utils import are_classes_equivalent+from tests.unit.pipelex.tools.typing.test_data import (+ CodeAliasedAsRef,+ Coded,+ ConstrainedPerson,+ GeneratedStructureTestCases,+ Invoice,+ Named,+ NamedWithHiddenBySchemaExtraClassmethod,+ NamedWithHiddenByTitle,+ NamedWithHiddenByTypedExtras,+ NamedWithHiddenListByTitle,+ NamedWithOmittingAwareDatetime,+ NamedWithOmittingAwareTime,+ NamedWithOmittingDefault,+ NamedWithSkippedField,+ NamedWithSkippedGeneric,+ NamedWithSkippedInteger,+ NamedWithTitledByInteger,+ Person,+ PersonTwin,+ PersonWithTextAge,+ StampedInUtc,+ Titled,+ TitledByAlias,+ TitledByAliasChoices,+ TitledByGenerator,+ TitledByOmission,+ TreeNode,+ UpperTitled,+ WithOmittingTypedExtras,+)++_SHORTCUT_NAME = "_are_property_sets_certainly_different"+++def _generate_structure_classes() -> list[type[BaseModel]]:+ """Generate the test structures the way a library load does, resolving their references to each other."""+ generated_classes: dict[str, type[BaseModel]] = {}+ for concept_code, structure_blueprint in GeneratedStructureTestCases.BLUEPRINTS.items():+ class_name = make_qualified_structure_class_name(domain_code=GeneratedStructureTestCases.DOMAIN, concept_code=concept_code)+ _, generated_class = StructureGenerator(local_domain=GeneratedStructureTestCases.DOMAIN).generate_from_structure_blueprint(+ class_name=class_name, structure_blueprint=structure_blueprint+ )+ assert issubclass(generated_class, BaseModel)+ generated_classes[class_name] = generated_class+ for generated_class in generated_classes.values():+ generated_class.model_rebuild(_types_namespace=generated_classes)+ assert generated_class.__pydantic_complete__+ return list(generated_classes.values())+++def _content_classes() -> list[type[BaseModel]]:+ """Every native content class pipelex ships as a pydantic model, and a handful of generated structures."""+ native_classes: list[type[BaseModel]] = []+ for native_code in NativeConceptCode:+ structure_class = native_code.structure_class+ if structure_class is not None and issubclass(structure_class, BaseModel) and structure_class not in native_classes:+ native_classes.append(structure_class)+ extra_classes: list[type[BaseModel]] = [StuffContent, StructuredContent, MermaidContent, ListContent, ListContent[TextContent]]+ return [*native_classes, *extra_classes, *_generate_structure_classes()]+++class TestClassEquivalenceShortcuts:+ def test_a_class_is_equivalent_to_itself_without_a_schema(self, mocker: MockerFixture):+ """Identity answers True before any JSON schema is generated, plain model or not."""+ schema_spy = mocker.spy(pydantic_main, "model_json_schema")++ for model_class in (Person, ImageContent, CompositeContent, NamedWithSkippedField, TitledByOmission):+ assert are_classes_equivalent(model_class, class_2=model_class) is True+ assert schema_spy.call_count == 0++ @pytest.mark.parametrize(+ ("class_1", "class_2"),+ [+ pytest.param(Person, Invoice, id="disjoint-fields"),+ pytest.param(Person, Named, id="subset-of-fields"),+ pytest.param(Invoice, ImageContent, id="structure-vs-native-image"),+ pytest.param(ConstrainedPerson, Named, id="inert-constraints-gt-strict-min-length"),+ pytest.param(TreeNode, Titled, id="self-referencing-model"),+ pytest.param(StampedInUtc, Person, id="aware-temporal-defaults-in-stdlib-time-zones"),+ ],+ )+ def test_plain_models_with_different_fields_are_rejected_without_a_schema(+ self, mocker: MockerFixture, class_1: type[BaseModel], class_2: type[BaseModel]+ ):+ """Two plain models whose field names differ are not equivalent, and that is settled without model_json_schema."""+ schema_spy = mocker.spy(pydantic_main, "model_json_schema")++ assert are_classes_equivalent(class_1, class_2=class_2) is False+ assert are_classes_equivalent(class_2, class_2=class_1) is False+ assert schema_spy.call_count == 0++ @pytest.mark.parametrize(+ ("class_1", "class_2", "expected"),+ [+ pytest.param(Person, PersonTwin, True, id="same-fields-same-types"),+ pytest.param(Person, PersonWithTextAge, False, id="same-fields-different-types"),+ ],+ )+ def test_plain_models_with_the_same_fields_are_decided_by_the_schema(+ self, mocker: MockerFixture, class_1: type[BaseModel], class_2: type[BaseModel], expected: bool+ ):+ """Same field names: the shortcut steps aside and the full schema comparison gives the verdict."""+ schema_spy = mocker.spy(pydantic_main, "model_json_schema")++ assert are_classes_equivalent(class_1, class_2=class_2) is expected+ assert schema_spy.call_count == 2++ @pytest.mark.parametrize(+ "class_1",+ [+ pytest.param(NamedWithSkippedGeneric, id="subscripted-generic-type-alias-to-skip-json-schema"),+ pytest.param(NamedWithSkippedInteger, id="type-alias-to-skip-json-schema"),+ pytest.param(NamedWithHiddenByTitle, id="nested-model-title-generator-omits"),+ pytest.param(NamedWithHiddenListByTitle, id="list-of-nested-model-title-generator-omits"),+ pytest.param(NamedWithHiddenBySchemaExtraClassmethod, id="nested-json-schema-extra-classmethod-omits"),+ pytest.param(NamedWithHiddenByTypedExtras, id="nested-typed-extras-omit"),+ pytest.param(NamedWithOmittingDefault, id="default-whose-type-omits"),+ pytest.param(NamedWithSkippedField, id="skip-json-schema-field"),+ pytest.param(NamedWithOmittingAwareDatetime, id="datetime-default-whose-time-zone-omits"),+ pytest.param(NamedWithOmittingAwareTime, id="time-default-whose-time-zone-omits"),+ ],+ )+ def test_a_field_pydantic_omits_falls_through_and_stays_equivalent(self, mocker: MockerFixture, class_1: type[BaseModel]):+ """The field names differ from `Named`'s but the published properties do not: the schema path decides, and says True."""+ assert set(class_1.model_fields) != set(Named.model_fields)+ assert set(class_1.model_json_schema()["properties"]) == set(Named.model_json_schema()["properties"])+ schema_spy = mocker.spy(pydantic_main, "model_json_schema")++ assert are_classes_equivalent(class_1, class_2=Named) is True+ assert are_classes_equivalent(Named, class_2=class_1) is True+ assert schema_spy.call_count == 4++ @pytest.mark.parametrize(+ ("class_1", "class_2", "expected_error"),+ [+ pytest.param(TitledByOmission, Named, PydanticOmit, id="model-title-generator-omits"),+ pytest.param(WithOmittingTypedExtras, Named, PydanticOmit, id="typed-extras-omit"),+ pytest.param(NamedWithTitledByInteger, Person, TypeError, id="nested-model-title-generator-returns-an-integer"),+ ],+ )+ def test_a_schema_that_cannot_be_generated_raises_instead_of_answering(+ self, class_1: type[BaseModel], class_2: type[BaseModel], expected_error: type[BaseException]+ ):+ """Where generating the schema raises, the shortcut does not answer False in its place: the error reaches the caller."""+ assert set(class_1.model_fields) != set(class_2.model_fields)++ with pytest.raises(expected_error):+ are_classes_equivalent(class_1, class_2=class_2)++ @pytest.mark.parametrize(+ ("class_1", "class_2", "expected"),+ [+ pytest.param(TitledByAlias, Titled, True, id="alias"),+ pytest.param(TitledByAliasChoices, Titled, True, id="alias-choices-first-single-string-path"),+ pytest.param(TitledByGenerator, UpperTitled, True, id="alias-generator"),+ pytest.param(CodeAliasedAsRef, Coded, False, id="same-field-names-different-aliases"),+ ],+ )+ def test_an_aliased_field_falls_through_to_the_schema(+ self, mocker: MockerFixture, class_1: type[BaseModel], class_2: type[BaseModel], expected: bool+ ):+ """An alias publishes a property under another name than the field's, so only the schemas can compare it."""+ schema_spy = mocker.spy(pydantic_main, "model_json_schema")++ assert are_classes_equivalent(class_1, class_2=class_2) is expected+ assert schema_spy.call_count == 2++ def test_the_shortcut_agrees_with_the_schema_on_pipelex_content_classes(self, mocker: MockerFixture):+ """Wherever the shortcut answers "certainly different" over pipelex's own content classes, the schema path says not equivalent.++ Both paths are symmetric in their two classes, so each unordered pair is asked once.+ """+ content_classes = _content_classes()+ shortcut_spy = mocker.spy(class_utils, _SHORTCUT_NAME)++ answered_pairs: list[tuple[type[BaseModel], type[BaseModel]]] = []+ for class_1, class_2 in itertools.combinations(content_classes, 2):+ are_classes_equivalent(class_1, class_2=class_2)+ if shortcut_spy.spy_return is True:+ answered_pairs.append((class_1, class_2))+ mocker.stop(shortcut_spy)+ assert any({class_1, class_2} == {ImageContent, TextContent} for class_1, class_2 in answered_pairs)++ mocker.patch.object(class_utils, _SHORTCUT_NAME, return_value=False)+ disagreements = [+ (class_1.__name__, class_2.__name__)+ for class_1, class_2 in answered_pairs+ if are_classes_equivalent(class_1, class_2=class_2) is not False+ ]+ assert not disagreementsdiff --git a/tests/unit/pipelex/tools/typing/test_data.py b/tests/unit/pipelex/tools/typing/test_data.pynew file mode 100644index 000000000..74f55bb7e--- /dev/null+++ b/tests/unit/pipelex/tools/typing/test_data.py@@ -0,0 +1,270 @@+import datetime+from typing import Annotated, Any, ClassVar, TypeVar, cast++from annotated_types import Gt+from pydantic import AliasChoices, AliasPath, BaseModel, ConfigDict, Field+from pydantic.config import JsonDict+from pydantic.json_schema import SkipJsonSchema+from pydantic.types import Strict+from pydantic_core import PydanticOmit+from typing_extensions import TypeAliasType, override++from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType+from pipelex.tools.typing.pydantic_utils import empty_list_factory_of++# Plain models: built only from the vocabulary the class-equivalence shortcut reads.+++class Person(BaseModel):+ name: str+ age: int+++class PersonTwin(BaseModel):+ name: str+ age: int+++class PersonWithTextAge(BaseModel):+ name: str+ age: str+++class Invoice(BaseModel):+ title: str+ summary: str+ score: int+ flag: bool+++class Named(BaseModel):+ name: str+++class Titled(BaseModel):+ title: str+++class StampedInUtc(BaseModel):+ name: str+ stamp: datetime.datetime = datetime.datetime(2020, 1, 1, tzinfo=datetime.UTC)+ alarm: datetime.time = datetime.time(12, 0, tzinfo=datetime.timezone(datetime.timedelta(hours=2)))+++class ConstrainedPerson(BaseModel):+ name: str = Field(min_length=1, strict=True)+ age: int = Field(default=1, gt=0)+ scores: list[Annotated[int, Gt(0), Strict()]] = Field(default_factory=empty_list_factory_of(int))+++class TreeNode(BaseModel):+ label: str+ children: list["TreeNode"] | None = None+++# Shapes a deny-list walk counted as published, though pydantic drops them from the JSON schema: each model here+# publishes the same properties as `Named`, so it is equivalent to it.++_T = TypeVar("_T")+SkippedGeneric = TypeAliasType("SkippedGeneric", SkipJsonSchema[_T], type_params=(_T,))+SkippedInteger = TypeAliasType("SkippedInteger", SkipJsonSchema[int])+++def omit_model_title(_model_class: type) -> str:+ raise PydanticOmit+++def omit_model_schema_extra(_model_class: type, _schema: JsonDict) -> None:+ raise PydanticOmit+++class HiddenByTitle(BaseModel):+ model_config = ConfigDict(model_title_generator=omit_model_title)+ value: int+++class HiddenBySchemaExtraClassmethod(BaseModel):+ model_config = ConfigDict(json_schema_extra=cast("Any", classmethod(omit_model_schema_extra)))+ value: int+++class HiddenByTypedExtras(BaseModel):+ model_config = ConfigDict(extra="allow")+ __pydantic_extra__: dict[str, SkipJsonSchema[int]] # pyright: ignore[reportIncompatibleVariableOverride]+ value: int+++class NamedWithSkippedGeneric(BaseModel):+ name: str+ hidden: SkippedGeneric[int] = 0+++class NamedWithSkippedInteger(BaseModel):+ name: str+ hidden: SkippedInteger = 0+++class NamedWithHiddenByTitle(BaseModel):+ name: str+ hidden: HiddenByTitle | None = None+++class NamedWithHiddenListByTitle(BaseModel):+ name: str+ hidden: list[HiddenByTitle] = Field(default_factory=empty_list_factory_of(HiddenByTitle))+++class NamedWithHiddenBySchemaExtraClassmethod(BaseModel):+ name: str+ hidden: HiddenBySchemaExtraClassmethod | None = None+++class NamedWithHiddenByTypedExtras(BaseModel):+ name: str+ hidden: HiddenByTypedExtras | None = None+++class OmittingDefault:+ """A value whose type drops any property it is the default of: pydantic encodes a default through its own type."""++ @classmethod+ def __get_pydantic_core_schema__(cls, source_type: Any, handler: Any) -> Any: # ruff: ignore[bad-dunder-method-name] - pydantic hook+ raise PydanticOmit+++class NamedWithOmittingDefault(BaseModel):+ name: str+ hidden: int = cast("int", OmittingDefault())+++class NamedWithSkippedField(BaseModel):+ name: str+ hidden: SkipJsonSchema[int] = 0+++class OmittingTimeZone(datetime.tzinfo):+ """A time zone that drops any property whose default it is set on: pydantic asks it for the offset to encode the default."""++ @override+ def utcoffset(self, _dt: datetime.datetime | None) -> datetime.timedelta:+ raise PydanticOmit++ @override+ def dst(self, _dt: datetime.datetime | None) -> datetime.timedelta:+ return datetime.timedelta(0)++ @override+ def tzname(self, _dt: datetime.datetime | None) -> str:+ return "Omitting"+++class NamedWithOmittingAwareDatetime(BaseModel):+ name: str+ hidden: datetime.datetime = datetime.datetime(2020, 1, 1, tzinfo=OmittingTimeZone())+++class NamedWithOmittingAwareTime(BaseModel):+ name: str+ hidden: datetime.time = datetime.time(12, 0, tzinfo=OmittingTimeZone())+++# Shapes whose JSON schema cannot be generated: the error must reach the caller, not be hidden behind a `False`.+++def integer_model_title(_model_class: type) -> Any:+ return 42+++class TitledByOmission(BaseModel):+ model_config = ConfigDict(model_title_generator=omit_model_title)+ name: str+ age: int+++class WithOmittingTypedExtras(BaseModel):+ model_config = ConfigDict(extra="allow")+ __pydantic_extra__: dict[str, SkipJsonSchema[int]] # pyright: ignore[reportIncompatibleVariableOverride]+ name: str+ age: int+++class TitledByInteger(BaseModel):+ model_config = ConfigDict(model_title_generator=integer_model_title)+ value: int+++class NamedWithTitledByInteger(BaseModel):+ name: str+ nested: TitledByInteger | None = None+++# Aliases: a published name that is not the field name, so the shortcut must leave these to the schemas.+++class TitledByAlias(BaseModel):+ name: str = Field(alias="title")+++class TitledByAliasChoices(BaseModel):+ name: str = Field(validation_alias=AliasChoices(AliasPath("payload", 0), "title"))+++class UpperTitled(BaseModel):+ TITLE: str+++class TitledByGenerator(BaseModel):+ model_config = ConfigDict(alias_generator=str.upper)+ title: str+++class CodeAliasedAsRef(BaseModel):+ code: str = Field(alias="ref")+++class Coded(BaseModel):+ code: str+++class GeneratedStructureTestCases:+ DOMAIN: ClassVar[str] = "shortcut"+ BLUEPRINTS: ClassVar[dict[str, dict[str, ConceptStructureBlueprint]]] = {+ "Scorecard": {+ "title": ConceptStructureBlueprint(description="The title", type=ConceptStructureBlueprintFieldType.TEXT, required=True),+ "summary": ConceptStructureBlueprint(description="The summary", type=ConceptStructureBlueprintFieldType.TEXT, required=True),+ "score": ConceptStructureBlueprint(description="The score", type=ConceptStructureBlueprintFieldType.INTEGER, required=True),+ "flag": ConceptStructureBlueprint(description="The flag", type=ConceptStructureBlueprintFieldType.BOOLEAN, required=True),+ },+ "Measurement": {+ "value": ConceptStructureBlueprint(description="The value", type=ConceptStructureBlueprintFieldType.NUMBER, required=True),+ "unit": ConceptStructureBlueprint(description="The unit", choices=["cm", "m"], required=True),+ "taken_on": ConceptStructureBlueprint(description="The day", type=ConceptStructureBlueprintFieldType.DATE),+ "taken_at": ConceptStructureBlueprint(description="The moment", type=ConceptStructureBlueprintFieldType.DATETIME),+ "time_of_day": ConceptStructureBlueprint(description="The time of day", type=ConceptStructureBlueprintFieldType.TIME),+ "notes": ConceptStructureBlueprint(description="Notes", type=ConceptStructureBlueprintFieldType.TEXT, default_value="none"),+ },+ "Tagged": {+ "tags": ConceptStructureBlueprint(description="The tags", type=ConceptStructureBlueprintFieldType.LIST, item_type="text"),+ "attributes": ConceptStructureBlueprint(+ description="The attributes", type=ConceptStructureBlueprintFieldType.DICT, key_type="text", value_type="integer"+ ),+ "payload": ConceptStructureBlueprint(description="Anything", type=ConceptStructureBlueprintFieldType.LIST),+ },+ "Illustrated": {+ "picture": ConceptStructureBlueprint(+ description="The picture", type=ConceptStructureBlueprintFieldType.CONCEPT, concept_ref="native.Image", required=True+ ),+ "gallery": ConceptStructureBlueprint(+ description="More pictures", type=ConceptStructureBlueprintFieldType.LIST, item_type="concept", item_concept_ref="native.Image"+ ),+ },+ "Node": {+ "label": ConceptStructureBlueprint(description="The label", type=ConceptStructureBlueprintFieldType.TEXT, required=True),+ "children": ConceptStructureBlueprint(+ description="The children", type=ConceptStructureBlueprintFieldType.LIST, item_type="concept", item_concept_ref="shortcut.Node"+ ),+ },+ "Quote": {+ "text": ConceptStructureBlueprint(description="The text", type=ConceptStructureBlueprintFieldType.TEXT, required=True),+ },+ }