Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Add is_crate_loaded query for parent-scope crate reuse (#1050)
Louis Choquele35538717590
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex bf811d76f..b5f2778e1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,11 @@# Changelog+## [Unreleased]++### Added++- **`LibraryManagerAbstract.is_crate_loaded(*, library_id, fingerprint)`** — public query over the per-library crate-fingerprint bookkeeping that already backs `load_from_crate`'s idempotency. A `True` answer means the library's ClassRegistry holds the crate's dynamic classes, so callers (e.g. `pipelex-transport`'s `scoped_library_for_crate`) can hydrate within an existing scope instead of opening a fresh one — preserving dynamic-class identity with instances that scope already produced. This is the core half of the fix for the PipeParallel same-concept combine failing over Temporal with `expected X, got X` pydantic model-type errors.+## [v0.39.1] - 2026-07-15### Changeddiff --git a/docs/under-the-hood/runtime-bridge-and-transport.md b/docs/under-the-hood/runtime-bridge-and-transport.mdindex f9bc6ec74..66955b4de 100644--- a/docs/under-the-hood/runtime-bridge-and-transport.md+++ b/docs/under-the-hood/runtime-bridge-and-transport.md@@ -93,7 +93,7 @@ A host runtime's data converter deserializes a payload (a job input, a return vaBoth `PipeJob` and `PipeOutput` carry a `working_memory_raw` field — a plain JSON dict — alongside the typed `working_memory`. The raw dict needs no class resolution to cross a boundary, which decouples transport from class registration.- **Input dehydration** — before dispatch, the working memory is moved into `working_memory_raw` (a plain dict produced by `WorkingMemory.dump_for_transport()`, which stays in open core at `pipelex/core/memory/working_memory.py`). After the worker loads the crate, the job's entry point hydrates the raw dict back to a typed `WorkingMemory`, inside the job's scope.-- **Output dehydration** — before a nested job returns, its working memory is dehydrated to `working_memory_raw`. The raw dict then flows **end-to-end** — nested job → parent → delivery — without intermediate rehydration, because parents and delivery workers generally have not loaded the crate. Only the **top-level submitter** rehydrates.+- **Output dehydration** — before a nested job returns, its working memory is dehydrated to `working_memory_raw`. A **parent job** that consumes the nested output (a controller combining branch results — PipeParallel, PipeBatch) rehydrates it on receive, inside its **own already-loaded scope**: the parent's library carries the same crate (checked by fingerprint via `is_crate_loaded`), so hydration rebinds the nested output to the parent's own dynamic classes — the identity the parent's models reference. **Delivery** workers never load the crate — they render from the raw dict, only best-effort hydrating the main stuff via globally registered classes (falling back to a generic dict render when the class isn't registered), so they stay crate-free. The **top-level submitter** rehydrates too, opening a fresh scoped library when it hasn't loaded the crate itself.The worker-side hydration helper (`pipelex/runtime_bridge/primitives/hydration.py`) is open core: it iterates the raw dict, uses `StuffContentFactory` to rebuild typed content from the registered classes, and preserves aliases. It also guards against cross-exec class-identity mismatches (`_validate_as_known_class` round-trips items through `model_dump()`): the dynamic classes a worker re-execs from the crate are new Python identities sharing a name with whatever a data converter eagerly rebuilt, and a naive `model_validate` would reject the older instance.@@ -103,7 +103,7 @@ The worker-side hydration helper (`pipelex/runtime_bridge/primitives/hydration.p|----------|-----------|--------------|----------------------|| Submitter → top worker job | input | `PipeJob` (`working_memory_raw` + `library_crate`) | Yes — worker loads crate, then hydrates in scope || Parent job → nested job | input | Same `PipeJob` shape (crate re-loaded idempotently) | Yes — in the nested job's own scope |-| Nested job → parent | output | `PipeOutput` (`working_memory_raw`) | **No** — parent forwards the raw dict |+| Nested job → parent | output | `PipeOutput` (`working_memory_raw`) | Yes — in the parent's own scope, whose library already carries the crate (fingerprint match), so branch instances rebind to the parent's class identities || Parent job → delivery | result arg | `PipeOutput` (`working_memory_raw`, no crate) | **No** — delivery renders from raw, best-effort local hydration of the main stuff using only globally registered classes || Top worker job → submitter | output | `PipeOutput` (`working_memory_raw`) | Yes — the submitter rehydrates, opening a fresh scoped library if a crate is available |diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.pyindex 2264b3d56..e61cbb426 100644--- a/pipelex/libraries/library_manager.py+++ b/pipelex/libraries/library_manager.py@@ -374,6 +374,18 @@ class LibraryManager(LibraryManagerAbstract):log.verbose(f"Loading MTHDS files from: {[str(p) for p in valid_mthds_paths]}")return self._load_mthds_files_into_library(library_id=library_id, valid_mthds_paths=valid_mthds_paths)+ @override+ def is_crate_loaded(self, *, library_id: str, fingerprint: str) -> bool:+ """Whether a crate with this fingerprint was already loaded into this library.++ Backed by the same per-library fingerprint bookkeeping that makes load_from_crate+ idempotent, so a True answer means the library's ClassRegistry already holds the+ crate's dynamic classes. Callers can use this to hydrate within an existing scope+ instead of opening a fresh one — preserving dynamic-class identity with instances+ the scope already produced.+ """+ return fingerprint in self._loaded_fingerprints.get(library_id, set())+@overridedef load_from_crate(self, *, library_id: str, crate: LibraryCrate) -> list[PipeAbstract]:"""Load a LibraryCrate into a live Library.@@ -523,7 +535,17 @@ class LibraryManager(LibraryManagerAbstract):)# Load from crate (domains, concepts, pipes, validation)- return self.load_from_crate(library_id=library_id, crate=crate)+ all_pipes = self.load_from_crate(library_id=library_id, crate=crate)++ # Also record the aggregate crate fingerprint: get_crate() rebuilds one crate from+ # ALL accumulated blueprints, so once the library holds more than one batch its+ # fingerprint differs from every per-batch fingerprint recorded by load_from_crate.+ # Recorded only after the load succeeds, so a failed batch (whose blueprints were+ # already accumulated above) never registers a phantom fingerprint.+ if aggregate_crate := self.get_crate(library_id=library_id):+ self._loaded_fingerprints.setdefault(library_id, set()).add(aggregate_crate.fingerprint)++ return all_pipesdef _load_concepts_from_blueprints(self,diff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.pyindex bf9b8171f..2e45e793b 100644--- a/pipelex/libraries/library_manager_abstract.py+++ b/pipelex/libraries/library_manager_abstract.py@@ -67,6 +67,23 @@ class LibraryManagerAbstract(ABC):"""return None+ def is_crate_loaded(self, *, library_id: str, fingerprint: str) -> bool: # noqa: ARG002+ """Whether a crate with this fingerprint was already loaded into this library.++ Returns False by default (managers that don't track crate fingerprints report+ nothing as loaded). Overridden by LibraryManager, which keeps per-library+ fingerprint bookkeeping for load idempotency.++ Args:+ library_id: The library to query.+ fingerprint: The crate fingerprint to look up.++ Returns:+ True when the crate is already loaded into the library, False otherwise+ (including when the library_id is unknown).+ """+ return False+@abstractmethoddef get_crate(self, library_id: str) -> LibraryCrate | None:"""Build a LibraryCrate from all accumulated blueprints for a given library_id.diff --git a/tests/unit/pipelex/libraries/test_library_crate_accumulation.py b/tests/unit/pipelex/libraries/test_library_crate_accumulation.pyindex a61e19dd0..9af0e00be 100644--- a/tests/unit/pipelex/libraries/test_library_crate_accumulation.py+++ b/tests/unit/pipelex/libraries/test_library_crate_accumulation.py@@ -97,6 +97,54 @@ class TestLibraryCrateAccumulation:library_manager.teardown(library_id=library_id)clear_current_library()+ def test_is_crate_loaded_tracks_fingerprint_lifecycle(self):+ """is_crate_loaded() is False before load, True after, and False again after teardown."""+ library_manager = get_library_manager()+ library_id, _ = library_manager.open_library()+ set_current_library(library_id=library_id)+ try:+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.SCORING_BUNDLE])++ crate = library_manager.get_crate(library_id=library_id)+ assert crate is not None++ second_library_id = "is-crate-loaded-test-lib"+ library_manager.open_library(library_id=second_library_id)++ assert library_manager.is_crate_loaded(library_id="nonexistent", fingerprint=crate.fingerprint) is False+ assert library_manager.is_crate_loaded(library_id=second_library_id, fingerprint=crate.fingerprint) is False++ library_manager.load_from_crate(library_id=second_library_id, crate=crate)+ assert library_manager.is_crate_loaded(library_id=second_library_id, fingerprint=crate.fingerprint) is True+ assert library_manager.is_crate_loaded(library_id=second_library_id, fingerprint="some-other-fingerprint") is False++ library_manager.teardown(library_id=second_library_id)+ assert library_manager.is_crate_loaded(library_id=second_library_id, fingerprint=crate.fingerprint) is False+ finally:+ library_manager.teardown(library_id=library_id)+ clear_current_library()++ def test_is_crate_loaded_matches_aggregate_crate_after_multi_batch_load(self):+ """is_crate_loaded() recognizes the aggregate crate fingerprint after multi-batch loads.++ get_crate() rebuilds a single crate from all accumulated blueprints, so once a library+ is populated in more than one load batch its fingerprint differs from every per-batch+ fingerprint. A caller transporting that aggregate crate must still get True.+ """+ library_manager = get_library_manager()+ library_id, _ = library_manager.open_library()+ set_current_library(library_id=library_id)+ try:+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.SCORING_BUNDLE])+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.ANALYTICS_BUNDLE])++ crate = library_manager.get_crate(library_id=library_id)+ assert crate is not None+ assert library_manager.is_crate_loaded(library_id=library_id, fingerprint=crate.fingerprint) is True+ finally:+ library_manager.teardown(library_id=library_id)+ clear_current_library()+def test_teardown_clears_blueprints_for_library_id(self):"""teardown(library_id) clears accumulated blueprints for that library_id."""library_manager = get_library_manager()