Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Add is_crate_loaded query for parent-scope crate reuse (#1050)

Louis Choquele35538717590
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bf811d76f..b5f2778e1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,11 @@
# Changelog
+## [Unreleased]
+
+### Added
+
+- **`LibraryManagerAbstract.is_crate_loaded(*, library_id, fingerprint)`** — public query over the per-library crate-fingerprint bookkeeping that already backs `load_from_crate`'s idempotency. A `True` answer means the library's ClassRegistry holds the crate's dynamic classes, so callers (e.g. `pipelex-transport`'s `scoped_library_for_crate`) can hydrate within an existing scope instead of opening a fresh one — preserving dynamic-class identity with instances that scope already produced. This is the core half of the fix for the PipeParallel same-concept combine failing over Temporal with `expected X, got X` pydantic model-type errors.
+
## [v0.39.1] - 2026-07-15
### Changed
diff --git a/docs/under-the-hood/runtime-bridge-and-transport.md b/docs/under-the-hood/runtime-bridge-and-transport.md
index f9bc6ec74..66955b4de 100644
--- a/docs/under-the-hood/runtime-bridge-and-transport.md
+++ b/docs/under-the-hood/runtime-bridge-and-transport.md
@@ -93,7 +93,7 @@ A host runtime's data converter deserializes a payload (a job input, a return va
Both `PipeJob` and `PipeOutput` carry a `working_memory_raw` field — a plain JSON dict — alongside the typed `working_memory`. The raw dict needs no class resolution to cross a boundary, which decouples transport from class registration.
- **Input dehydration** — before dispatch, the working memory is moved into `working_memory_raw` (a plain dict produced by `WorkingMemory.dump_for_transport()`, which stays in open core at `pipelex/core/memory/working_memory.py`). After the worker loads the crate, the job's entry point hydrates the raw dict back to a typed `WorkingMemory`, inside the job's scope.
-- **Output dehydration** — before a nested job returns, its working memory is dehydrated to `working_memory_raw`. The raw dict then flows **end-to-end** — nested job → parent → delivery — without intermediate rehydration, because parents and delivery workers generally have not loaded the crate. Only the **top-level submitter** rehydrates.
+- **Output dehydration** — before a nested job returns, its working memory is dehydrated to `working_memory_raw`. A **parent job** that consumes the nested output (a controller combining branch results — PipeParallel, PipeBatch) rehydrates it on receive, inside its **own already-loaded scope**: the parent's library carries the same crate (checked by fingerprint via `is_crate_loaded`), so hydration rebinds the nested output to the parent's own dynamic classes — the identity the parent's models reference. **Delivery** workers never load the crate — they render from the raw dict, only best-effort hydrating the main stuff via globally registered classes (falling back to a generic dict render when the class isn't registered), so they stay crate-free. The **top-level submitter** rehydrates too, opening a fresh scoped library when it hasn't loaded the crate itself.
The worker-side hydration helper (`pipelex/runtime_bridge/primitives/hydration.py`) is open core: it iterates the raw dict, uses `StuffContentFactory` to rebuild typed content from the registered classes, and preserves aliases. It also guards against cross-exec class-identity mismatches (`_validate_as_known_class` round-trips items through `model_dump()`): the dynamic classes a worker re-execs from the crate are new Python identities sharing a name with whatever a data converter eagerly rebuilt, and a naive `model_validate` would reject the older instance.
@@ -103,7 +103,7 @@ The worker-side hydration helper (`pipelex/runtime_bridge/primitives/hydration.p
|----------|-----------|--------------|----------------------|
| Submitter → top worker job | input | `PipeJob` (`working_memory_raw` + `library_crate`) | Yes — worker loads crate, then hydrates in scope |
| Parent job → nested job | input | Same `PipeJob` shape (crate re-loaded idempotently) | Yes — in the nested job's own scope |
-| Nested job → parent | output | `PipeOutput` (`working_memory_raw`) | **No** — parent forwards the raw dict |
+| Nested job → parent | output | `PipeOutput` (`working_memory_raw`) | Yes — in the parent's own scope, whose library already carries the crate (fingerprint match), so branch instances rebind to the parent's class identities |
| Parent job → delivery | result arg | `PipeOutput` (`working_memory_raw`, no crate) | **No** — delivery renders from raw, best-effort local hydration of the main stuff using only globally registered classes |
| Top worker job → submitter | output | `PipeOutput` (`working_memory_raw`) | Yes — the submitter rehydrates, opening a fresh scoped library if a crate is available |
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index 2264b3d56..e61cbb426 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -374,6 +374,18 @@ class LibraryManager(LibraryManagerAbstract):
log.verbose(f"Loading MTHDS files from: {[str(p) for p in valid_mthds_paths]}")
return self._load_mthds_files_into_library(library_id=library_id, valid_mthds_paths=valid_mthds_paths)
+ @override
+ def is_crate_loaded(self, *, library_id: str, fingerprint: str) -> bool:
+ """Whether a crate with this fingerprint was already loaded into this library.
+
+ Backed by the same per-library fingerprint bookkeeping that makes load_from_crate
+ idempotent, so a True answer means the library's ClassRegistry already holds the
+ crate's dynamic classes. Callers can use this to hydrate within an existing scope
+ instead of opening a fresh one — preserving dynamic-class identity with instances
+ the scope already produced.
+ """
+ return fingerprint in self._loaded_fingerprints.get(library_id, set())
+
@override
def load_from_crate(self, *, library_id: str, crate: LibraryCrate) -> list[PipeAbstract]:
"""Load a LibraryCrate into a live Library.
@@ -523,7 +535,17 @@ class LibraryManager(LibraryManagerAbstract):
)
# Load from crate (domains, concepts, pipes, validation)
- return self.load_from_crate(library_id=library_id, crate=crate)
+ all_pipes = self.load_from_crate(library_id=library_id, crate=crate)
+
+ # Also record the aggregate crate fingerprint: get_crate() rebuilds one crate from
+ # ALL accumulated blueprints, so once the library holds more than one batch its
+ # fingerprint differs from every per-batch fingerprint recorded by load_from_crate.
+ # Recorded only after the load succeeds, so a failed batch (whose blueprints were
+ # already accumulated above) never registers a phantom fingerprint.
+ if aggregate_crate := self.get_crate(library_id=library_id):
+ self._loaded_fingerprints.setdefault(library_id, set()).add(aggregate_crate.fingerprint)
+
+ return all_pipes
def _load_concepts_from_blueprints(
self,
diff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.py
index bf9b8171f..2e45e793b 100644
--- a/pipelex/libraries/library_manager_abstract.py
+++ b/pipelex/libraries/library_manager_abstract.py
@@ -67,6 +67,23 @@ class LibraryManagerAbstract(ABC):
"""
return None
+ def is_crate_loaded(self, *, library_id: str, fingerprint: str) -> bool: # noqa: ARG002
+ """Whether a crate with this fingerprint was already loaded into this library.
+
+ Returns False by default (managers that don't track crate fingerprints report
+ nothing as loaded). Overridden by LibraryManager, which keeps per-library
+ fingerprint bookkeeping for load idempotency.
+
+ Args:
+ library_id: The library to query.
+ fingerprint: The crate fingerprint to look up.
+
+ Returns:
+ True when the crate is already loaded into the library, False otherwise
+ (including when the library_id is unknown).
+ """
+ return False
+
@abstractmethod
def get_crate(self, library_id: str) -> LibraryCrate | None:
"""Build a LibraryCrate from all accumulated blueprints for a given library_id.
diff --git a/tests/unit/pipelex/libraries/test_library_crate_accumulation.py b/tests/unit/pipelex/libraries/test_library_crate_accumulation.py
index a61e19dd0..9af0e00be 100644
--- a/tests/unit/pipelex/libraries/test_library_crate_accumulation.py
+++ b/tests/unit/pipelex/libraries/test_library_crate_accumulation.py
@@ -97,6 +97,54 @@ class TestLibraryCrateAccumulation:
library_manager.teardown(library_id=library_id)
clear_current_library()
+ def test_is_crate_loaded_tracks_fingerprint_lifecycle(self):
+ """is_crate_loaded() is False before load, True after, and False again after teardown."""
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ try:
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.SCORING_BUNDLE])
+
+ crate = library_manager.get_crate(library_id=library_id)
+ assert crate is not None
+
+ second_library_id = "is-crate-loaded-test-lib"
+ library_manager.open_library(library_id=second_library_id)
+
+ assert library_manager.is_crate_loaded(library_id="nonexistent", fingerprint=crate.fingerprint) is False
+ assert library_manager.is_crate_loaded(library_id=second_library_id, fingerprint=crate.fingerprint) is False
+
+ library_manager.load_from_crate(library_id=second_library_id, crate=crate)
+ assert library_manager.is_crate_loaded(library_id=second_library_id, fingerprint=crate.fingerprint) is True
+ assert library_manager.is_crate_loaded(library_id=second_library_id, fingerprint="some-other-fingerprint") is False
+
+ library_manager.teardown(library_id=second_library_id)
+ assert library_manager.is_crate_loaded(library_id=second_library_id, fingerprint=crate.fingerprint) is False
+ finally:
+ library_manager.teardown(library_id=library_id)
+ clear_current_library()
+
+ def test_is_crate_loaded_matches_aggregate_crate_after_multi_batch_load(self):
+ """is_crate_loaded() recognizes the aggregate crate fingerprint after multi-batch loads.
+
+ get_crate() rebuilds a single crate from all accumulated blueprints, so once a library
+ is populated in more than one load batch its fingerprint differs from every per-batch
+ fingerprint. A caller transporting that aggregate crate must still get True.
+ """
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ try:
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.SCORING_BUNDLE])
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.ANALYTICS_BUNDLE])
+
+ crate = library_manager.get_crate(library_id=library_id)
+ assert crate is not None
+ assert library_manager.is_crate_loaded(library_id=library_id, fingerprint=crate.fingerprint) is True
+ finally:
+ library_manager.teardown(library_id=library_id)
+ clear_current_library()
+
def test_teardown_clears_blueprints_for_library_id(self):
"""teardown(library_id) clears accumulated blueprints for that library_id."""
library_manager = get_library_manager()