Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Pin kajson to 0.4.2 and scope dynamic enums per-payload (#859)

Louis Choquel631125cbd452
diff --git a/.badges/tests.json b/.badges/tests.json
index 8cfc219b6..f718e693f 100644
--- a/.badges/tests.json
+++ b/.badges/tests.json
@@ -1,7 +1,7 @@
{
"schemaVersion": 1,
"label": "tests",
- "message": "4194",
+ "message": "4854",
"color": "blue",
"cacheSeconds": 300
}
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9f4dd8edc..230acb044 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,24 +1,55 @@
# Changelog
-## [Unreleased]
+## [v0.26.0] - 2026-05-04
-### Added
+### Highlights
-- **`PipeOutput.graph_assembly_error`**: optional `str` field set when the Temporal `act_assemble_graph` activity raises so consumers can distinguish "graph never produced" from "graph assembly failed". Previously the failure was logged as a warning and `graph_spec` silently stayed `None`.
-- **`SecurityError(PipelexError)`**: new base class in `pipelex.base_exceptions` for security-policy violations. `UnsafeSchemaError` now extends it (was `ContentGenerationError`) so security signals are not silently swallowed by domain `except` handlers.
+- **Temporal distributed execution** — Pipelex pipelines can now run as Temporal workflows across worker processes. New `pipelex[temporal]` extra, `pipelex worker` CLI, `PipeRun` layer, and `LibraryCrate` IR for cross-process library propagation.
+- **Distributed tracing** — pluggable event-log backends (NDJSON, DynamoDB, in-memory, buffering) emit cross-worker `TraceEvent`s that reassemble offline into a complete `GraphSpec`. Enabled by default; works in single-process mode too.
+
+### Added
-- **PipeRun layer**: New `PipeRunProtocol` with `PipeRun` (direct) and `TemporalPipeRun` (Temporal) implementations. Wraps pipe execution + delivery in a single orchestration unit. In Temporal mode, `WfPipeRun` workflow orchestrates `WfPipeRouter` as a child workflow followed by delivery activities.
-- **Delivery framework**: `DeliveryAssignment` model with `WebhookDeliveryAssignment` (HTTP POST) and `StorageDeliveryAssignment` (S3/local). Deliveries run as Temporal activities with automatic retry, or inline in direct mode.
-- **`TemporalPipeRun.start()`**: Non-blocking start that returns `workflow_id` + `WorkflowHandle` immediately. Replaces the former `PipeRouterTop.start_pipe_job()`.
-- **Pre-generated `pipeline_run_id` support**: `pipeline_run_setup()`, `PipelineFactory`, and `PipelineManager` accept an optional `pipeline_run_id` parameter for external run record creation.
+#### Temporal distributed execution
+
+- **`pipelex[temporal]` extra and full Temporal integration package (`pipelex/temporal/`)**: includes `temporal_manager`, `temporal_connect`, `temporal_data_converter` (kajson-based JSON), `sandbox_manager`, `task_manager`, and a `pipelex worker` CLI subcommand that boots a worker against a configured queue. Toggle via `[temporal] is_enabled` in `pipelex.toml`. Bumps `temporalio` to `1.23.0`.
+- **`PipeRun` layer (`pipelex/pipe_run/`)**: new `PipeRunProtocol` with `PipeRun` (in-process) and `TemporalPipeRun` (workflow-orchestrated) implementations. Wraps pipe execution + delivery in a single orchestration unit. In Temporal mode, `WfPipeRun` orchestrates `WfPipeRouter` as a child workflow followed by delivery activities. `TemporalPipeRun.start()` is non-blocking and returns `workflow_id` + `WorkflowHandle` immediately.
+- **`TemporalPipeRouter`**: single router that auto-detects context (top-level vs child workflow) and dispatches accordingly.
+- **Delivery framework**: `DeliveryAssignment` model with `WebhookDeliveryAssignment` (HTTP POST) and `StorageDeliveryAssignment` (S3 / local). Deliveries run as Temporal activities with automatic retry, or inline in direct mode.
+- **`LibraryCrate` IR (`pipelex/libraries/library_crate.py`, `library_crate_factory.py`)**: serializable intermediate representation that lets a Temporal worker reconstruct the exact set of libraries (concepts, pipes, structures) required to execute a job — no shared filesystem assumed.
+- **`StoragePayloadCodec`**: offloads Temporal payloads larger than a configurable threshold (default 1 MiB) to external storage (S3 or local) and replaces them with lightweight URI references. Configurable via `[temporal.payload_codec_config]`. New `pipelex codec serve` CLI launches the codec server consumed by the Temporal Web UI for payload inspection.
+- **Schema-based dynamic model construction (`pipelex/cogt/content_generation/schema_to_model_factory.py`)**: rebuilds dynamic Pydantic models on a Temporal worker from the JSON schema embedded in `ObjectAssignment`, instead of relying on cross-process class identity. Cache bounded by an LRU (`_SCHEMA_CACHE_MAX_SIZE = 1024`); codegen serialized through a file lock to kill thundering-herd duplication on long-running workers receiving many distinct schemas. Adds `datamodel-code-generator>=0.55.0` as a runtime dependency.
+- **Worker scopes**: `pipelex worker --scope ...` lets a worker subscribe to a subset of activities/workflows so deployments can specialize workers (e.g. inference workers vs orchestration workers).
+- **Pre-generated `pipeline_run_id` support**: `pipeline_run_setup()`, `PipelineFactory`, and `PipelineManager` accept an optional `pipeline_run_id` parameter for external run-record creation.
- **`workflow_id` in start response**: `PipelexPipelineStartResponse` includes `workflow_id`.
- **Temporal `callbacks` passthrough**: `WorkflowExecutor.start_workflow()` accepts optional Temporal `Callback` objects.
-- **DynamoDB event log backend**: New `DynamoDBEventLog` implementation of `EventLogProtocol` for cloud-based trace event storage. Installable via `pip install "pipelex[dynamodb]"`.
-- **Event log DI**: Event log backend is now configurable via `TracingConfig.backend` (`"ndjson"` or `"dynamodb"`) and injectable via `Pipelex.setup(event_log=...)`. Factory function `make_event_log()` selects the backend from config or hub injection.
-- **Hub event_log support**: `PipelexHub.set_event_log()` / `get_event_log()` for dependency injection of custom event log backends.
+
+#### Distributed tracing (`pipelex/tracing/`)
+
+- **`EventLogProtocol` with pluggable backends**: `NdjsonEventLog`, `DynamoDBEventLog`, `InMemoryEventLog`, `BufferingEventLog`. Emits `TraceEvent`s as pipes start/finish across processes. Configured via `[pipelex.tracing_config]` (`backend = "ndjson" | "dynamodb"`). Tracing is **enabled by default** and decoupled from Temporal — single-process runs also produce a trace.
+- **`pipelex[dynamodb]` extra**: `pip install "pipelex[dynamodb]"` for the DynamoDB event log backend.
+- **`writer_id` on `TraceEvent`**: every event records which writer (worker / process) produced it, propagated through every backend. Enables proper cross-worker dedup and per-writer aggregation.
+- **`GraphSpecAssembler`**: rebuilds a complete `GraphSpec` offline from a stream of `TraceEvent`s — equivalence with the live `GraphTracer` is pinned by tests.
+- **`UsageAggregator`**: aggregates per-pipeline LLM/extract/img-gen usage across workers from the event log.
+- **Hub DI**: `PipelexHub.set_event_log()` / `get_event_log()` for dependency injection of custom event log backends. `make_event_log()` factory selects the backend from config or hub injection; the 3 prior call sites that hardcoded `NdjsonEventLog` now go through it.
+
+#### Inference error classification
+
+- **`InferenceErrorCategory` enum (`transient`, `configuration`, `content`, `capacity`)** with per-provider helpers that distinguish quota exhaustion from rate limits and detect content-policy violations — covers OpenAI, Anthropic, Google, Mistral, AWS Bedrock, and Pipelex Gateway.
+- **All inference workers attach error category + user action**: every LLM, image-gen, extract, and search worker across all providers raises exceptions with an `InferenceErrorCategory` and actionable `user_action` hint.
+- **Structured `ErrorReport`**: `PipelexError.to_error_report()` returns a dataclass with error type, category, retryable flag, user-action hint, model, and provider. CLI error handlers consume `to_error_report()` for consistent, structured display.
+- **`SecurityError(PipelexError)`**: new base class in `pipelex.base_exceptions` for security-policy violations. `UnsafeSchemaError` now extends it (was `ContentGenerationError`) so security signals are not silently swallowed by domain `except` handlers.
+
+#### Documentation
+
+- New "Under the Hood" pages: `pipe-routing-and-execution.md`, `temporal-integration.md`, `distributed-content-generation.md`. Cover routing/dispatch, Temporal architecture, and the dynamic-model + storage-codec serialization story.
+- New `AGENTS.md` at repo root, generated from the same source as `CLAUDE.md` via `make rules`.
+
+#### Other
+
+- **`PipeOutput.graph_assembly_error`**: optional `str` field set when the Temporal `act_assemble_graph` activity raises so consumers can distinguish "graph never produced" from "graph assembly failed". Previously the failure was logged as a warning and `graph_spec` silently stayed `None`.
+- **`pipe extract` web URL support** with full test coverage in `tests/integration/pipelex/pipes/pipe_operators/pipe_extract/`.
+- **`needs_inference` flag on CLI commands**: non-inference subcommands (e.g. `worker`, `doctor`) skip inference setup, shaving cold-start time.
- **Environment-specific config**: `RunEnvironment` enum values updated to full names (`"development"`, `"production"`). Config loaded from `ENVIRONMENT` env var (was `ENV`).
-- **Inference error classification**: New `InferenceErrorCategory` enum (`transient`, `configuration`, `content`, `capacity`) with per-provider helpers that distinguish quota exhaustion from rate limits and detect content policy violations — covers OpenAI, Anthropic, Google, Mistral, AWS Bedrock, and Gateway.
-- **Structured `ErrorReport`**: `PipelexError.to_error_report()` returns a dataclass with error type, category, retryable flag, user action hint, model, and provider.
### Changed
@@ -26,40 +57,26 @@
- Anthropic: bare `"credit"` token replaced with `"credit balance"`, `"out of credits"`, `"insufficient credit"`. No longer misclassifies `"your credit card was declined"` or `"we will credit your account"` as quota exhaustion.
- Google: bare `"billing"` token replaced with `"billing limit"`, `"billing quota"`, `"billing exceeded"`, and `"billing account"`. Continues to match real 429-class messages (`"Billing account is not active"`, `"billing limit reached"`); stops matching unrelated billing-mentioning text.
- Mistral / Gateway: bare `"billing"` likewise narrowed to `"billing limit"` / `"billing quota"`. Mistral additionally distinguishes `"out of credits"` vs `"insufficient credits"`.
-- **Schema codegen cache bounded with LRU eviction**: `SchemaToModelFactory._schema_cache` now uses an `OrderedDict` capped at `_SCHEMA_CACHE_MAX_SIZE = 1024` entries. Prevents unbounded memory growth in long-running Temporal workers receiving many distinct schemas.
- **Removed duplicate `ContentGenerationError` class**: the unused `pipelex.cogt.content_generation.exceptions.ContentGenerationError` was deleted (had zero external imports). Its subclasses re-parented: `NeitherUrlNorDataError` → `PipelexError`, `UnsafeSchemaError` → new `SecurityError`. The temporal `pipelex.temporal.exceptions.ContentGenerationError` is now the only class with that name.
-- **Narrowed `WfPipeRun` exception handling**: `except Exception:` replaced with `ChildWorkflowError` (child-workflow path) and `ActivityError` (graph-assembly path). Real workflow bugs (TypeError, AttributeError, etc.) are no longer masked as pipe failures.
-
-- **`PipeRouterTop` + `PipeRouterChild` merged into `TemporalPipeRouter`**: Single router that auto-detects context (top-level vs child workflow) and dispatches accordingly. Hub no longer needs `_pipe_router_top` — one `_pipe_router` field handles both modes.
-- **`PipelexRunner.execute_pipeline()` uses `get_pipe_run()`**: Runner now delegates to the PipeRun layer instead of calling `get_pipe_router()` directly.
-- **Tracing config structure**: `TracingConfig` now has `backend`, `ndjson` (with `traces_dir`), and `dynamodb` (with `table_name`, `region`) sub-configs. Tracing is enabled by default.
-- **Event log factory replaces hardcoded NdjsonEventLog**: The 3 call sites (`pipeline_run_setup.py`, `wf_pipe_router.py`, `runner.py`) now use `make_event_log()` instead of directly instantiating `NdjsonEventLog`.
-- **Tracing decoupled from Temporal**: Event log is created when `tracing_config.is_enabled`, regardless of whether Temporal is enabled.
+- **Storage providers unified**: `StorageConfig` flattened to inheritance-based config so providers (`local`, `s3`) share a single shape across delivery and trace storage.
- **`make rules` now generates both `CLAUDE.md` and `AGENTS.md` by default**: the `pipelex.kit_config.preferred_agent_target` setting has been renamed to `preferred_agent_targets` and is now a list. The default is `["claude", "agents"]`. Cursor remains exclusive (`["cursor"]`) and cannot be combined with single-file targets. Downstream projects overriding this setting must rename the key and wrap the value in a list.
-- **All inference workers attach error category and user action**: Every worker across all providers now raises exceptions with an `InferenceErrorCategory` and actionable `user_action` hint.
-- **CLI error output wired to `ErrorReport`**: Error handlers use `to_error_report()` for consistent, structured display.
+- **Cold start trimmed**: heavy SDK imports (`boto3`, `huggingface_hub`, etc.) deferred behind `TYPE_CHECKING` or function-local guards across multiple plugins.
- **Graph viewer updated to mthds-ui v0.3.4**: bumped from v0.3.0 — additional polish atop the resizable detail panel, escape-to-close, sticky header, prompt expand/collapse with copy button, concept refinement display.
-- **README install instructions**: Replaced step-by-step Claude Code setup with single copy-paste messages for Claude Code and Codex, added manual install section.
-
-### Removed
-
-- **`PipeRouterTop`** — replaced by `TemporalPipeRouter` + `TemporalPipeRun`.
-- **`PipeRouterChild`** — merged into `TemporalPipeRouter`.
-- **`PipelexHub.set_pipe_router_top()`** — no longer needed; single `set_pipe_router()` suffices.
+- **Default models**: small-vision and creative defaults now point to `gemini-3.0-flash-preview`. Added `claude-4.6-sonnet` and Bedrock token-auth support.
+- **`kajson` upgraded** from `0.3.1` to `0.4.2`.
+- **README install instructions**: replaced step-by-step Claude Code setup with single copy-paste messages for Claude Code and Codex, added manual install section.
### Fixed
- **Anthropic structured generation no longer flattens all errors to `CONTENT`.** `AnthropicLLMWorker._gen_object` runs requests through `instructor`, which wraps SDK exceptions in `InstructorRetryException`. The previous handler categorized every wrapped error as `CONTENT`, so genuine `RateLimitError`, `APITimeoutError`, `APIConnectionError`, `PermissionDeniedError`, and `AuthenticationError` cases never reached the typed branches — they were reported as content failures (non-retryable) instead of `TRANSIENT` / `CAPACITY` / `CONFIGURATION`. The handler now unwraps `InstructorRetryException.failed_attempts[-1].exception` (with a `__cause__`/`RetryError.last_attempt` fallback) and routes recognized SDK exceptions through a shared categorization helper; truly unrecognized errors keep the `CONTENT` fallback.
-- **ObjectAssignment deserialization on Temporal workers**: Removed `__init__` class registry check that blocked deserialization of `ObjectAssignment` before `library_crate` was loaded. Validation moved to `validate_before_execution()` method, following the existing codebase pattern.
-- **Dynamic concept classes in Temporal activities**: `WfPipeRouter` now propagates dynamically registered concept classes from the per-workflow registry to the global registry, so child workflows and activities can access them.
-- **Temporal nondeterminism in child workflow IDs**: Replaced `shortuuid.uuid()` with `workflow.uuid4()` in `TemporalPipeRouter` for deterministic child workflow ID generation. The old code caused TMPRL1100 errors on workflow replay.
-- **Enum fields (`choices`) breaking Temporal serialization**: Concepts with `choices` fields generated `Enum` classes that couldn't be resolved during `model_rebuild` or deserialized across Temporal workflow boundaries. Fixed `_exec_and_extract_class` to include all generated types (models + enums) in the forward reference namespace.
-- **Enum serialization in `ContentGeneratorChild`**: `model_dump()` now uses `mode="json"` to serialize enum values as plain strings, preventing type mismatch when validating across different model class instances.
-- **`Anything[]` hydration in Temporal**: Working memory hydration failed for `Anything` concept lists because `AnythingContent` doesn't exist as a class. Hydration now gracefully falls back to embedded `__class__` metadata or `TextContent` for items without type information.
- **Managed-deck detection no longer misclassifies user TOMLs.** `_is_managed_deck_filename` previously treated any non-`x_custom_*.toml` as kit-managed, so a user file like `my_overrides.toml` dropped into the deck dir would have been reported (and on `pipelex update` overwritten) as if it were a kit file. Now requires the numbered `^\d+_.*\.toml$` prefix that the kit actually ships.
- **`PipeLLM` output structure prompt used the unresolved concept ref.** When `is_structure_prompt_enabled`, `get_output_structure_prompt` was called with `pipe_run_params.dynamic_output_concept_ref or output_stuff_spec.concept.concept_ref`. The first form can be a bare code (no domain), which broke downstream concept resolution. Now passes the already-resolved `output_stuff_spec.concept.concept_ref` directly.
- **`PIPELEX_NO_DECK_NOTICE` suppression was too lax.** Any non-empty value silenced the deck-staleness notice — including `PIPELEX_NO_DECK_NOTICE=0`, which users might reasonably expect to _enable_ the notice. Now requires the documented `PIPELEX_NO_DECK_NOTICE=1`.
- **Shipped `.kit_manifest.json` carried `kit_version: "0.25.0"`** for the 0.25.1 release, which would have triggered false stale-detection on fresh installs. Bumped to `0.25.1` and re-synced into `pipelex/kit/configs/`.
+- **Stored-XSS guard**: HTML escaped in fallback `main_stuff.html`.
+- **`GatewayExtractWorker` teardown**: guarded done-callbacks against cancelled tasks to stop noisy teardown errors.
+- **URL checker**: 401/403 are now treated as OK for auth-walled URLs.
+- **Pipeline duplicate guard**: registering the same pipeline twice raises explicitly instead of silently shadowing.
### Documentation
@@ -68,6 +85,8 @@
### Security
- **`schema_to_model` rejects `x-python-*` codegen extensions and restricts `__import__` to an allowlist.** `datamodel-code-generator` honors the `x-python-import` JSON Schema extension by emitting arbitrary `from <module> import <name>` statements with no sanitization. Combined with the prior gap that `_make_restricted_builtins()` did not block `__import__`, an attacker able to plant a crafted `object_class_schema` on a Temporal payload (where `ObjectAssignment.object_class_schema: dict[str, Any]` round-trips untouched) could cause arbitrary modules to be imported during `exec()` of generated code. Two-layer defense added: (1) `_reject_unsafe_schema_extensions` raises `UnsafeSchemaError` if any `x-python-*` key is present anywhere in the schema; (2) `__import__` in the exec namespace is now wrapped to allow only `pydantic`, `typing`, `typing_extensions`, `enum`, `datetime`, `decimal`, `uuid`, `__future__`, `collections`, and `re`.
+- **Restricted exec builtins, path sanitization, and atomic fingerprint caching** in the schema-to-model pipeline.
+- **CORS hardened on codec server error paths**; no leaked exception strings; prefix guards on storage URIs.
## [v0.25.2] - 2026-04-30
diff --git a/pipelex/cogt/content_generation/schema_to_model_factory.py b/pipelex/cogt/content_generation/schema_to_model_factory.py
index 8cd3f33da..6a29bb9fe 100644
--- a/pipelex/cogt/content_generation/schema_to_model_factory.py
+++ b/pipelex/cogt/content_generation/schema_to_model_factory.py
@@ -34,6 +34,7 @@ import re
import tempfile
import threading
from collections import OrderedDict
+from enum import Enum
from pathlib import Path
from typing import Any, ClassVar, cast
@@ -72,6 +73,11 @@ class SchemaToModelFactory:
_schema_cache: ClassVar[OrderedDict[str, type[BaseModel]]] = OrderedDict()
_cache_lock: ClassVar[threading.Lock] = threading.Lock()
+ _SOURCE_CACHE_MAX_SIZE: ClassVar[int] = 1024
+
+ _source_cache: ClassVar[OrderedDict[str, dict[str, type[Any]]]] = OrderedDict()
+ _source_cache_lock: ClassVar[threading.Lock] = threading.Lock()
+
@classmethod
def make_from_json_schema(cls, schema: dict[str, Any], class_name: str) -> type[BaseModel]:
"""Reconstruct a BaseModel class from a JSON schema dict.
@@ -216,8 +222,13 @@ class SchemaToModelFactory:
return safe_builtins
@classmethod
- def _exec_and_extract_class(cls, source_code: str, class_name: str) -> type[BaseModel]:
- """Execute source code and extract the named BaseModel class."""
+ def _exec_source_to_types(cls, source_code: str) -> dict[str, type[Any]]:
+ """Exec source through restricted builtins, return all top-level BaseModel + Enum
+ subclasses with model_rebuild applied so forward references resolve.
+
+ Internal primitive shared by both make_types_from_source (receiver path) and
+ _exec_and_extract_class (sender path).
+ """
# TODO: replace exec() with programmatic construction via pydantic.create_model().
# Eliminates the exec primitive entirely, removes the need for both security
# layers, and lets us delete _make_restricted_builtins/_restricted_import.
@@ -226,11 +237,52 @@ class SchemaToModelFactory:
namespace: dict[str, Any] = {"__builtins__": cls._make_restricted_builtins()}
exec(compile(source_code, "<schema_to_model_factory>", "exec"), namespace)
- # Collect all BaseModel subclasses from the exec'd namespace
- model_classes: dict[str, type[BaseModel]] = {
- name: obj for name, obj in namespace.items() if isinstance(obj, type) and issubclass(obj, BaseModel) and obj is not BaseModel
+ all_user_types: dict[str, type[Any]] = {
+ name: obj
+ for name, obj in namespace.items()
+ if isinstance(obj, type)
+ and not name.startswith("_")
+ and (issubclass(obj, BaseModel) or issubclass(obj, Enum))
+ and obj is not BaseModel
+ and obj is not Enum
}
+ # datamodel-code-generator uses `from __future__ import annotations` which turns
+ # type annotations into strings. Rebuild every BaseModel so forward refs (including
+ # references to generated Enum classes for choices fields) resolve against the
+ # full type namespace.
+ for candidate in all_user_types.values():
+ if issubclass(candidate, BaseModel):
+ candidate.model_rebuild(_types_namespace=all_user_types)
+
+ return all_user_types
+
+ @classmethod
+ def make_types_from_source(cls, source_code: str) -> dict[str, type[Any]]:
+ """Receiver-side entry: exec generated source and return all top-level user-defined
+ types (BaseModel + Enum subclasses) keyed by class name.
+
+ Cached by sha256(source_code) so repeated payloads carrying the same dynamic
+ class don't re-exec. Returns a shallow copy so callers can mutate freely.
+ """
+ cache_key = hashlib.sha256(source_code.encode()).hexdigest()
+ with cls._source_cache_lock:
+ if cache_key in cls._source_cache:
+ cls._source_cache.move_to_end(cache_key)
+ return dict(cls._source_cache[cache_key])
+
+ types_dict = cls._exec_source_to_types(source_code)
+ if len(cls._source_cache) >= cls._SOURCE_CACHE_MAX_SIZE:
+ cls._source_cache.popitem(last=False)
+ cls._source_cache[cache_key] = types_dict
+ return dict(types_dict)
+
+ @classmethod
+ def _exec_and_extract_class(cls, source_code: str, class_name: str) -> type[BaseModel]:
+ """Execute source code and extract the named BaseModel class."""
+ all_user_types = cls.make_types_from_source(source_code)
+ model_classes: dict[str, type[BaseModel]] = {name: obj for name, obj in all_user_types.items() if issubclass(obj, BaseModel)}
+
# datamodel-code-generator normalizes schema titles to PascalCase class names
# (e.g. "dynamic_concept_test__Greeting" → "DynamicConceptTestGreeting"),
# so try both the original name and the normalized version.
@@ -243,13 +295,4 @@ class SchemaToModelFactory:
msg = f"Class '{class_name}' not found in generated source. Available BaseModel classes: {available}"
raise ValueError(msg)
- # datamodel-code-generator uses `from __future__ import annotations` which turns
- # type annotations into strings. For nested models, we need to rebuild so Pydantic
- # resolves the forward references against all classes in the generated namespace.
- # Include ALL user-defined types (models + enums) so forward references to
- # generated Enum classes (e.g. choices fields) can be resolved.
- all_generated_types: dict[str, Any] = {name: obj for name, obj in namespace.items() if isinstance(obj, type) and not name.startswith("_")}
- for model_cls in model_classes.values():
- model_cls.model_rebuild(_types_namespace=all_generated_types)
-
return extracted_class
diff --git a/pipelex/temporal/temporal_data_converter.py b/pipelex/temporal/temporal_data_converter.py
index d6d58dc70..bd840b927 100644
--- a/pipelex/temporal/temporal_data_converter.py
+++ b/pipelex/temporal/temporal_data_converter.py
@@ -28,6 +28,7 @@ For examples, see the tests in `test_top_crafter.py`:
from typing import Any, cast
from kajson import kajson
+from kajson.class_registry import ClassRegistry
from pydantic import BaseModel
from temporalio.api.common.v1 import Payload
from temporalio.converter import (
@@ -41,6 +42,7 @@ from temporalio.converter import (
from typing_extensions import override
from pipelex import log
+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
from pipelex.hub import get_class_registry
from pipelex.temporal.exceptions import TemporalFlowError
@@ -89,11 +91,22 @@ class BaseModelPayloadConverter(JSONPlainPayloadConverter):
log.verbose(f"unijson_deserialize_payload — data: {data}")
source_bytes = payload.metadata.get("kajson_class_source")
class_source_code = source_bytes.decode() if source_bytes else None
- pydantic_gizmo = kajson.loads(
- data,
- class_registry=get_class_registry(),
- class_source_code=class_source_code,
- )
+
+ global_registry = get_class_registry()
+ if class_source_code is not None:
+ # Build a per-call scoped registry overlaying the global one with the
+ # source-derived dynamic classes (BaseModel + Enum). Per-call so dynamic
+ # classes never persist across payloads — different schemas may reuse the
+ # same class name without collision.
+ source_types = SchemaToModelFactory.make_types_from_source(class_source_code)
+ scoped_registry = ClassRegistry()
+ scoped_registry.register_classes_dict(global_registry.get_classes_dict())
+ for type_name, type_obj in source_types.items():
+ scoped_registry.register_class(type_obj, name=type_name, should_warn_if_already_registered=False)
+ pydantic_gizmo = kajson.loads(data, class_registry=scoped_registry)
+ else:
+ pydantic_gizmo = kajson.loads(data, class_registry=global_registry)
+
if class_source_code is not None:
if isinstance(pydantic_gizmo, BaseModel):
self._restore_class_source(pydantic_gizmo, class_source_code)
diff --git a/pipelex/temporal/tprl_pipe/submitter_hydration.py b/pipelex/temporal/tprl_pipe/submitter_hydration.py
index 4c5015e32..7b4e43ba7 100644
--- a/pipelex/temporal/tprl_pipe/submitter_hydration.py
+++ b/pipelex/temporal/tprl_pipe/submitter_hydration.py
@@ -23,7 +23,6 @@ from typing import TYPE_CHECKING
from kajson.class_registry import ClassRegistry
from kajson.kajson_manager import KajsonManager
-from pipelex import log
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.hub import get_current_library, get_library_manager, set_current_library, teardown_current_library
from pipelex.temporal.tprl_pipe.hydration import hydrate_working_memory
@@ -66,10 +65,7 @@ def rehydrate_pipe_output_with_crate(
global_registry = KajsonManager.get_class_registry()
scoped_registry = ClassRegistry()
- if isinstance(global_registry, ClassRegistry):
- scoped_registry.register_classes_dict(dict(global_registry.root))
- else:
- log.warning(f"Global registry is {type(global_registry).__name__}, not ClassRegistry — cannot pre-seed rehydration registry")
+ scoped_registry.register_classes_dict(global_registry.get_classes_dict())
rehydration_library.set_class_registry(scoped_registry)
set_current_library(library_id=rehydration_library_id)
diff --git a/pipelex/temporal/tprl_pipe/wf_pipe_router.py b/pipelex/temporal/tprl_pipe/wf_pipe_router.py
index a728e3435..274d18b2e 100644
--- a/pipelex/temporal/tprl_pipe/wf_pipe_router.py
+++ b/pipelex/temporal/tprl_pipe/wf_pipe_router.py
@@ -52,10 +52,7 @@ class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):
# 1. Create per-workflow ClassRegistry pre-seeded from global
global_registry = KajsonManager.get_class_registry()
workflow_registry = ClassRegistry()
- if isinstance(global_registry, ClassRegistry):
- workflow_registry.register_classes_dict(dict(global_registry.root))
- else:
- workflow_log.warning("Global registry is not a ClassRegistry, cannot pre-seed workflow registry")
+ workflow_registry.register_classes_dict(global_registry.get_classes_dict())
# 2. Open library and attach registry to it
library_manager = get_library_manager()
diff --git a/pyproject.toml b/pyproject.toml
index 1d90e6acd..6fe687d86 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "pipelex"
-version = "0.25.2"
+version = "0.26.0"
description = "Execute composable AI methods declared in the MTHDS open standard"
authors = [{ name = "Evotis S.A.S.", email = "[email protected]" }]
maintainers = [{ name = "Pipelex staff", email = "[email protected]" }]
@@ -24,10 +24,10 @@ dependencies = [
"datamodel-code-generator>=0.55.0",
"filetype>=1.2.0",
"httpx>=0.23.0,<1.0.0",
- "instructor>=1.8.3,!=1.11.*,!=1.12.*", # 1.11.x caused typing errors with mypy
+ "instructor>=1.8.3,!=1.11.*,!=1.12.*", # 1.11.x caused typing errors with mypy
"jinja2>=3.1.4",
"json2html>=1.3.0",
- "kajson @ git+https://github.com/Pipelex/kajson.git@fix/enum-source-registry",
+ "kajson==0.5.0",
"markdown>=3.6",
"mthds>=0.3.0",
"networkx>=3.4.2",
@@ -44,7 +44,7 @@ dependencies = [
"pydantic>=2.10.6,<3.0.0",
"python-dotenv>=1.0.1",
"PyYAML>=6.0.2",
- "reportlab>=4.0,<5", # for generating synthetic PDFs
+ "reportlab>=4.0,<5", # for generating synthetic PDFs
"rich>=13.8.1",
"semantic-version>=2.10.0",
"shortuuid>=1.0.13",
diff --git a/tests/integration/pipelex/fixtures/pipe_job_helpers.py b/tests/integration/pipelex/fixtures/pipe_job_helpers.py
index fb0b10f3b..48498e9d4 100644
--- a/tests/integration/pipelex/fixtures/pipe_job_helpers.py
+++ b/tests/integration/pipelex/fixtures/pipe_job_helpers.py
@@ -67,8 +67,7 @@ def pipe_job_from_library(
# this scoped registry instead of the global, keeping the global clean.
global_registry = KajsonManager.get_class_registry()
scoped_registry = ClassRegistry()
- if isinstance(global_registry, ClassRegistry):
- scoped_registry.register_classes_dict(dict(global_registry.root))
+ scoped_registry.register_classes_dict(global_registry.get_classes_dict())
library.set_class_registry(scoped_registry)
set_current_library(library_id=library_id)
diff --git a/tests/integration/pipelex/temporal/data_converter/test_data_conv_enum_roundtrip.py b/tests/integration/pipelex/temporal/data_converter/test_data_conv_enum_roundtrip.py
new file mode 100644
index 000000000..3ad2c80af
--- /dev/null
+++ b/tests/integration/pipelex/temporal/data_converter/test_data_conv_enum_roundtrip.py
@@ -0,0 +1,57 @@
+from enum import Enum
+from typing import cast
+
+import pytest
+from pydantic import BaseModel
+
+from pipelex import pretty_print
+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
+from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter
+from pipelex.types import StrEnum
+
+
+class PetSpecies(StrEnum):
+ DOG = "dog"
+ CAT = "cat"
+
+
+class Pet(BaseModel):
+ name: str
+ species: PetSpecies
+
+
+class TestEnumRoundTrip:
+ def test_dynamic_class_with_enum_field_round_trips(
+ self,
+ payload_converter: BaseModelPayloadConverter,
+ ):
+ """A class with an Enum field generated via SchemaToModelFactory must survive a
+ full payload round-trip. Exercises the receiver-side exec path that registers
+ Enum subclasses in the per-call scoped ClassRegistry — without it the
+ deserializer cannot resolve the dynamic enum class and the round-trip fails.
+ """
+ schema = Pet.model_json_schema()
+ dynamic_pet_cls = SchemaToModelFactory.make_from_json_schema(schema, "Pet")
+ instance = dynamic_pet_cls(name="Rex", species=PetSpecies.DOG)
+ pretty_print(instance, title="instance")
+
+ payload = payload_converter.to_payload(instance)
+ pretty_print(payload, title="payload")
+ assert payload is not None
+ assert payload.metadata.get("kajson_class_source")
+
+ restored = cast("BaseModel", payload_converter.from_payload(payload, type_hint=BaseModel))
+ pretty_print(restored, title="restored")
+
+ restored_class: type[BaseModel] = type(restored)
+ assert restored_class.__name__ == "Pet"
+ assert restored.name == "Rex" # type: ignore[attr-defined]
+ # datamodel-code-generator emits `class PetSpecies(Enum)` (not StrEnum), so the
+ # dynamic enum is a distinct class from the static PetSpecies above. Assert on
+ # class name + value instead of identity equality.
+ species_value: Enum = restored.species # type: ignore[attr-defined]
+ assert isinstance(species_value, Enum)
+ assert type(species_value).__name__ == "PetSpecies"
+ assert species_value.value == "dog"
+ assert getattr(restored_class, "__kajson_class_source__", None)
diff --git a/uv.lock b/uv.lock
index 4539a3f18..b02cffc67 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1938,11 +1938,15 @@ wheels = [
[[package]]
name = "kajson"
-version = "0.4.2"
-source = { git = "https://github.com/Pipelex/kajson.git?rev=fix%2Fenum-source-registry#4e80ebf729e22f4f3a6a4fac846a024e38fc2fd0" }
+version = "0.5.0"
+source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pydantic" },
]
+sdist = { url = "https://files.pythonhosted.org/packages/91/9c/28a44cc51f328fe1076ea2114b6b7791ef49c54c24a07f11f86f83438da0/kajson-0.5.0.tar.gz", hash = "sha256:029b6a1b60ea28ec95b6f614656d1e48600e27c1f27a39c6e5681cd721854f64", size = 21969, upload-time = "2026-05-04T12:20:12.734Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2e/f4/6460fb07535ef71f87416e1d603f7eaaa9d2970161bec3e37fe08af74b45/kajson-0.5.0-py3-none-any.whl", hash = "sha256:53334fd7122e470ee560255eb5a8390b5ad60ea99bc0d6b75122b323379b553a", size = 28398, upload-time = "2026-05-04T12:20:11.202Z" },
+]
[[package]]
name = "latex2mathml"
@@ -3626,7 +3630,7 @@ wheels = [
[[package]]
name = "pipelex"
-version = "0.25.2"
+version = "0.26.0"
source = { editable = "." }
dependencies = [
{ name = "aiofiles" },
@@ -3774,7 +3778,7 @@ requires-dist = [
{ name = "instructor", extras = ["google-genai"], marker = "extra == 'google-genai'" },
{ name = "jinja2", specifier = ">=3.1.4" },
{ name = "json2html", specifier = ">=1.3.0" },
- { name = "kajson", git = "https://github.com/Pipelex/kajson.git?rev=fix%2Fenum-source-registry" },
+ { name = "kajson", specifier = "==0.5.0" },
{ name = "linkup-sdk", marker = "extra == 'linkup'", specifier = ">=0.12.0" },
{ name = "lxml", marker = "extra == 'docling'", specifier = ">=6.1.0" },
{ name = "markdown", specifier = ">=3.6" },