Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Feature/temporal merge 2 (#855)

Louis Choquelbefb0e7fc703
diff --git a/.claude/plans/spicy-chasing-sutton.md b/.claude/plans/spicy-chasing-sutton.md
new file mode 100644
index 000000000..d44398141
--- /dev/null
+++ b/.claude/plans/spicy-chasing-sutton.md
@@ -0,0 +1,239 @@
+# Plan: Fully align deep_flow content generators with ContentGeneratorProtocol
+
+## Context
+
+The `ContentGeneratorProtocol` (`pipelex/cogt/content_generation/content_generator_protocol.py`) evolved independently from the Temporal deep_flow implementations. The deep_flow files have broken references, wrong method names/signatures, missing methods, and wrong return types. The goal is full alignment — no `pyright: ignore[reportIncompatibleMethodOverride]` comments needed.
+
+## Reference files (read-only)
+
+- `pipelex/cogt/content_generation/content_generator_protocol.py` — the protocol to match
+- `pipelex/cogt/content_generation/content_generator.py` — non-Temporal reference implementation
+- `pipelex/cogt/content_generation/generated_content_factory.py` — `GeneratedContentFactory` for post-processing
+- `pipelex/cogt/content_generation/assignment_models.py` — `TemplatingAssignment` model (fields: `context`, `template`, `templating_style`, `category`)
+- `pipelex/cogt/templating/template_category.py` — `TemplateCategory` enum
+- `pipelex/cogt/templating/templating_style.py` — `TemplatingStyle` model
+- `pipelex/cogt/llm/llm_prompt_template.py` — `make_for_structuring_from_preliminary_text()` class method
+- `pipelex/deep_flow/tprl/workflow_caller.py` — `WorkflowExecutor` base class
+
+## Files to modify
+
+1. `pipelex/cogt/content_generation/content_generator_protocol.py` — add `wfid` param to all applicable methods
+2. `pipelex/cogt/content_generation/content_generator.py` — add `wfid` param (unused, for protocol compat)
+3. `pipelex/deep_flow/tprl_content_generation/content_generator_child.py`
+4. `pipelex/deep_flow/tprl_content_generation/content_generator_top.py`
+5. `pipelex/deep_flow/tprl_content_generation/content_generator_child_factory.py`
+6. `pipelex/deep_flow/tprl_content_generation/content_generator_top_factory.py`
+7. `pipelex/deep_flow/tprl_content_generation/wf_make_extract.py` — rename class `WfMakeOcr` → `WfMakeExtract`
+
+## Changes
+
+### 1. Add `wfid` to `ContentGeneratorProtocol`
+
+Add `wfid: str | None = None` as the last parameter to all methods that the deep_flow implementations use it with:
+- `make_llm_text`
+- `make_object_direct`
+- `make_text_then_object`
+- `make_object_list_direct`
+- `make_text_then_object_list`
+- `make_single_image`
+- `make_image_list`
+- `make_templated_text`
+- `make_extract_pages`
+- `make_render_page_views`
+
+This keeps `wfid` optional (defaulting to `None`) so the non-Temporal `ContentGenerator` doesn't break.
+
+### 2. Add `wfid` to non-Temporal `ContentGenerator`
+
+Add `wfid: str | None = None` parameter to all the same methods in `content_generator.py`. The param is unused in the non-Temporal implementation but satisfies the protocol.
+
+### 3. Rename `WfMakeOcr` → `WfMakeExtract` in `wf_make_extract.py`
+
+The class is still named `WfMakeOcr` despite the file being renamed. Rename to `WfMakeExtract`. The `@workflow.defn(name="wf_make_extract")` string is already correct.
+
+Update imports in `content_generator_child.py` and `content_generator_top.py`.
+
+### 4. Add `generated_content_factory` to `ContentGeneratorChild`
+
+Add `__init__` that accepts `generated_content_factory: GeneratedContentFactory` and stores it as `self._generated_content_factory`, calling `super().__init__(**kwargs)`.
+
+`ContentGeneratorChild` inherits from `WorkflowExecutor` which has `__init__(self, temporal_client, should_auto_connect_temporal, worker_environment, **kwargs)` → `WorkflowCaller.__init__(self, task_queue, workflow_execution_timeout, ...)`. So the child class init should be:
+
+```python
+def __init__(self, generated_content_factory: GeneratedContentFactory, **kwargs: Any) -> None:
+ super().__init__(**kwargs)
+ self._generated_content_factory = generated_content_factory
+```
+
+### 5. Add `generated_content_factory` to `ContentGeneratorTop`
+
+Same pattern as child.
+
+### 6. Update factories to pass `generated_content_factory`
+
+- `ContentGeneratorChildFactory.make_content_generator_child()` — add `generated_content_factory: GeneratedContentFactory` parameter and pass through
+- `ContentGeneratorTopFactory.make_content_generator_top()` — add `generated_content_factory: GeneratedContentFactory` parameter and pass through
+
+### 7. Rename `make_jinja2_text` → `make_templated_text` and align signature
+
+In both child and top, rename and update signature to match protocol:
+
+```python
+@override
+async def make_templated_text(
+ self,
+ context: dict[str, Any],
+ template: str,
+ templating_style: TemplatingStyle | None = None,
+ template_category: TemplateCategory | None = None,
+ wfid: str | None = None,
+) -> str:
+```
+
+Update `TemplatingAssignment` construction:
+```python
+templating_assignment = TemplatingAssignment(
+ context=context,
+ template=template,
+ templating_style=templating_style,
+ category=template_category or TemplateCategory.BASIC,
+)
+```
+
+Remove `PromptingTarget` import from both files. Add `TemplatingStyle` import. Remove `Jinja2TemplateCategory` references.
+
+### 8. Rename `make_extract_extract_pages` → `make_extract_pages` and change return type
+
+Change return type to `list[PageContent]`. After getting `ExtractOutput` from the workflow, call:
+```python
+return await self.make_page_contents(job_metadata=job_metadata, extract_output=extract_output)
+```
+
+Keep `extract_job_params` and `extract_job_config` as required params (matching protocol).
+
+### 9. Change `make_single_image` return type to `ImageContent`
+
+After getting `GeneratedImageRawDetails` from the workflow, convert:
+```python
+return await self.make_image_content(
+ job_metadata=job_metadata,
+ generated_image_raw_details=generated_image,
+ img_gen_prompt=img_gen_prompt,
+)
+```
+
+### 10. Change `make_image_list` return type to `list[ImageContent]`
+
+After getting `list[GeneratedImageRawDetails]`, convert each:
+```python
+return [
+ await self.make_image_content(
+ job_metadata=job_metadata,
+ generated_image_raw_details=raw_details,
+ img_gen_prompt=img_gen_prompt,
+ )
+ for raw_details in generated_image_list
+]
+```
+
+### 11. Add `make_image_content` method to both
+
+Same implementation as non-Temporal `ContentGenerator`:
+```python
+@override
+async def make_image_content(
+ self,
+ job_metadata: JobMetadata,
+ generated_image_raw_details: GeneratedImageRawDetails,
+ img_gen_prompt: ImgGenPrompt | None,
+) -> ImageContent:
+ image_content = await self._generated_content_factory.make_image_content(
+ primary_id=job_metadata.user_id,
+ secondary_id=job_metadata.pipeline_run_id,
+ raw_details=generated_image_raw_details,
+ )
+ if img_gen_prompt:
+ image_content.source_prompt = img_gen_prompt.positive_text
+ image_content.source_negative_prompt = img_gen_prompt.negative_text
+ return image_content
+```
+
+### 12. Add `make_page_contents` method to both
+
+```python
+@override
+async def make_page_contents(
+ self,
+ job_metadata: JobMetadata,
+ extract_output: ExtractOutput,
+) -> list[PageContent]:
+ return await self._generated_content_factory.make_page_contents(
+ primary_id=job_metadata.user_id,
+ secondary_id=job_metadata.pipeline_run_id,
+ extract_output=extract_output,
+ )
+```
+
+### 13. Add `make_render_page_views` method to both
+
+Same as non-Temporal version, using `pypdfium2_renderer`:
+```python
+@override
+async def make_render_page_views(
+ self,
+ job_metadata: JobMetadata,
+ extract_input: ExtractInput,
+ extract_handle: str,
+ extract_job_params: ExtractJobParams | None = None,
+ extract_job_config: ExtractJobConfig | None = None,
+ wfid: str | None = None,
+) -> list[ImageContent]:
+ if not extract_input.document_uri:
+ msg = "PDF URI is required to render page views"
+ raise ValueError(msg)
+ job_params = extract_job_params or ExtractJobParams.make_default_extract_job_params()
+ page_views_dpi = job_params.page_views_dpi or get_config().cogt.extract_config.default_page_views_dpi
+ page_view_images = await pypdfium2_renderer.render_pdf_pages_from_uri(
+ pdf_uri=extract_input.document_uri, dpi=page_views_dpi
+ )
+ return [
+ await self.make_image_content(
+ job_metadata=job_metadata,
+ generated_image_raw_details=GeneratedImageRawDetails.make_from_pil_image(
+ pil_image=page_view_image,
+ image_format=ImageFormat.PNG,
+ ),
+ img_gen_prompt=None,
+ )
+ for page_view_image in page_view_images
+ ]
+```
+
+### 14. Fix `LLMPromptTemplate` method name in `content_generator_top.py`
+
+Lines 138 and 217: `for_structure_from_preliminary_text()` → `make_for_structuring_from_preliminary_text()`
+
+### 15. Remove all `pyright: ignore[reportIncompatibleMethodOverride]` comments
+
+With full alignment, these are no longer needed.
+
+### 16. Fix imports in both deep_flow files
+
+**Add:**
+- `from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory`
+- `from pipelex.cogt.templating.templating_style import TemplatingStyle`
+- `from pipelex.core.stuffs.image_content import ImageContent`
+- `from pipelex.core.stuffs.page_content import PageContent`
+- `from pipelex.tools.misc.image_utils import ImageFormat`
+- `from pipelex.tools.pdf.pypdfium2_renderer import pypdfium2_renderer`
+
+**Remove:**
+- `from pipelex.cogt.model_backends.prompting_target import PromptingTarget` (child)
+- `from pipelex.tools.templating.templating_models import PromptingTarget` (top — wrong path anyway)
+
+**Fix:**
+- `WfMakeOcr` → `WfMakeExtract` in import from `wf_make_extract`
+
+## Verification
+
+Run `make agent-check` to verify no new lint/type errors are introduced.
diff --git a/.claude/skills/temporal-e2e-validate/SKILL.md b/.claude/skills/temporal-e2e-validate/SKILL.md
new file mode 100644
index 000000000..ac5bdb835
--- /dev/null
+++ b/.claude/skills/temporal-e2e-validate/SKILL.md
@@ -0,0 +1,742 @@
+---
+name: temporal-e2e-validate
+description: >
+ Full end-to-end validation of Temporal distributed execution (Phases 2-5).
+ Two modes: (1) pytest against a real Temporal server for detailed assertions on
+ all test cases, and (2) true 3-process setup (server + separate worker process
+ + submitter) that validates the actual deployment topology including cross-process
+ serialization, LibraryCrate propagation, deferred hydration, concurrent
+ isolation, image payload storage, and cross-worker graph tracing with GraphSpec
+ assembly. Includes image generation and image flow tests that verify large binary
+ payloads are stored at the activity level (not passed inline through Temporal).
+ Use when the user says "validate temporal", "e2e temporal",
+ "temporal regression", "temporal validation", "3-process test", "full temporal
+ test", "validate phases", "image temporal", or wants comprehensive verification
+ that distributed execution works end-to-end. Also use proactively after changes
+ to LibraryCrate, deferred hydration, ClassRegistry scoping, graph tracing,
+ image generation activities, content storage, or Temporal workflow code.
+allowed-tools:
+ - Bash(tmux *)
+ - Bash(curl *)
+ - Bash(.venv/bin/pytest *)
+ - Bash(.venv/bin/pipelex *)
+ - Bash(ls *)
+ - Bash(which *)
+ - Bash(open *)
+ - Bash(cat *)
+ - Bash(.venv/bin/python *)
+---
+
+# Temporal E2E Validation Suite
+
+This skill validates that Pipelex pipelines execute correctly when distributed across
+Temporal workers — separate processes that receive serialized work, run pipes, and return
+results. It covers the full chain: shipping pipe definitions to the worker (Phase 2),
+deserializing dynamic concepts the worker has never seen (Phase 3), isolating concurrent
+workflows so they don't corrupt each other (Phase 4), assembling an execution graph
+from trace events emitted across workers (Phase 4.5), and verifying that image-heavy
+pipelines (image generation, image-to-LLM flow) don't blow up Temporal's payload
+limits by ensuring images are stored at the activity level.
+
+## Important: surface results immediately
+
+After each command completes, **immediately tell the user** the outcome in plain text —
+PASS/FAIL, what it means, output paths. Do NOT silently move on to the next command.
+The user sees collapsed tool outputs by default and relies on your text messages.
+
+Use multi-line formatting with clean indentation — never cram everything onto one line:
+
+```
+Tier 1 PASS
+ The worker received the pipe definitions and executed a 2-step sequence correctly.
+ Output: results/native_text_sequence_output_01/
+ Graph: results/native_text_sequence_output_01/reactflow.html
+```
+
+For failures, include the error message and what it means:
+
+```
+Tier 2 FAIL
+ KajsonDecoderError: Class 'Greeting' not found
+ The worker tried to deserialize WorkingMemory before registering dynamic concepts.
+ Check worker output: tmux capture-pane -t temporal-worker -p -S -200
+```
+
+---
+
+## Prerequisites
+
+```bash
+which tmux && echo "ok" || echo "MISSING: brew install tmux"
+which temporal && echo "ok" || echo "MISSING: brew install temporal"
+.venv/bin/python -c "import temporalio; print(f'temporalio {temporalio.__version__}')"
+```
+
+---
+
+## Mode 1: Automated Test Suite (pytest)
+
+Runs integration tests against a real Temporal server (localhost:7233), but the worker
+runs in-process — no separate worker needed. This is the fast path for catching
+regressions.
+
+### Step 1: Ensure the Temporal dev server is running
+
+```bash
+curl -s http://localhost:8233 > /dev/null 2>&1 && echo "running" || echo "not running"
+```
+
+If not running:
+
+```bash
+tmux new-session -d -s temporal-server 'temporal server start-dev'
+sleep 3
+curl -s http://localhost:8233 > /dev/null 2>&1 && echo "running" || echo "FAILED"
+```
+
+Do NOT start if already running — it will fail with a bind error.
+
+### Step 2: Run the tests
+
+**Dry mode (fast, no LLM costs):**
+
+```bash
+.venv/bin/pytest -x -v tests/integration/pipelex/temporal/library_crate/ \
+ -m temporal --temporal-server local 2>&1
+```
+
+**Live mode (real LLM calls):**
+
+```bash
+.venv/bin/pytest -x -v tests/integration/pipelex/temporal/library_crate/ \
+ -m temporal --temporal-server local --pipe-run-mode live 2>&1
+```
+
+### Step 3: Report results
+
+Tell the user what each suite validated and whether it passed. Here is what the suites test:
+
+- **TestWfLibraryCrate** — Can a worker receive a "crate" (a portable bundle of pipe
+ definitions) and execute a PipeSequence from it? Also tests that submitting *without* a
+ crate correctly fails with `PipeNotFoundError`.
+- **TestWfDeferredHydration** — When a pipe creates a brand-new concept type at runtime
+ (e.g. `Greeting` with `message` and `language` fields), can the worker deserialize
+ WorkingMemory that contains instances of that concept, even though the worker has never
+ seen the class before?
+- **TestWfConcurrentConceptIsolation** — Two workflows run simultaneously on the same
+ worker, both defining a concept called `Result` — but with different fields (`score, label`
+ vs `value, confidence, is_valid`). Does each workflow get the right version, or do they
+ clobber each other?
+- **TestWfConcurrentPipeIsolation** — Same idea, but for pipes: two workflows both define
+ a pipe called `shared_step` with different prompts. Does each workflow execute its own
+ version?
+- **TestWfMultiConceptIsolation** — Worst case: two workflows define *two* overlapping
+ concepts each (`Profile` + `Summary`) with incompatible structures, running across 6
+ concurrent workflows.
+- **TestWfPipeParallel** — A PipeParallel controller dispatches branches as concurrent
+ child workflows. Do the branches execute independently and merge correctly?
+- **TestWfPipeCondition / PipeBatch / PipeCompose / CombinedPipeline** — Controller
+ coverage for condition routing, fan-out/fan-in, composition, and nested controller
+ combinations.
+
+**Known xfails:** Some controller execution tests fail in dry-run mode because
+`StuffArtefact` objects can't serialize through Temporal's data converter. The crate
+structure tests pass. These xfails will resolve once StuffArtefact serialization is fixed.
+
+The test `test_missing_crate_fails_pipe_resolution` is a **negative test** — it intentionally
+submits without a crate. The Temporal warning `Failed activation on workflow` with
+`RuntimeError: No current library set` is expected.
+
+---
+
+## Mode 2: True 3-Process Validation
+
+This is the real deployment test. Three separate OS processes — Temporal server, worker,
+and submitter — with no shared memory. The worker has its own Python runtime, its own
+`sys.modules`, its own ClassRegistry. This is the only way to catch bugs like:
+
+- The worker can't deserialize the PipeJob because concept classes aren't registered yet
+- The Kajson decoder bypasses class lookup when `__module__="builtins"`
+- The Temporal data converter silently drops fields during encoding/decoding
+
+Each command runs a pipeline through Temporal with `--graph`, which also validates that
+the worker emits NDJSON trace events and the submitter assembles them into a GraphSpec
+with an interactive ReactFlow HTML visualization.
+
+### Run mode: dry-run vs live
+
+By default, Mode 2 runs in **dry-run** mode (`--dry-run --mock-inputs`) — fast, no LLM
+costs, validates serialization and crate propagation. But dry-run produces tiny mock
+data, so it **cannot catch payload size issues** (e.g. large image payloads blowing up
+Temporal's data converter).
+
+Ask the user which mode they want. If they say "live", simply omit `--dry-run --mock-inputs`
+from all commands below. The `pipelex run bundle` CLI has no `--pipe-run-mode` flag — live
+is the default when neither `--dry-run` nor `--mock-inputs` is specified (note: pytest in
+Mode 1 does accept `--pipe-run-mode live`, but the CLI does not). Live mode makes real LLM and image
+generation calls — it costs money and is slower, but it's the only way to validate that
+real-sized payloads (especially images) flow correctly through Temporal.
+
+**This matters most for Tiers 4 and 5** (image generation and image flow). In dry-run,
+mock images are trivially small, so payload size bugs don't surface. In live mode,
+generated images are hundreds of KB to several MB — exactly the size that breaks Temporal
+if activity-level storage isn't implemented.
+
+### Step 1: Ensure Temporal server is running
+
+Same as Mode 1 Step 1.
+
+### Step 2: Start the worker processes
+
+The workers should NOT have test bundles in their PIPELEXPATH — the whole point is
+that bundles arrive via the LibraryCrate in the PipeJob.
+
+**Two scoped workers (cross-process regression setup — recommended).**
+Splits responsibilities so workflows and activities run in different processes:
+- `router` worker registers all workflows (`WfPipeRouter`, `WfPipeRun`, …),
+ `disable_all_activities = true`.
+- `runner` worker registers all activities (`act_deliver`, `act_llm_*`,
+ `act_assemble_graph`, `act_flush_trace_events`, …), `disable_all_workflows = true`.
+
+This forces every activity to be picked up by a *different* Python process than the
+workflow that scheduled it. The runner process never executes `WfPipeRouter`, so it
+never loads the LibraryCrate and its global `ClassRegistry` stays cold for any
+dynamic concept defined in the bundle.
+
+> ⚠️ **Important — what this setup does NOT reproduce on its own.**
+> Plain `pipelex run bundle` (Tiers 1–3 below) does NOT trigger the runner-side
+> registry decode bug, because:
+> - `WfPipeRouter` dehydrates `pipe_output` via `prepare_for_temporal()` before
+> returning, so workflow-level transit carries raw dicts (no class lookup).
+> - `WfPipeRun` rehydrates back on the *router* (same process that loaded the crate).
+> - Activities the runner actually executes in dry-run (`act_assemble_graph`,
+> `act_flush_trace_events`) operate on raw event records — no dynamic class needed.
+> - `act_deliver` is **only scheduled when `delivery_assignment is not None`**
+> (`wf_pipe_run.py:79`), and `pipelex run bundle` does not pass one.
+>
+> To deterministically force a hydrated `pipe_output` across the process boundary,
+> run **Tier 2b** below (mirrors the cloud / `start_pipeline` + webhook path).
+
+```bash
+tmux has-session -t temporal-worker-router 2>/dev/null && tmux kill-session -t temporal-worker-router
+tmux has-session -t temporal-worker-runner 2>/dev/null && tmux kill-session -t temporal-worker-runner
+tmux new-session -d -c "$PWD" -s temporal-worker-router \
+ '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope router'
+tmux new-session -d -c "$PWD" -s temporal-worker-runner \
+ '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner'
+sleep 4
+tmux capture-pane -t temporal-worker-router -p -S -30
+tmux capture-pane -t temporal-worker-runner -p -S -30
+```
+
+Look for `Temporal Worker started for 'temporal_task_queue'` in each session, plus
+`Temporal Worker scope: 'router'` and `'runner'` respectively.
+
+**Alternative — single full worker** (simpler, but masks distributed-execution bugs;
+use only when you don't need the regression coverage):
+
+```bash
+tmux has-session -t temporal-worker 2>/dev/null && tmux kill-session -t temporal-worker
+tmux new-session -d -c "$PWD" -s temporal-worker \
+ '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed'
+sleep 4
+tmux capture-pane -t temporal-worker -p -S -30
+```
+
+When using two scoped workers, replace any later capture commands like
+`tmux capture-pane -t temporal-worker ...` with the appropriate session name
+(`temporal-worker-router` or `temporal-worker-runner`).
+
+### Step 3: Sequential tests — run one at a time, report after each
+
+Do NOT clean previous results automatically — the user may want to compare runs.
+
+**Tier 1 — Can the worker execute a simple pipe sequence?**
+
+This sends a 2-step PipeSequence (step_one → step_two) to a worker that has never seen
+these pipes. The worker must unpack the LibraryCrate, register the pipes, and execute
+them in order. This is the most basic "does Temporal work at all" test.
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \
+ --pipe native_text_sequence \
+ --temporal --dry-run --mock-inputs --no-logo --graph
+```
+
+After this completes, tell the user: PASS/FAIL, output dir, graph file path.
+
+**Tier 2 — Can the worker handle concepts it has never seen? (in-process hydration)**
+
+This sends a pipeline that defines a custom concept (`Greeting` with `message` and
+`language` fields) at runtime. `WfPipeRouter` must register this dynamic class
+into the per-workflow registry before hydrating the WorkingMemory. If the hydration
+order is wrong **on the router**, this fails with
+`KajsonDecoderError: Class 'Greeting' not found`.
+
+> Note: in dry-run with `pipelex run bundle` (no `delivery_assignment`), this tier
+> only exercises hydration **on the router process**, not cross-process. It will
+> pass even if the runner's registry is cold — see Tier 2b for the deterministic
+> cross-process repro.
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \
+ --pipe dynamic_greeting_sequence \
+ --temporal --dry-run --mock-inputs --no-logo --graph
+```
+
+After this completes, tell the user: PASS/FAIL, output dir, graph file path.
+
+**Tier 2b — Cross-process registry regression (deterministic, dry-run).**
+
+Same bundle as Tier 2, but submitted with a `DeliveryAssignment` attached. This
+mirrors the cloud / `start_pipeline` + webhook path: `wf_pipe_run.py:79` then
+schedules `act_deliver` on the runner with `pipe_output` carrying a hydrated
+`WorkingMemory`. Temporal's data converter on the **runner process** must decode
+the dynamic concept class — which is the exact path the `wf_pipe_router.py:71-80`
+in-process propagation hack does NOT cover.
+
+If the registry propagation across processes is broken, this fails every time with:
+
+```
+KajsonDecoderError: Class '<bundle>__<DynamicConcept>' not found
+ in module 'builtins' or global registry
+ApplicationError: Failed decoding arguments
+ → temporalio/worker/_activity.py:566 (data_converter.decode_wrapper)
+```
+
+The error surfaces on the **runner** tmux session, raised by the activity worker
+*before* the activity body runs.
+
+```bash
+.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py
+```
+
+To reproduce against a different bundle:
+
+```bash
+.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py \
+ --bundle <path/to/bundle.mthds> --pipe <pipe_code>
+```
+
+After this completes, tell the user:
+- PASS/FAIL.
+- If FAIL: capture the runner log with
+ `tmux capture-pane -t temporal-worker-runner -p -S -300 | tail -120`
+ and quote the `KajsonDecoderError` and `ApplicationError: Failed decoding arguments`
+ lines verbatim, plus the activity name (`act_deliver`).
+
+**Tier 3 — Do parallel branches execute as separate child workflows?**
+
+This sends a PipeParallel controller that fans out into two branches (tone analysis and
+length analysis), each running as its own child workflow on the worker. The branches
+execute concurrently and their results merge back. The `--graph` flag here is especially
+interesting: it shows cross-worker execution with child workflow branches in the
+ReactFlow visualization.
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds \
+ --pipe temporal_parallel_sequence \
+ --temporal --dry-run --mock-inputs --no-logo --graph
+```
+
+After this completes, tell the user: PASS/FAIL, output dir, graph file path.
+
+**Tier 4 — Can the worker handle image generation pipelines?**
+
+**Important:** This tier only catches payload size bugs in **live mode**. In dry-run,
+mock images are tiny and will pass even without activity-level storage. If the user
+wants to validate image payload handling, they must run live.
+
+This is a critical payload-size test. Image generation produces large binary data
+(base64-encoded images) that must be stored at the activity level and passed as
+storage URIs through Temporal — not inline in the workflow payload. If storage is
+missing, this will either fail with a Temporal payload size error or trigger
+`PayloadSizeWarning` in the worker logs.
+
+The `generate_crazy_image` bundle runs a 2-step PipeSequence: an LLM imagines a
+scene description, then PipeImgGen renders it as an image. This exercises the full
+image generation path through Temporal, including the custom `ImagePrompt` concept
+(which refines `Text`).
+
+Dry-run:
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/pipes/pipelines/crazy_image_generation.mthds \
+ --pipe generate_crazy_image \
+ --temporal --dry-run --mock-inputs --no-logo --graph
+```
+
+Live (real image generation — required to catch payload size bugs):
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/pipes/pipelines/crazy_image_generation.mthds \
+ --pipe generate_crazy_image \
+ --temporal --no-logo --graph
+```
+
+After this completes, tell the user: PASS/FAIL, output dir, graph file path.
+Also check worker logs for `PayloadSizeWarning`:
+
+```bash
+tmux capture-pane -t temporal-worker -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings"
+```
+
+**Tier 5 — Can generated images flow between pipes in a sequence?**
+
+**Important:** Like Tier 4, this only catches real payload bugs in **live mode**.
+
+This is the "image out → image in" test. A PipeSequence first generates an image
+via PipeImgGen, then passes that image as input to a PipeLLM (vision model) that
+describes it. This exercises:
+- Image generation storage at the activity level
+- Image content serialization through Temporal's data converter
+- Image content deserialization on the worker for the next pipe step
+- Vision model receiving an image reference (URI) rather than inline base64
+
+If activity-level storage is not implemented, this will fail because the raw image
+bytes exceed Temporal's payload limit, or because the worker can't deserialize the
+`ImageContent` object from the previous step.
+
+Dry-run:
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/pipes/pipelines/test_image_out_in.mthds \
+ --pipe image_out_in \
+ --temporal --dry-run --mock-inputs --no-logo --graph
+```
+
+Live (real image generation + vision — required to catch payload size bugs):
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/pipes/pipelines/test_image_out_in.mthds \
+ --pipe image_out_in \
+ --temporal --no-logo --graph
+```
+
+After this completes, tell the user: PASS/FAIL, output dir, graph file path.
+Also check for payload warnings:
+
+```bash
+tmux capture-pane -t temporal-worker -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings"
+```
+
+If any tier hangs for more than 30 seconds, check worker output:
+
+```bash
+tmux capture-pane -t temporal-worker -p -S -100
+```
+
+### Step 4: Verify graph output
+
+Check that the `--graph` flag produced ReactFlow HTML files:
+
+```bash
+ls results/*/reactflow.html 2>/dev/null
+```
+
+Tell the user where the files are and propose opening the most interesting one (the
+PipeParallel graph shows cross-worker execution):
+
+```bash
+open results/temporal_parallel_sequence_output_01/reactflow.html
+```
+
+If no `reactflow.html` exists, GraphSpec assembly failed — check that tracing is enabled
+in `pipelex.toml` (`[pipelex.tracing_config]` with `is_enabled = true`).
+
+### Step 5: Concurrent isolation tests — can conflicting workflows coexist?
+
+These are the most important safety tests. They submit two workflows simultaneously to
+the same worker, where both workflows define identically-named concepts or pipes but
+with different structures. If per-workflow isolation is broken, one workflow will use
+the other's class definitions and either crash or silently produce wrong data.
+
+Each pair runs with `--graph`. The graph output from backgrounded jobs goes to separate
+`results/<pipe_code>_output_NN/` dirs.
+
+**Test A — Conflicting concept names:**
+Two workflows both define a concept called `Result`, but alpha's has `score` + `label`
+while beta's has `value` + `confidence` + `is_valid`. If isolation fails, one workflow
+will try to populate the wrong fields.
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/conflict_concept_alpha.mthds \
+ --pipe alpha_pipeline \
+ --temporal --dry-run --mock-inputs --no-logo --graph &
+PID_ALPHA=$!
+
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/conflict_concept_beta.mthds \
+ --pipe beta_pipeline \
+ --temporal --dry-run --mock-inputs --no-logo --graph &
+PID_BETA=$!
+
+wait $PID_ALPHA && echo "Alpha: PASS" || echo "Alpha: FAIL"
+wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"
+```
+
+After this, tell the user both results.
+
+**Test B — Conflicting pipe names:**
+Two workflows both define a pipe called `shared_step` but with different prompts.
+Does each workflow execute its own version?
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/conflict_pipe_alpha.mthds \
+ --pipe pipe_alpha_pipeline \
+ --temporal --dry-run --mock-inputs --no-logo --graph &
+PID_ALPHA=$!
+
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/conflict_pipe_beta.mthds \
+ --pipe pipe_beta_pipeline \
+ --temporal --dry-run --mock-inputs --no-logo --graph &
+PID_BETA=$!
+
+wait $PID_ALPHA && echo "Alpha: PASS" || echo "Alpha: FAIL"
+wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"
+```
+
+After this, tell the user both results.
+
+**Test C — Worst case: multiple conflicting concepts at high concurrency:**
+Two workflows define both `Profile` AND `Summary` with incompatible field structures.
+This is the hardest isolation test — if ContextVar scoping leaks between workflows,
+this is where it shows up.
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/multi_concept_alpha.mthds \
+ --pipe multi_alpha_pipeline \
+ --temporal --dry-run --mock-inputs --no-logo --graph &
+PID_ALPHA=$!
+
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/multi_concept_beta.mthds \
+ --pipe multi_beta_pipeline \
+ --temporal --dry-run --mock-inputs --no-logo --graph &
+PID_BETA=$!
+
+wait $PID_ALPHA && echo "Alpha: PASS" || echo "Alpha: FAIL"
+wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"
+```
+
+After this, tell the user both results.
+
+### Step 6: StoragePayloadCodec tests — does the codec work end-to-end?
+
+These tests validate Phase 5: large payloads are transparently offloaded to external
+storage by the `StoragePayloadCodec`, keeping Temporal's event history small. The worker
+and submitter both use the codec — it's wired at the data converter level.
+
+**Important:** The codec config is managed via `pipelex_temporary_override.toml` — a
+config layer that loads after `pipelex_override.toml` and takes highest priority. This
+file is ephemeral and gitignored. **NEVER modify `pipelex_override.toml`** — that is
+the user's personal config.
+
+**Step 6a: Create the temporary override to enable the codec**
+
+```bash
+cat > .pipelex/pipelex_temporary_override.toml << 'EOF'
+# Temporary override for E2E codec testing — delete when done
+[temporal.payload_codec_config]
+is_enabled = true
+EOF
+echo "Temporary override created"
+```
+
+This takes precedence over whatever codec setting exists in `pipelex_override.toml`.
+
+**Step 6b: Restart the worker** (so it picks up the new config)
+
+```bash
+tmux has-session -t temporal-worker 2>/dev/null && tmux kill-session -t temporal-worker
+tmux new-session -d -c "$PWD" -s temporal-worker \
+ '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed'
+sleep 4
+tmux capture-pane -t temporal-worker -p -S -30 | grep -i "payload codec\|Temporal Worker started"
+```
+
+Look for `Payload codec enabled` and `Temporal Worker started`.
+
+**Tier 6 — Codec transparency: existing pipelines work unchanged**
+
+Re-runs Tier 1 and Tier 2 with the codec on. If the codec is truly transparent, the
+same pipelines produce the same results. The only difference: payloads above 1MB are
+stored in `.pipelex/temporal-payload-store/` instead of inline in Temporal's event history.
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \
+ --pipe native_text_sequence \
+ --temporal --dry-run --mock-inputs --no-logo --graph
+```
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \
+ --pipe dynamic_greeting_sequence \
+ --temporal --dry-run --mock-inputs --no-logo --graph
+```
+
+After both complete, check that storage was used:
+
+```bash
+ls -la .pipelex/temporal-payload-store/ 2>/dev/null && echo "Storage files found" || echo "No storage files (payloads may be below threshold in dry-run)"
+```
+
+Tell the user: PASS/FAIL for each, and whether storage files were created. In dry-run
+mode, payloads may be below the 1MB threshold, so no storage files is expected. In live
+mode, larger payloads should trigger storage. Either way, the key assertion is that the
+pipelines complete successfully.
+
+**Tier 7 — Large payload stress test**
+
+A 3-step pipeline that accumulates verbose text output across steps. Each step's
+WorkingMemory carries the results of all previous steps, so the payload grows. In live
+mode this produces realistic multi-KB to multi-MB payloads. In dry-run mode, mock
+outputs are small but the codec path is still exercised.
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds \
+ --pipe large_payload_sequence \
+ --temporal --dry-run --mock-inputs --no-logo --graph
+```
+
+Live (real LLM calls — produces larger payloads, better stress test):
+
+```bash
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds \
+ --pipe large_payload_sequence \
+ --temporal --no-logo --graph
+```
+
+After this completes, check storage and worker logs:
+
+```bash
+ls -la .pipelex/temporal-payload-store/ 2>/dev/null | head -20
+tmux capture-pane -t temporal-worker -p -S -50 | grep -i "payload\|warning\|size\|codec" || echo "No payload warnings"
+```
+
+Tell the user: PASS/FAIL, output dir, graph file path, storage file count.
+
+**Step 6c: Remove the temporary override**
+
+```bash
+.venv/bin/python -c "from pathlib import Path; Path('.pipelex/pipelex_temporary_override.toml').unlink(missing_ok=True)"
+echo "Temporary override removed"
+```
+
+Optionally restart the worker to restore the base codec config, or leave it if done.
+
+### Step 7: Final report
+
+List all graph files and present a summary table with results:
+
+```bash
+ls results/*/reactflow.html
+```
+
+| Test | What it proved | Status | Graph | Payload warnings |
+|------|---------------|--------|-------|-----------------|
+| Tier 1: Sequence | Worker can unpack crate and run a pipe sequence | PASS/FAIL | path | — |
+| Tier 2: Hydration | Worker handles dynamic concepts it has never seen | PASS/FAIL | path | — |
+| Tier 2b: Cross-process registry | `act_deliver` decodes hydrated `pipe_output` on runner (forced via `delivery_assignment`) | PASS/FAIL | — | — |
+| Tier 3: Parallel | Branches execute as concurrent child workflows | PASS/FAIL | path | — |
+| Tier 4: ImgGen | Image generation pipeline works through Temporal | PASS/FAIL | path | yes/no |
+| Tier 5: Image flow | Generated image flows as input to next pipe step | PASS/FAIL | path | yes/no |
+| Tier 6: Codec transparency | Existing pipelines work unchanged with codec enabled | PASS/FAIL | path | — |
+| Tier 7: Large payload | Multi-step pipeline with codec stress test | PASS/FAIL | path | — |
+| Concept isolation (alpha) | Conflicting `Result` concepts don't cross-contaminate | PASS/FAIL | path | — |
+| Concept isolation (beta) | (same test, other side) | PASS/FAIL | path | — |
+| Pipe isolation (alpha) | Conflicting `shared_step` pipes use correct version | PASS/FAIL | path | — |
+| Pipe isolation (beta) | (same test, other side) | PASS/FAIL | path | — |
+| Multi-concept (alpha) | Two overlapping concepts with incompatible structures | PASS/FAIL | path | — |
+| Multi-concept (beta) | (same test, other side) | PASS/FAIL | path | — |
+
+After reporting, propose opening the PipeParallel graph (most interesting cross-worker view):
+
+```bash
+open results/temporal_parallel_sequence_output_01/reactflow.html
+```
+
+If any concurrent tests fail, capture worker output for diagnosis:
+
+```bash
+tmux capture-pane -t temporal-worker -p -S -200
+```
+
+---
+
+## Cleanup
+
+Propose these to the user — do NOT run them automatically:
+
+- Kill tmux sessions: `tmux kill-session -t temporal-worker` / `tmux kill-session -t temporal-server`
+- Clean results directory: `rm -rf results/`
+- Clean trace files: `rm -rf .pipelex/traces/`
+- Remove temporary override if still present: `.venv/bin/python -c "from pathlib import Path; Path('.pipelex/pipelex_temporary_override.toml').unlink(missing_ok=True)"`
+
+Leave the server running if the user plans to iterate.
+
+---
+
+## Interpreting failures
+
+| Error | What it means |
+|-------|--------------|
+| `PipeNotFoundError` on worker | The LibraryCrate didn't arrive or wasn't loaded — the worker can't find the pipes it's supposed to run |
+| `KajsonDecoderError: Class 'X' not found` | The worker tried to deserialize data containing a dynamic concept before registering that concept's class — hydration order bug |
+| `RuntimeError: Failed decoding arguments` | Temporal's data converter can't deserialize the PipeJob on the worker — usually a serialization format issue |
+| `WorkflowFailureError` wrapping `TemporalError` | The pipe itself failed during execution — read the inner error for the real cause |
+| `AssertionError: StructuredContent missing field` | Per-workflow ClassRegistry isolation failed — the worker used the wrong concept class (from another workflow's definitions) |
+| Submitter hangs indefinitely | The worker crashed during deserialization — check `tmux capture-pane -t temporal-worker -p -S -200` |
+| Both concurrent jobs succeed but wrong data | ContextVar leak between workflows — per-workflow scoping is broken, one workflow's class definitions bled into the other |
+| No `reactflow.html` generated | GraphSpec assembly failed — either tracing is disabled in `pipelex.toml` or NDJSON events weren't emitted by the worker |
+| `PayloadSizeWarning` in worker logs | Image data (base64) is being passed inline through Temporal payloads instead of being stored at the activity level — the fix is to call storage in the image generation activity before returning results |
+| `NotImplementedError` during image generation | The image generation or content storage path isn't wired up for the Temporal execution path — activity-level storage is missing |
+| Payload too large / `DataConverterError` on image pipes | Raw image bytes exceed Temporal's payload size limit (~2MB default) — images must be stored and referenced by URI, not passed inline |
+| `ImageContent` missing or has no `uri` field | The image generation activity returned raw image data instead of a stored `ImageContent` with a storage URI |
+| Codec enabled but no storage files | Payloads are below the threshold (1MB default) — expected in dry-run mode where mock data is small. Run live mode for realistic payload sizes |
+| `FileNotFoundError` in codec decode | The codec is trying to load a payload from storage that doesn't exist — storage root path may be misconfigured or files were cleaned up mid-run |
+
+---
+
+## Bundle reference
+
+**Crate/isolation bundles** — in `tests/integration/pipelex/temporal/library_crate/`:
+
+| Bundle | What it tests | Main pipe |
+|--------|--------------|-----------|
+| `native_text_sequence.mthds` | Basic crate propagation (native Text, 2 steps) | `native_text_sequence` |
+| `dynamic_concept_sequence.mthds` | Deferred hydration (Greeting concept created at runtime) | `dynamic_greeting_sequence` |
+| `conflict_concept_alpha.mthds` | Concept isolation — Result(score, label) | `alpha_pipeline` |
+| `conflict_concept_beta.mthds` | Concept isolation — Result(value, confidence, is_valid) | `beta_pipeline` |
+| `conflict_pipe_alpha.mthds` | Pipe isolation — shared_step with alpha prompt | `pipe_alpha_pipeline` |
+| `conflict_pipe_beta.mthds` | Pipe isolation — shared_step with beta prompt | `pipe_beta_pipeline` |
+| `multi_concept_alpha.mthds` | Multi-concept isolation — Profile(name, age) + Summary(headline, body) | `multi_alpha_pipeline` |
+| `multi_concept_beta.mthds` | Multi-concept isolation — Profile(title, department, level) + Summary(content) | `multi_beta_pipeline` |
+| `temporal_parallel.mthds` | PipeParallel — concurrent child workflows (ToneAnalysis + LengthAnalysis) | `temporal_parallel_sequence` |
+| `temporal_batch.mthds` | PipeBatch — fan-out to per-item child workflows | `temporal_batch_sequence` |
+| `temporal_condition.mthds` | PipeCondition — conditional routing via child workflow | `temporal_condition_sequence` |
+| `temporal_compose.mthds` | PipeCompose — operator composition + deferred hydration | `temporal_compose_sequence` |
+| `temporal_combined.mthds` | Nested PipeParallel + PipeCondition | `temporal_combined_pipeline` |
+| `large_payload_sequence.mthds` | Codec stress test — 3-step verbose sequence accumulating large WorkingMemory | `large_payload_sequence` |
+
+**Image payload bundles** — in `tests/integration/pipelex/pipes/pipelines/`:
+
+| Bundle | What it tests | Main pipe |
+|--------|--------------|-----------|
+| `crazy_image_generation.mthds` | Image generation pipeline — LLM imagines scene + PipeImgGen renders it, custom ImagePrompt concept | `generate_crazy_image` |
+| `test_image_out_in.mthds` | Image flow — PipeImgGen generates image, then PipeLLM (vision) describes it | `image_out_in` |
diff --git a/.claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py b/.claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py
new file mode 100644
index 000000000..afaaaaac7
--- /dev/null
+++ b/.claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py
@@ -0,0 +1,69 @@
+# ruff: noqa: INP001
+r"""Submitter that mirrors `pipelex run bundle` but attaches a DeliveryAssignment so
+`act_deliver` fires on the runner worker. This forces Temporal's data converter on
+the runner process to deserialize a hydrated PipeOutput — the path that surfaces
+the cross-process dynamic-class registry bug when the bundle defines a dynamic
+concept (e.g. Greeting) the runner has never registered.
+
+Equivalent to the cloud / webhook submission path (`start_pipeline` with a
+WebhookTarget): the runner must decode the dynamic concept class even though the
+crate was only loaded on the router process.
+
+Usage (with both scoped workers up — router + runner on temporal_task_queue):
+
+ .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py \\
+ --bundle tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \\
+ --pipe dynamic_greeting_sequence
+
+Defaults reproduce the canonical Greeting case.
+"""
+
+import argparse
+import asyncio
+from pathlib import Path
+
+from pipelex.cli.cli_factory import make_pipelex_for_cli
+from pipelex.cli.error_handlers import ErrorContext
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, StorageTarget
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipelex import Pipelex
+from pipelex.pipeline.runner import PipelexRunner
+
+DEFAULT_BUNDLE = "tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds"
+DEFAULT_PIPE = "dynamic_greeting_sequence"
+
+
+async def submit(bundle: str, pipe_code: str) -> None:
+ bundle_content = Path(bundle).read_text(encoding="utf-8")
+ runner = PipelexRunner(
+ bundle_uris=[bundle],
+ pipe_run_mode=PipeRunMode.DRY,
+ execution_config=None,
+ library_dirs=None,
+ )
+ delivery = DeliveryAssignment(storage=StorageTarget())
+ print(f"Submitting pipe='{pipe_code}' with delivery_assignment={delivery!r}")
+ response = await runner.execute_pipeline(
+ pipe_code=pipe_code,
+ mthds_contents=[bundle_content],
+ inputs=None,
+ delivery_assignment=delivery,
+ )
+ print(f"OK: pipeline_run_id={response.pipeline_run_id} state={response.pipeline_state}")
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--bundle", default=DEFAULT_BUNDLE)
+ parser.add_argument("--pipe", default=DEFAULT_PIPE, dest="pipe_code")
+ args = parser.parse_args()
+
+ make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN, library_dirs=None, temporal_enabled=True)
+ try:
+ asyncio.run(submit(bundle=args.bundle, pipe_code=args.pipe_code))
+ finally:
+ Pipelex.teardown_if_needed()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/.claude/skills/temporal-test-crate/SKILL.md b/.claude/skills/temporal-test-crate/SKILL.md
new file mode 100644
index 000000000..20629b463
--- /dev/null
+++ b/.claude/skills/temporal-test-crate/SKILL.md
@@ -0,0 +1,138 @@
+---
+name: temporal-test-crate
+description: "Run and diagnose the LibraryCrate integration tests for Temporal. Tests that PipeSequence controllers execute on Temporal workers via LibraryCrate propagation, including concurrent isolation tests for conflicting concepts and pipes. Use when the user says 'test temporal crate', 'run crate tests', 'isolation tests', or wants to verify LibraryCrate on Temporal works."
+---
+
+# Temporal LibraryCrate Integration Tests
+
+Run the integration tests that verify LibraryCrate propagation and per-workflow isolation through Temporal workflows.
+
+## How to narrate the test run
+
+Before running the tests, print a brief schematic explanation for the user so they understand what they're about to see. Use this narration:
+
+---
+
+### What we're testing
+
+These tests verify that **LibraryCrate** — the serializable snapshot of pipes, concepts, and dynamic classes — propagates correctly from the client to Temporal workers, and that concurrent workflows remain isolated from each other.
+
+**The components and how they interact:**
+
+```
+Client (test fixture) Temporal Worker
+───────────────────── ───────────────
+
+1. Load .mthds bundle
+2. Build LibraryCrate (pipes,
+ concepts, fingerprint)
+3. Build PipeJob with crate attached
+4. Submit workflow ──────────────────► 5. WfPipeRouter.run() receives PipeJob
+ 6. load_from_crate() → registers pipes
+ into a scoped Library (ContextVar)
+ 7. Dynamic concept classes generated
+ into a scoped ClassRegistry (ContextVar)
+ 8. WorkingMemory hydrated (deferred)
+ 9. PipeSequence executes child pipes
+ via get_required_pipe() from the
+ scoped library
+ 10. PipeOutput returned to client
+```
+
+**The 5 test suites (19 tests total):**
+
+| Suite | What it proves | Key risk if broken |
+|-------|---------------|--------------------|
+| **TestWfLibraryCrate** (5 tests) | Crate structure is valid, PipeSequence runs end-to-end via Temporal, both `library_dirs` and `mthds_content` loading paths work. Includes a **negative test** (`test_missing_crate_fails_pipe_resolution`) that verifies a PipeJob *without* a crate correctly fails. | Pipes can't resolve on worker |
+| **TestWfDeferredHydration** (2 tests) | Dynamic concept classes (e.g. `Greeting`) are generated on the worker from crate data, WorkingMemory is hydrated after class registration, StructuredContent fields are accessible. | `KajsonDecoderError: Class not found` |
+| **TestWfConcurrentConceptIsolation** (4 tests) | Two workflows define conflicting `Result` concepts (different fields) and run simultaneously on one worker. Per-workflow ClassRegistry scoping keeps them isolated. Repeated 5x to catch races. | Silent data corruption — wrong fields |
+| **TestWfConcurrentPipeIsolation** (4 tests) | Two workflows define conflicting `shared_step` pipes (different prompts) and run simultaneously. Per-workflow Library scoping ensures each resolves its own pipe. Repeated 5x. | Wrong pipe executed silently |
+| **TestWfMultiConceptIsolation** (4 tests) | Worst case: two workflows each define `Profile` AND `Summary` with incompatible structures. 6 concurrent workflows stress-test ContextVar scoping. | Multiple classes cross-contaminate |
+
+---
+
+After running the tests, report results with a summary table showing pass/fail per suite.
+
+## Run the tests
+
+### Step 1: Dry run (default)
+
+Always start with dry mode — fast, no API calls:
+
+```bash
+.venv/bin/pytest -x -v -s tests/integration/pipelex/temporal/library_crate/ -m temporal
+```
+
+To run only the isolation tests:
+
+```bash
+.venv/bin/pytest -x -v -s tests/integration/pipelex/temporal/library_crate/ -m temporal -k "Concurrent or Multi"
+```
+
+### Step 2: Live run (optional)
+
+After a successful dry run, ask the user if they want to run in **live mode**. Live mode makes real LLM API calls and exercises realistic concurrency timing — it's the closest thing to production behavior:
+
+```bash
+.venv/bin/pytest -x -v -s tests/integration/pipelex/temporal/library_crate/ -m temporal --pipe-run-mode live
+```
+
+When reporting live-mode results, show:
+- Pass/fail per test with the test name
+- For passing tests, confirm that structured outputs had the expected fields (e.g. "Alpha Result had score + label", "Beta Result had value + confidence + is_valid")
+- For concurrent tests, confirm that both workflows returned correct results simultaneously
+- Total wall-clock time (live mode is slower due to real API calls)
+
+## Expected warnings in test output
+
+`test_missing_crate_fails_pipe_resolution` is a **negative test** — it intentionally submits a PipeJob *without* a LibraryCrate and asserts that `WorkflowFailureError` is raised. During this test, Temporal logs a `WARNING Failed activation on workflow` with a full traceback ending in `RuntimeError: No current library set`. **This is expected behavior, not a failure.** The test passes (PASSED) because the expected error was correctly raised.
+
+## Prerequisites
+
+```bash
+# Verify venv and temporalio
+.venv/bin/python -c "import temporalio; print(f'temporalio {temporalio.__version__}')"
+# Verify test bundles exist
+ls tests/integration/pipelex/temporal/library_crate/*.mthds
+```
+
+## Interpreting failures
+
+| Error | Cause | Action |
+|-------|-------|--------|
+| `PipeNotFoundError` on worker | LibraryCrate not loaded or not propagated | Check `WfPipeRouter.run()` loads crate; check `library_crate` is set on PipeJob |
+| `KajsonDecoderError: Class 'X' not found` | Dynamic concept classes not registered on worker | Check deferred hydration path in `WfPipeRouter.run()` |
+| `RuntimeError: Failed decoding arguments` | Temporal can't deserialize PipeJob | Check Kajson data converter; may be a serialization issue with LibraryCrate |
+| `WorkflowFailureError` wrapping `TemporalError` | Pipe execution failed on worker | Read the inner error — it's the real cause |
+| Fixture setup error in `pipe_job_from_*` | Library loading failed before Temporal dispatch | Check bundle file exists and is valid MTHDS |
+| `AssertionError: StructuredContent missing field` | ClassRegistry scoping failed — wrong class used | Per-workflow ContextVar is leaking; check `set_current_library` / `teardown_current_library` |
+| `AssertionError` in repeated/high-concurrency tests | Intermittent ContextVar race condition | Scoping mechanism has a race under load |
+
+## Interactive debugging
+
+For tmux-based 3-terminal debugging (server + worker + submitter), use the `/temporal-e2e-validate` skill (Mode 2) instead.
+
+## Test files
+
+| File | Purpose |
+|------|---------|
+| `tests/integration/pipelex/temporal/library_crate/test_wf_library_crate.py` | Single-workflow crate propagation (native Text concepts) |
+| `tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py` | Deferred hydration with dynamic concept (Greeting) |
+| `tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_concept_isolation.py` | Two workflows with conflicting 'Result' concepts |
+| `tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_pipe_isolation.py` | Two workflows with conflicting 'shared_step' pipes |
+| `tests/integration/pipelex/temporal/library_crate/test_wf_multi_concept_isolation.py` | Worst-case multi-class conflicts (Profile + Summary) |
+| `tests/integration/pipelex/temporal/library_crate/conftest.py` | PipeJob fixtures with LibraryCrate for all scenarios |
+| `tests/integration/pipelex/temporal/test_data.py` | Test constants for all scenarios |
+
+## Bundle files
+
+| Bundle | Domain | Concepts | Pipes |
+|--------|--------|----------|-------|
+| `native_text_sequence.mthds` | `native_text_test` | (native Text only) | native_text_sequence, step_one, step_two |
+| `dynamic_concept_sequence.mthds` | `dynamic_concept_test` | Greeting(message, language) | dynamic_greeting_sequence, generate_greeting, summarize_greeting |
+| `conflict_concept_alpha.mthds` | `conflict_alpha` | Result(score, label) | alpha_pipeline, alpha_generate, alpha_summarize |
+| `conflict_concept_beta.mthds` | `conflict_beta` | Result(value, confidence, is_valid) | beta_pipeline, beta_generate, beta_summarize |
+| `conflict_pipe_alpha.mthds` | `pipe_conflict_alpha` | (native Text only) | pipe_alpha_pipeline, shared_step, alpha_finalize |
+| `conflict_pipe_beta.mthds` | `pipe_conflict_beta` | (native Text only) | pipe_beta_pipeline, shared_step, beta_finalize |
+| `multi_concept_alpha.mthds` | `multi_alpha` | Profile(name, age), Summary(headline, body) | multi_alpha_pipeline, generate_profile, generate_summary, finalize |
+| `multi_concept_beta.mthds` | `multi_beta` | Profile(title, department, level), Summary(content) | multi_beta_pipeline, generate_profile, generate_summary, finalize |
diff --git a/.github/workflows/graph-ui-check.yml b/.github/workflows/graph-ui-check.yml
index ca5bca976..558a238c0 100644
--- a/.github/workflows/graph-ui-check.yml
+++ b/.github/workflows/graph-ui-check.yml
@@ -16,6 +16,10 @@ jobs:
- name: Check graph UI assets are in sync
run: |
PINNED_VERSION=$(grep -o '"github:Pipelex/mthds-ui#[^"]*"' package.json | sed 's/.*#//;s/"//')
+ if [ -z "$PINNED_VERSION" ]; then
+ echo "::error::Could not parse pinned mthds-ui version from package.json (regex did not match)."
+ exit 1
+ fi
SYNCED_VERSION=$(cat pipelex/graph/reactflow/assets/.graph-ui-version 2>/dev/null | tr -d '[:space:]')
SYNCED_VERSION="${SYNCED_VERSION:-NONE}"
if [ "$PINNED_VERSION" != "$SYNCED_VERSION" ]; then
diff --git a/.gitignore b/.gitignore
index bc55bffb2..36d6030fe 100644
--- a/.gitignore
+++ b/.gitignore
@@ -45,6 +45,7 @@ results/
# personnal pipelex config files that overrides the default one
pipelex_super.toml
pipelex_override.toml
+pipelex_temporary_override.toml
telemetry_override.toml
# local pipelex service configuration
@@ -59,6 +60,12 @@ gcp_credentials.json
settings.local.json
structures/
.pipelex/storage
+.pipelex/traces
+.pipelex/temporal-payload-store
+
+# Pipelex runtime trace artifacts (workflow ndjson)
+**/traces/
+.pipelex/traces/
# Generated test fixtures (regenerate with `make regenerate-test-models`)
tests/integration/pipelex/fixtures/_generated_model_sets.py
diff --git a/.pipelex/inference/backends/google.toml b/.pipelex/inference/backends/google.toml
index e4fa75131..cad27a0cb 100644
--- a/.pipelex/inference/backends/google.toml
+++ b/.pipelex/inference/backends/google.toml
@@ -129,7 +129,6 @@ outputs = ["image"]
thinking_mode = "none"
costs = { input = 2.0, output = 120 }
-
[nano-banana-2]
model_type = "img_gen"
model_id = "gemini-3.1-flash-image-preview"
diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.toml
index 719a461f3..5cdd79efd 100644
--- a/.pipelex/pipelex.toml
+++ b/.pipelex/pipelex.toml
@@ -63,17 +63,36 @@ pan_to_top = true # Pan to show top of graph on load
theme = "dark"
palette = "dracula"
+####################################################################################################
+# Tracing Config (event log for distributed tracing across Temporal workers)
+####################################################################################################
+
+[pipelex.tracing_config]
+is_enabled = true
+backend = "ndjson"
+
+[pipelex.tracing_config.ndjson]
+traces_dir = ".pipelex/traces"
+
+[pipelex.tracing_config.dynamodb]
+table_name = ""
+region = ""
+
+[pipelex.tracing_config.temporal_dynamodb]
+table_name = ""
+region = ""
+
####################################################################################################
# Storage Config
####################################################################################################
[pipelex.storage_config]
-# Storage method: "local", "in_memory", "s3", or "gcp"
-method = "local"
# Whether to fetch remote HTTP URLs and store them locally
is_fetch_remote_content_enabled = true
# Whether to upload local file paths to storage and replace with pipelex-storage:// URIs
is_upload_local_content_enabled = true
+# Storage method: "local", "in_memory", "s3", or "gcp"
+method = "local"
[pipelex.storage_config.local]
# Local storage settings
@@ -193,3 +212,76 @@ is_dump_response_text_enabled = false
is_dump_kwargs_enabled = false
is_dump_response_enabled = false
is_dump_error_enabled = false
+
+####################################################################################################
+# Temporal Config
+####################################################################################################
+
+[temporal]
+is_enabled = false
+
+[temporal.payload_codec_config]
+is_enabled = false
+size_threshold = 1_048_576 # in bytes (~1 MB, well under Temporal's 2 MB hard limit)
+storage_prefix = "temporal-payloads/"
+
+[temporal.payload_codec_config.storage_provider_config]
+method = "local"
+
+[temporal.payload_codec_config.storage_provider_config.local]
+uri_format = "{hash}"
+local_storage_path = ".pipelex/temporal-payload-store"
+
+[temporal.worker_config]
+task_queue = "temporal_task_queue"
+workflow_execution_timeout = "1:00:00"
+run_timeout = "1:00:00"
+task_timeout = "0:00:10"
+start_delay = "0:00:00"
+rpc_timeout = "1:00:00"
+
+[temporal.worker_config.retry_policy_config]
+initial_interval = "0:00:03"
+backoff_coefficient = 2.0
+maximum_interval = "unlimited"
+maximum_attempts = 3
+non_retryable_error_types = [
+ "ExtractHandleNotFoundError",
+ "FileNotFoundError",
+ "ImgGenHandleNotFoundError",
+ "LLMHandleNotFoundError",
+ "ModelNotFoundError",
+ "ValidationError",
+]
+
+[temporal.temporal_config]
+selected_server = "local"
+
+[temporal.temporal_config.temporal_server_configs.local]
+description = "default localhost"
+target_host = "localhost:7233"
+namespace = "default"
+api_key_method = "none"
+api_key_id = ""
+
+[temporal.temporal_config.temporal_server_configs.testing]
+description = "Temporal server for testing"
+target_host = "${TESTING_TEMPORAL_TARGET_HOST}"
+namespace = "${TESTING_TEMPORAL_NAMESPACE}"
+api_key_method = "secret_provider"
+api_key_id = "${TESTING_TEMPORAL_MAIN_API_KEY}"
+
+[temporal.temporal_config.temporal_log_config]
+is_workflow_info_on_message = false
+is_workflow_info_on_extra = true
+is_full_workflow_info_on_extra = false
+is_activity_info_on_message = false
+is_activity_info_on_extra = true
+is_full_activity_info_on_extra = false
+is_formatter_enabled = true
+is_prefix_enabled = true
+managed_loggers = [
+ "temporalio.activity",
+ "temporalio.workflow",
+ "temporalio.worker._activity",
+]
diff --git a/.vscode/settings.json b/.vscode/settings.json
index 35c3e62b7..4c3156e1d 100644
--- a/.vscode/settings.json
+++ b/.vscode/settings.json
@@ -2,7 +2,12 @@
"python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
"python.analysis.extraPaths": ["./tests"],
"[python]": {
- "editor.defaultFormatter": "charliermarsh.ruff"
+ "editor.defaultFormatter": "charliermarsh.ruff",
+ "editor.formatOnSave": true,
+ "editor.codeActionsOnSave": {
+ "source.organizeImports": "explicit",
+ "source.fixAll": "explicit"
+ }
},
"ruff.configuration": "--config=pyproject.toml",
"mypy.enabled": true,
@@ -37,6 +42,5 @@
"[mthds]": {
"editor.defaultFormatter": "Pipelex.pipelex",
"editor.formatOnSave": true
- },
- "pipelex.graph.direction": "left_to_right"
+ }
}
\ No newline at end of file
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 000000000..7f1bf65b7
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,379 @@
+# Pipelex Coding Rules
+
+## Codex Cloud Commands
+
+### Linting
+
+ After making code changes, you must always lint using `make agent-check`.
+
+ ```bash
+ make agent-check
+ # If the current system doesn't have the `make` command,
+ # lookup the "agent-check" target in the Makefile and run the commands one by one (targets fix-unused-imports format lint pyright mypy)
+ ```
+
+ This runs multiple code quality tools:
+ - Pyright: Static type checking
+ - Ruff: Fix unused imports, lint, format
+ - Mypy: Static type checker
+ - plxt: Format and lint TOML, MTHDS, and PLX files
+
+ Always fix any issues reported by these tools before proceeding.
+
+### Cleaning Derived Files
+
+ If you need to clean derived files and caches, typically after you erased files or moved tests, the linters can get confused, the pytest collection can be off...
+
+ ```bash
+ make cleanderived
+ ```
+
+### Running Tests in Codex Cloud
+
+ To test everything that can be tested from within the Codex Cloud sandbox, run this:
+
+ ```bash
+ make codex-tests
+ # It's equivalent to running pytest with `-m "(dry_runnable or not inference) and not (pipelex_api or codex_disabled)"`
+ # If some test fails, re-run it with `-s -vv` to see more details
+ ```
+
+---
+
+### Prerequisites for running command lines: use virtual environment
+
+ **CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST use the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.
+
+ Call the CLI directly from the virtual environment:
+
+ ```bash
+ .venv/bin/pytest -s -v -k test_render_jinja2_from_text
+ .venv/bin/pipelex validate --all
+ ```
+
+ For standard installations, the virtual environment is named `.venv`. Always check this first. On Windows, the path is `.venv\Scripts\` instead of `.venv/bin/`.
+
+### Pipelex Dev CLI (`pipelex-dev`)
+
+ The `pipelex-dev` CLI provides internal development tools that are not distributed with the package. It is available in the virtual environment.
+
+ ```bash
+ .venv/bin/pipelex-dev --help
+ ```
+
+ Key commands:
+
+ - **`generate-mthds-schema`**: Regenerate the MTHDS JSON Schema (`derived/mthds_schema.json`). Run this after modifying `mthds_schema_generator.py`.
+
+ ```bash
+ .venv/bin/pipelex-dev generate-mthds-schema
+ ```
+
+### Pipelex CLI Commands
+
+ To run the Pipelex CLI commands without the logo, you can use the `--no-logo` flag, this will avoid useless tokens in the console output.
+
+ ```bash
+ .venv/bin/pipelex --help
+ .venv/bin/pipelex build --help --no-logo
+ .venv/bin/pipelex run --help --no-logo
+ .venv/bin/pipelex validate --help --no-logo
+ .venv/bin/pipelex doctor --help --no-logo
+ .venv/bin/pipelex init --help --no-logo
+ ```
+
+## Coding Standards & Best Practices for Python Code
+
+This document outlines the core coding standards, best practices, and quality control procedures for the codebase.
+
+### Python Version Compatibility
+
+ - The project supports Python 3.10+ (`requires-python = ">=3.10,<3.15"`). Never use features introduced after Python 3.10 without a compatibility fallback.
+ - Common pitfalls:
+ - `datetime.UTC` was added in Python 3.11. Use `datetime.timezone.utc` instead.
+ - `StrEnum` was added in Python 3.11. Always import it from `pipelex.types` which handles retrocompatibility.
+ - `type` statement (PEP 695) was added in Python 3.12. Use `TypeAlias` from `typing` instead.
+ - `ExceptionGroup` / `except*` was added in Python 3.11. Avoid unless using the `exceptiongroup` backport.
+
+### Variables, loops and indexes
+
+ - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.
+ - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.
+ - Avoid inline for loops, unless it's ultra-simple and holds on oneline.
+ - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.
+
+### Enums and tests
+
+ - When defining enums related to string values, always inherit from `StrEnum`
+ - When you need the enum value as a string, don't use `str(enum_var)` or `enum_var.value`, just use `enum_var` itself, that is the point of using StrEnum!
+ - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add @property methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases
+ - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.
+ - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import StrEnum
+ ```
+
+### Optionals
+
+- Don't write things like `a = b if b else c`, write `a = b or c` instead.
+
+### Imports
+
+#### **Imports at the top of the file**
+
+ - Avoid as much as possible import statements outside of a module's top-level scope.
+ - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting
+ - Do not bother with ordering the imports or removing unused imports, our Ruff linter will handle it for us.
+ - `if TYPE_CHECKING:` blocks must always be the **last** block in the imports section, placed after all regular imports.
+
+#### **Removing unused imports**
+
+ - To remove unused imports, run `make fix-unused-imports` or `make fui` (shorthand). This is faster and cheaper than rewriting with LLM.
+
+#### **No re-exports in `__init__.py`**
+
+ - Do NOT fill `__init__.py` files with re-exports.
+ - Always use direct full-path imports everywhere. For example:
+
+- **Logging and Pretty Printing**:
+
+ - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:
+ ```python
+ from pipelex import log, pretty_print
+
+ log.info("Hello, world!")
+ ```
+ - Both have a title arg which is handy when logging/printing objects:
+
+ ```python
+ log.verbose("Hello, world!", title="Your first Pipelex log")
+ pretty_print(output_object, title="Your first Pipelex output")
+ ```
+ - Both handle formatting json using Rich, pretty_print makes it prettier.
+
+- **StrEnum and Self type**:
+
+ - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):
+ ```python
+ from pipelex.types import Self, StrEnum
+ ```
+
+### Typing
+
+#### **Always Use Type Hints**
+
+ - Every function parameter must be typed
+ - Every function return must be typed
+ - Use type hints for all variables where type is not obvious
+ - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]
+ - Use type hints for all fields
+ - Use Field(default_factory=...) for mutable defaults
+ - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.
+
+#### **BaseModel / Pydantic Standards**
+
+ - Use `BaseModel` and respect Pydantic v2 standards
+ - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`
+ - Keep models focused and single-purpose
+ - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:
+ ```python
+ from pydantic import BaseModel, Field
+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+
+ class MyModel(BaseModel):
+ names: list[str] = Field(default_factory=list) # OK for strings
+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")
+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")
+ ```
+
+### Factory Pattern
+
+ - Use factory pattern for object creation when dealing with multiple implementations
+ - Our factory methods are named `make_from_...` and such
+
+### Protocols and Interfaces
+
+ - When a getter (e.g. `get_report_delegate()`, `get_library_manager()`) returns a `Protocol` type, callers must only rely on methods declared on that `Protocol`. If you need to call a method that lives on a concrete implementation, **extend the `Protocol`** — do not work around it with `isinstance(...)` checks, `cast(...)`, or inline imports of the concrete class.
+ - When extending the `Protocol`, also update every implementation (including no-op / null implementations) so structural typing is satisfied. For implementations where the method has nothing to do, give it a `pass` body — that is the whole point of having a `Protocol` rather than a concrete base class.
+ - Smell to watch for: an inline `from ... import ConcreteImpl # noqa: PLC0415` followed by `if isinstance(delegate, ConcreteImpl): delegate.some_method(...)`. This is almost always a missing method on the `Protocol`. Promote it.
+ - Use `@override` (from `typing_extensions`) on every implementation of a `Protocol` method, so that signature drift between the `Protocol` and its implementations is caught by the type checker.
+
+### Error Handling
+
+ - Always catch exceptions at the place where you can add useful context to it.
+ - Use try/except blocks with specific exceptions
+ - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
+ - **NEVER write `except Exception:` (or bare `except:`).** Catching the generic `Exception` is FORBIDDEN everywhere except at the absolute root of a CLI command or an API endpoint handler. There are NO other exceptions to this rule. The following are NOT valid justifications and the agent must reject them when tempted:
+ - "I want to log and re-raise" → just let it propagate; logging happens at the root.
+ - "I want to swallow it just in case" → don't. Unknown failures must surface.
+ - "I want to convert it to our error type" → only if you know which specific exception(s) the call can raise; catch those explicitly.
+ - "The third-party lib isn't well typed" → look up the actual exceptions it raises and catch those by name.
+ - "It's safer this way" → it isn't. It hides bugs. Crashing loudly on an unexpected exception is the desired behavior.
+ If you find yourself writing `except Exception`, STOP and ask the user before proceeding.
+ - **Do NOT add try/except speculatively.** Only add a `try`/`except` when (a) a specific exception type is documented or known to be raised by the call, AND (b) you have a concrete recovery path or a meaningful context to attach. Otherwise, let the error bubble up — that is the correct behavior, not a gap to fill.
+ - **Use `try`/`finally` (without `except`) for required cleanup.** When a piece of code MUST run on the way out — releasing a resource, closing a tracer, tearing down a library, clearing per-context state on a manager — put it in a `finally` block (or a `with`/context manager). This is the correct way to guarantee cleanup on both the happy path and the error path: it does NOT suppress the exception, it just ensures cleanup happens before the exception propagates. Do not invent an `except Exception` just to attach cleanup; use `finally` instead. Conversely, do not move cleanup into the success path "to keep things simple" — if it must run on errors too, it belongs in `finally`.
+ - When reviewing existing code, treat any `except Exception` you encounter as a bug to fix (replace it with the specific exception(s) actually raised, or remove the try/except entirely), not as a precedent to follow.
+ - Always add `from exc` to the exception raise statements
+ - Always write the error message as a variable before raising it, for cleaner error traces
+
+ ```python
+ try:
+ self.models_manager.setup()
+ except RoutingProfileLibraryNotFoundError as exc:
+ msg = "The routing library could not be found, please call `pipelex init config` to create it"
+ raise PipelexSetupError(msg) from exc
+ ```
+
+### Documentation
+
+1. **Docstring Format**
+ ```python
+ def process_image(image_path: str, size: tuple[int, int]) -> bytes:
+ """Process and resize an image.
+
+ Args:
+ image_path: Path to the source image
+ size: Tuple of (width, height) for resizing
+
+ Returns:
+ Processed image as bytes
+ """
+ pass
+ ```
+
+2. **Class Documentation**
+ ```python
+ class ImageProcessor:
+ """Handles image processing operations.
+
+ Provides methods for resizing, converting, and optimizing images.
+ """
+ ```
+
+## Standards related to developing the Pipelex codebase
+
+### Spec vs Blueprint Architecture
+
+- **Blueprints** (`pipelex/pipe_operators/`, `pipelex/pipe_controllers/`, `pipelex/core/`) are the MTHDS language reference — what `.mthds` files parse into.
+- **Specs** (`pipelex/builder/pipe/`) are a convenience authoring format for AI agents. Each spec has `to_blueprint()` that transforms it into the corresponding blueprint. Spec-level fields may differ from blueprint-level fields.
+
+When adding validation or fields, decide which layer they belong to. Language rules go on blueprints; authoring convenience goes on specs. See `pipelex/builder/CLAUDE.md` for details.
+
+### Main config
+
+- The main config model is defined using `ConfigModel` classes, derived from `pydantic BaseModel`
+- The model is defined in `pipelex/system/configuration/configs.py`, some of the submodels being defined in their respective sub-packages
+- When adding new configs, place them where it makes most sense, ask the user if you need arbitrage
+- As per our python standards, use StrEnum for multiple-value enums. In that case they must not be strict pydantic fields, i.e. add `= Field(strict=False)`
+- **Important**: NEVER EVER set default values for config attributes in the class definition. All the default values are defined in the main config file `pipelex/pipelex.toml`. The only exception si for Optional values which must be set to `None` in the class definition.
+- If (and only if) you add some config that will clearly make sense for client projects to override, for instance if it's a case of user preference, then you can also add a copy of the settings to the project override config file `.pipelex/pipelex.toml` but leaving them commented out, as an invitation to override.
+- The different `pipelex.toml` files and the python model `configs.py` must be up to date with each other in terms of structure and attributes, otherwise the loading of teh config fails. To check quickly that you're good, just run `make tb` which tests the boot sequence, which includes the config loading.
+
+## Writing tests
+
+### Unit test generalities
+
+NEVER USE unittest.mock. Instead YOU MUST USE pytest-mock: `from pytest_mock import MockerFixture`.
+NEVER EVER put more than one TestClass into a test module.
+
+#### Test file structure
+
+- Name test files with `test_` prefix
+- Place test files in the appropriate test category directory:
+ - `tests/unit/` - for unit tests that test individual functions/classes in isolation
+ - `tests/integration/` - for integration tests that test component interactions
+ - `tests/e2e/` - for end-to-end tests that test complete methods
+- Do NOT add `__init__.py` files to test directories. Test directories do not need to be Python packages.
+- Fixtures are defined in conftest.py modules at different levels of the hierarchy, their scope is handled by pytest
+- Test data is placed inside test_data.py at different levels of the hierarchy, they must be imported with package paths from the root like `from tests.integration.pipelex.cogt.test_data`. Their content is all constants, regrouped inside classes to keep things tidy.
+- Always put tests inside Test classes: 1 TestClass per module.
+- NEVER EVER put more than one TestClass into a test module.
+- Put fixtures into conftest.py files for easy sharing.
+
+#### Markers
+
+Apply the appropriate markers:
+- "gha_disabled: will not be able to run properly on GitHub Actions"
+- "llm: uses an LLM to generate text or objects"
+- "img_gen: uses an image generation AI"
+- "extract: uses text/image extraction from documents"
+- "inference: uses either an LLM or an image generation AI"
+- never add "@pytest.mark.dry_runnable" if you haven't set the "inference" marker
+
+Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.
+
+#### Important rules
+
+- Never use the unittest.mock. Use pytest-mock.
+
+#### Test Class Structure
+
+- Always group the tests of a module into a test class:
+
+```python
[email protected](loop_scope="class")
+class TestFooBar:
+ @pytest.mark.parametrize(
+ "topic, test_case_blueprint",
+ [
+ TestCases.CASE_1,
+ TestCases.CASE_2,
+ ],
+ )
+ async def test_pipe_processing(
+ self,
+ request: FixtureRequest,
+ topic: str,
+ test_case_blueprint: StuffBlueprint,
+ ):
+ # Test implementation
+```
+
+- Never more than 1 class per test module.
+- When testing one method, if possible, limit the number of test functions, but with different test cases in parameters
+
+#### Test Data Organization
+
+- If it's not already there, create a `test_data.py` file in the proper test directory
+- Note how we avoid initializing a default mutable value within a class instance, instead we use ClassVar.
+- Also note that we provide a topic for the test case, which is purely for convenience.
+
+### Best Practices for Testing
+
+- Use strong asserts: test value, not just type and presence.
+- Use parametrize for multiple test cases
+- Test both success and failure cases
+- Verify working memory state
+- Check output structure and content
+- Use meaningful test case names
+- Include concise docstrings explaining test purpose but not on top of the file and not on top of the class.
+- Log outputs for debugging
+
+## Writing Docs
+
+We use Material for MkDocs. All markdown in our docs must be compatible with Material for MkDocs and done using best practices to get the best results with Material for MkDocs.
+
+### MkDocs Markdown Requirements
+
+- Always add a blank line before any bullet lists or numbered lists in MkDocs markdown.
+
+## Test-Driven Development Guide
+
+This document outlines our test-driven development (TDD) process and the tools available for testing.
+
+### TDD Cycle
+
+1. **Write a Test First**
+
+2. **Write the Code**
+ - Implement the minimum amount of code needed to pass the test
+ - Follow the project's coding standards
+ - Keep it simple - don't write more than needed
+
+3. **Run Linting and Type Checking**
+
+4. **Validate tests**
+
+Remember: The key to TDD is writing the test first and letting it drive your implementation. Then, always run the full test suite and quality checks before considering a feature complete.
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 05934f328..9f4dd8edc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,8 +2,60 @@
## [Unreleased]
+### Added
+
+- **`PipeOutput.graph_assembly_error`**: optional `str` field set when the Temporal `act_assemble_graph` activity raises so consumers can distinguish "graph never produced" from "graph assembly failed". Previously the failure was logged as a warning and `graph_spec` silently stayed `None`.
+- **`SecurityError(PipelexError)`**: new base class in `pipelex.base_exceptions` for security-policy violations. `UnsafeSchemaError` now extends it (was `ContentGenerationError`) so security signals are not silently swallowed by domain `except` handlers.
+
+- **PipeRun layer**: New `PipeRunProtocol` with `PipeRun` (direct) and `TemporalPipeRun` (Temporal) implementations. Wraps pipe execution + delivery in a single orchestration unit. In Temporal mode, `WfPipeRun` workflow orchestrates `WfPipeRouter` as a child workflow followed by delivery activities.
+- **Delivery framework**: `DeliveryAssignment` model with `WebhookDeliveryAssignment` (HTTP POST) and `StorageDeliveryAssignment` (S3/local). Deliveries run as Temporal activities with automatic retry, or inline in direct mode.
+- **`TemporalPipeRun.start()`**: Non-blocking start that returns `workflow_id` + `WorkflowHandle` immediately. Replaces the former `PipeRouterTop.start_pipe_job()`.
+- **Pre-generated `pipeline_run_id` support**: `pipeline_run_setup()`, `PipelineFactory`, and `PipelineManager` accept an optional `pipeline_run_id` parameter for external run record creation.
+- **`workflow_id` in start response**: `PipelexPipelineStartResponse` includes `workflow_id`.
+- **Temporal `callbacks` passthrough**: `WorkflowExecutor.start_workflow()` accepts optional Temporal `Callback` objects.
+- **DynamoDB event log backend**: New `DynamoDBEventLog` implementation of `EventLogProtocol` for cloud-based trace event storage. Installable via `pip install "pipelex[dynamodb]"`.
+- **Event log DI**: Event log backend is now configurable via `TracingConfig.backend` (`"ndjson"` or `"dynamodb"`) and injectable via `Pipelex.setup(event_log=...)`. Factory function `make_event_log()` selects the backend from config or hub injection.
+- **Hub event_log support**: `PipelexHub.set_event_log()` / `get_event_log()` for dependency injection of custom event log backends.
+- **Environment-specific config**: `RunEnvironment` enum values updated to full names (`"development"`, `"production"`). Config loaded from `ENVIRONMENT` env var (was `ENV`).
+- **Inference error classification**: New `InferenceErrorCategory` enum (`transient`, `configuration`, `content`, `capacity`) with per-provider helpers that distinguish quota exhaustion from rate limits and detect content policy violations — covers OpenAI, Anthropic, Google, Mistral, AWS Bedrock, and Gateway.
+- **Structured `ErrorReport`**: `PipelexError.to_error_report()` returns a dataclass with error type, category, retryable flag, user action hint, model, and provider.
+
+### Changed
+
+- **Tightened inference quota classification patterns** to reduce false positives on auth/setup errors:
+ - Anthropic: bare `"credit"` token replaced with `"credit balance"`, `"out of credits"`, `"insufficient credit"`. No longer misclassifies `"your credit card was declined"` or `"we will credit your account"` as quota exhaustion.
+ - Google: bare `"billing"` token replaced with `"billing limit"`, `"billing quota"`, `"billing exceeded"`, and `"billing account"`. Continues to match real 429-class messages (`"Billing account is not active"`, `"billing limit reached"`); stops matching unrelated billing-mentioning text.
+ - Mistral / Gateway: bare `"billing"` likewise narrowed to `"billing limit"` / `"billing quota"`. Mistral additionally distinguishes `"out of credits"` vs `"insufficient credits"`.
+- **Schema codegen cache bounded with LRU eviction**: `SchemaToModelFactory._schema_cache` now uses an `OrderedDict` capped at `_SCHEMA_CACHE_MAX_SIZE = 1024` entries. Prevents unbounded memory growth in long-running Temporal workers receiving many distinct schemas.
+- **Removed duplicate `ContentGenerationError` class**: the unused `pipelex.cogt.content_generation.exceptions.ContentGenerationError` was deleted (had zero external imports). Its subclasses re-parented: `NeitherUrlNorDataError` → `PipelexError`, `UnsafeSchemaError` → new `SecurityError`. The temporal `pipelex.temporal.exceptions.ContentGenerationError` is now the only class with that name.
+- **Narrowed `WfPipeRun` exception handling**: `except Exception:` replaced with `ChildWorkflowError` (child-workflow path) and `ActivityError` (graph-assembly path). Real workflow bugs (TypeError, AttributeError, etc.) are no longer masked as pipe failures.
+
+- **`PipeRouterTop` + `PipeRouterChild` merged into `TemporalPipeRouter`**: Single router that auto-detects context (top-level vs child workflow) and dispatches accordingly. Hub no longer needs `_pipe_router_top` — one `_pipe_router` field handles both modes.
+- **`PipelexRunner.execute_pipeline()` uses `get_pipe_run()`**: Runner now delegates to the PipeRun layer instead of calling `get_pipe_router()` directly.
+- **Tracing config structure**: `TracingConfig` now has `backend`, `ndjson` (with `traces_dir`), and `dynamodb` (with `table_name`, `region`) sub-configs. Tracing is enabled by default.
+- **Event log factory replaces hardcoded NdjsonEventLog**: The 3 call sites (`pipeline_run_setup.py`, `wf_pipe_router.py`, `runner.py`) now use `make_event_log()` instead of directly instantiating `NdjsonEventLog`.
+- **Tracing decoupled from Temporal**: Event log is created when `tracing_config.is_enabled`, regardless of whether Temporal is enabled.
+- **`make rules` now generates both `CLAUDE.md` and `AGENTS.md` by default**: the `pipelex.kit_config.preferred_agent_target` setting has been renamed to `preferred_agent_targets` and is now a list. The default is `["claude", "agents"]`. Cursor remains exclusive (`["cursor"]`) and cannot be combined with single-file targets. Downstream projects overriding this setting must rename the key and wrap the value in a list.
+- **All inference workers attach error category and user action**: Every worker across all providers now raises exceptions with an `InferenceErrorCategory` and actionable `user_action` hint.
+- **CLI error output wired to `ErrorReport`**: Error handlers use `to_error_report()` for consistent, structured display.
+- **Graph viewer updated to mthds-ui v0.3.4**: bumped from v0.3.0 — additional polish atop the resizable detail panel, escape-to-close, sticky header, prompt expand/collapse with copy button, concept refinement display.
+- **README install instructions**: Replaced step-by-step Claude Code setup with single copy-paste messages for Claude Code and Codex, added manual install section.
+
+### Removed
+
+- **`PipeRouterTop`** — replaced by `TemporalPipeRouter` + `TemporalPipeRun`.
+- **`PipeRouterChild`** — merged into `TemporalPipeRouter`.
+- **`PipelexHub.set_pipe_router_top()`** — no longer needed; single `set_pipe_router()` suffices.
+
### Fixed
+- **Anthropic structured generation no longer flattens all errors to `CONTENT`.** `AnthropicLLMWorker._gen_object` runs requests through `instructor`, which wraps SDK exceptions in `InstructorRetryException`. The previous handler categorized every wrapped error as `CONTENT`, so genuine `RateLimitError`, `APITimeoutError`, `APIConnectionError`, `PermissionDeniedError`, and `AuthenticationError` cases never reached the typed branches — they were reported as content failures (non-retryable) instead of `TRANSIENT` / `CAPACITY` / `CONFIGURATION`. The handler now unwraps `InstructorRetryException.failed_attempts[-1].exception` (with a `__cause__`/`RetryError.last_attempt` fallback) and routes recognized SDK exceptions through a shared categorization helper; truly unrecognized errors keep the `CONTENT` fallback.
+- **ObjectAssignment deserialization on Temporal workers**: Removed `__init__` class registry check that blocked deserialization of `ObjectAssignment` before `library_crate` was loaded. Validation moved to `validate_before_execution()` method, following the existing codebase pattern.
+- **Dynamic concept classes in Temporal activities**: `WfPipeRouter` now propagates dynamically registered concept classes from the per-workflow registry to the global registry, so child workflows and activities can access them.
+- **Temporal nondeterminism in child workflow IDs**: Replaced `shortuuid.uuid()` with `workflow.uuid4()` in `TemporalPipeRouter` for deterministic child workflow ID generation. The old code caused TMPRL1100 errors on workflow replay.
+- **Enum fields (`choices`) breaking Temporal serialization**: Concepts with `choices` fields generated `Enum` classes that couldn't be resolved during `model_rebuild` or deserialized across Temporal workflow boundaries. Fixed `_exec_and_extract_class` to include all generated types (models + enums) in the forward reference namespace.
+- **Enum serialization in `ContentGeneratorChild`**: `model_dump()` now uses `mode="json"` to serialize enum values as plain strings, preventing type mismatch when validating across different model class instances.
+- **`Anything[]` hydration in Temporal**: Working memory hydration failed for `Anything` concept lists because `AnythingContent` doesn't exist as a class. Hydration now gracefully falls back to embedded `__class__` metadata or `TextContent` for items without type information.
- **Managed-deck detection no longer misclassifies user TOMLs.** `_is_managed_deck_filename` previously treated any non-`x_custom_*.toml` as kit-managed, so a user file like `my_overrides.toml` dropped into the deck dir would have been reported (and on `pipelex update` overwritten) as if it were a kit file. Now requires the numbered `^\d+_.*\.toml$` prefix that the kit actually ships.
- **`PipeLLM` output structure prompt used the unresolved concept ref.** When `is_structure_prompt_enabled`, `get_output_structure_prompt` was called with `pipe_run_params.dynamic_output_concept_ref or output_stuff_spec.concept.concept_ref`. The first form can be a bare code (no domain), which broke downstream concept resolution. Now passes the already-resolved `output_stuff_spec.concept.concept_ref` directly.
- **`PIPELEX_NO_DECK_NOTICE` suppression was too lax.** Any non-empty value silenced the deck-staleness notice — including `PIPELEX_NO_DECK_NOTICE=0`, which users might reasonably expect to _enable_ the notice. Now requires the documented `PIPELEX_NO_DECK_NOTICE=1`.
@@ -13,6 +65,10 @@
- **`docs/tools/cli/update.md` no longer claims the deck advisory fires on every CLI invocation.** Clarified that it is suppressed for `login`, `init`, `doctor`, `update`, and `which`.
+### Security
+
+- **`schema_to_model` rejects `x-python-*` codegen extensions and restricts `__import__` to an allowlist.** `datamodel-code-generator` honors the `x-python-import` JSON Schema extension by emitting arbitrary `from <module> import <name>` statements with no sanitization. Combined with the prior gap that `_make_restricted_builtins()` did not block `__import__`, an attacker able to plant a crafted `object_class_schema` on a Temporal payload (where `ObjectAssignment.object_class_schema: dict[str, Any]` round-trips untouched) could cause arbitrary modules to be imported during `exec()` of generated code. Two-layer defense added: (1) `_reject_unsafe_schema_extensions` raises `UnsafeSchemaError` if any `x-python-*` key is present anywhere in the schema; (2) `__import__` in the exec namespace is now wrapped to allow only `pydantic`, `typing`, `typing_extensions`, `enum`, `datetime`, `decimal`, `uuid`, `__future__`, `collections`, and `re`.
+
## [v0.25.2] - 2026-04-30
### Fixed
diff --git a/CLAUDE.md b/CLAUDE.md
index 16df444c9..f971f5618 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -30,9 +30,9 @@
### Running Tests
- After making code changes, run **targeted tests** based on what you changed. See `tests/CLAUDE.md` for the source-to-test mapping and the pytest command template.
+ `make agent-test` runs the full test suite and is **critical at the end of a coding session** to verify everything is good before wrapping up.
- Fall back to the full suite when changes are broad (3+ areas), touch shared infrastructure, or when preparing a release/push/commit to remote:
+ At intermediate steps during LOCAL development, it's OK to run only the tests relevant to your changes — either by calling pytest directly from the `.venv` (e.g. `.venv/bin/pytest -x -q tests/unit/path/to/test_module.py`) or using `make t TEST=TestClassName`. This applies only to local setups, not cloud agents.
```bash
make agent-test
@@ -76,6 +76,28 @@
```
Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.
+### Temporal Integration Test Options
+
+ The Temporal integration tests support different server modes via the `--temporal-server` pytest CLI option:
+
+ - `--temporal-server`: Which Temporal server to use
+ - `none` (default): in-process test server — no external dependencies, used in CI
+ - `time-skipping`: in-process server with deterministic time control
+ - A profile name from `temporal_server_configs` in `pipelex.toml` (e.g. `local`, `testing`): connects to a real Temporal server using the profile's host, namespace, and API key settings
+
+ ```bash
+ # CI default: in-process server
+ .venv/bin/pytest tests/integration/pipelex/temporal/
+
+ # Dev with local Temporal server
+ .venv/bin/pytest tests/integration/pipelex/temporal/ \
+ --temporal-server local
+
+ # Dev with cloud/testing server
+ .venv/bin/pytest tests/integration/pipelex/temporal/ \
+ --temporal-server testing
+ ```
+
---
### Prerequisites for running command lines: use virtual environment
@@ -216,12 +238,28 @@ This document outlines the core coding standards, best practices, and quality co
- Use factory pattern for object creation when dealing with multiple implementations
- Our factory methods are named `make_from_...` and such
+### Protocols and Interfaces
+
+ - When a getter (e.g. `get_report_delegate()`, `get_library_manager()`) returns a `Protocol` type, callers must only rely on methods declared on that `Protocol`. If you need to call a method that lives on a concrete implementation, **extend the `Protocol`** — do not work around it with `isinstance(...)` checks, `cast(...)`, or inline imports of the concrete class.
+ - When extending the `Protocol`, also update every implementation (including no-op / null implementations) so structural typing is satisfied. For implementations where the method has nothing to do, give it a `pass` body — that is the whole point of having a `Protocol` rather than a concrete base class.
+ - Smell to watch for: an inline `from ... import ConcreteImpl # noqa: PLC0415` followed by `if isinstance(delegate, ConcreteImpl): delegate.some_method(...)`. This is almost always a missing method on the `Protocol`. Promote it.
+ - Use `@override` (from `typing_extensions`) on every implementation of a `Protocol` method, so that signature drift between the `Protocol` and its implementations is caught by the type checker.
+
### Error Handling
- Always catch exceptions at the place where you can add useful context to it.
- Use try/except blocks with specific exceptions
- Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
- - NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands
+ - **NEVER write `except Exception:` (or bare `except:`).** Catching the generic `Exception` is FORBIDDEN everywhere except at the absolute root of a CLI command or an API endpoint handler. There are NO other exceptions to this rule. The following are NOT valid justifications and the agent must reject them when tempted:
+ - "I want to log and re-raise" → just let it propagate; logging happens at the root.
+ - "I want to swallow it just in case" → don't. Unknown failures must surface.
+ - "I want to convert it to our error type" → only if you know which specific exception(s) the call can raise; catch those explicitly.
+ - "The third-party lib isn't well typed" → look up the actual exceptions it raises and catch those by name.
+ - "It's safer this way" → it isn't. It hides bugs. Crashing loudly on an unexpected exception is the desired behavior.
+ If you find yourself writing `except Exception`, STOP and ask the user before proceeding.
+ - **Do NOT add try/except speculatively.** Only add a `try`/`except` when (a) a specific exception type is documented or known to be raised by the call, AND (b) you have a concrete recovery path or a meaningful context to attach. Otherwise, let the error bubble up — that is the correct behavior, not a gap to fill.
+ - **Use `try`/`finally` (without `except`) for required cleanup.** When a piece of code MUST run on the way out — releasing a resource, closing a tracer, tearing down a library, clearing per-context state on a manager — put it in a `finally` block (or a `with`/context manager). This is the correct way to guarantee cleanup on both the happy path and the error path: it does NOT suppress the exception, it just ensures cleanup happens before the exception propagates. Do not invent an `except Exception` just to attach cleanup; use `finally` instead. Conversely, do not move cleanup into the success path "to keep things simple" — if it must run on errors too, it belongs in `finally`.
+ - When reviewing existing code, treat any `except Exception` you encounter as a bug to fix (replace it with the specific exception(s) actually raised, or remove the try/except entirely), not as a precedent to follow.
- Always add `from exc` to the exception raise statements
- Always write the error message as a variable before raising it, for cleaner error traces
diff --git a/Makefile b/Makefile
index de566a374..5736f59f5 100644
--- a/Makefile
+++ b/Makefile
@@ -119,6 +119,8 @@ make test-extract - Run unit tests only for extract (with prints)
make te - Shorthand -> test-extract
make test-img-gen - Run unit tests only for img_gen (with prints)
make test-g - Shorthand -> test-img-gen
+make test-temporal - Run temporal tests (SRV=local|testing MODE=live REG=isolated)
+make ttm - Shorthand -> test-temporal
make check-unused-imports - Check for unused imports without fixing
make fix-unused-imports - Fix unused imports with ruff
@@ -144,6 +146,17 @@ make serve-graph - Start HTTP server to view ReactFlow graphs (PORT
make stop-graph-server - Stop the graph viewer HTTP server
make view-graph - Start server and open ReactFlow graph in browser
+make temporal-server - Start a local Temporal dev server (requires 'temporal' CLI)
+make ts - Shorthand -> temporal-server
+make temporal-stop - Kill the local Temporal dev server (port 7233)
+make tstop - Shorthand -> temporal-stop
+make temporal-worker - Start a Temporal worker (separate process)
+make tw - Shorthand -> temporal-worker
+make temporal-run - Run a pipe through Temporal (real LLM calls)
+make trun - Shorthand -> temporal-run
+make temporal-run-dry - Run a pipe through Temporal (dry run, no LLM)
+make trund - Shorthand -> temporal-run-dry
+
make check - Shorthand -> format lint mypy
make c - Shorthand -> check
make cc - Shorthand -> cleanderived check
@@ -169,8 +182,8 @@ export HELP
format lint ruff-format ruff-lint pyright mypy pylint plxt plxt-format plxt-lint \
rules up-kit-configs ukc check-config-sync ccs check-rules check-urls cu insert-skeleton \
cleanderived cleanenv cleanall \
- test test-xdist t test-quiet tq test-with-prints tp test-inference ti ticc \
- test-llm tl test-img-gen tg test-extract te codex-tests gha-tests \
+ test test-xdist t test-quiet tq test-with-prints tp test-inference ti \
+ test-llm tl test-img-gen tg test-extract te test-temporal ttm codex-tests gha-tests \
run-all-tests run-manual-trigger-gha-tests run-gha_disabled-tests \
validate v check c cc agent-check agent-test \
test-durations td test-durations-serial tds test-time tt test-time-serial tts \
@@ -182,6 +195,8 @@ export HELP
test-count check-test-badge \
serve-graph serve-graph-bg stop-graph-server view-graph sg vg \
sync-graph-ui sgui check-graph-ui-sync cguis \
+ temporal-server ts temporal-stop tstop temporal-worker tw temporal-worker-router twr temporal-worker-runner twn \
+ temporal-run trun temporal-run-dry trund \
docs-deploy-root
all help:
@@ -288,6 +303,8 @@ up-kit-configs:
--exclude='pipelex_override.toml' \
--exclude='telemetry_override.toml' \
--exclude='storage' \
+ --exclude='temporal-payload-store' \
+ --exclude='traces' \
--exclude='x_custom_llm_deck.toml' \
--exclude='x_custom_extract_deck.toml' \
--exclude='mthds_schema.json' \
@@ -428,7 +445,7 @@ gha-tests: env
@echo "• Regenerating test model fixtures with ci profile"
$(VENV_PIPELEX_DEV) preprocess-test-models --generate-fixtures --profile ci
@echo "• Running unit tests for github actions (excluding inference and gha_disabled)"
- $(VENV_PYTEST) -n auto --exitfirst --quiet -m "(dry_runnable or not inference) and not (gha_disabled or pipelex_api)" || [ $$? = 5 ]
+ $(VENV_PYTEST) -n auto --max-worker-restart=2 --timeout=180 --timeout-method=thread --tb=short -m "(dry_runnable or not inference) and not (gha_disabled or pipelex_api)" || [ $$? = 5 ]
run-all-tests: env
$(call PRINT_TITLE,"Running all unit tests")
@@ -628,6 +645,42 @@ test-img-gen: env
tg: test-img-gen
@echo "> done: tg = test-img-gen"
+SRV ?=
+MODE ?=
+REG ?=
+TEMPORAL_PYTEST_MARKERS := $(if $(filter live,$(MODE)),"temporal","temporal and (dry_runnable or not inference)")
+TEMPORAL_TESTS_DIR := tests/integration/pipelex/temporal/
+
+test-temporal: env
+ $(call PRINT_TITLE,"Unit testing Temporal")
+ @if [ -n "$(TEST)" ]; then \
+ if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \
+ $(VENV_PYTEST) --exitfirst -m $(TEMPORAL_PYTEST_MARKERS) -s --lf \
+ $(if $(SRV),--temporal-server $(SRV),) \
+ $(if $(REG),--class-registry $(REG),) \
+ $(if $(filter live,$(MODE)),--pipe-run-mode live,) \
+ $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))) \
+ $(TEMPORAL_TESTS_DIR); \
+ else \
+ $(VENV_PYTEST) --exitfirst -m $(TEMPORAL_PYTEST_MARKERS) -s -k "$(TEST)" \
+ $(if $(SRV),--temporal-server $(SRV),) \
+ $(if $(REG),--class-registry $(REG),) \
+ $(if $(filter live,$(MODE)),--pipe-run-mode live,) \
+ $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))) \
+ $(TEMPORAL_TESTS_DIR); \
+ fi; \
+ else \
+ $(VENV_PYTEST) --exitfirst -m $(TEMPORAL_PYTEST_MARKERS) -s \
+ $(if $(SRV),--temporal-server $(SRV),) \
+ $(if $(REG),--class-registry $(REG),) \
+ $(if $(filter live,$(MODE)),--pipe-run-mode live,) \
+ $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))) \
+ $(TEMPORAL_TESTS_DIR); \
+ fi
+
+ttm: test-temporal
+ @echo "> done: ttm = test-temporal"
+
test-pipelex-api: env
$(call PRINT_TITLE,"Unit testing")
@if [ -n "$(TEST)" ]; then \
@@ -950,6 +1003,67 @@ sg: serve-graph
vg: view-graph
@echo "> done: vg = view-graph"
+temporal-server:
+ $(call PRINT_TITLE,"Starting local Temporal dev server")
+ @if ! command -v temporal >/dev/null 2>&1; then \
+ echo "Error: 'temporal' CLI not found. Install it with: brew install temporal"; \
+ exit 1; \
+ fi
+ @echo "• Temporal Web UI will be available at http://localhost:8233"
+ @echo "• Temporal gRPC service at localhost:7233"
+ @echo "• Press Ctrl+C to stop"
+ temporal server start-dev
+
+ts: temporal-server
+
+temporal-stop:
+ $(call PRINT_TITLE,"Stopping local Temporal dev server")
+ @PID=$$(lsof -tiTCP:7233 -sTCP:LISTEN 2>/dev/null); \
+ if [ -z "$$PID" ]; then \
+ echo "• No process found on port 7233"; \
+ else \
+ kill $$PID && echo "• Killed Temporal server (PID $$PID)"; \
+ fi
+
+tstop: temporal-stop
+
+TEMPORAL_BUNDLE ?= tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_1.mthds
+TEMPORAL_PIPE ?= simple_text_sequence
+TEMPORAL_LIB ?=
+
+TEMPORAL_SCOPE ?=
+
+temporal-worker: env
+ $(call PRINT_TITLE,"Starting Temporal worker$(if $(TEMPORAL_SCOPE), (scope: $(TEMPORAL_SCOPE)),)")
+ $(if $(TEMPORAL_LIB),PIPELEXPATH=$(TEMPORAL_LIB),) $(VENV_PYTHON) -m pipelex.temporal.worker_cli --is-not-sandboxed \
+ $(if $(TEMPORAL_SCOPE),--scope $(TEMPORAL_SCOPE),)
+
+tw: temporal-worker
+
+temporal-worker-router: env
+ $(MAKE) temporal-worker TEMPORAL_SCOPE=router
+
+twr: temporal-worker-router
+
+temporal-worker-runner: env
+ $(MAKE) temporal-worker TEMPORAL_SCOPE=runner
+
+twn: temporal-worker-runner
+
+temporal-run: env
+ $(call PRINT_TITLE,"Running pipe through Temporal")
+ $(VENV_PIPELEX) run bundle $(TEMPORAL_BUNDLE) --temporal --mock-inputs --no-logo --graph \
+ $(if $(TEMPORAL_PIPE),--pipe $(TEMPORAL_PIPE),)
+
+trun: temporal-run
+
+temporal-run-dry: env
+ $(call PRINT_TITLE,"Running pipe through Temporal - dry run")
+ $(VENV_PIPELEX) run bundle $(TEMPORAL_BUNDLE) --temporal --dry-run --mock-inputs --no-logo --graph \
+ $(if $(TEMPORAL_PIPE),--pipe $(TEMPORAL_PIPE),)
+
+trund: temporal-run-dry
+
##########################################################################################
### GRAPH UI ASSET SYNC (from mthds-ui)
##########################################################################################
diff --git a/README.md b/README.md
index 427fa5f1f..94ef4d5ec 100644
--- a/README.md
+++ b/README.md
@@ -66,6 +66,18 @@ Paste this into Claude Code:
Install mthds: run bash -c 'npm install -g mthds && mthds-agent bootstrap && claude plugin marketplace add mthds-ai/mthds-plugins && claude plugin install mthds@mthds-plugins'. Then tell me to /reload-plugins.
```
+Build your first method:
+
+```
+/mthds-build A method to summarize articles with key takeaways for different audiences
+```
+
+Run it:
+
+```
+/mthds-run
+```
+
## With Codex
Paste this into Codex:
@@ -92,9 +104,9 @@ Verify everything is set up correctly:
pipelex doctor
```
-## Python Package Only
+## Standalone CLI
-If you just need the Pipelex runtime without agent integration:
+If you just need the Pipelex CLI without agent integration:
```bash
uv tool install pipelex
diff --git a/docs/stylesheets/mermaid.css b/docs/stylesheets/mermaid.css
index d9052745b..1f33c013b 100644
--- a/docs/stylesheets/mermaid.css
+++ b/docs/stylesheets/mermaid.css
@@ -1,13 +1,37 @@
/* docs/stylesheets/mermaid.css
- Tweak the colours Material exposes for Mermaid */
+ Tweak the colours Material exposes for Mermaid.
+ Material uses --md-mermaid-* for flowcharts and --md-mermaid-sequence-* for
+ sequence diagrams. Both sets must be defined for consistent styling. */
.mermaid {
+ /* Flowchart variables */
--md-mermaid-edge-color: #000000;
--md-mermaid-label-bg-color: #ffdbfa;
--md-mermaid-label-fg-color: #000000;
-
--md-mermaid-node-bg-color: #fff9db;
--md-mermaid-node-fg-color: #000000;
+
+ /* Sequence diagram variables */
+ --md-mermaid-sequence-actor-bg-color: #fff9db;
+ --md-mermaid-sequence-actor-border-color: #333;
+ --md-mermaid-sequence-actor-fg-color: #000;
+ --md-mermaid-sequence-actor-line-color: #666;
+ --md-mermaid-sequence-actorman-bg-color: #fff9db;
+ --md-mermaid-sequence-actorman-line-color: #666;
+ --md-mermaid-sequence-box-bg-color: #f5f5f5;
+ --md-mermaid-sequence-box-fg-color: #000;
+ --md-mermaid-sequence-label-bg-color: #ffdbfa;
+ --md-mermaid-sequence-label-fg-color: #000;
+ --md-mermaid-sequence-loop-bg-color: #f5f5f5;
+ --md-mermaid-sequence-loop-border-color: #666;
+ --md-mermaid-sequence-loop-fg-color: #000;
+ --md-mermaid-sequence-message-fg-color: #000;
+ --md-mermaid-sequence-message-line-color: #000;
+ --md-mermaid-sequence-note-bg-color: #ffdbfa;
+ --md-mermaid-sequence-note-border-color: #333;
+ --md-mermaid-sequence-note-fg-color: #000;
+ --md-mermaid-sequence-number-bg-color: #333;
+ --md-mermaid-sequence-number-fg-color: #fff;
}
/* Give the diagram itself a code-block like backdrop so it stands out
@@ -33,4 +57,4 @@
/* Ensure text inside nodes and labels is black */
.mermaid text {
fill: #000 !important;
-}
\ No newline at end of file
+}
diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.md
new file mode 100644
index 000000000..4397f9c5d
--- /dev/null
+++ b/docs/under-the-hood/distributed-content-generation.md
@@ -0,0 +1,234 @@
+---
+title: "Distributed Content Generation"
+description: "How Pipelex serializes dynamic Pydantic models and large binary content across Temporal workflow/activity boundaries — JSON schema propagation, class source injection, and storage-based payload offloading."
+---
+
+# Distributed Content Generation
+
+This page covers the serialization mechanisms that enable content generation (LLM structured output, image generation, PDF extraction) to run across Temporal worker processes. For the broader Temporal architecture — LibraryCrate propagation, deferred hydration, per-workflow scoping — see [Temporal Integration](./temporal-integration.md).
+
+---
+
+## The Two Serialization Challenges
+
+Content generation introduces two cross-cutting problems that don't exist in single-process mode:
+
+| Challenge | Trigger | Root cause | Solution |
+|-----------|---------|------------|----------|
+| **Dynamic class propagation** | LLM structured output (`PipeLLM` with object concept) | Submitter creates a Pydantic model at runtime from `.mthds` concept definitions; the worker process has no such class | Embed JSON schema in assignment, reconstruct class on worker, carry source code through Temporal payload metadata |
+| **Large payload management** | Image generation (`PipeImgGen`), PDF extraction (`PipeExtract`) | Binary data (base64 images, rendered pages) exceeds Temporal's ~2MB payload limit | Activities store binary data to external storage, return lightweight URI references |
+
+---
+
+## Dynamic Class Propagation
+
+### Schema embedding
+
+When a pipe produces structured output, the caller captures the Pydantic model's JSON schema into the assignment — a serializable Temporal argument that carries everything the worker needs:
+
+```python
+class ObjectAssignment(BaseModel):
+ object_class_name: str
+ object_class_schema: dict[str, Any]
+ llm_assignment_for_object: LLMAssignment
+
+ @staticmethod
+ def make_for_class(
+ object_class: type[BaseModel],
+ llm_assignment: LLMAssignment,
+ ) -> "ObjectAssignment":
+ return ObjectAssignment(
+ object_class_name=object_class.__name__,
+ object_class_schema=object_class.model_json_schema(),
+ llm_assignment_for_object=llm_assignment,
+ )
+```
+
+The schema is a plain `dict` — fully serializable, visible in the Temporal dashboard, and self-contained. No class reference crosses the process boundary.
+
+### Schema-to-model reconstruction
+
+On the worker, `model_class_from_json_schema()` rebuilds a live Pydantic class from the embedded schema:
+
+```python
+source_code = _generate_source_from_schema(schema) # datamodel-code-generator
+reconstructed_class = _exec_and_extract_class(source_code, class_name) # exec()
+reconstructed_class.__kajson_class_source__ = source_code # attached for downstream use
+```
+
+The reconstruction pipeline:
+
+1. **Generate** — `datamodel-code-generator` converts the JSON schema into Python source code defining a `BaseModel` subclass.
+2. **Execute** — `exec()` compiles the source in an isolated namespace and extracts the named class.
+3. **Cache** — A thread-safe SHA-256 hash cache (with double-check locking) avoids redundant generation for the same schema.
+4. **Tag** — The generated source code is attached as `__kajson_class_source__` on the class, enabling Kajson to carry it through Temporal payloads.
+
+!!! warning "Class name normalization"
+ `datamodel-code-generator` normalizes schema titles to PascalCase (e.g., `dynamic_concept_test__Greeting` becomes `DynamicConceptTestGreeting`). The code tries both the original and normalized names. If neither matches, it raises `ValueError` listing the available classes.
+
+### Source code injection into Temporal payloads
+
+When the activity returns a `BaseModel` whose class has `__kajson_class_source__`, the custom `BaseModelPayloadConverter` injects it into the Temporal payload metadata:
+
+**Serialization** (`to_payload`):
+
+```python
+class_source = getattr(type(value), "__kajson_class_source__", None)
+if class_source is not None:
+ metadata["kajson_class_source"] = class_source.encode()
+```
+
+**Deserialization** (`from_payload`):
+
+```python
+source_bytes = payload.metadata.get("kajson_class_source")
+class_source_code = source_bytes.decode() if source_bytes else None
+pydantic_gizmo = kajson.loads(
+ data,
+ class_registry=get_class_registry(),
+ class_source_code=class_source_code,
+)
+# Re-attach source so it survives further Temporal hops
+if class_source_code is not None:
+ if isinstance(pydantic_gizmo, BaseModel):
+ type(pydantic_gizmo).__kajson_class_source__ = class_source_code
+ elif isinstance(pydantic_gizmo, list):
+ for item in pydantic_gizmo:
+ if isinstance(item, BaseModel):
+ type(item).__kajson_class_source__ = class_source_code
+```
+
+The source code rides in Temporal metadata (not the payload data itself), keeping the JSON payload clean and the class reconstruction transparent. After `kajson.loads()` reconstructs the object, the converter re-attaches `__kajson_class_source__` to the deserialized class — this ensures the source survives if the object crosses another Temporal boundary (e.g. workflow returning a result to a parent activity that forwards it elsewhere).
+
+### The type bridge
+
+The class reconstructed on the worker is a structural match to the original, but it is a different Python class in memory. Standard `isinstance()` checks would fail. The caller bridges this gap with a `model_validate` round-trip:
+
+```python
+raw_obj = await llm_gen_object(object_assignment=object_assignment)
+return object_class.model_validate(raw_obj.model_dump(serialize_as_any=True))
+```
+
+`model_dump(serialize_as_any=True)` produces a plain dict from the reconstructed object, and `model_validate()` rebuilds it as a proper instance of the original class. This restores type safety for downstream code.
+
+### End-to-end flow
+
+```mermaid
+sequenceDiagram
+ participant S as Submitter
+ participant W as Worker Activity
+ participant LLM as LLM Provider
+
+ S->>S: object_class.model_json_schema()
+ S->>W: ObjectAssignment (class_name + schema)
+
+ Note over W: model_class_from_json_schema()
+ W->>W: datamodel-code-generator → source
+ W->>W: exec() → class + __kajson_class_source__
+
+ W->>LLM: gen_object(schema=reconstructed_class)
+ LLM-->>W: structured JSON
+
+ W-->>S: BaseModel result<br/>(metadata: kajson_class_source)
+
+ Note over S: kajson.loads(class_source_code=...)
+ S->>S: model_validate() → original type
+```
+
+---
+
+## Large Payload Management
+
+### The payload size constraint
+
+Temporal payloads have a practical limit around 2MB. A single generated image can exceed this in base64 encoding. PDF extraction produces multiple page images. Sending raw binary through Temporal would either fail with a `DataConverterError` or severely degrade workflow history performance.
+
+### The store-then-reference pattern
+
+Activities handle the full binary lifecycle internally — generate, store, return a lightweight reference:
+
+```python
+async def act_img_gen_images(img_gen_assignment: ImgGenAssignment) -> list[ImageContent]:
+ """Generate images and store them, returning lightweight ImageContent references.
+
+ Large binary data (base64/bytes) is stored within the activity and never crosses
+ the Temporal workflow boundary — only URLs are returned.
+ """
+ storage_provider = get_storage_provider()
+ generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)
+ return await img_gen_image_list_and_store(
+ img_gen_assignment=img_gen_assignment,
+ generated_content_factory=generated_content_factory,
+ )
+```
+
+`GeneratedContentFactory` uploads binary data via the configured `StorageProviderAbstract` (S3, local filesystem, etc.) and returns `ImageContent` or `PageContent` — Pydantic models carrying URIs, MIME types, and metadata, but never raw bytes.
+
+!!! info "What crosses the boundary"
+ `ImageContent` carries `url` (storage URI), `public_url`, `mime_type`, `size`, `caption` — but never raw bytes. The `url` can be an S3 URI, HTTP URL, or local file path depending on storage configuration.
+
+### Content type scenarios
+
+| Content type | Activity | What gets stored | What crosses the Temporal boundary |
+|-------------|----------|-----------------|-----------------------------------|
+| Generated image | `act_img_gen_images` | Base64/bytes → S3 via `GeneratedContentFactory` | `list[ImageContent]` with URIs |
+| Extracted pages | `act_extract_gen_extract_pages` | Extracted page images → S3 | `list[PageContent]` with URI references |
+| Rendered page views | `act_render_page_views` | PDF page renders → S3 | `list[ImageContent]` with URIs |
+| LLM text | `act_llm_gen_text` | Nothing stored (text is small) | Plain `str` |
+| LLM object | `act_llm_gen_object` | Nothing stored (JSON is small) | `BaseModel` + `__kajson_class_source__` in metadata |
+
+---
+
+## Kajson Class Resolution
+
+When `kajson.loads()` receives both a `class_registry` and `class_source_code`, the resolution follows a priority chain:
+
+```python
+if class_source_code is not None:
+ source_registry = _build_registry_from_source(class_source_code)
+ if class_registry is not None:
+ # Source-derived classes fill gaps; explicit registry takes priority
+ for name, cls in source_registry.root.items():
+ if not class_registry.has_class(name):
+ class_registry.register_class(cls, name=name, ...)
+ else:
+ class_registry = source_registry
+```
+
+Resolution order:
+
+1. **Explicit `class_registry`** — the per-workflow scoped registry (see [Temporal Integration](./temporal-integration.md#per-workflow-scoping))
+2. **Source-derived classes** — from `class_source_code`, filling gaps in the explicit registry
+3. **`sys.modules`** — already-imported classes
+4. **Dynamic import** — last resort, importing the module path from `__module__` metadata
+
+!!! note "exec() safety"
+ `_build_registry_from_source()` uses `exec()` internally. The source code originates from `datamodel-code-generator` running against the same JSON schema the submitter used — it is never user-supplied arbitrary code.
+
+---
+
+## File Reference
+
+| Component | File |
+|-----------|------|
+| Assignment models (schema embedding) | `pipelex/cogt/content_generation/assignment_models.py` |
+| Schema-to-model reconstruction | `pipelex/cogt/content_generation/schema_to_model.py` |
+| Content generator (type bridge) | `pipelex/cogt/content_generation/content_generator.py` |
+| LLM generation functions | `pipelex/cogt/content_generation/llm_generate.py` |
+| Generated content factory (storage) | `pipelex/cogt/content_generation/generated_content_factory.py` |
+| Temporal data converter | `pipelex/temporal/temporal_data_converter.py` |
+| Image generation activity | `pipelex/temporal/tprl_content_generation/act_img_gen_generate.py` |
+| Extract activity | `pipelex/temporal/tprl_content_generation/act_extract_generate.py` |
+| Page view rendering activity | `pipelex/temporal/tprl_content_generation/act_render_page_views.py` |
+| Kajson serialization | `kajson` (external PyPI package) |
+| ImageContent model | `pipelex/core/stuffs/image_content.py` |
+| PageContent model | `pipelex/core/stuffs/page_content.py` |
+
+---
+
+## Next Steps
+
+- [Temporal Integration](./temporal-integration.md) — LibraryCrate propagation, deferred hydration, per-workflow scoping
+- [Pipe Routing & Execution](./pipe-routing-and-execution.md) — how PipeJobs travel through the system
+- [Architecture Overview](./architecture-overview.md) — the two-layer design and how components fit together
diff --git a/docs/under-the-hood/index.md b/docs/under-the-hood/index.md
index d25894e1d..01e57c5f8 100644
--- a/docs/under-the-hood/index.md
+++ b/docs/under-the-hood/index.md
@@ -17,6 +17,7 @@ Welcome to the technical deep-dives of Pipelex. This section is for contributors
- **Test Profile Configuration** - How to configure which models are used in tests
- **Dry Run Mock Generation** - How mock objects satisfy field validation constraints
- **Init CLI Flows** - How `pipelex init` sets up the configuration directory
+- **Distributed Content Generation** - How dynamic classes and large payloads cross Temporal boundaries
- **Technical Design Decisions** - Why we chose X over Y
- **Module Deep-Dives** - Detailed explanations of specific subsystems
@@ -35,3 +36,6 @@ Welcome to the technical deep-dives of Pipelex. This section is for contributors
- [:material-test-tube: Test Profile Configuration](./test-profile-configuration.md){ .md-button }
- [:material-flask-outline: Dry Run Mock Generation](./dry-run-mock-generation.md){ .md-button }
- [:material-console: Init CLI Flows](./init-cli-flows.md){ .md-button }
+- [:material-pipe: Pipe Routing & Execution](./pipe-routing-and-execution.md){ .md-button }
+- [:material-cloud-sync: Temporal Integration](./temporal-integration.md){ .md-button }
+- [:material-swap-horizontal: Distributed Content Generation](./distributed-content-generation.md){ .md-button }
diff --git a/docs/under-the-hood/pipe-routing-and-execution.md b/docs/under-the-hood/pipe-routing-and-execution.md
new file mode 100644
index 000000000..379b47d94
--- /dev/null
+++ b/docs/under-the-hood/pipe-routing-and-execution.md
@@ -0,0 +1,346 @@
+---
+title: "Pipe Routing & Execution"
+description: "How PipeJobs are created, routed, and executed — covering both direct (single-process) and distributed (Temporal) execution modes."
+---
+
+# Pipe Routing & Execution
+
+Pipelex supports two execution modes for running pipes: **direct execution** (single-process) and **distributed execution** (via Temporal workers). Both modes share the same pipe definitions, library loading, and controller logic. The key difference is *where* the pipe runs and how the PipeJob travels to get there.
+
+---
+
+## Terminology
+
+| Term | Meaning |
+|------|---------|
+| **Direct execution** | All pipes run in the same Python process. Library, class registry, and pipe resolution are shared in-memory. |
+| **Distributed execution** | PipeJob is serialized and sent to a remote Temporal worker. The worker is a separate process (potentially on a different machine). |
+
+---
+
+## Design Principle
+
+Every pipe execution — regardless of mode — follows the same pattern:
+
+1. **Setup**: `pipeline_run_setup()` loads the library, resolves the pipe, initializes working memory, and creates a `PipeJob`
+2. **Route**: A router (`PipeRouter` or `PipeRouterTop`) receives the PipeJob and dispatches it
+3. **Execute**: The pipe's `run_pipe()` method runs, potentially resolving and executing child pipes
+
+The `PipeJob` is the universal unit of execution. It carries everything needed to run a single pipe.
+
+---
+
+## The PipeJob Model
+
+`PipeJob` encapsulates all information needed to execute a pipe.
+
+| Field | Type | Purpose |
+|-------|------|---------|
+| `pipe` | `PipeAbstract` | The resolved pipe object (concrete operator or controller) |
+| `working_memory` | `WorkingMemory \| None` | Runtime data store — typed `Stuff` objects keyed by variable name |
+| `working_memory_raw` | `dict \| None` | Raw JSON dict of WorkingMemory for deferred hydration (Temporal only) |
+| `pipe_run_params` | `PipeRunParams` | Execution config: run mode (LIVE/DRY), output multiplicity, pipe stack for cycle detection |
+| `job_metadata` | `JobMetadata` | Pipeline run ID, user ID, OTel tracing context, graph tracing context |
+| `output_name` | `str \| None` | Override for the output variable name |
+| `library_crate` | `LibraryCrate \| None` | Serializable library snapshot for distributed execution |
+
+`PipeJob` is created by `pipeline_run_setup()`, which handles library loading, pipe resolution, working memory initialization, and telemetry setup. For Temporal dispatch, `prepare_for_temporal()` moves `working_memory` to `working_memory_raw` (deferred hydration) and ensures the crate is attached.
+
+---
+
+## Library Loading
+
+Before a pipe can execute, the **library** must be loaded. The library contains:
+
+- **Pipes** — all pipe definitions (operators and controllers), resolved by code via `get_required_pipe()`
+- **Concepts** — semantic type definitions that determine what data a `Stuff` object holds
+- **Domains** — namespaces that group related pipes and concepts
+
+### Base vs Custom Libraries
+
+- **Base libraries** are loaded from directories listed in `PIPELEXPATH`. They contain shared pipe/concept definitions available to all executions.
+- **Custom bundles** are per-request `mthds_content` strings. Each API call can bring its own definitions.
+
+### Dynamic Class Generation
+
+When a `.mthds` file declares a concept like `RawText = "Raw input text..."`, the library loading process:
+
+1. `ConceptFactory._handle_basic_blueprint()` detects the concept declaration
+2. `StructureGenerator.generate_from_structure_blueprint()` dynamically creates a Python class (e.g., `RawText` inheriting from `TextContent`)
+3. The class is registered with the active `ClassRegistry` (accessed via `hub.get_class_registry()`) so it can be serialized/deserialized
+
+These dynamically-generated classes become the `content` type of `Stuff` objects in `WorkingMemory`.
+
+!!! info "Why Dynamic Classes Matter"
+ When a PipeJob is serialized (e.g., for Temporal transport), Kajson embeds `__class__` and `__module__` metadata. The receiving process must have these classes registered in its class registry to deserialize the payload.
+
+---
+
+## Direct Execution
+
+In direct execution, everything runs in a single Python process. This is the default mode when Temporal is not enabled.
+
+### Flow
+
+```
+pipeline_run_setup()
+ ├── Load library (library_manager singleton)
+ ├── Generate dynamic concept classes
+ ├── Register classes with Kajson
+ ├── Resolve pipe via get_required_pipe(pipe_code)
+ ├── Initialize WorkingMemory from inputs
+ └── Return PipeJob
+
+PipeRouter.run(pipe_job)
+ ├── Notify observers (before)
+ ├── _run_pipe_job(pipe_job)
+ │ └── pipe_job.pipe.run_pipe(...) ← delegates directly to the pipe
+ │ ├── Concrete pipe: execute operator logic (LLM call, template, etc.)
+ │ └── Controller: resolve child pipes via get_required_pipe(),
+ │ then recursively call child.run_pipe()
+ ├── Notify observers (after)
+ └── Return PipeOutput
+```
+
+### Router Selection
+
+The router is selected during `Pipelex.setup()`:
+
+```python
+if get_config().temporal.is_enabled:
+ effective_pipe_router = make_tprl_pipe_router_top() # Distributed
+else:
+ effective_pipe_router = PipeRouter(observer=...) # Direct
+```
+
+### How PipeRouter Works
+
+`PipeRouter` implements `PipeRouterProtocol` with a minimal `_run_pipe_job()`:
+
+```python
+async def _run_pipe_job(self, pipe_job, wfid=None):
+ return await pipe_job.pipe.run_pipe(
+ job_metadata=pipe_job.job_metadata,
+ working_memory=pipe_job.working_memory,
+ output_name=pipe_job.output_name,
+ pipe_run_params=pipe_job.pipe_run_params,
+ )
+```
+
+The router does not route by pipe type — it delegates to the pipe itself. Controllers handle their own orchestration internally.
+
+---
+
+## Pipe Controllers
+
+Controllers are pipes that orchestrate the execution of other pipes. They resolve child pipes at runtime via `get_required_pipe()` from the library.
+
+| Controller | Behavior |
+|------------|----------|
+| **PipeSequence** | Executes `sequential_sub_pipes` one after another. Each step receives working memory with outputs from previous steps. |
+| **PipeBatch** | Iterates over a `ListContent` input. For each item, loads `branch_pipe` and executes it with a deep copy of working memory. Items run concurrently via asyncio. |
+| **PipeCondition** | Evaluates a Jinja2 `expression` against working memory, maps the result via `outcome_map` to a pipe code, and executes that pipe. |
+| **PipeParallel** | Loads multiple child pipes and executes them concurrently, each with its own working memory copy. |
+
+All controllers follow the same pattern:
+
+1. Call `get_required_pipe(child_pipe_code)` to resolve the child pipe from the library
+2. Route through `get_pipe_router().run(PipeJob(...))` — the hub auto-selects the right router
+3. Aggregate results into working memory or output
+
+### Auto-Switching Router
+
+The hub (`get_pipe_router()`) automatically returns the correct router based on context:
+
+- **Direct execution**: Returns `PipeRouter` — child pipes run in-process
+- **Distributed execution, outside workflow** (submitter side): Returns `PipeRouterTop` — submits a top-level Temporal workflow
+- **Distributed execution, inside workflow** (worker side): Returns `PipeRouterChild` — creates a Temporal child workflow via `execute_child_workflow()`
+
+This means each child pipe in a controller gets its own Temporal workflow boundary in distributed mode — enabling independent retries, separate worker assignment, and per-pipe visibility in the Temporal UI.
+
+!!! note "Library Dependency"
+ Controllers depend on the library being loaded in the current process. `get_required_pipe()` queries the library scoped to the current workflow via `ContextVar`, which must have been populated by loading a `LibraryCrate`. In distributed execution, each `WfPipeRouter` loads the crate from the `PipeJob` into a per-workflow `Library` instance. Controllers call `get_required_pipe()` inside Temporal workflow code — accessing per-workflow state via ContextVar. The Temporal sandbox is disabled (`--is-not-sandboxed`) because library loading is a side effect incompatible with replay semantics.
+
+---
+
+## Distributed Execution
+
+In distributed execution, the PipeJob is serialized and sent to a Temporal worker for execution.
+
+### Flow
+
+```mermaid
+sequenceDiagram
+ participant S as Submitter (API/CLI)
+ participant T as Temporal Server
+ participant W as Worker
+
+ Note over S: pipeline_run_setup()
+ S->>S: Load library, generate dynamic classes
+ S->>S: Resolve pipe, build LibraryCrate
+ S->>S: Create PipeJob (crate attached)
+
+ Note over S: PipeRouterTop.run()
+ S->>S: prepare_for_temporal()<br/>(WM → working_memory_raw)
+ S->>T: Submit WfPipeRouter(PipeJob)
+ T->>W: Dispatch workflow
+
+ Note over W: WfPipeRouter.run()
+ W->>W: Create per-workflow ClassRegistry
+ W->>W: Load LibraryCrate (register classes)
+ W->>W: Hydrate working_memory_raw → WM
+ W->>W: pipe.run_pipe()
+
+ alt Concrete pipe
+ W->>W: Execute via Activity
+ else Controller pipe
+ W->>T: Child workflow (crate propagates)
+ T->>W: Child result
+ end
+
+ W->>W: Dehydrate PipeOutput<br/>(WM → working_memory_raw)
+ W->>T: Return dehydrated PipeOutput
+ T->>S: Workflow result
+ S->>S: Hydrate PipeOutput<br/>(working_memory_raw → WM)
+```
+
+### Key Components
+
+**PipeRouterTop** (`pipelex/temporal/tprl_pipe/pipe_router_top.py`)
+
+Implements `PipeRouterProtocol` like `PipeRouter`, but dispatches via Temporal instead of direct execution. Uses `WorkflowExecutor` to submit `WfPipeRouter` workflows.
+
+**WfPipeRouter** (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`)
+
+The Temporal workflow that runs on the worker. Receives a deserialized `PipeJob` and calls `pipe.run_pipe()` — exactly like the direct router.
+
+**Kajson Data Converter** (`pipelex/temporal/temporal_data_converter.py`)
+
+Custom Temporal payload converter that uses Kajson for serializing/deserializing Pydantic models. Preserves subclass types during transport, which is critical because `PipeJob.pipe` is a `PipeAbstract` subclass and `WorkingMemory` contains `Stuff` objects with concept-specific content classes.
+
+**Worker CLI** (`pipelex/temporal/worker_cli.py`)
+
+Entry point for the worker process. Calls `Pipelex.make(temporal_enabled=True)` to initialize the framework, then starts the Temporal worker with registered workflows and activities.
+
+### Content Generation Workflows
+
+Concrete pipe operators (PipeLLM, PipeCompose, PipeExtract, PipeImgGen) use dedicated Temporal workflows and activities for their actual work:
+
+| Workflow | Activity | Purpose |
+|----------|----------|---------|
+| `wf_make_llm_text` | `act_llm_gen_text` | LLM text generation |
+| `wf_make_object` | `act_llm_gen_object` | LLM structured output |
+| `wf_make_jinja2_text` | `act_jinja2_gen_text` | Jinja2 template rendering |
+| `wf_make_extract` | `act_extract_gen_extract_pages` | Document extraction |
+| `wf_make_image` | `act_img_gen_images` | Image generation |
+
+### Worker Environment
+
+The `@with_conditional_worker` decorator on `PipeRouterTop._run_pipe_job()` supports two environments:
+
+- **EXTERNAL** (production) — assumes a worker is already running, submits the workflow directly
+- **INTERNAL** (testing) — spins up an embedded worker for the duration of the execution
+
+!!! note "Future Improvement: Per-Pipe Routing"
+ Router selection is currently global and binary: either all pipes go through Temporal, or none do. A simple `PipeCompose` (microseconds of Jinja2 rendering) gets the same Temporal overhead as a `PipeLLM` (minutes of API call time). Per-pipe or per-type routing decisions could improve efficiency.
+
+---
+
+## LibraryCrate, Deferred Hydration, and Per-Workflow Isolation
+
+Distributed execution introduces three mechanisms that don't exist in direct mode. These solve the fundamental challenge: the worker is a separate process that doesn't share the submitter's library, class registry, or working memory state.
+
+For a detailed walkthrough of these mechanisms, see [Temporal Integration](./temporal-integration.md).
+
+### LibraryCrate Propagation
+
+The submitter builds a `LibraryCrate` — a serializable snapshot of all pipes and concepts — and attaches it to the `PipeJob`. Each `WfPipeRouter` on the worker loads the crate into a per-workflow scoped library, making all pipes resolvable via `get_required_pipe()`.
+
+### Deferred Hydration (Input and Output)
+
+Deferred hydration applies to both **PipeJob inputs** and **PipeOutput return values**. In both cases, `WorkingMemory` is serialized as a raw JSON dict (`working_memory_raw`) instead of a typed object, avoiding deserialization failures when dynamic concept classes aren't registered in the receiving process's class registry.
+
+- **Input**: `PipeJob.prepare_for_temporal()` moves `working_memory` to `working_memory_raw` before dispatch. The worker hydrates it after loading the crate.
+- **Output**: `PipeOutput.prepare_for_temporal()` does the same before returning from a child workflow. The parent (`PipeRouterChild` or `PipeRouterTop`) hydrates after receiving.
+
+### Per-Workflow ClassRegistry and Library Scoping
+
+Each workflow creates its own `ClassRegistry` (pre-seeded from the global registry) and its own `Library` instance, both scoped via `ContextVar`. This ensures concurrent workflows with conflicting concept names (e.g., two bundles that both define `Result` with different fields) don't cross-contaminate.
+
+### Known Limitation: StuffArtefact Serialization in Dry-Run
+
+PipeCondition and PipeCompose dispatch internal `WfMakeJinja2Text` sub-workflows that serialize working memory contents through the Temporal data converter. In dry-run mode, working memory contains `StuffArtefact` debug objects that are not JSON-serializable, causing these internal workflows to fail. PipeBatch also fails in dry-run (likely a related serialization issue in child PipeJob creation). PipeSequence and PipeParallel are unaffected — they dispatch child pipes through `SubPipe.run_pipe()` without internal templating workflows. See [Temporal Integration](./temporal-integration.md#known-limitation-stuffartefact-serialization-in-dry-run) for details and fix direction.
+
+---
+
+## Architecture Diagram
+
+```mermaid
+flowchart TB
+ subgraph Setup["Pipeline Setup (API / CLI Process)"]
+ S1["pipeline_run_setup()"]
+ S2["Load library"]
+ S3["Generate dynamic classes"]
+ S4["Resolve pipe"]
+ S5["Build LibraryCrate"]
+ S6["Create PipeJob (with crate)"]
+ S1 --> S2 --> S3 --> S4 --> S5 --> S6
+ end
+
+ S6 --> Decision{temporal.is_enabled?}
+
+ subgraph Direct["Direct Execution"]
+ D1["PipeRouter._run_pipe_job()"]
+ D2["pipe.run_pipe()"]
+ D3["Controller: get_required_pipe()"]
+ D4["Recursive child execution"]
+ D1 --> D2
+ D2 --> D3
+ D3 --> D4
+ end
+
+ subgraph Distributed["Distributed Execution (Temporal)"]
+ T1["PipeRouterTop: prepare_for_temporal()"]
+ T2["Kajson serialize PipeJob + crate"]
+ T3["Temporal Server"]
+ T4["Worker: Kajson deserialize"]
+ T5["WfPipeRouter: load crate, hydrate WM"]
+ T6["pipe.run_pipe()"]
+ T1 --> T2 --> T3 --> T4 --> T5 --> T6
+ end
+
+ Decision -- No --> D1
+ Decision -- Yes --> T1
+
+ D4 --> Result["PipeOutput"]
+ T6 --> Result
+```
+
+---
+
+## File Reference
+
+| Component | File |
+|-----------|------|
+| PipeJob model | `pipelex/pipe_run/pipe_job.py` |
+| Pipeline setup | `pipelex/pipeline/pipeline_run_setup.py` |
+| PipeRouter (direct) | `pipelex/pipe_run/pipe_router.py` |
+| PipeRouterProtocol | `pipelex/pipe_run/pipe_router_protocol.py` |
+| PipeRouterTop (distributed, outside workflow) | `pipelex/temporal/tprl_pipe/pipe_router_top.py` |
+| PipeRouterChild (distributed, inside workflow) | `pipelex/temporal/tprl_pipe/pipe_router_child.py` |
+| Auto-switch utility | `pipelex/temporal/temporal_workflow_utils.py` |
+| WfPipeRouter workflow | `pipelex/temporal/tprl_pipe/wf_pipe_router.py` |
+| Kajson data converter | `pipelex/temporal/temporal_data_converter.py` |
+| Worker CLI | `pipelex/temporal/worker_cli.py` |
+| Library manager | `pipelex/libraries/library_manager.py` |
+| ConceptFactory | `pipelex/core/concepts/concept_factory.py` |
+| StructureGenerator | `pipelex/core/concepts/structure_generation/generator.py` |
+| LibraryCrate model | `pipelex/libraries/library_crate.py` |
+| LibraryCrate factory | `pipelex/libraries/library_crate_factory.py` |
+| Deferred hydration utility | `pipelex/temporal/tprl_pipe/hydration.py` |
+| Hub (get_required_pipe, get_class_registry) | `pipelex/hub.py` |
+| PipeSequence | `pipelex/pipe_controllers/sequence/pipe_sequence.py` |
+| PipeCondition | `pipelex/pipe_controllers/condition/pipe_condition.py` |
+| PipeBatch | `pipelex/pipe_controllers/batch/pipe_batch.py` |
+| PipeParallel | `pipelex/pipe_controllers/parallel/pipe_parallel.py` |
+| Router selection | `pipelex/pipelex.py` (in `Pipelex.setup()`) |
diff --git a/docs/under-the-hood/temporal-integration.md b/docs/under-the-hood/temporal-integration.md
new file mode 100644
index 000000000..0e2a3c07f
--- /dev/null
+++ b/docs/under-the-hood/temporal-integration.md
@@ -0,0 +1,258 @@
+---
+title: "Temporal Integration"
+description: "How Pipelex distributes pipe execution across Temporal workers — LibraryCrate propagation, deferred hydration, per-workflow isolation, and the serialization chain."
+---
+
+# Temporal Integration
+
+This page covers the Temporal-specific mechanisms that enable distributed pipe execution. For the general pipe routing architecture (shared between direct and distributed modes), see [Pipe Routing & Execution](./pipe-routing-and-execution.md).
+
+---
+
+## The Three Challenges
+
+Distributing pipe execution across separate worker processes introduces three challenges that don't exist in direct (single-process) mode:
+
+1. **Pipe resolution** — Controllers call `get_required_pipe()` to resolve child pipes. The worker's library must contain those pipes.
+2. **Dynamic class deserialization** — `.mthds` bundles generate Python classes at runtime (e.g., `RawText` inheriting from `TextContent`). The worker must have those classes registered before deserializing `WorkingMemory`.
+3. **Concurrent isolation** — Multiple workflows running on the same worker may define conflicting class/pipe names. Each workflow needs its own scoped registry and library.
+
+These are solved by three mechanisms: LibraryCrate propagation, deferred hydration, and per-workflow scoping.
+
+---
+
+## LibraryCrate
+
+A `LibraryCrate` is a serializable snapshot of the library — two flat dicts of blueprints:
+
+```python
+class LibraryCrate(BaseModel):
+ concepts: dict[str, ConceptBlueprint | str] # concept_ref -> blueprint or description
+ pipes: dict[str, PipeBlueprintUnion] # pipe_ref -> blueprint
+ domains: dict[str, DomainBlueprint] # domain_code -> domain metadata
+ source_map: dict[str, str] # ref -> source file (error traceability)
+ fingerprint: str # SHA256 of serialized content
+```
+
+Domain is encoded in the dictionary keys (e.g., `scoring.WeightedScore`, `scoring.compute_score`), not in a structural container. `domains` carries per-domain metadata (description, system prompt). `source_map` enables error messages that trace back to origin files. The `fingerprint` enables idempotent loading — if a worker already loaded a crate with the same fingerprint, it skips the reload.
+
+### How it propagates
+
+```mermaid
+sequenceDiagram
+ participant S as Submitter
+ participant W as Worker
+
+ S->>S: Load library, build LibraryCrate
+ S->>W: PipeJob (crate attached)
+
+ Note over W: WfPipeRouter.run()
+ W->>W: load_from_crate(crate)
+ W->>W: pipe.run_pipe()
+
+ Note over W: Controller dispatches child
+ W->>W: PipeRouterChild → child workflow
+ Note over W: Child WfPipeRouter.run()
+ W->>W: load_from_crate(same crate)
+ W->>W: child pipe.run_pipe()
+
+ W->>S: PipeOutput (dehydrated)
+ S->>S: Hydrate PipeOutput
+```
+
+Every child workflow receives the crate through its `PipeJob`. Each `WfPipeRouter` loads the crate into its own scoped library. This ensures every workflow — parent or child, on any worker — can resolve all pipes.
+
+### Visibility
+
+The `LibraryCrate` is a Pydantic model. Temporal serializes it as structured JSON in the workflow input, making it fully visible in the Temporal dashboard — not opaque bytes.
+
+---
+
+## Deferred Hydration
+
+### The chicken-and-egg problem
+
+Temporal's data converter deserializes payloads (workflow inputs and return values) using Kajson, which needs dynamic concept classes registered to reconstruct typed `WorkingMemory`. But those classes only get registered when the `LibraryCrate` is loaded inside the workflow. This creates two deserialization gaps:
+
+1. **Input**: The `PipeJob` workflow input contains `WorkingMemory` with dynamic classes that don't exist yet on the worker.
+2. **Output**: A child workflow's `PipeOutput` return value contains `WorkingMemory` with dynamic classes. The parent's data converter runs in the Temporal SDK's event processing context (outside the workflow coroutine's `ContextVar` scope), so it can't access the parent's per-workflow class registry.
+
+### The solution
+
+Both `PipeJob` and `PipeOutput` carry a `working_memory_raw` field for deferred hydration:
+
+```python
+class PipeJob(BaseModel):
+ working_memory: WorkingMemory | None = None
+ working_memory_raw: dict[str, Any] | None = None
+ library_crate: LibraryCrate | None = None
+
+class PipeOutput(PipeOutputAbstract[WorkingMemory]):
+ working_memory: WorkingMemory = Field(default_factory=WorkingMemory)
+ working_memory_raw: dict[str, Any] | None = None
+```
+
+**Input dehydration/hydration cycle:**
+
+- Before dispatch, `PipeJob.prepare_for_temporal()` moves `working_memory` to `working_memory_raw` (a plain dict via `dump_for_temporal()`).
+- After the worker loads the crate, `WfPipeRouter` hydrates the raw dict back to typed `WorkingMemory`.
+
+**Output dehydration/hydration cycle:**
+
+- Before returning from a workflow, `WfPipeRouter` calls `PipeOutput.prepare_for_temporal()` to dehydrate `WorkingMemory` to `working_memory_raw`.
+- The raw dict flows **end-to-end**: child workflow → parent `WfPipeRun` → `act_deliver` activity, all without intermediate rehydration. Parents do not own a per-workflow registry that would let them rehydrate the child's output, and the activity worker is typically a different process that has not loaded the crate.
+- Only the **top-level submitter** rehydrates, via `rehydrate_pipe_output_with_crate()` (`pipelex/temporal/tprl_pipe/submitter_hydration.py`). When the submitter has not loaded the bundle locally (e.g., a remote API client that built the `PipeJob` from a serialized `LibraryCrate`), this function opens a fresh per-call scoped Library, loads the crate into a scoped `ClassRegistry` pre-seeded from the global, hydrates inside the scope, then tears down — leaving the submitter's global registry untouched.
+
+The hydration function (`pipelex/temporal/tprl_pipe/hydration.py`) iterates the raw dict, uses `StuffContentFactory` to reconstruct typed content objects using the registered classes, and preserves aliases. It also includes a small but unintuitive guard (`_validate_as_known_class`) that round-trips items through `model_dump()` to defeat cross-exec class-identity mismatches: kajson may eagerly rebuild instances using one class identity, then `load_from_crate` re-execs the source and replaces the registry entry with a new identity (same name, different `id()`), which would otherwise cause Pydantic's `model_validate` to reject the older instance.
+
+### Why output dehydration goes end-to-end
+
+Earlier designs had the parent `WfPipeRun` rehydrate the child's `PipeOutput` before scheduling `act_deliver`, then propagated dynamic classes from the per-workflow registry into the worker's global `KajsonManager` registry so the activity worker could decode the typed payload. This worked on a single worker but broke under distributed Temporal: any worker polling the task queue can pick up the activity, and `ContextVars` / per-worker globals do not cross processes. The current design eliminates that hack — `act_deliver` receives the raw dict directly, never needs class resolution for transport, and the activity worker does not need the crate loaded.
+
+### Activity-side rendering of the raw working memory
+
+`act_deliver` cannot assume dynamic concept classes are registered on its worker. `DeliveryExecutor.generate_result_files()` (`pipelex/pipe_run/delivery_executor.py`) therefore branches on which field is populated:
+
+- **Typed path** (`working_memory` set, in-process / same-worker): existing typed renderers run unchanged.
+- **Raw path** (`working_memory_raw` set, cross-process Temporal): `working_memory.json` is written directly from the raw dict; for the main stuff, `try_local_hydrate_stuff()` attempts a single-Stuff hydration using only globally registered classes (built-in `StuffContent` subclasses are always present). On success, typed renderers produce `main_stuff.{md,html}` and the viewer; on miss (dynamic concept class not registered locally), a generic field-walking fallback renders JSON-in-markdown and `<pre>`-in-HTML so users still get readable output. Misses log a warning so silent regressions on built-in hydration surface.
+
+Tradeoff: dynamic concepts lose typed rendering on the activity worker, but the activity worker no longer needs the crate, which is the precondition for true distributed execution.
+
+### Boundary-by-boundary summary
+
+| Boundary | Direction | What crosses | Hydration on receive? |
+|----------|-----------|--------------|------------------------|
+| Submitter → top `WfPipeRouter` | input | `PipeJob` with `working_memory_raw` + `library_crate` | Yes — worker loads crate, then hydrates inside scope |
+| Parent `WfPipeRouter` → child `WfPipeRouter` | input | Same `PipeJob` shape (crate re-loaded idempotently via fingerprint) | Yes — child worker loads crate, hydrates inside its own scope |
+| Child `WfPipeRouter` → parent `WfPipeRun` | output | `PipeOutput` with `working_memory_raw` | **No** — parent forwards raw dict |
+| Parent `WfPipeRun` → `act_deliver` | activity arg | `PipeOutput` with `working_memory_raw` (no crate) | **No** — activity renders from raw, with best-effort local hydration of the main stuff using only globally registered classes |
+| Top `WfPipeRouter` → submitter | output | `PipeOutput` with `working_memory_raw` | Yes — `rehydrate_pipe_output_with_crate()` opens a fresh scoped Library if a crate is available, otherwise falls back to the global registry |
+
+### Dashboard visibility
+
+`working_memory_raw` is a plain JSON dict — fully readable in the Temporal dashboard without needing class resolution.
+
+---
+
+## Per-Workflow Scoping
+
+### Why per-workflow isolation matters
+
+A single Temporal worker may process multiple concurrent workflows. If two workflows define a concept named `Result` with different structures (e.g., `Result(score, label)` vs `Result(value, confidence, is_valid)`), they must not share the same `ClassRegistry` or `Library`. Without isolation, the second workflow's class registration would overwrite the first's, causing silent data corruption.
+
+### How it works
+
+Each `WfPipeRouter.run()` creates its own scoped state:
+
+1. **ClassRegistry**: A new `ClassRegistry` pre-seeded from the global registry (which has base classes from `PIPELEXPATH`). Dynamic classes from the crate are registered here — not in the global registry.
+
+2. **Library**: A new `Library` instance opened via `library_manager.open_library()`, with the `ClassRegistry` attached as a `PrivateAttr`. The library is set as the current library via the `_library_id` `ContextVar`.
+
+3. **ClassRegistry lookup chain**: `hub.get_class_registry()` reads `_library_id` from the `ContextVar`, gets the library's attached `ClassRegistry`. Falls back to the global registry if no library is set.
+
+4. **Cleanup**: `library_manager.teardown(library_id)` deletes the library and its `ClassRegistry`. The `ContextVar` is reset via `teardown_current_library()`. No manual GC needed — the `ClassRegistry` is garbage-collected with the `Library`.
+
+### Kajson integration
+
+The Temporal data converter passes the active `ClassRegistry` to `kajson.loads()`:
+
+```python
+pydantic_gizmo = kajson.loads(data, class_registry=get_class_registry())
+```
+
+Inside a workflow coroutine, `get_class_registry()` returns the per-workflow scoped registry (via `_library_id` ContextVar). However, the data converter is also called from the Temporal SDK's event processing context (e.g., when deserializing child workflow return values), where the ContextVar is not set. In that case, it falls back to the global registry. This is why PipeOutput uses deferred hydration — the `working_memory_raw` dict doesn't require class resolution during deserialization.
+
+---
+
+## Workflow Topology
+
+A single pipe execution can create many Temporal workflows. Each level of nesting dispatches child workflows:
+
+```
+PipeSequence (3 steps)
+ → 1 parent WfPipeRouter
+ → 3 child WfPipeRouters (one per step)
+ Each child step (if PipeLLM):
+ → 1 child WfMakeLLMText + 1 ActLLMGenText activity
+
+PipeParallel (2 branches, then summary)
+ → 1 parent WfPipeRouter
+ → 3 child WfPipeRouters (2 branches concurrent + 1 summary)
+
+PipeBatch (3 items)
+ → 1 parent WfPipeRouter
+ → 1 child WfPipeRouter for the PipeBatch controller
+ → 3 child WfPipeRouters (one per item)
+```
+
+Each `WfPipeRouter` loads the `LibraryCrate` (idempotent via fingerprint), so pipe resolution works at every level.
+
+---
+
+## Controller Dispatch Patterns on Temporal
+
+Each controller type creates a different child workflow pattern:
+
+| Controller | Dispatch pattern | Child workflows |
+|------------|-----------------|----------------|
+| **PipeSequence** | Sequential: each step via `SubPipe.run_pipe()` → `PipeRouterChild` | N children (one per step), executed sequentially |
+| **PipeParallel** | Concurrent: all branches via `SubPipe.run_pipe()` + `asyncio.gather()` | N children (one per branch), executed concurrently |
+| **PipeBatch** | Fan-out: each list item via `SubPipe.run_pipe()` | N children (one per item), executed concurrently |
+| **PipeCondition** | Conditional: evaluates expression, dispatches selected outcome via `SubPipe.run_pipe()` | 1 child (the chosen outcome pipe), plus internal `WfMakeJinja2Text` for expression evaluation |
+
+### Internal sub-workflows
+
+Some controllers dispatch internal sub-workflows in addition to pipe routing:
+
+- **PipeCondition**: `WfMakeJinja2Text` to evaluate the expression template against working memory
+- **PipeCompose**: `WfMakeJinja2Text` to resolve construct field references (e.g., `{ from = "title_text.text" }`)
+
+These internal workflows carry working memory contents through the Temporal data converter, which creates a serialization dependency on working memory content types.
+
+---
+
+## Known Limitation: StuffArtefact Serialization in Dry-Run
+
+In dry-run mode, PipeLLM produces `StuffArtefact` debug objects as working memory content. When controllers dispatch internal sub-workflows that carry working memory contents through the Temporal data converter, Kajson fails with `TypeError: Type <class 'StuffArtefact'> is not JSON serializable`.
+
+`StuffArtefact` is not a Pydantic model — it's a plain object used for dry-run tracing that was never designed for cross-process serialization.
+
+**Confirmed affected**:
+
+- **PipeCondition**: expression evaluation dispatches `WfMakeJinja2Text`, which serializes working memory to evaluate the Jinja2 expression. Fails with `StuffArtefact` TypeError.
+- **PipeCompose**: construct field resolution dispatches `WfMakeJinja2Text` similarly.
+
+**Likely affected** (hangs in testing, root cause not fully diagnosed):
+
+- **PipeBatch**: fan-out creates child PipeJobs with copies of working memory. The hang may be caused by `StuffArtefact` in the serialized child PipeJob, or by a different serialization issue in the list content extraction.
+
+**Not affected**:
+
+- **PipeSequence**: no internal sub-workflows — child pipes are dispatched via `SubPipe.run_pipe()` → `PipeRouterChild`, which creates clean child PipeJobs.
+- **PipeParallel**: same dispatch pattern as PipeSequence — branches go through `SubPipe.run_pipe()` without internal templating workflows.
+- **All controllers in live mode**: `StuffArtefact` is only produced in dry-run. Live execution uses real content objects that serialize correctly.
+
+**Fix direction**: Either make `StuffArtefact` serializable (implement `__json__` or convert to Pydantic), or strip non-serializable objects from working memory before passing to internal sub-workflows.
+
+---
+
+## File Reference
+
+| Component | File |
+|-----------|------|
+| LibraryCrate model | `pipelex/libraries/library_crate.py` |
+| LibraryCrate factory | `pipelex/libraries/library_crate_factory.py` |
+| PipeJob (crate + raw WM fields) | `pipelex/pipe_run/pipe_job.py` |
+| PipeOutput (raw WM for deferred hydration) | `pipelex/core/pipes/pipe_output.py` |
+| PipeRouterTop (submitter-side dispatch) | `pipelex/temporal/tprl_pipe/pipe_router_top.py` |
+| PipeRouterChild (worker-side child dispatch) | `pipelex/temporal/tprl_pipe/pipe_router_child.py` |
+| WfPipeRouter (workflow entry point) | `pipelex/temporal/tprl_pipe/wf_pipe_router.py` |
+| Deferred hydration (worker-side) | `pipelex/temporal/tprl_pipe/hydration.py` |
+| Submitter-side rehydration | `pipelex/temporal/tprl_pipe/submitter_hydration.py` |
+| Activity-side delivery rendering (typed/raw dual path) | `pipelex/pipe_run/delivery_executor.py` |
+| WorkingMemory dehydration helper | `pipelex/core/memory/working_memory.py` (`dump_for_temporal`) |
+| Kajson data converter | `pipelex/temporal/temporal_data_converter.py` |
+| Hub (ClassRegistry + Library scoping) | `pipelex/hub.py` |
+| Library (PrivateAttr ClassRegistry) | `pipelex/libraries/library.py` |
+| Worker CLI | `pipelex/temporal/worker_cli.py` |
+| Temporal task manager | `pipelex/temporal/temporal_task_manager.py` |
diff --git a/mkdocs.yml b/mkdocs.yml
index 71b405b54..1ce9f4528 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -304,6 +304,9 @@ plugins:
- under-the-hood/stuffartefact-and-image-rendering.md: "StuffArtefact & Image Rendering"
- under-the-hood/test-profile-configuration.md: "Test Profile Configuration"
- under-the-hood/init-cli-flows.md: "Init CLI Flows"
+ - under-the-hood/pipe-routing-and-execution.md: "Pipe Routing & Execution"
+ - under-the-hood/temporal-integration.md: "Temporal Integration"
+ - under-the-hood/distributed-content-generation.md: "Distributed Content Generation"
Project:
- contributing.md: "Contributing"
- contribute/configuration-defaults-and-overrides.md: "Configuration Internals"
@@ -489,6 +492,9 @@ nav:
- StuffArtefact & Image Rendering: under-the-hood/stuffartefact-and-image-rendering.md
- Test Profile Configuration: under-the-hood/test-profile-configuration.md
- Init CLI Flows: under-the-hood/init-cli-flows.md
+ - Pipe Routing & Execution: under-the-hood/pipe-routing-and-execution.md
+ - Temporal Integration: under-the-hood/temporal-integration.md
+ - Distributed Content Generation: under-the-hood/distributed-content-generation.md
- Project:
- Contributing: contributing.md
- Configuration Internals: contribute/configuration-defaults-and-overrides.md
diff --git a/package.json b/package.json
index babf8bb18..4a9cb6f7e 100644
--- a/package.json
+++ b/package.json
@@ -7,4 +7,4 @@
"//dependencies": {
"@pipelex/mthds-ui": "v0.3.4 = 838c0a3e57c3e19cf9d9b5fea806ca915cb820d3"
}
-}
\ No newline at end of file
+}
diff --git a/pipelex/base_exceptions.py b/pipelex/base_exceptions.py
index 61ab5212f..a48dd4839 100644
--- a/pipelex/base_exceptions.py
+++ b/pipelex/base_exceptions.py
@@ -1,8 +1,47 @@
+from typing import cast
+
+from pydantic import TypeAdapter
+from pydantic.dataclasses import dataclass
+
+
+@dataclass(frozen=True, config={"extra": "forbid"})
+class ErrorReport:
+ """Structured error report — single source of truth for all error serialization.
+
+ Used by CLI JSON output, agent output, and Temporal error details.
+ """
+
+ error_type: str
+ message: str
+ error_category: str | None = None
+ retryable: bool | None = None
+ user_action: str | None = None
+ model: str | None = None
+ provider: str | None = None
+
+ def to_dict(self) -> dict[str, str | bool]:
+ """Return a dict with only non-None fields."""
+ return cast(
+ "dict[str, str | bool]",
+ TypeAdapter(type(self)).dump_python(self, mode="python", exclude_none=True),
+ )
+
+
class PipelexError(Exception):
def __init__(self, message: str):
super().__init__(message)
self.message = message
+ def to_error_report(self) -> ErrorReport:
+ """Return a structured error report.
+
+ Subclasses override to include additional fields (error_category, model, etc.).
+ """
+ return ErrorReport(
+ error_type=type(self).__name__,
+ message=self.message,
+ )
+
class PipelexUnexpectedError(PipelexError):
pass
@@ -14,3 +53,11 @@ class PipelexConfigError(PipelexError):
class PipelexSetupError(PipelexError):
pass
+
+
+class SecurityError(PipelexError):
+ """Base for security-policy violations.
+
+ Kept distinct from domain errors so security signals are not silently
+ swallowed by domain-level `except` handlers (e.g. `except PipelexError`).
+ """
diff --git a/pipelex/builder/operations/inputs_ops.py b/pipelex/builder/operations/inputs_ops.py
index de7759260..cdb8cd41f 100644
--- a/pipelex/builder/operations/inputs_ops.py
+++ b/pipelex/builder/operations/inputs_ops.py
@@ -6,6 +6,7 @@ import json
from typing import TYPE_CHECKING, Any
from pipelex.core.pipes.inputs.input_renderer import render_inputs
+from pipelex.core.pipes.pipe_factory import PipeFactory
from pipelex.hub import (
get_library_manager,
get_required_pipe,
@@ -51,7 +52,7 @@ async def build_inputs_for_pipe(
main_pipe_code: str | None = None
for blueprint in blueprints:
if blueprint.main_pipe:
- main_pipe_code = f"{blueprint.domain}.{blueprint.main_pipe}"
+ main_pipe_code = PipeFactory.make_pipe_ref_with_domain(domain_code=blueprint.domain, pipe_code=blueprint.main_pipe)
break
if not main_pipe_code:
msg = "Bundle does not declare a main_pipe. Specify a pipe code."
@@ -65,7 +66,7 @@ async def build_inputs_for_pipe(
if not main_pipe_code:
msg = f"Bundle '{bundle_path}' does not declare a main_pipe. Specify a pipe code."
raise ValueError(msg)
- pipe_code = f"{bundle_blueprint.domain}.{main_pipe_code}"
+ pipe_code = PipeFactory.make_pipe_ref_with_domain(domain_code=bundle_blueprint.domain, pipe_code=main_pipe_code)
else:
# No bundle - initialize the library manually
library_manager = get_library_manager()
diff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.py
index e95169d2c..797b7fb1d 100644
--- a/pipelex/cli/_cli.py
+++ b/pipelex/cli/_cli.py
@@ -17,6 +17,7 @@ from pipelex.cli.commands.show_cmd import show_app
from pipelex.cli.commands.update_cmd import update_cmd
from pipelex.cli.commands.validate.app import validate_app
from pipelex.cli.commands.which_cmd import which_cmd
+from pipelex.cli.commands.worker_cmd import worker_cmd
from pipelex.cli.deck_notice import warn_if_deck_stale
from pipelex.cli.readiness import check_readiness
from pipelex.hub import get_console
@@ -29,7 +30,7 @@ class PipelexCLI(TyperGroup):
@override
def list_commands(self, ctx: Context) -> list[str]:
# List the commands in the proper order because natural ordering doesn't work between Typer groups and commands
- return ["login", "init", "doctor", "update", "build", "validate", "run", "graph", "show", "which"]
+ return ["login", "init", "doctor", "update", "build", "validate", "run", "graph", "show", "which", "worker"]
@override
def get_command(self, ctx: Context, cmd_name: str) -> Command | None:
@@ -205,3 +206,4 @@ app.add_typer(run_app, name="run", help="Run a method or pipe, optionally provid
app.add_typer(graph_app, name="graph", help="Generate and render execution graphs")
app.add_typer(show_app, name="show", help="Show configuration, pipes, and list AI models")
app.command(name="which", help="Locate where a pipe is defined, similar to 'which' for executables")(which_cmd)
+app.command(name="worker", help="Start a Temporal worker for distributed workflow execution")(worker_cmd)
diff --git a/pipelex/cli/agent_cli/commands/agent_output.py b/pipelex/cli/agent_cli/commands/agent_output.py
index 0696b1faf..848fc9824 100644
--- a/pipelex/cli/agent_cli/commands/agent_output.py
+++ b/pipelex/cli/agent_cli/commands/agent_output.py
@@ -6,6 +6,7 @@ from typing import Any, NoReturn
import typer
+from pipelex.base_exceptions import PipelexError
from pipelex.pipeline.validate_bundle import ValidateBundleError
from pipelex.tools.misc.json_utils import clean_json_dumps
from pipelex.types import StrEnum
@@ -157,16 +158,51 @@ def agent_error(message: str, error_type: str, cause: BaseException | None = Non
"error_type": error_type,
"message": message,
}
- hint = AGENT_ERROR_HINTS.get(error_type)
+
+ # Extract structured data from PipelexError.to_error_report() when available
+ report_hint: str | None = None
+ report_retryable: bool | None = None
+ report_category: str | None = None
+ report_extras: dict[str, Any] = {}
+
+ if isinstance(cause, PipelexError):
+ report = cause.to_error_report()
+ report_hint = report.user_action
+ report_retryable = report.retryable
+ report_category = report.error_category
+ if report.model:
+ report_extras["model"] = report.model
+ if report.provider:
+ report_extras["provider"] = report.provider
+
+ # hint: report-first, fallback to lookup dict
+ hint = report_hint or AGENT_ERROR_HINTS.get(error_type)
if hint:
error_json["hint"] = hint
- if error_type in RETRYABLE_ERROR_TYPES:
+
+ # retryable: report takes precedence, fallback to lookup set (only emit when True)
+ if report_retryable is not None:
+ if report_retryable:
+ error_json["retryable"] = True
+ elif error_type in RETRYABLE_ERROR_TYPES:
error_json["retryable"] = True
+
+ # error_domain: always from lookup dict (independent of error_category)
domain = AGENT_ERROR_DOMAINS.get(error_type)
if domain:
error_json["error_domain"] = domain
+
+ # error_category: from report when available
+ if report_category:
+ error_json["error_category"] = report_category
+
+ # model/provider from report
+ error_json.update(report_extras)
+
if cause is not None:
error_json["error_source"] = _build_error_source(cause)
+
+ # **extra overrides everything
error_json.update(extra)
print(clean_json_dumps(error_json, indent=2), file=sys.stderr)
raise typer.Exit(1) from cause
diff --git a/pipelex/cli/cli_factory.py b/pipelex/cli/cli_factory.py
index aa981112b..921f2ed79 100644
--- a/pipelex/cli/cli_factory.py
+++ b/pipelex/cli/cli_factory.py
@@ -31,6 +31,7 @@ def make_pipelex_for_cli(
context: ErrorContext,
library_dirs: list[str] | list[Path] | None = None,
needs_inference: bool = True,
+ temporal_enabled: bool | None = None,
needs_model_specs: bool | None = None,
) -> Pipelex:
"""Initialize Pipelex for CLI commands with proper error handling.
@@ -42,6 +43,7 @@ def make_pipelex_for_cli(
context: The CLI context for error messages.
library_dirs: The library directories to use for the Pipelex instance.
needs_inference: When False, skip inference setup (credentials, gateway, telemetry).
+ temporal_enabled: When provided, overrides the temporal.is_enabled config value.
needs_model_specs: When True, load real model specs even without inference.
Returns:
@@ -52,7 +54,11 @@ def make_pipelex_for_cli(
"""
try:
return Pipelex.make(
- integration_mode=IntegrationMode.CLI, library_dirs=library_dirs, needs_inference=needs_inference, needs_model_specs=needs_model_specs
+ integration_mode=IntegrationMode.CLI,
+ library_dirs=library_dirs,
+ needs_inference=needs_inference,
+ temporal_enabled=temporal_enabled,
+ needs_model_specs=needs_model_specs,
)
except InferenceSetupRequiredError as exc:
handle_inference_setup_required_error(exc)
diff --git a/pipelex/cli/commands/build/structures_cmd.py b/pipelex/cli/commands/build/structures_cmd.py
index 8c6a59287..2695135ec 100644
--- a/pipelex/cli/commands/build/structures_cmd.py
+++ b/pipelex/cli/commands/build/structures_cmd.py
@@ -3,7 +3,6 @@ from pathlib import Path
from typing import TYPE_CHECKING, Annotated
import typer
-from kajson.kajson_manager import KajsonManager
from pipelex import log
from pipelex.base_exceptions import PipelexError
@@ -191,7 +190,7 @@ def generate_structures_from_blueprints(
raise PipelexError(msg) from exc
# Register the generated class so it can be used as a base class for refined concepts
- KajsonManager.get_class_registry().register_class(the_generated_class)
+ get_class_registry().register_class(the_generated_class)
concept_snake_case = pascal_case_to_snake_case(concept_code)
output_file = output_directory / f"{blueprint.domain}__{concept_snake_case}.py"
@@ -240,7 +239,7 @@ def generate_structures_from_blueprints(
raise PipelexError(msg) from exc
# Register the generated class so it can be used as a base class for other refined concepts
- KajsonManager.get_class_registry().register_class(the_generated_class)
+ get_class_registry().register_class(the_generated_class)
concept_snake_case = pascal_case_to_snake_case(concept_code)
output_file = output_directory / f"{blueprint.domain}__{concept_snake_case}.py"
@@ -267,7 +266,7 @@ def generate_structures_from_blueprints(
raise PipelexError(msg) from exc
# Register the generated class so it can be used as a base class for refined concepts
- KajsonManager.get_class_registry().register_class(the_generated_class)
+ get_class_registry().register_class(the_generated_class)
concept_snake_case = pascal_case_to_snake_case(concept_code)
output_file = output_directory / f"{blueprint.domain}__{concept_snake_case}.py"
diff --git a/pipelex/cli/commands/run/_run_core.py b/pipelex/cli/commands/run/_run_core.py
index a0bed6c81..9dacebfc6 100644
--- a/pipelex/cli/commands/run/_run_core.py
+++ b/pipelex/cli/commands/run/_run_core.py
@@ -246,13 +246,14 @@ def execute_run(
mock_inputs: bool,
library_dir: list[str] | None,
telemetry_command_label: str = COMMAND,
+ temporal: bool | None = None,
dynamic_output_concept_ref: str | None = None,
) -> None:
"""Synchronous entry point that wraps the async execution with Pipelex setup/teardown.
Shared between the ``method`` and ``pipe`` subcommands.
"""
- make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN, library_dirs=library_dir)
+ make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN, library_dirs=library_dir, temporal_enabled=temporal)
try:
with get_telemetry_manager().telemetry_context():
diff --git a/pipelex/cli/commands/run/bundle_cmd.py b/pipelex/cli/commands/run/bundle_cmd.py
index 2b7155184..745668069 100644
--- a/pipelex/cli/commands/run/bundle_cmd.py
+++ b/pipelex/cli/commands/run/bundle_cmd.py
@@ -69,6 +69,10 @@ def run_bundle_cmd(
list[str] | None,
typer.Option("--library-dir", "-L", help="Directory to search for pipe definitions (.mthds files). Can be specified multiple times."),
] = None,
+ temporal: Annotated[
+ bool | None,
+ typer.Option("--temporal/--no-temporal", help="Override config: enable or disable Temporal workflow execution"),
+ ] = None,
dynamic_output_concept_ref: Annotated[
str | None,
typer.Option(
@@ -168,5 +172,6 @@ def run_bundle_cmd(
mock_inputs=mock_inputs,
library_dir=library_dir,
telemetry_command_label=f"{COMMAND} bundle",
+ temporal=temporal,
dynamic_output_concept_ref=dynamic_output_concept_ref,
)
diff --git a/pipelex/cli/commands/run/method_cmd.py b/pipelex/cli/commands/run/method_cmd.py
index 1954eb19b..5e87e3f34 100644
--- a/pipelex/cli/commands/run/method_cmd.py
+++ b/pipelex/cli/commands/run/method_cmd.py
@@ -68,6 +68,10 @@ def run_method_cmd(
list[str] | None,
typer.Option("--library-dir", "-L", help="Directory to search for pipe definitions (.mthds files). Can be specified multiple times."),
] = None,
+ temporal: Annotated[
+ bool | None,
+ typer.Option("--temporal/--no-temporal", help="Override config: enable or disable Temporal workflow execution"),
+ ] = None,
dynamic_output_concept_ref: Annotated[
str | None,
typer.Option(
@@ -138,5 +142,6 @@ def run_method_cmd(
mock_inputs=mock_inputs,
library_dir=effective_library_dir,
telemetry_command_label=f"{COMMAND} method",
+ temporal=temporal,
dynamic_output_concept_ref=dynamic_output_concept_ref,
)
diff --git a/pipelex/cli/commands/run/pipe_cmd.py b/pipelex/cli/commands/run/pipe_cmd.py
index 90ba5cfc1..c963a7372 100644
--- a/pipelex/cli/commands/run/pipe_cmd.py
+++ b/pipelex/cli/commands/run/pipe_cmd.py
@@ -65,6 +65,10 @@ def run_pipe_cmd(
list[str] | None,
typer.Option("--library-dir", "-L", help="Directory to search for pipe definitions (.mthds files). Can be specified multiple times."),
] = None,
+ temporal: Annotated[
+ bool | None,
+ typer.Option("--temporal/--no-temporal", help="Override config: enable or disable Temporal workflow execution"),
+ ] = None,
dynamic_output_concept_ref: Annotated[
str | None,
typer.Option(
@@ -133,5 +137,6 @@ def run_pipe_cmd(
mock_inputs=mock_inputs,
library_dir=library_dir,
telemetry_command_label=f"{COMMAND} pipe",
+ temporal=temporal,
dynamic_output_concept_ref=dynamic_output_concept_ref,
)
diff --git a/pipelex/cli/commands/show_cmd.py b/pipelex/cli/commands/show_cmd.py
index 2a07542c8..ce1510740 100644
--- a/pipelex/cli/commands/show_cmd.py
+++ b/pipelex/cli/commands/show_cmd.py
@@ -272,7 +272,7 @@ def show_models_cmd(
pipelex show models openai
pipelex show models anthropic --flat
"""
- make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_SHOW_MODELS)
+ make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_SHOW_MODELS, needs_inference=True)
try:
with get_telemetry_manager().telemetry_context():
diff --git a/pipelex/cli/commands/worker_cmd.py b/pipelex/cli/commands/worker_cmd.py
new file mode 100644
index 000000000..71cd637f6
--- /dev/null
+++ b/pipelex/cli/commands/worker_cmd.py
@@ -0,0 +1,64 @@
+"""pipelex worker — Start a Temporal worker."""
+
+import asyncio
+from typing import Annotated
+
+import typer
+
+from pipelex import log
+from pipelex.cli.cli_factory import make_pipelex_for_cli
+from pipelex.cli.error_handlers import ErrorContext
+from pipelex.pipelex import Pipelex
+
+
+def worker_cmd(
+ is_not_sandboxed: Annotated[
+ bool,
+ typer.Option("--no-sandbox", help="Run workflows without Temporal sandbox restrictions"),
+ ] = False,
+ is_unit_testing: Annotated[
+ bool,
+ typer.Option("--unit-testing", help="Run in unit testing mode"),
+ ] = False,
+ task_queue: Annotated[
+ str | None,
+ typer.Option("--task-queue", help="Override the task queue name from config"),
+ ] = None,
+ scope: Annotated[
+ str | None,
+ typer.Option("--scope", help="Worker scope name from [temporal.worker_scopes.scopes] (defaults to default_scope)"),
+ ] = None,
+) -> None:
+ """Start a Temporal worker.
+
+ This command initializes Pipelex with Temporal enabled and starts
+ a worker that listens for workflow and activity tasks.
+
+ Examples:
+ pipelex worker
+ pipelex worker --no-sandbox
+ pipelex worker --task-queue my_queue
+ pipelex worker --scope router
+ """
+ make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN, temporal_enabled=True)
+
+ from pipelex.temporal.temporal_hub import get_task_manager # noqa: PLC0415
+
+ try:
+ log.info("Starting Temporal worker...")
+ asyncio.run(
+ get_task_manager().run_worker(
+ is_not_sandboxed=is_not_sandboxed,
+ is_unit_testing=is_unit_testing,
+ task_queue=task_queue,
+ scope_name=scope,
+ )
+ )
+ except KeyboardInterrupt:
+ log.info("Worker stopped by user")
+ except Exception as exc:
+ log.error(f"Worker failed: {exc}")
+ typer.secho(f"Worker failed: {exc}", fg=typer.colors.RED, err=True)
+ raise typer.Exit(1) from exc
+ finally:
+ Pipelex.teardown_if_needed()
diff --git a/pipelex/cli/dev_cli/commands/check_rules_sync_cmd.py b/pipelex/cli/dev_cli/commands/check_rules_sync_cmd.py
index 178e42762..8c9c29a92 100644
--- a/pipelex/cli/dev_cli/commands/check_rules_sync_cmd.py
+++ b/pipelex/cli/dev_cli/commands/check_rules_sync_cmd.py
@@ -4,6 +4,7 @@ from __future__ import annotations
import sys
from pathlib import Path
+from typing import TYPE_CHECKING
from rich.markup import escape
from rich.panel import Panel
@@ -14,20 +15,26 @@ from pipelex.kit.single_file_agent_rules import build_merged_rules, unified_diff
from pipelex.system.configuration.configs import AgentTarget
from pipelex.tools.misc.toml_utils import load_toml_from_path
+if TYPE_CHECKING:
+ from pipelex.kit.index_models import Target
-def _get_preferred_target_from_toml() -> AgentTarget:
- """Read preferred_agent_target directly from pipelex.toml without initializing Pipelex."""
+_DEFAULT_TARGETS: list[AgentTarget] = [AgentTarget.CLAUDE, AgentTarget.AGENTS]
+
+
+def _get_preferred_targets_from_toml() -> list[AgentTarget]:
+ """Read preferred_agent_targets directly from pipelex.toml without initializing Pipelex."""
pipelex_toml = Path("pipelex/pipelex.toml")
if not pipelex_toml.exists():
- return AgentTarget.CLAUDE # Default fallback
+ return _DEFAULT_TARGETS
config = load_toml_from_path(str(pipelex_toml))
try:
- target_str = config["pipelex"]["kit_config"]["preferred_agent_target"]
- return AgentTarget(target_str)
- except (KeyError, ValueError):
- return AgentTarget.CLAUDE # Default fallback
+ raw_targets = config["pipelex"]["kit_config"]["preferred_agent_targets"]
+ parsed_targets = [AgentTarget(item) for item in raw_targets]
+ except (KeyError, ValueError, TypeError):
+ return _DEFAULT_TARGETS
+ return parsed_targets or _DEFAULT_TARGETS
def check_rules_sync_cmd(show_diff: bool = True, quiet: bool = False) -> None:
@@ -41,24 +48,26 @@ def check_rules_sync_cmd(show_diff: bool = True, quiet: bool = False) -> None:
kit_index = load_index()
agent_set = "all"
- # Get preferred agent target from config (without initializing Pipelex)
- preferred_target = _get_preferred_target_from_toml()
-
- # Only check single-file targets (not Cursor)
- match preferred_target:
- case AgentTarget.CURSOR:
- # Cursor rules use a different mechanism, skip for now
- if quiet:
- console.print("[green]✓ Agent rules sync check: PASSED[/green] (Cursor target - skipped)")
- else:
- console.print("[dim]Cursor target selected - sync check not applicable[/dim]")
- return
- case AgentTarget.AGENTS | AgentTarget.CLAUDE:
- target_key = preferred_target
- if target_key not in kit_index.agent_rules.targets:
- console.print(f"[red]Target '{escape(preferred_target)}' not found in index.toml[/red]")
- sys.exit(1)
- targets_to_check = {target_key: kit_index.agent_rules.targets[target_key]}
+ # Get preferred agent targets from config (without initializing Pipelex)
+ preferred_targets = _get_preferred_targets_from_toml()
+
+ # Cursor is exclusive of the single-file targets per config validator.
+ if preferred_targets == [AgentTarget.CURSOR]:
+ if quiet:
+ console.print("[green]✓ Agent rules sync check: PASSED[/green] (Cursor target - skipped)")
+ else:
+ console.print("[dim]Cursor target selected - sync check not applicable[/dim]")
+ return
+ if AgentTarget.CURSOR in preferred_targets:
+ console.print("[red]Invalid config: preferred_agent_targets cannot mix 'cursor' with other targets[/red]")
+ sys.exit(1)
+
+ targets_to_check: dict[str, Target] = {}
+ for target_key in preferred_targets:
+ if target_key not in kit_index.agent_rules.targets:
+ console.print(f"[red]Target '{escape(target_key)}' not found in index.toml[/red]")
+ sys.exit(1)
+ targets_to_check[target_key] = kit_index.agent_rules.targets[target_key]
missing_targets: list[Path] = []
mismatches: list[tuple[Path, str, str]] = []
diff --git a/pipelex/cli/dev_cli/commands/check_urls_cmd.py b/pipelex/cli/dev_cli/commands/check_urls_cmd.py
index 4c3ec8262..65cb5c360 100644
--- a/pipelex/cli/dev_cli/commands/check_urls_cmd.py
+++ b/pipelex/cli/dev_cli/commands/check_urls_cmd.py
@@ -23,6 +23,9 @@ DEFAULT_TIMEOUT = 10
# - max_keepalive_connections: Connections to keep alive for reuse
HTTP_LIMITS = httpx.Limits(max_connections=20, max_keepalive_connections=10)
+# Status codes treated as OK: the URL exists but requires authentication (e.g. billing dashboards).
+AUTH_REQUIRED_STATUS_CODES = {401, 403}
+
class URLCheckResult(BaseModel):
"""Result of checking a single URL."""
@@ -71,7 +74,7 @@ async def check_single_url_async(
# Some servers don't support HEAD, fallback to GET
if response.status_code == 405:
response = await client.get(url)
- is_ok = response.status_code < 400
+ is_ok = response.status_code < 400 or response.status_code in AUTH_REQUIRED_STATUS_CODES
return URLCheckResult(
name=name,
url=url,
@@ -167,10 +170,12 @@ def check_urls_cmd(quiet: bool = False, timeout: int = DEFAULT_TIMEOUT) -> None:
# Display individual results if not quiet
if not quiet:
for result in results:
- if result.is_ok:
- console.print(f" [dim]{escape(result.name)}[/dim] [green]✓[/green] {result.status_code}")
- else:
+ if not result.is_ok:
console.print(f" [dim]{escape(result.name)}[/dim] [red]✗[/red] {escape(result.error_message or '')}")
+ elif result.status_code in AUTH_REQUIRED_STATUS_CODES:
+ console.print(f" [dim]{escape(result.name)}[/dim] [yellow]✓[/yellow] {result.status_code} (auth required)")
+ else:
+ console.print(f" [dim]{escape(result.name)}[/dim] [green]✓[/green] {result.status_code}")
# Count successes and failures
ok_count = sum(1 for result in results if result.is_ok)
diff --git a/pipelex/cli/dev_cli/commands/kit_cmd.py b/pipelex/cli/dev_cli/commands/kit_cmd.py
index 75d72e986..564ebd5f8 100644
--- a/pipelex/cli/dev_cli/commands/kit_cmd.py
+++ b/pipelex/cli/dev_cli/commands/kit_cmd.py
@@ -37,36 +37,37 @@ def _sync_agent_rules(
loaded_kit_index = load_index() if kit_index is None else kit_index
agent_set = agent_set or loaded_kit_index.agent_rules.default_set
- # Get preferred agent target from config
config = get_config()
- preferred_target = config.pipelex.kit_config.preferred_agent_target
-
- match preferred_target:
- case AgentTarget.CURSOR:
- typer.echo("Updating Cursor rules...")
- update_cursor_rules(resolved_repo_root, loaded_kit_index, agent_set=agent_set)
- case AgentTarget.AGENTS | AgentTarget.CLAUDE:
- typer.echo(f"Updating {preferred_target} rules...")
- all_targets = loaded_kit_index.agent_rules.targets
- if preferred_target in all_targets:
- filtered_targets: dict[str, Target] = {preferred_target: all_targets[preferred_target]}
- update_single_file_agent_rules(
- repo_root=resolved_repo_root,
- kit_index=loaded_kit_index,
- agent_set=agent_set,
- targets=filtered_targets,
- )
- else:
- msg = f"Target '{preferred_target}' not found in index.toml"
+ preferred_targets: list[AgentTarget] = list(config.pipelex.kit_config.preferred_agent_targets)
+
+ # The config validator guarantees CURSOR cannot coexist with other targets,
+ # so a simple membership check is enough to pick the branch.
+ if AgentTarget.CURSOR in preferred_targets:
+ typer.echo("Updating Cursor rules...")
+ update_cursor_rules(resolved_repo_root, loaded_kit_index, agent_set=agent_set)
+ else:
+ all_targets = loaded_kit_index.agent_rules.targets
+ filtered_targets: dict[str, Target] = {}
+ for target_key in preferred_targets:
+ if target_key not in all_targets:
+ msg = f"Target '{target_key}' not found in index.toml"
raise PipelexCLIError(msg)
+ filtered_targets[target_key] = all_targets[target_key]
+ names = ", ".join(sorted(filtered_targets.keys()))
+ typer.echo(f"Updating {names} rules...")
+ update_single_file_agent_rules(
+ repo_root=resolved_repo_root,
+ kit_index=loaded_kit_index,
+ agent_set=agent_set,
+ targets=filtered_targets,
+ )
- # Cleanup: remove rules from other targets
if cleanup:
typer.echo("Cleaning up rules from other targets...")
_cleanup_other_targets(
repo_root=resolved_repo_root,
kit_index=loaded_kit_index,
- preferred_target=preferred_target,
+ preferred_targets=preferred_targets,
)
typer.echo("Kit sync completed successfully")
@@ -75,30 +76,18 @@ def _sync_agent_rules(
def _cleanup_other_targets(
repo_root: Path,
kit_index: KitIndex,
- preferred_target: AgentTarget,
+ preferred_targets: list[AgentTarget],
) -> None:
- """Remove Pipelex rules from all targets except the preferred one.
+ """Remove Pipelex rules from all targets except the preferred ones.
For Cursor: deletes .mdc files
For single-file targets: deletes the files entirely
"""
- # If preferred target is NOT cursor, remove cursor rules
- match preferred_target:
- case AgentTarget.CURSOR:
- pass # Don't remove cursor rules since it's the preferred target
- case AgentTarget.AGENTS | AgentTarget.CLAUDE:
- # Remove cursor rules
- remove_cursor_rules(repo_root)
-
- # For single-file targets, delete files for all except the preferred one
- all_targets = kit_index.agent_rules.targets
- targets_to_clean: dict[str, Target] = {}
+ if AgentTarget.CURSOR not in preferred_targets:
+ remove_cursor_rules(repo_root)
- for target_key, target in all_targets.items():
- # Skip the preferred target
- if target_key == preferred_target:
- continue
- targets_to_clean[target_key] = target
+ all_targets = kit_index.agent_rules.targets
+ targets_to_clean: dict[str, Target] = {target_key: target for target_key, target in all_targets.items() if target_key not in preferred_targets}
if targets_to_clean:
remove_from_targets(
diff --git a/pipelex/cli/error_handlers.py b/pipelex/cli/error_handlers.py
index 4f4513fb8..f74f0c279 100644
--- a/pipelex/cli/error_handlers.py
+++ b/pipelex/cli/error_handlers.py
@@ -49,16 +49,15 @@ def handle_model_choice_error(exc: PipeOperatorModelChoiceError, context: ErrorC
exc: The model choice error exception
context: Context for the error message
"""
+ report = exc.to_error_report()
console = get_console()
console.print(f"\n[bold red]❌ {context} failed because of a model choice could not be interpreted correctly[/bold red]\n")
console.print(f"[bold cyan]Pipe:[/bold cyan] [yellow]'{escape(exc.pipe_code)}'[/yellow] [dim]({escape(exc.pipe_type)})[/dim]")
console.print(f"[bold cyan]Model Type:[/bold cyan] [yellow]'{escape(exc.model_type)}'[/yellow]")
console.print(f"[bold cyan]Model Choice:[/bold cyan] [yellow]'{escape(str(exc.model_choice))}'[/yellow]")
console.print(f"\n[bold red]Error:[/bold red] {escape(exc.message)}\n")
- console.print(
- f"[bold green]💡 Tip:[/bold green] Check your model configuration in [cyan].pipelex/inference/[/cyan] "
- f"or specify a different model in the [yellow]'{escape(exc.pipe_code)}'[/yellow] pipe."
- )
+ tip = report.user_action or (f"Check your model configuration in .pipelex/inference/ or specify a different model in the '{exc.pipe_code}' pipe.")
+ console.print(f"\n[bold green]💡 Tip:[/bold green] {escape(tip)}")
console.print(f"[dim]Learn more about the inference backend system: {URLs.backend_provider_docs}[/dim]")
console.print(f"[dim]Join our Discord for help: {URLs.discord}[/dim]\n")
raise typer.Exit(1) from exc
@@ -71,6 +70,7 @@ def handle_model_availability_error(exc: PipeOperatorModelAvailabilityError, con
exc: The model availability error exception
context: Context for the error message
"""
+ report = exc.to_error_report()
console = get_console()
console.print(f"\n[bold red]❌ {context} failed because a model wasn't available[/bold red]\n")
console.print(f"[bold cyan]Pipe:[/bold cyan] [yellow]'{escape(exc.pipe_code)}'[/yellow] [dim]({escape(exc.pipe_type)})[/dim]")
@@ -82,10 +82,8 @@ def handle_model_availability_error(exc: PipeOperatorModelAvailabilityError, con
stack_str = " [dim]→[/dim] ".join([f"[yellow]{escape(p)}[/yellow]" for p in exc.pipe_stack])
console.print(f"[bold cyan]Pipe Stack:[/bold cyan] {stack_str}")
console.print(f"\n[bold red]Error:[/bold red] {escape(str(exc))}\n")
- console.print(
- f"[bold green]💡 Tip:[/bold green] Check your model configuration in [cyan].pipelex/inference/[/cyan] "
- f"or specify a different model in the [yellow]'{escape(exc.pipe_code)}'[/yellow] pipe."
- )
+ tip = report.user_action or (f"Check your model configuration in .pipelex/inference/ or specify a different model in the '{exc.pipe_code}' pipe.")
+ console.print(f"[bold green]💡 Tip:[/bold green] {escape(tip)}")
console.print(f"[dim]Learn more about the inference backend system: {URLs.backend_provider_docs}[/dim]")
console.print(f"[dim]Join our Discord for help: {URLs.discord}[/dim]\n")
raise typer.Exit(1) from exc
@@ -98,6 +96,7 @@ def handle_model_deck_preset_error(exc: ModelDeckPresetValidatonError, context:
exc: The model deck preset validation error exception
context: Context for the error message
"""
+ report = exc.to_error_report()
console = get_console()
console.print(f"\n[bold red]❌ {context} failed due to model deck preset validation error[/bold red]\n")
console.print(f"[bold cyan]Preset ID:[/bold cyan] [yellow]'{escape(exc.preset_id)}'[/yellow]")
@@ -107,19 +106,22 @@ def handle_model_deck_preset_error(exc: ModelDeckPresetValidatonError, context:
backends_str = ", ".join([f"[yellow]{escape(b)}[/yellow]" for b in sorted(exc.enabled_backends)])
console.print(f"[bold cyan]Enabled Backends:[/bold cyan] {backends_str}")
console.print(f"\n[bold red]Error:[/bold red] {escape(exc.message)}\n")
- console.print(
- f"[bold green]💡 Tip:[/bold green] The preset [yellow]'{escape(exc.preset_id)}'[/yellow] references model handle "
- f"[yellow]'{escape(exc.model_handle)}'[/yellow] which is not available in any enabled backend."
- )
- if exc.enabled_backends:
- backends_str = ", ".join([f"[yellow]{escape(b)}[/yellow]" for b in sorted(exc.enabled_backends)])
- console.print(f"The enabled backends are: {backends_str}.")
- console.print(
- "[bold]Possible solutions:[/bold]\n"
- " 1. Update the preset to use a different model\n"
- f" 2. Configure model '{escape(exc.model_handle)}' in one of your enabled backends\n"
- f" 3. Enable a backend that supports [yellow]'{escape(exc.model_handle)}'[/yellow]"
- )
+ if report.user_action:
+ console.print(f"[bold green]💡 Tip:[/bold green] {escape(report.user_action)}")
+ else:
+ console.print(
+ f"[bold green]💡 Tip:[/bold green] The preset [yellow]'{escape(exc.preset_id)}'[/yellow] references model handle "
+ f"[yellow]'{escape(exc.model_handle)}'[/yellow] which is not available in any enabled backend."
+ )
+ if exc.enabled_backends:
+ backends_str = ", ".join([f"[yellow]{escape(b)}[/yellow]" for b in sorted(exc.enabled_backends)])
+ console.print(f"The enabled backends are: {backends_str}.")
+ console.print(
+ "[bold]Possible solutions:[/bold]\n"
+ " 1. Update the preset to use a different model\n"
+ f" 2. Configure model '{escape(exc.model_handle)}' in one of your enabled backends\n"
+ f" 3. Enable a backend that supports [yellow]'{escape(exc.model_handle)}'[/yellow]"
+ )
console.print(f"\n[dim]Learn more about the inference backend system: {URLs.backend_provider_docs}[/dim]")
console.print(f"[dim]Join our Discord for help: {URLs.discord}[/dim]\n")
raise typer.Exit(1) from exc
@@ -204,6 +206,7 @@ def handle_validate_bundle_error(exc: ValidateBundleError, bundle_path: Path | N
exc: The bundle validation error exception
bundle_path: Optional path to the bundle file being validated
"""
+ report = exc.to_error_report()
console = get_console()
console.print("\n[bold red]❌ Bundle validation failed[/bold red]\n")
@@ -213,10 +216,10 @@ def handle_validate_bundle_error(exc: ValidateBundleError, bundle_path: Path | N
_display_validation_error_details(console=console, exc=exc)
# Display helpful tips
- console.print(
- "[bold green]💡 Tip:[/bold green] Review the error messages above and check your pipeline configuration. "
- "Make sure all required fields are present and correctly formatted."
+ tip = report.user_action or (
+ "Review the error messages above and check your pipeline configuration. Make sure all required fields are present and correctly formatted."
)
+ console.print(f"[bold green]💡 Tip:[/bold green] {escape(tip)}")
console.print(f"[dim]Learn more: {URLs.documentation}[/dim]")
console.print(f"[dim]Join our Discord for help: {URLs.discord}[/dim]\n")
raise typer.Exit(1) from exc
diff --git a/pipelex/cogt/content_generation/assignment_models.py b/pipelex/cogt/content_generation/assignment_models.py
index 7f6471c52..e59887a12 100644
--- a/pipelex/cogt/content_generation/assignment_models.py
+++ b/pipelex/cogt/content_generation/assignment_models.py
@@ -3,7 +3,6 @@ from typing import Any
from pydantic import BaseModel
from typing_extensions import override
-from pipelex.cogt.exceptions import LLMAssignmentError
from pipelex.cogt.extract.extract_input import ExtractInput
from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
from pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParams
@@ -14,7 +13,6 @@ from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstrac
from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.cogt.templating.templating_style import TemplatingStyle
-from pipelex.hub import get_class_registry
from pipelex.pipeline.job_metadata import JobMetadata
@@ -85,34 +83,24 @@ class LLMAssignment(BaseModel):
class ObjectAssignment(BaseModel):
object_class_name: str
+ object_class_schema: dict[str, Any]
llm_assignment_for_object: LLMAssignment
- def __init__(self, **kwargs: Any):
- super().__init__(**kwargs)
- if not get_class_registry().has_class(name=self.object_class_name):
- error_msg = f"Could not create ObjectAssignment for class '{self.object_class_name}' because it is not in the class registry."
- raise LLMAssignmentError(error_msg)
-
@staticmethod
def make_for_class(
object_class: type[BaseModel],
llm_assignment: LLMAssignment,
) -> "ObjectAssignment":
- object_class_name = object_class.__name__
- get_class_registry().register_class(
- class_type=object_class,
- name=object_class_name,
- should_warn_if_already_registered=False,
- )
-
return ObjectAssignment(
- object_class_name=object_class_name,
+ object_class_name=object_class.__name__,
+ object_class_schema=object_class.model_json_schema(),
llm_assignment_for_object=llm_assignment,
)
class TextThenObjectAssignment(BaseModel):
object_class_name: str
+ object_class_schema: dict[str, Any]
llm_assignment_for_text: LLMAssignment
llm_assignment_factory_to_object: LLMAssignmentFactory
@@ -127,6 +115,7 @@ class ImgGenAssignment(BaseModel):
class TemplatingAssignment(BaseModel):
+ job_metadata: JobMetadata
context: dict[str, Any]
template: str
templating_style: TemplatingStyle | None = None
@@ -139,3 +128,9 @@ class ExtractAssignment(BaseModel):
extract_input: ExtractInput
extract_job_params: ExtractJobParams
extract_job_config: ExtractJobConfig
+
+
+class RenderPageViewsAssignment(BaseModel):
+ job_metadata: JobMetadata
+ document_uri: str
+ page_views_dpi: int
diff --git a/pipelex/cogt/content_generation/content_generator.py b/pipelex/cogt/content_generation/content_generator.py
index eb8721880..561370c04 100644
--- a/pipelex/cogt/content_generation/content_generator.py
+++ b/pipelex/cogt/content_generation/content_generator.py
@@ -1,4 +1,4 @@
-from typing import Any, cast
+from typing import Any
from typing_extensions import override
@@ -35,7 +35,6 @@ from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.page_content import PageContent
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.misc.image_utils import ImageFormat
-from pipelex.tools.pdf.pypdfium2_renderer import pypdfium2_renderer
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
@@ -50,6 +49,7 @@ class ContentGenerator(ContentGeneratorProtocol):
job_metadata: JobMetadata,
llm_setting_main: LLMSetting,
llm_prompt_for_text: LLMPrompt,
+ wfid: str | None = None,
) -> str:
log.verbose(f"{self.__class__.__name__} make_llm_text: {llm_prompt_for_text}")
log.verbose(f"llm_setting_main: {llm_setting_main}")
@@ -71,6 +71,7 @@ class ContentGenerator(ContentGeneratorProtocol):
object_class: type[BaseModelTypeVar],
llm_setting_for_object: LLMSetting,
llm_prompt_for_object: LLMPrompt,
+ wfid: str | None = None,
) -> BaseModelTypeVar:
log.verbose(f"{self.__class__.__name__} make_object_direct: {llm_prompt_for_object}")
llm_assignment_for_object = LLMAssignment.make_from_prompt(
@@ -82,9 +83,12 @@ class ContentGenerator(ContentGeneratorProtocol):
object_class=object_class,
llm_assignment=llm_assignment_for_object,
)
- obj = await llm_gen_object(object_assignment=object_assignment)
- log.verbose(f"{self.__class__.__name__} generated object direct: {obj}")
- return cast("BaseModelTypeVar", obj)
+ raw_obj = await llm_gen_object(object_assignment=object_assignment)
+ log.verbose(f"{self.__class__.__name__} generated object direct: {raw_obj}")
+ # llm_gen_object returns a plain BaseModel reconstructed from the JSON schema.
+ # Validate its data against the original class so the result is a proper subtype
+ # (e.g. StructuredContent) expected by the caller.
+ return object_class.model_validate(raw_obj.model_dump(serialize_as_any=True))
@override
@update_job_metadata
@@ -96,6 +100,7 @@ class ContentGenerator(ContentGeneratorProtocol):
llm_setting_for_object: LLMSetting,
llm_prompt_for_text: LLMPrompt,
llm_prompt_factory_for_object: LLMPromptFactoryAbstract | None = None,
+ wfid: str | None = None,
) -> BaseModelTypeVar:
log.verbose(llm_prompt_for_text.user_text, title="llm_prompt_for_text")
llm_assignment_for_text = LLMAssignment.make_from_prompt(
@@ -110,8 +115,10 @@ class ContentGenerator(ContentGeneratorProtocol):
llm_prompt_factory=llm_prompt_factory_for_object or LLMPromptTemplate.make_for_structuring_from_preliminary_text(),
)
+ object_class_schema = object_class.model_json_schema()
workflow_arg = TextThenObjectAssignment(
object_class_name=object_class.__name__,
+ object_class_schema=object_class_schema,
llm_assignment_for_text=llm_assignment_for_text,
llm_assignment_factory_to_object=llm_assignment_factory_to_object,
)
@@ -127,11 +134,12 @@ class ContentGenerator(ContentGeneratorProtocol):
fup_obj_assignment = ObjectAssignment(
llm_assignment_for_object=fup_llm_assignment,
object_class_name=object_class.__name__,
+ object_class_schema=object_class_schema,
)
- obj = await llm_gen_object(object_assignment=fup_obj_assignment)
- log.verbose(f"{self.__class__.__name__} generated object after text: {obj}")
- return cast("BaseModelTypeVar", obj)
+ raw_obj = await llm_gen_object(object_assignment=fup_obj_assignment)
+ log.verbose(f"{self.__class__.__name__} generated object after text: {raw_obj}")
+ return object_class.model_validate(raw_obj.model_dump(serialize_as_any=True))
@override
@update_job_metadata
@@ -142,6 +150,7 @@ class ContentGenerator(ContentGeneratorProtocol):
llm_setting_for_object_list: LLMSetting,
llm_prompt_for_object_list: LLMPrompt,
nb_items: int | None = None,
+ wfid: str | None = None,
) -> list[BaseModelTypeVar]:
llm_assignment_for_object = LLMAssignment.make_from_prompt(
job_metadata=job_metadata,
@@ -152,9 +161,9 @@ class ContentGenerator(ContentGeneratorProtocol):
object_class=object_class,
llm_assignment=llm_assignment_for_object,
)
- obj_list = await llm_gen_object_list(object_assignment=object_assignment)
- log.verbose(f"{self.__class__.__name__} generated object list direct: {obj_list}")
- return cast("list[BaseModelTypeVar]", obj_list)
+ raw_list = await llm_gen_object_list(object_assignment=object_assignment)
+ log.verbose(f"{self.__class__.__name__} generated object list direct: {raw_list}")
+ return [object_class.model_validate(raw_obj.model_dump(serialize_as_any=True)) for raw_obj in raw_list]
@override
@update_job_metadata
@@ -167,6 +176,7 @@ class ContentGenerator(ContentGeneratorProtocol):
llm_prompt_for_text: LLMPrompt,
llm_prompt_factory_for_object_list: LLMPromptFactoryAbstract | None = None,
nb_items: int | None = None,
+ wfid: str | None = None,
) -> list[BaseModelTypeVar]:
llm_assignment_for_text = LLMAssignment.make_from_prompt(
job_metadata=job_metadata,
@@ -179,8 +189,10 @@ class ContentGenerator(ContentGeneratorProtocol):
llm_setting=llm_setting_for_object_list,
llm_prompt_factory=llm_prompt_factory_for_object_list or LLMPromptTemplate.make_for_structuring_from_preliminary_text(),
)
+ object_class_schema = object_class.model_json_schema()
workflow_arg = TextThenObjectAssignment(
object_class_name=object_class.__name__,
+ object_class_schema=object_class_schema,
llm_assignment_for_text=llm_assignment_for_text,
llm_assignment_factory_to_object=llm_assignment_factory_to_object,
)
@@ -196,11 +208,12 @@ class ContentGenerator(ContentGeneratorProtocol):
fup_obj_assignment = ObjectAssignment(
llm_assignment_for_object=fup_llm_assignment,
object_class_name=object_class.__name__,
+ object_class_schema=object_class_schema,
)
- obj_list = await llm_gen_object_list(object_assignment=fup_obj_assignment)
- log.verbose(f"{self.__class__.__name__} generated object list after text: {obj_list}")
- return cast("list[BaseModelTypeVar]", obj_list)
+ raw_list = await llm_gen_object_list(object_assignment=fup_obj_assignment)
+ log.verbose(f"{self.__class__.__name__} generated object list after text: {raw_list}")
+ return [object_class.model_validate(raw_obj.model_dump(serialize_as_any=True)) for raw_obj in raw_list]
@override
async def make_image_content(
@@ -240,6 +253,7 @@ class ContentGenerator(ContentGeneratorProtocol):
img_gen_prompt: ImgGenPrompt,
img_gen_job_params: ImgGenJobParams | None = None,
img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
) -> ImageContent:
img_gen_config = get_config().cogt.img_gen_config
img_gen_job_params = img_gen_job_params or img_gen_config.make_default_img_gen_job_params()
@@ -270,6 +284,7 @@ class ContentGenerator(ContentGeneratorProtocol):
nb_images: int,
img_gen_job_params: ImgGenJobParams | None = None,
img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
) -> list[ImageContent]:
img_gen_config = get_config().cogt.img_gen_config
img_gen_assignment = ImgGenAssignment(
@@ -294,12 +309,15 @@ class ContentGenerator(ContentGeneratorProtocol):
@override
async def make_templated_text(
self,
+ job_metadata: JobMetadata,
context: dict[str, Any],
template: str,
templating_style: TemplatingStyle | None = None,
template_category: TemplateCategory | None = None,
+ wfid: str | None = None,
) -> str:
templating_assignment = TemplatingAssignment(
+ job_metadata=job_metadata,
context=context,
template=template,
templating_style=templating_style,
@@ -308,6 +326,7 @@ class ContentGenerator(ContentGeneratorProtocol):
return await templating_gen_text(templating_assignment=templating_assignment)
@override
+ @update_job_metadata
async def make_render_page_views(
self,
job_metadata: JobMetadata,
@@ -315,12 +334,16 @@ class ContentGenerator(ContentGeneratorProtocol):
extract_handle: str,
extract_job_params: ExtractJobParams | None = None,
extract_job_config: ExtractJobConfig | None = None,
+ wfid: str | None = None,
) -> list[ImageContent]:
if not extract_input.document_uri:
msg = "PDF URI is required to render page views"
raise ValueError(msg)
job_params = extract_job_params or ExtractJobParams.make_default_extract_job_params()
page_views_dpi = job_params.page_views_dpi or get_config().cogt.extract_config.default_page_views_dpi
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from pipelex.tools.pdf.pypdfium2_renderer import pypdfium2_renderer # noqa: PLC0415
+
page_view_images = await pypdfium2_renderer.render_pdf_pages_from_uri(pdf_uri=extract_input.document_uri, dpi=page_views_dpi)
page_view_images_resolved: list[ImageContent] = []
for page_view_image in page_view_images:
@@ -337,6 +360,7 @@ class ContentGenerator(ContentGeneratorProtocol):
return page_view_images_resolved
@override
+ @update_job_metadata
async def make_extract_pages(
self,
job_metadata: JobMetadata,
@@ -344,6 +368,7 @@ class ContentGenerator(ContentGeneratorProtocol):
extract_handle: str,
extract_job_params: ExtractJobParams | None = None,
extract_job_config: ExtractJobConfig | None = None,
+ wfid: str | None = None,
) -> list[PageContent]:
extract_job_params = extract_job_params or ExtractJobParams.make_default_extract_job_params()
extract_job_config = extract_job_config or ExtractJobConfig()
diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.py
index 983285462..19d94ff51 100644
--- a/pipelex/cogt/content_generation/content_generator_dry.py
+++ b/pipelex/cogt/content_generation/content_generator_dry.py
@@ -55,6 +55,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
job_metadata: JobMetadata,
llm_setting_main: LLMSetting,
llm_prompt_for_text: LLMPrompt,
+ wfid: str | None = None,
) -> str:
func_name = "make_llm_text"
log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")
@@ -69,6 +70,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
object_class: type[BaseModelTypeVar],
llm_setting_for_object: LLMSetting,
llm_prompt_for_object: LLMPrompt,
+ wfid: str | None = None,
) -> BaseModelTypeVar:
object_factory = DryRunFactory.make_dry_run_factory(object_class)
# We run validators to ensure mock data is valid. Fields with format constraints
@@ -86,6 +88,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
llm_setting_for_object: LLMSetting,
llm_prompt_for_text: LLMPrompt,
llm_prompt_factory_for_object: LLMPromptFactoryAbstract | None = None,
+ wfid: str | None = None,
) -> BaseModelTypeVar:
func_name = "make_text_then_object"
log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")
@@ -106,6 +109,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
llm_setting_for_object_list: LLMSetting,
llm_prompt_for_object_list: LLMPrompt,
nb_items: int | None = None,
+ wfid: str | None = None,
) -> list[BaseModelTypeVar]:
func_name = "make_object_list_direct"
log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")
@@ -136,6 +140,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
llm_prompt_for_text: LLMPrompt,
llm_prompt_factory_for_object_list: LLMPromptFactoryAbstract | None = None,
nb_items: int | None = None,
+ wfid: str | None = None,
) -> list[BaseModelTypeVar]:
func_name = "make_text_then_object_list"
log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")
@@ -197,6 +202,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
img_gen_prompt: ImgGenPrompt,
img_gen_job_params: ImgGenJobParams | None = None,
img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
) -> ImageContent:
func_name = "make_single_image"
log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")
@@ -222,6 +228,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
nb_images: int,
img_gen_job_params: ImgGenJobParams | None = None,
img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
) -> list[ImageContent]:
func_name = "make_image_list"
log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")
@@ -243,10 +250,12 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
@override
async def make_templated_text(
self,
+ job_metadata: JobMetadata,
context: dict[str, Any],
template: str,
templating_style: TemplatingStyle | None = None,
template_category: TemplateCategory | None = None,
+ wfid: str | None = None,
) -> str:
check_jinja2_parsing(template_source=template, template_category=template_category or TemplateCategory.BASIC)
func_name = "make_templated_text"
@@ -265,6 +274,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
extract_handle: str,
extract_job_params: ExtractJobParams | None = None,
extract_job_config: ExtractJobConfig | None = None,
+ wfid: str | None = None,
) -> list[ImageContent]:
if not extract_input.document_uri:
msg = "Document URI is required to render page views"
@@ -283,6 +293,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
return page_view_images_resolved
@override
+ @update_job_metadata
async def make_extract_pages(
self,
job_metadata: JobMetadata,
@@ -290,6 +301,7 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
extract_handle: str,
extract_job_params: ExtractJobParams | None = None,
extract_job_config: ExtractJobConfig | None = None,
+ wfid: str | None = None,
) -> list[PageContent]:
func_name = "make_extract_pages"
log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.{func_name}")
diff --git a/pipelex/cogt/content_generation/content_generator_protocol.py b/pipelex/cogt/content_generation/content_generator_protocol.py
index d73a3e2ed..81592b4b5 100644
--- a/pipelex/cogt/content_generation/content_generator_protocol.py
+++ b/pipelex/cogt/content_generation/content_generator_protocol.py
@@ -49,6 +49,7 @@ class ContentGeneratorProtocol(Protocol):
job_metadata: JobMetadata,
llm_setting_main: LLMSetting,
llm_prompt_for_text: LLMPrompt,
+ wfid: str | None = None,
) -> Coroutine[Any, Any, str]: ...
def make_object_direct(
@@ -57,6 +58,7 @@ class ContentGeneratorProtocol(Protocol):
object_class: type[BaseModelTypeVar],
llm_setting_for_object: LLMSetting,
llm_prompt_for_object: LLMPrompt,
+ wfid: str | None = None,
) -> Coroutine[Any, Any, BaseModelTypeVar]: ...
def make_text_then_object(
@@ -67,6 +69,7 @@ class ContentGeneratorProtocol(Protocol):
llm_setting_for_object: LLMSetting,
llm_prompt_for_text: LLMPrompt,
llm_prompt_factory_for_object: LLMPromptFactoryAbstract | None = None,
+ wfid: str | None = None,
) -> Coroutine[Any, Any, BaseModelTypeVar]: ...
def make_object_list_direct(
@@ -76,6 +79,7 @@ class ContentGeneratorProtocol(Protocol):
llm_setting_for_object_list: LLMSetting,
llm_prompt_for_object_list: LLMPrompt,
nb_items: int | None = None,
+ wfid: str | None = None,
) -> Coroutine[Any, Any, list[BaseModelTypeVar]]: ...
def make_text_then_object_list(
@@ -87,6 +91,7 @@ class ContentGeneratorProtocol(Protocol):
llm_prompt_for_text: LLMPrompt,
llm_prompt_factory_for_object_list: LLMPromptFactoryAbstract | None = None,
nb_items: int | None = None,
+ wfid: str | None = None,
) -> Coroutine[Any, Any, list[BaseModelTypeVar]]: ...
async def make_image_content(
@@ -109,6 +114,7 @@ class ContentGeneratorProtocol(Protocol):
img_gen_prompt: ImgGenPrompt,
img_gen_job_params: ImgGenJobParams | None = None,
img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
) -> Coroutine[Any, Any, ImageContent]: ...
def make_image_list(
@@ -119,24 +125,28 @@ class ContentGeneratorProtocol(Protocol):
nb_images: int,
img_gen_job_params: ImgGenJobParams | None = None,
img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
) -> Coroutine[Any, Any, list[ImageContent]]: ...
def make_templated_text(
self,
+ job_metadata: JobMetadata,
context: dict[str, Any],
template: str,
templating_style: TemplatingStyle | None = None,
template_category: TemplateCategory | None = None,
+ wfid: str | None = None,
) -> Coroutine[Any, Any, str]: ...
- async def make_render_page_views(
+ def make_render_page_views(
self,
job_metadata: JobMetadata,
extract_input: ExtractInput,
extract_handle: str,
extract_job_params: ExtractJobParams | None = None,
extract_job_config: ExtractJobConfig | None = None,
- ) -> list[ImageContent]: ...
+ wfid: str | None = None,
+ ) -> Coroutine[Any, Any, list[ImageContent]]: ...
def make_extract_pages(
self,
@@ -145,4 +155,5 @@ class ContentGeneratorProtocol(Protocol):
extract_handle: str,
extract_job_params: ExtractJobParams,
extract_job_config: ExtractJobConfig,
+ wfid: str | None = None,
) -> Coroutine[Any, Any, list[PageContent]]: ...
diff --git a/pipelex/cogt/content_generation/exceptions.py b/pipelex/cogt/content_generation/exceptions.py
index 0695fe3ed..6eafdbac0 100644
--- a/pipelex/cogt/content_generation/exceptions.py
+++ b/pipelex/cogt/content_generation/exceptions.py
@@ -1,9 +1,9 @@
-from pipelex.base_exceptions import PipelexError
+from pipelex.base_exceptions import PipelexError, SecurityError
-class ContentGenerationError(PipelexError):
+class NeitherUrlNorDataError(PipelexError):
pass
-class NeitherUrlNorDataError(ContentGenerationError):
+class UnsafeSchemaError(SecurityError):
pass
diff --git a/pipelex/cogt/content_generation/extract_generate.py b/pipelex/cogt/content_generation/extract_generate.py
index 43364c7c2..fd0846105 100644
--- a/pipelex/cogt/content_generation/extract_generate.py
+++ b/pipelex/cogt/content_generation/extract_generate.py
@@ -1,6 +1,8 @@
from pipelex.cogt.content_generation.assignment_models import ExtractAssignment
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
from pipelex.cogt.extract.extract_job_factory import ExtractJobFactory
from pipelex.cogt.extract.extract_output import ExtractOutput
+from pipelex.core.stuffs.page_content import PageContent
from pipelex.hub import get_extract_worker
@@ -13,3 +15,16 @@ async def extract_gen_pages(extract_assignment: ExtractAssignment) -> ExtractOut
job_metadata=extract_assignment.job_metadata,
)
return await extract_worker.extract_pages(extract_job=extract_job)
+
+
+async def extract_gen_pages_and_store(
+ extract_assignment: ExtractAssignment,
+ generated_content_factory: GeneratedContentFactory,
+) -> list[PageContent]:
+ """Extract pages and store extracted images, returning PageContent with URLs (no raw binary data)."""
+ extract_output = await extract_gen_pages(extract_assignment)
+ return await generated_content_factory.make_page_contents(
+ primary_id=extract_assignment.job_metadata.user_id,
+ secondary_id=extract_assignment.job_metadata.pipeline_run_id,
+ extract_output=extract_output,
+ )
diff --git a/pipelex/cogt/content_generation/img_gen_generate.py b/pipelex/cogt/content_generation/img_gen_generate.py
index de300b5d8..a45ab01f4 100644
--- a/pipelex/cogt/content_generation/img_gen_generate.py
+++ b/pipelex/cogt/content_generation/img_gen_generate.py
@@ -1,7 +1,9 @@
from pipelex import log
from pipelex.cogt.content_generation.assignment_models import ImgGenAssignment
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.cogt.img_gen.img_gen_job_factory import ImgGenJobFactory
+from pipelex.core.stuffs.image_content import ImageContent
from pipelex.hub import get_img_gen_worker
@@ -38,3 +40,38 @@ async def img_gen_image(img_gen_assignment: ImgGenAssignment) -> GeneratedImageR
if img_gen_assignment.nb_images > 1:
return await img_gen_image_list(img_gen_assignment)
return await img_gen_single_image(img_gen_assignment)
+
+
+async def img_gen_single_image_and_store(
+ img_gen_assignment: ImgGenAssignment,
+ generated_content_factory: GeneratedContentFactory,
+) -> ImageContent:
+ """Generate a single image and store it, returning an ImageContent with URLs (no raw binary data)."""
+ generated_image = await img_gen_single_image(img_gen_assignment)
+ image_content = await generated_content_factory.make_image_content(
+ primary_id=img_gen_assignment.job_metadata.user_id,
+ secondary_id=img_gen_assignment.job_metadata.pipeline_run_id,
+ raw_details=generated_image,
+ )
+ image_content.source_prompt = img_gen_assignment.img_gen_prompt.positive_text
+ image_content.source_negative_prompt = img_gen_assignment.img_gen_prompt.negative_text
+ return image_content
+
+
+async def img_gen_image_list_and_store(
+ img_gen_assignment: ImgGenAssignment,
+ generated_content_factory: GeneratedContentFactory,
+) -> list[ImageContent]:
+ """Generate multiple images and store them, returning ImageContent list with URLs (no raw binary data)."""
+ generated_image_list = await img_gen_image_list(img_gen_assignment)
+ image_contents: list[ImageContent] = []
+ for raw_details in generated_image_list:
+ image_content = await generated_content_factory.make_image_content(
+ primary_id=img_gen_assignment.job_metadata.user_id,
+ secondary_id=img_gen_assignment.job_metadata.pipeline_run_id,
+ raw_details=raw_details,
+ )
+ image_content.source_prompt = img_gen_assignment.img_gen_prompt.positive_text
+ image_content.source_negative_prompt = img_gen_assignment.img_gen_prompt.negative_text
+ image_contents.append(image_content)
+ return image_contents
diff --git a/pipelex/cogt/content_generation/llm_generate.py b/pipelex/cogt/content_generation/llm_generate.py
index e0d858100..cddddd719 100644
--- a/pipelex/cogt/content_generation/llm_generate.py
+++ b/pipelex/cogt/content_generation/llm_generate.py
@@ -4,8 +4,9 @@ from pydantic import BaseModel
from pipelex import log
from pipelex.cogt.content_generation.assignment_models import LLMAssignment, ObjectAssignment
+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
from pipelex.cogt.llm.llm_job_factory import LLMJobFactory
-from pipelex.hub import get_class_registry, get_llm_worker
+from pipelex.hub import get_llm_worker
async def llm_gen_text(llm_assignment: LLMAssignment) -> str:
@@ -28,8 +29,10 @@ async def llm_gen_object(object_assignment: ObjectAssignment) -> BaseModel:
llm_prompt=llm_assignment.llm_prompt,
llm_job_params=llm_assignment.llm_job_params,
)
- content_class_name = object_assignment.object_class_name
- content_class = get_class_registry().get_required_base_model(name=content_class_name)
+ content_class = SchemaToModelFactory.make_from_json_schema(
+ schema=object_assignment.object_class_schema,
+ class_name=object_assignment.object_class_name,
+ )
generated_object: BaseModel = await llm_worker.gen_object(
llm_job=llm_job,
schema=content_class,
@@ -47,7 +50,10 @@ async def llm_gen_object_list(object_assignment: ObjectAssignment) -> list[BaseM
llm_job_params=llm_assignment.llm_job_params,
)
item_class_name = object_assignment.object_class_name
- item_class = get_class_registry().get_required_class(name=item_class_name)
+ item_class = SchemaToModelFactory.make_from_json_schema(
+ schema=object_assignment.object_class_schema,
+ class_name=item_class_name,
+ )
class ListSchema(BaseModel):
items: list[item_class] # type: ignore[valid-type] # pyright: ignore[reportInvalidTypeForm]
diff --git a/pipelex/cogt/content_generation/schema_to_model_factory.py b/pipelex/cogt/content_generation/schema_to_model_factory.py
new file mode 100644
index 000000000..8cd3f33da
--- /dev/null
+++ b/pipelex/cogt/content_generation/schema_to_model_factory.py
@@ -0,0 +1,255 @@
+"""Reconstruct a Pydantic BaseModel class from a JSON schema using datamodel-code-generator.
+
+This module is the bridge between JSON schema (from model_json_schema()) and a live
+BaseModel class that can be used as a structured output schema for LLM generation.
+The generated class carries its own source code as __kajson_class_source__, enabling
+kajson to deserialize it across process boundaries without a class registry.
+
+Security model: this module exec()'s code generated from an attacker-influenceable JSON
+schema when schemas cross a process boundary (e.g. Temporal payloads). Two layers
+narrow the attack surface, but Layer 2 is defense-in-depth, NOT a sandbox:
+
+- Layer 1 — `_reject_unsafe_schema_extensions`: rejects `x-python-*` codegen extensions,
+ the only currently-known primitive that lets a JSON schema influence emitted imports.
+ This is the real defense.
+- Layer 2 — `_make_restricted_builtins`: removes a small allowlist of dangerous builtins
+ (eval/exec/compile/open/...) and restricts `__import__` to a fixed top-level allowlist.
+ This narrows the surface but does NOT contain a determined attacker: `__build_class__`,
+ `getattr`, `type`, and `object` remain reachable, so `().__class__.__base__.__subclasses__()`
+ can enumerate all loaded classes from inside the exec'd namespace. Any future codegen
+ vector that emits arbitrary Python (beyond the schema-driven `from X import Y`
+ statements we already block) would let an attacker pivot via that enumeration.
+
+Operationally: if Pipelex's threat model ever shifts to fully untrusted schemas (e.g.
+multi-tenant where one tenant can submit arbitrary schemas that another tenant's worker
+will codegen), replace exec() with a real sandbox (subprocess + seccomp) or migrate to
+programmatic class construction via `pydantic.create_model()` — see the TODO inside
+`_exec_and_extract_class`.
+"""
+
+import builtins
+import hashlib
+import json
+import re
+import tempfile
+import threading
+from collections import OrderedDict
+from pathlib import Path
+from typing import Any, ClassVar, cast
+
+from pydantic import BaseModel
+
+from pipelex.cogt.content_generation.exceptions import UnsafeSchemaError
+
+
+class SchemaToModelFactory:
+ """Reconstruct Pydantic BaseModel classes from JSON schemas.
+
+ See the module docstring for the threat model and security perimeter.
+ """
+
+ _UNSAFE_EXTENSION_PREFIX: ClassVar[str] = "x-python-"
+
+ _BLOCKED_BUILTINS: ClassVar[frozenset[str]] = frozenset({"eval", "exec", "compile", "open", "input", "breakpoint", "exit", "quit"})
+
+ _ALLOWED_IMPORT_TOP_LEVELS: ClassVar[frozenset[str]] = frozenset(
+ {
+ "pydantic",
+ "typing",
+ "typing_extensions",
+ "enum",
+ "datetime",
+ "decimal",
+ "uuid",
+ "__future__",
+ "collections",
+ "re",
+ }
+ )
+
+ _SCHEMA_CACHE_MAX_SIZE: ClassVar[int] = 1024
+
+ _schema_cache: ClassVar[OrderedDict[str, type[BaseModel]]] = OrderedDict()
+ _cache_lock: ClassVar[threading.Lock] = threading.Lock()
+
+ @classmethod
+ def make_from_json_schema(cls, schema: dict[str, Any], class_name: str) -> type[BaseModel]:
+ """Reconstruct a BaseModel class from a JSON schema dict.
+
+ Args:
+ schema: The JSON schema as returned by SomeModel.model_json_schema().
+ class_name: The expected class name (must match the schema's title).
+
+ Returns:
+ A BaseModel subclass with __kajson_class_source__ attached.
+ """
+ cache_key = hashlib.sha256(json.dumps(schema, sort_keys=True).encode()).hexdigest()
+
+ # Hold the lock for the entire call. This kills the same-schema thundering herd
+ # (N concurrent first-misses each paying a full codegen+exec round) at the cost
+ # of also serializing cache hits on other schemas behind any in-flight miss.
+ # That trade-off is acceptable because real workloads use a small bounded set
+ # of distinct schemas, each generated once and cached for the process lifetime —
+ # post-warmup, contention is limited to sub-millisecond cache lookups.
+ with cls._cache_lock:
+ if cache_key in cls._schema_cache:
+ cls._schema_cache.move_to_end(cache_key)
+ return cls._schema_cache[cache_key]
+
+ source_code = cls._generate_source_from_schema(schema)
+ reconstructed_class = cls._exec_and_extract_class(source_code, class_name)
+ reconstructed_class.__kajson_class_source__ = source_code # type: ignore[attr-defined]
+ if len(cls._schema_cache) >= cls._SCHEMA_CACHE_MAX_SIZE:
+ cls._schema_cache.popitem(last=False)
+ cls._schema_cache[cache_key] = reconstructed_class
+
+ return reconstructed_class
+
+ @classmethod
+ def _normalize_class_name(cls, title: str) -> str:
+ """Convert a schema title to a PascalCase class name (matching datamodel-code-generator behavior).
+
+ Splits on non-alphanumeric sequences, capitalizes the first char of each segment,
+ and preserves existing capitalization within segments (unlike str.title() which
+ lowercases everything after the first char).
+ """
+ segments = re.split(r"[^A-Za-z0-9]+", title)
+ return "".join(segment[0].upper() + segment[1:] if segment else "" for segment in segments)
+
+ @classmethod
+ def _collect_unsafe_extension_paths(cls, node: Any, path: str, found: list[str]) -> None:
+ """Walk a JSON-schema-shaped value, collecting dotted paths to any x-python-* key.
+
+ These extensions cause datamodel-code-generator to emit arbitrary `from X import Y`
+ statements with no sanitization, which is a code-injection primitive on any path
+ where the schema crosses an untrusted boundary (e.g. Temporal payloads).
+ """
+ if isinstance(node, dict):
+ node_dict = cast("dict[str, Any]", node)
+ for key, value in node_dict.items():
+ child_path = f"{path}.{key}" if path else key
+ if key.startswith(cls._UNSAFE_EXTENSION_PREFIX):
+ found.append(child_path)
+ cls._collect_unsafe_extension_paths(value, child_path, found)
+ elif isinstance(node, list):
+ node_list = cast("list[Any]", node)
+ for index, item in enumerate(node_list):
+ cls._collect_unsafe_extension_paths(item, f"{path}[{index}]", found)
+
+ @classmethod
+ def _reject_unsafe_schema_extensions(cls, schema: dict[str, Any]) -> None:
+ """Raise UnsafeSchemaError if the schema contains any x-python-* codegen extension.
+
+ Layer 1 of the SchemaToModelFactory security perimeter: kill the known exploit primitive
+ at the source rather than scrubbing it silently — silent stripping would turn a
+ malicious payload into a successful run with the attacker's payload merely ignored,
+ leaving no signal in logs for incident response.
+ """
+ found: list[str] = []
+ cls._collect_unsafe_extension_paths(schema, "", found)
+ if found:
+ msg = f"Unsafe codegen extensions found in schema: {found}"
+ raise UnsafeSchemaError(msg)
+
+ @classmethod
+ def _generate_source_from_schema(cls, schema: dict[str, Any]) -> str:
+ """Generate Python source code from a JSON schema using datamodel-code-generator."""
+ from datamodel_code_generator import InputFileType, generate # noqa: PLC0415
+ from datamodel_code_generator.enums import DataModelType # noqa: PLC0415
+
+ cls._reject_unsafe_schema_extensions(schema)
+
+ schema_str = json.dumps(schema)
+
+ with tempfile.NamedTemporaryFile(suffix=".py", delete=False) as tmp:
+ output_path = Path(tmp.name)
+
+ try:
+ generate(
+ input_=schema_str,
+ input_file_type=InputFileType.JsonSchema,
+ output=output_path,
+ output_model_type=DataModelType.PydanticV2BaseModel,
+ )
+ return output_path.read_text(encoding="utf-8")
+ finally:
+ output_path.unlink(missing_ok=True)
+
+ @classmethod
+ def _restricted_import(
+ cls,
+ name: str,
+ import_globals: dict[str, Any] | None = None,
+ import_locals: dict[str, Any] | None = None,
+ fromlist: tuple[str, ...] | None = (),
+ level: int = 0,
+ ) -> Any:
+ """Replacement for builtin __import__ that allows only an allowlist of modules.
+
+ Layer 2 of the SchemaToModelFactory security perimeter: even if a future codegen feature
+ in datamodel-code-generator emits new imports, this wrapper keeps the exec namespace
+ constrained to what the generator's output legitimately requires. The intended
+ failure mode is "test breaks loudly when the allowlist needs updating" rather than
+ "silent privilege expansion."
+ """
+ if level != 0:
+ msg = f"Relative imports are not allowed in generated code: {name!r}"
+ raise ImportError(msg)
+ top_level = name.split(".", 1)[0]
+ if top_level not in cls._ALLOWED_IMPORT_TOP_LEVELS:
+ msg = f"Import of {name!r} is not allowed in generated code"
+ raise ImportError(msg)
+ return builtins.__import__(name, import_globals, import_locals, fromlist or (), level)
+
+ @classmethod
+ def _make_restricted_builtins(cls) -> dict[str, Any]:
+ """Build a builtins dict that blocks dangerous functions and restricts __import__ to an allowlist.
+
+ This is defense-in-depth, not a sandbox. `__build_class__`, `getattr`, `type`,
+ and `object` remain reachable, so `().__class__.__base__.__subclasses__()` can
+ enumerate all loaded classes from inside the exec'd namespace. The real defense
+ is `_reject_unsafe_schema_extensions` (Layer 1), which prevents the codegen
+ from emitting attacker-controlled imports in the first place.
+ """
+ safe_builtins = {name: obj for name, obj in vars(builtins).items() if name not in cls._BLOCKED_BUILTINS}
+ safe_builtins["__import__"] = cls._restricted_import
+ return safe_builtins
+
+ @classmethod
+ def _exec_and_extract_class(cls, source_code: str, class_name: str) -> type[BaseModel]:
+ """Execute source code and extract the named BaseModel class."""
+ # TODO: replace exec() with programmatic construction via pydantic.create_model().
+ # Eliminates the exec primitive entirely, removes the need for both security
+ # layers, and lets us delete _make_restricted_builtins/_restricted_import.
+ # Non-trivial because nested models, enums, and forward references currently
+ # rely on datamodel-code-generator's output structure.
+ namespace: dict[str, Any] = {"__builtins__": cls._make_restricted_builtins()}
+ exec(compile(source_code, "<schema_to_model_factory>", "exec"), namespace)
+
+ # Collect all BaseModel subclasses from the exec'd namespace
+ model_classes: dict[str, type[BaseModel]] = {
+ name: obj for name, obj in namespace.items() if isinstance(obj, type) and issubclass(obj, BaseModel) and obj is not BaseModel
+ }
+
+ # datamodel-code-generator normalizes schema titles to PascalCase class names
+ # (e.g. "dynamic_concept_test__Greeting" → "DynamicConceptTestGreeting"),
+ # so try both the original name and the normalized version.
+ extracted_class = model_classes.get(class_name)
+ if extracted_class is None:
+ normalized_name = cls._normalize_class_name(class_name)
+ extracted_class = model_classes.get(normalized_name)
+ if extracted_class is None:
+ available = list(model_classes.keys())
+ msg = f"Class '{class_name}' not found in generated source. Available BaseModel classes: {available}"
+ raise ValueError(msg)
+
+ # datamodel-code-generator uses `from __future__ import annotations` which turns
+ # type annotations into strings. For nested models, we need to rebuild so Pydantic
+ # resolves the forward references against all classes in the generated namespace.
+ # Include ALL user-defined types (models + enums) so forward references to
+ # generated Enum classes (e.g. choices fields) can be resolved.
+ all_generated_types: dict[str, Any] = {name: obj for name, obj in namespace.items() if isinstance(obj, type) and not name.startswith("_")}
+ for model_cls in model_classes.values():
+ model_cls.model_rebuild(_types_namespace=all_generated_types)
+
+ return extracted_class
diff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.py
index c5654f102..90b6b6526 100644
--- a/pipelex/cogt/exceptions.py
+++ b/pipelex/cogt/exceptions.py
@@ -2,7 +2,9 @@ from __future__ import annotations
from typing import TYPE_CHECKING
-from pipelex.base_exceptions import PipelexError
+from typing_extensions import override
+
+from pipelex.base_exceptions import ErrorReport, PipelexError
from pipelex.types import StrEnum
if TYPE_CHECKING:
@@ -10,12 +12,54 @@ if TYPE_CHECKING:
from pipelex.cogt.models.model_reference import ModelReferenceKind
+class InferenceErrorCategory(StrEnum):
+ """Classifies inference errors for retry decisions and error reporting."""
+
+ TRANSIENT = "transient"
+ CONFIGURATION = "configuration"
+ CONTENT = "content"
+ CAPACITY = "capacity"
+
+ @property
+ def is_retryable(self) -> bool:
+ match self:
+ case InferenceErrorCategory.TRANSIENT:
+ return True
+ case InferenceErrorCategory.CONFIGURATION | InferenceErrorCategory.CONTENT | InferenceErrorCategory.CAPACITY:
+ return False
+
+
class CogtError(PipelexError):
- pass
+ error_category: InferenceErrorCategory | None = None
+ user_action: str | None = None
+
+ def __init__(
+ self,
+ message: str,
+ error_category: InferenceErrorCategory | None = None,
+ user_action: str | None = None,
+ ):
+ super().__init__(message)
+ if error_category is not None:
+ self.error_category = error_category
+ if user_action is not None:
+ self.user_action = user_action
+
+ @override
+ def to_error_report(self) -> ErrorReport:
+ return ErrorReport(
+ error_type=type(self).__name__,
+ message=self.message,
+ error_category=self.error_category,
+ retryable=self.error_category.is_retryable if self.error_category is not None else None,
+ user_action=self.user_action,
+ model=getattr(self, "model_handle", None),
+ provider=getattr(self, "backend_name", None),
+ )
class LLMConfigError(CogtError):
- pass
+ error_category = InferenceErrorCategory.CONFIGURATION
class ImageContentError(CogtError):
@@ -40,6 +84,8 @@ class ModelChoiceNotFoundError(CogtError):
Includes available options and migration hints in error message.
"""
+ error_category = InferenceErrorCategory.CONFIGURATION
+
def __init__(
self,
message: str,
@@ -78,15 +124,15 @@ class ModelChoiceNotFoundError(CogtError):
class LLMSettingsValidationError(CogtError):
- pass
+ error_category = InferenceErrorCategory.CONFIGURATION
class ImgGenSettingsValidationError(CogtError):
- pass
+ error_category = InferenceErrorCategory.CONFIGURATION
class ModelDeckValidatonError(CogtError):
- pass
+ error_category = InferenceErrorCategory.CONFIGURATION
class ModelDeckPresetValidatonError(ModelDeckValidatonError):
@@ -106,6 +152,8 @@ class ModelDeckPresetValidatonError(ModelDeckValidatonError):
class ModelNotFoundError(CogtError):
+ error_category = InferenceErrorCategory.CONFIGURATION
+
def __init__(self, message: str, model_handle: str):
self.model_handle = model_handle
super().__init__(message)
@@ -119,6 +167,8 @@ class ModelWaterfallError(ModelNotFoundError):
class LLMHandleNotFoundError(CogtError):
+ error_category = InferenceErrorCategory.CONFIGURATION
+
def __init__(self, message: str, preset_id: str, model_handle: str, enabled_backends: set[str] | None = None):
self.preset_id = preset_id
self.model_handle = model_handle
@@ -127,6 +177,8 @@ class LLMHandleNotFoundError(CogtError):
class ImgGenHandleNotFoundError(CogtError):
+ error_category = InferenceErrorCategory.CONFIGURATION
+
def __init__(self, message: str, preset_id: str, model_handle: str):
self.preset_id = preset_id
self.model_handle = model_handle
@@ -134,6 +186,8 @@ class ImgGenHandleNotFoundError(CogtError):
class ExtractHandleNotFoundError(CogtError):
+ error_category = InferenceErrorCategory.CONFIGURATION
+
def __init__(self, message: str, preset_id: str, model_handle: str):
self.preset_id = preset_id
self.model_handle = model_handle
@@ -141,6 +195,8 @@ class ExtractHandleNotFoundError(CogtError):
class SearchHandleNotFoundError(CogtError):
+ error_category = InferenceErrorCategory.CONFIGURATION
+
def __init__(self, message: str, preset_id: str, model_handle: str):
self.preset_id = preset_id
self.model_handle = model_handle
@@ -160,7 +216,7 @@ class LLMModelNotFoundError(ModelNotFoundError):
class LLMCapabilityError(CogtError):
- pass
+ error_category = InferenceErrorCategory.CONFIGURATION
class LLMCompletionError(CogtError):
@@ -216,13 +272,17 @@ class ImgGenGeneratedTypeError(ImgGenGenerationError):
class ExtractCapabilityError(CogtError):
- pass
+ error_category = InferenceErrorCategory.CONFIGURATION
class ExtractJobFailureError(CogtError):
pass
+class SearchJobFailureError(CogtError):
+ pass
+
+
class RoutingProfileLibraryNotFoundError(CogtError):
pass
@@ -254,14 +314,17 @@ class InferenceBackendCredentialsErrorType(StrEnum):
class InferenceBackendCredentialsError(CogtError):
+ error_category = InferenceErrorCategory.CONFIGURATION
+ user_action = "Check that the required API key environment variable is set"
+
def __init__(
self,
- error_type: InferenceBackendCredentialsErrorType,
+ credentials_error_type: InferenceBackendCredentialsErrorType,
backend_name: str,
message: str,
key_name: str,
):
- self.error_type = error_type
+ self.credentials_error_type = credentials_error_type
self.backend_name = backend_name
self.key_name = key_name
super().__init__(message)
@@ -272,7 +335,7 @@ class InferenceBackendLibraryError(CogtError):
class RoutingProfileDisabledBackendError(CogtError):
- pass
+ error_category = InferenceErrorCategory.CONFIGURATION
class ModelManagerError(CogtError):
diff --git a/pipelex/cogt/extract/extract_report.py b/pipelex/cogt/extract/extract_report.py
index 55e803ab3..7889a4d9a 100644
--- a/pipelex/cogt/extract/extract_report.py
+++ b/pipelex/cogt/extract/extract_report.py
@@ -1,4 +1,4 @@
-from typing import Any
+from typing import Any, Literal
from pydantic import BaseModel
@@ -32,7 +32,7 @@ class ExtractTokenCostReportField(StrEnum):
class ExtractTokenCostReport(BaseModel):
- model_type: str = "extract"
+ model_type: Literal["extract"] = "extract"
job_metadata: JobMetadata
inference_model_name: str
platform_model_id: str
@@ -64,7 +64,7 @@ class ExtractTokenCostReport(BaseModel):
class ExtractTokensUsage(BaseModel):
- model_type: str = "extract"
+ model_type: Literal["extract"] = "extract"
job_metadata: JobMetadata
inference_model_name: str
unit_costs: CostsByCategoryDict
diff --git a/pipelex/cogt/img_gen/img_gen_report.py b/pipelex/cogt/img_gen/img_gen_report.py
index 7390df79a..036b035a4 100644
--- a/pipelex/cogt/img_gen/img_gen_report.py
+++ b/pipelex/cogt/img_gen/img_gen_report.py
@@ -1,4 +1,4 @@
-from typing import Any
+from typing import Any, Literal
from pydantic import BaseModel
@@ -32,7 +32,7 @@ class ImgGenTokenCostReportField(StrEnum):
class ImgGenTokenCostReport(BaseModel):
- model_type: str = "img_gen"
+ model_type: Literal["img_gen"] = "img_gen"
job_metadata: JobMetadata
inference_model_name: str
platform_model_id: str
@@ -64,7 +64,7 @@ class ImgGenTokenCostReport(BaseModel):
class ImgGenTokensUsage(BaseModel):
- model_type: str = "img_gen"
+ model_type: Literal["img_gen"] = "img_gen"
job_metadata: JobMetadata
inference_model_name: str
unit_costs: CostsByCategoryDict
diff --git a/pipelex/cogt/img_gen/img_gen_worker_factory.py b/pipelex/cogt/img_gen/img_gen_worker_factory.py
index 1642e484b..77ffb8528 100644
--- a/pipelex/cogt/img_gen/img_gen_worker_factory.py
+++ b/pipelex/cogt/img_gen/img_gen_worker_factory.py
@@ -5,8 +5,6 @@ from pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstract
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.exceptions import MissingDependencyError
from pipelex.hub import get_models_manager, get_plugin_manager
-from pipelex.plugins.blackboxai.blackboxai_completions_factory import BlackboxaiCompletionsFactory
-from pipelex.plugins.openrouter.openrouter_completions_factory import OpenRouterCompletionsFactory
from pipelex.plugins.plugin_sdk_registry import Plugin
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.system.exceptions import CredentialsError
@@ -123,6 +121,9 @@ class ImgGenWorkerFactory:
),
)
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from pipelex.plugins.blackboxai.blackboxai_completions_factory import BlackboxaiCompletionsFactory # noqa: PLC0415
+
bbai_completions_factory = BlackboxaiCompletionsFactory(is_http_url_enabled=True)
img_gen_worker = OpenAICompletionsImgGenWorker(
@@ -195,6 +196,9 @@ class ImgGenWorkerFactory:
),
)
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from pipelex.plugins.openrouter.openrouter_completions_factory import OpenRouterCompletionsFactory # noqa: PLC0415
+
openrouter_completions_factory = OpenRouterCompletionsFactory(is_http_url_enabled=True)
img_gen_worker = OpenAICompletionsImgGenWorker(
diff --git a/pipelex/cogt/inference/error_classification.py b/pipelex/cogt/inference/error_classification.py
new file mode 100644
index 000000000..22bb4617d
--- /dev/null
+++ b/pipelex/cogt/inference/error_classification.py
@@ -0,0 +1,116 @@
+"""Helpers for classifying SDK errors into InferenceErrorCategory values.
+
+Pure functions that inspect error messages to discriminate between
+quota exhaustion vs rate limiting, and detect content policy violations.
+"""
+
+_OPENAI_QUOTA_PATTERNS: tuple[str, ...] = (
+ "insufficient_quota",
+ "exceeded your current quota",
+)
+
+_ANTHROPIC_QUOTA_PATTERNS: tuple[str, ...] = (
+ "quota exceeded",
+ "quota has been",
+ "credit balance",
+ "out of credits",
+ "insufficient credit",
+ "billing limit",
+ "billing issue",
+)
+
+_CONTENT_POLICY_PATTERNS: tuple[str, ...] = (
+ "content_policy",
+ "content_filter",
+ "safety system",
+ "safety filter",
+ "blocked by safety",
+)
+
+_GOOGLE_QUOTA_PATTERNS: tuple[str, ...] = (
+ "quota exceeded",
+ "resource has been exhausted",
+ "billing limit",
+ "billing quota",
+ "billing exceeded",
+ "billing account",
+)
+
+_MISTRAL_QUOTA_PATTERNS: tuple[str, ...] = (
+ "quota",
+ "billing limit",
+ "billing quota",
+ "out of credits",
+ "insufficient credits",
+)
+
+_AWS_QUOTA_PATTERNS: tuple[str, ...] = (
+ "quota",
+ "limit exceeded",
+ "service quota",
+)
+
+_GATEWAY_QUOTA_PATTERNS: tuple[str, ...] = (
+ "quota",
+ "billing limit",
+ "billing quota",
+ "insufficient_quota",
+ "insufficient credit",
+ "insufficient funds",
+ "insufficient balance",
+ "credits exhausted",
+)
+
+
+def is_quota_exhaustion_openai(error_message: str) -> bool:
+ """Check if an OpenAI error message indicates quota/credits exhaustion rather than rate limiting."""
+ lower_message = error_message.lower()
+ return any(pattern in lower_message for pattern in _OPENAI_QUOTA_PATTERNS)
+
+
+def is_quota_exhaustion_anthropic(error_message: str) -> bool:
+ """Check if an Anthropic error message indicates quota/credits exhaustion rather than rate limiting."""
+ lower_message = error_message.lower()
+ return any(pattern in lower_message for pattern in _ANTHROPIC_QUOTA_PATTERNS)
+
+
+def is_quota_exhaustion_google(error_message: str) -> bool:
+ """Check if a Google error message indicates quota/credits exhaustion rather than rate limiting."""
+ lower_message = error_message.lower()
+ return any(pattern in lower_message for pattern in _GOOGLE_QUOTA_PATTERNS)
+
+
+def is_quota_exhaustion_mistral(error_message: str, status_code: int) -> bool:
+ """Check if a Mistral error indicates quota/credits exhaustion.
+
+ HTTP 402 (Payment Required) is a definitive quota signal.
+ HTTP 429 requires message inspection to distinguish quota from rate limiting.
+ """
+ if status_code == 402:
+ return True
+ lower_message = error_message.lower()
+ return status_code == 429 and any(pattern in lower_message for pattern in _MISTRAL_QUOTA_PATTERNS)
+
+
+def is_quota_exhaustion_aws(error_message: str) -> bool:
+ """Check if an AWS error message indicates quota/credits exhaustion rather than rate limiting."""
+ lower_message = error_message.lower()
+ return any(pattern in lower_message for pattern in _AWS_QUOTA_PATTERNS)
+
+
+def is_quota_exhaustion_gateway(error_message: str, status_code: int) -> bool:
+ """Check if a Portkey/Gateway error indicates quota/credits exhaustion.
+
+ HTTP 402 (Payment Required) is a definitive quota signal.
+ HTTP 429 requires message inspection to distinguish quota from rate limiting.
+ """
+ if status_code == 402:
+ return True
+ lower_message = error_message.lower()
+ return status_code == 429 and any(pattern in lower_message for pattern in _GATEWAY_QUOTA_PATTERNS)
+
+
+def is_content_policy_violation(error_message: str) -> bool:
+ """Check if an error message indicates a content policy or safety filter violation."""
+ lower_message = error_message.lower()
+ return any(pattern in lower_message for pattern in _CONTENT_POLICY_PATTERNS)
diff --git a/pipelex/cogt/llm/llm_prompt_template.py b/pipelex/cogt/llm/llm_prompt_template.py
index 9c4ff0af3..e753117bd 100644
--- a/pipelex/cogt/llm/llm_prompt_template.py
+++ b/pipelex/cogt/llm/llm_prompt_template.py
@@ -9,10 +9,10 @@ from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
from pipelex.cogt.llm.llm_prompt_template_inputs import LLMPromptTemplateInputs
from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.cogt.templating.template_rendering import render_template
from pipelex.cogt.templating.templating_style import TagStyle, TemplatingStyle
from pipelex.cogt.templating.text_format import TextFormat
from pipelex.config import get_config
-from pipelex.hub import get_content_generator
from pipelex.tools.misc.string_utils import is_none_or_has_text
@@ -90,25 +90,23 @@ class LLMPromptTemplate(LLMPromptFactoryAbstract):
llm_prompt.user_images = user_images
# input variables can be applied to prompt texts used as templates
+ llm_prompt_templating_style = TemplatingStyle(
+ tag_style=TagStyle.XML,
+ text_format=TextFormat.MARKDOWN,
+ )
if llm_prompt.system_text:
- llm_prompt.system_text = await get_content_generator().make_templated_text(
- context=all_template_inputs.root,
+ llm_prompt.system_text = await render_template(
template=llm_prompt.system_text,
- templating_style=TemplatingStyle(
- tag_style=TagStyle.XML,
- text_format=TextFormat.MARKDOWN,
- ),
- template_category=TemplateCategory.LLM_PROMPT,
+ category=TemplateCategory.LLM_PROMPT,
+ context=all_template_inputs.root,
+ templating_style=llm_prompt_templating_style,
)
if llm_prompt.user_text:
- llm_prompt.user_text = await get_content_generator().make_templated_text(
- context=all_template_inputs.root,
+ llm_prompt.user_text = await render_template(
template=llm_prompt.user_text,
- templating_style=TemplatingStyle(
- tag_style=TagStyle.XML,
- text_format=TextFormat.MARKDOWN,
- ),
- template_category=TemplateCategory.LLM_PROMPT,
+ category=TemplateCategory.LLM_PROMPT,
+ context=all_template_inputs.root,
+ templating_style=llm_prompt_templating_style,
)
return llm_prompt
diff --git a/pipelex/cogt/llm/llm_report.py b/pipelex/cogt/llm/llm_report.py
index a72b9ce75..0bc4cc3a0 100644
--- a/pipelex/cogt/llm/llm_report.py
+++ b/pipelex/cogt/llm/llm_report.py
@@ -1,4 +1,4 @@
-from typing import Any
+from typing import Any, Literal
from pydantic import BaseModel
@@ -32,7 +32,7 @@ class LLMTokenCostReportField(StrEnum):
class LLMTokenCostReport(BaseModel):
- model_type: str = "llm"
+ model_type: Literal["llm"] = "llm"
job_metadata: JobMetadata
inference_model_name: str
platform_model_id: str
@@ -64,7 +64,7 @@ class LLMTokenCostReport(BaseModel):
class LLMTokensUsage(BaseModel):
- model_type: str = "llm"
+ model_type: Literal["llm"] = "llm"
job_metadata: JobMetadata
inference_model_name: str
unit_costs: CostsByCategoryDict
diff --git a/pipelex/cogt/model_backends/backend_factory.py b/pipelex/cogt/model_backends/backend_factory.py
index 0c2f6cd79..4168a0958 100644
--- a/pipelex/cogt/model_backends/backend_factory.py
+++ b/pipelex/cogt/model_backends/backend_factory.py
@@ -5,7 +5,6 @@ from pydantic import Field, field_validator
from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.cogt.model_backends.constraints import ListedConstraint, ValuedConstraint
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
-from pipelex.plugins.openai.vertexai_factory import VertexAIFactory
from pipelex.system.configuration.config_model import ConfigModel
from pipelex.tools.typing.pydantic_utils import empty_dict_factory_of, empty_list_factory_of
@@ -49,6 +48,9 @@ class InferenceBackendFactory:
# Deal with special authentication for some backends
match name:
case "vertexai":
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from pipelex.plugins.openai.vertexai_factory import VertexAIFactory # noqa: PLC0415
+
endpoint, api_key = VertexAIFactory.make_endpoint_and_api_key(extra_config=extra_config)
case _:
pass
diff --git a/pipelex/cogt/model_backends/backend_library.py b/pipelex/cogt/model_backends/backend_library.py
index bfa498698..32047b35f 100644
--- a/pipelex/cogt/model_backends/backend_library.py
+++ b/pipelex/cogt/model_backends/backend_library.py
@@ -113,7 +113,7 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
msg = f"Variable substitution failed due to a pattern error in file '{backends_library_path}':\n{var_fallback_pattern_exc}"
key_name = "unknown"
raise InferenceBackendCredentialsError(
- error_type=InferenceBackendCredentialsErrorType.VAR_FALLBACK_PATTERN,
+ credentials_error_type=InferenceBackendCredentialsErrorType.VAR_FALLBACK_PATTERN,
backend_name=backend_name,
message=msg,
key_name=key_name,
@@ -128,7 +128,7 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
f"{var_not_found_exc}\nRun mode: '{runtime_manager.run_mode}'"
)
raise InferenceBackendCredentialsError(
- error_type=InferenceBackendCredentialsErrorType.VAR_NOT_FOUND,
+ credentials_error_type=InferenceBackendCredentialsErrorType.VAR_NOT_FOUND,
backend_name=backend_name,
message=msg,
key_name=var_not_found_exc.var_name,
@@ -138,7 +138,7 @@ class InferenceBackendLibrary(RootModel[InferenceBackendLibraryRoot]):
log.verbose(f"Skipping backend '{backend_name}': unknown variable prefix for '{unknown_var_prefix_exc.var_name}'")
continue
raise InferenceBackendCredentialsError(
- error_type=InferenceBackendCredentialsErrorType.UNKNOWN_VAR_PREFIX,
+ credentials_error_type=InferenceBackendCredentialsErrorType.UNKNOWN_VAR_PREFIX,
backend_name=backend_name,
message=(
f"Variable substitution failed due to an unknown variable prefix error "
diff --git a/pipelex/cogt/search/search_report.py b/pipelex/cogt/search/search_report.py
index 5b57784eb..4f7354092 100644
--- a/pipelex/cogt/search/search_report.py
+++ b/pipelex/cogt/search/search_report.py
@@ -1,4 +1,4 @@
-from typing import Any
+from typing import Any, Literal
from pydantic import BaseModel
@@ -32,7 +32,7 @@ class SearchTokenCostReportField(StrEnum):
class SearchTokenCostReport(BaseModel):
- model_type: str = "search"
+ model_type: Literal["search"] = "search"
job_metadata: JobMetadata
inference_model_name: str
platform_model_id: str
@@ -64,7 +64,7 @@ class SearchTokenCostReport(BaseModel):
class SearchTokensUsage(BaseModel):
- model_type: str = "search"
+ model_type: Literal["search"] = "search"
job_metadata: JobMetadata
inference_model_name: str
unit_costs: CostsByCategoryDict
diff --git a/pipelex/core/concepts/concept.py b/pipelex/core/concepts/concept.py
index 478d8ce89..19ff104fe 100644
--- a/pipelex/core/concepts/concept.py
+++ b/pipelex/core/concepts/concept.py
@@ -1,6 +1,6 @@
from typing import Any, Callable
-from kajson.kajson_manager import KajsonManager
+from kajson.class_registry_abstract import ClassRegistryAbstract
from mthds.models.concept import ConceptAbstract
from pydantic import field_validator
@@ -23,6 +23,14 @@ from pipelex.tools.misc.string_utils import pascal_case_to_sentence
from pipelex.tools.typing.class_utils import are_classes_equivalent, has_compatible_field
+def _get_class_registry() -> ClassRegistryAbstract:
+ """Lazy import to break circular dependency between concept.py and hub.py."""
+ import importlib # noqa: PLC0415
+
+ hub = importlib.import_module("pipelex.hub")
+ return hub.get_class_registry() # type: ignore[no-any-return]
+
+
class Concept(ConceptAbstract):
@field_validator("code")
@classmethod
@@ -129,8 +137,8 @@ class Concept(ConceptAbstract):
# Check class-based compatibility
# This now works even when one or both concepts have refines, since we generate
# structure classes that inherit from the refined concept's class
- concept_1_class = KajsonManager.get_class_registry().get_class(name=concept_1.structure_class_name)
- concept_2_class = KajsonManager.get_class_registry().get_class(name=concept_2.structure_class_name)
+ concept_1_class = _get_class_registry().get_class(name=concept_1.structure_class_name)
+ concept_2_class = _get_class_registry().get_class(name=concept_2.structure_class_name)
if concept_1_class is None or concept_2_class is None:
return False
@@ -156,13 +164,9 @@ class Concept(ConceptAbstract):
@classmethod
def is_valid_structure_class(cls, structure_class_name: str) -> bool:
- # TODO: DO NOT use the KajsonManager here. Pipelex needs to be instantiated to use the get_class_registry.
- # And when we go through KajsonManager, no error raises if pipelex is not instantiated.
- # We get_class_registry directly from KajsonManager instead of pipelex hub to avoid circular import
- if KajsonManager.get_class_registry().has_subclass(name=structure_class_name, base_class=StuffContent):
+ if _get_class_registry().has_subclass(name=structure_class_name, base_class=StuffContent):
return True
- # We get_class_registry directly from KajsonManager instead of pipelex hub to avoid circular import
- if KajsonManager.get_class_registry().has_class(name=structure_class_name):
+ if _get_class_registry().has_class(name=structure_class_name):
log.warning(f"Concept class '{structure_class_name}' is registered but it's not a subclass of StuffContent")
return False
@@ -172,7 +176,7 @@ class Concept(ConceptAbstract):
Returns:
The StuffContent subclass, or None if not found
"""
- structure_class = KajsonManager.get_class_registry().get_class(name=self.structure_class_name)
+ structure_class = _get_class_registry().get_class(name=self.structure_class_name)
if structure_class is None:
msg = f"Concept class '{self.structure_class_name}' not found"
raise ConceptValueError(msg)
@@ -180,7 +184,7 @@ class Concept(ConceptAbstract):
def search_for_nested_image_fields_in_structure_class(self) -> list[str]:
"""Recursively search for image fields in a structure class."""
- structure_class = KajsonManager.get_class_registry().get_required_subclass(name=self.structure_class_name, base_class=StuffContent)
+ structure_class = _get_class_registry().get_required_subclass(name=self.structure_class_name, base_class=StuffContent)
if not issubclass(structure_class, StuffContent):
msg = f"Concept class '{self.structure_class_name}' is not a subclass of StuffContent"
raise PipelexUnexpectedError(msg)
diff --git a/pipelex/core/concepts/concept_factory.py b/pipelex/core/concepts/concept_factory.py
index eb188c2b3..46b76e30e 100644
--- a/pipelex/core/concepts/concept_factory.py
+++ b/pipelex/core/concepts/concept_factory.py
@@ -1,6 +1,6 @@
from typing import cast
-from kajson.kajson_manager import KajsonManager
+from kajson.class_registry_abstract import ClassRegistryAbstract
from pydantic import BaseModel
from pipelex.core.concepts.concept import Concept
@@ -10,7 +10,7 @@ from pipelex.core.concepts.exceptions import (
ConceptRefineError,
ConceptStringError,
)
-from pipelex.core.concepts.helpers import normalize_structure_blueprint
+from pipelex.core.concepts.helpers import make_qualified_structure_class_name, normalize_structure_blueprint
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.concepts.structure_generation.exceptions import ConceptStructureGeneratorError
from pipelex.core.concepts.structure_generation.generator import StructureGenerator
@@ -21,6 +21,14 @@ from pipelex.core.stuffs.text_content import TextContent
from pipelex.types import StrEnum
+def _get_class_registry() -> ClassRegistryAbstract:
+ """Lazy import to break circular dependency with hub.py."""
+ import importlib # noqa: PLC0415
+
+ hub = importlib.import_module("pipelex.hub")
+ return hub.get_class_registry() # type: ignore[no-any-return]
+
+
class ConceptDeclarationType(StrEnum):
"""Enum representing the 5 ways a concept can be declared in MTHDS files.
@@ -309,9 +317,10 @@ class ConceptFactory:
# Normalize the structure blueprint to ensure all values are ConceptStructureBlueprint objects
normalized_structure = normalize_structure_blueprint(blueprint.structure)
+ qualified_class_name = make_qualified_structure_class_name(domain_code, concept_code)
try:
_, the_generated_class = StructureGenerator().generate_from_structure_blueprint(
- class_name=concept_code,
+ class_name=qualified_class_name,
structure_blueprint=normalized_structure,
description=blueprint.description,
)
@@ -320,9 +329,9 @@ class ConceptFactory:
raise ConceptFactoryError(msg) from exc
# Register the generated class
- KajsonManager.get_class_registry().register_class(the_generated_class)
+ _get_class_registry().register_class(the_generated_class)
- return concept_code
+ return qualified_class_name
@classmethod
def _handle_basic_blueprint(
@@ -336,10 +345,15 @@ class ConceptFactory:
Returns:
Tuple of (structure_class_name, refine_string)
"""
- # Generate a new class that inherits from TextContent and register it
- # (unless a valid structure class already exists for this concept_code)
+ qualified_class_name = make_qualified_structure_class_name(domain_code, concept_code)
+
+ # Check if a valid structure class already exists — first by bare concept_code
+ # (for pre-existing Python classes registered under their own name), then by
+ # qualified name (for previously generated dynamic classes)
if Concept.is_valid_structure_class(structure_class_name=concept_code):
return concept_code, None
+ if Concept.is_valid_structure_class(structure_class_name=qualified_class_name):
+ return qualified_class_name, None
# Because native concepts have structure class names diffrent than other (with "Content")
if concept_code in NativeConceptCode.values_list():
@@ -347,7 +361,7 @@ class ConceptFactory:
try:
_, the_generated_class = StructureGenerator().generate_from_structure_blueprint(
- class_name=concept_code,
+ class_name=qualified_class_name,
structure_blueprint={},
base_class_name=TextContent.__name__,
description=description,
@@ -356,9 +370,9 @@ class ConceptFactory:
msg = f"Error generating structure class for concept '{concept_code}' in domain '{domain_code}': {exc}"
raise ConceptFactoryError(msg) from exc
# Register the generated class
- KajsonManager.get_class_registry().register_class(the_generated_class)
+ _get_class_registry().register_class(the_generated_class)
- return concept_code, NativeConceptCode.TEXT.concept_ref
+ return qualified_class_name, NativeConceptCode.TEXT.concept_ref
@classmethod
def _handle_refines(
@@ -385,13 +399,15 @@ class ConceptFactory:
msg = f"Could not validate refine '{blueprint.refines}' for concept '{concept_code}' in domain '{domain_code}': {exc}"
raise ConceptFactoryError(msg) from exc
+ qualified_class_name = make_qualified_structure_class_name(domain_code, concept_code)
+
# Cross-package refines: base class isn't available locally, so generate
# a standalone TextContent subclass. The refinement relationship is tracked
# in the concept model's refines field for runtime compatibility checks.
if QualifiedRef.has_cross_package_prefix(current_refine):
try:
_, the_generated_class = StructureGenerator().generate_from_structure_blueprint(
- class_name=concept_code,
+ class_name=qualified_class_name,
structure_blueprint={},
description=blueprint.description,
)
@@ -402,24 +418,25 @@ class ConceptFactory:
)
raise ConceptFactoryError(msg) from exc
- KajsonManager.get_class_registry().register_class(the_generated_class)
- return concept_code, current_refine
+ _get_class_registry().register_class(the_generated_class)
+ return qualified_class_name, current_refine
# Get the refined concept's structure class name
# For native concepts, the structure class name is "ConceptCode" + "Content" (e.g., TextContent)
- # For custom concepts, the structure class name is just the concept code (e.g., Customer)
+ # For custom concepts, the structure class name is domain-qualified (e.g., my_domain__Customer)
refined_ref = QualifiedRef.parse(current_refine)
refined_concept_code = refined_ref.local_code
if NativeConceptCode.is_native_concept_ref_or_code(concept_ref_or_code=current_refine):
refined_structure_class_name = refined_concept_code + "Content"
else:
- refined_structure_class_name = refined_concept_code
+ refined_domain_code = refined_ref.domain_path or domain_code
+ refined_structure_class_name = make_qualified_structure_class_name(refined_domain_code, refined_concept_code)
# Generate a new class that inherits from the refined structure class
# This creates an empty class that can be extended with additional fields in the future
try:
_, the_generated_class = StructureGenerator().generate_from_structure_blueprint(
- class_name=concept_code,
+ class_name=qualified_class_name,
structure_blueprint={}, # Empty structure - just inherits from refined class
base_class_name=refined_structure_class_name,
description=blueprint.description,
@@ -432,9 +449,9 @@ class ConceptFactory:
raise ConceptFactoryError(msg) from exc
# Register the generated class
- KajsonManager.get_class_registry().register_class(the_generated_class)
+ _get_class_registry().register_class(the_generated_class)
- return concept_code, current_refine
+ return qualified_class_name, current_refine
@classmethod
def make_from_blueprint(
diff --git a/pipelex/core/concepts/helpers.py b/pipelex/core/concepts/helpers.py
index bf17699a2..001e192aa 100644
--- a/pipelex/core/concepts/helpers.py
+++ b/pipelex/core/concepts/helpers.py
@@ -36,7 +36,7 @@ def get_structure_class_name_from_blueprint(
raise ValueError(msg)
# Extract concept_code from concept_ref_or_code
- ref = QualifiedRef.parse(concept_ref_or_code)
+ ref = QualifiedRef.parse_stripping_cross_package(concept_ref_or_code)
concept_code = ref.local_code
if isinstance(blueprint_or_string_description, str):
@@ -87,6 +87,26 @@ def normalize_structure_blueprint(structure_dict: dict[str, str | ConceptStructu
return normalized
+def make_qualified_structure_class_name(domain_code: str, concept_code: str) -> str:
+ """Build a domain-qualified class name for dynamically generated structure classes.
+
+ Uses double underscore as separator to produce a valid Python identifier
+ that is distinct from bare concept codes. Dots in hierarchical domain codes
+ are replaced with interpuncts (·, U+00B7) to preserve Python identifier
+ validity without colliding with domains that already contain underscores
+ (e.g., "a.b" → "a·b" stays distinct from "a_b").
+
+ Args:
+ domain_code: The domain code (e.g., "conflict_alpha" or "legal.contracts.shareholder")
+ concept_code: The concept code (e.g., "Result")
+
+ Returns:
+ Domain-qualified class name (e.g., "legal·contracts·shareholder__Result")
+ """
+ safe_domain = domain_code.replace(".", "·")
+ return f"{safe_domain}__{concept_code}"
+
+
def extract_concept_code_from_concept_ref_or_code(concept_ref_or_code: str) -> str:
"""Extract the concept code from a concept reference or code.
@@ -100,5 +120,5 @@ def extract_concept_code_from_concept_ref_or_code(concept_ref_or_code: str) -> s
msg = f"Invalid concept_ref_or_code: '{concept_ref_or_code}' for extracting concept code"
raise ValueError(msg)
- ref = QualifiedRef.parse(concept_ref_or_code)
+ ref = QualifiedRef.parse_stripping_cross_package(concept_ref_or_code)
return ref.local_code
diff --git a/pipelex/core/concepts/structure_generation/generator.py b/pipelex/core/concepts/structure_generation/generator.py
index a119f386b..d9d9e30ea 100644
--- a/pipelex/core/concepts/structure_generation/generator.py
+++ b/pipelex/core/concepts/structure_generation/generator.py
@@ -4,13 +4,14 @@ from datetime import datetime
from enum import Enum
from typing import Any, List, Literal, Optional
-from kajson.kajson_manager import KajsonManager
+from kajson.class_registry_abstract import ClassRegistryAbstract
from pydantic import Field
from pipelex.core.concepts.concept_structure_blueprint import ConceptStructureBlueprint, ConceptStructureBlueprintFieldType
-from pipelex.core.concepts.helpers import extract_concept_code_from_concept_ref_or_code
+from pipelex.core.concepts.helpers import extract_concept_code_from_concept_ref_or_code, make_qualified_structure_class_name
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.concepts.structure_generation.exceptions import ConceptStructureGeneratorError, ConceptStructureValidationError, SyntaxErrorData
+from pipelex.core.qualified_ref import QualifiedRef
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.core.stuffs.stuff_content import StuffContent
@@ -20,6 +21,14 @@ _MAX_LINE_LENGTH = 150
_WRAP_WIDTH = 120 # conservative: leaves headroom for indentation and syntax overhead
+def _get_class_registry() -> ClassRegistryAbstract:
+ """Lazy import to break circular dependency with hub.py."""
+ import importlib # noqa: PLC0415
+
+ hub = importlib.import_module("pipelex.hub")
+ return hub.get_class_registry() # type: ignore[no-any-return]
+
+
class ConceptClassInfo:
"""Information about a concept's structure class for import generation."""
@@ -434,9 +443,13 @@ class StructureGenerator:
# No module path: use forward reference (resolved via model_rebuild at runtime)
return f'"{class_info.class_name}"'
- # Default: extract concept code and use as forward reference
- # e.g., "myapp.Customer" -> '"Customer"'
+ # Default: use domain-qualified forward reference when domain is available
+ # e.g., "myapp.Customer" -> '"myapp__Customer"'
concept_code = extract_concept_code_from_concept_ref_or_code(concept_ref)
+ parsed_ref = QualifiedRef.parse_stripping_cross_package(concept_ref)
+ if parsed_ref.domain_path and not NativeConceptCode.is_native_concept_ref_or_code(concept_ref):
+ qualified_name = make_qualified_structure_class_name(parsed_ref.domain_path, concept_code)
+ return f'"{qualified_name}"'
return f'"{concept_code}"'
def _generate_field(self, field_name: str, field_def: dict[str, Any] | str) -> str:
@@ -626,7 +639,7 @@ class StructureGenerator:
if base_class_name:
if not NativeConceptCode.is_native_structure_class(base_class_name):
# Not a native class, provide it from registry
- custom_base_class = KajsonManager.get_class_registry().get_class(name=base_class_name)
+ custom_base_class = _get_class_registry().get_class(name=base_class_name)
if custom_base_class is None:
msg = f"Base class '{base_class_name}' not found in native classes or class registry"
raise ConceptStructureValidationError(msg)
diff --git a/pipelex/core/memory/working_memory.py b/pipelex/core/memory/working_memory.py
index bdb9454ee..66719fbe0 100644
--- a/pipelex/core/memory/working_memory.py
+++ b/pipelex/core/memory/working_memory.py
@@ -20,7 +20,7 @@ from pipelex.core.stuffs.mermaid_content import MermaidContent
from pipelex.core.stuffs.number_content import NumberContent
from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.stuff_artefact import StuffArtefact
-from pipelex.core.stuffs.stuff_content import StuffContentType
+from pipelex.core.stuffs.stuff_content import StuffContent, StuffContentType
from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
from pipelex.core.stuffs.text_content import TextContent
from pipelex.tools.misc.context_provider_abstract import ContextProviderAbstract
@@ -468,3 +468,28 @@ class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):
def smart_dump(self) -> dict[str, Any]:
"""Serialize the working memory as a dictionary."""
return self.model_dump(serialize_as_any=True)
+
+ def dump_for_temporal(self) -> dict[str, Any]:
+ """Serialize for Temporal transit with explicit ListContent representation.
+
+ Like smart_dump(), but serializes ListContent as a plain list instead of
+ ``{"items": [...]}``. This removes the ambiguity at hydration time: a
+ ``list`` content value is always a ListContent, a ``dict`` is always a
+ single StuffContent.
+ """
+ raw = self.model_dump(serialize_as_any=True)
+ raw_root = raw.get("root", {})
+ for stuff_name, stuff in self.root.items():
+ content = stuff.content
+ if isinstance(content, ListContent) and stuff_name in raw_root:
+ list_content = cast("ListContent[StuffContent]", content)
+ serialized_items: list[dict[str, Any]] = []
+ for item in list_content.items:
+ item_dict = item.model_dump(serialize_as_any=True)
+ # Preserve type metadata for items under Anything concepts so
+ # the hydration side can reconstruct the correct content class.
+ item_dict["__class__"] = type(item).__name__
+ item_dict["__module__"] = type(item).__module__
+ serialized_items.append(item_dict)
+ raw_root[stuff_name]["content"] = serialized_items
+ return raw
diff --git a/pipelex/core/pipes/pipe_abstract.py b/pipelex/core/pipes/pipe_abstract.py
index fc40435b2..3912b949a 100644
--- a/pipelex/core/pipes/pipe_abstract.py
+++ b/pipelex/core/pipes/pipe_abstract.py
@@ -38,6 +38,7 @@ from pipelex.types import Self
if TYPE_CHECKING:
from pipelex.graph.graph_context import GraphContext
+ from pipelex.libraries.library_crate import LibraryCrate
PipeAbstractType = type["PipeAbstract"]
@@ -97,7 +98,7 @@ class PipeAbstract(ABC, BaseModel):
if tracer_manager is None or graph_context.parent_node_id is None:
return
tracer_manager.register_execution_data(
- graph_id=graph_context.graph_id,
+ lookup_key=graph_context.lookup_key,
node_id=graph_context.parent_node_id,
execution_data=execution_data,
)
@@ -411,6 +412,7 @@ class PipeAbstract(ABC, BaseModel):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
pipe_run_params.push_pipe_to_stack(pipe_code=self.code)
@@ -476,6 +478,7 @@ class PipeAbstract(ABC, BaseModel):
working_memory=working_memory,
pipe_run_params=pipe_run_params,
output_name=output_name,
+ library_crate=library_crate,
)
case PipeRunMode.DRY:
pipe_output = await self.dry_run_pipe(
@@ -483,6 +486,7 @@ class PipeAbstract(ABC, BaseModel):
working_memory=working_memory,
pipe_run_params=pipe_run_params,
output_name=output_name,
+ library_crate=library_crate,
)
await self.validate_after_run(
job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
@@ -494,7 +498,7 @@ class PipeAbstract(ABC, BaseModel):
if parent_graph_context.data_inclusion.error_stack_traces:
error_stack = traceback.format_exc()
tracer_manager.on_pipe_end_error(
- graph_id=parent_graph_context.graph_id,
+ lookup_key=parent_graph_context.lookup_key,
node_id=graph_node_id,
ended_at=datetime.now(timezone.utc),
error_type=type(exc).__name__,
@@ -526,7 +530,7 @@ class PipeAbstract(ABC, BaseModel):
output_concept_data = self._make_single_concept_data_for_registry(main_stuff.concept)
tracer_manager.on_pipe_end_success(
- graph_id=parent_graph_context.graph_id,
+ lookup_key=parent_graph_context.lookup_key,
node_id=graph_node_id,
ended_at=datetime.now(timezone.utc),
output_spec=output_spec,
@@ -543,6 +547,7 @@ class PipeAbstract(ABC, BaseModel):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
log.info(self._format_pipe_run_info(pipe_run_params=pipe_run_params))
@@ -589,7 +594,11 @@ class PipeAbstract(ABC, BaseModel):
try:
pipe_output = await self._live_run_pipe(
- job_metadata=child_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
+ job_metadata=child_metadata,
+ working_memory=working_memory,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ library_crate=library_crate,
)
except Exception as exc:
self._end_pipe_span_error(span, error=exc, is_root_span=is_root_span)
@@ -608,11 +617,16 @@ class PipeAbstract(ABC, BaseModel):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
log.verbose(f"Dry run of {self.type}: '{self.code}'")
assert pipe_run_params.run_mode.is_dry, f"Dry run of {self.type} '{self.code}' called with run_mode = {pipe_run_params.run_mode}"
return await self._dry_run_pipe(
- job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ library_crate=library_crate,
)
@abstractmethod
@@ -622,6 +636,7 @@ class PipeAbstract(ABC, BaseModel):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
pass
@@ -632,6 +647,7 @@ class PipeAbstract(ABC, BaseModel):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
pass
diff --git a/pipelex/core/pipes/pipe_assignment.py b/pipelex/core/pipes/pipe_assignment.py
deleted file mode 100644
index 54ea86466..000000000
--- a/pipelex/core/pipes/pipe_assignment.py
+++ /dev/null
@@ -1,25 +0,0 @@
-"""Base Assignment class for the pipe execution pre_run/run/post_run pattern.
-
-An Assignment carries resolved runtime data through the execution lifecycle:
-- Created in prepare_assignment() (pre_run phase): models resolved, prompts rendered
-- Consumed in execute() (run phase): pure execution using pre-resolved data
-- Finalized in finalize_assignment() (post_run phase): execution metadata captured
-
-Each pipe subclass defines its own Assignment type with operator-specific fields.
-"""
-
-from typing import Any
-
-from pydantic import BaseModel, ConfigDict, Field
-
-
-class PipeAssignment(BaseModel):
- """Base Assignment for pipe execution.
-
- Subclasses add operator-specific fields (rendered prompts, resolved models, etc.).
- The execution_data dict collects metadata for the GraphSpec's execution_data field.
- """
-
- model_config = ConfigDict(arbitrary_types_allowed=True)
-
- execution_data: dict[str, Any] = Field(default_factory=dict)
diff --git a/pipelex/core/pipes/pipe_factory.py b/pipelex/core/pipes/pipe_factory.py
index f95c94f1f..068fea479 100644
--- a/pipelex/core/pipes/pipe_factory.py
+++ b/pipelex/core/pipes/pipe_factory.py
@@ -1,7 +1,6 @@
from typing import TYPE_CHECKING, Any, Generic, Protocol, TypeVar
from kajson.exceptions import ClassRegistryInheritanceError, ClassRegistryNotFoundError
-from kajson.kajson_manager import KajsonManager
from typing_extensions import runtime_checkable
from pipelex.core.concepts.helpers import strip_multiplicity_from_concept_ref_or_code
@@ -12,6 +11,7 @@ from pipelex.core.pipes.inputs.input_stuff_specs_factory import InputStuffSpecsF
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint, PipeType
from pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec
from pipelex.core.pipes.stuff_spec.stuff_spec_factory import StuffSpecFactory, StuffSpecFactoryError
+from pipelex.hub import get_class_registry
if TYPE_CHECKING:
from pipelex.core.pipes.pipe_abstract import PipeAbstract
@@ -37,6 +37,10 @@ class PipeFactoryProtocol(Protocol[PipeBlueprintType, PipeAbstractType]):
class PipeFactory(Generic[PipeAbstractType]):
+ @classmethod
+ def make_pipe_ref_with_domain(cls, domain_code: str, pipe_code: str) -> str:
+ return f"{domain_code}.{pipe_code}"
+
@classmethod
def make_from_blueprint(
cls,
@@ -103,7 +107,7 @@ class PipeFactory(Generic[PipeAbstractType]):
# The factory class name for that specific type of Pipe is the pipe class name with "Factory" suffix
factory_class_name = f"{pipe_type.value}Factory"
try:
- pipe_factory: type[PipeFactoryProtocol[Any, Any]] = KajsonManager.get_class_registry().get_required_subclass(
+ pipe_factory: type[PipeFactoryProtocol[Any, Any]] = get_class_registry().get_required_subclass(
name=factory_class_name,
base_class=PipeFactoryProtocol,
)
diff --git a/pipelex/core/pipes/pipe_output.py b/pipelex/core/pipes/pipe_output.py
index bbad9d0b1..3c976feee 100644
--- a/pipelex/core/pipes/pipe_output.py
+++ b/pipelex/core/pipes/pipe_output.py
@@ -1,4 +1,4 @@
-from typing import TypeVar
+from typing import Any, TypeVar
from mthds.models.pipe_output import PipeOutputAbstract
from pydantic import Field
@@ -14,13 +14,39 @@ from pipelex.core.stuffs.stuff_content import StuffContentType
from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
from pipelex.core.stuffs.text_content import TextContent
from pipelex.graph.graphspec import GraphSpec
+from pipelex.libraries.library_crate import LibraryCrate
from pipelex.pipeline.pipeline_models import SpecialPipelineId
class PipeOutput(PipeOutputAbstract[WorkingMemory]):
working_memory: WorkingMemory = Field(default_factory=WorkingMemory)
+ working_memory_raw: dict[str, Any] | None = None
pipeline_run_id: str = Field(default=SpecialPipelineId.UNTITLED)
graph_spec: GraphSpec | None = None
+ graph_assembly_error: str | None = None
+
+ def prepare_for_temporal(self, library_crate: LibraryCrate | None = None) -> "PipeOutput":
+ """Dehydrate WorkingMemory to raw dict for Temporal transit.
+
+ Returns a copy with working_memory serialized to a plain dict
+ (no dynamic class metadata), leaving the original unchanged.
+ The receiving side must call hydrate_working_memory() to reconstruct
+ the typed WorkingMemory after dynamic classes are registered.
+
+ Symmetric with `PipeJob.prepare_for_temporal()`: when `library_crate`
+ is None, dehydration is a no-op — there are no dynamic concept classes
+ to round-trip, so the typed WorkingMemory can travel as-is.
+ """
+ if library_crate is None:
+ return self
+ if not self.working_memory.root:
+ return self
+ return self.model_copy(
+ update={
+ "working_memory_raw": self.working_memory.dump_for_temporal(),
+ "working_memory": WorkingMemory(),
+ }
+ )
@property
def main_stuff(self) -> Stuff:
diff --git a/pipelex/core/qualified_ref.py b/pipelex/core/qualified_ref.py
index ab6a3aa32..073275857 100644
--- a/pipelex/core/qualified_ref.py
+++ b/pipelex/core/qualified_ref.py
@@ -60,6 +60,17 @@ class QualifiedRef(BaseModel):
domain_path, local_code = raw.rsplit(".", maxsplit=1)
return cls(domain_path=domain_path, local_code=local_code)
+ @classmethod
+ def parse_stripping_cross_package(cls, raw: str) -> "QualifiedRef":
+ """Strip any cross-package prefix ('alias->...') then parse.
+
+ Use this when the caller may receive a cross-package reference but only
+ needs the domain + local_code part, not the alias.
+ """
+ if cls.has_cross_package_prefix(raw):
+ _, raw = cls.split_cross_package_ref(raw)
+ return cls.parse(raw)
+
@classmethod
def parse_concept_ref(cls, raw: str) -> "QualifiedRef":
"""Parse a concept ref. Validates domain_path segments are snake_case, local_code is PascalCase.
diff --git a/pipelex/core/stuffs/stuff.py b/pipelex/core/stuffs/stuff.py
index 9a1c81ec4..5856fbad6 100644
--- a/pipelex/core/stuffs/stuff.py
+++ b/pipelex/core/stuffs/stuff.py
@@ -103,12 +103,22 @@ class Stuff(PrettyRenderable, CustomBaseModel, StuffAbstract[Concept, StuffConte
return content
# If isinstance failed, try model validation approach
- # This handles cases where the same class is loaded from different import paths
+ # This handles cases where the same class is loaded from different import paths,
+ # or where a domain-qualified dynamic class (e.g., "domain__Invoice") matches
+ # a pre-existing class with the bare name (e.g., "Invoice")
try:
- # Check if class names match (quick filter before attempting validation)
- if type(content).__name__ == content_type.__name__:
- # Use model_dump() instead of smart_dump() to ensure we get a dict
- # smart_dump() may return a string for some content types (e.g., TextContent)
+ actual_name = type(content).__name__
+ expected_name = content_type.__name__
+ # Check if class names match — exact match or domain-qualified match.
+ # Domain-qualified names have the format "domain__ConceptCode", so we
+ # extract the concept code (after the last "__") for comparison.
+ # Only allow the fallback when at least one side is bare (no domain prefix)
+ # to prevent silent cross-domain conversion between e.g. alpha__Result and beta__Result.
+ actual_code = actual_name.rsplit("__", 1)[-1]
+ expected_code = expected_name.rsplit("__", 1)[-1]
+ at_least_one_bare = ("__" not in actual_name) or ("__" not in expected_name)
+ names_match = (actual_name == expected_name) or (at_least_one_bare and actual_code == expected_code)
+ if names_match:
content_dict = content.smart_dump()
validated_content = content_type.model_validate(content_dict)
log.verbose(f"Model validation passed: converted {type(content).__name__} to {content_type.__name__}")
diff --git a/pipelex/graph/graph_context.py b/pipelex/graph/graph_context.py
index 3521e1c88..84e1b80ab 100644
--- a/pipelex/graph/graph_context.py
+++ b/pipelex/graph/graph_context.py
@@ -26,10 +26,16 @@ class GraphContext(BaseModel):
model_config = ConfigDict(strict=True, extra="forbid")
graph_id: str = Field(description="Unique identifier for the execution graph")
+ tracer_key: str | None = Field(default=None, description="Lookup key in GraphTracerManager. Defaults to graph_id when None.")
parent_node_id: str | None = Field(default=None, description="Node ID of the parent pipe, None for root")
node_sequence: int = Field(default=0, description="Monotonic counter for generating node IDs")
data_inclusion: DataInclusionConfig = Field(description="Controls which data formats to capture")
+ @property
+ def lookup_key(self) -> str:
+ """Key for looking up the tracer in GraphTracerManager."""
+ return self.tracer_key or self.graph_id
+
def make_node_id(self) -> str:
"""Generate a unique node ID within this graph.
@@ -50,6 +56,7 @@ class GraphContext(BaseModel):
"""
return GraphContext(
graph_id=self.graph_id,
+ tracer_key=self.tracer_key,
parent_node_id=child_node_id,
node_sequence=next_sequence,
data_inclusion=self.data_inclusion,
diff --git a/pipelex/graph/graph_tracer.py b/pipelex/graph/graph_tracer.py
index df146b21e..bcca1e633 100644
--- a/pipelex/graph/graph_tracer.py
+++ b/pipelex/graph/graph_tracer.py
@@ -21,6 +21,19 @@ from pipelex.graph.graphspec import (
PipelineRef,
TimingSpec,
)
+from pipelex.tracing.event_log_protocol import EventLogProtocol # noqa: TC001 - used in __init__ annotations
+from pipelex.tracing.trace_events import (
+ BatchAggregateEvent,
+ BatchItemEvent,
+ ControllerOutputEvent,
+ EdgeEvent,
+ ExecutionDataEvent,
+ ParallelCombineEvent,
+ PipeEndErrorEvent,
+ PipeEndSuccessEvent,
+ PipeStartEvent,
+ TraceEvent,
+)
class _MutableNodeData:
@@ -116,6 +129,14 @@ class GraphTracer(GraphTracerProtocol):
# The branch_producer_node_id is snapshotted at registration time, before register_controller_output
# overrides _stuff_producer_map to point branch stuff codes to the controller node
self._parallel_combine_map: dict[str, tuple[str, list[tuple[str, str]]]] = {}
+ # Event log for distributed tracing (None = no event emission, direct mode)
+ self._event_log: EventLogProtocol | None = None
+ # "direct" for single-process mode, full Temporal workflow ID otherwise
+ self._workflow_id: str = "direct"
+ # Pipeline run ID for event emission
+ self._pipeline_run_id: str | None = None
+ # Per-writer monotonic counter for event sequencing
+ self._event_sequence: int = 0
# Registries for pipe and concept data (keyed by pipe_ref and concept_ref)
self._pipe_registry: dict[str, dict[str, Any]] = {}
self._concept_registry: dict[str, dict[str, Any]] = {}
@@ -125,6 +146,49 @@ class GraphTracer(GraphTracerProtocol):
"""Whether tracing is currently active."""
return self._is_active
+ @property
+ def _event_pipeline_run_id(self) -> str:
+ """Pipeline run ID for event emission, falling back to graph_id."""
+ return self._pipeline_run_id or self._graph_id or ""
+
+ def _emit_event(self, event: TraceEvent) -> None:
+ """Emit a trace event to the event log.
+
+ Must only be called when self._event_log is not None.
+ """
+ assert self._event_log is not None
+ self._event_log.emit(event)
+
+ def _next_event_sequence(self) -> int:
+ """Return the next monotonic sequence number for event emission.
+
+ Delegates to the event log's shared counter so that all emitters
+ (GraphTracer, ReportingManager) produce unique sequences.
+ """
+ if self._event_log is not None:
+ return self._event_log.next_sequence()
+ seq = self._event_sequence
+ self._event_sequence += 1
+ return seq
+
+ def _make_node_id(self) -> str:
+ """Generate a node ID, including workflow_id when in Temporal mode."""
+ if self._workflow_id != "direct":
+ node_id = f"{self._graph_id}:{self._workflow_id}:node_{self._node_sequence}"
+ else:
+ node_id = f"{self._graph_id}:node_{self._node_sequence}"
+ self._node_sequence += 1
+ return node_id
+
+ def _make_edge_id(self) -> str:
+ """Generate an edge ID, including workflow_id when in Temporal mode."""
+ if self._workflow_id != "direct":
+ edge_id = f"{self._graph_id}:{self._workflow_id}:edge_{self._edge_sequence}"
+ else:
+ edge_id = f"{self._graph_id}:edge_{self._edge_sequence}"
+ self._edge_sequence += 1
+ return edge_id
+
@override
def setup(
self,
@@ -132,8 +196,23 @@ class GraphTracer(GraphTracerProtocol):
data_inclusion: DataInclusionConfig,
pipeline_ref_domain: str | None = None,
pipeline_ref_main_pipe: str | None = None,
+ event_log: "EventLogProtocol | None" = None,
+ workflow_id: str = "direct",
+ pipeline_run_id: str | None = None,
) -> GraphContext:
- """Initialize tracing for a new pipeline run."""
+ """Initialize tracing for a new pipeline run.
+
+ Args:
+ graph_id: Unique identifier for this execution graph.
+ data_inclusion: Configuration controlling which data formats to capture.
+ pipeline_ref_domain: Optional domain name for the pipeline.
+ pipeline_ref_main_pipe: Optional main pipe name.
+ event_log: Optional event log for distributed tracing. When set,
+ every tracing method emits a TraceEvent as a side effect.
+ workflow_id: Temporal workflow ID or "direct" for single-process mode.
+ When not "direct", node/edge IDs include the workflow_id segment.
+ pipeline_run_id: Pipeline run ID for event emission. Required when event_log is set.
+ """
self._is_active = True
self._graph_id = graph_id
self._pipeline_ref = PipelineRef(
@@ -149,6 +228,10 @@ class GraphTracer(GraphTracerProtocol):
self._batch_item_map = {}
self._batch_aggregate_map = {}
self._parallel_combine_map = {}
+ self._event_log = event_log
+ self._workflow_id = workflow_id
+ self._pipeline_run_id = pipeline_run_id
+ self._event_sequence = 0
self._pipe_registry = {}
self._concept_registry = {}
@@ -203,6 +286,12 @@ class GraphTracer(GraphTracerProtocol):
self._batch_item_map = {}
self._batch_aggregate_map = {}
self._parallel_combine_map = {}
+ if self._event_log is not None:
+ self._event_log.close()
+ self._event_log = None
+ self._workflow_id = "direct"
+ self._pipeline_run_id = None
+ self._event_sequence = 0
self._pipe_registry = {}
self._concept_registry = {}
@@ -359,6 +448,21 @@ class GraphTracer(GraphTracerProtocol):
# Note: We keep the first batch_controller_node_id registered for this list
# (all items for the same input list should come from the same batch controller)
+ # Emit BatchItemEvent
+ if self._event_log is not None:
+ self._emit_event(
+ BatchItemEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ workflow_id=self._workflow_id,
+ timestamp=datetime.now(timezone.utc),
+ sequence=self._next_event_sequence(),
+ list_stuff_code=list_stuff_code,
+ item_stuff_code=item_stuff_code,
+ item_index=item_index,
+ batch_controller_node_id=batch_controller_node_id or "",
+ )
+ )
+
@override
def register_batch_aggregation(
self,
@@ -386,6 +490,21 @@ class GraphTracer(GraphTracerProtocol):
# Note: We keep the first batch_controller_node_id registered for this output list
# (all items for the same output list should come from the same batch controller)
+ # Emit BatchAggregateEvent
+ if self._event_log is not None:
+ self._emit_event(
+ BatchAggregateEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ workflow_id=self._workflow_id,
+ timestamp=datetime.now(timezone.utc),
+ sequence=self._next_event_sequence(),
+ output_list_stuff_code=output_list_stuff_code,
+ item_stuff_code=item_stuff_code,
+ item_index=item_index,
+ batch_controller_node_id=batch_controller_node_id or "",
+ )
+ )
+
@override
def register_parallel_combine(
self,
@@ -412,6 +531,21 @@ class GraphTracer(GraphTracerProtocol):
branch_entries.append((branch_code, producer_id))
self._parallel_combine_map[combined_stuff_code] = (parallel_controller_node_id, branch_entries)
+ # Emit ParallelCombineEvent with snapshotted producer IDs
+ if self._event_log is not None:
+ self._emit_event(
+ ParallelCombineEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ workflow_id=self._workflow_id,
+ timestamp=datetime.now(timezone.utc),
+ sequence=self._next_event_sequence(),
+ combined_stuff_code=combined_stuff_code,
+ branch_stuff_codes=branch_stuff_codes,
+ parallel_controller_node_id=parallel_controller_node_id,
+ branch_producer_node_ids=branch_entries,
+ )
+ )
+
@override
def on_pipe_start(
self,
@@ -431,9 +565,8 @@ class GraphTracer(GraphTracerProtocol):
child_context = graph_context.copy_for_child(node_id, graph_context.node_sequence + 1)
return node_id, child_context
- # Generate node ID
- node_id = f"{self._graph_id}:node_{self._node_sequence}"
- self._node_sequence += 1
+ # Generate node ID (includes workflow_id in Temporal mode)
+ node_id = self._make_node_id()
# Create mutable node data
node_data = _MutableNodeData(
@@ -459,6 +592,25 @@ class GraphTracer(GraphTracerProtocol):
if concept_ref not in self._concept_registry:
self._concept_registry[concept_ref] = concept_item
+ # Emit PipeStartEvent (after registry accumulation so fields are populated)
+ if self._event_log is not None:
+ self._emit_event(
+ PipeStartEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ workflow_id=self._workflow_id,
+ timestamp=started_at,
+ sequence=self._next_event_sequence(),
+ node_id=node_id,
+ parent_node_id=graph_context.parent_node_id,
+ pipe_code=pipe_code,
+ pipe_type=pipe_type,
+ node_kind=node_kind,
+ input_specs=input_specs or [],
+ pipe_data=pipe_data or {},
+ concept_data=concept_data or [],
+ )
+ )
+
# Add containment edge from parent if this is a child pipe
if graph_context.parent_node_id is not None:
self.add_edge(
@@ -489,6 +641,19 @@ class GraphTracer(GraphTracerProtocol):
return
node_data.execution_data.update(execution_data)
+ # Emit ExecutionDataEvent
+ if self._event_log is not None:
+ self._emit_event(
+ ExecutionDataEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ workflow_id=self._workflow_id,
+ timestamp=datetime.now(timezone.utc),
+ sequence=self._next_event_sequence(),
+ node_id=node_id,
+ execution_data=execution_data,
+ )
+ )
+
@override
def on_pipe_end_success(
self,
@@ -534,6 +699,22 @@ class GraphTracer(GraphTracerProtocol):
if output_spec.digest:
self._stuff_producer_map[output_spec.digest] = node_id
+ # Emit PipeEndSuccessEvent
+ if self._event_log is not None:
+ self._emit_event(
+ PipeEndSuccessEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ workflow_id=self._workflow_id,
+ timestamp=ended_at,
+ sequence=self._next_event_sequence(),
+ node_id=node_id,
+ ended_at=ended_at,
+ output_spec=output_spec,
+ metrics=metrics or {},
+ output_concept_data=output_concept_data or {},
+ )
+ )
+
@override
def register_controller_output(
self,
@@ -561,6 +742,19 @@ class GraphTracer(GraphTracerProtocol):
if output_spec.digest:
self._stuff_producer_map[output_spec.digest] = node_id
+ # Emit ControllerOutputEvent
+ if self._event_log is not None:
+ self._emit_event(
+ ControllerOutputEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ workflow_id=self._workflow_id,
+ timestamp=datetime.now(timezone.utc),
+ sequence=self._next_event_sequence(),
+ node_id=node_id,
+ output_spec=output_spec,
+ )
+ )
+
@override
def on_pipe_end_error(
self,
@@ -580,11 +774,26 @@ class GraphTracer(GraphTracerProtocol):
node_data.ended_at = ended_at
node_data.status = NodeStatus.FAILED
- node_data.error = ErrorSpec(
+ error = ErrorSpec(
error_type=error_type,
message=error_message,
stack=error_stack,
)
+ node_data.error = error
+
+ # Emit PipeEndErrorEvent
+ if self._event_log is not None:
+ self._emit_event(
+ PipeEndErrorEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ workflow_id=self._workflow_id,
+ timestamp=ended_at,
+ sequence=self._next_event_sequence(),
+ node_id=node_id,
+ ended_at=ended_at,
+ error=error,
+ )
+ )
@override
def add_edge(
@@ -600,8 +809,7 @@ class GraphTracer(GraphTracerProtocol):
if not self._is_active:
return
- edge_id = f"{self._graph_id}:edge_{self._edge_sequence}"
- self._edge_sequence += 1
+ edge_id = self._make_edge_id()
edge = EdgeSpec(
edge_id=edge_id,
@@ -614,6 +822,24 @@ class GraphTracer(GraphTracerProtocol):
)
self._edges.append(edge)
+ # Emit EdgeEvent
+ if self._event_log is not None:
+ self._emit_event(
+ EdgeEvent(
+ pipeline_run_id=self._event_pipeline_run_id,
+ workflow_id=self._workflow_id,
+ timestamp=datetime.now(timezone.utc),
+ sequence=self._next_event_sequence(),
+ edge_id=edge_id,
+ source_node_id=source_node_id,
+ target_node_id=target_node_id,
+ edge_kind=edge_kind,
+ label=label,
+ source_stuff_digest=source_stuff_digest,
+ target_stuff_digest=target_stuff_digest,
+ )
+ )
+
def add_selected_outcome_edge(
self,
condition_node_id: str,
diff --git a/pipelex/graph/graph_tracer_manager.py b/pipelex/graph/graph_tracer_manager.py
index 3f24ba9da..4ea17e76b 100644
--- a/pipelex/graph/graph_tracer_manager.py
+++ b/pipelex/graph/graph_tracer_manager.py
@@ -9,6 +9,7 @@ from pipelex.graph.graph_tracer import GraphTracer
from pipelex.graph.graph_tracer_protocol import GraphTracerProtocol
from pipelex.graph.graphspec import EdgeKind, GraphSpec, IOSpec, NodeKind
from pipelex.system.registries.singleton import ABCSingletonMeta, MetaSingleton
+from pipelex.tracing.event_log_protocol import EventLogProtocol # noqa: TC001 - used in open_tracer signature
class GraphTracerManager(metaclass=ABCSingletonMeta):
@@ -53,22 +54,22 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
return instance
@classmethod
- def get_instance_tracer(cls, graph_id: str) -> GraphTracerProtocol | None:
- """Get the graph tracer for a specific graph_id from the singleton instance.
+ def get_instance_tracer(cls, lookup_key: str) -> GraphTracerProtocol | None:
+ """Get the graph tracer for a specific lookup key from the singleton instance.
This provides a way to access the tracer without importing from hub,
avoiding circular dependency issues.
Args:
- graph_id: The graph/pipeline run identifier.
+ lookup_key: The tracer lookup key (graph_id or tracer_key).
Returns:
- The tracer for the given graph_id, or None if not found.
+ The tracer for the given key, or None if not found.
"""
instance = cls.get_instance()
if instance is None:
return None
- return instance.get_tracer(graph_id)
+ return instance.get_tracer(lookup_key)
############################################################
# Private helpers
@@ -95,45 +96,64 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
data_inclusion: DataInclusionConfig,
pipeline_ref_domain: str | None = None,
pipeline_ref_main_pipe: str | None = None,
+ event_log: "EventLogProtocol | None" = None,
+ workflow_id: str = "direct",
+ pipeline_run_id: str | None = None,
+ tracer_key: str | None = None,
) -> GraphContext:
"""Create and initialize a new tracer for a pipeline run.
Args:
- graph_id: Unique identifier for this pipeline run.
+ graph_id: Unique identifier for this pipeline run (used in node ID generation).
data_inclusion: Configuration controlling which data formats to capture in IOSpec fields.
pipeline_ref_domain: Optional domain name for the pipeline.
pipeline_ref_main_pipe: Optional main pipe name.
+ event_log: Optional event log for distributed tracing. When set,
+ the tracer emits trace events as a side effect alongside in-memory accumulation.
+ workflow_id: Temporal workflow ID or "direct" for single-process mode.
+ pipeline_run_id: Pipeline run ID for event emission.
+ tracer_key: Lookup key for the tracer in the manager's dict. Defaults to graph_id.
+ In Temporal mode, use the workflow_id to avoid collisions when multiple
+ workflows share the same graph_id on the same process.
Returns:
Initial GraphContext to pass through JobMetadata.
Raises:
- ValueError: If a tracer for this graph_id already exists.
+ ValueError: If a tracer for this key already exists.
"""
- if graph_id in self._tracers:
- msg = f"Tracer for graph '{graph_id}' already exists"
+ key = tracer_key or graph_id
+ if key in self._tracers:
+ msg = f"Tracer for key '{key}' already exists"
raise ValueError(msg)
tracer = GraphTracer()
- self._tracers[graph_id] = tracer
+ self._tracers[key] = tracer
- return tracer.setup(
+ graph_context = tracer.setup(
graph_id=graph_id,
data_inclusion=data_inclusion,
pipeline_ref_domain=pipeline_ref_domain,
pipeline_ref_main_pipe=pipeline_ref_main_pipe,
+ event_log=event_log,
+ workflow_id=workflow_id,
+ pipeline_run_id=pipeline_run_id,
)
+ # Set the tracer_key on the GraphContext so downstream lookups use the same key
+ if tracer_key is not None:
+ graph_context = graph_context.model_copy(update={"tracer_key": tracer_key})
+ return graph_context
- def close_tracer(self, graph_id: str) -> GraphSpec | None:
+ def close_tracer(self, tracer_key: str) -> GraphSpec | None:
"""Finalize tracing for a specific pipeline run and return its GraphSpec.
Args:
- graph_id: The graph/pipeline run identifier.
+ tracer_key: The tracer lookup key (graph_id or workflow_id).
Returns:
- The completed GraphSpec, or None if no tracer found for this graph_id.
+ The completed GraphSpec, or None if no tracer found for this key.
"""
- tracer = self._tracers.pop(graph_id, None)
+ tracer = self._tracers.pop(tracer_key, None)
if tracer is None:
return None
return tracer.teardown()
@@ -194,7 +214,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
Returns:
Tuple of (node_id, child_graph_context) if tracing is active, (None, None) otherwise.
"""
- tracer = self._get_tracer(graph_context.graph_id)
+ tracer = self._get_tracer(graph_context.lookup_key)
if tracer is None:
return None, None
@@ -211,7 +231,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
def on_pipe_end_success(
self,
- graph_id: str,
+ lookup_key: str,
node_id: str | None,
ended_at: datetime,
output_preview: str | None = None,
@@ -222,7 +242,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
"""Record successful completion of a pipe execution.
Args:
- graph_id: The graph identifier.
+ lookup_key: The tracer lookup key.
node_id: The node ID returned from on_pipe_start.
ended_at: When the pipe finished executing.
output_preview: Optional truncated preview of the output.
@@ -233,7 +253,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
if node_id is None:
return
- tracer = self._get_tracer(graph_id)
+ tracer = self._get_tracer(lookup_key)
if tracer is None:
return
@@ -248,27 +268,27 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
def register_execution_data(
self,
- graph_id: str,
+ lookup_key: str,
node_id: str | None,
execution_data: dict[str, Any],
) -> None:
"""Register execution metadata for a node.
Args:
- graph_id: The graph identifier.
+ lookup_key: The tracer lookup key (graph_id or workflow_id).
node_id: The node ID to attach execution data to.
execution_data: Dictionary of execution metadata.
"""
if node_id is None:
return
- tracer = self._get_tracer(graph_id)
+ tracer = self._get_tracer(lookup_key)
if tracer is None:
return
tracer.register_execution_data(node_id=node_id, execution_data=execution_data)
def on_pipe_end_error(
self,
- graph_id: str,
+ lookup_key: str,
node_id: str | None,
ended_at: datetime,
error_type: str,
@@ -278,7 +298,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
"""Record failed completion of a pipe execution.
Args:
- graph_id: The graph identifier.
+ lookup_key: The tracer lookup key.
node_id: The node ID returned from on_pipe_start.
ended_at: When the pipe failed.
error_type: The exception type name.
@@ -288,7 +308,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
if node_id is None:
return
- tracer = self._get_tracer(graph_id)
+ tracer = self._get_tracer(lookup_key)
if tracer is None:
return
@@ -302,7 +322,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
def add_edge(
self,
- graph_id: str,
+ lookup_key: str,
source_node_id: str,
target_node_id: str,
edge_kind: EdgeKind,
@@ -311,13 +331,13 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
"""Add an edge between two nodes.
Args:
- graph_id: The graph identifier.
+ lookup_key: The tracer lookup key.
source_node_id: The source node ID.
target_node_id: The target node ID.
edge_kind: The type of edge.
label: Optional label for the edge.
"""
- tracer = self._get_tracer(graph_id)
+ tracer = self._get_tracer(lookup_key)
if tracer is None:
return
@@ -330,18 +350,18 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
def register_controller_output(
self,
- graph_id: str,
+ lookup_key: str,
node_id: str,
output_spec: IOSpec,
) -> None:
"""Register an additional output for a controller node.
Args:
- graph_id: The graph identifier.
+ lookup_key: The tracer lookup key.
node_id: The controller node ID.
output_spec: The IOSpec describing the output.
"""
- tracer = self._get_tracer(graph_id)
+ tracer = self._get_tracer(lookup_key)
if tracer is None:
return
tracer.register_controller_output(
@@ -351,7 +371,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
def register_batch_item_extraction(
self,
- graph_id: str,
+ lookup_key: str,
list_stuff_code: str,
item_stuff_code: str,
item_index: int,
@@ -360,14 +380,14 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
"""Register that a list stuff produced an item stuff during batch iteration.
Args:
- graph_id: The graph identifier.
+ lookup_key: The tracer lookup key.
list_stuff_code: The stuff_code of the input list.
item_stuff_code: The stuff_code of the extracted item.
item_index: The index of the item in the list.
batch_controller_node_id: The node_id of the PipeBatch controller performing the fan-out.
If provided, this will be used as the source node for BATCH_ITEM edges in controller-centric mode.
"""
- tracer = self._get_tracer(graph_id)
+ tracer = self._get_tracer(lookup_key)
if tracer is None:
return
tracer.register_batch_item_extraction(
@@ -379,7 +399,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
def register_batch_aggregation(
self,
- graph_id: str,
+ lookup_key: str,
output_list_stuff_code: str,
item_stuff_code: str,
item_index: int,
@@ -388,14 +408,14 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
"""Register that an item stuff will be aggregated into an output list.
Args:
- graph_id: The graph identifier.
+ lookup_key: The tracer lookup key.
output_list_stuff_code: The stuff_code of the output list.
item_stuff_code: The stuff_code of the item to aggregate.
item_index: The index of the item in the output list.
batch_controller_node_id: The node_id of the PipeBatch controller that will produce the output list.
If provided, this will be used as the target node for BATCH_AGGREGATE edges.
"""
- tracer = self._get_tracer(graph_id)
+ tracer = self._get_tracer(lookup_key)
if tracer is None:
return
tracer.register_batch_aggregation(
@@ -407,7 +427,7 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
def register_parallel_combine(
self,
- graph_id: str,
+ lookup_key: str,
combined_stuff_code: str,
branch_stuff_codes: list[str],
parallel_controller_node_id: str,
@@ -415,12 +435,12 @@ class GraphTracerManager(metaclass=ABCSingletonMeta):
"""Register that branch outputs are combined into a single output in PipeParallel.
Args:
- graph_id: The graph identifier.
+ lookup_key: The tracer lookup key.
combined_stuff_code: The stuff_code of the combined output.
branch_stuff_codes: The stuff_codes of the individual branch outputs.
parallel_controller_node_id: The node_id of the PipeParallel controller.
"""
- tracer = self._get_tracer(graph_id)
+ tracer = self._get_tracer(lookup_key)
if tracer is None:
return
tracer.register_parallel_combine(
diff --git a/pipelex/graph/graph_tracer_protocol.py b/pipelex/graph/graph_tracer_protocol.py
index 4a8b70d1c..215e17262 100644
--- a/pipelex/graph/graph_tracer_protocol.py
+++ b/pipelex/graph/graph_tracer_protocol.py
@@ -6,6 +6,7 @@ from typing_extensions import override
from pipelex.graph.graph_config import DataInclusionConfig
from pipelex.graph.graph_context import GraphContext
from pipelex.graph.graphspec import EdgeKind, GraphSpec, IOSpec, NodeKind
+from pipelex.tracing.event_log_protocol import EventLogProtocol # noqa: TC001 - used in setup signature
class GraphTracerProtocol(Protocol):
@@ -21,6 +22,9 @@ class GraphTracerProtocol(Protocol):
data_inclusion: DataInclusionConfig,
pipeline_ref_domain: str | None = None,
pipeline_ref_main_pipe: str | None = None,
+ event_log: "EventLogProtocol | None" = None,
+ workflow_id: str = "direct",
+ pipeline_run_id: str | None = None,
) -> GraphContext:
"""Initialize tracing for a new pipeline run.
@@ -29,6 +33,9 @@ class GraphTracerProtocol(Protocol):
data_inclusion: Configuration controlling which data formats to capture in IOSpec fields.
pipeline_ref_domain: Optional domain name for the pipeline.
pipeline_ref_main_pipe: Optional main pipe name.
+ event_log: Optional event log for distributed tracing.
+ workflow_id: Temporal workflow ID or "direct" for single-process mode.
+ pipeline_run_id: Pipeline run ID for event emission.
Returns:
Initial GraphContext to pass through JobMetadata.
@@ -230,6 +237,9 @@ class GraphTracerNoOp(GraphTracerProtocol):
data_inclusion: DataInclusionConfig,
pipeline_ref_domain: str | None = None,
pipeline_ref_main_pipe: str | None = None,
+ event_log: "EventLogProtocol | None" = None,
+ workflow_id: str = "direct",
+ pipeline_run_id: str | None = None,
) -> GraphContext:
return GraphContext(
graph_id=graph_id,
diff --git a/pipelex/graph/graphspec.py b/pipelex/graph/graphspec.py
index d471a9d4a..3cbb02cdc 100644
--- a/pipelex/graph/graphspec.py
+++ b/pipelex/graph/graphspec.py
@@ -16,6 +16,10 @@ from pipelex.types import Self, StrEnum
MAX_PREVIEW_LENGTH = 200
MAX_STACK_LENGTH = 2000
+# Format tag stored in GraphSpec.meta so external tooling (e.g. the VS Code
+# graph viewer) can recognize a JSON file as a Pipelex execution graph.
+GRAPHSPEC_FORMAT = "mthds"
+
class NodeKind(StrEnum):
"""Types of nodes in the execution graph."""
@@ -231,10 +235,10 @@ class NodeSpec(BaseModel):
model_config = ConfigDict(extra="forbid", strict=True, populate_by_name=True)
node_id: str = Field(validation_alias="id", serialization_alias="id")
- kind: NodeKind
+ kind: NodeKind = Field(strict=False)
pipe_code: str | None = None
pipe_type: str | None = None
- status: NodeStatus
+ status: NodeStatus = Field(strict=False)
timing: TimingSpec | None = None
node_io: NodeIOSpec = Field(
default_factory=NodeIOSpec,
@@ -255,7 +259,7 @@ class EdgeSpec(BaseModel):
edge_id: str = Field(validation_alias="id", serialization_alias="id")
source: str
target: str
- kind: EdgeKind
+ kind: EdgeKind = Field(strict=False)
label: str | None = None
# For batch edges, specify the stuff digests for renderers to connect stuff nodes directly
source_stuff_digest: str | None = None
@@ -281,5 +285,15 @@ class GraphSpec(BaseModel):
pipe_registry: dict[str, dict[str, Any]] = Field(default_factory=dict)
concept_registry: dict[str, dict[str, Any]] = Field(default_factory=dict)
+ @model_validator(mode="after")
+ def ensure_format_meta(self) -> Self:
+ existing_format = self.meta.get("format")
+ if existing_format is None:
+ self.meta["format"] = GRAPHSPEC_FORMAT
+ elif existing_format != GRAPHSPEC_FORMAT:
+ msg = f"GraphSpec.meta['format'] must be '{GRAPHSPEC_FORMAT}', got '{existing_format}'"
+ raise ValueError(msg)
+ return self
+
def to_json(self) -> str:
return self.model_dump_json(serialize_as_any=True, by_alias=True, indent=2)
diff --git a/pipelex/graph/reactflow/standalone_assets.py b/pipelex/graph/reactflow/standalone_assets.py
index cac1f1baf..b0f9dd50a 100644
--- a/pipelex/graph/reactflow/standalone_assets.py
+++ b/pipelex/graph/reactflow/standalone_assets.py
@@ -5,12 +5,13 @@ in Jinja2 templates via the standard template rendering pipeline.
"""
import importlib.resources
-from functools import cache
+from functools import lru_cache
_ASSET_PACKAGE = "pipelex.graph.reactflow.assets"
-@cache
+# @lru_cache(maxsize=1) memoizes the no-arg call: file is read once, then served from cache.
+@lru_cache(maxsize=1)
def get_standalone_js() -> str:
"""Load the pre-built GraphViewer JS bundle (IIFE).
@@ -21,7 +22,7 @@ def get_standalone_js() -> str:
return (package_files / "graph-viewer.js").read_text(encoding="utf-8")
-@cache
+@lru_cache(maxsize=1)
def get_standalone_css() -> str:
"""Load the pre-built GraphViewer CSS bundle.
diff --git a/pipelex/hub.py b/pipelex/hub.py
index 7ada0b9c0..31817d38b 100644
--- a/pipelex/hub.py
+++ b/pipelex/hub.py
@@ -2,10 +2,10 @@ import sys
from collections.abc import Sequence
from contextvars import ContextVar
from pathlib import Path
-from typing import ClassVar, Optional
+from typing import TYPE_CHECKING, Any, ClassVar, Optional
from kajson.class_registry_abstract import ClassRegistryAbstract
-from opentelemetry.trace import Tracer as OTelTracer
+from kajson.kajson_manager import KajsonManager
from rich.console import Console
from pipelex import log
@@ -38,10 +38,16 @@ from pipelex.system.configuration.config_root import ConfigRoot
from pipelex.system.console_target import ConsoleTarget
from pipelex.system.environment import PIPELEXPATH_ENV_KEY, get_pipelexpath_dirs
from pipelex.system.registries.func_registry import FuncRegistry
-from pipelex.system.telemetry.telemetry_manager import TelemetryManagerAbstract
+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract
from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
+if TYPE_CHECKING:
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from opentelemetry.trace import Tracer as OTelTracer
+
+ from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
+
class PipelexHub:
"""PipelexHub serves as a central dependency manager to break cyclic imports between components.
@@ -74,6 +80,7 @@ class PipelexHub:
self._concept_library: ConceptLibraryAbstract | None = None
self._pipe_library: PipeLibraryAbstract | None = None
self._pipe_router: PipeRouterProtocol | None = None
+ self._pipe_run: PipeRunProtocol | None = None
# pipeline
self._pipeline_manager: PipelineManagerAbstract | None = None
@@ -104,13 +111,16 @@ class PipelexHub:
# tools
- def setup_config(self, config_cls: type[ConfigRoot]):
+ def setup_config(self, config_cls: type[ConfigRoot], config_overrides: dict[str, Any] | None = None):
"""Set the global configuration instance.
- # Args:
- # config (Config): The configuration instance to set.
+ Args:
+ config_cls: The config root class to validate against.
+ config_overrides: Optional dict deep-merged on top of all TOML layers
+ as the highest-priority override. Useful for tests that need
+ specific config without editing TOML files.
"""
- config_dict = config_manager.load_config()
+ config_dict = config_manager.load_config(extra_overrides=config_overrides)
self.set_config(config=config_cls.model_validate(config_dict))
def set_config(self, config: ConfigRoot):
@@ -180,6 +190,9 @@ class PipelexHub:
def set_pipe_router(self, pipe_router: PipeRouterProtocol):
self._pipe_router = pipe_router
+ def set_pipe_run(self, pipe_run: "PipeRunProtocol") -> None:
+ self._pipe_run = pipe_run
+
def set_pipeline_manager(self, pipeline_manager: PipelineManagerAbstract):
self._pipeline_manager = pipeline_manager
@@ -297,6 +310,12 @@ class PipelexHub:
raise RuntimeError(msg)
return self._pipe_router
+ def get_required_pipe_run(self) -> "PipeRunProtocol":
+ if self._pipe_run is None:
+ msg = "PipeRun is not initialized"
+ raise RuntimeError(msg)
+ return self._pipe_run
+
def get_required_pipeline_manager(self) -> PipelineManagerAbstract:
if self._pipeline_manager is None:
msg = "PipelineManager is not initialized"
@@ -363,7 +382,17 @@ def get_storage_provider() -> StorageProviderAbstract:
def get_class_registry() -> ClassRegistryAbstract:
- return get_pipelex_hub().get_required_class_registry()
+ """Return the active class registry, respecting per-workflow library scoping.
+
+ When a library_id is set in the current async context (e.g. inside a Temporal workflow),
+ returns the library's scoped ClassRegistry. Otherwise, returns the global registry.
+ """
+ library_id = _library_id.get()
+ if library_id is not None:
+ registry = get_library_manager().get_library_class_registry(library_id)
+ if registry is not None:
+ return registry
+ return KajsonManager.get_class_registry()
def get_func_registry() -> FuncRegistry:
@@ -374,7 +403,7 @@ def get_telemetry_manager() -> TelemetryManagerAbstract:
return get_pipelex_hub().get_telemetry_manager()
-def get_otel_tracer() -> OTelTracer | None:
+def get_otel_tracer() -> "OTelTracer | None":
return get_telemetry_manager().get_otel_tracer()
@@ -540,6 +569,10 @@ def get_pipe_router() -> PipeRouterProtocol:
return get_pipelex_hub().get_required_pipe_router()
+def get_pipe_run() -> "PipeRunProtocol":
+ return get_pipelex_hub().get_required_pipe_run()
+
+
def get_pipeline_manager() -> PipelineManagerAbstract:
return get_pipelex_hub().get_required_pipeline_manager()
diff --git a/pipelex/kit/agent_rules/commands.md b/pipelex/kit/agent_rules/commands.md
index 9f042b7a1..18a229ed4 100644
--- a/pipelex/kit/agent_rules/commands.md
+++ b/pipelex/kit/agent_rules/commands.md
@@ -28,9 +28,9 @@
## Running Tests
- After making code changes, run **targeted tests** based on what you changed. See `tests/CLAUDE.md` for the source-to-test mapping and the pytest command template.
+ `make agent-test` runs the full test suite and is **critical at the end of a coding session** to verify everything is good before wrapping up.
- Fall back to the full suite when changes are broad (3+ areas), touch shared infrastructure, or when preparing a release/push/commit to remote:
+ At intermediate steps during LOCAL development, it's OK to run only the tests relevant to your changes — either by calling pytest directly from the `.venv` (e.g. `.venv/bin/pytest -x -q tests/unit/path/to/test_module.py`) or using `make t TEST=TestClassName`. This applies only to local setups, not cloud agents.
```bash
make agent-test
@@ -74,6 +74,28 @@
```
Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.
+## Temporal Integration Test Options
+
+ The Temporal integration tests support different server modes via the `--temporal-server` pytest CLI option:
+
+ - `--temporal-server`: Which Temporal server to use
+ - `none` (default): in-process test server — no external dependencies, used in CI
+ - `time-skipping`: in-process server with deterministic time control
+ - A profile name from `temporal_server_configs` in `pipelex.toml` (e.g. `local`, `testing`): connects to a real Temporal server using the profile's host, namespace, and API key settings
+
+ ```bash
+ # CI default: in-process server
+ .venv/bin/pytest tests/integration/pipelex/temporal/
+
+ # Dev with local Temporal server
+ .venv/bin/pytest tests/integration/pipelex/temporal/ \
+ --temporal-server local
+
+ # Dev with cloud/testing server
+ .venv/bin/pytest tests/integration/pipelex/temporal/ \
+ --temporal-server testing
+ ```
+
---
## Prerequisites for running command lines: use virtual environment
diff --git a/pipelex/kit/agent_rules/python_standards.md b/pipelex/kit/agent_rules/python_standards.md
index 70716420f..73ce518c6 100644
--- a/pipelex/kit/agent_rules/python_standards.md
+++ b/pipelex/kit/agent_rules/python_standards.md
@@ -107,12 +107,28 @@ This document outlines the core coding standards, best practices, and quality co
- Use factory pattern for object creation when dealing with multiple implementations
- Our factory methods are named `make_from_...` and such
+## Protocols and Interfaces
+
+ - When a getter (e.g. `get_report_delegate()`, `get_library_manager()`) returns a `Protocol` type, callers must only rely on methods declared on that `Protocol`. If you need to call a method that lives on a concrete implementation, **extend the `Protocol`** — do not work around it with `isinstance(...)` checks, `cast(...)`, or inline imports of the concrete class.
+ - When extending the `Protocol`, also update every implementation (including no-op / null implementations) so structural typing is satisfied. For implementations where the method has nothing to do, give it a `pass` body — that is the whole point of having a `Protocol` rather than a concrete base class.
+ - Smell to watch for: an inline `from ... import ConcreteImpl # noqa: PLC0415` followed by `if isinstance(delegate, ConcreteImpl): delegate.some_method(...)`. This is almost always a missing method on the `Protocol`. Promote it.
+ - Use `@override` (from `typing_extensions`) on every implementation of a `Protocol` method, so that signature drift between the `Protocol` and its implementations is caught by the type checker.
+
## Error Handling
- Always catch exceptions at the place where you can add useful context to it.
- Use try/except blocks with specific exceptions
- Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError
- - NEVER catch the generic Exception, only catch specific exceptions, except at the root of CLI commands
+ - **NEVER write `except Exception:` (or bare `except:`).** Catching the generic `Exception` is FORBIDDEN everywhere except at the absolute root of a CLI command or an API endpoint handler. There are NO other exceptions to this rule. The following are NOT valid justifications and the agent must reject them when tempted:
+ - "I want to log and re-raise" → just let it propagate; logging happens at the root.
+ - "I want to swallow it just in case" → don't. Unknown failures must surface.
+ - "I want to convert it to our error type" → only if you know which specific exception(s) the call can raise; catch those explicitly.
+ - "The third-party lib isn't well typed" → look up the actual exceptions it raises and catch those by name.
+ - "It's safer this way" → it isn't. It hides bugs. Crashing loudly on an unexpected exception is the desired behavior.
+ If you find yourself writing `except Exception`, STOP and ask the user before proceeding.
+ - **Do NOT add try/except speculatively.** Only add a `try`/`except` when (a) a specific exception type is documented or known to be raised by the call, AND (b) you have a concrete recovery path or a meaningful context to attach. Otherwise, let the error bubble up — that is the correct behavior, not a gap to fill.
+ - **Use `try`/`finally` (without `except`) for required cleanup.** When a piece of code MUST run on the way out — releasing a resource, closing a tracer, tearing down a library, clearing per-context state on a manager — put it in a `finally` block (or a `with`/context manager). This is the correct way to guarantee cleanup on both the happy path and the error path: it does NOT suppress the exception, it just ensures cleanup happens before the exception propagates. Do not invent an `except Exception` just to attach cleanup; use `finally` instead. Conversely, do not move cleanup into the success path "to keep things simple" — if it must run on errors too, it belongs in `finally`.
+ - When reviewing existing code, treat any `except Exception` you encounter as a bug to fix (replace it with the specific exception(s) actually raised, or remove the try/except entirely), not as a precedent to follow.
- Always add `from exc` to the exception raise statements
- Always write the error message as a variable before raising it, for cleaner error traces
diff --git a/pipelex/kit/configs/inference/backends/google.toml b/pipelex/kit/configs/inference/backends/google.toml
index e4fa75131..cad27a0cb 100644
--- a/pipelex/kit/configs/inference/backends/google.toml
+++ b/pipelex/kit/configs/inference/backends/google.toml
@@ -129,7 +129,6 @@ outputs = ["image"]
thinking_mode = "none"
costs = { input = 2.0, output = 120 }
-
[nano-banana-2]
model_type = "img_gen"
model_id = "gemini-3.1-flash-image-preview"
diff --git a/pipelex/kit/configs/inference/backends/pipelex_gateway_models.md b/pipelex/kit/configs/inference/backends/pipelex_gateway_models.md
index d497327a9..ddb1b79f8 100644
--- a/pipelex/kit/configs/inference/backends/pipelex_gateway_models.md
+++ b/pipelex/kit/configs/inference/backends/pipelex_gateway_models.md
@@ -312,7 +312,7 @@ For configuration details, see the [documentation](https://docs.pipelex.com/late
<td>gpt-5.4</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
-<td style="text-align:center;background-color:rgba(33,150,243,0.15)">❌</td>
+<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
</tr>
@@ -320,7 +320,7 @@ For configuration details, see the [documentation](https://docs.pipelex.com/late
<td>gpt-5.4-mini</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
-<td style="text-align:center;background-color:rgba(33,150,243,0.15)">❌</td>
+<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
</tr>
@@ -328,7 +328,7 @@ For configuration details, see the [documentation](https://docs.pipelex.com/late
<td>gpt-5.4-nano</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
-<td style="text-align:center;background-color:rgba(33,150,243,0.15)">❌</td>
+<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
</tr>
@@ -336,7 +336,7 @@ For configuration details, see the [documentation](https://docs.pipelex.com/late
<td>gpt-5.4-pro</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
-<td style="text-align:center;background-color:rgba(33,150,243,0.15)">❌</td>
+<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
</tr>
@@ -344,7 +344,7 @@ For configuration details, see the [documentation](https://docs.pipelex.com/late
<td>gpt-5.5</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
-<td style="text-align:center;background-color:rgba(33,150,243,0.15)">❌</td>
+<td style="text-align:center;background-color:rgba(33,150,243,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
<td style="text-align:center;background-color:rgba(76,175,80,0.15)">✅</td>
</tr>
@@ -598,6 +598,6 @@ For configuration details, see the [documentation](https://docs.pipelex.com/late
> **AUTO-GENERATED FILE** - Do not edit manually.
-> Last updated: 2026-04-28T11:06:01Z
+> Last updated: 2026-04-28T12:50:14Z
>
> Run `pipelex-dev update-gateway-models` or `make ugm` to regenerate.
\ No newline at end of file
diff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.toml
index 719a461f3..5cdd79efd 100644
--- a/pipelex/kit/configs/pipelex.toml
+++ b/pipelex/kit/configs/pipelex.toml
@@ -63,17 +63,36 @@ pan_to_top = true # Pan to show top of graph on load
theme = "dark"
palette = "dracula"
+####################################################################################################
+# Tracing Config (event log for distributed tracing across Temporal workers)
+####################################################################################################
+
+[pipelex.tracing_config]
+is_enabled = true
+backend = "ndjson"
+
+[pipelex.tracing_config.ndjson]
+traces_dir = ".pipelex/traces"
+
+[pipelex.tracing_config.dynamodb]
+table_name = ""
+region = ""
+
+[pipelex.tracing_config.temporal_dynamodb]
+table_name = ""
+region = ""
+
####################################################################################################
# Storage Config
####################################################################################################
[pipelex.storage_config]
-# Storage method: "local", "in_memory", "s3", or "gcp"
-method = "local"
# Whether to fetch remote HTTP URLs and store them locally
is_fetch_remote_content_enabled = true
# Whether to upload local file paths to storage and replace with pipelex-storage:// URIs
is_upload_local_content_enabled = true
+# Storage method: "local", "in_memory", "s3", or "gcp"
+method = "local"
[pipelex.storage_config.local]
# Local storage settings
@@ -193,3 +212,76 @@ is_dump_response_text_enabled = false
is_dump_kwargs_enabled = false
is_dump_response_enabled = false
is_dump_error_enabled = false
+
+####################################################################################################
+# Temporal Config
+####################################################################################################
+
+[temporal]
+is_enabled = false
+
+[temporal.payload_codec_config]
+is_enabled = false
+size_threshold = 1_048_576 # in bytes (~1 MB, well under Temporal's 2 MB hard limit)
+storage_prefix = "temporal-payloads/"
+
+[temporal.payload_codec_config.storage_provider_config]
+method = "local"
+
+[temporal.payload_codec_config.storage_provider_config.local]
+uri_format = "{hash}"
+local_storage_path = ".pipelex/temporal-payload-store"
+
+[temporal.worker_config]
+task_queue = "temporal_task_queue"
+workflow_execution_timeout = "1:00:00"
+run_timeout = "1:00:00"
+task_timeout = "0:00:10"
+start_delay = "0:00:00"
+rpc_timeout = "1:00:00"
+
+[temporal.worker_config.retry_policy_config]
+initial_interval = "0:00:03"
+backoff_coefficient = 2.0
+maximum_interval = "unlimited"
+maximum_attempts = 3
+non_retryable_error_types = [
+ "ExtractHandleNotFoundError",
+ "FileNotFoundError",
+ "ImgGenHandleNotFoundError",
+ "LLMHandleNotFoundError",
+ "ModelNotFoundError",
+ "ValidationError",
+]
+
+[temporal.temporal_config]
+selected_server = "local"
+
+[temporal.temporal_config.temporal_server_configs.local]
+description = "default localhost"
+target_host = "localhost:7233"
+namespace = "default"
+api_key_method = "none"
+api_key_id = ""
+
+[temporal.temporal_config.temporal_server_configs.testing]
+description = "Temporal server for testing"
+target_host = "${TESTING_TEMPORAL_TARGET_HOST}"
+namespace = "${TESTING_TEMPORAL_NAMESPACE}"
+api_key_method = "secret_provider"
+api_key_id = "${TESTING_TEMPORAL_MAIN_API_KEY}"
+
+[temporal.temporal_config.temporal_log_config]
+is_workflow_info_on_message = false
+is_workflow_info_on_extra = true
+is_full_workflow_info_on_extra = false
+is_activity_info_on_message = false
+is_activity_info_on_extra = true
+is_full_activity_info_on_extra = false
+is_formatter_enabled = true
+is_prefix_enabled = true
+managed_loggers = [
+ "temporalio.activity",
+ "temporalio.workflow",
+ "temporalio.worker._activity",
+]
diff --git a/pipelex/kit/paths.py b/pipelex/kit/paths.py
index 7a3f54f6c..266b57871 100644
--- a/pipelex/kit/paths.py
+++ b/pipelex/kit/paths.py
@@ -9,6 +9,7 @@ from pipelex.types import Traversable
# - telemetry_override.toml: Personal telemetry settings
GIT_IGNORED_CONFIG_FILES: frozenset[str] = frozenset(
{
+ ".DS_Store",
"pipelex_service.toml",
"pipelex_override.toml",
"telemetry_override.toml",
@@ -29,9 +30,12 @@ CONFIG_SYNC_EXCLUDED_FILES: frozenset[str] = GIT_IGNORED_CONFIG_FILES
# Directories that should not be synced between .pipelex and kit/configs.
# These are runtime directories created locally:
# - storage: Local storage directory for runtime data
+# - traces: Local directory for execution traces
GIT_IGNORED_CONFIG_DIRS: frozenset[str] = frozenset(
{
"storage",
+ "temporal-payload-store",
+ "traces",
}
)
diff --git a/pipelex/libraries/library.py b/pipelex/libraries/library.py
index 94b6852b7..c13dc0424 100644
--- a/pipelex/libraries/library.py
+++ b/pipelex/libraries/library.py
@@ -1,7 +1,8 @@
from pathlib import Path
from typing import TYPE_CHECKING
-from pydantic import BaseModel, Field
+from kajson.class_registry import ClassRegistry
+from pydantic import BaseModel, Field, PrivateAttr
from pipelex import log
from pipelex.base_exceptions import PipelexUnexpectedError
@@ -35,6 +36,13 @@ class Library(BaseModel):
pipe_library: PipeLibrary
loaded_mthds_paths: list[Path] = Field(default_factory=empty_list_factory_of(Path))
dependency_libraries: dict[str, "Library"] = Field(default_factory=dict)
+ _class_registry: ClassRegistry | None = PrivateAttr(default=None)
+
+ def get_class_registry(self) -> ClassRegistry | None:
+ return self._class_registry
+
+ def set_class_registry(self, class_registry: ClassRegistry) -> None:
+ self._class_registry = class_registry
def get_domain_library(self) -> DomainLibrary:
return self.domain_library
diff --git a/pipelex/libraries/library_crate.py b/pipelex/libraries/library_crate.py
new file mode 100644
index 000000000..622a5db85
--- /dev/null
+++ b/pipelex/libraries/library_crate.py
@@ -0,0 +1,58 @@
+import hashlib
+import json
+
+from pydantic import BaseModel, ConfigDict, Field
+
+from pipelex.core.bundles.pipelex_bundle_blueprint import PipeBlueprintUnion
+from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
+from pipelex.core.domains.domain_blueprint import DomainBlueprint
+
+
+class LibraryCrate(BaseModel):
+ """Complete library content as qualified blueprints, ready to load into a live Library.
+
+ The crate is a flat, domain-agnostic snapshot of all concepts, pipes, and domain metadata.
+ Domain is encoded in the dictionary keys (e.g. 'scoring.WeightedScore',
+ 'scoring.compute_score'), not in a structural container.
+
+ Includes source information so that validation errors can trace back to origin files.
+ """
+
+ model_config = ConfigDict(extra="forbid")
+
+ concepts: dict[str, ConceptBlueprint | str] = Field(default_factory=dict)
+ """concept_ref (domain.ConceptCode) -> ConceptBlueprint or string description"""
+
+ pipes: dict[str, PipeBlueprintUnion] = Field(default_factory=dict)
+ """pipe_ref (domain.pipe_code) -> PipeBlueprintUnion"""
+
+ domains: dict[str, DomainBlueprint] = Field(default_factory=dict)
+ """domain_code -> DomainBlueprint (first-write-wins across bundles)"""
+
+ source_map: dict[str, str] = Field(default_factory=dict)
+ """concept_ref or pipe_ref -> source file path (for error reporting)"""
+
+ fingerprint: str = ""
+ """SHA-256 hex digest of the serialized concepts + pipes content."""
+
+ @staticmethod
+ def compute_fingerprint_from_content(
+ concepts: dict[str, "ConceptBlueprint | str"],
+ pipes: dict[str, PipeBlueprintUnion],
+ ) -> str:
+ """Compute SHA-256 fingerprint from concepts and pipes content.
+
+ Uses deterministic JSON serialization of concepts and pipes only
+ (excludes domains, source_map, and fingerprint itself).
+ """
+ concepts_json: dict[str, object] = {}
+ for ref, value in sorted(concepts.items()):
+ concepts_json[ref] = value if isinstance(value, str) else value.model_dump(mode="json")
+ pipes_json = {ref: blueprint.model_dump(mode="json") for ref, blueprint in sorted(pipes.items())}
+ payload = {"concepts": concepts_json, "pipes": pipes_json}
+ serialized = json.dumps(payload, sort_keys=True, ensure_ascii=True)
+ return hashlib.sha256(serialized.encode()).hexdigest()
+
+ def compute_fingerprint(self) -> str:
+ """Compute SHA-256 fingerprint from this crate's concepts and pipes."""
+ return self.compute_fingerprint_from_content(concepts=self.concepts, pipes=self.pipes)
diff --git a/pipelex/libraries/library_crate_factory.py b/pipelex/libraries/library_crate_factory.py
new file mode 100644
index 000000000..22b4c51dc
--- /dev/null
+++ b/pipelex/libraries/library_crate_factory.py
@@ -0,0 +1,123 @@
+from typing import TYPE_CHECKING
+
+from pipelex import log
+from pipelex.core.bundles.pipelex_bundle_blueprint import PipeBlueprintUnion, PipelexBundleBlueprint
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.domains.domain_blueprint import DomainBlueprint
+from pipelex.core.pipes.pipe_factory import PipeFactory
+from pipelex.libraries.concept.exceptions import ConceptLibraryError
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.libraries.pipe.exceptions import PipeLibraryError
+
+if TYPE_CHECKING:
+ from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
+
+
+class LibraryCrateFactory:
+ """Builds a LibraryCrate from parsed bundle blueprints."""
+
+ @classmethod
+ def make_from_blueprints(
+ cls,
+ blueprints: list[PipelexBundleBlueprint],
+ ) -> LibraryCrate:
+ """Build a LibraryCrate from parsed bundle blueprints.
+
+ For each bundle:
+ 1. Qualify concept codes with the bundle's domain -> concept_ref keys
+ 2. Qualify pipe codes with the bundle's domain -> pipe_ref keys
+ 3. Preserve string-described concepts as-is (no normalization to ConceptBlueprint)
+ 4. Collect domain metadata (first-write-wins per domain code)
+ 5. Track source file for each concept_ref and pipe_ref
+ 6. Detect duplicate refs across bundles (raise ConceptLibraryError / PipeLibraryError)
+ 7. Compute SHA-256 fingerprint
+
+ Args:
+ blueprints: List of parsed MTHDS bundle blueprints
+
+ Returns:
+ A LibraryCrate with all content merged and fingerprinted
+ """
+ concepts: dict[str, ConceptBlueprint | str] = {}
+ pipes: dict[str, PipeBlueprintUnion] = {}
+ domains: dict[str, DomainBlueprint] = {}
+ source_map: dict[str, str] = {}
+
+ for blueprint in blueprints:
+ domain_code = blueprint.domain
+ source = blueprint.source
+
+ # Domain metadata: first-write-wins
+ if domain_code not in domains:
+ domains[domain_code] = DomainBlueprint(
+ source=source,
+ code=domain_code,
+ description=blueprint.description or "",
+ system_prompt=blueprint.system_prompt,
+ main_pipe=blueprint.main_pipe,
+ )
+ else:
+ existing = domains[domain_code]
+ new_description = blueprint.description or ""
+ if existing.description != new_description:
+ log.warning(
+ f"Domain '{domain_code}' declared with different descriptions: "
+ f"'{existing.description}' vs '{new_description}'. Keeping the first.",
+ )
+ if existing.system_prompt != blueprint.system_prompt:
+ log.warning(
+ f"Domain '{domain_code}' declared with different system_prompts. Keeping the first.",
+ )
+
+ # Concepts
+ if blueprint.concept is not None:
+ for concept_code, value in blueprint.concept.items():
+ concept_ref = ConceptFactory.make_concept_ref_with_domain(domain_code=domain_code, concept_code=concept_code)
+ if concept_ref in concepts:
+ existing_source = source_map.get(concept_ref)
+ if existing_source is not None and existing_source == source:
+ msg = (
+ f"Concept '{concept_ref}' is declared twice in the same bundle file: '{existing_source}'. "
+ "Please remove the duplicate declaration."
+ )
+ else:
+ msg = (
+ f"Concept '{concept_ref}' is declared in two different bundle files: "
+ f"'{existing_source or 'unknown'}' and '{source or 'unknown'}'. "
+ "Please remove one of the declarations or rename one of the concepts."
+ )
+ raise ConceptLibraryError(msg)
+ concepts[concept_ref] = value
+ if source:
+ source_map[concept_ref] = source
+
+ # Pipes
+ if blueprint.pipe is not None:
+ for pipe_code, pipe_blueprint in blueprint.pipe.items():
+ pipe_ref = PipeFactory.make_pipe_ref_with_domain(domain_code=domain_code, pipe_code=pipe_code)
+ if pipe_ref in pipes:
+ existing_source = source_map.get(pipe_ref)
+ if existing_source is not None and existing_source == source:
+ msg = (
+ f"Pipe '{pipe_ref}' is declared twice in the same bundle file: '{existing_source}'. "
+ "Please remove the duplicate declaration."
+ )
+ else:
+ msg = (
+ f"Pipe '{pipe_ref}' is declared in two different bundle files: "
+ f"'{existing_source or 'unknown'}' and '{source or 'unknown'}'. "
+ "Please remove one of the declarations or rename one of the pipes."
+ )
+ raise PipeLibraryError(msg)
+ pipes[pipe_ref] = pipe_blueprint
+ if source:
+ source_map[pipe_ref] = source
+
+ fingerprint = LibraryCrate.compute_fingerprint_from_content(concepts=concepts, pipes=pipes)
+ return LibraryCrate(
+ concepts=concepts,
+ pipes=pipes,
+ domains=domains,
+ source_map=source_map,
+ fingerprint=fingerprint,
+ )
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index 83622afc7..8c8ba7f7f 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -3,7 +3,7 @@ from graphlib import CycleError, TopologicalSorter
from pathlib import Path
from typing import TYPE_CHECKING
-from kajson.kajson_manager import KajsonManager
+from kajson.class_registry import ClassRegistry
from mthds.package.dependency_resolver import ResolvedDependency, determine_exported_pipes, resolve_all_dependencies
from mthds.package.discovery import find_package_manifest
from mthds.package.exceptions import DependencyResolveError, ManifestError
@@ -15,9 +15,9 @@ import pipelex.builder as builder_pkg # package import — used for __file__ pa
from pipelex import log
from pipelex.cli.installed_methods import find_method_by_full_address
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
+from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.concepts.concept_factory import ConceptFactory
from pipelex.core.concepts.native.concept_native import NativeConceptCode
-from pipelex.core.domains.domain import Domain
from pipelex.core.domains.domain_blueprint import DomainBlueprint
from pipelex.core.domains.domain_factory import DomainFactory
from pipelex.core.interpreter.exceptions import PipelexInterpreterError
@@ -27,13 +27,15 @@ from pipelex.core.pipes.pipe_factory import PipeFactory
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.core.validation import report_validation_error
-from pipelex.hub import get_current_library
+from pipelex.hub import get_class_registry, get_current_library
from pipelex.libraries.concept.exceptions import ConceptLibraryError
from pipelex.libraries.exceptions import (
LibraryError,
LibraryLoadingError,
)
from pipelex.libraries.library import Library
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.libraries.library_crate_factory import LibraryCrateFactory
from pipelex.libraries.library_factory import LibraryFactory
from pipelex.libraries.library_manager_abstract import LibraryManagerAbstract
from pipelex.libraries.library_utils import (
@@ -46,8 +48,9 @@ from pipelex.system.registries.func_registry_utils import FuncRegistryUtils
from pipelex.tools.misc.semver import SemVerError, parse_constraint, parse_version, version_satisfies
if TYPE_CHECKING:
+ from collections.abc import Mapping
+
from pipelex.core.concepts.concept import Concept
- from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
from pipelex.core.domains.domain import Domain
MTHDS_METHODS_DIRNAME = ".mthds/methods"
@@ -93,6 +96,9 @@ class LibraryManager(LibraryManagerAbstract):
# UNTITLED library is the fallback library for all others
self._libraries: dict[str, Library] = {}
self._pipe_source_map: dict[str, Path] = {} # pipe_ref (domain.pipe_code) -> source .mthds file
+ self._blueprints: dict[str, list[PipelexBundleBlueprint]] = {} # library_id -> accumulated blueprints
+ self._crate_cache: dict[str, LibraryCrate] = {} # library_id -> cached crate from get_crate()
+ self._loaded_fingerprints: dict[str, set[str]] = {} # library_id -> set of loaded crate fingerprints
############################################################
# Manager lifecycle
@@ -116,12 +122,18 @@ class LibraryManager(LibraryManagerAbstract):
self._pipe_source_map.pop(pipe_ref, None)
library.teardown()
del self._libraries[library_id]
+ self._blueprints.pop(library_id, None)
+ self._crate_cache.pop(library_id, None)
+ self._loaded_fingerprints.pop(library_id, None)
return
for library in self._libraries.values():
library.teardown()
self._libraries = {}
self._pipe_source_map = {}
+ self._blueprints = {}
+ self._crate_cache = {}
+ self._loaded_fingerprints = {}
@override
def reset(self) -> None:
@@ -145,6 +157,14 @@ class LibraryManager(LibraryManagerAbstract):
# Public library accessors
############################################################
+ @override
+ def get_library_class_registry(self, library_id: str) -> ClassRegistry | None:
+ """Get the ClassRegistry associated with a library, if any."""
+ library = self._libraries.get(library_id)
+ if library is not None:
+ return library.get_class_registry()
+ return None
+
@override
def get_library(self, library_id: str) -> Library:
if library_id not in self._libraries:
@@ -182,6 +202,18 @@ class LibraryManager(LibraryManagerAbstract):
return matches[0]
return None
+ @override
+ def get_crate(self, library_id: str) -> LibraryCrate | None:
+ cached = self._crate_cache.get(library_id)
+ if cached is not None:
+ return cached
+ accumulated = self._blueprints.get(library_id)
+ if not accumulated:
+ return None
+ crate = LibraryCrateFactory.make_from_blueprints(blueprints=accumulated)
+ self._crate_cache[library_id] = crate
+ return crate
+
############################################################
# Private methods
############################################################
@@ -335,41 +367,41 @@ class LibraryManager(LibraryManagerAbstract):
return all_concepts
@override
- def load_from_blueprints(self, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> list[PipeAbstract]:
- """Load domains, concepts, and pipes from a list of blueprints.
+ def load_from_crate(self, library_id: str, crate: LibraryCrate) -> list[PipeAbstract]:
+ """Load a LibraryCrate into a live Library.
+
+ Fingerprint idempotency: if a crate with the same fingerprint was already loaded
+ into this library_id, the load is skipped and an empty list is returned.
+
+ Note: This method does NOT resolve cross-package address-based dependencies.
+ Callers must handle dependency loading before calling this method (e.g. via
+ _load_address_based_dependencies). The load_from_blueprints method does this
+ automatically before delegating here.
Args:
- library_id: The ID of the library to load into
- blueprints: List of parsed MTHDS blueprints to load
+ library_id: The library to load into
+ crate: The LibraryCrate containing qualified blueprints, domain metadata, and source info
Returns:
- List of all pipes that were loaded
+ List of all pipes that were loaded, or empty list if already loaded
"""
- # Discover and load address-based cross-package dependencies before loading pipes
- self._load_address_based_dependencies(
- library_id=library_id,
- blueprints=blueprints,
- )
-
+ # Fingerprint idempotency: skip if this crate was already loaded into this library
+ fingerprint = crate.fingerprint
+ loaded_set = self._loaded_fingerprints.setdefault(library_id, set())
+ if fingerprint in loaded_set:
+ log.verbose(f"Crate with fingerprint {fingerprint[:12]}... already loaded into '{library_id}', skipping")
+ return []
library = self.get_library(library_id=library_id)
- all_pipes: list[PipeAbstract] = []
- # Load all domains first
+ # Load domains from crate metadata
all_domains: list[Domain] = []
- for blueprint in blueprints:
- domain = DomainFactory.make_from_blueprint(
- blueprint=DomainBlueprint(
- source=blueprint.source,
- code=blueprint.domain,
- description=blueprint.description or "",
- system_prompt=blueprint.system_prompt,
- ),
- )
+ for domain_blueprint in crate.domains.values():
+ domain = DomainFactory.make_from_blueprint(blueprint=domain_blueprint)
all_domains.append(domain)
library.domain_library.add_domains(domains=all_domains)
- # Load concepts (forward references resolved after all are loaded)
- all_concepts = self._load_concepts_from_blueprints(blueprints)
+ # Load concepts in topological order
+ all_concepts = self._load_concepts_from_crate(crate.concepts)
library.concept_library.add_concepts(concepts=all_concepts)
# Resolve forward references in dynamically generated structure classes
@@ -378,47 +410,77 @@ class LibraryManager(LibraryManagerAbstract):
# Detect cycles in concept references (A -> B -> A is forbidden)
self._detect_concept_cycles(all_concepts)
- # Load all pipes, detecting duplicate declarations across bundles in this library
- pipe_source_in_this_load: dict[str, Path | None] = {}
- for blueprint in blueprints:
- pipes: list[PipeAbstract] = []
- new_source = Path(blueprint.source) if blueprint.source else None
- if blueprint.pipe is not None:
- for pipe_code, pipe_blueprint in blueprint.pipe.items():
- pipe_ref = f"{blueprint.domain}.{pipe_code}"
- # Detect duplicate pipe declarations across different bundles in the same library
- if pipe_ref in pipe_source_in_this_load:
- existing_source = pipe_source_in_this_load[pipe_ref]
- if existing_source == new_source:
- msg = (
- f"Pipe '{pipe_ref}' is declared twice in the same bundle file: '{existing_source}'. "
- "Please remove the duplicate declaration."
- )
- else:
- msg = (
- f"Pipe '{pipe_ref}' is declared in two different bundle files: "
- f"'{existing_source}' and '{new_source}'. "
- "Please remove one of the declarations or rename one of the pipes."
- )
- raise PipeLibraryError(msg)
- pipe_source_in_this_load[pipe_ref] = new_source
- pipe = PipeFactory[PipeAbstract].make_from_blueprint(
- domain_code=blueprint.domain,
- pipe_code=pipe_code,
- blueprint=pipe_blueprint,
- concept_codes_from_the_same_domain=[the_concept.code for the_concept in all_concepts],
- )
- pipes.append(pipe)
- # Track source file for this pipe (used by get_pipe_source)
- if new_source:
- self._pipe_source_map[pipe_ref] = new_source
- all_pipes.extend(pipes)
+ # Precompute domain -> concept local codes mapping (avoids O(N*M) re-parsing per pipe)
+ domain_concept_codes: dict[str, list[str]] = {}
+ for concept_ref in crate.concepts:
+ parsed_concept = QualifiedRef.parse_concept_ref(raw=concept_ref)
+ if parsed_concept.domain_path is not None:
+ domain_concept_codes.setdefault(parsed_concept.domain_path, []).append(parsed_concept.local_code)
+
+ # Load pipes with domain-filtered concept codes
+ all_pipes: list[PipeAbstract] = []
+ for pipe_ref, pipe_blueprint in crate.pipes.items():
+ parsed = QualifiedRef.parse_pipe_ref(raw=pipe_ref)
+ if parsed.domain_path is None:
+ msg = f"Crate pipe_ref '{pipe_ref}' must be domain-qualified"
+ raise PipeLibraryError(msg)
+ domain_code = parsed.domain_path
+ pipe_code = parsed.local_code
+
+ concept_codes_for_domain = domain_concept_codes.get(domain_code, [])
+
+ pipe = PipeFactory[PipeAbstract].make_from_blueprint(
+ domain_code=domain_code,
+ pipe_code=pipe_code,
+ blueprint=pipe_blueprint,
+ concept_codes_from_the_same_domain=concept_codes_for_domain,
+ )
+ all_pipes.append(pipe)
+
+ # Track source file for this pipe (used by get_pipe_source)
+ source = crate.source_map.get(pipe_ref)
+ if source:
+ self._pipe_source_map[pipe_ref] = Path(source)
library.pipe_library.add_pipes(pipes=all_pipes)
library.validate_library()
+
+ # Only cache fingerprint after the entire load succeeds — if loading fails
+ # with an exception, subsequent retries must not be skipped.
+ loaded_set.add(fingerprint)
return all_pipes
+ @override
+ def load_from_blueprints(self, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> list[PipeAbstract]:
+ """Load domains, concepts, and pipes from a list of blueprints.
+
+ Delegates through LibraryCrate: builds a crate from blueprints, then loads from the crate.
+ Also accumulates blueprints for later crate retrieval via get_crate().
+
+ Args:
+ library_id: The ID of the library to load into
+ blueprints: List of parsed MTHDS blueprints to load
+
+ Returns:
+ List of all pipes that were loaded
+ """
+ # Accumulate blueprints for later crate construction via get_crate()
+ self._blueprints.setdefault(library_id, []).extend(blueprints)
+ self._crate_cache.pop(library_id, None)
+
+ # Discover and load address-based cross-package dependencies before loading pipes
+ self._load_address_based_dependencies(
+ library_id=library_id,
+ blueprints=blueprints,
+ )
+
+ # Build the crate (merges, qualifies, detects duplicates)
+ crate = LibraryCrateFactory.make_from_blueprints(blueprints=blueprints)
+
+ # Load from crate (domains, concepts, pipes, validation)
+ return self.load_from_crate(library_id=library_id, crate=crate)
+
@override
def load_concepts_only_from_blueprints(
self,
@@ -449,6 +511,7 @@ class LibraryManager(LibraryManagerAbstract):
code=blueprint.domain,
description=blueprint.description or "",
system_prompt=blueprint.system_prompt,
+ main_pipe=blueprint.main_pipe,
),
)
all_domains.append(domain)
@@ -520,7 +583,53 @@ class LibraryManager(LibraryManagerAbstract):
concept_blueprint,
)
- # Step 2: Build dependency graph and topologically sort using graphlib
+ return self._topological_load_concepts(ref_to_entry)
+
+ def _load_concepts_from_crate(
+ self,
+ concepts: dict[str, ConceptBlueprint | str],
+ ) -> list["Concept"]:
+ """Load concepts from a crate's flat concept dict in topological order.
+
+ Builds the entry map from the crate's flat dict, then delegates to
+ _topological_load_concepts for sorting and instantiation.
+ Duplicate detection is already done by LibraryCrateFactory.
+
+ Args:
+ concepts: Flat dict mapping concept_ref to ConceptBlueprint or string description
+
+ Returns:
+ List of loaded concepts in topological order
+ """
+ ref_to_entry: dict[str, tuple[str, str, ConceptBlueprint | str]] = {}
+ for concept_ref, concept_blueprint in concepts.items():
+ parsed = QualifiedRef.parse_concept_ref(raw=concept_ref)
+ if parsed.domain_path is None:
+ msg = f"Crate concept_ref '{concept_ref}' must be domain-qualified"
+ raise ConceptLibraryError(msg)
+ ref_to_entry[concept_ref] = (
+ parsed.domain_path,
+ parsed.local_code,
+ concept_blueprint,
+ )
+
+ return self._topological_load_concepts(ref_to_entry)
+
+ def _topological_load_concepts(
+ self,
+ ref_to_entry: "Mapping[str, tuple[str, str, ConceptBlueprint | str]]",
+ ) -> list["Concept"]:
+ """Sort concepts by refines dependencies and instantiate them.
+
+ Shared helper used by both _load_concepts_from_blueprints (blueprint path)
+ and _load_concepts_from_crate (crate path).
+
+ Args:
+ ref_to_entry: Mapping of concept_ref to (domain_code, concept_code, blueprint_or_string)
+
+ Returns:
+ List of loaded concepts in topological order
+ """
sorter: TopologicalSorter[str] = TopologicalSorter()
for concept_ref, (
@@ -548,14 +657,12 @@ class LibraryManager(LibraryManagerAbstract):
sorter.add(concept_ref, *dependencies)
- # Step 3: Get sorted order (raises CycleError if cycles detected)
try:
sorted_refs = list(sorter.static_order())
except CycleError as exc:
msg = f"Cycle detected in concept refines dependencies: {exc.args[1]}"
raise LibraryLoadingError(msg) from exc
- # Step 4: Load concepts in topological order
all_concepts: list[Concept] = []
for concept_ref in sorted_refs:
domain_code, concept_code, concept_blueprint = ref_to_entry[concept_ref]
@@ -997,7 +1104,9 @@ class LibraryManager(LibraryManagerAbstract):
def _remove_pipes_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> None:
library = self.get_current_library()
if blueprint.pipe is not None:
- pipe_refs_to_remove = [f"{blueprint.domain}.{pipe_code}" for pipe_code in blueprint.pipe]
+ pipe_refs_to_remove = [
+ PipeFactory.make_pipe_ref_with_domain(domain_code=blueprint.domain, pipe_code=pipe_code) for pipe_code in blueprint.pipe
+ ]
library.pipe_library.remove_pipes_by_refs(pipe_refs=pipe_refs_to_remove)
def _remove_concepts_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> None:
@@ -1032,7 +1141,7 @@ class LibraryManager(LibraryManagerAbstract):
"""
# Build namespace with all structure class names
namespace: dict[str, type] = {}
- class_registry = KajsonManager.get_class_registry()
+ class_registry = get_class_registry()
for concept in concepts:
structure_class = class_registry.get_class(name=concept.structure_class_name)
@@ -1063,7 +1172,7 @@ class LibraryManager(LibraryManagerAbstract):
# TODO: Refactor to inspect ConceptStructureBlueprint directly (concept_ref and item_concept_ref fields)
# instead of the generated Python types. This would be more direct and wouldn't depend on
# how types are generated (e.g., Optional wrappers for non-required fields).
- class_registry = KajsonManager.get_class_registry()
+ class_registry = get_class_registry()
# Build mappings from class names to concept refs
class_to_concept: dict[str, str] = {}
diff --git a/pipelex/libraries/library_manager_abstract.py b/pipelex/libraries/library_manager_abstract.py
index 22893bd3b..aa1f2ad4c 100644
--- a/pipelex/libraries/library_manager_abstract.py
+++ b/pipelex/libraries/library_manager_abstract.py
@@ -2,8 +2,11 @@ from abc import ABC, abstractmethod
from pathlib import Path
from typing import TYPE_CHECKING
+from kajson.class_registry import ClassRegistry
+
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.core.pipes.pipe_abstract import PipeAbstract
+from pipelex.libraries.library_crate import LibraryCrate
if TYPE_CHECKING:
from pipelex.core.concepts.concept import Concept
@@ -35,6 +38,13 @@ class LibraryManagerAbstract(ABC):
def get_current_library(self) -> "Library":
"""Get the Library object for the current library."""
+ def get_library_class_registry(self, library_id: str) -> ClassRegistry | None: # noqa: ARG002
+ """Get the ClassRegistry associated with a library, if any.
+
+ Returns None by default. Overridden by LibraryManager.
+ """
+ return None
+
def get_pipe_source(self, pipe_code: str) -> Path | None: # noqa: ARG002
"""Get the source file path for a pipe.
@@ -46,6 +56,31 @@ class LibraryManagerAbstract(ABC):
"""
return None
+ @abstractmethod
+ def get_crate(self, library_id: str) -> LibraryCrate | None:
+ """Build a LibraryCrate from all accumulated blueprints for a given library_id.
+
+ Returns None if the library_id is unknown or no blueprints were loaded.
+
+ Args:
+ library_id: The library to get the crate for
+
+ Returns:
+ A LibraryCrate built from all accumulated blueprints, or None
+ """
+
+ @abstractmethod
+ def load_from_crate(self, library_id: str, crate: LibraryCrate) -> list[PipeAbstract]:
+ """Load a LibraryCrate into a live Library.
+
+ Note: This method does NOT resolve cross-package address-based dependencies.
+ Callers must handle dependency loading before calling this method.
+
+ Args:
+ library_id: The library to load into
+ crate: The LibraryCrate containing qualified blueprints, domain metadata, and source info
+ """
+
@abstractmethod
def load_from_blueprints(self, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> list[PipeAbstract]:
pass
diff --git a/pipelex/pipe_controllers/batch/pipe_batch.py b/pipelex/pipe_controllers/batch/pipe_batch.py
index e4ea1968f..dc96e978c 100644
--- a/pipelex/pipe_controllers/batch/pipe_batch.py
+++ b/pipelex/pipe_controllers/batch/pipe_batch.py
@@ -9,9 +9,10 @@ from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.graph.graph_tracer_manager import GraphTracerManager
-from pipelex.hub import get_required_pipe
+from pipelex.hub import get_pipe_router, get_required_pipe
from pipelex.pipe_controllers.pipe_controller import PipeController
from pipelex.pipe_run.exceptions import PipeRunError
+from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunParams
from pipelex.pipeline.job_metadata import JobMetadata
@@ -19,6 +20,7 @@ if TYPE_CHECKING:
from collections.abc import Coroutine
from pipelex.core.stuffs.stuff_content import StuffContent
+ from pipelex.libraries.library_crate import LibraryCrate
class PipeBatch(PipeController):
@@ -98,6 +100,7 @@ class PipeBatch(PipeController):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
batch_params = pipe_run_params.batch_params or self.batch_params or BatchParams.make_default()
input_item_stuff_name = batch_params.input_item_stuff_name
@@ -133,7 +136,7 @@ class PipeBatch(PipeController):
# Pass this PipeBatch's node_id so BATCH_ITEM edges can source from the controller
batch_controller_node_id = job_metadata.graph_context.parent_node_id
tracer_manager.register_batch_item_extraction(
- graph_id=job_metadata.graph_context.graph_id,
+ lookup_key=job_metadata.graph_context.lookup_key,
list_stuff_code=input_stuff.stuff_code,
item_stuff_code=branch_input_item_code,
item_index=branch_index,
@@ -156,11 +159,15 @@ class PipeBatch(PipeController):
},
)
branch_pipe_run_params.run_mode = pipe_run_params.run_mode
- task = sub_pipe.run_pipe(
- job_metadata=job_metadata,
- working_memory=branch_memory,
- output_name=None,
- pipe_run_params=branch_pipe_run_params,
+ task = get_pipe_router().run(
+ pipe_job=PipeJobFactory.make_pipe_job(
+ pipe=sub_pipe,
+ job_metadata=job_metadata,
+ working_memory=branch_memory,
+ pipe_run_params=branch_pipe_run_params,
+ output_name=None,
+ library_crate=library_crate,
+ ),
)
tasks.append(task)
@@ -191,7 +198,7 @@ class PipeBatch(PipeController):
batch_controller_node_id = job_metadata.graph_context.parent_node_id
for agg_index, item_stuff_code in enumerate(branch_output_stuff_codes):
tracer_manager.register_batch_aggregation(
- graph_id=job_metadata.graph_context.graph_id,
+ lookup_key=job_metadata.graph_context.lookup_key,
output_list_stuff_code=output_stuff.stuff_code,
item_stuff_code=item_stuff_code,
item_index=agg_index,
@@ -222,12 +229,14 @@ class PipeBatch(PipeController):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
pipe_output = await self._live_run_controller_pipe(
job_metadata=job_metadata,
working_memory=working_memory,
pipe_run_params=pipe_run_params,
output_name=output_name,
+ library_crate=library_crate,
)
# For dry run coordination: set the first item's pipe_code to "mock_main"
# to match the BundleHeaderSpec.main_pipe examples=["mock_main"]
diff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.py
index 162b7c404..4cde8037e 100644
--- a/pipelex/pipe_controllers/condition/pipe_condition.py
+++ b/pipelex/pipe_controllers/condition/pipe_condition.py
@@ -1,16 +1,17 @@
-from typing import Any, Literal
+from typing import TYPE_CHECKING, Any, Literal
from typing_extensions import override
from pipelex import log
from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.cogt.templating.template_rendering import render_template
from pipelex.core.concepts.native.concept_native import NativeConceptCode
from pipelex.core.memory.working_memory import WorkingMemory, WorkingMemoryStuffNotFoundError
from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorType
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.inputs.input_stuff_specs_factory import InputStuffSpecsFactory
from pipelex.core.pipes.pipe_output import PipeOutput
-from pipelex.hub import get_content_generator, get_optional_pipe, get_pipe_router, get_required_pipe
+from pipelex.hub import get_optional_pipe, get_pipe_router, get_required_pipe
from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome
from pipelex.pipe_controllers.pipe_controller import PipeController
from pipelex.pipe_run.exceptions import PipeRunError
@@ -21,6 +22,9 @@ from pipelex.tools.jinja2.jinja2_errors import Jinja2DetectVariablesError
from pipelex.tools.jinja2.jinja2_required_variables import detect_jinja2_required_variables
from pipelex.tools.misc.string_utils import get_root_from_dotted_path
+if TYPE_CHECKING:
+ from pipelex.libraries.library_crate import LibraryCrate
+
ConditionOutcomeMap = dict[str, str | SpecialOutcome]
@@ -171,10 +175,10 @@ class PipeCondition(PipeController):
async def _validate_before_run(
self, job_metadata: JobMetadata, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None = None
):
- evaluated_expression = await get_content_generator().make_templated_text(
- context=working_memory.generate_context(),
+ evaluated_expression = await render_template(
template=self.expression,
- template_category=TemplateCategory.EXPRESSION,
+ category=TemplateCategory.EXPRESSION,
+ context=working_memory.generate_context(),
)
if not evaluated_expression or evaluated_expression == "None":
error_msg = f"PipeCondition '{self.code}': Conditional expression returned no result"
@@ -202,10 +206,10 @@ class PipeCondition(PipeController):
Returns:
The evaluated expression
"""
- evaluated_expression = await get_content_generator().make_templated_text(
- context=working_memory.generate_context(),
+ evaluated_expression = await render_template(
template=self.expression,
- template_category=TemplateCategory.EXPRESSION,
+ category=TemplateCategory.EXPRESSION,
+ context=working_memory.generate_context(),
)
log.verbose(f"add_alias: {evaluated_expression} -> {self.add_alias_from_expression_to}")
@@ -224,6 +228,7 @@ class PipeCondition(PipeController):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
evaluated_expression = await self._evaluate_expression(working_memory=working_memory)
# Select the outcome based on the evaluated expression
@@ -269,6 +274,7 @@ class PipeCondition(PipeController):
working_memory=working_memory,
pipe_run_params=pipe_run_params,
output_name=output_name,
+ library_crate=library_crate,
),
)
self._register_execution_data(job_metadata, execution_data_dict)
@@ -281,6 +287,7 @@ class PipeCondition(PipeController):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
# Validate that the expression template is valid
try:
@@ -320,6 +327,8 @@ class PipeCondition(PipeController):
job_metadata=job_metadata,
working_memory=working_memory,
pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ library_crate=library_crate,
)
execution_data_dict: dict[str, Any] = {
"evaluated_expression": "dry_run",
diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.py
index 1a279efb9..323889e89 100644
--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py
+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py
@@ -30,6 +30,7 @@ if TYPE_CHECKING:
from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.stuff_content import StuffContent
+ from pipelex.libraries.library_crate import LibraryCrate
class PipeParallel(PipeController):
@@ -137,6 +138,7 @@ class PipeParallel(PipeController):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
if pipe_run_params.final_stuff_code:
log.verbose(f"PipeBatch.run_pipe() final_stuff_code: {pipe_run_params.final_stuff_code}")
@@ -151,6 +153,7 @@ class PipeParallel(PipeController):
job_metadata=job_metadata,
working_memory=working_memory.make_deep_copy(),
sub_pipe_run_params=pipe_run_params.make_deep_copy(),
+ library_crate=library_crate,
),
)
@@ -228,6 +231,7 @@ class PipeParallel(PipeController):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
# 1. Validate that all sub-pipes exist
for sub_pipe in self.parallel_sub_pipes:
@@ -247,6 +251,7 @@ class PipeParallel(PipeController):
job_metadata=job_metadata,
working_memory=working_memory.make_deep_copy(),
sub_pipe_run_params=pipe_run_params.make_deep_copy(),
+ library_crate=library_crate,
),
)
@@ -348,7 +353,7 @@ class PipeParallel(PipeController):
data_html=output_stuff.content.rendered_pretty_html() if graph_context.data_inclusion.stuff_html_content else None,
)
tracer_manager.register_controller_output(
- graph_id=graph_context.graph_id,
+ lookup_key=graph_context.lookup_key,
node_id=graph_context.parent_node_id,
output_spec=output_spec,
)
@@ -377,7 +382,7 @@ class PipeParallel(PipeController):
return
branch_stuff_codes = [stuff.stuff_code for stuff in branch_stuffs.values()]
tracer_manager.register_parallel_combine(
- graph_id=graph_context.graph_id,
+ lookup_key=graph_context.lookup_key,
combined_stuff_code=combined_stuff.stuff_code,
branch_stuff_codes=branch_stuff_codes,
parallel_controller_node_id=graph_context.parent_node_id,
diff --git a/pipelex/pipe_controllers/pipe_controller.py b/pipelex/pipe_controllers/pipe_controller.py
index 444fc15c8..f6d36c919 100644
--- a/pipelex/pipe_controllers/pipe_controller.py
+++ b/pipelex/pipe_controllers/pipe_controller.py
@@ -1,5 +1,5 @@
from abc import abstractmethod
-from typing import Literal, final
+from typing import TYPE_CHECKING, Literal, final
from typing_extensions import override
@@ -9,6 +9,9 @@ from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipeline.job_metadata import JobMetadata
+if TYPE_CHECKING:
+ from pipelex.libraries.library_crate import LibraryCrate
+
class PipeController(PipeAbstract):
pipe_category: Literal["PipeController"] = "PipeController"
@@ -33,18 +36,32 @@ class PipeController(PipeAbstract):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
return await self._live_run_controller_pipe(
- job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ library_crate=library_crate,
)
@final
@override
async def _dry_run_pipe(
- self, job_metadata: JobMetadata, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None = None
+ self,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
return await self._dry_run_controller_pipe(
- job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ library_crate=library_crate,
)
@abstractmethod
@@ -54,6 +71,7 @@ class PipeController(PipeAbstract):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
pass
@@ -64,5 +82,6 @@ class PipeController(PipeAbstract):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
pass
diff --git a/pipelex/pipe_controllers/sequence/pipe_sequence.py b/pipelex/pipe_controllers/sequence/pipe_sequence.py
index cb5a85d09..b92f6f8a2 100644
--- a/pipelex/pipe_controllers/sequence/pipe_sequence.py
+++ b/pipelex/pipe_controllers/sequence/pipe_sequence.py
@@ -1,4 +1,4 @@
-from typing import Any, Literal
+from typing import TYPE_CHECKING, Any, Literal
from pydantic import field_validator
from typing_extensions import override
@@ -20,6 +20,9 @@ from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.misc.string_utils import get_root_from_dotted_path
+if TYPE_CHECKING:
+ from pipelex.libraries.library_crate import LibraryCrate
+
class PipeSequence(PipeController):
type: Literal["PipeSequence"] = "PipeSequence"
@@ -178,6 +181,7 @@ class PipeSequence(PipeController):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
evolving_memory = working_memory
@@ -192,6 +196,7 @@ class PipeSequence(PipeController):
working_memory=evolving_memory,
job_metadata=job_metadata,
sub_pipe_run_params=sub_pipe_run_params,
+ library_crate=library_crate,
)
evolving_memory = pipe_output.working_memory
# Capture execution data for the graph tracer
@@ -212,12 +217,14 @@ class PipeSequence(PipeController):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
return await self._live_run_controller_pipe(
job_metadata=job_metadata,
working_memory=working_memory,
pipe_run_params=pipe_run_params,
output_name=output_name,
+ library_crate=library_crate,
)
@override
diff --git a/pipelex/pipe_controllers/sub_pipe.py b/pipelex/pipe_controllers/sub_pipe.py
index dfdb503e1..6c2a4b7d4 100644
--- a/pipelex/pipe_controllers/sub_pipe.py
+++ b/pipelex/pipe_controllers/sub_pipe.py
@@ -1,4 +1,4 @@
-from typing import cast
+from typing import TYPE_CHECKING, cast
from pydantic import BaseModel
@@ -12,14 +12,18 @@ from pipelex.core.pipes.variable_multiplicity import VariableMultiplicity
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
-from pipelex.hub import get_required_pipe
+from pipelex.hub import get_pipe_router, get_required_pipe
from pipelex.pipe_controllers.batch.pipe_batch import PipeBatch
from pipelex.pipe_controllers.batch.pipe_batch_blueprint import PipeBatchBlueprint
from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition
+from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
from pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunParams
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.misc.string_utils import get_root_from_dotted_path
+if TYPE_CHECKING:
+ from pipelex.libraries.library_crate import LibraryCrate
+
class SubPipe(BaseModel):
pipe_code: str
@@ -33,6 +37,7 @@ class SubPipe(BaseModel):
working_memory: WorkingMemory,
job_metadata: JobMetadata,
sub_pipe_run_params: PipeRunParams,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
if self.output_multiplicity:
sub_pipe_run_params.output_multiplicity = self.output_multiplicity
@@ -171,11 +176,15 @@ class SubPipe(BaseModel):
concept_codes_from_the_same_domain=[concept.code for concept in sub_pipe.concept_dependencies],
)
try:
- pipe_output = await pipe_batch.run_pipe(
- job_metadata=job_metadata,
- working_memory=working_memory,
- pipe_run_params=sub_pipe_run_params,
- output_name=self.output_name,
+ pipe_output = await get_pipe_router().run(
+ pipe_job=PipeJobFactory.make_pipe_job(
+ pipe=pipe_batch,
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=sub_pipe_run_params,
+ output_name=self.output_name,
+ library_crate=library_crate,
+ ),
)
finally:
# Clean up synthetic stuff injected for dotted-path resolution
@@ -183,11 +192,15 @@ class SubPipe(BaseModel):
working_memory.remove_stuff(name=synthetic_flat_name)
# Case 2: Condition processing
elif isinstance(sub_pipe, PipeCondition):
- pipe_output = await sub_pipe.run_pipe(
- job_metadata=job_metadata,
- working_memory=working_memory,
- pipe_run_params=sub_pipe_run_params,
- output_name=self.output_name,
+ pipe_output = await get_pipe_router().run(
+ pipe_job=PipeJobFactory.make_pipe_job(
+ pipe=sub_pipe,
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=sub_pipe_run_params,
+ output_name=self.output_name,
+ library_crate=library_crate,
+ ),
)
else:
# Case 3: Normal processing
@@ -213,11 +226,15 @@ class SubPipe(BaseModel):
concept_code=None,
) from exc
log.verbose(required_stuffs, title=f"Required stuffs for {self.pipe_code}")
- pipe_output = await sub_pipe.run_pipe(
- job_metadata=job_metadata,
- working_memory=working_memory,
- pipe_run_params=sub_pipe_run_params,
- output_name=self.output_name,
+ pipe_output = await get_pipe_router().run(
+ pipe_job=PipeJobFactory.make_pipe_job(
+ pipe=sub_pipe,
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ pipe_run_params=sub_pipe_run_params,
+ output_name=self.output_name,
+ library_crate=library_crate,
+ ),
)
return pipe_output
diff --git a/pipelex/pipe_operators/compose/pipe_compose.py b/pipelex/pipe_operators/compose/pipe_compose.py
index f6869d9d2..dc4abe5be 100644
--- a/pipelex/pipe_operators/compose/pipe_compose.py
+++ b/pipelex/pipe_operators/compose/pipe_compose.py
@@ -1,5 +1,6 @@
from typing import Any, Literal
+from pydantic import Field
from typing_extensions import override
from pipelex import log
@@ -7,6 +8,7 @@ from pipelex.cogt.content_generation.content_generator_dry import ContentGenerat
from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol
from pipelex.cogt.content_generation.dry_run_factory import DryRunFactory
from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.cogt.templating.template_rendering import render_template
from pipelex.cogt.templating.templating_style import TemplatingStyle
from pipelex.config import get_config
from pipelex.core.concepts.native.concept_native import NativeConceptCode
@@ -40,7 +42,7 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
# Template mode fields (used when template is provided)
template: str | None = None
templating_style: TemplatingStyle | None = None
- category: TemplateCategory = TemplateCategory.BASIC
+ category: TemplateCategory = Field(default=TemplateCategory.BASIC, strict=False)
extra_context: dict[str, Any] | None = None
# Construct mode field (used when construct is provided)
@@ -163,7 +165,6 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
working_memory=working_memory,
pipe_run_params=pipe_run_params,
output_name=output_name,
- content_generator=content_generator,
)
async def _run_template_mode(
@@ -172,7 +173,6 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None,
- content_generator: ContentGeneratorProtocol,
) -> PipeComposeOutput:
"""Run PipeCompose in template mode (produces Text or Html output)."""
if self.template is None:
@@ -185,14 +185,14 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
if self.extra_context:
context.update(**self.extra_context)
- jinja2_text = await content_generator.make_templated_text(
- context=context,
+ rendered_template_text = await render_template(
template=self.template,
+ category=self.category,
+ context=context,
templating_style=self.templating_style,
- template_category=self.category,
)
- log.verbose(f"Jinja2 rendered text:\n{jinja2_text}")
- assert isinstance(jinja2_text, str)
+ log.verbose(f"Template rendered text:\n{rendered_template_text}")
+ assert isinstance(rendered_template_text, str)
# Get the structure class from the registry (might be a subclass of TextContent or HtmlContent)
structure_class = get_class_registry().get_required_subclass(
@@ -202,9 +202,9 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
# Construct content based on the structure class type
if issubclass(structure_class, HtmlContent):
- the_content = structure_class(inner_html=jinja2_text, css_class="")
+ the_content = structure_class(inner_html=rendered_template_text, css_class="")
else:
- the_content = structure_class(text=jinja2_text)
+ the_content = structure_class(text=rendered_template_text)
output_stuff = StuffFactory.make_stuff(concept=self.output.concept, content=the_content, name=output_name)
@@ -216,7 +216,7 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
# Capture execution data for the graph tracer
execution_data_dict: dict[str, Any] = {
"compose_mode": "template",
- "rendered_text": jinja2_text,
+ "rendered_text": rendered_template_text,
}
self._register_execution_data(job_metadata, execution_data_dict)
@@ -269,9 +269,9 @@ class PipeCompose(PipeOperator[PipeComposeOutput]):
name=output_name,
)
- # Capture execution data for the graph tracer
execution_data_dict: dict[str, Any] = {
"compose_mode": "construct",
+ "fields": composer.field_resolutions,
}
self._register_execution_data(job_metadata, execution_data_dict)
diff --git a/pipelex/pipe_operators/compose/pipe_compose_blueprint.py b/pipelex/pipe_operators/compose/pipe_compose_blueprint.py
index fb2b41e21..5ba0d5769 100644
--- a/pipelex/pipe_operators/compose/pipe_compose_blueprint.py
+++ b/pipelex/pipe_operators/compose/pipe_compose_blueprint.py
@@ -31,14 +31,23 @@ class PipeComposeBlueprint(PipeBlueprint):
@model_validator(mode="before")
@classmethod
def validate_template_or_construct(cls, values: dict[str, Any]) -> dict[str, Any]:
- """Validate that exactly one of template or construct is provided."""
+ """Validate that exactly one of template or construct is provided.
+
+ Handles two deserialization paths:
+ - MTHDS loading: 'construct' key with raw dict data → needs make_from_raw
+ - Kajson round-trip: 'construct_blueprint' key with ConstructBlueprint object
+ (Kajson serializes using field names, not aliases)
+ """
has_template = values.get("template") is not None
construct_raw = values.get("construct")
+ construct_bp = values.get("construct_blueprint")
+
+ has_construct = construct_raw is not None or construct_bp is not None
- if not has_template and construct_raw is None:
+ if not has_template and not has_construct:
msg = "PipeComposeBlueprint requires either 'template' or 'construct' to be provided"
raise ValueError(msg)
- if has_template and construct_raw is not None:
+ if has_template and has_construct:
msg = "PipeComposeBlueprint cannot have both 'template' and 'construct' - use one or the other"
raise ValueError(msg)
diff --git a/pipelex/pipe_operators/compose/structured_content_composer.py b/pipelex/pipe_operators/compose/structured_content_composer.py
index 04bff8498..55502ef81 100644
--- a/pipelex/pipe_operators/compose/structured_content_composer.py
+++ b/pipelex/pipe_operators/compose/structured_content_composer.py
@@ -7,7 +7,7 @@ from pydantic import ValidationError
from pipelex import log
from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol
from pipelex.cogt.templating.template_category import TemplateCategory
-from pipelex.cogt.templating.template_preprocessor import preprocess_template
+from pipelex.cogt.templating.template_rendering import render_template
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_content import StuffContent
@@ -60,6 +60,10 @@ class StructuredContentComposer:
self.extra_context = extra_context or {}
self.content_generator = content_generator or get_content_generator()
self.pipe_run_params = pipe_run_params
+ # Per-field record of how each field was built. Populated as fields resolve.
+ # Shape per entry: {"method": ConstructFieldMethod, "rendered": str (templates only)}.
+ # Nested fields record only their method; their sub-fields are not surfaced.
+ self.field_resolutions: dict[str, dict[str, Any]] = {}
async def compose(self) -> StuffContent:
"""Compose the StructuredContent asynchronously.
@@ -127,6 +131,7 @@ class StructuredContentComposer:
Returns:
The resolved value for the field
"""
+ self.field_resolutions[field_name] = {"method": field_blueprint.method}
match field_blueprint.method:
case ConstructFieldMethod.FIXED:
return field_blueprint.fixed_value
@@ -135,7 +140,9 @@ class StructuredContentComposer:
return self._resolve_from_var(field_blueprint=field_blueprint, field_name=field_name)
case ConstructFieldMethod.TEMPLATE:
- return await self._resolve_template(field_blueprint=field_blueprint)
+ rendered_text = await self._resolve_template(field_blueprint=field_blueprint)
+ self.field_resolutions[field_name]["rendered"] = rendered_text
+ return rendered_text
case ConstructFieldMethod.NESTED:
return await self._resolve_nested(field_blueprint=field_blueprint, field_name=field_name)
@@ -650,14 +657,11 @@ class StructuredContentComposer:
if self.extra_context:
context.update(**self.extra_context)
- # Preprocess the template (handles $ -> {{ }} conversion)
- preprocessed = preprocess_template(field_blueprint.template)
-
- # Render the template using the provided content generator (supports dry run mode)
- return await self.content_generator.make_templated_text(
+ # TODO: dry-run templating is being removed — this direct render_template call is intentional
+ return await render_template(
+ template=field_blueprint.template,
+ category=TemplateCategory.BASIC,
context=context,
- template=preprocessed,
- template_category=TemplateCategory.BASIC,
)
async def _resolve_nested(self, field_blueprint: ConstructFieldBlueprint, field_name: str) -> StuffContent:
diff --git a/pipelex/pipe_operators/func/pipe_func.py b/pipelex/pipe_operators/func/pipe_func.py
index 45b0cc999..39f33d4e4 100644
--- a/pipelex/pipe_operators/func/pipe_func.py
+++ b/pipelex/pipe_operators/func/pipe_func.py
@@ -1,8 +1,7 @@
import asyncio
import inspect
-from typing import Literal, cast, get_args, get_origin, get_type_hints
+from typing import Any, Literal, cast, get_args, get_origin, get_type_hints
-from kajson.kajson_manager import KajsonManager
from pydantic import field_validator
from typing_extensions import override
@@ -16,6 +15,7 @@ from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_class_registry
from pipelex.pipe_operators.pipe_operator import PipeOperator
from pipelex.pipe_run.exceptions import PipeRunError
from pipelex.pipe_run.pipe_run_params import PipeRunParams
@@ -101,7 +101,7 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):
raise TypeError(msg)
# Validate that the function's return type matches the concept's structure class
- concept_structure_class = KajsonManager.get_class_registry().get_class(name=self.output.concept.structure_class_name)
+ concept_structure_class = get_class_registry().get_class(name=self.output.concept.structure_class_name)
if concept_structure_class is None:
msg = (
f"PipeFunc '{self.code}' failed to validate output with library: "
@@ -229,6 +229,18 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):
name=output_name,
)
+ # Capture execution data for the graph tracer. PipeFunc has no prompts or
+ # models to record, but the sidepanel should still show *what* function ran
+ # and what kind of content it returned — useful for debugging and for
+ # distinguishing multiple PipeFunc nodes in a graph.
+ execution_data_dict: dict[str, Any] = {
+ "function_name": self.function_name,
+ "function_module": getattr(function, "__module__", None),
+ "function_qualname": getattr(function, "__qualname__", self.function_name),
+ "output_content_type": type(the_content).__name__,
+ }
+ self._register_execution_data(job_metadata, execution_data_dict)
+
return PipeFuncOutput(
working_memory=working_memory,
pipeline_run_id=job_metadata.pipeline_run_id,
@@ -277,6 +289,18 @@ class PipeFunc(PipeOperator[PipeFuncOutput]):
name=output_name,
)
+ # Capture execution data for the graph tracer. Dry run does NOT invoke the
+ # real function — we flag that explicitly so the sidepanel can show a
+ # "mock output" indicator and explain why no real function_result is present.
+ execution_data_dict: dict[str, Any] = {
+ "function_name": self.function_name,
+ "function_module": getattr(function, "__module__", None),
+ "function_qualname": getattr(function, "__qualname__", self.function_name),
+ "output_content_type": type(mock_content).__name__,
+ "is_mock_output": True,
+ }
+ self._register_execution_data(job_metadata, execution_data_dict)
+
return PipeFuncOutput(
working_memory=working_memory,
pipeline_run_id=job_metadata.pipeline_run_id,
diff --git a/pipelex/pipe_operators/llm/helpers.py b/pipelex/pipe_operators/llm/helpers.py
index 4d7b39a9e..81eb48557 100644
--- a/pipelex/pipe_operators/llm/helpers.py
+++ b/pipelex/pipe_operators/llm/helpers.py
@@ -1,9 +1,9 @@
from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.cogt.templating.template_rendering import render_template
from pipelex.config import get_config
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.hub import (
get_class_registry,
- get_content_generator,
get_required_concept,
)
from pipelex.tools.typing.structure_printer import StructurePrinter
@@ -26,10 +26,10 @@ async def get_output_structure_prompt(concept_ref: str, is_with_preliminary_text
else:
template_source = llm_config.get_template(template_name="output_structure_prompt_no_preliminary_text")
- return await get_content_generator().make_templated_text(
+ return await render_template(
+ template=template_source,
+ category=TemplateCategory.LLM_PROMPT,
context={
"class_structure_str": class_structure_str,
},
- template=template_source,
- template_category=TemplateCategory.LLM_PROMPT,
)
diff --git a/pipelex/pipe_operators/pipe_operator.py b/pipelex/pipe_operators/pipe_operator.py
index eff33318e..7dc8ede41 100644
--- a/pipelex/pipe_operators/pipe_operator.py
+++ b/pipelex/pipe_operators/pipe_operator.py
@@ -14,6 +14,7 @@ from pipelex.pipeline.job_metadata import JobMetadata
if TYPE_CHECKING:
from pipelex.core.stuffs.list_content import ListContent
from pipelex.core.stuffs.stuff_content import StuffContent
+ from pipelex.libraries.library_crate import LibraryCrate
PipeOperatorOutputType = TypeVar("PipeOperatorOutputType", bound=PipeOutput)
@@ -33,6 +34,7 @@ class PipeOperator(PipeAbstract, Generic[PipeOperatorOutputType]):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
try:
pipe_output = await self._live_run_operator_pipe(
@@ -82,7 +84,12 @@ class PipeOperator(PipeAbstract, Generic[PipeOperatorOutputType]):
@final
@override
async def _dry_run_pipe(
- self, job_metadata: JobMetadata, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None = None
+ self,
+ job_metadata: JobMetadata,
+ working_memory: WorkingMemory,
+ pipe_run_params: PipeRunParams,
+ output_name: str | None = None,
+ library_crate: "LibraryCrate | None" = None,
) -> PipeOutput:
return await self._dry_run_operator_pipe(
job_metadata=job_metadata, working_memory=working_memory, pipe_run_params=pipe_run_params, output_name=output_name
diff --git a/pipelex/pipe_run/delivery_assignment.py b/pipelex/pipe_run/delivery_assignment.py
new file mode 100644
index 000000000..2312cfec8
--- /dev/null
+++ b/pipelex/pipe_run/delivery_assignment.py
@@ -0,0 +1,45 @@
+from __future__ import annotations
+
+from typing import Any
+
+from pydantic import BaseModel, Field, field_validator
+
+from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+from pipelex.types import StrEnum
+
+
+class DeliveryStatus(StrEnum):
+ COMPLETED = "COMPLETED"
+ FAILED = "FAILED"
+
+
+class WebhookTarget(BaseModel):
+ """A webhook endpoint to notify on delivery."""
+
+ url: str
+ headers: dict[str, str] = Field(default_factory=dict)
+ payload: dict[str, Any] = Field(default_factory=dict)
+
+
+class StorageTarget(BaseModel):
+ """Storage configuration for persisting pipe output."""
+
+ key_prefix: str | None = None
+
+ @field_validator("key_prefix", mode="after")
+ @classmethod
+ def normalize_key_prefix(cls, value: str | None) -> str | None:
+ if value is not None and value != "" and not value.endswith("/"):
+ return f"{value}/"
+ return value
+
+
+class DeliveryAssignment(BaseModel):
+ """Configures the full delivery behavior for a pipe run.
+
+ Execution order: storage first (persist output), then webhooks (notify consumers).
+ The result_url from storage is automatically injected into webhook payloads.
+ """
+
+ storage: StorageTarget | None = None
+ webhooks: list[WebhookTarget] = Field(default_factory=empty_list_factory_of(WebhookTarget))
diff --git a/pipelex/pipe_run/delivery_executor.py b/pipelex/pipe_run/delivery_executor.py
new file mode 100644
index 000000000..025c32b44
--- /dev/null
+++ b/pipelex/pipe_run/delivery_executor.py
@@ -0,0 +1,268 @@
+from __future__ import annotations
+
+import html
+from typing import TYPE_CHECKING, Any, Awaitable, NamedTuple, cast
+
+import httpx
+from kajson.exceptions import KajsonException
+from pydantic import ValidationError
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.core.concepts.concept import Concept
+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.stuff_content import StuffContent
+from pipelex.core.stuffs.stuff_viewer import render_stuff_viewer
+from pipelex.graph.graph_factory import generate_graph_outputs
+from pipelex.hub import get_class_registry, get_storage_provider
+from pipelex.pipe_run.exceptions import PipeJobError, StorageDeliveryError, WebhookDeliveryError
+from pipelex.temporal.tprl_pipe.hydration import hydrate_content
+from pipelex.tools.misc.json_utils import clean_json_dumps
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+ from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus, StorageTarget, WebhookTarget
+
+
+class ResultFile(NamedTuple):
+ """A generated result file with its bytes and MIME type, ready to store."""
+
+ data: bytes
+ content_type: str
+
+
+class DeliveryExecutor:
+ """Handles the full delivery lifecycle: generate result files, store them, notify webhooks."""
+
+ async def execute(
+ self,
+ pipe_output: PipeOutput | None,
+ user_id: str,
+ pipeline_run_id: str,
+ delivery_assignment: DeliveryAssignment,
+ status: DeliveryStatus,
+ ) -> None:
+ """Execute a full delivery: generate result files, store them, then notify webhooks."""
+ # Step 1: Persist the result files to storage (only on success with output)
+ result_url: str | None = None
+ if delivery_assignment.storage is not None and pipe_output is not None:
+ result_url = await self._store_results(pipe_output, user_id, pipeline_run_id, delivery_assignment.storage)
+
+ # Step 2: Notify webhooks with status + result_url (always, even on failure)
+ for webhook in delivery_assignment.webhooks:
+ await self._notify_webhook(pipeline_run_id, status, result_url, webhook)
+
+ # ---- Result file generation ----
+
+ async def generate_result_files(self, pipe_output: PipeOutput) -> dict[str, ResultFile]:
+ """Generate the full set of result files from a PipeOutput.
+
+ Produces: working_memory.json, main_stuff (json/md/html/viewer),
+ graph outputs (mermaidflow, reactflow, graphspec).
+
+ Supports two `pipe_output` shapes:
+ - Typed: `working_memory` populated (in-process / same-worker path).
+ - Raw: `working_memory_raw` populated (cross-process Temporal path,
+ where the activity worker may not have the dynamic concept classes
+ loaded). The activity tries to locally hydrate using only globally
+ registered classes; on failure it falls back to a generic dict
+ render so built-in content types still get typed rendering and
+ dynamic concepts produce a readable JSON dump.
+ """
+ files: dict[str, ResultFile] = {}
+
+ if pipe_output.working_memory_raw is not None:
+ files["working_memory.json"] = ResultFile(
+ data=clean_json_dumps(pipe_output.working_memory_raw, indent=2).encode("utf-8"),
+ content_type="application/json",
+ )
+ raw_main_stuff = self._get_raw_main_stuff_dict(pipe_output.working_memory_raw)
+ main_stuff = self.try_local_hydrate_stuff(raw_main_stuff) if raw_main_stuff is not None else None
+ if main_stuff is not None:
+ await self._generate_main_stuff_files(main_stuff, files)
+ elif raw_main_stuff is not None:
+ self._generate_main_stuff_files_from_raw(raw_main_stuff, files)
+ else:
+ files["working_memory.json"] = ResultFile(
+ data=clean_json_dumps(pipe_output.working_memory.smart_dump(), indent=2).encode("utf-8"),
+ content_type="application/json",
+ )
+ main_stuff = pipe_output.working_memory.get_optional_main_stuff()
+ if main_stuff is not None:
+ await self._generate_main_stuff_files(main_stuff, files)
+
+ graph_spec = pipe_output.graph_spec
+ if graph_spec:
+ await self._generate_graph_files(graph_spec, files)
+
+ return files
+
+ @classmethod
+ def _get_raw_main_stuff_dict(cls, working_memory_raw: dict[str, Any]) -> dict[str, Any] | None:
+ """Extract the main stuff dict from a raw working_memory, following aliases."""
+ root: dict[str, Any] = working_memory_raw.get("root", {})
+ aliases: dict[str, str] = working_memory_raw.get("aliases", {})
+ target_name = aliases.get(MAIN_STUFF_NAME, MAIN_STUFF_NAME)
+ candidate = root.get(target_name)
+ if isinstance(candidate, dict):
+ return cast("dict[str, Any]", candidate)
+ return None
+
+ @classmethod
+ def try_local_hydrate_stuff(cls, stuff_raw: dict[str, Any]) -> Stuff | None:
+ """Attempt to hydrate a single Stuff dict using only globally-registered classes.
+
+ Returns None when the structure class isn't available locally (typically
+ a dynamic concept class missing from the activity worker's registry) —
+ callers then fall back to a generic raw-dict render. A warning is
+ emitted on the fallback path so silent regressions on built-in
+ hydration surface in logs.
+ """
+ try:
+ concept = Concept.model_validate(stuff_raw["concept"])
+ registry = get_class_registry()
+ item_class = registry.get_class(name=concept.structure_class_name)
+ if item_class is None or not issubclass(item_class, StuffContent):
+ log.warning(
+ f"Local hydration failed for delivery main stuff, falling back to raw render: "
+ f"class '{concept.structure_class_name}' not registered locally"
+ )
+ return None
+ content = hydrate_content(concept=concept, raw_content=stuff_raw["content"])
+ return Stuff(
+ stuff_code=stuff_raw["stuff_code"],
+ stuff_name=stuff_raw.get("stuff_name"),
+ concept=concept,
+ content=content,
+ )
+ except (PipeJobError, ValidationError, KajsonException, KeyError, TypeError) as exc:
+ log.warning(f"Local hydration failed for delivery main stuff, falling back to raw render: {exc}")
+ return None
+
+ @classmethod
+ def _generate_main_stuff_files_from_raw(cls, raw_main_stuff: dict[str, Any], files: dict[str, ResultFile]) -> None:
+ """Generic fallback rendering of a main stuff that we couldn't locally hydrate.
+
+ Produces JSON-in-markdown and `<pre>`-in-HTML so the user still gets
+ readable output for dynamic concepts the activity worker doesn't know about.
+ """
+ content_dict = raw_main_stuff.get("content", raw_main_stuff)
+ json_text = clean_json_dumps(content_dict, indent=2)
+ files["main_stuff.json"] = ResultFile(data=json_text.encode("utf-8"), content_type="application/json")
+ files["main_stuff.md"] = ResultFile(data=f"```json\n{json_text}\n```\n".encode(), content_type="text/markdown")
+ # Escape HTML-special chars: json.dumps does not escape <, >, &, so embedding
+ # raw user-controlled JSON inside <pre> would allow stored XSS via strings
+ # like "</pre><script>...</script>" in pipeline outputs.
+ files["main_stuff.html"] = ResultFile(data=f"<pre>{html.escape(json_text)}</pre>".encode(), content_type="text/html")
+
+ async def _generate_main_stuff_files(self, main_stuff: Stuff, files: dict[str, ResultFile]) -> None:
+ content = main_stuff.content
+ await self._try_add_rendered_file(files, "main_stuff.json", content.rendered_json_async(), "application/json")
+ await self._try_add_rendered_file(files, "main_stuff.md", content.rendered_markdown_async(), "text/markdown")
+ await self._try_add_rendered_file(files, "main_stuff.html", content.rendered_html_async(), "text/html")
+ await self._try_add_rendered_file(files, "main_stuff_viewer.html", render_stuff_viewer(main_stuff), "text/html")
+
+ async def _generate_graph_files(self, graph_spec: Any, files: dict[str, ResultFile]) -> None:
+ try:
+ graph_config = get_config().pipelex.pipeline_execution_config.graph_config
+ graph_outputs = await generate_graph_outputs(
+ graph_spec=graph_spec,
+ graph_config=graph_config,
+ )
+ self._add_optional_text_file(files, "graphspec.json", graph_outputs.graphspec_json, "application/json")
+ self._add_optional_text_file(files, "mermaidflow.mmd", graph_outputs.mermaidflow_mmd, "text/plain")
+ self._add_optional_text_file(files, "mermaidflow.html", graph_outputs.mermaidflow_html, "text/html")
+ self._add_optional_text_file(files, "reactflow.html", graph_outputs.reactflow_html, "text/html")
+ except Exception:
+ # TODO: wip - do not catch all exceptions
+ log.warning("Failed to generate graph outputs")
+
+ @classmethod
+ async def _try_add_rendered_file(
+ cls,
+ files: dict[str, ResultFile],
+ filename: str,
+ render: Awaitable[str],
+ content_type: str,
+ ) -> None:
+ """Await a render coroutine and store the encoded result; log a warning on failure."""
+ try:
+ text = await render
+ except Exception:
+ # TODO: wip - do not catch all exceptions
+ log.warning(f"Failed to render {filename}")
+ return
+ files[filename] = ResultFile(data=text.encode("utf-8"), content_type=content_type)
+
+ @classmethod
+ def _add_optional_text_file(cls, files: dict[str, ResultFile], filename: str, text: str | None, content_type: str) -> None:
+ """Encode `text` and store it under `filename`. No-op when `text` is None."""
+ if text is not None:
+ files[filename] = ResultFile(data=text.encode("utf-8"), content_type=content_type)
+
+ # ---- Storage ----
+
+ async def _store_results(
+ self,
+ pipe_output: PipeOutput,
+ user_id: str,
+ pipeline_run_id: str,
+ storage: StorageTarget,
+ ) -> str:
+ """Generate all result files and store them. Returns the base result URL."""
+ try:
+ storage_provider = get_storage_provider()
+ prefix: str = storage.key_prefix or ""
+ base_key: str = f"{user_id}/{prefix}{pipeline_run_id}"
+
+ result_files = await self.generate_result_files(pipe_output)
+
+ for filename, result_file in result_files.items():
+ key: str = f"{base_key}/{filename}"
+ await storage_provider.store(data=result_file.data, key=key, content_type=result_file.content_type)
+ log.debug(f"Stored: {key}")
+
+ # TODO: include the full S3 URI (s3://bucket/key/) so result_url is
+ # self-contained and doesn't depend on knowing the bucket externally.
+ result_url: str = f"{base_key}/"
+ log.info(f"Storage delivery completed: pipeline_run_id={pipeline_run_id}, files={len(result_files)}")
+ return result_url
+ except Exception as exc:
+ # TODO: wip - do not catch all exceptions
+ msg = f"Storage delivery failed for pipeline_run_id={pipeline_run_id}"
+ raise StorageDeliveryError(msg) from exc
+
+ # ---- Webhooks ----
+
+ async def _notify_webhook(
+ self,
+ pipeline_run_id: str,
+ status: DeliveryStatus,
+ result_url: str | None,
+ webhook: WebhookTarget,
+ ) -> None:
+ """POST status + result_url to a webhook URL."""
+ try:
+ payload: dict[str, Any] = dict(webhook.payload)
+ payload["pipeline_run_id"] = pipeline_run_id
+ payload["status"] = status
+ if result_url is not None:
+ payload["result_url"] = result_url
+
+ async with httpx.AsyncClient() as client:
+ response = await client.post(
+ webhook.url,
+ json=payload,
+ headers=webhook.headers,
+ timeout=30.0,
+ )
+ response.raise_for_status()
+ log.info(f"Webhook delivery completed: pipeline_run_id={pipeline_run_id}, url={webhook.url}")
+ except httpx.HTTPStatusError as exc:
+ msg = f"Webhook delivery failed for pipeline_run_id={pipeline_run_id}: HTTP {exc.response.status_code}"
+ raise WebhookDeliveryError(msg) from exc
+ except Exception as exc:
+ # TODO: wip - do not catch all exceptions
+ msg = f"Webhook delivery failed for pipeline_run_id={pipeline_run_id}: {exc}"
+ raise WebhookDeliveryError(msg) from exc
diff --git a/pipelex/pipe_run/dry_pipe_router.py b/pipelex/pipe_run/dry_pipe_router.py
new file mode 100644
index 000000000..b797e9e1b
--- /dev/null
+++ b/pipelex/pipe_run/dry_pipe_router.py
@@ -0,0 +1,25 @@
+from typing_extensions import override
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.observer.observer_protocol import ObserverNoOp, ObserverProtocol
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol
+
+
+class DryPipeRouter(PipeRouterProtocol):
+ def __init__(self, observer: ObserverProtocol | None = None):
+ self.observer = observer or ObserverNoOp()
+
+ @override
+ async def _run_pipe_job(
+ self,
+ pipe_job: PipeJob,
+ wfid: str | None = None,
+ ) -> PipeOutput:
+ return await pipe_job.pipe.dry_run_pipe(
+ job_metadata=pipe_job.job_metadata,
+ working_memory=pipe_job.get_working_memory(),
+ output_name=pipe_job.output_name,
+ pipe_run_params=pipe_job.pipe_run_params,
+ library_crate=pipe_job.library_crate,
+ )
diff --git a/pipelex/pipe_run/dry_run_pipeline.py b/pipelex/pipe_run/dry_run_pipeline.py
index ff1c21a11..ed07f16d3 100644
--- a/pipelex/pipe_run/dry_run_pipeline.py
+++ b/pipelex/pipe_run/dry_run_pipeline.py
@@ -8,6 +8,7 @@ from pipelex.base_exceptions import PipelexError
from pipelex.config import get_config
from pipelex.core.interpreter.exceptions import PipelexInterpreterError
from pipelex.core.interpreter.interpreter import PipelexInterpreter
+from pipelex.core.pipes.pipe_factory import PipeFactory
from pipelex.graph.graphspec import GraphSpec
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
from pipelex.pipeline.runner import PipelexRunner
@@ -53,7 +54,7 @@ async def dry_run_pipeline(
bundle_blueprint = PipelexInterpreter.make_pipelex_bundle_blueprint(mthds_content=content)
if bundle_blueprint.main_pipe and main_pipe_code is None:
# Domain-qualify to avoid ambiguity across multiple domains
- main_pipe_code = f"{bundle_blueprint.domain}.{bundle_blueprint.main_pipe}"
+ main_pipe_code = PipeFactory.make_pipe_ref_with_domain(domain_code=bundle_blueprint.domain, pipe_code=bundle_blueprint.main_pipe)
if not main_pipe_code:
msg = "Bundle does not declare a main_pipe, cannot generate graph"
diff --git a/pipelex/pipe_run/exceptions.py b/pipelex/pipe_run/exceptions.py
index 6bdbb538d..ea21a6a72 100644
--- a/pipelex/pipe_run/exceptions.py
+++ b/pipelex/pipe_run/exceptions.py
@@ -17,6 +17,22 @@ class PipeRunError(PipelexError):
super().__init__(message)
+class PipeJobError(PipelexError):
+ pass
+
+
+class DeliveryError(PipelexError):
+ pass
+
+
+class WebhookDeliveryError(DeliveryError):
+ pass
+
+
+class StorageDeliveryError(DeliveryError):
+ pass
+
+
class PipeRouterError(PipelexError):
def __init__(
self,
diff --git a/pipelex/pipe_run/graph_assembly.py b/pipelex/pipe_run/graph_assembly.py
new file mode 100644
index 000000000..14dd5db54
--- /dev/null
+++ b/pipelex/pipe_run/graph_assembly.py
@@ -0,0 +1,67 @@
+"""Shared graph assembly logic for PipeRun implementations.
+
+After pipe execution, assembles the full GraphSpec from trace events.
+Works with any event log backend (NDJSON, DynamoDB, etc.) — the backend
+is determined by the tracing config, not by the caller.
+"""
+
+import json
+
+from pydantic import ValidationError
+
+from pipelex import log
+from pipelex.base_exceptions import PipelexConfigError
+from pipelex.config import get_config
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.graph.graphspec import PipelineRef
+from pipelex.system.exceptions import MissingDependencyError
+from pipelex.tracing.event_log_factory import make_event_log
+from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
+
+
+def assemble_graph_on_output(
+ pipe_output: PipeOutput,
+ pipeline_run_id: str,
+ domain_code: str | None = None,
+ main_pipe_code: str | None = None,
+) -> None:
+ """Assemble the full graph from trace events and set it on pipe_output.
+
+ Reads all events for the pipeline_run_id from the configured event log
+ backend and uses GraphSpecAssembler to build the complete cross-worker graph.
+ Falls back to whatever graph_spec is already on pipe_output if assembly fails.
+
+ Tracing is observability and treated as best-effort: runtime I/O issues,
+ malformed event data, and broken tracing infrastructure (config errors,
+ missing optional dependencies) are caught and warned about, not propagated.
+ Programming bugs (KeyError, AttributeError, etc.) propagate so they surface
+ during development.
+
+ Args:
+ pipe_output: The pipe output to set graph_spec on.
+ pipeline_run_id: The pipeline run ID to query events for.
+ domain_code: Domain code for the pipeline ref.
+ main_pipe_code: Main pipe code for the pipeline ref.
+ """
+ tracing_config = get_config().pipelex.tracing_config
+ if not tracing_config.is_enabled:
+ return
+
+ try:
+ event_log = make_event_log(tracing_config)
+ try:
+ events = event_log.read_events(pipeline_run_id)
+ if events:
+ pipe_output.graph_spec = GraphSpecAssembler.assemble(
+ events=events,
+ graph_id=pipeline_run_id,
+ pipeline_ref=PipelineRef(
+ domain=domain_code,
+ main_pipe=main_pipe_code,
+ ),
+ )
+ log.debug(f"Graph assembled from {len(events)} events for pipeline_run_id={pipeline_run_id}")
+ finally:
+ event_log.close()
+ except (OSError, json.JSONDecodeError, ValidationError, PipelexConfigError, MissingDependencyError) as graph_assembly_error:
+ log.warning(f"Graph assembly failed, using existing graph: {graph_assembly_error}")
diff --git a/pipelex/pipe_run/pipe_job.py b/pipelex/pipe_run/pipe_job.py
index 7bd33cf65..1c533bd76 100644
--- a/pipelex/pipe_run/pipe_job.py
+++ b/pipelex/pipe_run/pipe_job.py
@@ -1,18 +1,49 @@
-from pydantic import BaseModel, Field
+from typing import Any
+
+from pydantic import BaseModel
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.pipe_abstract import PipeAbstract
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.pipe_run.exceptions import PipeJobError
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipeline.job_metadata import JobMetadata
class PipeJob(BaseModel):
pipe: PipeAbstract
- working_memory: WorkingMemory = Field(default_factory=WorkingMemory)
+ working_memory: WorkingMemory | None = None
+ working_memory_raw: dict[str, Any] | None = None
pipe_run_params: PipeRunParams
job_metadata: JobMetadata
output_name: str | None = None
+ library_crate: LibraryCrate | None = None
@property
def pipe_type(self) -> str:
return self.pipe.__class__.__name__
+
+ def get_working_memory(self) -> WorkingMemory:
+ if self.working_memory is not None:
+ return self.working_memory
+ if self.working_memory_raw is not None:
+ msg = "WorkingMemory is in raw form and has not been hydrated yet"
+ raise PipeJobError(msg)
+ return WorkingMemory()
+
+ def prepare_for_temporal(self) -> "PipeJob":
+ """Return a PipeJob ready for Temporal dispatch, with WorkingMemory serialized to raw dict.
+
+ Returns a copy when mutation is needed, leaving the original unchanged.
+ Returns self when no transformation is required.
+ """
+ if self.library_crate is None:
+ return self
+ if self.working_memory is not None:
+ return self.model_copy(
+ update={
+ "working_memory_raw": self.working_memory.dump_for_temporal(),
+ "working_memory": None,
+ }
+ )
+ return self
diff --git a/pipelex/pipe_run/pipe_job_factory.py b/pipelex/pipe_run/pipe_job_factory.py
index 7c05cdf08..22bba9751 100644
--- a/pipelex/pipe_run/pipe_job_factory.py
+++ b/pipelex/pipe_run/pipe_job_factory.py
@@ -1,6 +1,7 @@
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
from pipelex.core.pipes.pipe_abstract import PipeAbstract
+from pipelex.libraries.library_crate import LibraryCrate
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
@@ -16,6 +17,7 @@ class PipeJobFactory:
pipe_run_params: PipeRunParams | None = None,
working_memory: WorkingMemory | None = None,
output_name: str | None = None,
+ library_crate: LibraryCrate | None = None,
) -> PipeJob:
working_memory = working_memory or WorkingMemoryFactory.make_empty()
if not pipe_run_params:
@@ -26,4 +28,5 @@ class PipeJobFactory:
pipe_run_params=pipe_run_params,
pipe=pipe,
output_name=output_name,
+ library_crate=library_crate,
)
diff --git a/pipelex/pipe_run/pipe_router.py b/pipelex/pipe_run/pipe_router.py
index 7db704620..6d0304b4b 100644
--- a/pipelex/pipe_run/pipe_router.py
+++ b/pipelex/pipe_run/pipe_router.py
@@ -14,10 +14,12 @@ class PipeRouter(PipeRouterProtocol):
async def _run_pipe_job(
self,
pipe_job: PipeJob,
+ wfid: str | None = None,
) -> PipeOutput:
return await pipe_job.pipe.run_pipe(
job_metadata=pipe_job.job_metadata,
- working_memory=pipe_job.working_memory,
+ working_memory=pipe_job.get_working_memory(),
output_name=pipe_job.output_name,
pipe_run_params=pipe_job.pipe_run_params,
+ library_crate=pipe_job.library_crate,
)
diff --git a/pipelex/pipe_run/pipe_router_protocol.py b/pipelex/pipe_run/pipe_router_protocol.py
index daf3d8b43..1112779e7 100644
--- a/pipelex/pipe_run/pipe_router_protocol.py
+++ b/pipelex/pipe_run/pipe_router_protocol.py
@@ -47,11 +47,12 @@ class PipeRouterProtocol(Protocol):
async def run(
self,
pipe_job: PipeJob,
+ wfid: str | None = None,
) -> PipeOutput:
await self._before_run(pipe_job)
try:
- pipe_output = await self._run_pipe_job(pipe_job)
+ pipe_output = await self._run_pipe_job(pipe_job, wfid=wfid)
except PipeRunError as exc:
await self._after_failing_run(pipe_job, exc)
raise PipeRouterError(
@@ -70,4 +71,5 @@ class PipeRouterProtocol(Protocol):
async def _run_pipe_job(
self,
pipe_job: PipeJob,
+ wfid: str | None = None,
) -> PipeOutput: ...
diff --git a/pipelex/pipe_run/pipe_run.py b/pipelex/pipe_run/pipe_run.py
new file mode 100644
index 000000000..38834e151
--- /dev/null
+++ b/pipelex/pipe_run/pipe_run.py
@@ -0,0 +1,91 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.graph.graph_tracer_manager import GraphTracerManager
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus
+from pipelex.pipe_run.delivery_executor import DeliveryExecutor
+from pipelex.pipe_run.exceptions import DeliveryError
+from pipelex.pipe_run.graph_assembly import assemble_graph_on_output
+from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+ from pipelex.pipe_run.pipe_job import PipeJob
+ from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol
+
+
+class PipeRun(PipeRunProtocol):
+ """Direct-mode PipeRun: executes the pipe inline, then delivers results."""
+
+ def __init__(self, pipe_router: PipeRouterProtocol) -> None:
+ self._pipe_router = pipe_router
+ self._delivery_executor = DeliveryExecutor()
+
+ @override
+ async def run(
+ self,
+ pipe_job: PipeJob,
+ delivery_assignment: DeliveryAssignment | None = None,
+ wfid: str | None = None,
+ ) -> PipeOutput:
+ pipeline_run_id: str = pipe_job.job_metadata.pipeline_run_id
+ status: DeliveryStatus = DeliveryStatus.COMPLETED
+ pipe_output: PipeOutput | None = None
+ execution_error: Exception | None = None
+
+ try:
+ pipe_output = await self._pipe_router.run(pipe_job, wfid=wfid)
+ except Exception as exc:
+ status = DeliveryStatus.FAILED
+ execution_error = exc
+ log.error(f"Pipe execution failed for pipeline_run_id={pipeline_run_id}: {exc}")
+ finally:
+ tracer_manager = GraphTracerManager.get_instance()
+ if tracer_manager is not None:
+ try:
+ tracer_manager.close_tracer(pipeline_run_id)
+ except OSError as tracer_close_error:
+ if execution_error is None:
+ raise
+ log.error(
+ f"close_tracer also failed for pipeline_run_id={pipeline_run_id} "
+ f"after pipe execution failure; raising original execution error. "
+ f"Suppressed tracer close error: {tracer_close_error}"
+ )
+
+ if pipe_output is not None:
+ assemble_graph_on_output(
+ pipe_output=pipe_output,
+ pipeline_run_id=pipeline_run_id,
+ domain_code=pipe_job.pipe.domain_code,
+ main_pipe_code=pipe_job.pipe.code,
+ )
+
+ if delivery_assignment is not None:
+ log.debug(f"Executing delivery for pipeline_run_id={pipeline_run_id}, status={status}")
+ try:
+ await self._delivery_executor.execute(
+ pipe_output=pipe_output,
+ user_id=pipe_job.job_metadata.user_id,
+ pipeline_run_id=pipeline_run_id,
+ delivery_assignment=delivery_assignment,
+ status=status,
+ )
+ except DeliveryError as delivery_error:
+ if execution_error is None:
+ raise
+ log.error(
+ f"Delivery also failed for pipeline_run_id={pipeline_run_id} "
+ f"after pipe execution failure; raising original execution error. "
+ f"Suppressed delivery error: {delivery_error}"
+ )
+
+ if execution_error is not None:
+ raise execution_error
+
+ assert pipe_output is not None
+ return pipe_output
diff --git a/pipelex/pipe_run/pipe_run_protocol.py b/pipelex/pipe_run/pipe_run_protocol.py
new file mode 100644
index 000000000..b641ec134
--- /dev/null
+++ b/pipelex/pipe_run/pipe_run_protocol.py
@@ -0,0 +1,31 @@
+from __future__ import annotations
+
+from abc import abstractmethod
+from typing import TYPE_CHECKING, Protocol
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+ from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
+ from pipelex.pipe_run.pipe_job import PipeJob
+
+
+class PipeRunProtocol(Protocol):
+ """Protocol for executing a pipe job and delivering the output.
+
+ Wraps pipe execution (via PipeRouter) with delivery of the output
+ to storage providers, webhooks, or other targets.
+ """
+
+ @abstractmethod
+ async def run(
+ self,
+ pipe_job: PipeJob,
+ delivery_assignment: DeliveryAssignment | None = None,
+ wfid: str | None = None,
+ ) -> PipeOutput:
+ """Execute a pipe job and, if `delivery_assignment` is provided, deliver the output.
+
+ When `delivery_assignment` is None, no storage or webhook delivery happens.
+ Pass `DeliveryAssignment(storage=StorageTarget())` for default storage-only delivery.
+ """
+ ...
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index cb105805f..ed51ace3f 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -44,6 +44,7 @@ from pipelex.observer.multi_observer import MultiObserver
from pipelex.observer.observer_protocol import ObserverNoOp, ObserverProtocol
from pipelex.pipe_run.pipe_router import PipeRouter
from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol
+from pipelex.pipe_run.pipe_run import PipeRun
from pipelex.pipeline.pipeline_manager import PipelineManager
from pipelex.pipeline.pipeline_manager_abstract import PipelineManagerAbstract
from pipelex.plugins.plugin_manager import PluginManager
@@ -95,6 +96,7 @@ class Pipelex(metaclass=MetaSingleton):
self,
config_dir_path: Path | None = None,
config_cls: type[ConfigRoot] | None = None,
+ config_overrides: dict[str, Any] | None = None,
) -> None:
self.is_pipelex_service_enabled = False # Will be set during setup
self.config_dir_path = config_dir_path or config_manager.pipelex_config_dir
@@ -103,7 +105,7 @@ class Pipelex(metaclass=MetaSingleton):
# tools
try:
- self.pipelex_hub.setup_config(config_cls=config_cls or PipelexConfig)
+ self.pipelex_hub.setup_config(config_cls=config_cls or PipelexConfig, config_overrides=config_overrides)
except ValidationError as validation_error:
validation_error_msg = report_validation_error(category="config", validation_error=validation_error)
msg = f"Could not setup config because of: {validation_error_msg}"
@@ -158,6 +160,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self,
integration_mode: IntegrationMode,
needs_inference: bool = True,
+ temporal_enabled: bool | None = None,
needs_model_specs: bool | None = None,
class_registry: ClassRegistryAbstract | None = None,
secrets_provider: SecretsProviderAbstract | None = None,
@@ -179,6 +182,14 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
msg = f"The base setup method does not support any additional arguments: {kwargs}"
raise PipelexSetupError(msg)
+ # Override temporal.is_enabled if temporal_enabled is explicitly provided
+ if temporal_enabled is not None:
+ config = get_config()
+ updated_temporal = config.temporal.model_copy(update={"is_enabled": temporal_enabled})
+ config.temporal = updated_temporal
+
+ self._temporal_task_manager: object | None = None
+
# Initialize secrets provider early - needed for gateway check
secrets_provider = secrets_provider or EnvSecretsProvider()
@@ -327,6 +338,13 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
if content_generator is None:
if not needs_inference:
content_generator = ContentGeneratorDry()
+ elif get_config().temporal.is_enabled:
+ from pipelex.temporal.tprl_content_generation.content_generator_child_factory import ContentGeneratorChildFactory # noqa: PLC0415
+
+ generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)
+ content_generator = ContentGeneratorChildFactory.make_content_generator_child(
+ generated_content_factory=generated_content_factory,
+ )
else:
generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)
content_generator = ContentGenerator(generated_content_factory=generated_content_factory)
@@ -375,13 +393,55 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
multi_observer = MultiObserver(observers=observers)
self.pipelex_hub.set_observer(observer=multi_observer)
+ # --- Temporal --------------------------------------------------------------------------
+
+ if get_config().temporal.is_enabled:
+ from pipelex.temporal.tasks import Tasks # noqa: PLC0415
+ from pipelex.temporal.temporal_hub import temporal_hub # noqa: PLC0415
+ from pipelex.temporal.temporal_task_manager import TemporalTaskManager # noqa: PLC0415
+
+ temporal_task_manager = TemporalTaskManager()
+ temporal_hub.set_task_manager(temporal_task_manager)
+ temporal_task_manager.complement_catalog(
+ extra_catalog=Tasks.TASK_PACKS,
+ extra_workflows=[],
+ extra_activities=[],
+ )
+ temporal_task_manager.setup()
+ self._temporal_task_manager = temporal_task_manager
+
# --- Pipe Router -----------------------------------------------------------------------
- self.pipelex_hub.set_pipe_router(pipe_router or PipeRouter(observer=multi_observer))
+ if pipe_router:
+ self.pipelex_hub.set_pipe_router(pipe_router)
+ elif get_config().temporal.is_enabled:
+ from pipelex.temporal.tprl_pipe.temporal_pipe_router import make_temporal_pipe_router # noqa: PLC0415
+
+ self.pipelex_hub.set_pipe_router(make_temporal_pipe_router())
+ else:
+ self.pipelex_hub.set_pipe_router(PipeRouter(observer=multi_observer))
+
+ # --- Pipe Run --------------------------------------------------------------------------
+
+ if get_config().temporal.is_enabled:
+ from pipelex.temporal.tprl_pipe.temporal_pipe_run import make_temporal_pipe_run # noqa: PLC0415
+
+ self.pipelex_hub.set_pipe_run(make_temporal_pipe_run())
+ else:
+ self.pipelex_hub.set_pipe_run(PipeRun(pipe_router=self.pipelex_hub.get_required_pipe_router()))
log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} setup done")
def teardown(self):
+ # temporal
+ if self._temporal_task_manager is not None:
+ from pipelex.temporal.temporal_hub import temporal_hub # noqa: PLC0415
+ from pipelex.temporal.temporal_task_manager import TemporalTaskManager # noqa: PLC0415
+
+ if isinstance(self._temporal_task_manager, TemporalTaskManager):
+ self._temporal_task_manager.teardown()
+ temporal_hub.reset()
+
# pipelex
self.pipeline_manager.teardown()
if self.telemetry_manager:
@@ -418,6 +478,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
cls,
integration_mode: IntegrationMode = IntegrationMode.PYTHON,
needs_inference: bool = True,
+ temporal_enabled: bool | None = None,
needs_model_specs: bool | None = None,
class_registry: ClassRegistryAbstract | None = None,
secrets_provider: SecretsProviderAbstract | None = None,
@@ -432,6 +493,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
telemetry_manager: TelemetryManagerAbstract | None = None,
observers: dict[str, ObserverProtocol] | None = None,
library_dirs: list[str] | list[Path] | None = None,
+ config_overrides: dict[str, Any] | None = None,
**kwargs: Any,
) -> Self:
"""Create and initialize a Pipelex singleton instance.
@@ -446,6 +508,8 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
content generator and loading backends leniently (skipping those with missing
credentials). This skips gateway terms check and model deck validation.
Useful for commands like validate/show that don't call inference APIs.
+ temporal_enabled: When provided, overrides the temporal.is_enabled config value.
+ True forces Temporal workflow execution, False forces direct execution.
needs_model_specs: When True, load real model specs even if needs_inference
is False. When None (default), follows needs_inference. Useful for validate
commands that need gateway-provided model specs without enabling full inference.
@@ -464,6 +528,9 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
library_dirs: Default library directories for pipeline execution. If provided, these
directories will be used instead of the PIPELEXPATH environment variable.
Per-call library_dirs in execute_pipeline/start_pipeline will override this default.
+ config_overrides: Optional dict deep-merged on top of all TOML config layers
+ as the highest-priority override. Useful for tests that need specific
+ config without editing TOML files.
**kwargs: Additional configuration options, only supported by your own subclass of Pipelex if you really need one
Returns:
@@ -477,11 +544,12 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
msg = "Pipelex is already initialized"
raise PipelexSetupError(msg)
- pipelex_instance = cls()
+ pipelex_instance = cls(config_overrides=config_overrides)
try:
pipelex_instance.setup(
integration_mode=integration_mode,
needs_inference=needs_inference,
+ temporal_enabled=temporal_enabled,
needs_model_specs=needs_model_specs,
class_registry=class_registry,
secrets_provider=secrets_provider,
diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.toml
index f1f88fe34..ad496af35 100644
--- a/pipelex/pipelex.toml
+++ b/pipelex/pipelex.toml
@@ -356,6 +356,25 @@ show_batch_controller = false
theme = "dark"
palette = "dracula"
+####################################################################################################
+# Tracing config (event log for distributed tracing across Temporal workers)
+####################################################################################################
+
+[pipelex.tracing_config]
+is_enabled = true
+backend = "ndjson"
+
+[pipelex.tracing_config.ndjson]
+traces_dir = ".pipelex/traces"
+
+[pipelex.tracing_config.dynamodb]
+table_name = ""
+region = ""
+
+[pipelex.tracing_config.temporal_dynamodb]
+table_name = ""
+region = ""
+
####################################################################################################
# Dry run config
####################################################################################################
@@ -432,10 +451,117 @@ parallels = "branches"
####################################################################################################
[pipelex.kit_config]
-# Preferred agent for rules installation (used by `pipelex kit rules`)
-# Options: "cursor", "agents", "claude"
-# - "cursor" installs .cursor/rules/*.mdc files
-# - Other options install a single markdown file for that agent
-preferred_agent_target = "claude"
+# Agent rule files to install (used by `pipelex-dev kit rules`)
+# Each entry is one of: "cursor", "agents", "claude"
+# - "cursor" installs .cursor/rules/*.mdc files (cannot be combined with others)
+# - "agents" / "claude" install AGENTS.md / CLAUDE.md respectively — both can be listed together
+preferred_agent_targets = ["claude", "agents"]
[migration.migration_maps.telemetry]
+
+####################################################################################################
+# Temporal Config
+####################################################################################################
+
+[temporal]
+is_enabled = false
+
+[temporal.payload_codec_config]
+is_enabled = false
+size_threshold = 1_048_576
+storage_prefix = "temporal-payloads/"
+
+[temporal.payload_codec_config.storage_provider_config]
+method = "local"
+
+[temporal.payload_codec_config.storage_provider_config.local]
+uri_format = "{hash}"
+local_storage_path = ".pipelex/temporal-payload-store"
+
+[temporal.worker_config]
+task_queue = "temporal_task_queue"
+workflow_execution_timeout = "1:00:00"
+run_timeout = "1:00:00"
+task_timeout = "0:00:10"
+start_delay = "0:00:00"
+rpc_timeout = "1:00:00"
+
+[temporal.worker_config.retry_policy_config]
+initial_interval = "0:00:03"
+backoff_coefficient = 2.0
+maximum_interval = "unlimited"
+maximum_attempts = 3
+non_retryable_error_types = [
+ "ExtractHandleNotFoundError",
+ "FileNotFoundError",
+ "ImgGenHandleNotFoundError",
+ "LLMHandleNotFoundError",
+ "ModelNotFoundError",
+ "ValidationError",
+]
+
+[temporal.temporal_config]
+selected_server = "local"
+
+[temporal.temporal_config.temporal_server_configs.local]
+description = "default localhost"
+target_host = "localhost:7233"
+namespace = "default"
+api_key_method = "none"
+api_key_id = ""
+
+[temporal.temporal_config.temporal_server_configs.testing]
+description = "Temporal server for testing"
+target_host = "${TESTING_TEMPORAL_TARGET_HOST}"
+namespace = "${TESTING_TEMPORAL_NAMESPACE}"
+api_key_method = "secret_provider"
+api_key_id = "${TESTING_TEMPORAL_MAIN_API_KEY}"
+
+[temporal.temporal_config.temporal_log_config]
+is_workflow_info_on_message = false
+is_workflow_info_on_extra = true
+is_full_workflow_info_on_extra = false
+is_activity_info_on_message = false
+is_activity_info_on_extra = true
+is_full_activity_info_on_extra = false
+is_formatter_enabled = true
+is_prefix_enabled = true
+managed_loggers = [
+ "temporalio.activity",
+ "temporalio.workflow",
+ "temporalio.worker._activity",
+]
+
+[temporal.worker_scopes]
+default_scope = "full"
+
+# Default — single worker handling everything.
+[temporal.worker_scopes.scopes.full]
+required_tasks_packs = ["crafting", "pipe"]
+required_workflows = []
+required_activities = []
+excluded_workflows = []
+excluded_activities = []
+disable_all_workflows = false
+disable_all_activities = false
+
+# Workflow-only worker. Pair with `runner` on the same task queue to force
+# activity dispatch to a different process (deterministic cross-worker hops).
+[temporal.worker_scopes.scopes.router]
+required_tasks_packs = ["crafting", "pipe"]
+required_workflows = []
+required_activities = []
+excluded_workflows = []
+excluded_activities = []
+disable_all_workflows = false
+disable_all_activities = true
+
+# Activity-only worker. Counterpart to `router`.
+[temporal.worker_scopes.scopes.runner]
+required_tasks_packs = ["crafting", "pipe"]
+required_workflows = []
+required_activities = []
+excluded_workflows = []
+excluded_activities = []
+disable_all_workflows = true
+disable_all_activities = false
diff --git a/pipelex/pipeline/pipeline_factory.py b/pipelex/pipeline/pipeline_factory.py
index 749c6d33a..b040ea526 100644
--- a/pipelex/pipeline/pipeline_factory.py
+++ b/pipelex/pipeline/pipeline_factory.py
@@ -6,8 +6,8 @@ from pipelex.pipeline.pipeline import Pipeline
class PipelineFactory:
@classmethod
- def make_pipeline(cls) -> Pipeline:
- pipeline_run_id = cls.make_pipeline_run_id()
+ def make_pipeline(cls, pipeline_run_id: str | None = None) -> Pipeline:
+ pipeline_run_id = pipeline_run_id or cls.make_pipeline_run_id()
log.verbose(f"Making new pipeline with run id: {pipeline_run_id}")
return Pipeline(
pipeline_run_id=pipeline_run_id,
diff --git a/pipelex/pipeline/pipeline_manager.py b/pipelex/pipeline/pipeline_manager.py
index 40a0b500e..1c2bfb92b 100644
--- a/pipelex/pipeline/pipeline_manager.py
+++ b/pipelex/pipeline/pipeline_manager.py
@@ -13,6 +13,10 @@ class PipelineManagerNotFoundError(PipelexError):
pass
+class PipelineManagerAlreadyExistsError(PipelexError):
+ pass
+
+
class PipelineManager(PipelineManagerAbstract, RootModel[PipelineManagerRoot]):
root: PipelineManagerRoot = Field(default_factory=dict)
@@ -41,7 +45,10 @@ class PipelineManager(PipelineManagerAbstract, RootModel[PipelineManagerRoot]):
return pipeline
@override
- def add_new_pipeline(self, pipe_code: str | None) -> Pipeline:
- pipeline = PipelineFactory.make_pipeline()
+ def add_new_pipeline(self, pipe_code: str | None, pipeline_run_id: str | None = None) -> Pipeline:
+ if pipeline_run_id is not None and pipeline_run_id in self.root:
+ msg = f"Pipeline {pipeline_run_id} already exists"
+ raise PipelineManagerAlreadyExistsError(msg)
+ pipeline = PipelineFactory.make_pipeline(pipeline_run_id=pipeline_run_id)
self._set_pipeline(pipeline_run_id=pipeline.pipeline_run_id, pipeline=pipeline)
return pipeline
diff --git a/pipelex/pipeline/pipeline_manager_abstract.py b/pipelex/pipeline/pipeline_manager_abstract.py
index 3b1916234..c87d5f7ed 100644
--- a/pipelex/pipeline/pipeline_manager_abstract.py
+++ b/pipelex/pipeline/pipeline_manager_abstract.py
@@ -21,5 +21,5 @@ class PipelineManagerAbstract(ABC):
pass
@abstractmethod
- def add_new_pipeline(self, pipe_code: str | None) -> Pipeline:
+ def add_new_pipeline(self, pipe_code: str | None, pipeline_run_id: str | None = None) -> Pipeline:
pass
diff --git a/pipelex/pipeline/pipeline_response.py b/pipelex/pipeline/pipeline_response.py
index 907dd37b8..ca7311c45 100644
--- a/pipelex/pipeline/pipeline_response.py
+++ b/pipelex/pipeline/pipeline_response.py
@@ -26,4 +26,4 @@ class PipelexPipelineExecuteResponse(PipelineExecuteResponse[PipeOutput]):
class PipelexPipelineStartResponse(PipelineStartResponse[PipeOutput]):
- pass
+ workflow_id: str | None = None
diff --git a/pipelex/pipeline/pipeline_run_setup.py b/pipelex/pipeline/pipeline_run_setup.py
index e75172a11..9adb75ea6 100644
--- a/pipelex/pipeline/pipeline_run_setup.py
+++ b/pipelex/pipeline/pipeline_run_setup.py
@@ -4,9 +4,11 @@ from typing import TYPE_CHECKING
from mthds.models.pipeline_inputs import PipelineInputs
from pipelex import log
+from pipelex.config import get_config
from pipelex.core.interpreter.interpreter import PipelexInterpreter
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_factory import PipeFactory
from pipelex.graph.graph_tracer_manager import GraphTracerManager
from pipelex.hub import (
get_library_manager,
@@ -36,11 +38,13 @@ from pipelex.system.environment import get_optional_env
from pipelex.system.telemetry.events import EventName, EventProperty
from pipelex.system.telemetry.otel_constants import OTelConstants
from pipelex.system.telemetry.otel_factory import OtelFactory
+from pipelex.tracing.event_log_factory import make_event_log
if TYPE_CHECKING:
from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
from pipelex.core.pipes.pipe_abstract import PipeAbstract
from pipelex.graph.graph_context import GraphContext
+ from pipelex.tracing.event_log_protocol import EventLogProtocol
async def pipeline_run_setup(
@@ -57,6 +61,7 @@ async def pipeline_run_setup(
pipe_run_mode: PipeRunMode | None = None,
search_domain_codes: list[str] | None = None,
user_id: str | None = None,
+ pipeline_run_id: str | None = None,
) -> tuple[PipeJob, str, str]:
"""Set up a pipeline for execution.
@@ -111,6 +116,10 @@ async def pipeline_run_setup(
added if not already present.
user_id:
Unique identifier for the user (optional).
+ pipeline_run_id:
+ Pre-generated pipeline run ID. If provided, this ID is used instead of
+ generating a new one. Use this when the run record has already been created
+ externally (e.g., by an API Gateway).
Returns:
-------
@@ -124,7 +133,7 @@ async def pipeline_run_setup(
msg = "Either pipe_code or mthds_contents must be provided to the pipeline API."
raise ValueError(msg)
- pipeline = get_pipeline_manager().add_new_pipeline(pipe_code=pipe_code)
+ pipeline = get_pipeline_manager().add_new_pipeline(pipe_code=pipe_code, pipeline_run_id=pipeline_run_id)
pipeline_run_id = pipeline.pipeline_run_id
if not library_id:
@@ -183,7 +192,7 @@ async def pipeline_run_setup(
qualified_main_pipe: str | None = None
for blueprint in all_blueprints:
if blueprint.main_pipe:
- qualified_main_pipe = f"{blueprint.domain}.{blueprint.main_pipe}"
+ qualified_main_pipe = PipeFactory.make_pipe_ref_with_domain(domain_code=blueprint.domain, pipe_code=blueprint.main_pipe)
break
if not qualified_main_pipe:
msg = "No pipe_code provided and no main_pipe found in any of the MTHDS contents."
@@ -204,13 +213,23 @@ async def pipeline_run_setup(
# Initialize graph tracing if requested (after pipe is loaded so we have domain info)
graph_context: GraphContext | None = None
+ event_log: EventLogProtocol | None = None
if execution_config.is_generate_graph:
+ # Create event log when tracing is enabled
+ config = get_config()
+ tracing_config = config.pipelex.tracing_config
+ if tracing_config.is_enabled:
+ event_log = make_event_log(tracing_config)
+
graph_tracer_manager = GraphTracerManager.get_or_create_instance()
graph_context = graph_tracer_manager.open_tracer(
graph_id=pipeline_run_id,
data_inclusion=execution_config.graph_config.data_inclusion,
pipeline_ref_domain=pipe.domain_code,
pipeline_ref_main_pipe=pipe_code,
+ event_log=event_log,
+ workflow_id="direct",
+ pipeline_run_id=pipeline_run_id,
)
try:
@@ -258,6 +277,15 @@ async def pipeline_run_setup(
get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)
+ # Set event log on the report delegate for distributed usage event emission
+ if event_log is not None:
+ get_report_delegate().set_event_log(
+ context_key=pipeline_run_id,
+ event_log=event_log,
+ workflow_id="direct",
+ pipeline_run_id=pipeline_run_id,
+ )
+
# Initialize OtelContext if telemetry is enabled (not dry mode and tracer available)
# The trace_id is computed once here; span_id uses OTEL_VIRTUAL_ROOT_PARENT_SPAN_ID for root
otel_context: OtelContext | None = None
@@ -287,12 +315,16 @@ async def pipeline_run_setup(
pipe_run_mode=pipe_run_mode,
)
+ # Build the library crate from all accumulated blueprints for Temporal dispatch
+ library_crate = library_manager.get_crate(library_id=library_id)
+
pipe_job = PipeJobFactory.make_pipe_job(
pipe=pipe,
pipe_run_params=pipe_run_params,
job_metadata=job_metadata,
working_memory=working_memory,
output_name=output_name,
+ library_crate=library_crate,
)
properties = {
@@ -308,6 +340,9 @@ async def pipeline_run_setup(
tracer_manager = GraphTracerManager.get_instance()
if tracer_manager is not None:
tracer_manager.close_tracer(pipeline_run_id)
+ # Cleanup event log state from the report delegate
+ if event_log is not None:
+ get_report_delegate().clear_event_log(context_key=pipeline_run_id)
# Cleanup library
library_manager.teardown(library_id=library_id)
teardown_current_library()
diff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.py
index d5ca8e510..d52190b46 100644
--- a/pipelex/pipeline/runner.py
+++ b/pipelex/pipeline/runner.py
@@ -13,7 +13,8 @@ from pipelex.config import get_config
from pipelex.graph.graph_tracer_manager import GraphTracerManager
from pipelex.hub import (
get_library_manager,
- get_pipe_router,
+ get_pipe_run,
+ get_report_delegate,
get_telemetry_manager,
teardown_current_library,
)
@@ -34,8 +35,10 @@ if TYPE_CHECKING:
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.pipe_output import PipeOutput
+ from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+ from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
from pipelex.system.configuration.configs import PipelineExecutionConfig
@@ -56,6 +59,7 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):
search_domain_codes: list[str] | None = None,
user_id: str | None = None,
execution_config: PipelineExecutionConfig | None = None,
+ pipe_run: PipeRunProtocol | None = None,
):
self.library_id = library_id
self.library_dirs = library_dirs
@@ -64,6 +68,7 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):
self.search_domain_codes = search_domain_codes
self.user_id = user_id
self.execution_config = execution_config
+ self._pipe_run = pipe_run
self._running_tasks: dict[str, asyncio.Task[PipeOutput]] = {}
@override
@@ -75,6 +80,7 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):
output_name: str | None = None,
output_multiplicity: VariableMultiplicity | None = None,
dynamic_output_concept_ref: str | None = None,
+ delivery_assignment: DeliveryAssignment | None = None,
) -> PipelexPipelineExecuteResponse:
"""Execute a pipeline and wait for its completion.
@@ -123,7 +129,6 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):
pipelex_inputs: PipelineInputs | WorkingMemory | None = cast("PipelineInputs | WorkingMemory | None", inputs)
properties: dict[EventProperty, Any]
- graph_spec_result = None
# These variables are set in pipeline_run_setup and needed in finally/except blocks
pipeline_run_id: str | None = None
library_id_resolved: str | None = None
@@ -144,9 +149,10 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):
search_domain_codes=self.search_domain_codes,
user_id=self.user_id,
)
- pipe_output = await get_pipe_router().run(pipe_job)
+ effective_pipe_run = self._pipe_run or get_pipe_run()
+ pipe_output = await effective_pipe_run.run(pipe_job, delivery_assignment=delivery_assignment)
except PipeRouterError as exc:
- # PipeRouterError can only be raised by get_pipe_router().run(), so pipe_job is guaranteed to exist
+ # PipeRouterError can only be raised by get_pipe_run().run(), so pipe_job is guaranteed to exist
assert pipe_job is not None # for type checker
properties = {
EventProperty.PIPELINE_RUN_ID: pipeline_run_id,
@@ -186,23 +192,23 @@ class PipelexRunner(RunnerProtocol["PipeOutput"]):
msg = f"Input validation failed for '{model_name}': {formatted_error}"
raise PipeExecutionError(message=msg) from exc
finally:
- # Close graph tracer if it was opened (capture graph even on failure)
- # pipeline_run_id may be None if pipeline_run_setup failed early
+ # Close graph tracer if it was opened (cleanup only — the PipeRun is
+ # responsible for capturing the graph spec on pipe_output)
if execution_config.is_generate_graph and pipeline_run_id is not None:
tracer_manager = GraphTracerManager.get_instance()
if tracer_manager is not None:
- graph_spec_result = tracer_manager.close_tracer(pipeline_run_id)
+ tracer_manager.close_tracer(pipeline_run_id)
+
+ # Clear event log state from the report delegate (direct execution path)
+ if pipeline_run_id is not None:
+ get_report_delegate().clear_event_log(context_key=pipeline_run_id)
# Only teardown library if it was successfully created
if library_id_resolved is not None:
- library = get_library_manager().get_library(library_id=library_id_resolved)
- library.teardown()
+ get_library_manager().teardown(library_id=library_id_resolved)
teardown_current_library()
- # Assign graph spec to output (only reached on success, when pipe_output is bound)
- if graph_spec_result is not None:
- pipe_output.graph_spec = graph_spec_result
-
+ assert pipe_job is not None # for type checker, success path requires a resolved job
properties = {
EventProperty.PIPELINE_RUN_ID: pipeline_run_id,
EventProperty.PIPE_TYPE: pipe_job.pipe.pipe_type,
diff --git a/pipelex/pipeline/validate_bundle.py b/pipelex/pipeline/validate_bundle.py
index 16127d094..43ade781a 100644
--- a/pipelex/pipeline/validate_bundle.py
+++ b/pipelex/pipeline/validate_bundle.py
@@ -1,3 +1,4 @@
+import asyncio
from pathlib import Path
from typing import Sequence
@@ -112,6 +113,7 @@ async def validate_bundle(
loaded_pipes: list[PipeAbstract] | None = None
loaded_blueprints: list[PipelexBundleBlueprint] | None = None
+ await asyncio.sleep(0) # Yield to event loop (keeps function async-compatible)
try:
if effective_dirs:
log.verbose(f"Loading libraries from {len(effective_dirs)} directory(ies) ({source_label}) for validation")
@@ -124,14 +126,16 @@ async def validate_bundle(
if blueprints is not None:
loaded_blueprints = blueprints
loaded_pipes = library_manager.load_from_blueprints(library_id=library_id, blueprints=blueprints)
- dry_run_results = await dry_run_pipes(pipes=loaded_pipes, raise_on_failure=True)
- return ValidateBundleResult(blueprints=loaded_blueprints, pipes=loaded_pipes, dry_run_result=dry_run_results)
+ # TODO: wip - restore or refactor dry run
+ # dry_run_results = await dry_run_pipes(pipes=loaded_pipes, raise_on_failure=True)
+ return ValidateBundleResult(blueprints=loaded_blueprints, pipes=loaded_pipes, dry_run_result={})
elif mthds_contents is not None:
loaded_blueprints = [PipelexInterpreter.make_pipelex_bundle_blueprint(mthds_content=content) for content in mthds_contents]
loaded_pipes = library_manager.load_from_blueprints(library_id=library_id, blueprints=loaded_blueprints)
- dry_run_results = await dry_run_pipes(pipes=loaded_pipes, raise_on_failure=True)
- return ValidateBundleResult(blueprints=loaded_blueprints, pipes=loaded_pipes, dry_run_result=dry_run_results)
+ # TODO: wip - restore or refactor dry run
+ # dry_run_results = await dry_run_pipes(pipes=loaded_pipes, raise_on_failure=True)
+ return ValidateBundleResult(blueprints=loaded_blueprints, pipes=loaded_pipes, dry_run_result={})
else:
assert mthds_file_path is not None
@@ -146,8 +150,9 @@ async def validate_bundle(
pipe_codes = list(blueprint.pipe.keys()) if blueprint.pipe else []
loaded_pipes = [library.pipe_library.get_required_pipe(pipe_code=code) for code in pipe_codes]
- dry_run_results = await dry_run_pipes(pipes=loaded_pipes, raise_on_failure=True)
- return ValidateBundleResult(blueprints=loaded_blueprints, pipes=loaded_pipes, dry_run_result=dry_run_results)
+ # TODO: wip - restore or refactor dry run
+ # dry_run_results = await dry_run_pipes(pipes=loaded_pipes, raise_on_failure=True)
+ return ValidateBundleResult(blueprints=loaded_blueprints, pipes=loaded_pipes, dry_run_result={})
except PipelexInterpreterError as interpreter_error:
raise ValidateBundleError(
diff --git a/pipelex/plugins/anthropic/anthropic_exceptions.py b/pipelex/plugins/anthropic/anthropic_exceptions.py
index 4f1622e95..7bc36d0b1 100644
--- a/pipelex/plugins/anthropic/anthropic_exceptions.py
+++ b/pipelex/plugins/anthropic/anthropic_exceptions.py
@@ -1,4 +1,9 @@
-from pipelex.cogt.exceptions import CogtError
+from pipelex.cogt.exceptions import CogtError, InferenceErrorCategory
+
+
+class AnthropicCredentialsError(CogtError):
+ error_category = InferenceErrorCategory.CONFIGURATION
+ user_action = "Check that your Anthropic API key is valid and correctly configured"
class AnthropicModelListingError(CogtError):
diff --git a/pipelex/plugins/anthropic/anthropic_factory.py b/pipelex/plugins/anthropic/anthropic_factory.py
index 1ecd845da..7b49335e9 100644
--- a/pipelex/plugins/anthropic/anthropic_factory.py
+++ b/pipelex/plugins/anthropic/anthropic_factory.py
@@ -6,10 +6,6 @@ from anthropic.types import Usage
from anthropic.types.document_block_param import DocumentBlockParam
from anthropic.types.image_block_param import ImageBlockParam
from anthropic.types.message_param import MessageParam
-from openai.types.chat import (
- ChatCompletionMessageParam,
- ChatCompletionSystemMessageParam,
-)
from pipelex.cogt.document.prompt_document_utils import prep_prompt_documents
from pipelex.cogt.exceptions import CogtError
@@ -23,6 +19,10 @@ from pipelex.tools.aws.aws_config import BedrockAccessVariant
from pipelex.tools.uri.prepared_file import PreparedFile, PreparedFileBase64, PreparedFileHttpUrl, PreparedFileLocalPath
from pipelex.types import StrEnum
+if TYPE_CHECKING:
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from openai.types.chat import ChatCompletionMessageParam
+
if TYPE_CHECKING:
from anthropic.types.text_block_param import TextBlockParam
@@ -166,7 +166,7 @@ class AnthropicFactory:
user_content_txt: str,
prepped_user_images: list[PreparedFile] | None = None,
prepped_user_documents: list[tuple[PreparedFile, str | None]] | None = None,
- ) -> ChatCompletionMessageParam:
+ ) -> "ChatCompletionMessageParam":
text_block_param: TextBlockParam = {"type": "text", "text": user_content_txt}
message: MessageParam
@@ -193,8 +193,11 @@ class AnthropicFactory:
async def make_simple_messages(
cls,
llm_job: LLMJob,
- ) -> list[ChatCompletionMessageParam]:
+ ) -> 'list["ChatCompletionMessageParam"]':
"""Makes a list of messages with a system message (if provided) and followed by a user message."""
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from openai.types.chat import ChatCompletionSystemMessageParam # noqa: PLC0415
+
llm_prompt = llm_job.llm_prompt
messages: list[ChatCompletionMessageParam] = []
if system_content := llm_prompt.system_text:
diff --git a/pipelex/plugins/anthropic/anthropic_llm_worker.py b/pipelex/plugins/anthropic/anthropic_llm_worker.py
index f221a8276..5f0b623b3 100644
--- a/pipelex/plugins/anthropic/anthropic_llm_worker.py
+++ b/pipelex/plugins/anthropic/anthropic_llm_worker.py
@@ -1,12 +1,26 @@
from typing import TYPE_CHECKING, Any
-from anthropic import APIConnectionError, AsyncAnthropic, AsyncAnthropicBedrock, AuthenticationError, BadRequestError, omit
+from anthropic import (
+ APIConnectionError,
+ APITimeoutError,
+ AsyncAnthropic,
+ AsyncAnthropicBedrock,
+ AuthenticationError,
+ BadRequestError,
+ PermissionDeniedError,
+ RateLimitError,
+ omit,
+)
from anthropic.types import OutputConfigParam, ThinkingConfigParam
from pydantic.dataclasses import dataclass as pydantic_dataclass
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import LLMCapabilityError, LLMCompletionError, SdkTypeError
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCapabilityError, LLMCompletionError, SdkTypeError
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_anthropic,
+)
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_job_components import LLMJobParams, ReasoningEffort
from pipelex.cogt.llm.llm_utils import (
@@ -20,6 +34,7 @@ from pipelex.cogt.model_backends.constraints import ListedConstraint
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.config import get_config
from pipelex.plugins.anthropic.anthropic_exceptions import (
+ AnthropicCredentialsError,
AnthropicWorkerConfigurationError,
)
from pipelex.plugins.anthropic.anthropic_factory import (
@@ -27,8 +42,8 @@ from pipelex.plugins.anthropic.anthropic_factory import (
AnthropicSdkVariant,
)
from pipelex.reporting.reporting_protocol import ReportingProtocol
-from pipelex.system.exceptions import CredentialsError
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+from pipelex.urls import URLs
if TYPE_CHECKING:
from anthropic.types import Message
@@ -43,10 +58,6 @@ class _ThinkingParams:
suppress_temperature: bool
-class AnthropicCredentialsError(CredentialsError):
- pass
-
-
class AnthropicLLMWorker(LLMWorkerInternalAbstract):
def __init__(
self,
@@ -213,6 +224,99 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):
msg = f"Model '{self.inference_model.desc}' does not support reasoning (thinking_mode=none)"
raise LLMCapabilityError(msg)
+ @staticmethod
+ def _extract_underlying_sdk_exception(instructor_exc: Any) -> BaseException | None:
+ """Recover the SDK exception that caused an ``InstructorRetryException``.
+
+ instructor's retry loop wraps the last failed attempt's exception inside
+ ``InstructorRetryException``. We prefer ``failed_attempts[-1].exception``
+ (the documented public attribute) and fall back to walking ``__cause__``
+ (a tenacity ``RetryError`` whose ``last_attempt._exception`` holds the
+ original exception) when ``failed_attempts`` is unset.
+ """
+ failed_attempts = getattr(instructor_exc, "failed_attempts", None)
+ if failed_attempts:
+ last_exc = getattr(failed_attempts[-1], "exception", None)
+ if isinstance(last_exc, BaseException):
+ return last_exc
+ cause = instructor_exc.__cause__
+ last_attempt = getattr(cause, "last_attempt", None)
+ if last_attempt is not None:
+ underlying = getattr(last_attempt, "_exception", None)
+ if isinstance(underlying, BaseException):
+ return underlying
+ return None
+
+ def _raise_categorized_anthropic_sdk_error(
+ self,
+ sdk_exc: BaseException,
+ chain_from: BaseException | None = None,
+ ) -> None:
+ """Categorize an Anthropic SDK exception and raise the matching pipelex error.
+
+ Args:
+ sdk_exc: The SDK exception to categorize. If it is not one of the
+ recognized SDK exception types this method returns None and the
+ caller is responsible for the fallback.
+ chain_from: Override for ``raise ... from`` chaining (defaults to ``sdk_exc``).
+ Used when the SDK exception was unwrapped from a wrapper (e.g.
+ ``InstructorRetryException``) so the traceback preserves the wrapper.
+
+ """
+ cause = chain_from if chain_from is not None else sdk_exc
+
+ if isinstance(sdk_exc, RateLimitError):
+ error_message = str(sdk_exc)
+ if is_quota_exhaustion_anthropic(error_message):
+ msg = f"Anthropic quota exhausted for model '{self.inference_model.desc}': {sdk_exc}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your Anthropic account has exceeded its quota — check billing at {URLs.anthropic_billing}",
+ ) from cause
+ msg = f"Anthropic rate limit exceeded for model '{self.inference_model.desc}': {sdk_exc}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by Anthropic — the system will retry automatically",
+ ) from cause
+
+ if isinstance(sdk_exc, APITimeoutError):
+ msg = f"Anthropic API request timed out for model '{self.inference_model.desc}': {sdk_exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from cause
+
+ if isinstance(sdk_exc, BadRequestError):
+ error_message = str(sdk_exc)
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {sdk_exc}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ ) from cause
+ msg = f"Anthropic bad request error: {sdk_exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from cause
+
+ if isinstance(sdk_exc, APIConnectionError):
+ msg = f"Anthropic API connection error: {sdk_exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from cause
+
+ if isinstance(sdk_exc, PermissionDeniedError):
+ error_message = str(sdk_exc)
+ if is_quota_exhaustion_anthropic(error_message):
+ msg = f"Anthropic quota exhausted: {sdk_exc}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your Anthropic account has exceeded its quota — check billing at {URLs.anthropic_billing}",
+ ) from cause
+ msg = f"Anthropic permission denied: {sdk_exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from cause
+
+ if isinstance(sdk_exc, AuthenticationError):
+ msg = f"Anthropic credentials error: {sdk_exc}"
+ raise AnthropicCredentialsError(msg) from cause
+
@override
async def _gen_text(
self,
@@ -239,15 +343,16 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):
output_config=thinking_params.output_config or omit,
) as stream:
final_message: Message = await stream.get_final_message()
- except BadRequestError as exc:
- msg = f"Anthropic bad request error: {exc}"
- raise LLMCompletionError(msg) from exc
- except APIConnectionError as exc:
- msg = f"Anthropic API connection error: {exc}"
- raise LLMCompletionError(msg) from exc
- except AuthenticationError as exc:
- msg = f"Anthropic credentials error: {exc}"
- raise AnthropicCredentialsError(msg) from exc
+ except (
+ RateLimitError,
+ APITimeoutError,
+ BadRequestError,
+ APIConnectionError,
+ PermissionDeniedError,
+ AuthenticationError,
+ ) as sdk_exc:
+ self._raise_categorized_anthropic_sdk_error(sdk_exc=sdk_exc)
+ raise # unreachable: helper always raises for these types
# Collect all text blocks (adaptive thinking enables interleaved thinking,
# so the response may contain multiple text blocks interspersed with thinking blocks)
@@ -302,6 +407,9 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):
requested_max_tokens = job_params.max_tokens or self.default_max_tokens
effective_max_tokens = min(requested_max_tokens, safe_max_tokens)
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from instructor.core import InstructorRetryException # noqa: PLC0415
+
temperature_unsupported = ListedConstraint.TEMPERATURE_UNSUPPORTED in self.inference_model.listed_constraints
try:
result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
@@ -313,15 +421,27 @@ class AnthropicLLMWorker(LLMWorkerInternalAbstract):
max_tokens=effective_max_tokens,
timeout=float(timeout_seconds), # Explicit timeout disables SDK's long-request protection
)
- except BadRequestError as exc:
- msg = f"Anthropic bad request error: {exc}"
- raise LLMCompletionError(msg) from exc
- except APIConnectionError as exc:
- msg = f"Anthropic API connection error: {exc}"
- raise LLMCompletionError(msg) from exc
- except AuthenticationError as exc:
- msg = f"Anthropic credentials error: {exc}"
- raise AnthropicCredentialsError(msg) from exc
+ except InstructorRetryException as instructor_exc:
+ # instructor wraps SDK exceptions during retries; recover the underlying
+ # one so transient/capacity/auth errors aren't all flattened to CONTENT.
+ underlying_exc = self._extract_underlying_sdk_exception(instructor_exc=instructor_exc)
+ if underlying_exc is not None:
+ self._raise_categorized_anthropic_sdk_error(sdk_exc=underlying_exc, chain_from=instructor_exc)
+ msg = (
+ f"Anthropic structured generation via 'instructor' failed with model: {self.inference_model.desc} "
+ f"trying to generate schema: {schema} with error: {instructor_exc}"
+ )
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from instructor_exc
+ except (
+ RateLimitError,
+ APITimeoutError,
+ BadRequestError,
+ APIConnectionError,
+ PermissionDeniedError,
+ AuthenticationError,
+ ) as sdk_exc:
+ self._raise_categorized_anthropic_sdk_error(sdk_exc=sdk_exc)
+ raise # unreachable: helper always raises for these types
if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := completion.usage):
llm_tokens_usage.nb_tokens_by_category = AnthropicFactory.make_nb_tokens_by_category(usage=usage)
diff --git a/pipelex/plugins/azure_rest/azure_img_gen_worker.py b/pipelex/plugins/azure_rest/azure_img_gen_worker.py
index 664e1790d..1052af3f2 100644
--- a/pipelex/plugins/azure_rest/azure_img_gen_worker.py
+++ b/pipelex/plugins/azure_rest/azure_img_gen_worker.py
@@ -1,12 +1,13 @@
import httpx
from typing_extensions import override
-from pipelex.cogt.exceptions import CogtError, ImgGenGenerationError, ImgGenParameterError
+from pipelex.cogt.exceptions import CogtError, ImgGenGenerationError, ImgGenParameterError, InferenceErrorCategory
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.cogt.image.image_size import ImageSize
from pipelex.cogt.img_gen.img_gen_args_factory import ImgGenArgsFactory
from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
from pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstract
+from pipelex.cogt.inference.error_classification import is_content_policy_violation
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.hub import get_models_manager
@@ -90,18 +91,56 @@ class AzureImgGenWorker(ImgGenWorkerAbstract):
params = f"?api-version={self.api_version}"
generation_url = f"{self.endpoint}/{base_path}/generations{params}"
- async with httpx.AsyncClient() as client:
- response = await client.post(
- generation_url,
- headers={
- "Api-Key": self.api_key,
- "Content-Type": "application/json",
- },
- json=args_dict,
- timeout=600.0,
- )
- response.raise_for_status()
- response_dict = response.json()
+ try:
+ async with httpx.AsyncClient() as client:
+ response = await client.post(
+ generation_url,
+ headers={
+ "Api-Key": self.api_key,
+ "Content-Type": "application/json",
+ },
+ json=args_dict,
+ timeout=600.0,
+ )
+ response.raise_for_status()
+ response_dict = response.json()
+ except httpx.HTTPStatusError as exc:
+ status_code = exc.response.status_code
+ error_body = exc.response.text
+ if status_code == 429:
+ msg = f"Azure rate limit exceeded for model '{self.inference_model.desc}': {error_body}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by Azure — the system will retry automatically",
+ ) from exc
+ if status_code == 402:
+ msg = f"Azure quota exhausted for model '{self.inference_model.desc}': {error_body}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CAPACITY) from exc
+ if status_code in {401, 403}:
+ msg = f"Azure authentication error for model '{self.inference_model.desc}': {error_body}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from exc
+ if status_code == 400:
+ if is_content_policy_violation(error_body):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {error_body}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ ) from exc
+ msg = f"Azure bad request for model '{self.inference_model.desc}': {error_body}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+ if status_code >= 500:
+ msg = f"Azure server error ({status_code}) for model '{self.inference_model.desc}': {error_body}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+ msg = f"Azure API error ({status_code}) for model '{self.inference_model.desc}': {error_body}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from exc
+ except httpx.ConnectError as exc:
+ msg = f"Azure connection error for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+ except httpx.TimeoutException as exc:
+ msg = f"Azure request timed out for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
# Extract usage tokens if available
if (usage_dict := response_dict.get("usage")) and (img_gen_tokens_usage := img_gen_job.job_report.img_gen_tokens_usage):
diff --git a/pipelex/plugins/bedrock/bedrock_llm_worker.py b/pipelex/plugins/bedrock/bedrock_llm_worker.py
index d2a5e696a..ab21adf3f 100644
--- a/pipelex/plugins/bedrock/bedrock_llm_worker.py
+++ b/pipelex/plugins/bedrock/bedrock_llm_worker.py
@@ -1,9 +1,13 @@
from typing import Any
+from botocore.exceptions import ClientError
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import CogtError, LLMCapabilityError, SdkTypeError
+from pipelex.cogt.exceptions import CogtError, InferenceErrorCategory, LLMCapabilityError, LLMCompletionError, SdkTypeError
+from pipelex.cogt.inference.error_classification import (
+ is_quota_exhaustion_aws,
+)
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_job_components import LLMJobParams
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
@@ -12,6 +16,7 @@ from pipelex.plugins.bedrock.bedrock_client_protocol import BedrockClientProtoco
from pipelex.plugins.bedrock.bedrock_factory import BedrockFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+from pipelex.urls import URLs
class BedrockWorkerConfigurationError(CogtError):
@@ -62,13 +67,57 @@ class BedrockLLMWorker(LLMWorkerInternalAbstract):
log.verbose(self.inference_model.model_id)
- bedrock_response_text, nb_tokens_by_category = await self.bedrock_client_for_text.chat(
- messages=message.to_dict_list(),
- system_text=llm_job.llm_prompt.system_text,
- model=self.inference_model.model_id,
- temperature=job_params.temperature,
- max_tokens=job_params.max_tokens or self.default_max_tokens,
- )
+ try:
+ bedrock_response_text, nb_tokens_by_category = await self.bedrock_client_for_text.chat(
+ messages=message.to_dict_list(),
+ system_text=llm_job.llm_prompt.system_text,
+ model=self.inference_model.model_id,
+ temperature=job_params.temperature,
+ max_tokens=job_params.max_tokens or self.default_max_tokens,
+ )
+ except ClientError as exc:
+ error_code = exc.response.get("Error", {}).get("Code", "")
+ error_msg = exc.response.get("Error", {}).get("Message", str(exc))
+
+ if error_code == "ServiceQuotaExceededException":
+ msg = f"AWS service quota exceeded for model '{self.inference_model.desc}': {error_msg}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your AWS account has exceeded its service quota — check billing at {URLs.aws_billing}",
+ ) from exc
+
+ if error_code == "ThrottlingException":
+ if is_quota_exhaustion_aws(error_msg):
+ msg = f"AWS quota exhausted for model '{self.inference_model.desc}': {error_msg}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your AWS account has exceeded its quota — check billing at {URLs.aws_billing}",
+ ) from exc
+ msg = f"AWS rate limit exceeded for model '{self.inference_model.desc}': {error_msg}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by AWS — the system will retry automatically",
+ ) from exc
+
+ if error_code == "AccessDeniedException":
+ msg = f"AWS access denied for model '{self.inference_model.desc}': {error_msg}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from exc
+
+ if error_code == "ValidationException":
+ msg = f"AWS validation error for model '{self.inference_model.desc}': {error_msg}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+
+ if error_code in {"ModelNotReadyException", "ServiceUnavailableException"}:
+ msg = f"AWS service unavailable for model '{self.inference_model.desc}': {error_msg}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+
+ # Fallback for unknown AWS errors
+ msg = f"AWS error for model '{self.inference_model.desc}' ({error_code}): {error_msg}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+
if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and nb_tokens_by_category:
llm_tokens_usage.nb_tokens_by_category = nb_tokens_by_category
return bedrock_response_text
diff --git a/pipelex/plugins/docling/docling_extract_worker.py b/pipelex/plugins/docling/docling_extract_worker.py
index 0dfe6e617..9e431579b 100644
--- a/pipelex/plugins/docling/docling_extract_worker.py
+++ b/pipelex/plugins/docling/docling_extract_worker.py
@@ -7,7 +7,7 @@ from typing import Any
import aiofiles
from typing_extensions import override
-from pipelex.cogt.exceptions import SdkTypeError
+from pipelex.cogt.exceptions import ExtractJobFailureError, InferenceErrorCategory, SdkTypeError
from pipelex.cogt.extract.extract_input import ExtractInputError
from pipelex.cogt.extract.extract_job import ExtractJob
from pipelex.cogt.extract.extract_output import ExtractOutput
@@ -89,6 +89,22 @@ class DoclingExtractWorker(ExtractWorkerAbstract):
# Run synchronous Docling conversion in a thread pool to avoid blocking
conversion_result = await asyncio.to_thread(self.docling_sdk.document_converter.convert, docling_source)
return DoclingFactory.make_extract_output_from_docling_document(doc=conversion_result.document)
+ except ExtractJobFailureError:
+ raise
+ # The exceptions below are raised by docling's document_converter.convert() for
+ # corrupt/unreadable documents, unsupported formats, or file system issues.
+ except FileNotFoundError as exc:
+ msg = f"Document file not found for Docling extraction: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from exc
+ except ValueError as exc:
+ msg = f"Invalid document format for Docling extraction: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+ except RuntimeError as exc:
+ msg = f"Docling conversion failed: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+ except OSError as exc:
+ msg = f"I/O error during Docling extraction: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
finally:
if temp_path:
temp_path.unlink(missing_ok=True)
diff --git a/pipelex/plugins/docling/docling_factory.py b/pipelex/plugins/docling/docling_factory.py
index 16bf117a2..fdfe643bb 100644
--- a/pipelex/plugins/docling/docling_factory.py
+++ b/pipelex/plugins/docling/docling_factory.py
@@ -1,9 +1,12 @@
-from docling_core.transforms.serializer.markdown import MarkdownDocSerializer, MarkdownParams
-from docling_core.types.doc.document import DoclingDocument
+from typing import TYPE_CHECKING
from pipelex.cogt.extract.extract_output import ExtractOutput, Page
from pipelex.plugins.docling.docling_sdk import DoclingSdk
+if TYPE_CHECKING:
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from docling_core.types.doc.document import DoclingDocument
+
class DoclingFactory:
@classmethod
@@ -13,7 +16,7 @@ class DoclingFactory:
@classmethod
def make_extract_output_from_docling_document(
cls,
- doc: DoclingDocument,
+ doc: "DoclingDocument",
) -> ExtractOutput:
"""Convert a Docling document to ExtractOutput with markdown text per page.
@@ -23,6 +26,9 @@ class DoclingFactory:
Returns:
ExtractOutput with pages dict (0-indexed).
"""
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from docling_core.transforms.serializer.markdown import MarkdownDocSerializer, MarkdownParams # noqa: PLC0415
+
pages: dict[int, Page] = {}
# doc.pages is a dictionary where keys are page numbers (1-based in Docling)
diff --git a/pipelex/plugins/docling/docling_sdk.py b/pipelex/plugins/docling/docling_sdk.py
index ed7553d17..922f216ca 100644
--- a/pipelex/plugins/docling/docling_sdk.py
+++ b/pipelex/plugins/docling/docling_sdk.py
@@ -1,6 +1,6 @@
-from docling.document_converter import DocumentConverter
-
-
class DoclingSdk:
def __init__(self):
- self.document_converter = DocumentConverter()
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from docling.document_converter import DocumentConverter # noqa: PLC0415
+
+ self.document_converter: DocumentConverter = DocumentConverter()
diff --git a/pipelex/plugins/fal/fal_img_gen_worker.py b/pipelex/plugins/fal/fal_img_gen_worker.py
index 707564927..1c04266ca 100644
--- a/pipelex/plugins/fal/fal_img_gen_worker.py
+++ b/pipelex/plugins/fal/fal_img_gen_worker.py
@@ -1,10 +1,12 @@
from typing import Any
from fal_client import AsyncClient, InProgress
+from fal_client.auth import MissingCredentialsError
+from fal_client.client import FalClientError, FalClientHTTPError, FalClientTimeoutError
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import ImgGenParameterError, SdkTypeError
+from pipelex.cogt.exceptions import ImgGenGenerationError, ImgGenParameterError, InferenceErrorCategory, SdkTypeError
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.cogt.img_gen.img_gen_args_factory import ImgGenArgsFactory
from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
@@ -12,6 +14,7 @@ from pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstract
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.plugins.fal.fal_factory import FalFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.urls import URLs
class FalImgGenWorker(ImgGenWorkerAbstract):
@@ -49,22 +52,60 @@ class FalImgGenWorker(ImgGenWorkerAbstract):
msg = f"Model '{self.inference_model.name}' rules must include a 'model_choice' entry"
raise ImgGenParameterError(msg)
log.verbose(args_dict, title=f"Fal arguments, application={fal_application}")
- handler = await self.fal_async_client.submit(
- application=fal_application,
- arguments=args_dict,
- )
+ try:
+ handler = await self.fal_async_client.submit(
+ application=fal_application,
+ arguments=args_dict,
+ )
- log_index = 0
- async for event in handler.iter_events(with_logs=True):
- if isinstance(event, InProgress):
- if not event.logs:
- continue
- new_logs = event.logs[log_index:]
- for event_log in new_logs:
- log.verbose(event_log["message"], title="FAL Log")
- log_index = len(event.logs)
+ log_index = 0
+ async for event in handler.iter_events(with_logs=True):
+ if isinstance(event, InProgress):
+ if not event.logs:
+ continue
+ new_logs = event.logs[log_index:]
+ for event_log in new_logs:
+ log.verbose(event_log["message"], title="FAL Log")
+ log_index = len(event.logs)
- return await handler.get()
+ return await handler.get()
+ except MissingCredentialsError as exc:
+ msg = f"FAL API key not configured for model '{self.inference_model.desc}'"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.CONFIGURATION,
+ user_action="Check that the FAL_KEY environment variable is set",
+ ) from exc
+ except FalClientHTTPError as exc:
+ status_code = exc.status_code
+ if status_code == 402:
+ msg = f"FAL quota exhausted for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your FAL account has exceeded its quota — check billing at {URLs.fal_billing}",
+ ) from exc
+ if status_code == 429:
+ msg = f"FAL rate limit exceeded for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by FAL — the system will retry automatically",
+ ) from exc
+ if status_code in {401, 403}:
+ msg = f"FAL authentication error for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from exc
+ if status_code == 400:
+ msg = f"FAL bad request for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+ msg = f"FAL API error ({status_code}) for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+ except FalClientTimeoutError as exc:
+ msg = f"FAL request timed out for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+ except FalClientError as exc:
+ msg = f"FAL error for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
@override
async def _gen_image(
diff --git a/pipelex/plugins/gateway/gateway_completions_factory.py b/pipelex/plugins/gateway/gateway_completions_factory.py
index eb651de76..a6cfbe170 100644
--- a/pipelex/plugins/gateway/gateway_completions_factory.py
+++ b/pipelex/plugins/gateway/gateway_completions_factory.py
@@ -3,16 +3,6 @@ from __future__ import annotations
import json
from typing import TYPE_CHECKING, Any, cast
-import openai
-from openai.types.chat import (
- ChatCompletionContentPartImageParam,
- ChatCompletionContentPartParam,
- ChatCompletionContentPartTextParam,
- ChatCompletionMessageParam,
- ChatCompletionSystemMessageParam,
- ChatCompletionUserMessageParam,
-)
-from openai.types.chat.chat_completion_content_part_image_param import ImageURL as OpenAIImageURL
from portkey_ai import (
createHeaders, # type: ignore[reportUnknownVariableType]
)
@@ -35,6 +25,9 @@ from pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsF
from pipelex.tools.uri.prepared_file import PreparedFileBase64, PreparedFileHttpUrl, PreparedFileLocalPath
if TYPE_CHECKING:
+ # Deferred imports: avoid pulling heavy SDK at module-load time
+ import openai
+ from openai.types.chat import ChatCompletionMessageParam
from portkey_ai.api_resources.utils import GenericResponse
from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
@@ -51,6 +44,16 @@ class GatewayCompletionsFactory(OpenAICompletionsFactory):
llm_job: LLMJob,
) -> list[ChatCompletionMessageParam]:
"""Override to use image_url format for documents which Portkey/Gateway translates correctly."""
+ # Deferred imports: avoid pulling heavy SDK at module-load time
+ from openai.types.chat import ( # noqa: PLC0415
+ ChatCompletionContentPartImageParam,
+ ChatCompletionContentPartParam,
+ ChatCompletionContentPartTextParam,
+ ChatCompletionSystemMessageParam,
+ ChatCompletionUserMessageParam,
+ )
+ from openai.types.chat.chat_completion_content_part_image_param import ImageURL as OpenAIImageURL # noqa: PLC0415
+
llm_prompt = llm_job.llm_prompt
messages: list[ChatCompletionMessageParam] = []
user_contents: list[ChatCompletionContentPartParam] = []
@@ -102,6 +105,9 @@ class GatewayCompletionsFactory(OpenAICompletionsFactory):
plugin: Plugin,
backend: InferenceBackend,
) -> openai.AsyncOpenAI:
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ import openai # noqa: PLC0415
+
is_debug_enabled = GatewayFactory.is_debug_enabled(backend=backend)
endpoint = GatewayFactory.get_endpoint(backend=backend)
api_key = GatewayFactory.get_api_key(backend=backend)
diff --git a/pipelex/plugins/gateway/gateway_extract_worker.py b/pipelex/plugins/gateway/gateway_extract_worker.py
index 8c64b120c..77d7b1c60 100644
--- a/pipelex/plugins/gateway/gateway_extract_worker.py
+++ b/pipelex/plugins/gateway/gateway_extract_worker.py
@@ -1,3 +1,4 @@
+import asyncio
from typing import Any
from portkey_ai import AsyncPortkey
@@ -47,6 +48,35 @@ class GatewayExtractWorker(ExtractWorkerAbstract):
self.portkey_client: AsyncPortkey = sdk_instance
self._tenacity_config = get_config().cogt.tenacity_config
+ @override
+ def teardown(self):
+ """Close the portkey client's underlying httpx connection pool.
+
+ The httpx client holds TCP connections bound to a specific event loop.
+ If the worker is reused across event loop boundaries (e.g. between test
+ modules with different class-scoped loops), stale connections cause
+ 'Event loop is closed' errors. Closing prevents this.
+ """
+ try:
+ httpx_client = getattr(self.portkey_client, "_client", None)
+ if httpx_client is not None and hasattr(httpx_client, "is_closed") and not httpx_client.is_closed:
+ try:
+ loop = asyncio.get_running_loop()
+ task = loop.create_task(httpx_client.aclose())
+ task.add_done_callback(
+ lambda task_result: log.debug(f"Portkey httpx client cleanup error: {task_result.exception()}")
+ if not task_result.cancelled() and task_result.exception()
+ else None
+ )
+ except RuntimeError:
+ # No running event loop, best-effort sync close
+ try:
+ asyncio.run(httpx_client.aclose())
+ except Exception as exc:
+ log.debug(f"Error closing portkey httpx client during teardown: {exc}")
+ except Exception as exc:
+ log.debug(f"Error during GatewayExtractWorker teardown: {exc}")
+
def _make_retryer(self) -> AsyncRetrying:
"""Create a fresh AsyncRetrying instance for each extraction call.
@@ -151,11 +181,12 @@ class GatewayExtractWorker(ExtractWorkerAbstract):
)
except portkey_exceptions.APIError as exc:
error_summary = GatewayFactory.make_error_summary_from_portkey_error(exc)
+ error_category = GatewayFactory.classify_error_category(exc)
msg = (
f"Web fetch service error for URL '{document_uri}' via model '{self.inference_model.tag}' "
f"after {attempt_number} attempt(s): {error_summary}"
)
- raise ExtractJobFailureError(msg) from exc
+ raise ExtractJobFailureError(msg, error_category=error_category) from exc
if response is None:
msg = f"Could not get a response for model '{self.inference_model.tag}' via Portkey after {attempt_number} attempts"
@@ -217,8 +248,9 @@ class GatewayExtractWorker(ExtractWorkerAbstract):
)
except portkey_exceptions.APIError as exc:
error_summary = GatewayFactory.make_error_summary_from_portkey_error(exc)
+ error_category = GatewayFactory.classify_error_category(exc)
msg = f"Extract service error for model '{self.inference_model.tag}' after {attempt_number} attempt(s): {error_summary}"
- raise ExtractJobFailureError(msg) from exc
+ raise ExtractJobFailureError(msg, error_category=error_category) from exc
if response is None:
msg = f"Could not get a response for model '{self.inference_model.tag}' via Portkey after {attempt_number} attempts"
diff --git a/pipelex/plugins/gateway/gateway_factory.py b/pipelex/plugins/gateway/gateway_factory.py
index a4dbb8a1e..4d69817cc 100644
--- a/pipelex/plugins/gateway/gateway_factory.py
+++ b/pipelex/plugins/gateway/gateway_factory.py
@@ -10,8 +10,10 @@ from portkey_ai import (
from portkey_ai.api_resources import exceptions as portkey_exc
from pipelex import log
+from pipelex.cogt.exceptions import InferenceErrorCategory
from pipelex.cogt.extract.extract_job import ExtractJob
from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
+from pipelex.cogt.inference.error_classification import is_quota_exhaustion_gateway
from pipelex.cogt.inference.inference_constants import InferenceOutputType
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.hub import get_telemetry_manager
@@ -115,6 +117,32 @@ class GatewayFactory:
return extra_headers, extra_body
+ @classmethod
+ def classify_error_category(cls, exc: portkey_exceptions.APIError) -> InferenceErrorCategory:
+ """Classify a Portkey API error into an InferenceErrorCategory.
+
+ Args:
+ exc: The Portkey API error
+
+ Returns:
+ The appropriate InferenceErrorCategory for retry decisions and error reporting
+ """
+ if isinstance(exc, portkey_exc.APIStatusError):
+ status_code = exc.status_code
+ if is_quota_exhaustion_gateway(str(exc), status_code):
+ return InferenceErrorCategory.CAPACITY
+ if isinstance(exc, portkey_exc.RateLimitError):
+ return InferenceErrorCategory.TRANSIENT
+ if isinstance(exc, (portkey_exc.AuthenticationError, portkey_exc.PermissionDeniedError)):
+ return InferenceErrorCategory.CONFIGURATION
+ if isinstance(exc, portkey_exc.BadRequestError):
+ return InferenceErrorCategory.CONTENT
+ if isinstance(exc, portkey_exc.NotFoundError):
+ return InferenceErrorCategory.CONFIGURATION
+ if isinstance(exc, (portkey_exc.APITimeoutError, portkey_exc.APIConnectionError)):
+ return InferenceErrorCategory.TRANSIENT
+ return InferenceErrorCategory.TRANSIENT
+
@classmethod
def make_error_summary_from_portkey_error(cls, exc: portkey_exceptions.APIError) -> str:
"""Extract a clean, human-readable error summary from a Portkey API error.
diff --git a/pipelex/plugins/gateway/gateway_img_gen_worker.py b/pipelex/plugins/gateway/gateway_img_gen_worker.py
index db690e957..100e00eea 100644
--- a/pipelex/plugins/gateway/gateway_img_gen_worker.py
+++ b/pipelex/plugins/gateway/gateway_img_gen_worker.py
@@ -81,8 +81,9 @@ class GatewayImgGenWorker(ImgGenWorkerAbstract):
)
except portkey_exceptions.APIError as exc:
error_summary = GatewayFactory.make_error_summary_from_portkey_error(exc)
+ error_category = GatewayFactory.classify_error_category(exc)
msg = f"Image generation service error for model '{self.inference_model.model_id}': {error_summary}"
- raise ImgGenGenerationError(msg) from exc
+ raise ImgGenGenerationError(msg, error_category=error_category) from exc
if response is None:
msg = f"Could not get a response for model '{self.inference_model.model_id}' via Portkey"
diff --git a/pipelex/plugins/gateway/gateway_search_worker.py b/pipelex/plugins/gateway/gateway_search_worker.py
index 4a3e2928e..28f23b92a 100644
--- a/pipelex/plugins/gateway/gateway_search_worker.py
+++ b/pipelex/plugins/gateway/gateway_search_worker.py
@@ -180,8 +180,9 @@ class GatewaySearchWorker(SearchWorkerAbstract):
)
except portkey_exceptions.APIError as exc:
error_summary = GatewayFactory.make_error_summary_from_portkey_error(exc)
+ error_category = GatewayFactory.classify_error_category(exc)
msg = f"Search service error for model '{model}' after {attempt_number} attempt(s): {error_summary}"
- raise GatewaySearchResponseError(msg) from exc
+ raise GatewaySearchResponseError(msg, error_category=error_category) from exc
if response is None:
msg = f"Could not get a response for model '{model}' via Portkey after {attempt_number} attempts"
diff --git a/pipelex/plugins/google/google_factory.py b/pipelex/plugins/google/google_factory.py
index b34433304..e5b495cd6 100644
--- a/pipelex/plugins/google/google_factory.py
+++ b/pipelex/plugins/google/google_factory.py
@@ -1,8 +1,8 @@
import asyncio
import base64
-from google import genai
from google.genai import types as genai_types
+from google.genai.client import Client as GoogleGenAiClient
from pipelex.cogt.document.prompt_document import PromptDocument
from pipelex.cogt.document.prompt_document_utils import prepare_prompt_document_as_base64
@@ -16,9 +16,9 @@ from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCateg
class GoogleFactory:
@classmethod
- def make_google_client(cls, backend: InferenceBackend) -> genai.Client:
+ def make_google_client(cls, backend: InferenceBackend) -> GoogleGenAiClient:
"""Create a Google Gemini API client."""
- return genai.Client(api_key=backend.api_key)
+ return GoogleGenAiClient(api_key=backend.api_key)
@classmethod
async def prepare_image_part(cls, prompt_image: PromptImage) -> genai_types.Part:
diff --git a/pipelex/plugins/google/google_img_gen_worker.py b/pipelex/plugins/google/google_img_gen_worker.py
index 67d6f4919..c8d8303f1 100644
--- a/pipelex/plugins/google/google_img_gen_worker.py
+++ b/pipelex/plugins/google/google_img_gen_worker.py
@@ -1,21 +1,27 @@
import asyncio
from typing import Any
-from google import genai
+from google.genai import errors as genai_errors
from google.genai import types as genai_types
+from google.genai.client import Client as GoogleGenAiClient
from typing_extensions import override
from pipelex import log
from pipelex.base_exceptions import PipelexError
-from pipelex.cogt.exceptions import ImgGenGenerationError, SdkTypeError
+from pipelex.cogt.exceptions import ImgGenGenerationError, InferenceErrorCategory, SdkTypeError
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.cogt.image.image_size import ImageSize
from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
from pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstract
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_google,
+)
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.plugins.google.google_factory import GoogleFactory
from pipelex.plugins.google.google_img_gen_factory import GoogleImgGenFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.urls import URLs
class GoogleImgGenWorkerError(PipelexError):
@@ -33,11 +39,11 @@ class GoogleImgGenWorker(ImgGenWorkerAbstract):
):
super().__init__(inference_model=inference_model, reporting_delegate=reporting_delegate)
- if not isinstance(sdk_instance, genai.Client):
- msg = f"Provided ImgGen sdk_instance is not of type genai.Client: it's a '{type(sdk_instance)}'"
+ if not isinstance(sdk_instance, GoogleGenAiClient):
+ msg = f"Provided ImgGen sdk_instance is not of type GoogleGenAiClient: it's a '{type(sdk_instance)}'"
raise SdkTypeError(msg)
- self.genai_client: genai.Client = sdk_instance
+ self.genai_client: GoogleGenAiClient = sdk_instance
self.genai_async_client = sdk_instance.aio
# Capture the event loop at creation time if one is running
@@ -81,6 +87,49 @@ class GoogleImgGenWorker(ImgGenWorkerAbstract):
# Log but don't fail teardown if cleanup has issues
log.debug(f"Error during Google async client teardown: {exc}")
+ def _classify_google_client_error(self, exc: genai_errors.ClientError) -> ImgGenGenerationError:
+ """Classify a Google GenAI ClientError into a categorized ImgGenGenerationError."""
+ error_message = str(exc)
+ status_code = exc.code
+
+ if status_code == 404:
+ msg = f"Google model '{self.inference_model.desc}' not found: {exc}"
+ return ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONFIGURATION)
+
+ if status_code in {401, 403}:
+ msg = f"Google API permission denied for model '{self.inference_model.desc}': {exc}"
+ return ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONFIGURATION)
+
+ if status_code == 429:
+ if is_quota_exhaustion_google(error_message):
+ msg = f"Google quota exhausted for model '{self.inference_model.desc}': {exc}"
+ return ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your Google Cloud account has exceeded its quota — check billing at {URLs.google_billing}",
+ )
+ msg = f"Google rate limit exceeded for model '{self.inference_model.desc}': {exc}"
+ return ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by Google — the system will retry automatically",
+ )
+
+ if status_code == 400:
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {exc}"
+ return ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ )
+ msg = f"Google bad request error for model '{self.inference_model.desc}': {exc}"
+ return ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONTENT)
+
+ # Fallback for other 4xx errors
+ msg = f"Google API client error for model '{self.inference_model.desc}': {exc}"
+ return ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT)
+
@override
async def _gen_image(
self,
@@ -108,11 +157,17 @@ class GoogleImgGenWorker(ImgGenWorkerAbstract):
)
# Generate content using async client
- response = await self.genai_async_client.models.generate_content(
- model=self.inference_model.model_id,
- contents=prompt_text,
- config=generation_config,
- )
+ try:
+ response = await self.genai_async_client.models.generate_content(
+ model=self.inference_model.model_id,
+ contents=prompt_text,
+ config=generation_config,
+ )
+ except genai_errors.ServerError as exc:
+ msg = f"Google API server error for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+ except genai_errors.ClientError as exc:
+ raise self._classify_google_client_error(exc) from exc
usage_metadata: genai_types.GenerateContentResponseUsageMetadata | None = response.usage_metadata
if not usage_metadata:
diff --git a/pipelex/plugins/google/google_llm_worker.py b/pipelex/plugins/google/google_llm_worker.py
index 7e1da160f..5eac8b8f3 100644
--- a/pipelex/plugins/google/google_llm_worker.py
+++ b/pipelex/plugins/google/google_llm_worker.py
@@ -1,13 +1,18 @@
import asyncio
from typing import TYPE_CHECKING, cast
-from google import genai
+from google.genai import errors as genai_errors
from google.genai import types as genai_types
+from google.genai.client import Client as GoogleGenAiClient
from typing_extensions import override
from pipelex import log
from pipelex.base_exceptions import PipelexError
-from pipelex.cogt.exceptions import LLMCapabilityError
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCapabilityError, LLMCompletionError
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_google,
+)
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_job_components import LLMJobParams, ReasoningEffort
from pipelex.cogt.llm.llm_utils import dump_error, dump_kwargs, dump_response_from_structured_gen
@@ -19,6 +24,7 @@ from pipelex.config import get_config
from pipelex.plugins.google.google_factory import GoogleFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+from pipelex.urls import URLs
if TYPE_CHECKING:
from openai.types.chat import ChatCompletionMessageParam
@@ -31,7 +37,7 @@ class GoogleLLMWorkerError(PipelexError):
class GoogleLLMWorker(LLMWorkerInternalAbstract):
def __init__(
self,
- sdk_instance: genai.Client,
+ sdk_instance: GoogleGenAiClient,
inference_model: InferenceModelSpec,
reporting_delegate: ReportingProtocol | None = None,
):
@@ -39,7 +45,7 @@ class GoogleLLMWorker(LLMWorkerInternalAbstract):
inference_model=inference_model,
reporting_delegate=reporting_delegate,
)
- genai_client: genai.Client = sdk_instance
+ genai_client: GoogleGenAiClient = sdk_instance
self.genai_async_client = genai_client.aio
from instructor import from_genai # noqa: PLC0415
@@ -97,6 +103,53 @@ class GoogleLLMWorker(LLMWorkerInternalAbstract):
# Log but don't fail teardown if cleanup has issues
log.debug(f"Error during Google async client teardown: {exc}")
+ #########################################################
+ # Error classification
+ #########################################################
+
+ def _classify_google_client_error(self, exc: genai_errors.ClientError) -> LLMCompletionError:
+ """Classify a Google GenAI ClientError into a categorized LLMCompletionError."""
+ error_message = str(exc)
+ status_code = exc.code
+
+ if status_code == 404:
+ msg = f"Google model '{self.inference_model.desc}' not found: {exc}"
+ return LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION)
+
+ if status_code in {401, 403}:
+ msg = f"Google API permission denied for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION)
+
+ if status_code == 429:
+ if is_quota_exhaustion_google(error_message):
+ msg = f"Google quota exhausted for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your Google Cloud account has exceeded its quota — check billing at {URLs.google_billing}",
+ )
+ msg = f"Google rate limit exceeded for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by Google — the system will retry automatically",
+ )
+
+ if status_code == 400:
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ )
+ msg = f"Google bad request error for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT)
+
+ # Fallback for other 4xx errors
+ msg = f"Google API client error for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT)
+
#########################################################
# Reasoning helpers
#########################################################
@@ -205,11 +258,17 @@ class GoogleLLMWorker(LLMWorkerInternalAbstract):
generation_config.system_instruction = llm_job.llm_prompt.system_text
# Generate content using async client
- response = await self.genai_async_client.models.generate_content(
- model=self.inference_model.model_id,
- contents=contents,
- config=generation_config,
- )
+ try:
+ response = await self.genai_async_client.models.generate_content(
+ model=self.inference_model.model_id,
+ contents=contents,
+ config=generation_config,
+ )
+ except genai_errors.ServerError as exc:
+ msg = f"Google API server error for model '{self.inference_model.desc}': {exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+ except genai_errors.ClientError as exc:
+ raise self._classify_google_client_error(exc) from exc
# Extract text from response (skips thinking parts)
text_content = GoogleFactory.extract_text_from_response(response=response, model_desc=self.inference_model.desc)
@@ -239,13 +298,28 @@ class GoogleLLMWorker(LLMWorkerInternalAbstract):
candidate_count=1,
)
- result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
- messages=[cast("ChatCompletionMessageParam", contents)],
- response_model=schema,
- max_retries=llm_job.job_config.max_retries,
- model=self.inference_model.model_id,
- generation_config=generation_config,
- )
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from instructor.exceptions import InstructorRetryException # noqa: PLC0415
+
+ try:
+ result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
+ messages=[cast("ChatCompletionMessageParam", contents)],
+ response_model=schema,
+ max_retries=llm_job.job_config.max_retries,
+ model=self.inference_model.model_id,
+ generation_config=generation_config,
+ )
+ except genai_errors.ServerError as exc:
+ msg = f"Google API server error for model '{self.inference_model.desc}': {exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+ except genai_errors.ClientError as exc:
+ raise self._classify_google_client_error(exc) from exc
+ except LLMCompletionError:
+ raise
+ except InstructorRetryException as exc:
+ msg = f"Google structured generation failed after retries for model '{self.inference_model.desc}': {exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+
if not isinstance(result_object, schema):
msg = f"Google Gemini API returned an object that is not of type {schema}: {result_object}"
raise GoogleLLMWorkerError(msg)
diff --git a/pipelex/plugins/huggingface/huggingface_img_gen_worker.py b/pipelex/plugins/huggingface/huggingface_img_gen_worker.py
index 6c2a74536..a8c8952c6 100644
--- a/pipelex/plugins/huggingface/huggingface_img_gen_worker.py
+++ b/pipelex/plugins/huggingface/huggingface_img_gen_worker.py
@@ -1,11 +1,12 @@
from typing import Any
from huggingface_hub import AsyncInferenceClient
+from huggingface_hub.errors import HfHubHTTPError, InferenceTimeoutError
from PIL import Image
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import ImgGenParameterError, SdkTypeError
+from pipelex.cogt.exceptions import ImgGenGenerationError, ImgGenParameterError, InferenceErrorCategory, SdkTypeError
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.cogt.img_gen.img_gen_args_factory import ImgGenArgsFactory
from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
@@ -49,11 +50,37 @@ class HuggingFaceImgGenWorker(ImgGenWorkerAbstract):
if model_id is None:
msg = f"Model '{self.inference_model.name}' rules must include a 'model_choice' entry"
raise ImgGenParameterError(msg)
- return await self.hf_async_client.text_to_image(
- prompt=prompt,
- model=model_id,
- extra_body=args_dict,
- )
+ try:
+ return await self.hf_async_client.text_to_image(
+ prompt=prompt,
+ model=model_id,
+ extra_body=args_dict,
+ )
+ except InferenceTimeoutError as exc:
+ msg = f"HuggingFace request timed out for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+ except HfHubHTTPError as exc:
+ status_code: int | None = None
+ if hasattr(exc, "response") and exc.response is not None:
+ status_code = exc.response.status_code
+ if status_code == 429:
+ msg = f"HuggingFace rate limit exceeded for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by HuggingFace — the system will retry automatically",
+ ) from exc
+ if status_code == 402:
+ msg = f"HuggingFace quota exhausted for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CAPACITY) from exc
+ if status_code in {401, 403}:
+ msg = f"HuggingFace authentication error for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from exc
+ if status_code == 400:
+ msg = f"HuggingFace bad request for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+ msg = f"HuggingFace API error for model '{self.inference_model.desc}': {exc}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
@override
async def _gen_image(
diff --git a/pipelex/plugins/linkup/linkup_extract_worker.py b/pipelex/plugins/linkup/linkup_extract_worker.py
index 78439dbd6..f186fe9ee 100644
--- a/pipelex/plugins/linkup/linkup_extract_worker.py
+++ b/pipelex/plugins/linkup/linkup_extract_worker.py
@@ -1,8 +1,22 @@
from typing import Any
-from linkup import LinkupClient, LinkupFetchResponse
+from linkup import (
+ LinkupAuthenticationError,
+ LinkupClient,
+ LinkupFailedFetchError,
+ LinkupFetchResponse,
+ LinkupFetchResponseTooLargeError,
+ LinkupFetchUrlIsFileError,
+ LinkupInsufficientCreditError,
+ LinkupInvalidRequestError,
+ LinkupNoResultError,
+ LinkupTimeoutError,
+ LinkupTooManyRequestsError,
+ LinkupUnknownError,
+)
from typing_extensions import override
+from pipelex.cogt.exceptions import ExtractJobFailureError, InferenceErrorCategory
from pipelex.cogt.extract.extract_job import ExtractJob
from pipelex.cogt.extract.extract_output import ExtractedImageFromPage, ExtractOutput, Page
from pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstract
@@ -10,6 +24,7 @@ from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.hub import get_secrets_provider
from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.urls import URLs
class LinkupExtractWorker(ExtractWorkerAbstract):
@@ -41,12 +56,46 @@ class LinkupExtractWorker(ExtractWorkerAbstract):
job_params = extract_job.job_params
extract_images = job_params.max_nb_images is None or job_params.max_nb_images > 0
- response: LinkupFetchResponse = await self._linkup_client.async_fetch(
- url=document_uri,
- render_js=job_params.render_js,
- include_raw_html=job_params.include_raw_html,
- extract_images=extract_images,
- )
+ try:
+ response: LinkupFetchResponse = await self._linkup_client.async_fetch(
+ url=document_uri,
+ render_js=job_params.render_js,
+ include_raw_html=job_params.include_raw_html,
+ extract_images=extract_images,
+ )
+ except LinkupAuthenticationError as exc:
+ msg = f"Linkup authentication error: {exc}"
+ raise ExtractJobFailureError(
+ msg,
+ error_category=InferenceErrorCategory.CONFIGURATION,
+ user_action="Check that the LINKUP_API_KEY environment variable is set",
+ ) from exc
+ except LinkupInsufficientCreditError as exc:
+ msg = f"Linkup credits exhausted: {exc}"
+ raise ExtractJobFailureError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your Linkup account has insufficient credits — check billing at {URLs.linkup_billing}",
+ ) from exc
+ except LinkupTooManyRequestsError as exc:
+ msg = f"Linkup rate limit exceeded: {exc}"
+ raise ExtractJobFailureError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by Linkup — the system will retry automatically",
+ ) from exc
+ except LinkupTimeoutError as exc:
+ msg = f"Linkup request timed out: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
+ except LinkupInvalidRequestError as exc:
+ msg = f"Linkup invalid request: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+ except (LinkupFetchResponseTooLargeError, LinkupFetchUrlIsFileError) as exc:
+ msg = f"Linkup fetch error: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+ except (LinkupFailedFetchError, LinkupNoResultError, LinkupUnknownError) as exc:
+ msg = f"Linkup error: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
# Per-request cost model: costs are defined per million, so 1 request = 1_000_000
if extract_tokens_usage := extract_job.job_report.extract_tokens_usage:
diff --git a/pipelex/plugins/linkup/linkup_search_worker.py b/pipelex/plugins/linkup/linkup_search_worker.py
index e26ca680f..14849c909 100644
--- a/pipelex/plugins/linkup/linkup_search_worker.py
+++ b/pipelex/plugins/linkup/linkup_search_worker.py
@@ -1,9 +1,20 @@
from datetime import date
from typing import Any
-from linkup import LinkupClient, LinkupSourcedAnswer
+from linkup import (
+ LinkupAuthenticationError,
+ LinkupClient,
+ LinkupInsufficientCreditError,
+ LinkupInvalidRequestError,
+ LinkupNoResultError,
+ LinkupSourcedAnswer,
+ LinkupTimeoutError,
+ LinkupTooManyRequestsError,
+ LinkupUnknownError,
+)
from typing_extensions import override
+from pipelex.cogt.exceptions import InferenceErrorCategory, SearchJobFailureError
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.search.search_depth import SearchDepth
from pipelex.cogt.search.search_job import SearchJob
@@ -14,6 +25,7 @@ from pipelex.core.stuffs.search_result_content import SearchResultContent
from pipelex.hub import get_secrets_provider
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+from pipelex.urls import URLs
class LinkupSearchWorker(SearchWorkerAbstract):
@@ -26,6 +38,41 @@ class LinkupSearchWorker(SearchWorkerAbstract):
api_key = get_secrets_provider().get_secret(secret_id="LINKUP_API_KEY")
self._linkup_client = LinkupClient(api_key=api_key)
+ def _classify_linkup_error(self, exc: Exception) -> SearchJobFailureError:
+ """Classify a Linkup SDK error into a categorized SearchJobFailureError."""
+ if isinstance(exc, LinkupAuthenticationError):
+ msg = f"Linkup authentication error: {exc}"
+ return SearchJobFailureError(
+ msg,
+ error_category=InferenceErrorCategory.CONFIGURATION,
+ user_action="Check that the LINKUP_API_KEY environment variable is set",
+ )
+ if isinstance(exc, LinkupInsufficientCreditError):
+ msg = f"Linkup credits exhausted: {exc}"
+ return SearchJobFailureError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your Linkup account has insufficient credits — check billing at {URLs.linkup_billing}",
+ )
+ if isinstance(exc, LinkupTooManyRequestsError):
+ msg = f"Linkup rate limit exceeded: {exc}"
+ return SearchJobFailureError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by Linkup — the system will retry automatically",
+ )
+ if isinstance(exc, LinkupTimeoutError):
+ msg = f"Linkup request timed out: {exc}"
+ return SearchJobFailureError(msg, error_category=InferenceErrorCategory.TRANSIENT)
+ if isinstance(exc, LinkupInvalidRequestError):
+ msg = f"Linkup invalid request: {exc}"
+ return SearchJobFailureError(msg, error_category=InferenceErrorCategory.CONTENT)
+ if isinstance(exc, (LinkupNoResultError, LinkupUnknownError)):
+ msg = f"Linkup error: {exc}"
+ return SearchJobFailureError(msg, error_category=InferenceErrorCategory.TRANSIENT)
+ msg = f"Linkup error: {exc}"
+ return SearchJobFailureError(msg, error_category=InferenceErrorCategory.TRANSIENT)
+
def _parse_date(self, date_str: str | None) -> date | None:
if date_str is None:
return None
@@ -40,18 +87,29 @@ class LinkupSearchWorker(SearchWorkerAbstract):
search_setting = job_params.search_setting
depth_value = SearchDepth(self.inference_model.model_id.rsplit("/", 1)[-1])
- response: LinkupSourcedAnswer = await self._linkup_client.async_search(
- query=search_job.query,
- depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
- output_type="sourcedAnswer",
- include_images=search_setting.include_images,
- include_inline_citations=search_setting.include_inline_citations,
- max_results=search_setting.max_results,
- include_domains=job_params.include_domains,
- exclude_domains=job_params.exclude_domains,
- from_date=self._parse_date(job_params.from_date),
- to_date=self._parse_date(job_params.to_date),
- )
+ try:
+ response: LinkupSourcedAnswer = await self._linkup_client.async_search(
+ query=search_job.query,
+ depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
+ output_type="sourcedAnswer",
+ include_images=search_setting.include_images,
+ include_inline_citations=search_setting.include_inline_citations,
+ max_results=search_setting.max_results,
+ include_domains=job_params.include_domains,
+ exclude_domains=job_params.exclude_domains,
+ from_date=self._parse_date(job_params.from_date),
+ to_date=self._parse_date(job_params.to_date),
+ )
+ except (
+ LinkupAuthenticationError,
+ LinkupInsufficientCreditError,
+ LinkupTooManyRequestsError,
+ LinkupTimeoutError,
+ LinkupInvalidRequestError,
+ LinkupNoResultError,
+ LinkupUnknownError,
+ ) as exc:
+ raise self._classify_linkup_error(exc) from exc
# Per-request cost model: costs are defined per million, so 1 request = 1_000_000
if search_tokens_usage := search_job.job_report.search_tokens_usage:
@@ -84,19 +142,30 @@ class LinkupSearchWorker(SearchWorkerAbstract):
search_setting = job_params.search_setting
depth_value = SearchDepth(self.inference_model.model_id.rsplit("/", 1)[-1])
- response = await self._linkup_client.async_search(
- query=search_job.query,
- depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
- output_type="structured",
- structured_output_schema=schema,
- include_images=search_setting.include_images,
- max_results=search_setting.max_results,
- include_domains=job_params.include_domains,
- exclude_domains=job_params.exclude_domains,
- from_date=self._parse_date(job_params.from_date),
- to_date=self._parse_date(job_params.to_date),
- include_sources=True,
- )
+ try:
+ response = await self._linkup_client.async_search(
+ query=search_job.query,
+ depth=depth_value, # type: ignore[arg-type] # pyright: ignore[reportArgumentType]
+ output_type="structured",
+ structured_output_schema=schema,
+ include_images=search_setting.include_images,
+ max_results=search_setting.max_results,
+ include_domains=job_params.include_domains,
+ exclude_domains=job_params.exclude_domains,
+ from_date=self._parse_date(job_params.from_date),
+ to_date=self._parse_date(job_params.to_date),
+ include_sources=True,
+ )
+ except (
+ LinkupAuthenticationError,
+ LinkupInsufficientCreditError,
+ LinkupTooManyRequestsError,
+ LinkupTimeoutError,
+ LinkupInvalidRequestError,
+ LinkupNoResultError,
+ LinkupUnknownError,
+ ) as exc:
+ raise self._classify_linkup_error(exc) from exc
# Per-request cost model: costs are defined per million, so 1 request = 1_000_000
if search_tokens_usage := search_job.job_report.search_tokens_usage:
diff --git a/pipelex/plugins/mistral/mistral_extract_worker.py b/pipelex/plugins/mistral/mistral_extract_worker.py
index d92f979d9..1034ed830 100644
--- a/pipelex/plugins/mistral/mistral_extract_worker.py
+++ b/pipelex/plugins/mistral/mistral_extract_worker.py
@@ -1,17 +1,22 @@
from typing import Any
-from mistralai import Mistral
+from mistralai import Mistral, MistralError
from typing_extensions import override
-from pipelex.cogt.exceptions import ExtractCapabilityError, SdkTypeError
+from pipelex.cogt.exceptions import ExtractCapabilityError, ExtractJobFailureError, InferenceErrorCategory, SdkTypeError
from pipelex.cogt.extract.extract_input import ExtractInputError
from pipelex.cogt.extract.extract_job import ExtractJob
from pipelex.cogt.extract.extract_job_components import ExtractJobParams
from pipelex.cogt.extract.extract_output import ExtractOutput
from pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstract
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_mistral,
+)
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.plugins.mistral.mistral_factory import MistralFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.urls import URLs
class MistralExtractWorker(ExtractWorkerAbstract):
@@ -55,15 +60,65 @@ class MistralExtractWorker(ExtractWorkerAbstract):
raise ExtractInputError(msg)
return extract_output
+ def _classify_mistral_error(self, exc: MistralError) -> ExtractJobFailureError:
+ """Classify a Mistral SDK error into a categorized ExtractJobFailureError."""
+ error_message = str(exc)
+ status_code = exc.status_code
+
+ if is_quota_exhaustion_mistral(error_message, status_code):
+ msg = f"Mistral quota exhausted for model '{self.inference_model.desc}': {exc}"
+ return ExtractJobFailureError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your Mistral account has exceeded its quota — check billing at {URLs.mistral_billing}",
+ )
+
+ if status_code in {401, 403}:
+ msg = f"Mistral authentication error for model '{self.inference_model.desc}': {exc}"
+ return ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONFIGURATION)
+
+ if status_code == 404:
+ msg = f"Mistral model '{self.inference_model.desc}' not found: {exc}"
+ return ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONFIGURATION)
+
+ if status_code == 429:
+ msg = f"Mistral rate limit exceeded for model '{self.inference_model.desc}': {exc}"
+ return ExtractJobFailureError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by Mistral — the system will retry automatically",
+ )
+
+ if status_code == 400:
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {exc}"
+ return ExtractJobFailureError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the input",
+ )
+ msg = f"Mistral bad request error for model '{self.inference_model.desc}': {exc}"
+ return ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONTENT)
+
+ if status_code >= 500:
+ msg = f"Mistral server error for model '{self.inference_model.desc}': {exc}"
+ return ExtractJobFailureError(msg, error_category=InferenceErrorCategory.TRANSIENT)
+
+ msg = f"Mistral API error for model '{self.inference_model.desc}': {exc}"
+ return ExtractJobFailureError(msg, error_category=InferenceErrorCategory.TRANSIENT)
+
async def _extract_page_from_image(
self,
image_uri: str,
) -> ExtractOutput:
document = await MistralFactory.make_mistral_image_url_chunk_from_uri(uri=image_uri)
- extract_response = await self.mistral_client.ocr.process_async(
- model=self.inference_model.model_id,
- document=document,
- )
+ try:
+ extract_response = await self.mistral_client.ocr.process_async(
+ model=self.inference_model.model_id,
+ document=document,
+ )
+ except MistralError as exc:
+ raise self._classify_mistral_error(exc) from exc
return await MistralFactory.make_extract_output_from_mistral_response(
mistral_extract_response=extract_response,
)
@@ -88,13 +143,16 @@ class MistralExtractWorker(ExtractWorkerAbstract):
# include_image_base64 specifies return format; image_limit=0 means no images extracted
include_image_base64 = True
- extract_response = await self.mistral_client.ocr.process_async(
- model=self.inference_model.model_id,
- document=document,
- include_image_base64=include_image_base64,
- image_limit=image_limit,
- image_min_size=image_min_size,
- )
+ try:
+ extract_response = await self.mistral_client.ocr.process_async(
+ model=self.inference_model.model_id,
+ document=document,
+ include_image_base64=include_image_base64,
+ image_limit=image_limit,
+ image_min_size=image_min_size,
+ )
+ except MistralError as exc:
+ raise self._classify_mistral_error(exc) from exc
return await MistralFactory.make_extract_output_from_mistral_response(
mistral_extract_response=extract_response,
diff --git a/pipelex/plugins/mistral/mistral_factory.py b/pipelex/plugins/mistral/mistral_factory.py
index 202b1f174..1644975fb 100644
--- a/pipelex/plugins/mistral/mistral_factory.py
+++ b/pipelex/plugins/mistral/mistral_factory.py
@@ -1,6 +1,7 @@
import asyncio
import base64
import os
+from typing import TYPE_CHECKING
import aiofiles
import mistralai
@@ -17,15 +18,10 @@ from mistralai.models import (
UsageInfo,
UserMessage,
)
-from openai.types.chat import (
- ChatCompletionContentPartImageParam,
- ChatCompletionContentPartParam,
- ChatCompletionContentPartTextParam,
- ChatCompletionMessageParam,
- ChatCompletionSystemMessageParam,
- ChatCompletionUserMessageParam,
-)
-from openai.types.chat.chat_completion_content_part_image_param import ImageURL as OpenAIImageURL
+
+if TYPE_CHECKING:
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from openai.types.chat import ChatCompletionMessageParam
from pipelex import log
from pipelex.cogt.document.prompt_document import PromptDocument
@@ -125,12 +121,23 @@ class MistralFactory:
async def make_simple_messages_openai_typed(
self,
llm_job: LLMJob,
- ) -> list[ChatCompletionMessageParam]:
+ ) -> 'list["ChatCompletionMessageParam"]':
"""Makes a list of messages with a system message (if provided) and followed by a user message.
Uses the unified prep_prompt_images() which supports all URI types
including pipelex-storage://.
"""
+ # Deferred imports: avoid pulling heavy SDK at module-load time
+ from openai.types.chat import ( # noqa: PLC0415
+ ChatCompletionContentPartImageParam,
+ ChatCompletionContentPartParam,
+ ChatCompletionContentPartTextParam,
+ ChatCompletionMessageParam,
+ ChatCompletionSystemMessageParam,
+ ChatCompletionUserMessageParam,
+ )
+ from openai.types.chat.chat_completion_content_part_image_param import ImageURL as OpenAIImageURL # noqa: PLC0415
+
llm_prompt = llm_job.llm_prompt
messages: list[ChatCompletionMessageParam] = []
user_contents: list[ChatCompletionContentPartParam] = []
diff --git a/pipelex/plugins/mistral/mistral_llm_worker.py b/pipelex/plugins/mistral/mistral_llm_worker.py
index 2c9892585..7bc229a18 100644
--- a/pipelex/plugins/mistral/mistral_llm_worker.py
+++ b/pipelex/plugins/mistral/mistral_llm_worker.py
@@ -1,12 +1,16 @@
from typing import TYPE_CHECKING, Any
-from mistralai import Mistral
+from mistralai import Mistral, MistralError
from mistralai.models import MistralPromptMode, TextChunk, ThinkChunk
from mistralai.types import UNSET
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import LLMCapabilityError, LLMCompletionError, SdkTypeError
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCapabilityError, LLMCompletionError, SdkTypeError
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_mistral,
+)
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_job_components import LLMJobParams
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
@@ -17,6 +21,7 @@ from pipelex.plugins.mistral.mistral_exceptions import MistralWorkerConfiguratio
from pipelex.plugins.mistral.mistral_factory import MistralFactory
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+from pipelex.urls import URLs
if TYPE_CHECKING:
from mistralai.models import ChatCompletionResponse
@@ -55,6 +60,54 @@ class MistralLLMWorker(LLMWorkerInternalAbstract):
else:
self.instructor_for_objects = from_mistral(client=sdk_instance, use_async=True)
+ def _classify_mistral_error(self, exc: MistralError) -> LLMCompletionError:
+ """Classify a Mistral SDK error into a categorized LLMCompletionError."""
+ error_message = str(exc)
+ status_code = exc.status_code
+
+ if is_quota_exhaustion_mistral(error_message, status_code):
+ msg = f"Mistral quota exhausted for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your Mistral account has exceeded its quota — check billing at {URLs.mistral_billing}",
+ )
+
+ if status_code in {401, 403}:
+ msg = f"Mistral authentication error for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION)
+
+ if status_code == 404:
+ msg = f"Mistral model '{self.inference_model.desc}' not found: {exc}"
+ return LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION)
+
+ if status_code == 429:
+ msg = f"Mistral rate limit exceeded for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by Mistral — the system will retry automatically",
+ )
+
+ if status_code == 400:
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ )
+ msg = f"Mistral bad request error for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT)
+
+ if status_code >= 500:
+ msg = f"Mistral server error for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT)
+
+ # Fallback
+ msg = f"Mistral API error for model '{self.inference_model.desc}': {exc}"
+ return LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT)
+
def _resolve_prompt_mode(self, job_params: LLMJobParams) -> "OptionalNullable[MistralPromptMode]":
"""Resolve reasoning parameters to a Mistral prompt_mode value.
@@ -106,13 +159,17 @@ class MistralLLMWorker(LLMWorkerInternalAbstract):
job_params = llm_job.applied_job_params or llm_job.job_params
messages = await self.mistral_factory.make_simple_messages(llm_job=llm_job)
prompt_mode = self._resolve_prompt_mode(job_params=job_params)
- response: ChatCompletionResponse | None = await self.mistral_client_for_text.chat.complete_async(
- messages=messages,
- model=self.inference_model.model_id,
- temperature=job_params.temperature,
- max_tokens=job_params.max_tokens or self.default_max_tokens,
- prompt_mode=prompt_mode,
- )
+ try:
+ response: ChatCompletionResponse | None = await self.mistral_client_for_text.chat.complete_async(
+ messages=messages,
+ model=self.inference_model.model_id,
+ temperature=job_params.temperature,
+ max_tokens=job_params.max_tokens or self.default_max_tokens,
+ prompt_mode=prompt_mode,
+ )
+ except MistralError as exc:
+ raise self._classify_mistral_error(exc) from exc
+
if not response:
msg = "Mistral response is None"
raise LLMCompletionError(msg)
@@ -157,13 +214,25 @@ class MistralLLMWorker(LLMWorkerInternalAbstract):
job_params = llm_job.applied_job_params or llm_job.job_params
self._validate_no_reasoning_for_structured_gen(job_params=job_params)
messages = await self.mistral_factory.make_simple_messages_openai_typed(llm_job=llm_job)
- result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
- response_model=schema,
- messages=messages,
- model=self.inference_model.model_id,
- temperature=job_params.temperature,
- max_tokens=job_params.max_tokens or self.default_max_tokens,
- )
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from instructor.exceptions import InstructorRetryException # noqa: PLC0415
+
+ try:
+ result_object, completion = await self.instructor_for_objects.chat.completions.create_with_completion(
+ response_model=schema,
+ messages=messages,
+ model=self.inference_model.model_id,
+ temperature=job_params.temperature,
+ max_tokens=job_params.max_tokens or self.default_max_tokens,
+ )
+ except MistralError as exc:
+ raise self._classify_mistral_error(exc) from exc
+ except LLMCompletionError:
+ raise
+ except InstructorRetryException as exc:
+ msg = f"Mistral structured generation failed after retries for model '{self.inference_model.desc}': {exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+
if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := completion.usage):
llm_tokens_usage.nb_tokens_by_category = self.mistral_factory.make_nb_tokens_by_category(usage=usage)
diff --git a/pipelex/plugins/openai/openai_completions_factory.py b/pipelex/plugins/openai/openai_completions_factory.py
index 317a6f972..d3a290222 100644
--- a/pipelex/plugins/openai/openai_completions_factory.py
+++ b/pipelex/plugins/openai/openai_completions_factory.py
@@ -1,16 +1,5 @@
-from typing import Any
+from typing import TYPE_CHECKING, Any
-from openai.types.chat import (
- ChatCompletionContentPartImageParam,
- ChatCompletionContentPartParam,
- ChatCompletionContentPartTextParam,
- ChatCompletionMessageParam,
- ChatCompletionSystemMessageParam,
- ChatCompletionUserMessageParam,
-)
-from openai.types.chat.chat_completion_content_part_image_param import ImageURL as OpenAIImageURL
-from openai.types.chat.chat_completion_content_part_param import File as ChatCompletionContentPartFileParam
-from openai.types.completion_usage import CompletionUsage
from typing_extensions import override
from pipelex.cogt.document.prompt_document import PromptDocument
@@ -24,6 +13,11 @@ from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCateg
from pipelex.plugins.plugin_factory_abstract import PluginFactoryAbstract
from pipelex.tools.uri.prepared_file import PreparedFileBase64, PreparedFileHttpUrl, PreparedFileLocalPath
+if TYPE_CHECKING:
+ # Deferred imports: avoid pulling heavy SDK at module-load time
+ from openai.types.chat import ChatCompletionMessageParam
+ from openai.types.completion_usage import CompletionUsage
+
class OpenAICompletionsFactory(PluginFactoryAbstract):
def __init__(self, is_http_url_enabled: bool):
@@ -33,8 +27,20 @@ class OpenAICompletionsFactory(PluginFactoryAbstract):
async def make_simple_messages(
self,
llm_job: LLMJob,
- ) -> list[ChatCompletionMessageParam]:
+ ) -> 'list["ChatCompletionMessageParam"]':
"""Makes a list of messages with a system message (if provided) and followed by a user message."""
+ # Deferred imports: avoid pulling heavy SDK at module-load time
+ from openai.types.chat import ( # noqa: PLC0415
+ ChatCompletionContentPartImageParam,
+ ChatCompletionContentPartParam,
+ ChatCompletionContentPartTextParam,
+ ChatCompletionMessageParam,
+ ChatCompletionSystemMessageParam,
+ ChatCompletionUserMessageParam,
+ )
+ from openai.types.chat.chat_completion_content_part_image_param import ImageURL as OpenAIImageURL # noqa: PLC0415
+ from openai.types.chat.chat_completion_content_part_param import File as ChatCompletionContentPartFileParam # noqa: PLC0415
+
llm_prompt = llm_job.llm_prompt
messages: list[ChatCompletionMessageParam] = []
user_contents: list[ChatCompletionContentPartParam] = []
@@ -90,7 +96,7 @@ class OpenAICompletionsFactory(PluginFactoryAbstract):
# Note: we hardocde the extension to pdf because OpenAI Chat Completions API only supports PDF files at this stage
return f"document_{prompt_document.get_content_hash(length=12)}.pdf"
- def make_nb_tokens_by_category(self, usage: CompletionUsage) -> NbTokensByCategoryDict:
+ def make_nb_tokens_by_category(self, usage: "CompletionUsage") -> NbTokensByCategoryDict:
nb_tokens_by_category: NbTokensByCategoryDict = {
TokenCategory.INPUT: usage.prompt_tokens,
TokenCategory.OUTPUT: usage.completion_tokens,
diff --git a/pipelex/plugins/openai/openai_completions_img_gen_worker.py b/pipelex/plugins/openai/openai_completions_img_gen_worker.py
index 431bb7c85..8b87a8ac8 100644
--- a/pipelex/plugins/openai/openai_completions_img_gen_worker.py
+++ b/pipelex/plugins/openai/openai_completions_img_gen_worker.py
@@ -1,15 +1,19 @@
from typing import TYPE_CHECKING, Any, cast
import openai
-from openai import APIConnectionError, BadRequestError, NotFoundError
+from openai import APIConnectionError, APITimeoutError, AuthenticationError, BadRequestError, NotFoundError, RateLimitError
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import ImgGenGenerationError, ImgGenModelNotFoundError, ImgGenParameterError, SdkTypeError
+from pipelex.cogt.exceptions import ImgGenGenerationError, ImgGenModelNotFoundError, ImgGenParameterError, InferenceErrorCategory, SdkTypeError
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.cogt.image.prompt_image_utils import prep_prompt_images
from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
from pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstract
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_openai,
+)
from pipelex.cogt.inference.inference_constants import InferenceOutputType
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsFactory
@@ -17,6 +21,7 @@ from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.misc.base64_utils import extract_base64_str_from_base64_url_if_possible
from pipelex.tools.misc.image_utils import ImageFormat
from pipelex.tools.uri.prepared_file import PreparedFileBase64, PreparedFileHttpUrl
+from pipelex.urls import URLs
if TYPE_CHECKING:
from openai.types.chat import ChatCompletionMessage
@@ -81,12 +86,41 @@ class OpenAICompletionsImgGenWorker(ImgGenWorkerAbstract):
except NotFoundError as not_found_error:
msg = f"ImgGen model or deployment not found:\n{self.inference_model.desc}\nmodel: {self.inference_model.desc}\n{not_found_error}"
raise ImgGenModelNotFoundError(message=msg, model_handle=self.inference_model.name) from not_found_error
+ except RateLimitError as rate_limit_error:
+ error_message = str(rate_limit_error)
+ if is_quota_exhaustion_openai(error_message):
+ msg = f"OpenAI quota exhausted for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your OpenAI account has exceeded its quota — check billing at {URLs.openai_billing}",
+ ) from rate_limit_error
+ msg = f"OpenAI rate limit exceeded for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by OpenAI — the system will retry automatically",
+ ) from rate_limit_error
+ except APITimeoutError as timeout_error:
+ msg = f"OpenAI API request timed out for model '{self.inference_model.desc}': {timeout_error}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from timeout_error
except APIConnectionError as api_connection_error:
msg = f"ImgGen API connection error: {api_connection_error}"
- raise ImgGenGenerationError(msg) from api_connection_error
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from api_connection_error
except BadRequestError as bad_request_error:
+ error_message = str(bad_request_error)
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {bad_request_error}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ ) from bad_request_error
msg = f"ImgGen bad request error with model: {self.inference_model.desc}:\n{bad_request_error}"
- raise ImgGenGenerationError(msg) from bad_request_error
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONTENT) from bad_request_error
+ except AuthenticationError as authentication_error:
+ msg = f"ImgGen authentication error: {authentication_error}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from authentication_error
openai_message: ChatCompletionMessage = response.choices[0].message
actual_url: str | None = None
diff --git a/pipelex/plugins/openai/openai_completions_llm_worker.py b/pipelex/plugins/openai/openai_completions_llm_worker.py
index adb6ad589..9da3389c5 100644
--- a/pipelex/plugins/openai/openai_completions_llm_worker.py
+++ b/pipelex/plugins/openai/openai_completions_llm_worker.py
@@ -1,12 +1,16 @@
from typing import TYPE_CHECKING, Any
import openai
-from openai import NOT_GIVEN, APIConnectionError, AuthenticationError, BadRequestError, NotFoundError, omit
+from openai import NOT_GIVEN, APIConnectionError, APITimeoutError, AuthenticationError, BadRequestError, NotFoundError, RateLimitError, omit
from openai.types.chat import ChatCompletionReasoningEffort
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import LLMCapabilityError, LLMCompletionError, SdkTypeError
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCapabilityError, LLMCompletionError, SdkTypeError
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_openai,
+)
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_job_components import LLMJobParams
from pipelex.cogt.llm.llm_utils import dump_error, dump_kwargs, dump_response_from_structured_gen
@@ -19,6 +23,7 @@ from pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsF
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.system.telemetry.otel_constants import InferenceOutputType
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+from pipelex.urls import URLs
if TYPE_CHECKING:
from openai.types.chat import ChatCompletionMessage
@@ -141,21 +146,56 @@ class OpenAICompletionsLLMWorker(LLMWorkerInternalAbstract):
)
except NotFoundError as exc:
msg = f"LLM model or deployment '{self.inference_model.model_id}' not found: {exc}"
- raise LLMCompletionError(msg) from exc
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from exc
+ except RateLimitError as rate_limit_error:
+ error_message = str(rate_limit_error)
+ if is_quota_exhaustion_openai(error_message):
+ msg = f"OpenAI quota exhausted for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your OpenAI account has exceeded its quota — check billing at {URLs.openai_billing}",
+ ) from rate_limit_error
+ msg = f"OpenAI rate limit exceeded for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by OpenAI — the system will retry automatically",
+ ) from rate_limit_error
+ except APITimeoutError as timeout_error:
+ msg = f"OpenAI API request timed out for model '{self.inference_model.desc}': {timeout_error}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from timeout_error
except APIConnectionError as api_connection_error:
msg = f"LLM API connection error: {api_connection_error}"
- raise LLMCompletionError(msg) from api_connection_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from api_connection_error
except BadRequestError as bad_request_error:
+ error_message = str(bad_request_error)
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {bad_request_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ ) from bad_request_error
msg = f"LLM bad request error with model: {self.inference_model.desc}:\n{bad_request_error}"
- raise LLMCompletionError(msg) from bad_request_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from bad_request_error
except AuthenticationError as authentication_error:
msg = f"LLM authentication error: {authentication_error}"
- raise LLMCompletionError(msg) from authentication_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from authentication_error
if not response.choices:
msg = f"OpenAI chat completion response choices are empty with model: {self.inference_model.desc}"
raise LLMCompletionError(msg)
+ finish_reason = response.choices[0].finish_reason
+ if finish_reason == "content_filter":
+ msg = f"OpenAI response was filtered by content policy for model: {self.inference_model.desc}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ )
+
openai_message: ChatCompletionMessage = response.choices[0].message
response_text = openai_message.content
if response_text is None:
@@ -199,19 +239,45 @@ class OpenAICompletionsLLMWorker(LLMWorkerInternalAbstract):
f"LLM structured generation via 'instructor' failed with model: {self.inference_model.desc} "
f"trying to generate schema: {schema} with error: {exc}"
)
- raise LLMCompletionError(msg) from exc
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
except NotFoundError as exc:
msg = f"LLM model or deployment '{self.inference_model.model_id}' not found: {exc}"
- raise LLMCompletionError(msg) from exc
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from exc
+ except RateLimitError as rate_limit_error:
+ error_message = str(rate_limit_error)
+ if is_quota_exhaustion_openai(error_message):
+ msg = f"OpenAI quota exhausted for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your OpenAI account has exceeded its quota — check billing at {URLs.openai_billing}",
+ ) from rate_limit_error
+ msg = f"OpenAI rate limit exceeded for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by OpenAI — the system will retry automatically",
+ ) from rate_limit_error
+ except APITimeoutError as timeout_error:
+ msg = f"OpenAI API request timed out for model '{self.inference_model.desc}': {timeout_error}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from timeout_error
except APIConnectionError as api_connection_error:
msg = f"LLM API connection error: {api_connection_error}"
- raise LLMCompletionError(msg) from api_connection_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from api_connection_error
except BadRequestError as bad_request_error:
+ error_message = str(bad_request_error)
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {bad_request_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ ) from bad_request_error
msg = f"LLM bad request error with model: {self.inference_model.desc}:\n{bad_request_error}"
- raise LLMCompletionError(msg) from bad_request_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from bad_request_error
except AuthenticationError as authentication_error:
msg = f"LLM authentication error: {authentication_error}"
- raise LLMCompletionError(msg) from authentication_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from authentication_error
if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and (usage := completion.usage):
llm_tokens_usage.nb_tokens_by_category = self.openai_completions_factory.make_nb_tokens_by_category(usage=usage)
diff --git a/pipelex/plugins/openai/openai_config.py b/pipelex/plugins/openai/openai_config.py
index 50bca7bf3..7653f0dd4 100644
--- a/pipelex/plugins/openai/openai_config.py
+++ b/pipelex/plugins/openai/openai_config.py
@@ -1,6 +1,5 @@
-from typing import cast
+from typing import TYPE_CHECKING, cast
-from openai.types.chat import ChatCompletionReasoningEffort
from pydantic import field_validator
from pipelex.cogt.llm.llm_job_components import ReasoningEffort
@@ -8,6 +7,10 @@ from pipelex.cogt.llm.reasoning_config_base import EffortToLevelMap, get_reasoni
from pipelex.system.configuration.config_model import ConfigModel
from pipelex.types import StrEnum
+if TYPE_CHECKING:
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from openai.types.chat import ChatCompletionReasoningEffort
+
class OpenAIReasoningLevel(StrEnum):
NONE = "none"
@@ -26,7 +29,7 @@ class OpenAIConfig(ConfigModel):
def validate_effort_map(cls, value: EffortToLevelMap) -> EffortToLevelMap:
return validate_effort_to_level_map(value, "openai_config", level_type=OpenAIReasoningLevel)
- def get_reasoning_level(self, effort: ReasoningEffort) -> ChatCompletionReasoningEffort | None:
+ def get_reasoning_level(self, effort: ReasoningEffort) -> "ChatCompletionReasoningEffort | None":
"""Resolve a ReasoningEffort to an OpenAI ChatCompletionReasoningEffort value.
Returns:
diff --git a/pipelex/plugins/openai/openai_img_gen_worker.py b/pipelex/plugins/openai/openai_img_gen_worker.py
index 154f5aabf..6a7cb50c3 100644
--- a/pipelex/plugins/openai/openai_img_gen_worker.py
+++ b/pipelex/plugins/openai/openai_img_gen_worker.py
@@ -2,21 +2,27 @@ import base64
from typing import TYPE_CHECKING, Any, cast
import openai
+from openai import APIConnectionError, APITimeoutError, AuthenticationError, BadRequestError, NotFoundError, RateLimitError
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import ImgGenGenerationError, ImgGenParameterError, SdkTypeError
+from pipelex.cogt.exceptions import ImgGenGenerationError, ImgGenModelNotFoundError, ImgGenParameterError, InferenceErrorCategory, SdkTypeError
from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
from pipelex.cogt.image.image_size import ImageSize
from pipelex.cogt.img_gen.img_gen_args_factory import ImgGenArgsFactory
from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
from pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstract
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_openai,
+)
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.tools.misc.base64_utils import extract_base64_str_from_base64_url_if_possible
from pipelex.tools.misc.filetype_utils import detect_file_type_from_bytes
from pipelex.tools.misc.image_utils import ImageFormat
+from pipelex.urls import URLs
if TYPE_CHECKING:
from openai.types.images_response import ImagesResponse, Usage
@@ -64,13 +70,52 @@ class OpenAIImgGenWorker(ImgGenWorkerAbstract):
)
images_response: ImagesResponse
- if image_arg := args_dict.get("image"):
- args_dict["image"] = self._convert_image_data_urls_for_openai_sdk(image_arg=image_arg)
- if args_dict.pop("moderation", None) is not None:
- log.warning("OpenAI images.edit does not accept 'moderation'; dropping the kwarg")
- images_response = cast("ImagesResponse", await self.openai_client.images.edit(**args_dict))
- else:
- images_response = cast("ImagesResponse", await self.openai_client.images.generate(**args_dict))
+ try:
+ if image_arg := args_dict.get("image"):
+ args_dict["image"] = self._convert_image_data_urls_for_openai_sdk(image_arg=image_arg)
+ if args_dict.pop("moderation", None) is not None:
+ log.warning("OpenAI images.edit does not accept 'moderation'; dropping the kwarg")
+ images_response = cast("ImagesResponse", await self.openai_client.images.edit(**args_dict))
+ else:
+ images_response = cast("ImagesResponse", await self.openai_client.images.generate(**args_dict))
+ except NotFoundError as not_found_error:
+ msg = f"ImgGen model or deployment not found: {self.inference_model.desc}: {not_found_error}"
+ raise ImgGenModelNotFoundError(message=msg, model_handle=self.inference_model.name) from not_found_error
+ except RateLimitError as rate_limit_error:
+ error_message = str(rate_limit_error)
+ if is_quota_exhaustion_openai(error_message):
+ msg = f"OpenAI quota exhausted for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your OpenAI account has exceeded its quota — check billing at {URLs.openai_billing}",
+ ) from rate_limit_error
+ msg = f"OpenAI rate limit exceeded for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by OpenAI — the system will retry automatically",
+ ) from rate_limit_error
+ except APITimeoutError as timeout_error:
+ msg = f"OpenAI API request timed out for model '{self.inference_model.desc}': {timeout_error}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from timeout_error
+ except APIConnectionError as api_connection_error:
+ msg = f"ImgGen API connection error: {api_connection_error}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.TRANSIENT) from api_connection_error
+ except BadRequestError as bad_request_error:
+ error_message = str(bad_request_error)
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {bad_request_error}"
+ raise ImgGenGenerationError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ ) from bad_request_error
+ msg = f"ImgGen bad request error with model: {self.inference_model.desc}:\n{bad_request_error}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONTENT) from bad_request_error
+ except AuthenticationError as authentication_error:
+ msg = f"ImgGen authentication error: {authentication_error}"
+ raise ImgGenGenerationError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from authentication_error
if not images_response.data:
msg = "No result from OpenAI"
diff --git a/pipelex/plugins/openai/openai_responses_llm_worker.py b/pipelex/plugins/openai/openai_responses_llm_worker.py
index 8368242bd..a8ddc45ab 100644
--- a/pipelex/plugins/openai/openai_responses_llm_worker.py
+++ b/pipelex/plugins/openai/openai_responses_llm_worker.py
@@ -3,17 +3,22 @@ from __future__ import annotations
from typing import TYPE_CHECKING, Any, cast
import openai
-from openai import NOT_GIVEN, APIConnectionError, AuthenticationError, BadRequestError, NotFoundError, omit
+from openai import NOT_GIVEN, APIConnectionError, APITimeoutError, AuthenticationError, BadRequestError, NotFoundError, RateLimitError, omit
from openai.types.shared_params import Reasoning
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.exceptions import LLMCapabilityError, LLMCompletionError, LLMModelNotFoundError, SdkTypeError
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCapabilityError, LLMCompletionError, LLMModelNotFoundError, SdkTypeError
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_openai,
+)
from pipelex.cogt.llm.llm_utils import dump_error, dump_kwargs, dump_response_from_structured_gen
from pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstract
from pipelex.cogt.llm.thinking_mode import ThinkingMode
from pipelex.config import get_config
from pipelex.system.telemetry.otel_constants import InferenceOutputType
+from pipelex.urls import URLs
if TYPE_CHECKING:
from openai.types.chat import ChatCompletionMessageParam
@@ -142,15 +147,41 @@ class OpenAIResponsesLLMWorker(LLMWorkerInternalAbstract):
f"OpenAI Responses model or deployment not found:\n{self.inference_model.desc}\nmodel: {self.inference_model.desc}\n{not_found_error}"
)
raise LLMModelNotFoundError(message=msg, model_handle=self.inference_model.name) from not_found_error
+ except RateLimitError as rate_limit_error:
+ error_message = str(rate_limit_error)
+ if is_quota_exhaustion_openai(error_message):
+ msg = f"OpenAI quota exhausted for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your OpenAI account has exceeded its quota — check billing at {URLs.openai_billing}",
+ ) from rate_limit_error
+ msg = f"OpenAI rate limit exceeded for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by OpenAI — the system will retry automatically",
+ ) from rate_limit_error
+ except APITimeoutError as timeout_error:
+ msg = f"OpenAI API request timed out for model '{self.inference_model.desc}': {timeout_error}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from timeout_error
except APIConnectionError as api_connection_error:
msg = f"OpenAI API connection error: {api_connection_error}"
- raise LLMCompletionError(msg) from api_connection_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from api_connection_error
except BadRequestError as bad_request_error:
+ error_message = str(bad_request_error)
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {bad_request_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ ) from bad_request_error
msg = f"OpenAI bad request error with model: {self.inference_model.desc}:\n{bad_request_error}"
- raise LLMCompletionError(msg) from bad_request_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from bad_request_error
except AuthenticationError as authentication_error:
msg = f"Authentication error: {authentication_error}"
- raise LLMCompletionError(msg) from authentication_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from authentication_error
if not response.output_text:
msg = f"OpenAI Responses message content is empty: {response}\nmodel: {self.inference_model.desc}"
@@ -191,16 +222,45 @@ class OpenAIResponsesLLMWorker(LLMWorkerInternalAbstract):
) # type: ignore[arg-type,misc]
except InstructorRetryException as exc:
msg = f"OpenAI instructor failed with model: {self.inference_model.desc} trying to generate schema: {schema} with error: {exc}"
- raise LLMCompletionError(msg) from exc
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
except NotFoundError as exc:
msg = f"OpenAI Responses model or deployment '{self.inference_model.model_id}' not found: {exc}"
- raise LLMCompletionError(msg) from exc
+ raise LLMModelNotFoundError(message=msg, model_handle=self.inference_model.name) from exc
+ except RateLimitError as rate_limit_error:
+ error_message = str(rate_limit_error)
+ if is_quota_exhaustion_openai(error_message):
+ msg = f"OpenAI quota exhausted for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CAPACITY,
+ user_action=f"Your OpenAI account has exceeded its quota — check billing at {URLs.openai_billing}",
+ ) from rate_limit_error
+ msg = f"OpenAI rate limit exceeded for model '{self.inference_model.desc}': {rate_limit_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Rate limited by OpenAI — the system will retry automatically",
+ ) from rate_limit_error
+ except APITimeoutError as timeout_error:
+ msg = f"OpenAI API request timed out for model '{self.inference_model.desc}': {timeout_error}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from timeout_error
+ except APIConnectionError as api_connection_error:
+ msg = f"OpenAI API connection error: {api_connection_error}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from api_connection_error
except BadRequestError as bad_request_error:
+ error_message = str(bad_request_error)
+ if is_content_policy_violation(error_message):
+ msg = f"Content rejected by safety filters for model '{self.inference_model.desc}': {bad_request_error}"
+ raise LLMCompletionError(
+ msg,
+ error_category=InferenceErrorCategory.CONTENT,
+ user_action="Content was rejected by safety filters — revise the prompt",
+ ) from bad_request_error
msg = f"OpenAI bad request error with model: {self.inference_model.desc}:\n{bad_request_error}"
- raise LLMCompletionError(msg) from bad_request_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from bad_request_error
except AuthenticationError as authentication_error:
msg = f"Authentication error: {authentication_error}"
- raise LLMCompletionError(msg) from authentication_error
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from authentication_error
if (llm_tokens_usage := llm_job.job_report.llm_tokens_usage) and hasattr(completion, "usage"):
completion_usage = completion.usage
diff --git a/pipelex/plugins/portkey/portkey_completions_factory.py b/pipelex/plugins/portkey/portkey_completions_factory.py
index 04c9a4465..a35e8dd3a 100644
--- a/pipelex/plugins/portkey/portkey_completions_factory.py
+++ b/pipelex/plugins/portkey/portkey_completions_factory.py
@@ -2,16 +2,6 @@ from __future__ import annotations
from typing import TYPE_CHECKING, Any
-import openai
-from openai.types.chat import (
- ChatCompletionContentPartImageParam,
- ChatCompletionContentPartParam,
- ChatCompletionContentPartTextParam,
- ChatCompletionMessageParam,
- ChatCompletionSystemMessageParam,
- ChatCompletionUserMessageParam,
-)
-from openai.types.chat.chat_completion_content_part_image_param import ImageURL as OpenAIImageURL
from portkey_ai import (
createHeaders, # type: ignore[reportUnknownVariableType]
)
@@ -27,6 +17,10 @@ from pipelex.plugins.portkey.portkey_factory import PortkeyFactory
from pipelex.tools.uri.prepared_file import PreparedFileBase64, PreparedFileHttpUrl, PreparedFileLocalPath
if TYPE_CHECKING:
+ # Deferred imports: avoid pulling heavy SDK at module-load time
+ import openai
+ from openai.types.chat import ChatCompletionMessageParam
+
from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.model_backends.backend import InferenceBackend
@@ -41,6 +35,16 @@ class PortkeyCompletionsFactory(OpenAICompletionsFactory):
llm_job: LLMJob,
) -> list[ChatCompletionMessageParam]:
"""Override to use image_url format for documents which Portkey translates correctly."""
+ # Deferred imports: avoid pulling heavy SDK at module-load time
+ from openai.types.chat import ( # noqa: PLC0415
+ ChatCompletionContentPartImageParam,
+ ChatCompletionContentPartParam,
+ ChatCompletionContentPartTextParam,
+ ChatCompletionSystemMessageParam,
+ ChatCompletionUserMessageParam,
+ )
+ from openai.types.chat.chat_completion_content_part_image_param import ImageURL as OpenAIImageURL # noqa: PLC0415
+
llm_prompt = llm_job.llm_prompt
messages: list[ChatCompletionMessageParam] = []
user_contents: list[ChatCompletionContentPartParam] = []
@@ -92,6 +96,9 @@ class PortkeyCompletionsFactory(OpenAICompletionsFactory):
plugin: Plugin,
backend: InferenceBackend,
) -> openai.AsyncOpenAI:
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ import openai # noqa: PLC0415
+
is_debug_enabled = PortkeyFactory.is_debug_enabled(backend=backend)
endpoint = PortkeyFactory.get_endpoint(backend=backend)
api_key = PortkeyFactory.get_api_key(backend=backend)
diff --git a/pipelex/plugins/pypdfium2/pypdfium2_worker.py b/pipelex/plugins/pypdfium2/pypdfium2_worker.py
index 66aa89918..ffe700a03 100644
--- a/pipelex/plugins/pypdfium2/pypdfium2_worker.py
+++ b/pipelex/plugins/pypdfium2/pypdfium2_worker.py
@@ -4,6 +4,7 @@ from typing import Any
from typing_extensions import override
from pipelex import log
+from pipelex.cogt.exceptions import ExtractJobFailureError, InferenceErrorCategory
from pipelex.cogt.extract.extract_input import ExtractInputError
from pipelex.cogt.extract.extract_job import ExtractJob
from pipelex.cogt.extract.extract_output import ExtractOutput, Page
@@ -78,12 +79,26 @@ class Pypdfium2Worker(ExtractWorkerAbstract):
max_nb_images = extract_job.job_params.max_nb_images
should_extract_images = max_nb_images is None or max_nb_images > 0
- if should_extract_images:
- all_page_images = await pypdfium2_renderer.extract_embedded_images_from_pdf(pdf_input=pdf_input)
- else:
- all_page_images = {}
-
- all_page_texts = await pypdfium2_renderer.extract_text_from_pdf_pages(pdf_input=pdf_input)
+ try:
+ if should_extract_images:
+ all_page_images = await pypdfium2_renderer.extract_embedded_images_from_pdf(pdf_input=pdf_input)
+ else:
+ all_page_images = {}
+
+ all_page_texts = await pypdfium2_renderer.extract_text_from_pdf_pages(pdf_input=pdf_input)
+ # The exceptions below are raised by pypdfium2_renderer for corrupt/invalid PDFs or file system issues.
+ except FileNotFoundError as exc:
+ msg = f"PDF file not found: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONFIGURATION) from exc
+ except ValueError as exc:
+ msg = f"Invalid PDF format: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+ except RuntimeError as exc:
+ msg = f"PDF extraction failed: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.CONTENT) from exc
+ except OSError as exc:
+ msg = f"I/O error during PDF extraction: {exc}"
+ raise ExtractJobFailureError(msg, error_category=InferenceErrorCategory.TRANSIENT) from exc
pages: dict[int, Page] = {}
total_images_count = 0
diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.py
index 765d29785..490227793 100644
--- a/pipelex/reporting/reporting_manager.py
+++ b/pipelex/reporting/reporting_manager.py
@@ -1,3 +1,6 @@
+from collections.abc import Sequence
+from dataclasses import dataclass
+from datetime import datetime, timezone
from typing import TYPE_CHECKING
from pydantic import Field, RootModel
@@ -6,25 +9,34 @@ from typing_extensions import override
from pipelex import log
from pipelex.cogt.exceptions import ReportingManagerError
from pipelex.cogt.extract.extract_job import ExtractJob
-from pipelex.cogt.extract.extract_report import ExtractTokensUsage
from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
-from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage
from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
from pipelex.cogt.llm.llm_job import LLMJob
from pipelex.cogt.llm.llm_report import LLMTokensUsage
from pipelex.cogt.search.search_job import SearchJob
-from pipelex.cogt.search.search_report import SearchTokensUsage
from pipelex.cogt.usage.cost_registry import CostRegistry
from pipelex.config import get_config
from pipelex.pipeline.pipeline_models import SpecialPipelineId
from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.reporting.reporting_types import AnyTokensUsage, TokensUsage
from pipelex.tools.misc.file_utils import ensure_path, get_incremental_file_path
from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+from pipelex.tracing.event_log_protocol import EventLogProtocol
+from pipelex.tracing.trace_events import UsageReportEvent
if TYPE_CHECKING:
from pathlib import Path
-TokensUsage = LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage
+
+@dataclass
+class _EventLogContext:
+ """Per-workflow/run event log state. Private to ReportingManager."""
+
+ event_log: EventLogProtocol
+ workflow_id: str
+ pipeline_run_id: str
+
+
UsageRegistryRoot = list[TokensUsage]
@@ -42,6 +54,40 @@ class ReportingManager(ReportingProtocol):
def __init__(self):
self._reporting_config = get_config().pipelex.reporting_config
self._usage_registries: dict[str, UsageRegistry] = {}
+ # Per-context event log state, keyed by graph_context.lookup_key.
+ # Each concurrent workflow/run gets its own isolated context.
+ self._event_log_contexts: dict[str, _EventLogContext] = {}
+
+ ############################################################
+ # Event log configuration
+ ############################################################
+
+ @override
+ def set_event_log(
+ self,
+ context_key: str,
+ event_log: EventLogProtocol,
+ workflow_id: str,
+ pipeline_run_id: str,
+ ) -> None:
+ """Configure event log for a specific workflow/run context.
+
+ Args:
+ context_key: Unique key for this context (graph_context.lookup_key).
+ event_log: The event log backend for emitting UsageReportEvents.
+ workflow_id: Temporal workflow ID or "direct".
+ pipeline_run_id: Pipeline run ID for event correlation.
+ """
+ self._event_log_contexts[context_key] = _EventLogContext(
+ event_log=event_log,
+ workflow_id=workflow_id,
+ pipeline_run_id=pipeline_run_id,
+ )
+
+ @override
+ def clear_event_log(self, context_key: str) -> None:
+ """Remove event log configuration for a completed workflow/run."""
+ self._event_log_contexts.pop(context_key, None)
############################################################
# Manager lifecycle
@@ -55,6 +101,7 @@ class ReportingManager(ReportingProtocol):
@override
def teardown(self):
self._usage_registries.clear()
+ self._event_log_contexts.clear()
############################################################
# Private methods
@@ -62,8 +109,12 @@ class ReportingManager(ReportingProtocol):
def _get_registry(self, pipeline_run_id: str) -> UsageRegistry:
if pipeline_run_id not in self._usage_registries:
- msg = f"Registry for pipeline '{pipeline_run_id}' does not exist"
- raise ReportingManagerError(msg)
+ # Auto-create registry for unknown pipeline IDs. This happens when
+ # Activities report inference jobs on a Temporal worker where
+ # open_registry() was never called (it runs on the API process).
+
+ # TODO: replace with proper distributed reporting system
+ self._usage_registries[pipeline_run_id] = UsageRegistry()
return self._usage_registries[pipeline_run_id]
def _report_llm_job(self, llm_job: LLMJob):
@@ -75,6 +126,7 @@ class ReportingManager(ReportingProtocol):
pipeline_run_id = llm_job.job_metadata.pipeline_run_id
self._get_registry(pipeline_run_id).add_tokens_usage(llm_tokens_usage)
+ self._emit_usage_event(llm_job, llm_tokens_usage)
if self._reporting_config.is_log_costs_to_console:
llm_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=llm_tokens_usage)
@@ -89,6 +141,7 @@ class ReportingManager(ReportingProtocol):
pipeline_run_id = img_gen_job.job_metadata.pipeline_run_id
self._get_registry(pipeline_run_id).add_tokens_usage(img_gen_tokens_usage)
+ self._emit_usage_event(img_gen_job, img_gen_tokens_usage)
if self._reporting_config.is_log_costs_to_console:
img_gen_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=img_gen_tokens_usage)
@@ -103,6 +156,7 @@ class ReportingManager(ReportingProtocol):
pipeline_run_id = extract_job.job_metadata.pipeline_run_id
self._get_registry(pipeline_run_id).add_tokens_usage(extract_tokens_usage)
+ self._emit_usage_event(extract_job, extract_tokens_usage)
if self._reporting_config.is_log_costs_to_console:
extract_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=extract_tokens_usage)
@@ -117,6 +171,7 @@ class ReportingManager(ReportingProtocol):
pipeline_run_id = search_job.job_metadata.pipeline_run_id
self._get_registry(pipeline_run_id).add_tokens_usage(search_tokens_usage)
+ self._emit_usage_event(search_job, search_tokens_usage)
if self._reporting_config.is_log_costs_to_console:
search_token_cost_report = CostRegistry.complete_cost_report(tokens_usage=search_tokens_usage)
@@ -133,6 +188,47 @@ class ReportingManager(ReportingProtocol):
raise ReportingManagerError(msg)
self._usage_registries[pipeline_run_id] = UsageRegistry()
+ def inject_tokens_usages(self, pipeline_run_id: str, tokens_usages: Sequence[AnyTokensUsage]) -> None:
+ """Inject externally-collected token usage records into a pipeline's registry.
+
+ Used after assembling usage data from distributed trace events, so that
+ generate_report() can produce a complete cost report across all workers.
+
+ Args:
+ pipeline_run_id: The pipeline run to add usage data to.
+ tokens_usages: Token usage records to inject.
+ """
+ registry = self._get_registry(pipeline_run_id)
+ for tokens_usage in tokens_usages:
+ registry.add_tokens_usage(tokens_usage)
+
+ def _emit_usage_event(self, inference_job: InferenceJobAbstract, tokens_usage: AnyTokensUsage) -> None:
+ """Emit a UsageReportEvent if event log is configured for this job's context."""
+ graph_context = inference_job.job_metadata.graph_context
+ if graph_context is None:
+ return
+
+ context = self._event_log_contexts.get(graph_context.lookup_key)
+ if context is None:
+ return
+
+ # Determine the node_id from graph context (the pipe that dispatched this inference)
+ node_id: str = "unknown"
+ if graph_context.parent_node_id is not None:
+ node_id = graph_context.parent_node_id
+
+ seq = context.event_log.next_sequence()
+
+ event = UsageReportEvent(
+ pipeline_run_id=context.pipeline_run_id,
+ workflow_id=context.workflow_id,
+ timestamp=datetime.now(timezone.utc),
+ sequence=seq,
+ node_id=node_id,
+ tokens_usage=tokens_usage,
+ )
+ context.event_log.emit(event)
+
@override
def report_inference_job(self, inference_job: InferenceJobAbstract):
log.verbose(f"Inference job '{inference_job.job_metadata.unit_job_id}' completed in {inference_job.job_metadata.duration:.2f} seconds")
diff --git a/pipelex/reporting/reporting_protocol.py b/pipelex/reporting/reporting_protocol.py
index 9d8664a5e..b772d09e1 100644
--- a/pipelex/reporting/reporting_protocol.py
+++ b/pipelex/reporting/reporting_protocol.py
@@ -3,6 +3,7 @@ from typing import Protocol
from typing_extensions import override
from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstract
+from pipelex.tracing.event_log_protocol import EventLogProtocol
class ReportingProtocol(Protocol):
@@ -18,6 +19,16 @@ class ReportingProtocol(Protocol):
def teardown(self): ...
+ def set_event_log(
+ self,
+ context_key: str,
+ event_log: EventLogProtocol,
+ workflow_id: str,
+ pipeline_run_id: str,
+ ) -> None: ...
+
+ def clear_event_log(self, context_key: str) -> None: ...
+
class ReportingNoOp(ReportingProtocol):
@override
@@ -43,3 +54,17 @@ class ReportingNoOp(ReportingProtocol):
@override
def teardown(self):
pass
+
+ @override
+ def set_event_log(
+ self,
+ context_key: str,
+ event_log: EventLogProtocol,
+ workflow_id: str,
+ pipeline_run_id: str,
+ ) -> None:
+ pass
+
+ @override
+ def clear_event_log(self, context_key: str) -> None:
+ pass
diff --git a/pipelex/reporting/reporting_types.py b/pipelex/reporting/reporting_types.py
new file mode 100644
index 000000000..ef5c47e23
--- /dev/null
+++ b/pipelex/reporting/reporting_types.py
@@ -0,0 +1,16 @@
+"""Type aliases for reporting token usage across inference backends."""
+
+from typing import Annotated
+
+from pydantic import Field
+
+from pipelex.cogt.extract.extract_report import ExtractTokensUsage
+from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.search.search_report import SearchTokensUsage
+
+AnyTokensUsage = Annotated[
+ LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage,
+ Field(discriminator="model_type"),
+]
+TokensUsage = AnyTokensUsage
diff --git a/pipelex/system/configuration/config_loader.py b/pipelex/system/configuration/config_loader.py
index 87f89748d..4bde9346e 100644
--- a/pipelex/system/configuration/config_loader.py
+++ b/pipelex/system/configuration/config_loader.py
@@ -3,6 +3,7 @@ from pathlib import Path
from typing import Any
from pipelex.system.runtime import runtime_manager
+from pipelex.tools.misc.json_utils import deep_update
from pipelex.tools.misc.toml_utils import load_toml_from_path_and_merge_with_overrides
CONFIG_DIR_NAME = ".pipelex"
@@ -165,7 +166,7 @@ class ConfigLoader:
write_manifest(global_dir / "inference" / "deck", compute_kit_manifest())
- def load_config(self) -> dict[str, Any]:
+ def load_config(self, extra_overrides: dict[str, Any] | None = None) -> dict[str, Any]:
"""Load and merge configurations from pipelex and local config files.
The configuration is loaded and merged in the following order:
@@ -176,7 +177,8 @@ class ConfigLoader:
- pipelex_local.toml (local execution)
- pipelex_{environment}.toml
- pipelex_{run_mode}.toml
- - pipelex_override.toml (final override)
+ - pipelex_override.toml (user's final override)
+ - pipelex_temporary_override.toml (ephemeral, safe for tools to create/delete)
Returns:
dict[str, Any]: The merged configuration dictionary
@@ -214,7 +216,13 @@ class ConfigLoader:
# Final override
list_of_configs.append(effective_config_dir / "pipelex_override.toml")
- return load_toml_from_path_and_merge_with_overrides(paths=list_of_configs)
+ # Temporary override (e.g. for testing tools — safe to create/delete)
+ list_of_configs.append(effective_config_dir / "pipelex_temporary_override.toml")
+
+ merged = load_toml_from_path_and_merge_with_overrides(paths=list_of_configs)
+ if extra_overrides:
+ deep_update(merged, extra_overrides)
+ return merged
config_manager = ConfigLoader()
diff --git a/pipelex/system/configuration/configs.py b/pipelex/system/configuration/configs.py
index 13e3afacb..f87d5c894 100644
--- a/pipelex/system/configuration/configs.py
+++ b/pipelex/system/configuration/configs.py
@@ -1,5 +1,5 @@
import shortuuid
-from pydantic import Field, field_validator
+from pydantic import Field, field_validator, model_validator
from pipelex.base_exceptions import PipelexConfigError
from pipelex.cogt.config_cogt import Cogt
@@ -9,6 +9,7 @@ from pipelex.graph.graph_config import GraphConfig
from pipelex.language.mthds_config import MthdsConfig
from pipelex.system.configuration.config_model import ConfigModel
from pipelex.system.configuration.config_root import ConfigRoot
+from pipelex.temporal.config_temporal import Temporal
from pipelex.tools.aws.aws_config import AwsConfig
from pipelex.tools.log.log_config import LogConfig
from pipelex.tools.storage.storage_config import StorageConfig
@@ -33,7 +34,24 @@ class AgentTarget(StrEnum):
class KitConfig(ConfigModel):
- preferred_agent_target: AgentTarget = Field(strict=False)
+ preferred_agent_targets: list[AgentTarget] = Field(strict=False)
+
+ @field_validator("preferred_agent_targets", mode="before")
+ @classmethod
+ def _coerce_targets(cls, value: object) -> object:
+ if isinstance(value, list):
+ return [AgentTarget(item) if isinstance(item, str) else item for item in value] # pyright: ignore[reportUnknownVariableType]
+ return value
+
+ @model_validator(mode="after")
+ def _validate_targets(self) -> Self:
+ if not self.preferred_agent_targets:
+ msg = "preferred_agent_targets must contain at least one target"
+ raise ValueError(msg)
+ if AgentTarget.CURSOR in self.preferred_agent_targets and len(self.preferred_agent_targets) > 1:
+ msg = "preferred_agent_targets cannot mix 'cursor' with other targets (cursor uses a separate mechanism)"
+ raise ValueError(msg)
+ return self
class PipeRunConfig(ConfigModel):
@@ -84,6 +102,34 @@ class ReportingConfig(ConfigModel):
cost_report_unit_scale: float
+class TracingBackend(StrEnum):
+ NDJSON = "ndjson"
+ DYNAMODB = "dynamodb"
+ TEMPORAL_DYNAMODB = "temporal_dynamodb"
+
+
+class NdjsonTracingConfig(ConfigModel):
+ traces_dir: str
+
+
+class DynamoDBTracingConfig(ConfigModel):
+ table_name: str
+ region: str
+
+
+class TemporalDynamoDBTracingConfig(ConfigModel):
+ table_name: str
+ region: str
+
+
+class TracingConfig(ConfigModel):
+ is_enabled: bool
+ backend: TracingBackend = Field(strict=False)
+ ndjson: NdjsonTracingConfig | None = None
+ dynamodb: DynamoDBTracingConfig | None = None
+ temporal_dynamodb: TemporalDynamoDBTracingConfig | None = None
+
+
class ObserverConfig(ConfigModel):
observer_dir: str
@@ -168,6 +214,7 @@ class Pipelex(ConfigModel):
pipe_run_config: PipeRunConfig
pipeline_execution_config: PipelineExecutionConfig
reporting_config: ReportingConfig
+ tracing_config: TracingConfig
observer_config: ObserverConfig
scan_config: ScanConfig
builder_config: BuilderConfig
@@ -193,5 +240,6 @@ class MigrationConfig(ConfigModel):
class PipelexConfig(ConfigRoot):
session_id: str = shortuuid.uuid()
cogt: Cogt
+ temporal: Temporal
pipelex: Pipelex
migration: MigrationConfig
diff --git a/pipelex/system/registries/class_registry_utils.py b/pipelex/system/registries/class_registry_utils.py
index cc62f280e..26a76f12b 100644
--- a/pipelex/system/registries/class_registry_utils.py
+++ b/pipelex/system/registries/class_registry_utils.py
@@ -3,10 +3,9 @@ import sys
import warnings
from typing import Any
-from kajson.kajson_manager import KajsonManager
-
from pipelex import log
from pipelex.config import get_config
+from pipelex.hub import get_class_registry
from pipelex.tools.misc.file_utils import find_files_in_dir
from pipelex.tools.typing.module_inspector import (
ModuleFileError,
@@ -34,7 +33,7 @@ class ClassRegistryUtils:
include_imported=is_include_imported,
)
- KajsonManager.get_class_registry().register_classes(classes=classes_to_register)
+ get_class_registry().register_classes(classes=classes_to_register)
@classmethod
def register_classes_in_folder(
@@ -145,7 +144,7 @@ class ClassRegistryUtils:
"""
registered_count = 0
- class_registry = KajsonManager.get_class_registry()
+ class_registry = get_class_registry()
# Create a snapshot of modules to avoid "dictionary changed size during iteration" error
# (inspect.getmembers can trigger imports which modify sys.modules)
diff --git a/pipelex/system/runtime.py b/pipelex/system/runtime.py
index bfec9bba1..fd75dc641 100644
--- a/pipelex/system/runtime.py
+++ b/pipelex/system/runtime.py
@@ -64,13 +64,13 @@ class WorkerMode(StrEnum):
class RunEnvironment(StrEnum):
LOCAL = "local"
- DEV = "dev"
+ DEVELOPMENT = "development"
STAGING = "staging"
- PROD = "prod"
+ PRODUCTION = "production"
@classmethod
def get_from_env_var(cls) -> "RunEnvironment":
- return RunEnvironment(get_optional_env("ENV") or RunEnvironment.DEV)
+ return RunEnvironment(get_optional_env("ENVIRONMENT") or RunEnvironment.DEVELOPMENT)
class ProblemReaction(StrEnum):
diff --git a/pipelex/system/telemetry/exception_capture.py b/pipelex/system/telemetry/exception_capture.py
index ac14521c6..4e3d7fe0f 100644
--- a/pipelex/system/telemetry/exception_capture.py
+++ b/pipelex/system/telemetry/exception_capture.py
@@ -1,11 +1,14 @@
import sys
import threading
from types import TracebackType
-
-from posthog import Posthog
+from typing import TYPE_CHECKING
from pipelex import log
+if TYPE_CHECKING:
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from posthog import Posthog
+
class DualClientExceptionCapture:
"""Captures unhandled exceptions and sends them to multiple PostHog clients.
@@ -16,9 +19,9 @@ class DualClientExceptionCapture:
def __init__(
self,
- custom_posthog_client: Posthog | None,
+ custom_posthog_client: "Posthog | None",
custom_distinct_id: str | None,
- pipelex_posthog_client: Posthog | None,
+ pipelex_posthog_client: "Posthog | None",
pipelex_distinct_id: str | None,
):
self._custom_client = custom_posthog_client
diff --git a/pipelex/system/telemetry/otel_factory.py b/pipelex/system/telemetry/otel_factory.py
index 2acf5535c..85a24fc01 100644
--- a/pipelex/system/telemetry/otel_factory.py
+++ b/pipelex/system/telemetry/otel_factory.py
@@ -5,16 +5,7 @@ This module provides helpers for instrumenting LLM operations with OpenTelemetry
import base64
import hashlib
-from typing import Any
-
-from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
-from opentelemetry.sdk.resources import Resource as OTelResource
-from opentelemetry.sdk.trace import TracerProvider as OTelTracerProvider
-from opentelemetry.sdk.trace.export import BatchSpanProcessor as OTelBatchSpanProcessor
-from opentelemetry.semconv._incubating.attributes import deployment_attributes # noqa: PLC2701
-from opentelemetry.semconv.attributes import service_attributes
-from opentelemetry.trace import Tracer as OTelTracer
-from posthog import Posthog # type: ignore[attr-defined]
+from typing import TYPE_CHECKING, Any
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.pipe_output import PipeOutput
@@ -22,13 +13,19 @@ from pipelex.system.environment import get_optional_env
from pipelex.system.runtime import RunEnvironment
from pipelex.system.telemetry.exceptions import LangfuseCredentialsError
from pipelex.system.telemetry.otel_constants import OTelConstants
-from pipelex.system.telemetry.posthog_span_exporter import PostHogSpanExporter
from pipelex.system.telemetry.telemetry_config import LangfuseConfig, OtlpExporterConfig, TelemetryRedactionConfig
from pipelex.tools.log.log import log
from pipelex.tools.misc.hash_utils import hash_md5_to_int
from pipelex.tools.misc.json_utils import JsonContent, pure_json_str
from pipelex.tools.misc.package_utils import get_package_version
+if TYPE_CHECKING:
+ # Deferred imports: avoid pulling heavy SDKs at module-load time
+ from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
+ from opentelemetry.sdk.trace import TracerProvider as OTelTracerProvider
+ from opentelemetry.trace import Tracer as OTelTracer
+ from posthog import Posthog # type: ignore[attr-defined]
+
class OtelFactory:
@classmethod
@@ -154,7 +151,7 @@ class OtelFactory:
return value is not None and value.startswith("${")
@classmethod
- def make_langfuse_exporter(cls, langfuse_config: LangfuseConfig) -> OTLPSpanExporter:
+ def make_langfuse_exporter(cls, langfuse_config: LangfuseConfig) -> "OTLPSpanExporter":
"""Create a Langfuse OTLP exporter using config credentials.
Credentials can be provided via config (with env var substitution) or
@@ -187,6 +184,9 @@ class OtelFactory:
# Build Basic auth header
langfuse_auth = base64.b64encode(f"{public_key}:{secret_key}".encode()).decode()
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter # noqa: PLC0415
+
return OTLPSpanExporter(
endpoint=f"{endpoint}/api/public/otel/v1/traces",
headers={"Authorization": f"Basic {langfuse_auth}"},
@@ -196,14 +196,14 @@ class OtelFactory:
def make_ai_tracer(
cls,
user_id: str | None,
- custom_posthog_client: Posthog | None,
+ custom_posthog_client: "Posthog | None",
custom_redaction_config: TelemetryRedactionConfig,
- pipelex_posthog_client: Posthog | None,
+ pipelex_posthog_client: "Posthog | None",
pipelex_gateway_redaction_config: TelemetryRedactionConfig,
pipelex_distinct_id: str | None,
otlp_exporters: list[OtlpExporterConfig] | None,
langfuse_config: LangfuseConfig | None,
- ) -> tuple[OTelTracer, OTelTracerProvider]:
+ ) -> "tuple[OTelTracer, OTelTracerProvider]":
"""Create an isolated OpenTelemetry Tracer for GenAI instrumentation.
This creates a dedicated TracerProvider that does NOT register itself as the
@@ -229,6 +229,16 @@ class OtelFactory:
A tuple of (Tracer, TracerProvider). The caller should call
provider.shutdown() during teardown to flush pending spans.
"""
+ # Deferred imports: avoid pulling heavy SDKs at module-load time
+ from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter # noqa: PLC0415
+ from opentelemetry.sdk.resources import Resource as OTelResource # noqa: PLC0415
+ from opentelemetry.sdk.trace import TracerProvider as OTelTracerProvider # noqa: PLC0415
+ from opentelemetry.sdk.trace.export import BatchSpanProcessor as OTelBatchSpanProcessor # noqa: PLC0415
+ from opentelemetry.semconv._incubating.attributes import deployment_attributes # noqa: PLC0415, PLC2701
+ from opentelemetry.semconv.attributes import service_attributes # noqa: PLC0415
+
+ from pipelex.system.telemetry.posthog_span_exporter import PostHogSpanExporter # noqa: PLC0415
+
# Define Resource (Identity)
resource = OTelResource.create(
attributes={
diff --git a/pipelex/system/telemetry/telemetry_factory.py b/pipelex/system/telemetry/telemetry_factory.py
index 2ba94b337..41f33c17a 100644
--- a/pipelex/system/telemetry/telemetry_factory.py
+++ b/pipelex/system/telemetry/telemetry_factory.py
@@ -10,7 +10,6 @@ from pipelex.system.pipelex_service.remote_config import RemoteConfig
from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.otel_constants import OTelConstants
from pipelex.system.telemetry.telemetry_config import PostHogMode, TelemetryConfig, load_telemetry_config
-from pipelex.system.telemetry.telemetry_manager import TelemetryManager
from pipelex.system.telemetry.telemetry_manager_abstract import (
TelemetryManagerAbstract,
TelemetryManagerNoOp,
@@ -48,6 +47,9 @@ class TelemetryFactory:
allows_custom_telemetry = telemetry_config.is_custom_telemetry_allowed_for_mode(integration_mode)
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from pipelex.system.telemetry.telemetry_manager import TelemetryManager # noqa: PLC0415
+
chosen_telemetry_manager: TelemetryManagerAbstract
if allows_custom_telemetry or is_pipelex_telemetry_enabled:
# Always respect DO_NOT_TRACK env var
diff --git a/pipelex/system/telemetry/telemetry_manager_abstract.py b/pipelex/system/telemetry/telemetry_manager_abstract.py
index 2b49faae2..cb0f97576 100644
--- a/pipelex/system/telemetry/telemetry_manager_abstract.py
+++ b/pipelex/system/telemetry/telemetry_manager_abstract.py
@@ -1,14 +1,17 @@
from abc import abstractmethod
from contextlib import contextmanager
-from typing import Any, Generator
+from typing import TYPE_CHECKING, Any, Generator
-from opentelemetry.trace import Tracer as OTelTracer
from typing_extensions import override
from pipelex.system.registries.singleton import ABCSingletonMeta, MetaSingleton
from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.events import EventName, EventProperty
+if TYPE_CHECKING:
+ # Deferred import: avoid pulling heavy SDK at module-load time
+ from opentelemetry.trace import Tracer as OTelTracer
+
class TelemetryManagerAbstract(metaclass=ABCSingletonMeta):
@classmethod
@@ -26,7 +29,7 @@ class TelemetryManagerAbstract(metaclass=ABCSingletonMeta):
return MetaSingleton.get_subclass_instance(TelemetryManagerAbstract) # type: ignore[type-abstract]
@classmethod
- def get_instance_tracer(cls) -> OTelTracer | None:
+ def get_instance_tracer(cls) -> "OTelTracer | None":
"""Get the tracer from the singleton instance.
This provides a way to access the tracer without importing from hub,
@@ -116,7 +119,7 @@ class TelemetryManagerAbstract(metaclass=ABCSingletonMeta):
pass
@abstractmethod
- def get_otel_tracer(self) -> OTelTracer | None:
+ def get_otel_tracer(self) -> "OTelTracer | None":
"""Get the OpenTelemetry tracer for GenAI spans, if configured."""
@property
@@ -197,7 +200,7 @@ class TelemetryManagerNoOp(TelemetryManagerAbstract):
return False
@override
- def get_otel_tracer(self) -> OTelTracer | None:
+ def get_otel_tracer(self) -> "OTelTracer | None":
return None
@property
diff --git a/pipelex/temporal/__init__.py b/pipelex/temporal/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/temporal/codec/__init__.py b/pipelex/temporal/codec/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/temporal/codec/codec_factory.py b/pipelex/temporal/codec/codec_factory.py
new file mode 100644
index 000000000..b9f3d94c6
--- /dev/null
+++ b/pipelex/temporal/codec/codec_factory.py
@@ -0,0 +1,23 @@
+"""Factory for building a :class:`StoragePayloadCodec` from the Pipelex config."""
+
+from __future__ import annotations
+
+from pipelex.config import get_config
+from pipelex.temporal.codec.storage_payload_codec import StoragePayloadCodec
+from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_config
+
+
+def make_codec_from_config() -> StoragePayloadCodec:
+ """Build a :class:`StoragePayloadCodec` from the current Pipelex config.
+
+ Returns:
+ A fully configured codec backed by the storage provider specified in
+ ``temporal.payload_codec_config``.
+ """
+ payload_codec_config = get_config().temporal.payload_codec_config
+ storage_provider = make_storage_provider_from_config(payload_codec_config.storage_provider_config)
+ return StoragePayloadCodec(
+ storage_provider=storage_provider,
+ size_threshold=payload_codec_config.size_threshold,
+ storage_prefix=payload_codec_config.storage_prefix,
+ )
diff --git a/pipelex/temporal/codec/codec_server.py b/pipelex/temporal/codec/codec_server.py
new file mode 100644
index 000000000..d23533725
--- /dev/null
+++ b/pipelex/temporal/codec/codec_server.py
@@ -0,0 +1,110 @@
+"""HTTP codec server for the Temporal Web UI.
+
+Exposes ``POST /encode`` and ``POST /decode`` endpoints that the Temporal Web UI
+can call to transparently decode payloads offloaded by :class:`StoragePayloadCodec`.
+Without this server, the UI shows opaque ``binary/storage-ref`` blobs instead of
+the actual workflow data.
+
+Usage::
+
+ python -m pipelex.temporal.codec.codec_server_cli
+"""
+
+from __future__ import annotations
+
+from functools import partial
+from typing import TYPE_CHECKING, Awaitable, Callable
+
+from aiohttp import hdrs, web
+from google.protobuf import json_format
+from temporalio.api.common.v1 import Payload, Payloads
+
+from pipelex import log
+from pipelex.tools.storage.exceptions import StorageFileNotFoundError
+
+if TYPE_CHECKING:
+ from collections.abc import Sequence
+
+ from pipelex.temporal.codec.storage_payload_codec import StoragePayloadCodec
+
+
+def _set_cors_headers(response: web.Response, request_origin: str, cors_origins: list[str]) -> None:
+ """Set CORS headers on the response if the request origin is allowed."""
+ if request_origin in cors_origins:
+ response.headers[hdrs.ACCESS_CONTROL_ALLOW_ORIGIN] = request_origin
+ response.headers[hdrs.ACCESS_CONTROL_ALLOW_METHODS] = "POST"
+ response.headers[hdrs.ACCESS_CONTROL_ALLOW_HEADERS] = "content-type,x-namespace"
+
+
+def _make_cors_handler(cors_origins: list[str]) -> Callable[[web.Request], Awaitable[web.Response]]:
+ """Create a CORS preflight handler for the given origins."""
+
+ async def cors_options(request: web.Request) -> web.Response: # noqa: RUF029 — must be async for aiohttp
+ response = web.Response()
+ _set_cors_headers(response, request.headers.get(hdrs.ORIGIN, ""), cors_origins)
+ return response
+
+ return cors_options
+
+
+def _error_response(request: web.Request, cors_origins: list[str], status: int, text: str) -> web.Response:
+ """Build an error response with CORS headers so browsers see the diagnostic message."""
+ response = web.Response(status=status, text=text)
+ _set_cors_headers(response, request.headers.get(hdrs.ORIGIN, ""), cors_origins)
+ return response
+
+
+async def _apply(
+ codec_fn: Callable[[Sequence[Payload]], Awaitable[list[Payload]]],
+ cors_origins: list[str],
+ request: web.Request,
+) -> web.Response:
+ """Generic handler: parse Payloads JSON, apply codec function, return Payloads JSON."""
+ if request.content_type != "application/json":
+ return _error_response(request, cors_origins, 415, "Expected application/json")
+
+ try:
+ payloads_proto = json_format.Parse(await request.read(), Payloads())
+ except json_format.ParseError:
+ return _error_response(request, cors_origins, 400, "Malformed Payloads JSON")
+
+ try:
+ result_payloads = await codec_fn(payloads_proto.payloads)
+ except StorageFileNotFoundError as exc:
+ log.error(f"Codec storage lookup failed: {exc}")
+ return _error_response(request, cors_origins, 404, "Storage object not found")
+ except (OSError, FileNotFoundError) as exc:
+ log.error(f"Codec I/O error: {exc}")
+ return _error_response(request, cors_origins, 502, "Storage I/O error")
+
+ result_proto = Payloads(payloads=result_payloads)
+
+ response = web.Response()
+ _set_cors_headers(response, request.headers.get(hdrs.ORIGIN, ""), cors_origins)
+ response.content_type = "application/json"
+ response.text = json_format.MessageToJson(result_proto)
+ return response
+
+
+def build_codec_server(codec: StoragePayloadCodec, cors_origins: list[str]) -> web.Application:
+ """Build an aiohttp application implementing the Temporal codec server protocol.
+
+ Args:
+ codec: The payload codec to use for encode/decode operations.
+ cors_origins: Allowed CORS origins (e.g. ``["http://localhost:8233"]``).
+
+ Returns:
+ A configured aiohttp web application ready to run.
+ """
+ cors_handler = _make_cors_handler(cors_origins)
+ application = web.Application()
+ application.add_routes(
+ [
+ web.post("/encode", partial(_apply, codec.encode, cors_origins)),
+ web.post("/decode", partial(_apply, codec.decode, cors_origins)),
+ web.options("/encode", cors_handler),
+ web.options("/decode", cors_handler),
+ ]
+ )
+ log.info(f"Codec server configured — CORS origins: {cors_origins}")
+ return application
diff --git a/pipelex/temporal/codec/codec_server_cli.py b/pipelex/temporal/codec/codec_server_cli.py
new file mode 100644
index 000000000..415f85553
--- /dev/null
+++ b/pipelex/temporal/codec/codec_server_cli.py
@@ -0,0 +1,55 @@
+"""CLI entry point for the Temporal codec server.
+
+Usage::
+
+ python -m pipelex.temporal.codec.codec_server_cli
+ python -m pipelex.temporal.codec.codec_server_cli --port 8082
+ python -m pipelex.temporal.codec.codec_server_cli --cors-origin http://localhost:8233 --cors-origin https://cloud.temporal.io
+"""
+
+from __future__ import annotations
+
+from typing import Annotated
+
+import typer
+from aiohttp import web
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.pipelex import Pipelex
+from pipelex.temporal.codec.codec_factory import make_codec_from_config
+from pipelex.temporal.codec.codec_server import build_codec_server
+from pipelex.temporal.exceptions import TemporalConfigError
+
+app = typer.Typer()
+
+DEFAULT_CORS_ORIGINS = ["http://localhost:8233"]
+
+
+def start(
+ host: Annotated[str, typer.Option(help="Host to bind the codec server to")] = "127.0.0.1",
+ port: Annotated[int, typer.Option(help="Port to bind the codec server to")] = 8081,
+ cors_origin: Annotated[list[str] | None, typer.Option(help="Allowed CORS origin (repeatable)")] = None,
+) -> None:
+ """Start the Temporal codec server for Web UI payload decoding."""
+ Pipelex.make(temporal_enabled=True, needs_inference=False)
+
+ payload_codec_config = get_config().temporal.payload_codec_config
+ if not payload_codec_config.is_enabled:
+ msg = (
+ "Payload codec is not enabled in the config (temporal.payload_codec_config.is_enabled = false). "
+ "Enable it in your pipelex.toml before starting the codec server."
+ )
+ raise TemporalConfigError(msg)
+
+ codec = make_codec_from_config()
+ cors_origins = cors_origin or DEFAULT_CORS_ORIGINS
+ application = build_codec_server(codec=codec, cors_origins=cors_origins)
+
+ log.info(f"Starting Temporal codec server on {host}:{port}")
+ web.run_app(application, host=host, port=port)
+
+
+if __name__ == "__main__":
+ app()
diff --git a/pipelex/temporal/codec/storage_payload_codec.py b/pipelex/temporal/codec/storage_payload_codec.py
new file mode 100644
index 000000000..9ec643c5b
--- /dev/null
+++ b/pipelex/temporal/codec/storage_payload_codec.py
@@ -0,0 +1,117 @@
+from __future__ import annotations
+
+import json
+import re
+from hashlib import sha256
+from typing import TYPE_CHECKING, Any, cast
+
+from temporalio.api.common.v1 import Payload
+from temporalio.converter import PayloadCodec
+from typing_extensions import override
+
+from pipelex import log
+
+if TYPE_CHECKING:
+ from collections.abc import Sequence
+
+ from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
+
+STORAGE_REF_ENCODING = b"binary/storage-ref"
+JSON_PLAIN_ENCODING = b"json/plain"
+
+
+class StoragePayloadCodec(PayloadCodec):
+ """Codec that offloads large Temporal payloads to external storage.
+
+ Payloads whose serialized size meets or exceeds ``size_threshold`` are
+ stored via the provided :class:`StorageProviderAbstract` and replaced with
+ a lightweight reference payload. Payloads below the threshold pass through
+ unchanged. Content-addressed keys (SHA-256) give natural deduplication.
+
+ Storage keys are structured as ``{user_id}/{pipeline_run_id}/{hash}`` when
+ the payload contains ``job_metadata``, enabling per-client cleanup.
+ """
+
+ def __init__(
+ self,
+ storage_provider: StorageProviderAbstract,
+ size_threshold: int,
+ storage_prefix: str,
+ ) -> None:
+ self._storage = storage_provider
+ self._size_threshold = size_threshold
+ self._storage_prefix = f"{storage_prefix}/" if storage_prefix and not storage_prefix.endswith("/") else storage_prefix
+
+ @staticmethod
+ def _extract_job_routing(payload: Payload) -> tuple[str, str] | None:
+ """Extract user_id and pipeline_run_id from a JSON payload's job_metadata.
+
+ Returns:
+ A (user_id, pipeline_run_id) tuple, or None if the payload is not
+ JSON or does not contain job_metadata at the top level.
+ """
+ if payload.metadata.get("encoding") != JSON_PLAIN_ENCODING:
+ return None
+ try:
+ data = json.loads(payload.data)
+ except (json.JSONDecodeError, UnicodeDecodeError):
+ return None
+ if not isinstance(data, dict):
+ return None
+ typed_data = cast("dict[str, Any]", data)
+ job_metadata = typed_data.get("job_metadata")
+ if not isinstance(job_metadata, dict):
+ return None
+ user_id: str | None = job_metadata.get("user_id") # type: ignore[assignment]
+ pipeline_run_id: str | None = job_metadata.get("pipeline_run_id") # type: ignore[assignment]
+ if isinstance(user_id, str) and isinstance(pipeline_run_id, str):
+ return user_id, pipeline_run_id
+ return None
+
+ @staticmethod
+ def _sanitize_path_segment(segment: str) -> str:
+ """Sanitize a path segment to prevent traversal and encoding issues."""
+ return re.sub(r"[^A-Za-z0-9_\-]", "_", segment)
+
+ def _build_storage_key(self, payload: Payload, hash_hex: str) -> str:
+ """Build a storage key, structured by job routing when available."""
+ routing = self._extract_job_routing(payload)
+ if routing:
+ user_id, pipeline_run_id = routing
+ safe_user_id = self._sanitize_path_segment(user_id)
+ safe_run_id = self._sanitize_path_segment(pipeline_run_id)
+ return f"{self._storage_prefix}{safe_user_id}/{safe_run_id}/{hash_hex}"
+ return f"{self._storage_prefix}{hash_hex}"
+
+ @override
+ async def encode(self, payloads: Sequence[Payload]) -> list[Payload]:
+ encoded: list[Payload] = []
+ for payload in payloads:
+ serialized = payload.SerializeToString()
+ if len(serialized) < self._size_threshold:
+ encoded.append(payload)
+ continue
+ hash_hex = sha256(serialized).hexdigest()
+ key = self._build_storage_key(payload, hash_hex)
+ uri = await self._storage.store(data=serialized, key=key)
+ log.dev(f"Payload offloaded to storage: key='{key}'")
+ ref_payload = Payload(
+ metadata={"encoding": STORAGE_REF_ENCODING},
+ data=uri.encode(),
+ )
+ encoded.append(ref_payload)
+ return encoded
+
+ @override
+ async def decode(self, payloads: Sequence[Payload]) -> list[Payload]:
+ decoded: list[Payload] = []
+ for payload in payloads:
+ if payload.metadata.get("encoding") != STORAGE_REF_ENCODING:
+ decoded.append(payload)
+ continue
+ uri = payload.data.decode()
+ original_bytes = await self._storage.load(uri)
+ original_payload = Payload()
+ original_payload.ParseFromString(original_bytes)
+ decoded.append(original_payload)
+ return decoded
diff --git a/pipelex/temporal/config_temporal.py b/pipelex/temporal/config_temporal.py
new file mode 100644
index 000000000..4444decc4
--- /dev/null
+++ b/pipelex/temporal/config_temporal.py
@@ -0,0 +1,211 @@
+from datetime import timedelta
+from typing import TYPE_CHECKING, Literal, Union
+
+from pydantic import Field, model_validator
+
+from pipelex.system.configuration.config_model import ConfigModel
+from pipelex.temporal.exceptions import TemporalConfigError
+from pipelex.tools.storage.storage_config import StorageProviderConfig
+from pipelex.types import Self, StrEnum
+
+if TYPE_CHECKING:
+ from temporalio.common import RetryPolicy
+
+
+class SecretMethod(StrEnum):
+ NONE = "none"
+ ENV_VAR = "env_var"
+ SECRET_PROVIDER = "secret_provider"
+
+
+class TemporalServerConfig(ConfigModel):
+ """Configuration model for Temporal server settings."""
+
+ description: str
+ target_host: str
+ namespace: str
+ api_key_method: SecretMethod = Field(strict=False)
+ api_key_id: str
+
+ @property
+ def full_description(self) -> str:
+ """Generate a full description of the Temporal server configuration.
+
+ Returns:
+ str: A detailed description including non-default target host and namespace.
+ """
+ desc = self.description
+ if self.target_host != "localhost:7233":
+ desc += f", target host: {self.target_host}"
+ if self.namespace != "default":
+ desc += f", namespace: {self.namespace}"
+ return desc
+
+
+class TemporalLogConfig(ConfigModel):
+ """Configuration model for Temporal logging settings."""
+
+ is_workflow_info_on_message: bool
+ is_workflow_info_on_extra: bool
+ is_full_workflow_info_on_extra: bool
+ is_activity_info_on_message: bool
+ is_activity_info_on_extra: bool
+ is_full_activity_info_on_extra: bool
+ is_formatter_enabled: bool
+ is_prefix_enabled: bool
+ managed_loggers: list[str]
+
+
+class WorkerScope(ConfigModel):
+ """Defines the subset of workflows and activities a worker registers.
+
+ Resolution order: pack contents → required_* additions → excluded_* subtractions →
+ disable_all_* clearing. Names match TaskPack keys, workflow class __name__, and
+ activity function __name__.
+
+ `disable_all_workflows` / `disable_all_activities` are useful to split workflow-only
+ and activity-only workers on the same task queue (Temporal SDK rejects multiple
+ workers with overlapping task types on the same queue).
+ """
+
+ required_tasks_packs: list[str]
+ required_workflows: list[str]
+ required_activities: list[str]
+ excluded_workflows: list[str]
+ excluded_activities: list[str]
+ disable_all_workflows: bool
+ disable_all_activities: bool
+
+
+class WorkerScopesConfig(ConfigModel):
+ """Named worker scopes selectable via --scope on the worker CLI."""
+
+ default_scope: str
+ scopes: dict[str, WorkerScope]
+
+ @model_validator(mode="after")
+ def validate_default_scope(self) -> Self:
+ if self.default_scope not in self.scopes:
+ msg = f"default_scope '{self.default_scope}' not found in scopes (known: {sorted(self.scopes.keys())})"
+ raise TemporalConfigError(msg)
+ return self
+
+
+class TemporalConfig(ConfigModel):
+ """Configuration model for overall Temporal settings."""
+
+ temporal_server_configs: dict[str, TemporalServerConfig]
+ selected_server: str
+ temporal_log_config: TemporalLogConfig
+
+ @model_validator(mode="after")
+ def validate_selected_server(self) -> Self:
+ """Validate that the selected server exists in the server configurations.
+
+ Raises:
+ TemporalConfigError: If the selected server is not found in the configurations.
+
+ Returns:
+ Self: The validated instance.
+ """
+ if self.selected_server not in self.temporal_server_configs:
+ msg = f"Selected server '{self.selected_server}' not found in temporal_server_configs"
+ raise TemporalConfigError(msg)
+ return self
+
+
+class RetryPolicyConfig(ConfigModel):
+ """Configuration model for retry policy settings."""
+
+ initial_interval: timedelta = Field(strict=False)
+ backoff_coefficient: float
+ maximum_interval: Union[timedelta, Literal["unlimited"]]
+ maximum_attempts: Union[int, Literal["unlimited"]]
+ non_retryable_error_types: list[str]
+
+ def make_retry_policy(self) -> "RetryPolicy":
+ """Create a RetryPolicy instance based on the configuration.
+
+ Returns:
+ RetryPolicy: A configured RetryPolicy object.
+ """
+ from temporalio.common import RetryPolicy as _RetryPolicy # noqa: PLC0415
+
+ maximum_attempts: int
+ if self.maximum_attempts == "unlimited":
+ # This is according to the Temporal SDK's documentation
+ maximum_attempts = 0
+ else:
+ maximum_attempts = self.maximum_attempts
+
+ maximum_interval: timedelta | None
+ # this test is in two steps because timedelta's are actually read from the config as strings
+ if isinstance(self.maximum_interval, str) and self.maximum_interval == "unlimited":
+ maximum_interval = None
+ else:
+ maximum_interval = self.maximum_interval
+
+ return _RetryPolicy(
+ initial_interval=self.initial_interval,
+ backoff_coefficient=self.backoff_coefficient,
+ maximum_interval=maximum_interval,
+ maximum_attempts=maximum_attempts,
+ non_retryable_error_types=self.non_retryable_error_types,
+ )
+
+
+# class PipelexWorkflowsConfig(ConfigModel):
+# """Configuration model for workflow settings."""
+
+# jinja2_activity_timeout: timedelta = Field(strict=False)
+# jinja2_retry_policy_config: RetryPolicyConfig
+
+# @property
+# def jinja2_retry_policy(self) -> RetryPolicy:
+# """
+# Create a RetryPolicy for LLM generation based on the configuration.
+
+# Returns:
+# RetryPolicy: A configured RetryPolicy object for LLM generation.
+# """
+# return self.jinja2_retry_policy_config.make_retry_policy()
+
+
+class WorkerConfig(ConfigModel):
+ """Configuration model for workflow execution settings."""
+
+ task_queue: str
+ workflow_execution_timeout: timedelta = Field(strict=False)
+ run_timeout: timedelta | None = Field(default=None, strict=False)
+ task_timeout: timedelta | None = Field(default=None, strict=False)
+ start_delay: timedelta | None = Field(default=None, strict=False)
+ rpc_timeout: timedelta | None = Field(default=None, strict=False)
+ retry_policy_config: RetryPolicyConfig
+
+ @property
+ def retry_policy(self) -> "RetryPolicy":
+ """Create a RetryPolicy based on the configuration.
+
+ Returns:
+ RetryPolicy: A configured RetryPolicy object.
+ """
+ return self.retry_policy_config.make_retry_policy()
+
+
+class PayloadCodecConfig(ConfigModel):
+ """Configuration for the storage-based payload codec that offloads large payloads."""
+
+ is_enabled: bool
+ size_threshold: int
+ storage_prefix: str
+ storage_provider_config: StorageProviderConfig
+
+
+class Temporal(ConfigModel):
+ """Main configuration model for Temporal."""
+
+ is_enabled: bool
+ temporal_config: TemporalConfig
+ worker_config: WorkerConfig
+ worker_scopes: WorkerScopesConfig
+ payload_codec_config: PayloadCodecConfig
diff --git a/pipelex/temporal/exceptions.py b/pipelex/temporal/exceptions.py
new file mode 100644
index 000000000..70bb726cd
--- /dev/null
+++ b/pipelex/temporal/exceptions.py
@@ -0,0 +1,29 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class TemporalFlowError(PipelexError):
+ pass
+
+
+class WorkflowInputError(TemporalFlowError):
+ pass
+
+
+class WorkflowExecutionError(TemporalFlowError):
+ pass
+
+
+class ContentGenerationError(TemporalFlowError):
+ pass
+
+
+class TemporalConfigError(ValueError, TemporalFlowError):
+ pass
+
+
+class TemporalServerError(TemporalFlowError):
+ pass
+
+
+class WorkerScopeConfigError(TemporalConfigError):
+ pass
diff --git a/pipelex/temporal/log_formatter.py b/pipelex/temporal/log_formatter.py
new file mode 100644
index 000000000..1cfcc26c1
--- /dev/null
+++ b/pipelex/temporal/log_formatter.py
@@ -0,0 +1,52 @@
+import logging
+from typing import Any
+
+from typing_extensions import override
+
+from pipelex.config import get_config
+from pipelex.tools.log.log_formatter import LevelAndEmojiLogFormatter, log_level_color, log_level_tag
+from pipelex.tools.misc.terminal_utils import RESET_FONT, TerminalColor
+
+
+class TemporalLogFormatter(LevelAndEmojiLogFormatter):
+ @override
+ def format(self, record: logging.LogRecord):
+ temporal_log_config = get_config().temporal.temporal_config.temporal_log_config
+ if record.name not in temporal_log_config.managed_loggers:
+ return super().format(record)
+
+ # these are prints because we are configuring logs
+ # they are commmented but remain as a practical solution to check what temporal injects in logs in various contexts
+ # print(f"\n\n{record.name=}")
+ # print(f"\n\n{json.dumps(vars(record), default=str)}\n\n")
+
+ prefix = ""
+
+ if temporal_log_config.is_prefix_enabled:
+ wf_color = TerminalColor.MAGENTA
+ act_color = TerminalColor.CYAN
+ if temporal_workflow := getattr(record, "temporal_workflow", None):
+ workflow_dict: dict[str, Any] = temporal_workflow
+ if workflow_type := workflow_dict.get("workflow_type"):
+ prefix += f"{wf_color}[{workflow_type}]{RESET_FONT}"
+
+ if temporal_activity := getattr(record, "temporal_activity", None):
+ activity_dict: dict[str, Any] = temporal_activity
+ if workflow_type := activity_dict.get("workflow_type"):
+ prefix += f"{wf_color}[{workflow_type}]{RESET_FONT}"
+ if activity_type := activity_dict.get("activity_type"):
+ prefix += f"{act_color}[{activity_type}]{RESET_FONT}"
+
+ if prefix:
+ prefix += " "
+
+ color = log_level_color.get(record.levelno, RESET_FONT)
+ tag = log_level_tag.get(record.levelno, "?????")
+
+ if record.levelno in {logging.WARNING, logging.ERROR, logging.CRITICAL}:
+ record.msg = f"{color}{record.msg}{RESET_FONT}"
+
+ log_fmt = f"{color}{tag}:{prefix}{RESET_FONT} %(message)s"
+
+ formatter = logging.Formatter(log_fmt)
+ return formatter.format(record)
diff --git a/pipelex/temporal/log_temporal.py b/pipelex/temporal/log_temporal.py
new file mode 100644
index 000000000..fd620b5e5
--- /dev/null
+++ b/pipelex/temporal/log_temporal.py
@@ -0,0 +1,109 @@
+import logging
+from typing import Any, Union
+
+from temporalio import activity, workflow
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.temporal.log_formatter import TemporalLogFormatter
+from pipelex.tools.log.log_levels import LOGGING_LEVEL_DEV, LOGGING_LEVEL_VERBOSE
+
+
+def configure_temporal_logs():
+ """Configure Temporal logging settings based on the application configuration.
+
+ This function sets up the log formatter, safety callable, and various logging options
+ for both workflow and activity loggers.
+ """
+ temporal_log_config = get_config().temporal.temporal_config.temporal_log_config
+ log.set_poor_log_formatter(TemporalLogFormatter())
+
+ workflow.logger.workflow_info_on_message = temporal_log_config.is_workflow_info_on_message
+ workflow.logger.workflow_info_on_extra = temporal_log_config.is_workflow_info_on_extra
+ workflow.logger.full_workflow_info_on_extra = temporal_log_config.is_full_workflow_info_on_extra
+
+ activity.logger.activity_info_on_message = temporal_log_config.is_activity_info_on_message
+ activity.logger.activity_info_on_extra = temporal_log_config.is_activity_info_on_extra
+ activity.logger.full_activity_info_on_extra = temporal_log_config.is_full_activity_info_on_extra
+
+
+class WorkflowLog:
+ """A class for logging messages in Temporal workflows with different severity levels."""
+
+ def verbose(self, content: Union[str, Any]):
+ """Log a verbose-level message in a workflow."""
+ severity = LOGGING_LEVEL_VERBOSE
+ workflow.logger.log(level=severity, msg=content)
+
+ def debug(self, content: Union[str, Any]):
+ """Log a debug-level message in a workflow."""
+ severity = logging.DEBUG
+ workflow.logger.log(level=severity, msg=content)
+
+ def dev(self, content: Union[str, Any]):
+ """Log a development-level message in a workflow."""
+ severity = LOGGING_LEVEL_DEV
+ workflow.logger.log(level=severity, msg=content)
+
+ def info(self, content: Union[str, Any]):
+ """Log an info-level message in a workflow."""
+ severity = logging.INFO
+ workflow.logger.log(level=severity, msg=content)
+
+ def warning(self, content: Union[str, Any]):
+ """Log a warning-level message in a workflow."""
+ severity = logging.WARNING
+ workflow.logger.log(level=severity, msg=content)
+
+ def error(self, content: Union[str, Any]):
+ """Log an error-level message in a workflow."""
+ severity = logging.ERROR
+ workflow.logger.log(level=severity, msg=content)
+
+ def critical(self, content: Union[str, Any]):
+ """Log a critical-level message in a workflow."""
+ severity = logging.CRITICAL
+ workflow.logger.log(level=severity, msg=content)
+
+
+class ActivityLog:
+ """A class for logging messages in Temporal activities with different severity levels."""
+
+ def verbose(self, content: Union[str, Any]):
+ """Log a verbose-level message in an activity."""
+ severity = LOGGING_LEVEL_VERBOSE
+ activity.logger.log(level=severity, msg=content)
+
+ def debug(self, content: Union[str, Any]):
+ """Log a debug-level message in an activity."""
+ severity = logging.DEBUG
+ activity.logger.log(level=severity, msg=content)
+
+ def dev(self, content: Union[str, Any]):
+ """Log a development-level message in an activity."""
+ severity = LOGGING_LEVEL_DEV
+ activity.logger.log(level=severity, msg=content)
+
+ def info(self, content: Union[str, Any]):
+ """Log an info-level message in an activity."""
+ severity = logging.INFO
+ activity.logger.log(level=severity, msg=content)
+
+ def warning(self, content: Union[str, Any]):
+ """Log a warning-level message in an activity."""
+ severity = logging.WARNING
+ activity.logger.log(level=severity, msg=content)
+
+ def error(self, content: Union[str, Any]):
+ """Log an error-level message in an activity."""
+ severity = logging.ERROR
+ activity.logger.log(level=severity, msg=content)
+
+ def critical(self, content: Union[str, Any]):
+ """Log a critical-level message in an activity."""
+ severity = logging.CRITICAL
+ activity.logger.log(level=severity, msg=content)
+
+
+workflow_log = WorkflowLog()
+activity_log = ActivityLog()
diff --git a/pipelex/temporal/sandbox_manager.py b/pipelex/temporal/sandbox_manager.py
new file mode 100644
index 000000000..38059d4ae
--- /dev/null
+++ b/pipelex/temporal/sandbox_manager.py
@@ -0,0 +1,29 @@
+from typing import Callable
+
+from pydantic import BaseModel
+
+
+class SandboxManager(BaseModel):
+ is_in_sandbox_callable: Callable[[], bool] | None = None
+
+ def set_sandbox_callable(self, sandbox_callable: Callable[[], bool]):
+ """Sets a callable function to determine if we're in a sandbox, which would prevent importing unsafe modules,
+ and in particular prevent logging with default methods.
+
+ Args:
+ sandbox_callable: A function that returns True if we're in a sandbox, False otherwise.
+ """
+ self.is_in_sandbox_callable = sandbox_callable
+
+ def is_in_sandbox(self) -> bool:
+ """Returns:
+ bool: True if we're in a sandbox, False otherwise.
+ """
+ if self.is_in_sandbox_callable is None:
+ # by default, we are not in a sandbox
+ return False
+ else:
+ return self.is_in_sandbox_callable()
+
+
+sandbox_manager = SandboxManager()
diff --git a/pipelex/temporal/task_manager.py b/pipelex/temporal/task_manager.py
new file mode 100644
index 000000000..43f26bd94
--- /dev/null
+++ b/pipelex/temporal/task_manager.py
@@ -0,0 +1,51 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Protocol
+
+if TYPE_CHECKING:
+ from temporalio.client import Client as TemporalClient
+ from temporalio.worker import Worker
+
+ from pipelex.temporal.config_temporal import WorkerScope
+ from pipelex.temporal.temporal_tasks import TaskPack
+ from pipelex.temporal.temporal_types import ActivityList, ActivityType, WorkflowList, WorkflowType
+
+
+class TaskManager(Protocol):
+ def complement_catalog(
+ self,
+ extra_catalog: dict[str, TaskPack],
+ extra_workflows: list[WorkflowType],
+ extra_activities: list[ActivityType],
+ ): ...
+
+ def make_worker(
+ self,
+ temporal_client: TemporalClient,
+ task_queue: str,
+ is_not_sandboxed: bool = False,
+ scope: WorkerScope | None = None,
+ substitute_activities: dict[ActivityType, ActivityType] | None = None,
+ test_workflows: WorkflowList | None = None,
+ test_activities: ActivityList | None = None,
+ ) -> Worker: ...
+
+ async def run_worker(
+ self,
+ is_not_sandboxed: bool,
+ is_unit_testing: bool,
+ task_queue: str | None = None,
+ scope_name: str | None = None,
+ ): ...
+
+ def task_packs(self) -> list[str]: ...
+
+ def workflows_and_activities(
+ self,
+ scope: WorkerScope | None = None,
+ test_workflows: WorkflowList | None = None,
+ test_activities: ActivityList | None = None,
+ substitute_activities: dict[ActivityType, ActivityType] | None = None,
+ ) -> tuple[list[WorkflowType], list[ActivityType]]: ...
+
+ def workflows_and_activities_str(self) -> tuple[list[str], list[str]]: ...
diff --git a/pipelex/temporal/tasks.py b/pipelex/temporal/tasks.py
new file mode 100644
index 000000000..e05c77129
--- /dev/null
+++ b/pipelex/temporal/tasks.py
@@ -0,0 +1,63 @@
+from typing import ClassVar
+
+from pipelex.temporal.temporal_tasks import TaskPack
+from pipelex.temporal.tprl_content_generation.act_extract_generate import act_extract_gen_extract_pages
+from pipelex.temporal.tprl_content_generation.act_img_gen_generate import act_img_gen_images
+from pipelex.temporal.tprl_content_generation.act_jinja2_generate import act_jinja2_gen_text
+from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_object, act_llm_gen_object_list, act_llm_gen_text
+from pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views
+from pipelex.temporal.tprl_content_generation.wf_make_extract import WfMakeExtract
+from pipelex.temporal.tprl_content_generation.wf_make_images import WfMakeImages
+from pipelex.temporal.tprl_content_generation.wf_make_jinja2_text import WfMakeJinja2Text
+from pipelex.temporal.tprl_content_generation.wf_make_llm_text import WfMakeLLMText
+from pipelex.temporal.tprl_content_generation.wf_make_object import WfMakeObject, WfMakeObjectList, WfMakeTextThenObject, WfMakeTextThenObjectList
+from pipelex.temporal.tprl_content_generation.wf_render_page_views import WfRenderPageViews
+from pipelex.temporal.tprl_pipe.act_assemble_graph import act_assemble_graph
+from pipelex.temporal.tprl_pipe.act_deliver import act_deliver
+from pipelex.temporal.tprl_pipe.act_flush_trace_events import act_flush_trace_events
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun
+from pipelex.types import StrEnum
+
+
+class PackName(StrEnum):
+ CRAFTING = "crafting"
+ PIPE = "pipe"
+
+
+class Tasks:
+ TASK_PACKS: ClassVar[dict[str, TaskPack]] = {
+ PackName.CRAFTING: TaskPack(
+ workflow_list=[
+ WfMakeObject,
+ WfMakeLLMText,
+ WfMakeTextThenObject,
+ WfMakeObjectList,
+ WfMakeTextThenObjectList,
+ WfMakeImages,
+ WfMakeExtract,
+ WfMakeJinja2Text,
+ WfRenderPageViews,
+ ],
+ activity_list=[
+ act_llm_gen_object,
+ act_llm_gen_text,
+ act_llm_gen_object_list,
+ act_img_gen_images,
+ act_jinja2_gen_text,
+ act_extract_gen_extract_pages,
+ act_render_page_views,
+ ],
+ ),
+ PackName.PIPE: TaskPack(
+ workflow_list=[
+ WfPipeRouter,
+ WfPipeRun,
+ ],
+ activity_list=[
+ act_assemble_graph,
+ act_deliver,
+ act_flush_trace_events,
+ ],
+ ),
+ }
diff --git a/pipelex/temporal/temporal_connect.py b/pipelex/temporal/temporal_connect.py
new file mode 100644
index 000000000..886238f10
--- /dev/null
+++ b/pipelex/temporal/temporal_connect.py
@@ -0,0 +1,103 @@
+from typing import TYPE_CHECKING, Mapping
+
+from temporalio.client import Client as TemporalClient
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.hub import get_secret
+from pipelex.system.environment import get_required_env
+from pipelex.temporal.codec.codec_factory import make_codec_from_config
+from pipelex.temporal.config_temporal import SecretMethod, TemporalConfigError, TemporalServerConfig
+from pipelex.temporal.exceptions import TemporalServerError
+from pipelex.temporal.temporal_data_converter import data_converter, make_data_converter
+
+if TYPE_CHECKING:
+ from temporalio.converter import DataConverter
+
+
+async def connect_to_temporal_server(server_config: TemporalServerConfig, name: str | None = None) -> TemporalClient:
+ """Connect to Temporal using the provided server config."""
+ api_key: str | None
+ match server_config.api_key_method:
+ case SecretMethod.NONE:
+ api_key = None
+ case SecretMethod.ENV_VAR:
+ if api_key_id := server_config.api_key_id:
+ api_key = get_required_env(api_key_id)
+ else:
+ msg = "api_key_id is required for api_key_method=ENV_VAR"
+ raise TemporalConfigError(msg)
+ case SecretMethod.SECRET_PROVIDER:
+ if api_key_id := server_config.api_key_id:
+ api_key = get_secret(secret_id=api_key_id)
+ else:
+ msg = "api_key_id is required for api_key_method=SECRET_PROVIDER"
+ raise TemporalConfigError(msg)
+
+ tls: bool
+ rpc_metadata: Mapping[str, str]
+ if api_key:
+ rpc_metadata = {"temporal-namespace": server_config.namespace}
+ tls = True
+ else:
+ rpc_metadata = {}
+ tls = False
+
+ # Build the data converter, optionally with a storage-based payload codec.
+ payload_codec_config = get_config().temporal.payload_codec_config
+ converter: DataConverter
+ if payload_codec_config.is_enabled:
+ payload_codec = make_codec_from_config()
+ converter = make_data_converter(payload_codec=payload_codec)
+ log.info(f"Payload codec enabled — threshold={payload_codec_config.size_threshold}, prefix='{payload_codec_config.storage_prefix}'")
+ else:
+ converter = data_converter
+
+ log.info(f"Connecting to Temporal server: {server_config.full_description}")
+ log.info(
+ f"""Establishing connection to Temporal...
+ - Selected Server Config : {name}
+ - Target Host : {server_config.target_host}
+ - Namespace : {server_config.namespace}
+ - TLS Enabled : {"Yes" if tls else "No"}
+ - API Key : {"Provided" if api_key else "Not Provided"}
+ - RPC Metadata : {rpc_metadata or "None"}
+ """,
+ )
+
+ try:
+ temporal_client: TemporalClient = await TemporalClient.connect(
+ target_host=server_config.target_host,
+ namespace=server_config.namespace,
+ rpc_metadata=rpc_metadata,
+ api_key=api_key,
+ data_converter=converter,
+ tls=tls,
+ )
+ except RuntimeError as exc:
+ msg = f"Failed to connect to Temporal server using config: {server_config.full_description}\nError: {exc}"
+ raise TemporalServerError(msg) from exc
+ return temporal_client
+
+
+async def connect_to_temporal_selected_server(
+ selected_server_config: str,
+) -> TemporalClient:
+ """Connect to Temporal using the server selected by argument."""
+ temporal_config = get_config().temporal.temporal_config
+ log.dev(f"Using Temporal server config named: '{selected_server_config}'")
+ server_config = temporal_config.temporal_server_configs.get(selected_server_config)
+ if not server_config:
+ msg = f"Server config not found for selected server: '{selected_server_config}'"
+ raise TemporalConfigError(msg)
+ temporal_client = await connect_to_temporal_server(server_config=server_config, name=selected_server_config)
+ log.info(f"Connected to Temporal server: '{selected_server_config}' = {server_config.full_description}")
+ return temporal_client
+
+
+async def connect_to_temporal() -> TemporalClient:
+ """Connect to Temporal using the server selected from the config."""
+ temporal_config = get_config().temporal.temporal_config
+ return await connect_to_temporal_selected_server(
+ selected_server_config=temporal_config.selected_server,
+ )
diff --git a/pipelex/temporal/temporal_data_converter.py b/pipelex/temporal/temporal_data_converter.py
new file mode 100644
index 000000000..d6d58dc70
--- /dev/null
+++ b/pipelex/temporal/temporal_data_converter.py
@@ -0,0 +1,175 @@
+"""This module was entirely rewritten based on Temporal's sample "pydantic_converter":
+https://github.com/temporalio/samples-python/tree/main/pydantic_converter
+
+The purpose of this converter is to serialize and deserialize inputs and outputs of Temporal activities and workflows.
+
+Our implementation includes several key improvements:
+- Compatibility with Pydantic v2
+- Support for universal JSON serialization/deserialization via `pipelex.tools.serde.kajson`
+- Preservation of subclass types
+
+Please note that, while Temporal does not officially support Generics,
+this converter serves as a reliable workaround for Pydantic models.
+It also supports lists such as `List[BaseModel]` and `List[BaseModelSubclass]`.
+
+For examples, see the tests in `test_top_crafter.py`:
+
+1. `make ti TEST=test_make_object_direct`
+ - This test uses `WfCraftObject`, which returns a `BaseModel`.
+ - However, the actual class of the returned object is determined by `object_assignment.object_class`,
+ allowing the calling method to cast it to the required generic type with `cast(BaseModelType, obj)`.
+
+2. `make ti TEST=test_make_object_list_direct`
+ - This test uses `WfCraftObjectList`, which returns a `List[BaseModel]`.
+ - Similarly, the items in the returned list use the class specified in `object_assignment.object_class`,
+ allowing the calling method to cast the list to the required generic type with `cast(List[BaseModelType], obj_list)`.
+"""
+
+from typing import Any, cast
+
+from kajson import kajson
+from pydantic import BaseModel
+from temporalio.api.common.v1 import Payload
+from temporalio.converter import (
+ CompositePayloadConverter,
+ DataConverter,
+ DefaultPayloadConverter,
+ EncodingPayloadConverter,
+ JSONPlainPayloadConverter,
+ PayloadCodec,
+)
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.hub import get_class_registry
+from pipelex.temporal.exceptions import TemporalFlowError
+
+
+class BaseModelPayloadConverterError(TemporalFlowError):
+ pass
+
+
+class BaseModelPayloadConverter(JSONPlainPayloadConverter):
+ """Universal Pydantic JSON payload converter.
+
+ This extends the :py:class:`JSONPlainPayloadConverter` to override :py:meth:`to_payload`.
+ """
+
+ @override
+ def to_payload(self, value: Any) -> Payload | None:
+ if isinstance(value, BaseModel):
+ payload_str: str = kajson.dumps(value)
+ metadata: dict[str, bytes] = {"encoding": self.encoding.encode()}
+ class_source = getattr(type(value), "__kajson_class_source__", None)
+ if class_source is not None:
+ metadata["kajson_class_source"] = class_source.encode()
+ return Payload(
+ metadata=metadata,
+ data=payload_str.encode(),
+ )
+ elif isinstance(value, list) and value and isinstance(value[0], BaseModel):
+ list_payload_str: str = kajson.dumps(value)
+ metadata = {"encoding": self.encoding.encode()}
+ class_source = getattr(type(value[0]), "__kajson_class_source__", None)
+ if class_source is not None:
+ metadata["kajson_class_source"] = class_source.encode()
+ return Payload(
+ metadata=metadata,
+ data=list_payload_str.encode(),
+ )
+ else:
+ return super().to_payload(value)
+
+ def _restore_class_source(self, value: BaseModel, class_source_code: str) -> None:
+ value_class = cast("Any", type(value))
+ value_class.__kajson_class_source__ = class_source_code
+
+ def _kajson_deserialize_from_payload(self, payload: Payload) -> Any:
+ data = payload.data.decode()
+ log.verbose(f"unijson_deserialize_payload — data: {data}")
+ source_bytes = payload.metadata.get("kajson_class_source")
+ class_source_code = source_bytes.decode() if source_bytes else None
+ pydantic_gizmo = kajson.loads(
+ data,
+ class_registry=get_class_registry(),
+ class_source_code=class_source_code,
+ )
+ if class_source_code is not None:
+ if isinstance(pydantic_gizmo, BaseModel):
+ self._restore_class_source(pydantic_gizmo, class_source_code)
+ elif isinstance(pydantic_gizmo, list):
+ for item in cast("list[Any]", pydantic_gizmo):
+ if isinstance(item, BaseModel):
+ self._restore_class_source(item, class_source_code)
+ log.verbose(f"unijson_deserialize_payload — pydantic_gizmo: {pydantic_gizmo}")
+ return cast("Any", pydantic_gizmo)
+
+ @staticmethod
+ def _unwrap_optional_base_model(type_hint: type[Any] | None) -> bool:
+ """Check if type_hint is Optional[BaseModel] / BaseModel | None."""
+ if type_hint is None:
+ return False
+ args = getattr(type_hint, "__args__", None)
+ if args is None:
+ return False
+ # Optional[X] is Union[X, None] — check if any non-None arg is a BaseModel subclass
+ for arg in args:
+ if arg is type(None):
+ continue
+ if isinstance(arg, type) and issubclass(arg, BaseModel):
+ return True
+ return False
+
+ @override
+ def from_payload(
+ self,
+ payload: Payload,
+ type_hint: type[Any] | None = None,
+ ) -> Any:
+ # BaseModel case
+ if isinstance(type_hint, type) and issubclass(type_hint, BaseModel): # pyright: ignore[reportUnnecessaryIsInstance]
+ return self._kajson_deserialize_from_payload(payload=payload)
+
+ # Optional[BaseModel] case (e.g. GraphSpec | None)
+ if self._unwrap_optional_base_model(type_hint):
+ return self._kajson_deserialize_from_payload(payload=payload)
+
+ # BaseModel list case
+ origin = getattr(type_hint, "__origin__", None)
+ args = getattr(type_hint, "__args__", None)
+ log.verbose(f"Type hint origin: {origin}, args: {args}")
+ if origin is list and args and len(args) == 1 and issubclass(args[0], BaseModel):
+ log.debug(f"Type hint is a List of BaseModel: {args[0]}")
+ return self._kajson_deserialize_from_payload(payload=payload)
+
+ log.verbose(f"The type hint '{type_hint}' is neither a subclass of BaseModel nor a List of BaseModel")
+ return super().from_payload(payload, type_hint) # pyright: ignore[reportUnknownMemberType]
+
+
+class PydanticCompositePayloadConverter(CompositePayloadConverter):
+ """Payload converter that replaces Temporal JSON conversion without BaseModelPayloadConverter JSON conversion."""
+
+ def __init__(self) -> None:
+ converters: list[EncodingPayloadConverter] = []
+ for converter in DefaultPayloadConverter.default_encoding_payload_converters:
+ if isinstance(converter, JSONPlainPayloadConverter):
+ converters.append(BaseModelPayloadConverter())
+ else:
+ converters.append(converter)
+
+ super().__init__(*converters)
+
+
+def make_data_converter(payload_codec: PayloadCodec | None = None) -> DataConverter:
+ """Create a DataConverter with our Pydantic-aware payload converter and an optional codec.
+
+ Args:
+ payload_codec: Optional codec for offloading large payloads to external storage.
+ """
+ return DataConverter(
+ payload_converter_class=PydanticCompositePayloadConverter,
+ payload_codec=payload_codec,
+ )
+
+
+data_converter = make_data_converter()
diff --git a/pipelex/temporal/temporal_hub.py b/pipelex/temporal/temporal_hub.py
new file mode 100644
index 000000000..956363f1d
--- /dev/null
+++ b/pipelex/temporal/temporal_hub.py
@@ -0,0 +1,41 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from pipelex.temporal.task_manager import TaskManager
+
+
+class TemporalHub:
+ """TemporalHub serves as a central dependency manager to break cyclic imports between components.
+ It provides access to core providers and factories through a singleton instance,
+ allowing components to retrieve dependencies based on protocols without direct imports that could create cycles.
+ """
+
+ def __init__(self):
+ self._task_manager: TaskManager | None = None
+
+ # Setters
+
+ def set_task_manager(self, task_manager: TaskManager):
+ self._task_manager = task_manager
+
+ def reset(self):
+ self._task_manager = None
+
+ # Getters
+
+ def get_required_task_manager(self) -> TaskManager:
+ if self._task_manager is None:
+ msg = "TaskManager instance is not set. You must initialize Pipelex first."
+ raise RuntimeError(msg)
+ return self._task_manager
+
+
+temporal_hub = TemporalHub()
+
+# root convenience functions
+
+
+def get_task_manager() -> TaskManager:
+ return temporal_hub.get_required_task_manager()
diff --git a/pipelex/temporal/temporal_manager.py b/pipelex/temporal/temporal_manager.py
new file mode 100644
index 000000000..91c594e0e
--- /dev/null
+++ b/pipelex/temporal/temporal_manager.py
@@ -0,0 +1,118 @@
+from typing import ClassVar, Optional
+
+import shortuuid
+from temporalio.client import Client as TemporalClient
+
+from pipelex import log
+from pipelex.system.runtime import RunMode, runtime_manager
+from pipelex.temporal.config_temporal import TemporalServerConfig
+from pipelex.temporal.temporal_connect import connect_to_temporal, connect_to_temporal_selected_server, connect_to_temporal_server
+from pipelex.types import StrEnum
+
+
+class TemporalWorkerEnvironment(StrEnum):
+ INTERNAL = "internal"
+ EXTERNAL = "external"
+
+
+class TemporalManager:
+ _shared_instance: ClassVar[Optional["TemporalManager"]] = None
+
+ def __init__(self, session_id: str) -> None:
+ if TemporalManager._shared_instance is not None:
+ msg = "TemporalManager is a singleton. Use get_instance() to access it."
+ raise RuntimeError(msg)
+ super().__init__()
+ self.session_id = session_id
+ self.temporal_client: TemporalClient | None = None
+ TemporalManager._shared_instance = self
+
+ @classmethod
+ def get_instance(cls) -> "TemporalManager":
+ if cls._shared_instance is None:
+ msg = "Shared instance is not set. You must call TemporalManager.setup() once."
+ raise RuntimeError(msg)
+ return cls._shared_instance
+
+ @classmethod
+ def setup(cls, session_id: str) -> None:
+ if cls._shared_instance is not None:
+ cls.teardown()
+ cls._shared_instance = cls(session_id=session_id)
+ log.debug(f"TemporalManager setup done, session_id={session_id}")
+
+ @classmethod
+ def teardown(cls) -> None:
+ cls._shared_instance = None
+
+ async def connect_temporal(
+ self,
+ temporal_client: TemporalClient | None = None,
+ temporal_server_config: TemporalServerConfig | None = None,
+ temporal_selected_server: str | None = None,
+ ) -> TemporalClient:
+ """One remark first: this method is async only because Temporal's Client.connect() method is async.
+
+ This method is only passing your settings or using defaults from pipelex.temporal's config.
+ To init the temporal client, it will use one of the provided arguments if available, in the following order:
+ - temporal_client
+ - temporal_server_config
+ - temporal_selected_server
+ ... otherwise it will use the default connection set by temporal's config.
+ """
+ # select temporal client based on the available arguments
+ the_temporal_client: TemporalClient
+ if temporal_client:
+ log.dev("Connecting temporal using provided Temporal client")
+ the_temporal_client = temporal_client
+ elif temporal_server_config:
+ log.dev(f"Connecting temporal using provided Temporal server config: {temporal_server_config.description}")
+ the_temporal_client = await connect_to_temporal_server(
+ server_config=temporal_server_config,
+ )
+ elif temporal_selected_server:
+ log.dev(f"Connecting temporal using selected Temporal server: {temporal_selected_server}")
+ the_temporal_client = await connect_to_temporal_selected_server(
+ selected_server_config=temporal_selected_server,
+ )
+ else:
+ log.dev("Connecting temporal using default Temporal client")
+ # use automatic default connection
+ the_temporal_client = await connect_to_temporal()
+ self.temporal_client = the_temporal_client
+ log.debug(f"Temporal client connected, session_id={self.session_id}")
+ return the_temporal_client
+
+ async def get_temporal_client(self, should_auto_connect: bool) -> TemporalClient:
+ if self.temporal_client is not None:
+ return self.temporal_client
+ elif should_auto_connect:
+ return await self.connect_temporal()
+ else:
+ msg = "Temporal client not connected. Enable should_auto_connect or call TemporalManager.connect_temporal() first."
+ raise RuntimeError(msg)
+
+ def make_top_workflow_id(self, base_id: str) -> str:
+ prefix: str
+ match runtime_manager.run_mode:
+ case RunMode.UNIT_TEST:
+ prefix = "ut-"
+ case RunMode.NORMAL:
+ prefix = ""
+ case RunMode.CI_TEST:
+ prefix = "ci-"
+ case RunMode.CODEX_CLOUD:
+ prefix = "cc-"
+ case RunMode.CODEX_CLOUD_TEST:
+ prefix = "cct-"
+ session_part = self.session_id[:5]
+ random_part = shortuuid.uuid()[:5]
+ return f"{prefix}{session_part}-{random_part}-{base_id}"
+
+
+def get_temporal_manager() -> TemporalManager:
+ return TemporalManager.get_instance()
+
+
+async def get_temporal_client(should_auto_connect: bool) -> TemporalClient:
+ return await get_temporal_manager().get_temporal_client(should_auto_connect=should_auto_connect)
diff --git a/pipelex/temporal/temporal_registry_models.py b/pipelex/temporal/temporal_registry_models.py
new file mode 100644
index 000000000..ecead90a1
--- /dev/null
+++ b/pipelex/temporal/temporal_registry_models.py
@@ -0,0 +1,7 @@
+from typing import ClassVar
+
+from pipelex.system.registries.registry_base import ModelType, RegistryModels
+
+
+class TemporalRegistryModels(RegistryModels):
+ GENERIC: ClassVar[list[ModelType]] = []
diff --git a/pipelex/temporal/temporal_task_manager.py b/pipelex/temporal/temporal_task_manager.py
new file mode 100644
index 000000000..e65091eaa
--- /dev/null
+++ b/pipelex/temporal/temporal_task_manager.py
@@ -0,0 +1,216 @@
+import asyncio
+from datetime import timedelta
+
+from temporalio import workflow
+from temporalio.client import Client as TemporalClient
+from temporalio.worker import UnsandboxedWorkflowRunner, Worker, WorkflowRunner
+from temporalio.worker.workflow_sandbox import (
+ SandboxedWorkflowRunner,
+ SandboxRestrictions,
+)
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.hub import get_class_registry
+from pipelex.system.runtime import WorkerMode, runtime_manager
+from pipelex.temporal.config_temporal import WorkerScope
+from pipelex.temporal.exceptions import WorkerScopeConfigError
+from pipelex.temporal.log_temporal import configure_temporal_logs
+from pipelex.temporal.sandbox_manager import sandbox_manager
+from pipelex.temporal.task_manager import TaskManager
+from pipelex.temporal.temporal_connect import connect_to_temporal
+from pipelex.temporal.temporal_manager import TemporalManager
+from pipelex.temporal.temporal_registry_models import TemporalRegistryModels
+from pipelex.temporal.temporal_tasks import TaskPack, TemporalTasks
+from pipelex.temporal.temporal_types import (
+ ActivityList,
+ ActivityType,
+ WorkflowList,
+ WorkflowType,
+)
+from pipelex.temporal.test_extras.temporal_registry_test_models import TemporalTestModels
+
+
+def is_in_temporal_sandbox() -> bool:
+ """Check if we're in a Temporal workflow sandbox.
+
+ Returns:
+ bool: True if in a workflow sandbox, False otherwise.
+ """
+ return workflow.unsafe.in_sandbox()
+
+
+class TemporalTaskManager(TaskManager):
+ def __init__(self):
+ self.temporal_tasks = self.__class__.create_temporal_tasks()
+
+ @classmethod
+ def create_temporal_tasks(cls) -> TemporalTasks:
+ return TemporalTasks()
+
+ def setup(self):
+ get_class_registry().register_classes(TemporalRegistryModels.get_all_models())
+ if runtime_manager.is_unit_testing:
+ log.debug("Registering test models for unit testing")
+ get_class_registry().register_classes(TemporalTestModels.get_all_models())
+ configure_temporal_logs()
+ # TODO: use direct tweaking of settings in Pipelex to apply the sandbox restrictions to loggers etc.
+ sandbox_manager.set_sandbox_callable(sandbox_callable=is_in_temporal_sandbox)
+ TemporalManager.setup(session_id=get_config().session_id)
+ log.info("TemporalTaskManager setup done")
+
+ def teardown(self):
+ TemporalManager.teardown()
+ log.info("TemporalTaskManager teardown done")
+
+ @override
+ def complement_catalog(
+ self,
+ extra_catalog: dict[str, TaskPack],
+ extra_workflows: list[WorkflowType],
+ extra_activities: list[ActivityType],
+ ):
+ self.temporal_tasks.complement_catalog(
+ extra_catalog=extra_catalog,
+ extra_workflows=extra_workflows,
+ extra_activities=extra_activities,
+ )
+
+ @override
+ def make_worker(
+ self,
+ temporal_client: TemporalClient,
+ task_queue: str,
+ is_not_sandboxed: bool = False,
+ scope: WorkerScope | None = None,
+ substitute_activities: dict[ActivityType, ActivityType] | None = None,
+ test_workflows: WorkflowList | None = None,
+ test_activities: ActivityList | None = None,
+ ) -> Worker:
+ workflows, activities = self.temporal_tasks.workflows_and_activities(
+ scope=scope,
+ test_activities=test_activities,
+ test_workflows=test_workflows,
+ substitute_activities=substitute_activities,
+ )
+ workflow_runner: WorkflowRunner
+ if is_not_sandboxed:
+ workflow_runner = UnsandboxedWorkflowRunner()
+ else:
+ workflow_runner = SandboxedWorkflowRunner(
+ restrictions=SandboxRestrictions.default.with_passthrough_modules(
+ "pydantic",
+ "pydantic.root_model",
+ "bson",
+ "pymongo",
+ "networkx",
+ "google.auth",
+ "google.cloud.logging",
+ "cogt.oai.azure_openai_config",
+ "pipelex.tools.log.log",
+ "pipelex.tools.log.log_config",
+ "pipelex.tools.log.log_dispatch",
+ "pipelex.tools.environment",
+ "citadel.mdb.models",
+ "citadel.mdb.connector",
+ "citadel.gcp.gsecret_helpers",
+ )
+ )
+ return Worker(
+ temporal_client,
+ task_queue=task_queue,
+ workflows=workflows,
+ activities=activities,
+ workflow_runner=workflow_runner,
+ max_cached_workflows=10000,
+ max_concurrent_workflow_tasks=1000,
+ max_concurrent_activities=1000,
+ max_concurrent_local_activities=1000,
+ max_concurrent_workflow_task_polls=100,
+ max_concurrent_activity_task_polls=100,
+ sticky_queue_schedule_to_start_timeout=timedelta(minutes=30),
+ max_heartbeat_throttle_interval=timedelta(minutes=60),
+ default_heartbeat_throttle_interval=timedelta(minutes=60),
+ graceful_shutdown_timeout=timedelta(minutes=30),
+ max_activities_per_second=1000,
+ max_task_queue_activities_per_second=1000,
+ )
+
+ @override
+ async def run_worker(
+ self,
+ is_not_sandboxed: bool,
+ is_unit_testing: bool,
+ task_queue: str | None = None,
+ scope_name: str | None = None,
+ ):
+ try:
+ test_workflows: WorkflowList | None = None
+ test_activities: ActivityList | None = None
+ if is_unit_testing:
+ log.debug(f"is_unit_testing={is_unit_testing} Registering test models")
+ runtime_manager.set_worker_mode(worker_mode=WorkerMode.UNIT_TEST)
+ from pipelex.temporal.test_extras.temporal_test_tasks import ( # noqa: PLC0415
+ TEMPORAL_TEST_ACTIVITIES,
+ TEMPORAL_TEST_WORKFLOWS,
+ )
+
+ test_workflows = TEMPORAL_TEST_WORKFLOWS
+ test_activities = TEMPORAL_TEST_ACTIVITIES
+ else:
+ log.debug(f"is_unit_testing={is_unit_testing} Setting worker mode to NORMAL")
+ runtime_manager.set_worker_mode(worker_mode=WorkerMode.NORMAL)
+ temporal_client = await connect_to_temporal()
+ worker_config = get_config().temporal.worker_config
+ task_queue = task_queue or worker_config.task_queue
+ scope = self._resolve_scope_by_name(scope_name=scope_name)
+ log.info(f"Temporal Worker scope: '{scope_name or get_config().temporal.worker_scopes.default_scope}'")
+ async with self.make_worker(
+ temporal_client=temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=is_not_sandboxed,
+ scope=scope,
+ test_workflows=test_workflows,
+ test_activities=test_activities,
+ ):
+ log.info(f"Temporal Worker started for '{task_queue}'")
+ if is_not_sandboxed:
+ log.warning("Worker is running without sandbox")
+ await asyncio.Event().wait()
+ except asyncio.CancelledError:
+ log.info("Main coroutine was cancelled.")
+ finally:
+ log.info("Shutting down")
+
+ @staticmethod
+ def _resolve_scope_by_name(scope_name: str | None) -> WorkerScope:
+ worker_scopes = get_config().temporal.worker_scopes
+ effective_name = scope_name or worker_scopes.default_scope
+ if effective_name not in worker_scopes.scopes:
+ msg = f"Unknown worker scope '{effective_name}' (known: {sorted(worker_scopes.scopes.keys())})"
+ raise WorkerScopeConfigError(msg)
+ return worker_scopes.scopes[effective_name]
+
+ @override
+ def task_packs(self) -> list[str]:
+ return self.temporal_tasks.task_packs()
+
+ @override
+ def workflows_and_activities(
+ self,
+ scope: WorkerScope | None = None,
+ test_workflows: WorkflowList | None = None,
+ test_activities: ActivityList | None = None,
+ substitute_activities: dict[ActivityType, ActivityType] | None = None,
+ ) -> tuple[list[WorkflowType], list[ActivityType]]:
+ return self.temporal_tasks.workflows_and_activities(
+ scope=scope,
+ test_workflows=test_workflows,
+ test_activities=test_activities,
+ substitute_activities=substitute_activities,
+ )
+
+ @override
+ def workflows_and_activities_str(self) -> tuple[list[str], list[str]]:
+ return self.temporal_tasks.workflows_and_activities_str()
diff --git a/pipelex/temporal/temporal_tasks.py b/pipelex/temporal/temporal_tasks.py
new file mode 100644
index 000000000..273c70b27
--- /dev/null
+++ b/pipelex/temporal/temporal_tasks.py
@@ -0,0 +1,178 @@
+from pipelex import log
+from pipelex.temporal.config_temporal import WorkerScope
+from pipelex.temporal.exceptions import TemporalConfigError, WorkerScopeConfigError
+from pipelex.temporal.temporal_types import ActivityList, ActivityType, WorkflowList, WorkflowType
+
+
+class TaskPack:
+ def __init__(self, workflow_list: WorkflowList, activity_list: ActivityList):
+ self.workflow_list = workflow_list
+ self.activity_list = activity_list
+
+
+class TemporalTasks:
+ def __init__(self):
+ self.tasks_catalog: dict[str, TaskPack] = {}
+ self.extra_workflows: list[WorkflowType] = []
+ self.extra_activities: list[ActivityType] = []
+
+ def complement_catalog(
+ self,
+ extra_catalog: dict[str, TaskPack],
+ extra_workflows: list[WorkflowType],
+ extra_activities: list[ActivityType],
+ ):
+ log.verbose(extra_catalog, title="Complementing catalog with extra_catalog")
+ for pack_name, task_pack in extra_catalog.items():
+ if pack_name in self.tasks_catalog:
+ msg = f"Task pack with the same name '{pack_name}' already exists, no merging or overriding allowed for task packs"
+ raise RuntimeError(msg)
+ self.tasks_catalog[pack_name] = task_pack
+
+ log.verbose(extra_workflows, title="Complementing catalog with extra_workflows")
+ self.extra_workflows = list(set(self.extra_workflows + extra_workflows))
+ log.verbose(extra_activities, title="Complementing catalog with extra_activities")
+ self.extra_activities = list(set(self.extra_activities + extra_activities))
+
+ def replace_catalog(
+ self,
+ new_catalog: dict[str, TaskPack],
+ new_workflows: list[WorkflowType],
+ new_activities: list[ActivityType],
+ ):
+ self.tasks_catalog = new_catalog
+ self.extra_workflows = new_workflows
+ self.extra_activities = new_activities
+
+ def task_packs(self) -> list[str]:
+ return list(self.tasks_catalog.keys())
+
+ def _full_catalog(self) -> tuple[dict[str, WorkflowType], dict[str, ActivityType]]:
+ all_workflows: dict[str, WorkflowType] = {}
+ all_activities: dict[str, ActivityType] = {}
+ for pack_name, task_pack in self.tasks_catalog.items():
+ source = f"task pack '{pack_name}'"
+ for workflow in task_pack.workflow_list:
+ self._register_workflow(all_workflows=all_workflows, workflow=workflow, source=source)
+ for activity in task_pack.activity_list:
+ self._register_activity(all_activities=all_activities, activity=activity, source=source)
+ for workflow in self.extra_workflows:
+ self._register_workflow(all_workflows=all_workflows, workflow=workflow, source="extra_workflows")
+ for activity in self.extra_activities:
+ self._register_activity(all_activities=all_activities, activity=activity, source="extra_activities")
+ return all_workflows, all_activities
+
+ @classmethod
+ def _register_workflow(cls, all_workflows: dict[str, WorkflowType], workflow: WorkflowType, source: str) -> None:
+ existing = all_workflows.get(workflow.__name__)
+ if existing is not None and existing is not workflow:
+ msg = (
+ f"Workflow name collision: '{workflow.__name__}' is registered as two different callables "
+ f"(latest source: {source}). Temporal requires a unique name per workflow."
+ )
+ raise TemporalConfigError(msg)
+ all_workflows[workflow.__name__] = workflow
+
+ @classmethod
+ def _register_activity(cls, all_activities: dict[str, ActivityType], activity: ActivityType, source: str) -> None:
+ existing = all_activities.get(activity.__name__)
+ if existing is not None and existing is not activity:
+ msg = (
+ f"Activity name collision: '{activity.__name__}' is registered as two different callables "
+ f"(latest source: {source}). Temporal requires a unique name per activity."
+ )
+ raise TemporalConfigError(msg)
+ all_activities[activity.__name__] = activity
+
+ def workflows_and_activities(
+ self,
+ scope: WorkerScope | None = None,
+ test_workflows: WorkflowList | None = None,
+ test_activities: ActivityList | None = None,
+ substitute_activities: dict[ActivityType, ActivityType] | None = None,
+ ) -> tuple[list[WorkflowType], list[ActivityType]]:
+ all_workflows, all_activities = self._full_catalog()
+
+ the_workflows: set[WorkflowType]
+ the_activities: set[ActivityType]
+ if scope is None:
+ the_workflows = set(all_workflows.values())
+ the_activities = set(all_activities.values())
+ else:
+ the_workflows, the_activities = self._resolve_scope(
+ scope=scope,
+ all_workflows=all_workflows,
+ all_activities=all_activities,
+ )
+
+ if test_workflows:
+ the_workflows.update(test_workflows)
+ if test_activities:
+ the_activities.update(test_activities)
+ if substitute_activities:
+ for old_activity, new_activity in substitute_activities.items():
+ if old_activity in the_activities:
+ the_activities.remove(old_activity)
+ the_activities.add(new_activity)
+ return list(the_workflows), list(the_activities)
+
+ def _resolve_scope(
+ self,
+ scope: WorkerScope,
+ all_workflows: dict[str, WorkflowType],
+ all_activities: dict[str, ActivityType],
+ ) -> tuple[set[WorkflowType], set[ActivityType]]:
+ known_packs = set(self.tasks_catalog.keys())
+ unknown_packs = [pack for pack in scope.required_tasks_packs if pack not in known_packs]
+ if unknown_packs:
+ msg = f"Unknown task pack(s) in scope: {unknown_packs} (known: {sorted(known_packs)})"
+ raise WorkerScopeConfigError(msg)
+
+ unknown_required_workflows = [name for name in scope.required_workflows if name not in all_workflows]
+ if unknown_required_workflows:
+ msg = f"Unknown workflow(s) in scope.required_workflows: {unknown_required_workflows} (known: {sorted(all_workflows.keys())})"
+ raise WorkerScopeConfigError(msg)
+ unknown_required_activities = [name for name in scope.required_activities if name not in all_activities]
+ if unknown_required_activities:
+ msg = f"Unknown activity(ies) in scope.required_activities: {unknown_required_activities} (known: {sorted(all_activities.keys())})"
+ raise WorkerScopeConfigError(msg)
+ unknown_excluded_workflows = [name for name in scope.excluded_workflows if name not in all_workflows]
+ if unknown_excluded_workflows:
+ msg = f"Unknown workflow(s) in scope.excluded_workflows: {unknown_excluded_workflows} (known: {sorted(all_workflows.keys())})"
+ raise WorkerScopeConfigError(msg)
+ unknown_excluded_activities = [name for name in scope.excluded_activities if name not in all_activities]
+ if unknown_excluded_activities:
+ msg = f"Unknown activity(ies) in scope.excluded_activities: {unknown_excluded_activities} (known: {sorted(all_activities.keys())})"
+ raise WorkerScopeConfigError(msg)
+
+ the_workflows: set[WorkflowType] = set()
+ the_activities: set[ActivityType] = set()
+ for pack_name in scope.required_tasks_packs:
+ task_pack = self.tasks_catalog[pack_name]
+ the_workflows.update(task_pack.workflow_list)
+ the_activities.update(task_pack.activity_list)
+
+ the_workflows.update(all_workflows[name] for name in scope.required_workflows)
+ the_activities.update(all_activities[name] for name in scope.required_activities)
+
+ the_workflows.difference_update(all_workflows[name] for name in scope.excluded_workflows)
+ the_activities.difference_update(all_activities[name] for name in scope.excluded_activities)
+
+ if scope.disable_all_workflows:
+ the_workflows.clear()
+ if scope.disable_all_activities:
+ the_activities.clear()
+
+ if not the_workflows and not the_activities:
+ msg = (
+ f"Worker scope resolves to an empty set (packs={scope.required_tasks_packs}, "
+ f"required_workflows={scope.required_workflows}, required_activities={scope.required_activities}, "
+ f"excluded_workflows={scope.excluded_workflows}, excluded_activities={scope.excluded_activities})"
+ )
+ raise WorkerScopeConfigError(msg)
+
+ return the_workflows, the_activities
+
+ def workflows_and_activities_str(self) -> tuple[list[str], list[str]]:
+ workflows, activities = self.workflows_and_activities(test_workflows=None, test_activities=None)
+ return [str(w.__name__) for w in workflows], [str(a.__name__) for a in activities]
diff --git a/pipelex/temporal/temporal_types.py b/pipelex/temporal/temporal_types.py
new file mode 100644
index 000000000..014c3eb35
--- /dev/null
+++ b/pipelex/temporal/temporal_types.py
@@ -0,0 +1,6 @@
+from typing import Any, Callable
+
+WorkflowType = type[Any]
+WorkflowList = list[WorkflowType]
+ActivityType = Callable[[Any], Any]
+ActivityList = list[ActivityType]
diff --git a/pipelex/temporal/temporal_workflow_utils.py b/pipelex/temporal/temporal_workflow_utils.py
new file mode 100644
index 000000000..fabe3844e
--- /dev/null
+++ b/pipelex/temporal/temporal_workflow_utils.py
@@ -0,0 +1,16 @@
+from temporalio import workflow
+from temporalio.exceptions import TemporalError
+
+
+def is_in_temporal_workflow() -> bool:
+ """Check if the current code is running inside a Temporal workflow.
+
+ Uses workflow.info() which raises if not inside a workflow context.
+ Used by `TemporalPipeRouter` and `TemporalPipeRun` to decide between top-level dispatch
+ (`execute_workflow`) and child dispatch (`execute_child_workflow`).
+ """
+ try:
+ workflow.info()
+ return True
+ except (TemporalError, RuntimeError):
+ return False
diff --git a/pipelex/temporal/test_extras/__init__.py b/pipelex/temporal/test_extras/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/temporal/test_extras/models_for_tests.py b/pipelex/temporal/test_extras/models_for_tests.py
new file mode 100644
index 000000000..44b559620
--- /dev/null
+++ b/pipelex/temporal/test_extras/models_for_tests.py
@@ -0,0 +1,5 @@
+from pipelex.core.stuffs.structured_content import StructuredContent
+
+
+class Mock(StructuredContent):
+ pass
diff --git a/pipelex/temporal/test_extras/temporal_registry_test_models.py b/pipelex/temporal/test_extras/temporal_registry_test_models.py
new file mode 100644
index 000000000..f9f4a9a5a
--- /dev/null
+++ b/pipelex/temporal/test_extras/temporal_registry_test_models.py
@@ -0,0 +1,17 @@
+from typing import ClassVar
+
+from pydantic import BaseModel, ConfigDict
+
+from pipelex.system.registries.registry_base import ModelType, RegistryModels
+
+
+# for testing & examples
+class Person(BaseModel):
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+ name: str
+ age: int
+ job: str
+
+
+class TemporalTestModels(RegistryModels):
+ TEST_MODELS: ClassVar[list[ModelType]] = [Person]
diff --git a/pipelex/temporal/test_extras/temporal_test_tasks.py b/pipelex/temporal/test_extras/temporal_test_tasks.py
new file mode 100644
index 000000000..2361c88a8
--- /dev/null
+++ b/pipelex/temporal/test_extras/temporal_test_tasks.py
@@ -0,0 +1,10 @@
+from pipelex.temporal.temporal_types import ActivityList, WorkflowList
+from pipelex.temporal.test_extras.wf_test_child_pipe import WfTestChildPipeLLMObject, WfTestChildPipeLLMText
+from pipelex.temporal.test_extras.wf_test_content_generator_child import WfTestContentGeneratorChild
+
+TEMPORAL_TEST_WORKFLOWS: WorkflowList = [WfTestContentGeneratorChild]
+TEMPORAL_TEST_ACTIVITIES: ActivityList = []
+
+
+PIPELEX_TEMPORAL_TEST_WORKFLOWS: WorkflowList = [WfTestChildPipeLLMText, WfTestChildPipeLLMObject]
+PIPELEX_TEMPORAL_TEST_ACTIVITIES: ActivityList = []
diff --git a/pipelex/temporal/test_extras/wf_test_child_pipe.py b/pipelex/temporal/test_extras/wf_test_child_pipe.py
new file mode 100644
index 000000000..8343a9d9b
--- /dev/null
+++ b/pipelex/temporal/test_extras/wf_test_child_pipe.py
@@ -0,0 +1,87 @@
+from temporalio import workflow
+
+with workflow.unsafe.imports_passed_through():
+ from pipelex.core.concepts.native.concept_native import NativeConceptCode
+ from pipelex.core.pipes.pipe_factory import PipeFactory
+ from pipelex.core.pipes.pipe_output import PipeOutput # noqa: TC001
+ from pipelex.hub import get_pipe_library, get_pipe_router
+ from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
+ from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+ from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
+ from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+ from pipelex.pipeline.job_metadata import JobMetadata
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.test_extras.models_for_tests import Mock
+
+
+class WfTestChildPipeTestCases:
+ SYSTEM_PROMPT = "You are a pirate, you always talk like a pirate."
+ USER_PROMPT = "In 3 sentences, tell me about the sea."
+ USER_TEXT_TRICKY_1 = """
+When my son was 7 he was 3ft tall. When he was 8 he was 4ft tall. When he was 9 he was 5ft tall.
+How tall do you think he was when he was 12? and at 15?
+"""
+
+
[email protected](name="wf_test_child_pipe_llm_text")
+class WfTestChildPipeLLMText:
+ @workflow.run
+ async def run(self) -> str | None:
+ workflow_log.debug("Workflow start")
+ pipe_llm_blueprint = PipeLLMBlueprint(
+ description="LLM test for basic text generation",
+ output=NativeConceptCode.TEXT,
+ system_prompt=WfTestChildPipeTestCases.SYSTEM_PROMPT,
+ prompt=WfTestChildPipeTestCases.USER_PROMPT,
+ )
+ pipe = PipeFactory[PipeLLM].make_from_blueprint(
+ domain_code="adhoc",
+ pipe_code="adhoc_for_test_child_pipe_llm_text",
+ blueprint=pipe_llm_blueprint,
+ )
+ pipe_library = get_pipe_library()
+ pipe_library.add_new_pipe(pipe)
+
+ pipe_job = PipeJobFactory.make_pipe_job(
+ pipe=pipe,
+ pipe_run_params=PipeRunParamsFactory.make_run_params(),
+ job_metadata=JobMetadata(
+ user_id="temporal_test",
+ pipeline_run_id=workflow.info().workflow_id,
+ ),
+ )
+ pipe_output: PipeOutput = await get_pipe_router().run(pipe_job=pipe_job)
+ return pipe_output.main_stuff_as_text.text
+
+
[email protected](name="wf_test_child_pipe_llm_object")
+class WfTestChildPipeLLMObject:
+ @workflow.run
+ async def run(self) -> Mock:
+ workflow_log.debug("Workflow start")
+ pipe_llm_blueprint = PipeLLMBlueprint(
+ description="LLM test for object generation",
+ output="tests.Mock",
+ prompt=WfTestChildPipeTestCases.USER_TEXT_TRICKY_1,
+ model_to_structure="llm_to_reason",
+ )
+ pipe = PipeFactory[PipeLLM].make_from_blueprint(
+ domain_code="adhoc",
+ pipe_code="adhoc_for_test_child_pipe_llm_object",
+ blueprint=pipe_llm_blueprint,
+ )
+ pipe_library = get_pipe_library()
+ pipe_library.add_new_pipe(pipe)
+
+ pipe_job = PipeJobFactory.make_pipe_job(
+ pipe=pipe,
+ pipe_run_params=PipeRunParamsFactory.make_run_params(),
+ job_metadata=JobMetadata(
+ user_id="temporal_test",
+ pipeline_run_id=workflow.info().workflow_id,
+ ),
+ )
+ pipe_output: PipeOutput = await get_pipe_router().run(pipe_job=pipe_job)
+ llm_generated_object = pipe_output.main_stuff_as(content_type=Mock)
+ assert isinstance(llm_generated_object, Mock)
+ return llm_generated_object
diff --git a/pipelex/temporal/test_extras/wf_test_content_generator_child.py b/pipelex/temporal/test_extras/wf_test_content_generator_child.py
new file mode 100644
index 000000000..5d63c459c
--- /dev/null
+++ b/pipelex/temporal/test_extras/wf_test_content_generator_child.py
@@ -0,0 +1,142 @@
+from temporalio import workflow
+
+with workflow.unsafe.imports_passed_through():
+ from pipelex import pretty_print
+ from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry
+ from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol # noqa: TC001
+ from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+ from pipelex.cogt.extract.extract_input import ExtractInput
+ from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
+ from pipelex.cogt.llm.llm_prompt import LLMPrompt
+ from pipelex.hub import get_model_deck
+ from pipelex.pipeline.job_metadata import JobMetadata
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.test_extras.temporal_registry_test_models import Person
+ from pipelex.temporal.tprl_content_generation.content_generator_child_factory import ContentGeneratorChildFactory
+ from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
+ from tests.integration.pipelex.temporal.test_data import PipeTestCases
+
+
+USER_TEXT_FOR_BASE = """
+Write a detailed description of a woman's clothing in the style of a 19th-century novel.
+Keep it short: 3 sentences max
+"""
+
+USER_TEXT_FOR_SINGLE_PERSON = "name: John, age: 30, job: bank teller"
+USER_TEXT_FOR_SINGLE_PERSON_TEXT_THEN_OBJECT = """
+Imagine a female character that decides to become a cop once reaching middle age.
+Present this character in a couple of very short sentences.
+Be sure to include the character's full name, age and job.
+"""
+MULTIPLE_USER_TEXTS_FOR_PEOPLE = [
+ "name: Bob, age: 25, job: banker",
+ "name: Maria, age: 35, job: consultant",
+ "name: SLartiblfastikur, age: 30, job: fizzy buzzer",
+ "name: Alice, age: 40, job: developer",
+ "name: Tom, age: 45, job: TV presenter",
+ "name: Jerry, age: 50, job: nurse",
+]
+USER_TEXTS_FOR_PEOPLE_STR = "\n".join(MULTIPLE_USER_TEXTS_FOR_PEOPLE)
+USER_TEXT_FOR_MULTIPLE_PEOPLE_TEXT_THEN_OBJECT = """
+Imagine the 4 main characters for a sitcom in Paris.
+Present each character in one very short sentence.
+Be sure to include each character's full name, age and job.
+"""
+
+USER_TEXT_FOR_HAIKU = """
+Write a haiku about the meaning of life
+"""
+
+POSITIVE_TEXT_FOR_IMAGE = "A group of dogs with sunglasses playing beer pong with drinks and snacks"
+
+
[email protected](name="wf_test_content_generator_child")
+class WfTestContentGeneratorChild:
+ @workflow.run
+ async def run(self, is_dry_run: bool = False):
+ workflow_log.debug("Workflow start")
+ content_generator: ContentGeneratorProtocol
+ if is_dry_run:
+ content_generator = ContentGeneratorDry()
+ else:
+ generated_content_factory = GeneratedContentFactory(storage_provider=InMemoryStorageProvider())
+ content_generator = ContentGeneratorChildFactory.make_content_generator_child(
+ generated_content_factory=generated_content_factory,
+ )
+
+ job_metadata = JobMetadata(
+ user_id="temporal-test",
+ pipeline_run_id=workflow.info().workflow_id,
+ )
+
+ llm_setting_for_text = get_model_deck().get_llm_setting(llm_choice="$testing-text")
+ crafted_text = await content_generator.make_llm_text(
+ job_metadata=job_metadata,
+ llm_setting_main=llm_setting_for_text,
+ llm_prompt_for_text=LLMPrompt(user_text=USER_TEXT_FOR_BASE),
+ )
+ pretty_print(crafted_text, title="make_llm_text")
+
+ llm_setting_for_object = get_model_deck().get_llm_setting(llm_choice="$testing-structured")
+ crafted_object_direct = await content_generator.make_object_direct(
+ job_metadata=job_metadata,
+ object_class=Person,
+ llm_setting_for_object=llm_setting_for_object,
+ llm_prompt_for_object=LLMPrompt(user_text=USER_TEXT_FOR_SINGLE_PERSON),
+ )
+ pretty_print(crafted_object_direct, title="make_object_direct")
+
+ crafted_object = await content_generator.make_text_then_object(
+ job_metadata=job_metadata,
+ object_class=Person,
+ llm_setting_main=llm_setting_for_text,
+ llm_setting_for_object=llm_setting_for_object,
+ llm_prompt_for_text=LLMPrompt(user_text=USER_TEXT_FOR_SINGLE_PERSON_TEXT_THEN_OBJECT),
+ )
+ pretty_print(crafted_object, title="make_text_then_object")
+
+ crafted_object_list_direct = await content_generator.make_object_list_direct(
+ job_metadata=job_metadata,
+ object_class=Person,
+ llm_setting_for_object_list=llm_setting_for_object,
+ llm_prompt_for_object_list=LLMPrompt(user_text=USER_TEXTS_FOR_PEOPLE_STR),
+ )
+ pretty_print(crafted_object_list_direct, title="make_object_list_direct")
+
+ crafted_object_list = await content_generator.make_text_then_object_list(
+ job_metadata=job_metadata,
+ object_class=Person,
+ llm_setting_main=llm_setting_for_text,
+ llm_setting_for_object_list=llm_setting_for_object,
+ llm_prompt_for_text=LLMPrompt(user_text=USER_TEXT_FOR_MULTIPLE_PEOPLE_TEXT_THEN_OBJECT),
+ )
+ pretty_print(crafted_object_list, title="make_text_then_object_list")
+
+ # TODO: fix this
+ # crafted_image = await content_generator.craft_image(
+ # job_metadata=job_metadata,
+ # img_gen_prompt=ImgGenPrompt(positive_text=POSITIVE_TEXT_FOR_IMAGE),
+ # )
+ # pretty_print(crafted_image, title="craft_image")
+ context = {
+ "the_answer": "elementary, my dear Watson",
+ }
+ jinja2_text = await content_generator.make_templated_text(
+ job_metadata=job_metadata,
+ context=context,
+ template="♦️♦️ {{ the_answer }} ♦️♦️",
+ )
+ pretty_print(jinja2_text, title="templated_text")
+
+ page_contents = await content_generator.make_extract_pages(
+ extract_input=ExtractInput(
+ image_uri=PipeTestCases.IMG_EXPENSE_REPORT_1,
+ ),
+ extract_handle="azure-document-intelligence",
+ job_metadata=job_metadata,
+ extract_job_params=ExtractJobParams.make_default_extract_job_params(),
+ extract_job_config=ExtractJobConfig(),
+ )
+ pretty_print(page_contents, title="make_extract_pages")
+
+ workflow_log.debug("Workflow complete")
diff --git a/pipelex/temporal/test_helpers/__init__.py b/pipelex/temporal/test_helpers/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/temporal/test_helpers/temporal_pytest_plugins.py b/pipelex/temporal/test_helpers/temporal_pytest_plugins.py
new file mode 100644
index 000000000..4d37f67fd
--- /dev/null
+++ b/pipelex/temporal/test_helpers/temporal_pytest_plugins.py
@@ -0,0 +1,17 @@
+from pytest import Parser
+
+from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment
+from pipelex.types import StrEnum
+
+
+class TemporalPytestOption(StrEnum):
+ WORKER_ENVIRONMENT = "--worker"
+
+
+def pytest_addoption(parser: Parser):
+ parser.addoption(
+ TemporalPytestOption.WORKER_ENVIRONMENT,
+ type=TemporalWorkerEnvironment,
+ default=TemporalWorkerEnvironment.INTERNAL,
+ help="Which temporal worker environment to use ('internal', 'external')",
+ )
diff --git a/pipelex/temporal/tprl/__init__.py b/pipelex/temporal/tprl/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/temporal/tprl/conditional_worker.py b/pipelex/temporal/tprl/conditional_worker.py
new file mode 100644
index 000000000..fe388968d
--- /dev/null
+++ b/pipelex/temporal/tprl/conditional_worker.py
@@ -0,0 +1,41 @@
+from functools import wraps
+from typing import Any, Awaitable, Callable, TypeVar, cast
+
+import shortuuid
+
+from pipelex import log
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor, WorkflowInput, WorkflowOutput
+
+FuncExecuteWorkflow = TypeVar("FuncExecuteWorkflow", bound=Callable[..., Awaitable[Any]])
+
+
+def with_conditional_worker(execute_workflow: FuncExecuteWorkflow) -> FuncExecuteWorkflow:
+ @wraps(execute_workflow)
+ async def wrapper(
+ self: WorkflowExecutor[WorkflowInput, WorkflowOutput],
+ *args: Any,
+ **kwargs: Any,
+ ) -> Any:
+ log.debug(f"worker_environment: {self.worker_environment}")
+ match self.worker_environment:
+ case TemporalWorkerEnvironment.INTERNAL:
+ # add some random to task_queue to avoid this worker to take over other (possibly failed) tasks scheduled by preceding tests
+ original_task_queue = self.task_queue
+ self.task_queue = f"{self.task_queue}-{shortuuid.uuid()[:5]}"
+
+ temporal_client = await self.temporal_client()
+ try:
+ async with get_task_manager().make_worker(
+ temporal_client=temporal_client,
+ task_queue=self.task_queue,
+ is_not_sandboxed=True,
+ ):
+ return await execute_workflow(self, *args, **kwargs)
+ finally:
+ self.task_queue = original_task_queue
+ case TemporalWorkerEnvironment.EXTERNAL:
+ return await execute_workflow(self, *args, **kwargs)
+
+ return cast("FuncExecuteWorkflow", wrapper)
diff --git a/pipelex/temporal/tprl/temporal_error.py b/pipelex/temporal/tprl/temporal_error.py
new file mode 100644
index 000000000..f64ee49af
--- /dev/null
+++ b/pipelex/temporal/tprl/temporal_error.py
@@ -0,0 +1,40 @@
+from temporalio.exceptions import ApplicationError
+
+from pipelex.base_exceptions import PipelexError
+from pipelex.config import get_config
+from pipelex.temporal.log_temporal import workflow_log
+from pipelex.types import Self
+
+
+class TemporalError(ApplicationError):
+ def __init__(self, message: str, error_type: str | None):
+ super().__init__(
+ message=message,
+ type=error_type,
+ )
+
+ @classmethod
+ def from_app_error(cls, exc: ApplicationError) -> Self:
+ message = exc.message
+ error_type = exc.type
+ if error_type in get_config().temporal.worker_config.retry_policy_config.non_retryable_error_types:
+ workflow_log.critical(f"Non retryable error from ApplicationError[{error_type}]: {message}")
+ else:
+ workflow_log.error(f"Error from ApplicationError[{error_type}]: {message}")
+ return cls(
+ message=message,
+ error_type=error_type,
+ )
+
+ @classmethod
+ def from_message_exception(cls, exc: PipelexError) -> Self:
+ message = exc.message
+ error_type = exc.__class__.__name__
+ if error_type in get_config().temporal.worker_config.retry_policy_config.non_retryable_error_types:
+ workflow_log.critical(f"Non retryable error from PipelexError[{error_type}]: {message}")
+ else:
+ workflow_log.error(f"Retryable error from PipelexError[{error_type}]: {message}")
+ return cls(
+ message=message,
+ error_type=error_type,
+ )
diff --git a/pipelex/temporal/tprl/workflow_caller.py b/pipelex/temporal/tprl/workflow_caller.py
new file mode 100644
index 000000000..267ac2255
--- /dev/null
+++ b/pipelex/temporal/tprl/workflow_caller.py
@@ -0,0 +1,248 @@
+from collections.abc import Sequence
+from datetime import timedelta
+from typing import Any, Callable, Coroutine, Generic, Protocol, TypeVar, Union, cast
+
+from pydantic import BaseModel
+from temporalio import workflow
+from temporalio.client import Callback, WorkflowHandle
+from temporalio.client import Client as TemporalClient
+from temporalio.common import RetryPolicy
+from temporalio.exceptions import ApplicationError, ChildWorkflowError
+from temporalio.workflow import ChildWorkflowHandle
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.temporal.exceptions import WorkflowExecutionError
+from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment, get_temporal_client, get_temporal_manager
+
+T = TypeVar("T")
+WorkflowOutput = TypeVar("WorkflowOutput", covariant=True)
+WorkflowInput = TypeVar("WorkflowInput", bound=Union[BaseModel, dict[Any, Any]], contravariant=True)
+
+
+class WorkflowCaller:
+ def __init__(
+ self,
+ task_queue: str | None = None,
+ workflow_execution_timeout: timedelta | None = None,
+ run_timeout: timedelta | None = None,
+ task_timeout: timedelta | None = None,
+ retry_policy: RetryPolicy | None = None,
+ start_delay: timedelta | None = None,
+ rpc_timeout: timedelta | None = None,
+ ):
+ self.task_queue: str | None = task_queue
+ self.execution_timeout: timedelta | None = workflow_execution_timeout
+ self.run_timeout: timedelta | None = run_timeout
+ self.task_timeout: timedelta | None = task_timeout
+ self.retry_policy: RetryPolicy | None = retry_policy
+ self.start_delay: timedelta | None = start_delay
+ self.rpc_timeout: timedelta | None = rpc_timeout
+
+
+class WorkflowClass(Protocol[WorkflowInput, WorkflowOutput]):
+ @workflow.run
+ async def run(self, workflow_arg: WorkflowInput) -> WorkflowOutput: ...
+
+
+class WorkflowExecutor(WorkflowCaller, Generic[WorkflowInput, WorkflowOutput]):
+ """A class that provides methods to execute and start workflows, both top-level and child workflows."""
+
+ def __init__(
+ self,
+ temporal_client: TemporalClient | None = None,
+ should_auto_connect_temporal: bool = False,
+ worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,
+ **kwargs: Any,
+ ):
+ super().__init__(**kwargs)
+ self._temporal_client = temporal_client
+ self.should_auto_connect_temporal = should_auto_connect_temporal
+ self.worker_environment = worker_environment
+
+ log.dev(f"{self.__class__.__name__} initialized for task queue '{self.task_queue}'")
+ log.dev(f"worker_environment='{self.worker_environment}', auto_connect_temporal={self.should_auto_connect_temporal}")
+
+ @property
+ def class_name(self) -> str:
+ return self.__class__.__name__
+
+ async def temporal_client(self) -> TemporalClient:
+ """Get the temporal client, raising an error if not set."""
+ if not get_config().temporal.is_enabled:
+ msg = "Temporal is not enabled. Set temporal.is_enabled = true in pipelex.toml or use --temporal CLI flag."
+ raise WorkflowExecutionError(msg)
+ if self._temporal_client is None:
+ if self.should_auto_connect_temporal:
+ log.debug(f"{self.class_name} auto-connecting to Temporal server")
+ self._temporal_client = await get_temporal_client(should_auto_connect=self.should_auto_connect_temporal)
+ else:
+ msg = "temporal_client is not connected and is not allowed to auto-connect"
+ raise WorkflowExecutionError(msg)
+ return self._temporal_client
+
+ def make_workflow_id(self, base_id: str) -> str:
+ workflow_id = get_temporal_manager().make_top_workflow_id(base_id=base_id)
+ log.debug(f"Top workflow_id: {workflow_id}")
+ return workflow_id
+
+ async def execute_workflow(
+ self,
+ workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]],
+ workflow_arg: WorkflowInput,
+ workflow_id: str,
+ ) -> WorkflowOutput:
+ """Execute a workflow and wait for its completion."""
+ try:
+ client = await self.temporal_client()
+ return await cast(
+ "Callable[..., Coroutine[Any, Any, WorkflowOutput]]",
+ client.execute_workflow, # pyright: ignore[reportUnknownMemberType]
+ )(
+ workflow=workflow_class.run,
+ arg=workflow_arg,
+ id=workflow_id,
+ task_queue=self.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ run_timeout=self.run_timeout,
+ task_timeout=self.task_timeout,
+ start_delay=self.start_delay,
+ rpc_timeout=self.rpc_timeout,
+ )
+ except Exception as exc:
+ # TODO: wip - do not catch all exceptions
+ log.error(f"Failed to execute workflow {workflow_class.__name__}: {exc}")
+ msg = f"Failed to execute workflow {workflow_class.__name__}"
+ raise WorkflowExecutionError(msg) from exc
+
+ async def start_workflow(
+ self,
+ workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]],
+ workflow_arg: WorkflowInput,
+ workflow_id: str,
+ callbacks: Sequence[Callback] | None = None,
+ ) -> WorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]:
+ """Start a workflow without waiting for its completion."""
+ try:
+ client = await self.temporal_client()
+ return await cast(
+ "Callable[..., Coroutine[Any, Any, WorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]]]",
+ client.start_workflow, # pyright: ignore[reportUnknownMemberType]
+ )(
+ workflow=workflow_class.run,
+ arg=workflow_arg,
+ id=workflow_id,
+ task_queue=self.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ run_timeout=self.run_timeout,
+ task_timeout=self.task_timeout,
+ start_delay=self.start_delay,
+ rpc_timeout=self.rpc_timeout,
+ callbacks=callbacks or [],
+ )
+ except Exception as exc:
+ log.error(f"Failed to start workflow {workflow_class.__name__}: {exc}")
+ msg = f"Failed to start workflow {workflow_class.__name__}"
+ raise WorkflowExecutionError(msg) from exc
+
+ async def execute_child_workflow(
+ self,
+ workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]],
+ workflow_arg: WorkflowInput,
+ workflow_id: str,
+ child_task_queue: str | None = None,
+ ) -> WorkflowOutput:
+ """Execute a child workflow and wait for its completion."""
+ try:
+ return await cast(
+ "Callable[..., Coroutine[Any, Any, WorkflowOutput]]",
+ workflow.execute_child_workflow, # pyright: ignore[reportUnknownMemberType]
+ )(
+ workflow=workflow_class.run,
+ arg=workflow_arg,
+ id=workflow_id,
+ task_queue=child_task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ run_timeout=self.run_timeout,
+ task_timeout=self.task_timeout,
+ )
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError in {workflow_class.__name__} caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ msg = f"Application error in child workflow {workflow_class.__name__}"
+ raise WorkflowExecutionError(msg) from exc.cause
+ msg = f"Failed to execute child workflow {workflow_class.__name__}"
+ raise WorkflowExecutionError(msg) from exc
+ except Exception as exc:
+ log.error(f"Failed to execute child workflow {workflow_class.__name__}: {exc}")
+ msg = f"Failed to execute child workflow {workflow_class.__name__}"
+ raise WorkflowExecutionError(msg) from exc
+
+ async def start_child_workflow(
+ self,
+ workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]],
+ workflow_arg: WorkflowInput,
+ workflow_id: str,
+ child_task_queue: str | None = None,
+ ) -> ChildWorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]:
+ """Start a child workflow without waiting for its completion."""
+ try:
+ return await cast(
+ "Callable[..., Coroutine[Any, Any, ChildWorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]]]",
+ workflow.start_child_workflow, # pyright: ignore[reportUnknownMemberType]
+ )(
+ workflow=workflow_class.run,
+ arg=workflow_arg,
+ id=workflow_id,
+ task_queue=child_task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ run_timeout=self.run_timeout,
+ task_timeout=self.task_timeout,
+ )
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError in {workflow_class.__name__} caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ msg = f"Application error in child workflow {workflow_class.__name__}"
+ raise WorkflowExecutionError(msg) from exc.cause
+ msg = f"Failed to start child workflow {workflow_class.__name__}"
+ raise WorkflowExecutionError(msg) from exc
+ except Exception as exc:
+ log.error(f"Failed to start child workflow {workflow_class.__name__}: {exc}")
+ msg = f"Failed to start child workflow {workflow_class.__name__}"
+ raise WorkflowExecutionError(msg) from exc
+
+
+class WorkflowExecutorFactory(Generic[WorkflowInput, WorkflowOutput]):
+ @classmethod
+ def create_executor(
+ cls,
+ task_queue: str | None = None,
+ workflow_execution_timeout: timedelta | None = None,
+ retry_policy: RetryPolicy | None = None,
+ run_timeout: timedelta | None = None,
+ task_timeout: timedelta | None = None,
+ start_delay: timedelta | None = None,
+ rpc_timeout: timedelta | None = None,
+ temporal_client: TemporalClient | None = None,
+ should_auto_connect_temporal: bool = False,
+ worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,
+ ) -> WorkflowExecutor[WorkflowInput, WorkflowOutput]:
+ """Creates a WorkflowExecutor with configuration from pipelex.temporal's config if not provided."""
+ config = get_config().temporal.worker_config
+
+ return WorkflowExecutor[WorkflowInput, WorkflowOutput](
+ task_queue=task_queue or config.task_queue,
+ workflow_execution_timeout=workflow_execution_timeout or config.workflow_execution_timeout,
+ retry_policy=retry_policy or config.retry_policy,
+ run_timeout=run_timeout or config.run_timeout,
+ task_timeout=task_timeout or config.task_timeout,
+ start_delay=start_delay or config.start_delay,
+ rpc_timeout=rpc_timeout or config.rpc_timeout,
+ temporal_client=temporal_client,
+ should_auto_connect_temporal=should_auto_connect_temporal,
+ worker_environment=worker_environment,
+ )
diff --git a/pipelex/temporal/tprl_content_generation/__init__.py b/pipelex/temporal/tprl_content_generation/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/temporal/tprl_content_generation/act_extract_generate.py b/pipelex/temporal/tprl_content_generation/act_extract_generate.py
new file mode 100644
index 000000000..d0cdfcf7e
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/act_extract_generate.py
@@ -0,0 +1,24 @@
+from temporalio import activity
+
+from pipelex import log
+from pipelex.cogt.content_generation.assignment_models import ExtractAssignment
+from pipelex.cogt.content_generation.extract_generate import extract_gen_pages_and_store
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.core.stuffs.page_content import PageContent
+from pipelex.hub import get_storage_provider
+
+
+async def act_extract_gen_extract_pages(extract_assignment: ExtractAssignment) -> list[PageContent]:
+ """Extract pages and store extracted images, returning lightweight PageContent references.
+
+ Large binary data (extracted images) is stored within the activity and never crosses
+ the Temporal workflow boundary — only URLs are returned.
+ """
+ log.dev("act_extract_gen_extract_pages")
+ storage_provider = get_storage_provider()
+ generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)
+ return await extract_gen_pages_and_store(
+ extract_assignment=extract_assignment,
+ generated_content_factory=generated_content_factory,
+ )
diff --git a/pipelex/temporal/tprl_content_generation/act_img_gen_generate.py b/pipelex/temporal/tprl_content_generation/act_img_gen_generate.py
new file mode 100644
index 000000000..3ba484e9b
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/act_img_gen_generate.py
@@ -0,0 +1,30 @@
+from temporalio import activity
+
+from pipelex import log
+from pipelex.cogt.content_generation.assignment_models import ImgGenAssignment
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.cogt.content_generation.img_gen_generate import img_gen_image_list_and_store, img_gen_single_image_and_store
+from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.hub import get_storage_provider
+
+
+async def act_img_gen_images(img_gen_assignment: ImgGenAssignment) -> list[ImageContent]:
+ """Generate images and store them, returning lightweight ImageContent references.
+
+ Large binary data (base64/bytes) is stored within the activity and never crosses
+ the Temporal workflow boundary — only URLs are returned.
+ """
+ log.dev("act_img_gen_images")
+ storage_provider = get_storage_provider()
+ generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)
+ if img_gen_assignment.nb_images > 1:
+ return await img_gen_image_list_and_store(
+ img_gen_assignment=img_gen_assignment,
+ generated_content_factory=generated_content_factory,
+ )
+ image_content = await img_gen_single_image_and_store(
+ img_gen_assignment=img_gen_assignment,
+ generated_content_factory=generated_content_factory,
+ )
+ return [image_content]
diff --git a/pipelex/temporal/tprl_content_generation/act_jinja2_generate.py b/pipelex/temporal/tprl_content_generation/act_jinja2_generate.py
new file mode 100644
index 000000000..fc594d8c3
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/act_jinja2_generate.py
@@ -0,0 +1,11 @@
+from temporalio import activity
+
+from pipelex import log
+from pipelex.cogt.content_generation.assignment_models import TemplatingAssignment
+from pipelex.cogt.content_generation.templating_generate import templating_gen_text
+
+
+async def act_jinja2_gen_text(jinja2_assignment: TemplatingAssignment) -> str:
+ log.dev("act_jinja2_gen_text")
+ return await templating_gen_text(templating_assignment=jinja2_assignment)
diff --git a/pipelex/temporal/tprl_content_generation/act_llm_generate.py b/pipelex/temporal/tprl_content_generation/act_llm_generate.py
new file mode 100644
index 000000000..22f608a37
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/act_llm_generate.py
@@ -0,0 +1,24 @@
+from pydantic import BaseModel
+from temporalio import activity
+
+from pipelex import log
+from pipelex.cogt.content_generation.assignment_models import LLMAssignment, ObjectAssignment
+from pipelex.cogt.content_generation.llm_generate import llm_gen_object, llm_gen_object_list, llm_gen_text
+
+
+async def act_llm_gen_text(llm_assignment: LLMAssignment) -> str:
+ log.dev("act_llm_gen_text")
+ return await llm_gen_text(llm_assignment=llm_assignment)
+
+
+async def act_llm_gen_object(object_assignment: ObjectAssignment) -> BaseModel:
+ log.dev("act_llm_gen_object")
+ return await llm_gen_object(object_assignment=object_assignment)
+
+
+async def act_llm_gen_object_list(object_assignment: ObjectAssignment) -> list[BaseModel]:
+ log.dev("act_llm_gen_object_list")
+ return await llm_gen_object_list(object_assignment=object_assignment)
diff --git a/pipelex/temporal/tprl_content_generation/act_render_page_views.py b/pipelex/temporal/tprl_content_generation/act_render_page_views.py
new file mode 100644
index 000000000..636dde4e7
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/act_render_page_views.py
@@ -0,0 +1,39 @@
+from temporalio import activity
+
+from pipelex import log
+from pipelex.cogt.content_generation.assignment_models import RenderPageViewsAssignment
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
+from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.hub import get_storage_provider
+from pipelex.tools.misc.image_utils import ImageFormat
+from pipelex.tools.pdf.pypdfium2_renderer import pypdfium2_renderer
+
+
+async def act_render_page_views(render_assignment: RenderPageViewsAssignment) -> list[ImageContent]:
+ """Render PDF pages as images, store them, and return lightweight ImageContent references.
+
+ Performs pypdfium2 rendering and S3 storage within the activity so that
+ large binary data never crosses a Temporal workflow boundary.
+ """
+ log.dev("act_render_page_views")
+ page_view_images = await pypdfium2_renderer.render_pdf_pages_from_uri(
+ pdf_uri=render_assignment.document_uri,
+ dpi=render_assignment.page_views_dpi,
+ )
+ storage_provider = get_storage_provider()
+ generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)
+ image_contents: list[ImageContent] = []
+ for page_view_image in page_view_images:
+ raw_details = GeneratedImageRawDetails.make_from_pil_image(
+ pil_image=page_view_image,
+ image_format=ImageFormat.PNG,
+ )
+ image_content = await generated_content_factory.make_image_content(
+ primary_id=render_assignment.job_metadata.user_id,
+ secondary_id=render_assignment.job_metadata.pipeline_run_id,
+ raw_details=raw_details,
+ )
+ image_contents.append(image_content)
+ return image_contents
diff --git a/pipelex/temporal/tprl_content_generation/content_generator_child.py b/pipelex/temporal/tprl_content_generation/content_generator_child.py
new file mode 100644
index 000000000..159f4f75a
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/content_generator_child.py
@@ -0,0 +1,532 @@
+from typing import Any
+
+from pydantic import BaseModel
+from temporalio import workflow
+from temporalio.exceptions import ApplicationError, ChildWorkflowError
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.base_exceptions import PipelexError
+from pipelex.cogt.content_generation.assignment_models import (
+ ExtractAssignment,
+ ImgGenAssignment,
+ LLMAssignment,
+ LLMAssignmentFactory,
+ ObjectAssignment,
+ RenderPageViewsAssignment,
+ TemplatingAssignment,
+ TextThenObjectAssignment,
+)
+from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.cogt.extract.extract_input import ExtractInput
+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
+from pipelex.cogt.extract.extract_output import ExtractOutput
+from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
+from pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParams
+from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
+from pipelex.cogt.llm.llm_prompt_template import LLMPromptTemplate
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.cogt.templating.templating_style import TemplatingStyle
+from pipelex.config import get_config
+from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.core.stuffs.page_content import PageContent
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.system.runtime import WorkerMode, runtime_manager
+from pipelex.temporal.exceptions import ContentGenerationError
+from pipelex.temporal.tprl.temporal_error import TemporalError
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor, WorkflowExecutorFactory
+from pipelex.temporal.tprl_content_generation.content_generator_models import AssignmentType, ResultType
+from pipelex.temporal.tprl_content_generation.wf_make_extract import WfMakeExtract
+from pipelex.temporal.tprl_content_generation.wf_make_images import WfMakeImages
+from pipelex.temporal.tprl_content_generation.wf_make_jinja2_text import WfMakeJinja2Text
+from pipelex.temporal.tprl_content_generation.wf_make_llm_text import WfMakeLLMText
+from pipelex.temporal.tprl_content_generation.wf_make_object import (
+ WfMakeObject,
+ WfMakeObjectList,
+ WfMakeTextThenObject,
+ WfMakeTextThenObjectList,
+)
+from pipelex.temporal.tprl_content_generation.wf_render_page_views import WfRenderPageViews
+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+
+
+def make_child_workflow_id(base_id: str) -> str:
+ prefix: str
+ if worker_mode := runtime_manager.worker_mode:
+ match worker_mode:
+ case WorkerMode.UNIT_TEST:
+ prefix = "utw-"
+ case WorkerMode.NORMAL:
+ prefix = ""
+ else:
+ prefix = ""
+
+ info = workflow.info()
+ workflow_id = f"{prefix}{info.workflow_id}-{base_id}"
+ log.debug(f"Child workflow_id: {workflow_id}")
+ return workflow_id
+
+
+class ContentGeneratorChild(WorkflowExecutor[AssignmentType, ResultType], ContentGeneratorProtocol):
+ def __init__(self, generated_content_factory: GeneratedContentFactory, **kwargs: Any) -> None:
+ super().__init__(**kwargs)
+ self._generated_content_factory = generated_content_factory
+
+ @override
+ @update_job_metadata
+ async def make_llm_text(
+ self,
+ job_metadata: JobMetadata,
+ llm_setting_main: LLMSetting,
+ llm_prompt_for_text: LLMPrompt,
+ wfid: str | None = None,
+ ) -> str:
+ log.debug(f"ContentGeneratorChild make_llm_text: {llm_prompt_for_text}")
+ log.verbose(f"llm_setting_main: {llm_setting_main}")
+ try:
+ llm_assignment = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_main,
+ llm_prompt=llm_prompt_for_text,
+ )
+ log.verbose(llm_assignment.desc, title="llm_assignment")
+
+ generated_text = (
+ await WorkflowExecutorFactory[LLMAssignment, str]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfMakeLLMText,
+ workflow_arg=llm_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "craft-text"),
+ )
+ )
+
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorChild generated text: {generated_text}")
+ return generated_text
+
+ @override
+ @update_job_metadata
+ async def make_object_direct(
+ self,
+ job_metadata: JobMetadata,
+ object_class: type[BaseModelTypeVar],
+ llm_setting_for_object: LLMSetting,
+ llm_prompt_for_object: LLMPrompt,
+ wfid: str | None = None,
+ ) -> BaseModelTypeVar:
+ log.verbose(f"ContentGeneratorChild make_object_direct: {llm_prompt_for_object}")
+ try:
+ llm_assignment_for_object = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_for_object,
+ llm_prompt=llm_prompt_for_object,
+ )
+ object_assignment = ObjectAssignment.make_for_class(
+ object_class=object_class,
+ llm_assignment=llm_assignment_for_object,
+ )
+
+ obj = (
+ await WorkflowExecutorFactory[ObjectAssignment, BaseModel]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfMakeObject,
+ workflow_arg=object_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "craft-object-direct"),
+ )
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorChild generated object direct: {obj}")
+ return object_class.model_validate(obj.model_dump(mode="json", serialize_as_any=True))
+
+ @override
+ @update_job_metadata
+ async def make_text_then_object(
+ self,
+ job_metadata: JobMetadata,
+ object_class: type[BaseModelTypeVar],
+ llm_setting_main: LLMSetting,
+ llm_setting_for_object: LLMSetting,
+ llm_prompt_for_text: LLMPrompt,
+ llm_prompt_factory_for_object: LLMPromptFactoryAbstract | None = None,
+ wfid: str | None = None,
+ ) -> BaseModelTypeVar:
+ try:
+ llm_assignment_for_text = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_main,
+ llm_prompt=llm_prompt_for_text,
+ )
+
+ llm_assignment_factory_to_object = LLMAssignmentFactory(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_for_object,
+ llm_prompt_factory=llm_prompt_factory_for_object or LLMPromptTemplate.make_for_structuring_from_preliminary_text(),
+ )
+ tto_assignment = TextThenObjectAssignment(
+ object_class_name=object_class.__name__,
+ object_class_schema=object_class.model_json_schema(),
+ llm_assignment_for_text=llm_assignment_for_text,
+ llm_assignment_factory_to_object=llm_assignment_factory_to_object,
+ )
+
+ obj = (
+ await WorkflowExecutorFactory[TextThenObjectAssignment, BaseModel]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfMakeTextThenObject,
+ workflow_arg=tto_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "craft-text-then-object"),
+ )
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorChild generated object after text: {obj}")
+ return object_class.model_validate(obj.model_dump(mode="json", serialize_as_any=True))
+
+ @override
+ @update_job_metadata
+ async def make_object_list_direct(
+ self,
+ job_metadata: JobMetadata,
+ object_class: type[BaseModelTypeVar],
+ llm_setting_for_object_list: LLMSetting,
+ llm_prompt_for_object_list: LLMPrompt,
+ nb_items: int | None = None,
+ wfid: str | None = None,
+ ) -> list[BaseModelTypeVar]:
+ try:
+ llm_assignment_for_object = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_for_object_list,
+ llm_prompt=llm_prompt_for_object_list,
+ )
+ object_assignment = ObjectAssignment.make_for_class(
+ object_class=object_class,
+ llm_assignment=llm_assignment_for_object,
+ )
+
+ obj_list = (
+ await WorkflowExecutorFactory[ObjectAssignment, list[BaseModel]]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfMakeObjectList,
+ workflow_arg=object_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "craft-object-list-direct"),
+ )
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorChild generated object list direct: {obj_list}")
+ return [object_class.model_validate(raw_obj.model_dump(serialize_as_any=True)) for raw_obj in obj_list]
+
+ @override
+ @update_job_metadata
+ async def make_text_then_object_list(
+ self,
+ job_metadata: JobMetadata,
+ object_class: type[BaseModelTypeVar],
+ llm_setting_main: LLMSetting,
+ llm_setting_for_object_list: LLMSetting,
+ llm_prompt_for_text: LLMPrompt,
+ llm_prompt_factory_for_object_list: LLMPromptFactoryAbstract | None = None,
+ nb_items: int | None = None,
+ wfid: str | None = None,
+ ) -> list[BaseModelTypeVar]:
+ try:
+ llm_assignment_for_text = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_main,
+ llm_prompt=llm_prompt_for_text,
+ )
+
+ llm_assignment_factory_to_object = LLMAssignmentFactory(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_for_object_list,
+ llm_prompt_factory=llm_prompt_factory_for_object_list or LLMPromptTemplate.make_for_structuring_from_preliminary_text(),
+ )
+ tto_assignment = TextThenObjectAssignment(
+ object_class_name=object_class.__name__,
+ object_class_schema=object_class.model_json_schema(),
+ llm_assignment_for_text=llm_assignment_for_text,
+ llm_assignment_factory_to_object=llm_assignment_factory_to_object,
+ )
+
+ obj_list = (
+ await WorkflowExecutorFactory[TextThenObjectAssignment, list[BaseModel]]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfMakeTextThenObjectList,
+ workflow_arg=tto_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "craft-text-then-object-list"),
+ )
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorChild generated object list after text: {obj_list}")
+ return [object_class.model_validate(raw_obj.model_dump(serialize_as_any=True)) for raw_obj in obj_list]
+
+ @override
+ async def make_image_content(
+ self,
+ job_metadata: JobMetadata,
+ generated_image_raw_details: GeneratedImageRawDetails,
+ img_gen_prompt: ImgGenPrompt | None,
+ ) -> ImageContent:
+ image_content = await self._generated_content_factory.make_image_content(
+ primary_id=job_metadata.user_id,
+ secondary_id=job_metadata.pipeline_run_id,
+ raw_details=generated_image_raw_details,
+ )
+ if img_gen_prompt:
+ image_content.source_prompt = img_gen_prompt.positive_text
+ image_content.source_negative_prompt = img_gen_prompt.negative_text
+ return image_content
+
+ @override
+ async def make_page_contents(
+ self,
+ job_metadata: JobMetadata,
+ extract_output: ExtractOutput,
+ ) -> list[PageContent]:
+ return await self._generated_content_factory.make_page_contents(
+ primary_id=job_metadata.user_id,
+ secondary_id=job_metadata.pipeline_run_id,
+ extract_output=extract_output,
+ )
+
+ @override
+ @update_job_metadata
+ async def make_single_image(
+ self,
+ job_metadata: JobMetadata,
+ img_gen_handle: str,
+ img_gen_prompt: ImgGenPrompt,
+ img_gen_job_params: ImgGenJobParams | None = None,
+ img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
+ ) -> ImageContent:
+ img_gen_config = get_config().cogt.img_gen_config
+ try:
+ img_gen_assignment = ImgGenAssignment(
+ job_metadata=job_metadata,
+ img_gen_handle=img_gen_handle,
+ img_gen_prompt=img_gen_prompt,
+ img_gen_job_params=img_gen_job_params or img_gen_config.make_default_img_gen_job_params(),
+ img_gen_job_config=img_gen_job_config or img_gen_config.img_gen_job_config,
+ nb_images=1,
+ )
+
+ image_content_list = (
+ await WorkflowExecutorFactory[ImgGenAssignment, list[ImageContent]]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfMakeImages,
+ workflow_arg=img_gen_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "craft-image"),
+ )
+ )
+ if len(image_content_list) != 1:
+ msg = f"Expected 1 image, got {len(image_content_list)}"
+ raise ContentGenerationError(msg)
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ image_content = image_content_list[0]
+ log.verbose(f"ContentGeneratorChild generated image: {image_content}")
+ return image_content
+
+ @override
+ @update_job_metadata
+ async def make_image_list(
+ self,
+ job_metadata: JobMetadata,
+ img_gen_handle: str,
+ img_gen_prompt: ImgGenPrompt,
+ nb_images: int,
+ img_gen_job_params: ImgGenJobParams | None = None,
+ img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
+ ) -> list[ImageContent]:
+ img_gen_config = get_config().cogt.img_gen_config
+ try:
+ img_gen_assignment = ImgGenAssignment(
+ job_metadata=job_metadata,
+ img_gen_handle=img_gen_handle,
+ img_gen_prompt=img_gen_prompt,
+ img_gen_job_params=img_gen_job_params or img_gen_config.make_default_img_gen_job_params(),
+ img_gen_job_config=img_gen_job_config or img_gen_config.img_gen_job_config,
+ nb_images=nb_images,
+ )
+
+ image_content_list = (
+ await WorkflowExecutorFactory[ImgGenAssignment, list[ImageContent]]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfMakeImages,
+ workflow_arg=img_gen_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "craft-image"),
+ )
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorChild generated image list: {image_content_list}")
+ return image_content_list
+
+ @override
+ async def make_templated_text(
+ self,
+ job_metadata: JobMetadata,
+ context: dict[str, Any],
+ template: str,
+ templating_style: TemplatingStyle | None = None,
+ template_category: TemplateCategory | None = None,
+ wfid: str | None = None,
+ ) -> str:
+ try:
+ templating_assignment = TemplatingAssignment(
+ job_metadata=job_metadata,
+ context=context,
+ template=template,
+ templating_style=templating_style,
+ category=template_category or TemplateCategory.BASIC,
+ )
+ jinja2_text = (
+ await WorkflowExecutorFactory[TemplatingAssignment, str]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfMakeJinja2Text,
+ workflow_arg=templating_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "jinja2-text"),
+ )
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorChild templated text: {jinja2_text}")
+ return jinja2_text
+
+ @override
+ @update_job_metadata
+ async def make_render_page_views(
+ self,
+ job_metadata: JobMetadata,
+ extract_input: ExtractInput,
+ extract_handle: str,
+ extract_job_params: ExtractJobParams | None = None,
+ extract_job_config: ExtractJobConfig | None = None,
+ wfid: str | None = None,
+ ) -> list[ImageContent]:
+ if not extract_input.document_uri:
+ msg = "PDF URI is required to render page views"
+ raise ValueError(msg)
+ job_params = extract_job_params or ExtractJobParams.make_default_extract_job_params()
+ page_views_dpi = job_params.page_views_dpi or get_config().cogt.extract_config.default_page_views_dpi
+ try:
+ render_assignment = RenderPageViewsAssignment(
+ job_metadata=job_metadata,
+ document_uri=extract_input.document_uri,
+ page_views_dpi=page_views_dpi,
+ )
+ image_content_list = (
+ await WorkflowExecutorFactory[RenderPageViewsAssignment, list[ImageContent]]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfRenderPageViews,
+ workflow_arg=render_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "render-page-views"),
+ )
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorChild rendered page views: {image_content_list}")
+ return image_content_list
+
+ @override
+ @update_job_metadata
+ async def make_extract_pages(
+ self,
+ job_metadata: JobMetadata,
+ extract_input: ExtractInput,
+ extract_handle: str,
+ extract_job_params: ExtractJobParams,
+ extract_job_config: ExtractJobConfig,
+ wfid: str | None = None,
+ ) -> list[PageContent]:
+ try:
+ extract_assignment = ExtractAssignment(
+ job_metadata=job_metadata,
+ extract_handle=extract_handle,
+ extract_input=extract_input,
+ extract_job_params=extract_job_params,
+ extract_job_config=extract_job_config,
+ )
+
+ page_content_list = (
+ await WorkflowExecutorFactory[ExtractAssignment, list[PageContent]]
+ .create_executor()
+ .execute_child_workflow(
+ workflow_class=WfMakeExtract,
+ workflow_arg=extract_assignment,
+ workflow_id=make_child_workflow_id(base_id=wfid or "extract"),
+ )
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ChildWorkflowError as exc:
+ log.error(f"ChildWorkflowError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorChild generated page contents: {page_content_list}")
+ return page_content_list
diff --git a/pipelex/temporal/tprl_content_generation/content_generator_child_factory.py b/pipelex/temporal/tprl_content_generation/content_generator_child_factory.py
new file mode 100644
index 000000000..9f2f6b9a6
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/content_generator_child_factory.py
@@ -0,0 +1,34 @@
+from datetime import timedelta
+
+from temporalio.common import RetryPolicy
+
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.config import get_config
+from pipelex.temporal.tprl_content_generation.content_generator_child import ContentGeneratorChild
+
+
+class ContentGeneratorChildFactory:
+ @classmethod
+ def make_content_generator_child(
+ cls,
+ generated_content_factory: GeneratedContentFactory,
+ task_queue: str | None = None,
+ workflow_execution_timeout: timedelta | None = None,
+ retry_policy: RetryPolicy | None = None,
+ run_timeout: timedelta | None = None,
+ task_timeout: timedelta | None = None,
+ start_delay: timedelta | None = None,
+ rpc_timeout: timedelta | None = None,
+ ) -> ContentGeneratorChild:
+ config = get_config().temporal.worker_config
+
+ return ContentGeneratorChild(
+ generated_content_factory=generated_content_factory,
+ task_queue=task_queue or config.task_queue,
+ workflow_execution_timeout=workflow_execution_timeout or config.workflow_execution_timeout,
+ retry_policy=retry_policy or config.retry_policy,
+ run_timeout=run_timeout or config.run_timeout,
+ task_timeout=task_timeout or config.task_timeout,
+ start_delay=start_delay or config.start_delay,
+ rpc_timeout=rpc_timeout or config.rpc_timeout,
+ )
diff --git a/pipelex/temporal/tprl_content_generation/content_generator_models.py b/pipelex/temporal/tprl_content_generation/content_generator_models.py
new file mode 100644
index 000000000..879903b4a
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/content_generator_models.py
@@ -0,0 +1,20 @@
+from typing import Union
+
+from pydantic import BaseModel
+
+from pipelex.cogt.content_generation.assignment_models import ImgGenAssignment, LLMAssignment, ObjectAssignment, TextThenObjectAssignment
+from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
+
+AssignmentType = Union[
+ LLMAssignment,
+ ObjectAssignment,
+ TextThenObjectAssignment,
+ ImgGenAssignment,
+]
+
+ResultType = Union[
+ str,
+ BaseModel,
+ list[BaseModel],
+ GeneratedImageRawDetails,
+]
diff --git a/pipelex/temporal/tprl_content_generation/content_generator_top.py b/pipelex/temporal/tprl_content_generation/content_generator_top.py
new file mode 100644
index 000000000..9c65a9205
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/content_generator_top.py
@@ -0,0 +1,447 @@
+from typing import Any, cast
+
+from pydantic import BaseModel # noqa: TC002
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.cogt.content_generation.assignment_models import (
+ ExtractAssignment,
+ ImgGenAssignment,
+ LLMAssignment,
+ LLMAssignmentFactory,
+ ObjectAssignment,
+ RenderPageViewsAssignment,
+ TemplatingAssignment,
+ TextThenObjectAssignment,
+)
+from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.cogt.extract.extract_input import ExtractInput
+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
+from pipelex.cogt.extract.extract_output import ExtractOutput
+from pipelex.cogt.image.generated_image import GeneratedImageRawDetails
+from pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParams
+from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_prompt_factory_abstract import LLMPromptFactoryAbstract
+from pipelex.cogt.llm.llm_prompt_template import LLMPromptTemplate
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.cogt.templating.template_category import TemplateCategory
+from pipelex.cogt.templating.templating_style import TemplatingStyle
+from pipelex.config import get_config
+from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.core.stuffs.page_content import PageContent
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.temporal.exceptions import ContentGenerationError
+from pipelex.temporal.tprl.conditional_worker import with_conditional_worker
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor
+from pipelex.temporal.tprl_content_generation.content_generator_models import AssignmentType, ResultType
+from pipelex.temporal.tprl_content_generation.wf_make_extract import WfMakeExtract
+from pipelex.temporal.tprl_content_generation.wf_make_images import WfMakeImages
+from pipelex.temporal.tprl_content_generation.wf_make_jinja2_text import WfMakeJinja2Text
+from pipelex.temporal.tprl_content_generation.wf_make_llm_text import WfMakeLLMText
+from pipelex.temporal.tprl_content_generation.wf_make_object import (
+ WfMakeObject,
+ WfMakeObjectList,
+ WfMakeTextThenObject,
+ WfMakeTextThenObjectList,
+)
+from pipelex.temporal.tprl_content_generation.wf_render_page_views import WfRenderPageViews
+from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
+
+
+class ContentGeneratorTop(WorkflowExecutor[AssignmentType, ResultType], ContentGeneratorProtocol):
+ def __init__(self, generated_content_factory: GeneratedContentFactory, **kwargs: Any) -> None:
+ super().__init__(**kwargs)
+ self._generated_content_factory = generated_content_factory
+
+ @override
+ @with_conditional_worker
+ @update_job_metadata
+ async def make_llm_text(
+ self,
+ job_metadata: JobMetadata,
+ llm_setting_main: LLMSetting,
+ llm_prompt_for_text: LLMPrompt,
+ wfid: str | None = None,
+ ) -> str:
+ log.verbose(f"TopCrafter make_llm_text: {llm_prompt_for_text}")
+ log.verbose(f"llm_setting_main: {llm_setting_main}")
+ workflow_id = self.make_workflow_id(base_id=wfid or "craft-text")
+ llm_assignment = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_main,
+ llm_prompt=llm_prompt_for_text,
+ )
+ log.verbose(llm_assignment.desc, title="llm_assignment")
+ temporal_client = await self.temporal_client()
+ generated_text = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfMakeLLMText.run,
+ arg=llm_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ log.verbose(f"TopCrafter generated text: {generated_text}")
+ return generated_text
+
+ @override
+ @with_conditional_worker
+ @update_job_metadata
+ async def make_object_direct(
+ self,
+ job_metadata: JobMetadata,
+ object_class: type[BaseModelTypeVar],
+ llm_setting_for_object: LLMSetting,
+ llm_prompt_for_object: LLMPrompt,
+ wfid: str | None = None,
+ ) -> BaseModelTypeVar:
+ log.verbose(f"TopCrafter make_object_direct: {llm_prompt_for_object}")
+ workflow_id = self.make_workflow_id(base_id=wfid or "craft-object-direct")
+ llm_assignment_for_object = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_for_object,
+ llm_prompt=llm_prompt_for_object,
+ )
+ object_assignment = ObjectAssignment.make_for_class(
+ object_class=object_class,
+ llm_assignment=llm_assignment_for_object,
+ )
+ temporal_client = await self.temporal_client()
+ obj = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfMakeObject.run,
+ arg=object_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ log.verbose(f"TopCrafter generated object direct: {obj}")
+ return cast("BaseModelTypeVar", obj)
+
+ @override
+ @with_conditional_worker
+ @update_job_metadata
+ async def make_text_then_object(
+ self,
+ job_metadata: JobMetadata,
+ object_class: type[BaseModelTypeVar],
+ llm_setting_main: LLMSetting,
+ llm_setting_for_object: LLMSetting,
+ llm_prompt_for_text: LLMPrompt,
+ llm_prompt_factory_for_object: LLMPromptFactoryAbstract | None = None,
+ wfid: str | None = None,
+ ) -> BaseModelTypeVar:
+ workflow_id = self.make_workflow_id(base_id=wfid or "craft-text-then-object")
+
+ llm_assignment_for_text = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_main,
+ llm_prompt=llm_prompt_for_text,
+ )
+
+ llm_assignment_factory_to_object = LLMAssignmentFactory(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_for_object,
+ llm_prompt_factory=llm_prompt_factory_for_object or LLMPromptTemplate.make_for_structuring_from_preliminary_text(),
+ )
+ tto_assignment = TextThenObjectAssignment(
+ object_class_name=object_class.__name__,
+ object_class_schema=object_class.model_json_schema(),
+ llm_assignment_for_text=llm_assignment_for_text,
+ llm_assignment_factory_to_object=llm_assignment_factory_to_object,
+ )
+
+ temporal_client = await self.temporal_client()
+ obj = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfMakeTextThenObject.run,
+ arg=tto_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ log.verbose(f"TopCrafter generated object after text: {obj}")
+ return cast("BaseModelTypeVar", obj)
+
+ @override
+ @with_conditional_worker
+ @update_job_metadata
+ async def make_object_list_direct(
+ self,
+ job_metadata: JobMetadata,
+ object_class: type[BaseModelTypeVar],
+ llm_setting_for_object_list: LLMSetting,
+ llm_prompt_for_object_list: LLMPrompt,
+ nb_items: int | None = None,
+ wfid: str | None = None,
+ ) -> list[BaseModelTypeVar]:
+ workflow_id = self.make_workflow_id(base_id=wfid or "craft-object-list-direct")
+ llm_assignment_for_object = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_for_object_list,
+ llm_prompt=llm_prompt_for_object_list,
+ )
+ object_assignment = ObjectAssignment.make_for_class(
+ object_class=object_class,
+ llm_assignment=llm_assignment_for_object,
+ )
+ temporal_client = await self.temporal_client()
+ obj_list: list[BaseModel] = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfMakeObjectList.run,
+ arg=object_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ log.verbose(f"TopCrafter generated object list direct: {obj_list}")
+ return cast("list[BaseModelTypeVar]", obj_list)
+
+ @override
+ @with_conditional_worker
+ @update_job_metadata
+ async def make_text_then_object_list(
+ self,
+ job_metadata: JobMetadata,
+ object_class: type[BaseModelTypeVar],
+ llm_setting_main: LLMSetting,
+ llm_setting_for_object_list: LLMSetting,
+ llm_prompt_for_text: LLMPrompt,
+ llm_prompt_factory_for_object_list: LLMPromptFactoryAbstract | None = None,
+ nb_items: int | None = None,
+ wfid: str | None = None,
+ ) -> list[BaseModelTypeVar]:
+ workflow_id = self.make_workflow_id(base_id=wfid or "craft-text-then-object-list")
+
+ llm_assignment_for_text = LLMAssignment(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_main,
+ llm_prompt=llm_prompt_for_text,
+ )
+
+ llm_assignment_factory_to_object = LLMAssignmentFactory(
+ job_metadata=job_metadata,
+ llm_setting=llm_setting_for_object_list,
+ llm_prompt_factory=llm_prompt_factory_for_object_list or LLMPromptTemplate.make_for_structuring_from_preliminary_text(),
+ )
+ tto_assignment = TextThenObjectAssignment(
+ object_class_name=object_class.__name__,
+ object_class_schema=object_class.model_json_schema(),
+ llm_assignment_for_text=llm_assignment_for_text,
+ llm_assignment_factory_to_object=llm_assignment_factory_to_object,
+ )
+
+ temporal_client = await self.temporal_client()
+ obj_list: list[BaseModel] = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfMakeTextThenObjectList.run,
+ arg=tto_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ log.verbose(f"TopCrafter generated object list after text: {obj_list}")
+ return cast("list[BaseModelTypeVar]", obj_list)
+
+ @override
+ async def make_image_content(
+ self,
+ job_metadata: JobMetadata,
+ generated_image_raw_details: GeneratedImageRawDetails,
+ img_gen_prompt: ImgGenPrompt | None,
+ ) -> ImageContent:
+ image_content = await self._generated_content_factory.make_image_content(
+ primary_id=job_metadata.user_id,
+ secondary_id=job_metadata.pipeline_run_id,
+ raw_details=generated_image_raw_details,
+ )
+ if img_gen_prompt:
+ image_content.source_prompt = img_gen_prompt.positive_text
+ image_content.source_negative_prompt = img_gen_prompt.negative_text
+ return image_content
+
+ @override
+ async def make_page_contents(
+ self,
+ job_metadata: JobMetadata,
+ extract_output: ExtractOutput,
+ ) -> list[PageContent]:
+ return await self._generated_content_factory.make_page_contents(
+ primary_id=job_metadata.user_id,
+ secondary_id=job_metadata.pipeline_run_id,
+ extract_output=extract_output,
+ )
+
+ @override
+ @with_conditional_worker
+ @update_job_metadata
+ async def make_single_image(
+ self,
+ job_metadata: JobMetadata,
+ img_gen_handle: str,
+ img_gen_prompt: ImgGenPrompt,
+ img_gen_job_params: ImgGenJobParams | None = None,
+ img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
+ ) -> ImageContent:
+ workflow_id = self.make_workflow_id(base_id=wfid or "craft-image")
+ img_gen_config = get_config().cogt.img_gen_config
+ # We're using workflowWfMakeImages which can generate several images but we're asking for only one
+ img_gen_assignment = ImgGenAssignment(
+ job_metadata=job_metadata,
+ img_gen_handle=img_gen_handle,
+ img_gen_prompt=img_gen_prompt,
+ img_gen_job_params=img_gen_job_params or img_gen_config.make_default_img_gen_job_params(),
+ img_gen_job_config=img_gen_job_config or img_gen_config.img_gen_job_config,
+ nb_images=1,
+ )
+ temporal_client = await self.temporal_client()
+ image_content_list = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfMakeImages.run,
+ arg=img_gen_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ if len(image_content_list) != 1:
+ msg = f"Expected 1 image, got {len(image_content_list)}"
+ raise ContentGenerationError(msg)
+ image_content = image_content_list[0]
+ log.dev(f"TopCrafter generated image: {image_content}")
+ return image_content
+
+ @override
+ @with_conditional_worker
+ @update_job_metadata
+ async def make_image_list(
+ self,
+ job_metadata: JobMetadata,
+ img_gen_handle: str,
+ img_gen_prompt: ImgGenPrompt,
+ nb_images: int,
+ img_gen_job_params: ImgGenJobParams | None = None,
+ img_gen_job_config: ImgGenJobConfig | None = None,
+ wfid: str | None = None,
+ ) -> list[ImageContent]:
+ workflow_id = self.make_workflow_id(base_id=wfid or "craft-image")
+ img_gen_config = get_config().cogt.img_gen_config
+ img_gen_assignment = ImgGenAssignment(
+ job_metadata=job_metadata,
+ img_gen_handle=img_gen_handle,
+ img_gen_prompt=img_gen_prompt,
+ img_gen_job_params=img_gen_job_params or img_gen_config.make_default_img_gen_job_params(),
+ img_gen_job_config=img_gen_job_config or img_gen_config.img_gen_job_config,
+ nb_images=nb_images,
+ )
+ temporal_client = await self.temporal_client()
+ image_content_list = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfMakeImages.run,
+ arg=img_gen_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ log.dev(f"TopCrafter generated image list: {image_content_list}")
+ return image_content_list
+
+ @override
+ @with_conditional_worker
+ async def make_templated_text(
+ self,
+ job_metadata: JobMetadata,
+ context: dict[str, Any],
+ template: str,
+ templating_style: TemplatingStyle | None = None,
+ template_category: TemplateCategory | None = None,
+ wfid: str | None = None,
+ ) -> str:
+ workflow_id = self.make_workflow_id(base_id=wfid or "jinja2-text")
+ templating_assignment = TemplatingAssignment(
+ job_metadata=job_metadata,
+ context=context,
+ template=template,
+ templating_style=templating_style,
+ category=template_category or TemplateCategory.BASIC,
+ )
+ temporal_client = await self.temporal_client()
+ jinja2_text = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ WfMakeJinja2Text.run,
+ arg=templating_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ log.dev(f"TopCrafter templated text: {jinja2_text}")
+ return jinja2_text
+
+ @override
+ @with_conditional_worker
+ @update_job_metadata
+ async def make_render_page_views(
+ self,
+ job_metadata: JobMetadata,
+ extract_input: ExtractInput,
+ extract_handle: str,
+ extract_job_params: ExtractJobParams | None = None,
+ extract_job_config: ExtractJobConfig | None = None,
+ wfid: str | None = None,
+ ) -> list[ImageContent]:
+ if not extract_input.document_uri:
+ msg = "PDF URI is required to render page views"
+ raise ValueError(msg)
+ job_params = extract_job_params or ExtractJobParams.make_default_extract_job_params()
+ page_views_dpi = job_params.page_views_dpi or get_config().cogt.extract_config.default_page_views_dpi
+ workflow_id = self.make_workflow_id(base_id=wfid or "render-page-views")
+ render_assignment = RenderPageViewsAssignment(
+ job_metadata=job_metadata,
+ document_uri=extract_input.document_uri,
+ page_views_dpi=page_views_dpi,
+ )
+ temporal_client = await self.temporal_client()
+ image_content_list = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfRenderPageViews.run,
+ arg=render_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ log.dev(f"TopCrafter rendered page views: {image_content_list}")
+ return image_content_list
+
+ @override
+ @with_conditional_worker
+ @update_job_metadata
+ async def make_extract_pages(
+ self,
+ job_metadata: JobMetadata,
+ extract_input: ExtractInput,
+ extract_handle: str,
+ extract_job_params: ExtractJobParams,
+ extract_job_config: ExtractJobConfig,
+ wfid: str | None = None,
+ ) -> list[PageContent]:
+ workflow_id = self.make_workflow_id(base_id=wfid or "extract")
+ extract_assignment = ExtractAssignment(
+ job_metadata=job_metadata,
+ extract_handle=extract_handle,
+ extract_input=extract_input,
+ extract_job_params=extract_job_params,
+ extract_job_config=extract_job_config,
+ )
+ temporal_client = await self.temporal_client()
+ page_content_list = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfMakeExtract.run,
+ arg=extract_assignment,
+ id=workflow_id,
+ task_queue=self.task_queue or get_config().temporal.worker_config.task_queue,
+ execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ )
+ log.dev(f"TopCrafter page contents: {page_content_list}")
+ return page_content_list
diff --git a/pipelex/temporal/tprl_content_generation/content_generator_top_factory.py b/pipelex/temporal/tprl_content_generation/content_generator_top_factory.py
new file mode 100644
index 000000000..6d24aba24
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/content_generator_top_factory.py
@@ -0,0 +1,55 @@
+from datetime import timedelta
+
+from temporalio.client import Client as TemporalClient
+from temporalio.common import RetryPolicy
+
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.config import get_config
+from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment
+from pipelex.temporal.tprl_content_generation.content_generator_top import ContentGeneratorTop
+
+
+class ContentGeneratorTopFactory:
+ @classmethod
+ def make_content_generator_top(
+ cls,
+ generated_content_factory: GeneratedContentFactory,
+ workflow_execution_timeout: timedelta | None = None,
+ retry_policy: RetryPolicy | None = None,
+ task_queue: str | None = None,
+ worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,
+ temporal_client: TemporalClient | None = None,
+ ) -> ContentGeneratorTop:
+ """This factory is only passing your settings or using defaults from pipelex.temporal's config.
+ Don't hesitate to create your own factory or your own TopCrafter according to your needs and context.
+ """
+ worker_config = get_config().temporal.worker_config
+ return ContentGeneratorTop(
+ generated_content_factory=generated_content_factory,
+ task_queue=task_queue or worker_config.task_queue,
+ workflow_execution_timeout=workflow_execution_timeout or worker_config.workflow_execution_timeout,
+ retry_policy=retry_policy or worker_config.retry_policy_config.make_retry_policy(),
+ temporal_client=temporal_client,
+ should_auto_connect_temporal=temporal_client is None,
+ worker_environment=worker_environment,
+ )
+
+ # @classmethod
+ # async def make_content_generator_top_connected_to_temporal(
+ # cls,
+ # task_queue: Optional[str] = None,
+ # workflow_execution_timeout: Optional[timedelta] = None,
+ # retry_policy: Optional[RetryPolicy] = None,
+ # worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,
+ # ) -> TopCrafterTemporal:
+ # """
+ # This factory uses above make_top_crafter but then also makes sure that the temporal client is connected.
+ # """
+ # top_crafter_temporal = cls.make_top_crafter_temporal(
+ # task_queue=task_queue,
+ # workflow_execution_timeout=workflow_execution_timeout,
+ # retry_policy=retry_policy,
+ # worker_environment=worker_environment,
+ # )
+ # await top_crafter_temporal.temporal_client()
+ # return top_crafter_temporal
diff --git a/pipelex/temporal/tprl_content_generation/wf_make_extract.py b/pipelex/temporal/tprl_content_generation/wf_make_extract.py
new file mode 100644
index 000000000..35735c61e
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/wf_make_extract.py
@@ -0,0 +1,40 @@
+from temporalio import workflow
+from temporalio.exceptions import ActivityError, ApplicationError
+from typing_extensions import override
+
+with workflow.unsafe.imports_passed_through():
+ from pipelex import log
+ from pipelex.cogt.content_generation.assignment_models import ExtractAssignment
+ from pipelex.config import get_config
+ from pipelex.core.stuffs.page_content import PageContent
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.tprl.temporal_error import TemporalError
+ from pipelex.temporal.tprl.workflow_caller import WorkflowClass
+ from pipelex.temporal.tprl_content_generation.act_extract_generate import act_extract_gen_extract_pages
+
+
[email protected](name="wf_make_extract")
+class WfMakeExtract(WorkflowClass[ExtractAssignment, list[PageContent]]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: ExtractAssignment,
+ ) -> list[PageContent]:
+ workflow_log.debug("Workflow start")
+ worker_config = get_config().temporal.worker_config
+ try:
+ page_content_list = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_extract_gen_extract_pages,
+ arg=workflow_arg,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+ except ActivityError as exc:
+ log.error(f"ActivityError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+
+ workflow_log.debug("Workflow complete")
+ return page_content_list
diff --git a/pipelex/temporal/tprl_content_generation/wf_make_images.py b/pipelex/temporal/tprl_content_generation/wf_make_images.py
new file mode 100644
index 000000000..e38dcedf4
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/wf_make_images.py
@@ -0,0 +1,40 @@
+from temporalio import workflow
+from temporalio.exceptions import ActivityError, ApplicationError
+from typing_extensions import override
+
+with workflow.unsafe.imports_passed_through():
+ from pipelex import log
+ from pipelex.cogt.content_generation.assignment_models import ImgGenAssignment
+ from pipelex.config import get_config
+ from pipelex.core.stuffs.image_content import ImageContent
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.tprl.temporal_error import TemporalError
+ from pipelex.temporal.tprl.workflow_caller import WorkflowClass
+ from pipelex.temporal.tprl_content_generation.act_img_gen_generate import act_img_gen_images
+
+
[email protected](name="wf_make_image")
+class WfMakeImages(WorkflowClass[ImgGenAssignment, list[ImageContent]]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: ImgGenAssignment,
+ ) -> list[ImageContent]:
+ workflow_log.debug("Workflow start")
+ worker_config = get_config().temporal.worker_config
+ try:
+ image_content_list = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_img_gen_images,
+ arg=workflow_arg,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+ except ActivityError as exc:
+ log.error(f"ActivityError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+
+ workflow_log.debug("Workflow complete")
+ return image_content_list
diff --git a/pipelex/temporal/tprl_content_generation/wf_make_jinja2_text.py b/pipelex/temporal/tprl_content_generation/wf_make_jinja2_text.py
new file mode 100644
index 000000000..813d5f947
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/wf_make_jinja2_text.py
@@ -0,0 +1,40 @@
+from temporalio import workflow
+from temporalio.exceptions import ActivityError, ApplicationError
+from typing_extensions import override
+
+with workflow.unsafe.imports_passed_through():
+ from pipelex.cogt.content_generation.assignment_models import (
+ TemplatingAssignment,
+ )
+ from pipelex.config import get_config
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.tprl.temporal_error import TemporalError
+ from pipelex.temporal.tprl.workflow_caller import WorkflowClass
+ from pipelex.temporal.tprl_content_generation.act_jinja2_generate import (
+ act_jinja2_gen_text,
+ )
+
+
[email protected](name="wf_make_jinja2_text")
+class WfMakeJinja2Text(WorkflowClass[TemplatingAssignment, str]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: TemplatingAssignment,
+ ) -> str:
+ workflow_log.debug("Workflow start")
+ worker_config = get_config().temporal.worker_config
+ try:
+ jinja2_text: str = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_jinja2_gen_text,
+ arg=workflow_arg,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+ except ActivityError as exc:
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ workflow_log.debug("Workflow complete")
+ return jinja2_text
diff --git a/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py b/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py
new file mode 100644
index 000000000..f635e5bed
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/wf_make_llm_text.py
@@ -0,0 +1,36 @@
+from temporalio import workflow
+from temporalio.exceptions import ActivityError, ApplicationError
+from typing_extensions import override
+
+with workflow.unsafe.imports_passed_through():
+ from pipelex.cogt.content_generation.assignment_models import LLMAssignment
+ from pipelex.config import get_config
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.tprl.temporal_error import TemporalError
+ from pipelex.temporal.tprl.workflow_caller import WorkflowClass
+ from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text
+
+
[email protected](name="wf_make_llm_text")
+class WfMakeLLMText(WorkflowClass[LLMAssignment, str]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: LLMAssignment,
+ ) -> str:
+ workflow_log.debug("Workflow start")
+ worker_config = get_config().temporal.worker_config
+ try:
+ crafted_text = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_llm_gen_text,
+ arg=workflow_arg,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+ except ActivityError as exc:
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ workflow_log.debug("Workflow complete")
+ return crafted_text
diff --git a/pipelex/temporal/tprl_content_generation/wf_make_object.py b/pipelex/temporal/tprl_content_generation/wf_make_object.py
new file mode 100644
index 000000000..8424970fe
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/wf_make_object.py
@@ -0,0 +1,169 @@
+from temporalio import workflow
+from temporalio.exceptions import ActivityError, ApplicationError
+from typing_extensions import override
+
+with workflow.unsafe.imports_passed_through():
+ from pydantic import BaseModel
+
+ from pipelex.base_exceptions import PipelexError
+ from pipelex.cogt.content_generation.assignment_models import (
+ ObjectAssignment,
+ TextThenObjectAssignment,
+ )
+ from pipelex.config import get_config
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.tprl.temporal_error import TemporalError
+ from pipelex.temporal.tprl.workflow_caller import WorkflowClass
+ from pipelex.temporal.tprl_content_generation.act_llm_generate import (
+ act_llm_gen_object,
+ act_llm_gen_object_list,
+ act_llm_gen_text,
+ )
+
+
[email protected](name="wf_make_object")
+class WfMakeObject(WorkflowClass[ObjectAssignment, BaseModel]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: ObjectAssignment,
+ ) -> BaseModel:
+ workflow_log.debug("Workflow start")
+ worker_config = get_config().temporal.worker_config
+ try:
+ obj = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_llm_gen_object,
+ arg=workflow_arg,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+ except ActivityError as exc:
+ workflow_log.error(f"ActivityError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ workflow_log.debug("Workflow complete")
+ return obj
+
+
[email protected](name="wf_make_object_list")
+class WfMakeObjectList(WorkflowClass[ObjectAssignment, list[BaseModel]]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: ObjectAssignment,
+ ) -> list[BaseModel]:
+ workflow_log.debug("Workflow start")
+ worker_config = get_config().temporal.worker_config
+ try:
+ obj_list: list[BaseModel] = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_llm_gen_object_list,
+ arg=workflow_arg,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+ except ActivityError as exc:
+ workflow_log.error(f"ActivityError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ workflow_log.debug("Workflow complete")
+ return obj_list
+
+
[email protected](name="wf_make_text_then_object")
+class WfMakeTextThenObject(WorkflowClass[TextThenObjectAssignment, BaseModel]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: TextThenObjectAssignment,
+ ) -> BaseModel:
+ workflow_log.debug("Workflow start")
+ worker_config = get_config().temporal.worker_config
+ try:
+ preliminary_text = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_llm_gen_text,
+ arg=workflow_arg.llm_assignment_for_text,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+
+ workflow_log.dev(f"preliminary_text: {preliminary_text}")
+
+ fup_llm_assignment = await workflow_arg.llm_assignment_factory_to_object.make_llm_assignment(
+ preliminary_text=preliminary_text,
+ )
+
+ fup_obj_assignment = ObjectAssignment(
+ llm_assignment_for_object=fup_llm_assignment,
+ object_class_name=workflow_arg.object_class_name,
+ object_class_schema=workflow_arg.object_class_schema,
+ )
+
+ obj: BaseModel = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_llm_gen_object,
+ arg=fup_obj_assignment,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ActivityError as exc:
+ workflow_log.error(f"ActivityError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ workflow_log.debug("Workflow complete")
+ return obj
+
+
[email protected](name="wf_make_text_then_object_list")
+class WfMakeTextThenObjectList(WorkflowClass[TextThenObjectAssignment, list[BaseModel]]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: TextThenObjectAssignment,
+ ) -> list[BaseModel]:
+ workflow_log.debug("Workflow start")
+ worker_config = get_config().temporal.worker_config
+ try:
+ preliminary_text = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_llm_gen_text,
+ arg=workflow_arg.llm_assignment_for_text,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+
+ workflow_log.debug(f"preliminary_text: {preliminary_text}")
+
+ llm_assignment_for_object = await workflow_arg.llm_assignment_factory_to_object.make_llm_assignment(
+ preliminary_text=preliminary_text,
+ )
+
+ object_assignment = ObjectAssignment(
+ object_class_name=workflow_arg.object_class_name,
+ object_class_schema=workflow_arg.object_class_schema,
+ llm_assignment_for_object=llm_assignment_for_object,
+ )
+
+ obj_list: list[BaseModel] = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_llm_gen_object_list,
+ arg=object_assignment,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+ except PipelexError as exc:
+ raise TemporalError.from_message_exception(exc) from exc
+ except ActivityError as exc:
+ workflow_log.error(f"ActivityError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+
+ workflow_log.debug(f"obj_list: {obj_list}")
+ workflow_log.debug("Workflow complete")
+ return obj_list
diff --git a/pipelex/temporal/tprl_content_generation/wf_render_page_views.py b/pipelex/temporal/tprl_content_generation/wf_render_page_views.py
new file mode 100644
index 000000000..88e803bf0
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/wf_render_page_views.py
@@ -0,0 +1,40 @@
+from temporalio import workflow
+from temporalio.exceptions import ActivityError, ApplicationError
+from typing_extensions import override
+
+with workflow.unsafe.imports_passed_through():
+ from pipelex import log
+ from pipelex.cogt.content_generation.assignment_models import RenderPageViewsAssignment
+ from pipelex.config import get_config
+ from pipelex.core.stuffs.image_content import ImageContent
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.tprl.temporal_error import TemporalError
+ from pipelex.temporal.tprl.workflow_caller import WorkflowClass
+ from pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views
+
+
[email protected](name="wf_render_page_views")
+class WfRenderPageViews(WorkflowClass[RenderPageViewsAssignment, list[ImageContent]]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: RenderPageViewsAssignment,
+ ) -> list[ImageContent]:
+ workflow_log.debug("Workflow start")
+ worker_config = get_config().temporal.worker_config
+ try:
+ image_content_list = await workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=act_render_page_views,
+ arg=workflow_arg,
+ start_to_close_timeout=worker_config.workflow_execution_timeout,
+ retry_policy=worker_config.retry_policy,
+ )
+ except ActivityError as exc:
+ log.error(f"ActivityError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+
+ workflow_log.debug("Workflow complete")
+ return image_content_list
diff --git a/pipelex/temporal/tprl_pipe/__init__.py b/pipelex/temporal/tprl_pipe/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/temporal/tprl_pipe/act_assemble_graph.py b/pipelex/temporal/tprl_pipe/act_assemble_graph.py
new file mode 100644
index 000000000..b5dd63555
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/act_assemble_graph.py
@@ -0,0 +1,55 @@
+"""Temporal activity for assembling the graph spec from trace events.
+
+Reads trace events from the configured backend (DynamoDB) and assembles
+the full GraphSpec. Runs as an activity because DynamoDB reads are I/O
+that cannot happen inside a workflow thread.
+"""
+
+from pydantic import BaseModel
+from temporalio import activity
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.graph.graphspec import GraphSpec, PipelineRef
+from pipelex.tracing.event_log_factory import make_event_log
+from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
+
+
+class AssembleGraphArg(BaseModel):
+ """Input for the graph assembly activity."""
+
+ pipeline_run_id: str
+ domain_code: str | None = None
+ main_pipe_code: str | None = None
+
+
[email protected](name="act_assemble_graph")
+async def act_assemble_graph(arg: AssembleGraphArg) -> GraphSpec | None: # noqa: RUF029
+ """Read trace events and assemble GraphSpec. Returns None if no events found."""
+ tracing_config = get_config().pipelex.tracing_config
+ if not tracing_config.is_enabled:
+ return None
+
+ try:
+ event_log = make_event_log(tracing_config)
+ try:
+ events = event_log.read_events(arg.pipeline_run_id)
+ if not events:
+ log.debug(f"No trace events found for pipeline_run_id={arg.pipeline_run_id}")
+ return None
+ graph_spec = GraphSpecAssembler.assemble(
+ events=events,
+ graph_id=arg.pipeline_run_id,
+ pipeline_ref=PipelineRef(
+ domain=arg.domain_code,
+ main_pipe=arg.main_pipe_code,
+ ),
+ )
+ log.debug(f"Graph assembled from {len(events)} events for pipeline_run_id={arg.pipeline_run_id}")
+ return graph_spec
+ finally:
+ event_log.close()
+ except Exception as exc:
+ # TODO: wip - do not catch all exceptions
+ log.warning(f"Graph assembly activity failed: {exc}")
+ return None
diff --git a/pipelex/temporal/tprl_pipe/act_deliver.py b/pipelex/temporal/tprl_pipe/act_deliver.py
new file mode 100644
index 000000000..100353cfc
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/act_deliver.py
@@ -0,0 +1,29 @@
+from pydantic import BaseModel
+from temporalio import activity
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus
+from pipelex.pipe_run.delivery_executor import DeliveryExecutor
+
+
+class DeliveryActivityArg(BaseModel):
+ """Input for the act_deliver activity."""
+
+ pipe_output: PipeOutput | None = None
+ user_id: str
+ pipeline_run_id: str
+ delivery_assignment: DeliveryAssignment
+ status: DeliveryStatus = DeliveryStatus.COMPLETED
+
+
[email protected](name="act_deliver")
+async def act_deliver(arg: DeliveryActivityArg) -> None:
+ """Temporal activity that executes the full delivery (storage + webhooks)."""
+ executor = DeliveryExecutor()
+ await executor.execute(
+ pipe_output=arg.pipe_output,
+ user_id=arg.user_id,
+ pipeline_run_id=arg.pipeline_run_id,
+ delivery_assignment=arg.delivery_assignment,
+ status=arg.status,
+ )
diff --git a/pipelex/temporal/tprl_pipe/act_flush_trace_events.py b/pipelex/temporal/tprl_pipe/act_flush_trace_events.py
new file mode 100644
index 000000000..2e68730d2
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/act_flush_trace_events.py
@@ -0,0 +1,41 @@
+"""Temporal activity for flushing buffered trace events to the configured backend.
+
+Called from wf_pipe_router after pipe execution to persist trace events
+that were collected in-memory by BufferingEventLog during the workflow.
+"""
+
+from pydantic import BaseModel, Field
+from temporalio import activity
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+from pipelex.tracing.event_log_factory import make_event_log
+from pipelex.tracing.trace_events import TraceEvent
+
+
+class FlushTraceEventsArg(BaseModel):
+ """Input for the act_flush_trace_events activity."""
+
+ events: list[TraceEvent] = Field(default_factory=empty_list_factory_of(TraceEvent))
+
+
[email protected](name="act_flush_trace_events")
+async def act_flush_trace_events(arg: FlushTraceEventsArg) -> None: # noqa: RUF029
+ """Write buffered trace events to the configured backend. Runs as a Temporal
+ activity so synchronous I/O (boto3, file writes) doesn't block the workflow thread.
+ """
+ if not arg.events:
+ return
+
+ tracing_config = get_config().pipelex.tracing_config
+ if not tracing_config.is_enabled:
+ return
+
+ event_log = make_event_log(tracing_config)
+ try:
+ for event in arg.events:
+ event_log.emit(event)
+ log.debug(f"Flushed {len(arg.events)} trace events to backend={tracing_config.backend}")
+ finally:
+ event_log.close()
diff --git a/pipelex/temporal/tprl_pipe/hydration.py b/pipelex/temporal/tprl_pipe/hydration.py
new file mode 100644
index 000000000..6eee8452d
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/hydration.py
@@ -0,0 +1,134 @@
+from typing import Any, cast
+
+from kajson.exceptions import KajsonException
+from pydantic import ValidationError
+
+from pipelex.core.concepts.concept import Concept
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.stuff_content import StuffContent
+from pipelex.core.stuffs.stuff_content_factory import StuffContentFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_class_registry
+from pipelex.pipe_run.exceptions import PipeJobError
+
+
+def _validate_as_known_class(item_class: type[StuffContent], raw_item: StuffContent | dict[str, Any]) -> StuffContent:
+ """Validate raw_item into item_class, tolerating cross-exec instances.
+
+ When a child workflow ships a ListContent back, ``dump_for_temporal`` embeds
+ ``__class__`` metadata on each item. kajson (in the Temporal data converter)
+ eagerly rebuilds class instances from that metadata *before* the parent
+ workflow runs ``load_from_crate``. That later crate load re-execs the
+ dynamic source and replaces the class in the registry — same name, new
+ Python identity. Pydantic's ``model_validate`` then rejects the old
+ instance because ``type(raw_item) is not item_class``. To bridge the gap,
+ we round-trip through ``smart_dump()`` (serialize_as_any) when we detect a
+ cross-exec instance, so nested subclass fields survive the round-trip.
+ """
+ if isinstance(raw_item, StuffContent):
+ if type(raw_item) is item_class:
+ return raw_item
+ else:
+ return item_class.model_validate(raw_item.smart_dump())
+ else:
+ return item_class.model_validate(raw_item)
+
+
+def _hydrate_list_item(raw_item: dict[str, Any] | str | StuffContent) -> StuffContent:
+ """Hydrate a single list item for Anything[] results.
+
+ Tries __class__ metadata first, falls back to TextContent for plain text or
+ dicts with a 'text' key (the common case for Anything outputs).
+ """
+ if isinstance(raw_item, str):
+ return TextContent(text=raw_item)
+
+ # Already hydrated by kajson via the Temporal data converter. The instance
+ # may come from a previous exec of the dynamic source — normalize through
+ # the registry's current class so downstream type checks stay consistent.
+ if isinstance(raw_item, StuffContent):
+ class_name_from_instance = type(raw_item).__name__
+ item_class_or_none = get_class_registry().get_class(name=class_name_from_instance)
+ if item_class_or_none is not None and issubclass(item_class_or_none, StuffContent):
+ return _validate_as_known_class(item_class_or_none, raw_item)
+ return raw_item
+
+ class_name = raw_item.get("__class__")
+ if class_name is not None:
+ item_class = get_class_registry().get_required_subclass(name=class_name, base_class=StuffContent)
+ clean_item = {key: val for key, val in raw_item.items() if key not in {"__class__", "__module__"}}
+ return cast("StuffContent", item_class.model_validate(clean_item))
+
+ # No __class__ metadata — fall back to TextContent for simple text dicts.
+ # This handles legacy payloads serialized before __class__ metadata was added.
+ if "text" in raw_item:
+ return TextContent.model_validate(raw_item)
+
+ msg = f"Cannot hydrate Anything list item: no __class__ metadata and no recognized content structure. Keys: {list(raw_item.keys())}"
+ raise PipeJobError(msg)
+
+
+def hydrate_content(concept: Concept, raw_content: list[Any] | dict[str, Any] | str) -> StuffContent:
+ """Hydrate a single StuffContent from a raw value.
+
+ Handles both plain content and ListContent. The Temporal serialization
+ format (produced by ``WorkingMemory.dump_for_temporal()``) encodes
+ ListContent as a plain JSON list and single StuffContent as a dict,
+ so the type check is unambiguous — no heuristic required.
+
+ The concept's ``structure_class_name`` always refers to the *item* type,
+ even when the stuff holds a list of those items.
+ """
+ if isinstance(raw_content, list):
+ registry = get_class_registry()
+ item_class_or_none = registry.get_class(name=concept.structure_class_name)
+ if item_class_or_none is not None and issubclass(item_class_or_none, StuffContent):
+ # Known content class (e.g. TextContent, PageContent, or a dynamic
+ # structured concept class). Use _validate_as_known_class so that
+ # cross-exec instances rebuilt by kajson during Temporal transit
+ # get normalized through a dict round-trip.
+ items = [_validate_as_known_class(item_class_or_none, raw_item) for raw_item in raw_content]
+ else:
+ # Anything or unknown concept — resolve each item by its embedded type metadata
+ raw_items = cast("list[dict[str, Any]]", raw_content)
+ items = [_hydrate_list_item(raw_item) for raw_item in raw_items]
+ return ListContent(items=items)
+
+ return StuffContentFactory.make_stuff_content_from_concept_required(
+ concept=concept,
+ value=raw_content,
+ )
+
+
+def hydrate_working_memory(working_memory_raw: dict[str, Any]) -> WorkingMemory:
+ """Reconstruct typed WorkingMemory from a raw dict.
+
+ Must be called AFTER load_from_crate() has registered dynamic classes
+ in the scoped ClassRegistry. Uses concept.structure_class_name to look up
+ the correct StuffContent subclass from the registry.
+ """
+ working_memory = WorkingMemory()
+
+ raw_root = working_memory_raw.get("root", {})
+ for stuff_name, stuff_dict in raw_root.items():
+ try:
+ concept = Concept.model_validate(stuff_dict["concept"])
+ content = hydrate_content(concept=concept, raw_content=stuff_dict["content"])
+ stuff = Stuff(
+ stuff_code=stuff_dict["stuff_code"],
+ stuff_name=stuff_dict.get("stuff_name"),
+ concept=concept,
+ content=content,
+ )
+ except KeyError as exc:
+ msg = f"Failed to hydrate stuff '{stuff_name}': missing key {exc} in raw dict"
+ raise PipeJobError(msg) from exc
+ except (ValidationError, KajsonException) as exc:
+ msg = f"Failed to hydrate stuff '{stuff_name}': {exc}"
+ raise PipeJobError(msg) from exc
+ working_memory.root[stuff_name] = stuff
+
+ working_memory.aliases = working_memory_raw.get("aliases", {})
+ return working_memory
diff --git a/pipelex/temporal/tprl_pipe/pipe_run_arg.py b/pipelex/temporal/tprl_pipe/pipe_run_arg.py
new file mode 100644
index 000000000..e4519ca86
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/pipe_run_arg.py
@@ -0,0 +1,14 @@
+from pydantic import BaseModel
+
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
+from pipelex.pipe_run.pipe_job import PipeJob
+
+
+class PipeRunArg(BaseModel):
+ """Workflow input for WfPipeRun: bundles PipeJob + delivery assignment for Temporal serialization."""
+
+ pipe_job: PipeJob
+ delivery_assignment: DeliveryAssignment | None = None
+
+ def prepare_for_temporal(self) -> "PipeRunArg":
+ return self.model_copy(update={"pipe_job": self.pipe_job.prepare_for_temporal()})
diff --git a/pipelex/temporal/tprl_pipe/submitter_hydration.py b/pipelex/temporal/tprl_pipe/submitter_hydration.py
new file mode 100644
index 000000000..4c5015e32
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/submitter_hydration.py
@@ -0,0 +1,99 @@
+"""Submitter-side rehydration of PipeOutput received from a Temporal workflow.
+
+The workflow side (`WfPipeRouter`) dehydrates `PipeOutput.working_memory` into
+`working_memory_raw` for transit. The submitter must rebuild the typed
+`WorkingMemory` after the workflow returns.
+
+When the submitter has loaded the bundle locally (via `PIPELEXPATH` or by
+explicit setup), the dynamic concept classes live in the global class registry
+and a plain `hydrate_working_memory(...)` call works. But when the submitter is
+a remote API client that built the `PipeJob` from a serialized `LibraryCrate`
+without loading it into the global registry, hydration would fail (or worse,
+silently degrade) on dynamic concept fields.
+
+`rehydrate_pipe_output_with_crate` mirrors what `WfPipeRouter` does on the
+worker side: open a per-call scoped Library, pre-seed a scoped `ClassRegistry`
+from the global, load the crate into it, hydrate inside the scope, then tear
+down — leaving the submitter's global registry untouched.
+"""
+
+import uuid
+from typing import TYPE_CHECKING
+
+from kajson.class_registry import ClassRegistry
+from kajson.kajson_manager import KajsonManager
+
+from pipelex import log
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.hub import get_current_library, get_library_manager, set_current_library, teardown_current_library
+from pipelex.temporal.tprl_pipe.hydration import hydrate_working_memory
+
+if TYPE_CHECKING:
+ from pipelex.libraries.library_crate import LibraryCrate
+
+
+def rehydrate_pipe_output_with_crate(
+ pipe_output: PipeOutput,
+ library_crate: "LibraryCrate | None",
+) -> PipeOutput:
+ """Rehydrate `pipe_output.working_memory_raw` into a typed WorkingMemory.
+
+ When `library_crate` is provided, opens a fresh per-call scoped Library,
+ loads the crate into it, hydrates inside the scope, and tears down — so
+ callers that did not load the bundle into their global registry still get
+ a correct typed WorkingMemory. When `library_crate` is None, falls back to
+ the active class registry (built-ins + whatever PIPELEXPATH loaded).
+
+ Mutates `pipe_output` in place: sets `working_memory`, clears
+ `working_memory_raw`. Returns the same instance for call-chain ergonomics.
+ """
+ if pipe_output.working_memory_raw is None:
+ return pipe_output
+
+ if library_crate is None:
+ pipe_output.working_memory = hydrate_working_memory(pipe_output.working_memory_raw)
+ pipe_output.working_memory_raw = None
+ return pipe_output
+
+ library_manager = get_library_manager()
+ rehydration_library_id = f"rehydrate_{uuid.uuid4().hex[:8]}"
+ prev_library_id = _get_current_library_id_or_none()
+ library_opened = False
+ library_set_as_current = False
+ try:
+ _lib_id, rehydration_library = library_manager.open_library(library_id=rehydration_library_id)
+ library_opened = True
+
+ global_registry = KajsonManager.get_class_registry()
+ scoped_registry = ClassRegistry()
+ if isinstance(global_registry, ClassRegistry):
+ scoped_registry.register_classes_dict(dict(global_registry.root))
+ else:
+ log.warning(f"Global registry is {type(global_registry).__name__}, not ClassRegistry — cannot pre-seed rehydration registry")
+ rehydration_library.set_class_registry(scoped_registry)
+
+ set_current_library(library_id=rehydration_library_id)
+ library_set_as_current = True
+ library_manager.load_from_crate(library_id=rehydration_library_id, crate=library_crate)
+ pipe_output.working_memory = hydrate_working_memory(pipe_output.working_memory_raw)
+ pipe_output.working_memory_raw = None
+ finally:
+ try:
+ if library_opened:
+ library_manager.teardown(library_id=rehydration_library_id)
+ finally:
+ if library_set_as_current:
+ if prev_library_id is not None:
+ set_current_library(library_id=prev_library_id)
+ else:
+ teardown_current_library()
+
+ return pipe_output
+
+
+def _get_current_library_id_or_none() -> str | None:
+ """Get the current library_id without raising if none is set."""
+ try:
+ return get_current_library()
+ except RuntimeError:
+ return None
diff --git a/pipelex/temporal/tprl_pipe/temporal_pipe_router.py b/pipelex/temporal/tprl_pipe/temporal_pipe_router.py
new file mode 100644
index 000000000..b0b7a9cb1
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/temporal_pipe_router.py
@@ -0,0 +1,109 @@
+from datetime import timedelta
+
+from temporalio import workflow
+from temporalio.common import RetryPolicy
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.observer.observer_protocol import ObserverNoOp
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol
+from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment
+from pipelex.temporal.temporal_workflow_utils import is_in_temporal_workflow
+from pipelex.temporal.tprl.conditional_worker import with_conditional_worker
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor, WorkflowExecutorFactory
+from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+
+
+class TemporalPipeRouter(WorkflowExecutor[PipeJob, PipeOutput], PipeRouterProtocol):
+ """Temporal pipe router: auto-detects whether to dispatch as top-level or child workflow.
+
+ When called outside a Temporal workflow, dispatches via execute_workflow (top-level).
+ When called inside a Temporal workflow, dispatches via execute_child_workflow (child).
+ """
+
+ def __init__(
+ self,
+ task_queue: str,
+ workflow_execution_timeout: timedelta | None = None,
+ retry_policy: RetryPolicy | None = None,
+ should_auto_connect_temporal: bool = False,
+ worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,
+ ):
+ log.debug(f"TemporalPipeRouter init with worker_environment: {worker_environment}")
+ super().__init__(
+ workflow_execution_timeout=workflow_execution_timeout,
+ retry_policy=retry_policy,
+ task_queue=task_queue,
+ should_auto_connect_temporal=should_auto_connect_temporal,
+ worker_environment=worker_environment,
+ )
+ self.observer = ObserverNoOp()
+
+ @override
+ @with_conditional_worker
+ async def _run_pipe_job(
+ self,
+ pipe_job: PipeJob,
+ wfid: str | None = None,
+ ) -> PipeOutput:
+ pipe_job = pipe_job.prepare_for_temporal()
+
+ if is_in_temporal_workflow():
+ # Child workflow dispatch (inside a Temporal workflow)
+ # Use deterministic ID derived from parent workflow to avoid Temporal nondeterminism errors
+ log.debug("TemporalPipeRouter: child workflow dispatch")
+ parent_workflow_id = workflow.info().workflow_id
+ child_unique_id = wfid or str(workflow.uuid4())
+ child_workflow_id = f"{parent_workflow_id}-{child_unique_id}"
+ executor = WorkflowExecutorFactory[PipeJob, PipeOutput]().create_executor(task_queue=None)
+ pipe_output = await executor.execute_child_workflow(
+ workflow_class=WfPipeRouter,
+ workflow_id=child_workflow_id,
+ workflow_arg=pipe_job,
+ )
+ else:
+ # Top-level dispatch (outside a Temporal workflow)
+ log.debug(f"TemporalPipeRouter: top-level dispatch, task_queue={self.task_queue}")
+ executor = WorkflowExecutorFactory[PipeJob, PipeOutput]().create_executor(
+ task_queue=self.task_queue,
+ workflow_execution_timeout=self.execution_timeout,
+ retry_policy=self.retry_policy,
+ run_timeout=self.run_timeout,
+ task_timeout=self.task_timeout,
+ start_delay=self.start_delay,
+ rpc_timeout=self.rpc_timeout,
+ should_auto_connect_temporal=self.should_auto_connect_temporal,
+ worker_environment=self.worker_environment,
+ )
+ pipe_output = await executor.execute_workflow(
+ workflow_class=WfPipeRouter,
+ workflow_id=self.make_workflow_id(base_id=wfid or self.class_name),
+ workflow_arg=pipe_job,
+ )
+
+ # Rehydrate PipeOutput: reconstruct typed WorkingMemory from raw dict.
+ # Uses a per-call scoped library when the pipe_job carries a crate so the
+ # submitter does not need to have pre-loaded the bundle into its global registry.
+ return rehydrate_pipe_output_with_crate(pipe_output, pipe_job.library_crate)
+
+
+def make_temporal_pipe_router(
+ task_queue: str | None = None,
+ workflow_execution_timeout: timedelta | None = None,
+ retry_policy: RetryPolicy | None = None,
+ should_auto_connect_temporal: bool = True,
+ worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,
+) -> TemporalPipeRouter:
+ """Factory: creates a TemporalPipeRouter from config defaults."""
+ worker_config = get_config().temporal.worker_config
+ return TemporalPipeRouter(
+ task_queue=task_queue or worker_config.task_queue,
+ workflow_execution_timeout=workflow_execution_timeout or worker_config.workflow_execution_timeout,
+ retry_policy=retry_policy or worker_config.retry_policy,
+ should_auto_connect_temporal=should_auto_connect_temporal,
+ worker_environment=worker_environment,
+ )
diff --git a/pipelex/temporal/tprl_pipe/temporal_pipe_run.py b/pipelex/temporal/tprl_pipe/temporal_pipe_run.py
new file mode 100644
index 000000000..23e01a537
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/temporal_pipe_run.py
@@ -0,0 +1,119 @@
+from datetime import timedelta
+
+from temporalio.client import WorkflowHandle
+from temporalio.common import RetryPolicy
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
+from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment
+from pipelex.temporal.tprl.conditional_worker import with_conditional_worker
+from pipelex.temporal.tprl.workflow_caller import WorkflowClass, WorkflowExecutor, WorkflowExecutorFactory
+from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg
+from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
+from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun
+
+
+class TemporalPipeRun(WorkflowExecutor[PipeRunArg, PipeOutput], PipeRunProtocol):
+ """Temporal-mode PipeRun: dispatches WfPipeRun workflow which orchestrates execution + delivery."""
+
+ def __init__(
+ self,
+ task_queue: str,
+ workflow_execution_timeout: timedelta | None = None,
+ retry_policy: RetryPolicy | None = None,
+ should_auto_connect_temporal: bool = False,
+ worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,
+ ) -> None:
+ log.debug(f"TemporalPipeRun init with worker_environment: {worker_environment}")
+ super().__init__(
+ workflow_execution_timeout=workflow_execution_timeout,
+ retry_policy=retry_policy,
+ task_queue=task_queue,
+ should_auto_connect_temporal=should_auto_connect_temporal,
+ worker_environment=worker_environment,
+ )
+
+ @override
+ @with_conditional_worker
+ async def run(
+ self,
+ pipe_job: PipeJob,
+ delivery_assignment: DeliveryAssignment | None = None,
+ wfid: str | None = None,
+ ) -> PipeOutput:
+ """Execute a pipe run via Temporal (blocking — waits for completion)."""
+ pipe_run_arg = PipeRunArg(
+ pipe_job=pipe_job,
+ delivery_assignment=delivery_assignment,
+ )
+ pipe_run_arg = pipe_run_arg.prepare_for_temporal()
+
+ executor = WorkflowExecutorFactory[PipeRunArg, PipeOutput]().create_executor(
+ task_queue=self.task_queue,
+ should_auto_connect_temporal=self.should_auto_connect_temporal,
+ worker_environment=self.worker_environment,
+ )
+ pipe_output = await executor.execute_workflow(
+ workflow_class=WfPipeRun,
+ workflow_id=self.make_workflow_id(base_id=wfid or self.class_name),
+ workflow_arg=pipe_run_arg,
+ )
+
+ # Rehydrate PipeOutput on the submitter. When the pipe_job carries a crate,
+ # the helper opens a per-call scoped library so the submitter does not need
+ # the bundle pre-loaded in its global registry.
+ return rehydrate_pipe_output_with_crate(pipe_output, pipe_job.library_crate)
+
+ @with_conditional_worker
+ async def start(
+ self,
+ pipe_job: PipeJob,
+ delivery_assignment: DeliveryAssignment | None = None,
+ wfid: str | None = None,
+ ) -> tuple[str, WorkflowHandle[WorkflowClass[PipeRunArg, PipeOutput], PipeOutput]]:
+ """Start a pipe run without waiting for completion.
+
+ Returns the workflow_id and a WorkflowHandle that can be awaited later.
+ """
+ log.debug(f"TemporalPipeRun start using task_queue: {self.task_queue}")
+ pipe_run_arg = PipeRunArg(
+ pipe_job=pipe_job,
+ delivery_assignment=delivery_assignment,
+ )
+ pipe_run_arg = pipe_run_arg.prepare_for_temporal()
+
+ executor = WorkflowExecutorFactory[PipeRunArg, PipeOutput]().create_executor(
+ task_queue=self.task_queue,
+ should_auto_connect_temporal=self.should_auto_connect_temporal,
+ worker_environment=self.worker_environment,
+ )
+ workflow_id = self.make_workflow_id(base_id=wfid or self.class_name)
+ handle = await executor.start_workflow(
+ workflow_class=WfPipeRun,
+ workflow_id=workflow_id,
+ workflow_arg=pipe_run_arg,
+ )
+ return workflow_id, handle
+
+
+def make_temporal_pipe_run(
+ task_queue: str | None = None,
+ workflow_execution_timeout: timedelta | None = None,
+ retry_policy: RetryPolicy | None = None,
+ should_auto_connect_temporal: bool = True,
+ worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,
+) -> TemporalPipeRun:
+ """Factory: creates a TemporalPipeRun from config defaults."""
+ worker_config = get_config().temporal.worker_config
+ return TemporalPipeRun(
+ task_queue=task_queue or worker_config.task_queue,
+ workflow_execution_timeout=workflow_execution_timeout or worker_config.workflow_execution_timeout,
+ retry_policy=retry_policy or worker_config.retry_policy,
+ should_auto_connect_temporal=should_auto_connect_temporal,
+ worker_environment=worker_environment,
+ )
diff --git a/pipelex/temporal/tprl_pipe/wf_pipe_router.py b/pipelex/temporal/tprl_pipe/wf_pipe_router.py
new file mode 100644
index 000000000..a728e3435
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/wf_pipe_router.py
@@ -0,0 +1,188 @@
+from datetime import timedelta
+
+from temporalio import workflow
+from temporalio.common import RetryPolicy
+from temporalio.exceptions import ActivityError, ApplicationError
+from typing_extensions import override
+
+with workflow.unsafe.imports_passed_through():
+ from kajson.class_registry import ClassRegistry
+ from kajson.kajson_manager import KajsonManager
+
+ from pipelex.config import get_config
+ from pipelex.core.pipes.pipe_output import PipeOutput
+ from pipelex.graph.graph_tracer_manager import GraphTracerManager
+ from pipelex.hub import get_library_manager, get_report_delegate, set_current_library, teardown_current_library
+ from pipelex.pipe_run.pipe_job import PipeJob
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.tprl.temporal_error import TemporalError
+ from pipelex.temporal.tprl.workflow_caller import WorkflowClass
+ from pipelex.temporal.tprl_pipe.act_flush_trace_events import FlushTraceEventsArg, act_flush_trace_events
+ from pipelex.temporal.tprl_pipe.hydration import hydrate_working_memory
+ from pipelex.tracing.buffering_event_log import BufferingEventLog
+
+
[email protected](name="wf_pipe_router")
+class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):
+ @override
+ @workflow.run
+ async def run(
+ self,
+ workflow_arg: PipeJob,
+ ) -> PipeOutput:
+ workflow_log.debug("Workflow start")
+
+ pipe = workflow_arg.pipe
+ workflow_log.verbose(f"Routing {pipe.__class__.__name__} pipe '{workflow_arg.pipe.code}': {pipe.description}")
+
+ # Set up per-workflow library if a library crate is present
+ library_crate = workflow_arg.library_crate
+ wf_library_id: str | None = None
+
+ # Per-workflow tracing state (declared before try for finally block access)
+ event_log = None
+ wf_graph_tracer_manager: GraphTracerManager | None = None
+ wf_tracer_key: str | None = None
+ graph_context = workflow_arg.job_metadata.graph_context
+
+ pipe_output: PipeOutput | None = None
+
+ try:
+ if library_crate is not None:
+ # 1. Create per-workflow ClassRegistry pre-seeded from global
+ global_registry = KajsonManager.get_class_registry()
+ workflow_registry = ClassRegistry()
+ if isinstance(global_registry, ClassRegistry):
+ workflow_registry.register_classes_dict(dict(global_registry.root))
+ else:
+ workflow_log.warning("Global registry is not a ClassRegistry, cannot pre-seed workflow registry")
+
+ # 2. Open library and attach registry to it
+ library_manager = get_library_manager()
+ wf_library_id = f"wf_{workflow.info().workflow_id}"
+ _wf_library_id, wf_library = library_manager.open_library(library_id=wf_library_id)
+ wf_library.set_class_registry(workflow_registry)
+ set_current_library(library_id=wf_library_id)
+
+ # 3. Load crate (registers dynamic classes into workflow_registry via hub.get_class_registry())
+ library_manager.load_from_crate(library_id=wf_library_id, crate=library_crate)
+
+ # 4. Hydrate WorkingMemory if needed
+ if workflow_arg.working_memory_raw is not None:
+ workflow_arg.working_memory = hydrate_working_memory(workflow_arg.working_memory_raw)
+ workflow_arg.working_memory_raw = None
+
+ # Set up per-workflow graph tracing if enabled
+ pipeline_run_id = workflow_arg.job_metadata.pipeline_run_id
+ wf_workflow_id = workflow.info().workflow_id
+
+ tracing_config = get_config().pipelex.tracing_config
+ if tracing_config.is_enabled and graph_context is not None:
+ try:
+ # Use BufferingEventLog inside workflows (no I/O allowed).
+ # Events are flushed to the real backend via act_flush_trace_events.
+ event_log = BufferingEventLog()
+ wf_graph_tracer_manager = GraphTracerManager.get_or_create_instance()
+ wf_tracer_key = wf_workflow_id
+ wf_graph_context = wf_graph_tracer_manager.open_tracer(
+ graph_id=graph_context.graph_id,
+ data_inclusion=graph_context.data_inclusion,
+ event_log=event_log,
+ workflow_id=wf_workflow_id,
+ pipeline_run_id=pipeline_run_id,
+ tracer_key=wf_tracer_key,
+ )
+ # Update job_metadata with the per-workflow graph_context (carries tracer_key),
+ # but preserve parent_node_id from the incoming context so CONTAINS edges
+ # link back to the parent workflow's controller node.
+ wf_graph_context = wf_graph_context.model_copy(
+ update={"parent_node_id": graph_context.parent_node_id},
+ )
+ workflow_arg.job_metadata = workflow_arg.job_metadata.model_copy(
+ update={"graph_context": wf_graph_context},
+ )
+ # Configure the report delegate for usage event emission
+ get_report_delegate().set_event_log(
+ context_key=wf_workflow_id,
+ event_log=event_log,
+ workflow_id=wf_workflow_id,
+ pipeline_run_id=pipeline_run_id,
+ )
+ except Exception as exc:
+ workflow_log.warning(f"Failed to set up per-workflow tracing, continuing without: {exc}")
+ # Clean up partially initialized resources before nulling (the finally block
+ # won't be able to clean up after we null these references)
+ if wf_graph_tracer_manager is not None and wf_tracer_key is not None:
+ try:
+ wf_graph_tracer_manager.close_tracer(wf_tracer_key)
+ except Exception as tracer_exc:
+ workflow_log.warning(f"Failed to close partially initialized tracer: {tracer_exc}")
+ if event_log is not None:
+ try:
+ event_log.close()
+ except Exception: # noqa: S110
+ pass
+ get_report_delegate().clear_event_log(context_key=wf_workflow_id)
+ event_log = None
+ wf_graph_tracer_manager = None
+
+ working_memory = workflow_arg.get_working_memory()
+ pipe_output = await pipe.run_pipe(
+ job_metadata=workflow_arg.job_metadata,
+ working_memory=working_memory,
+ output_name=workflow_arg.output_name,
+ pipe_run_params=workflow_arg.pipe_run_params,
+ library_crate=library_crate,
+ )
+ except ActivityError as exc:
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ finally:
+ # Close per-workflow graph tracer (collects in-memory graph spec)
+ if wf_graph_tracer_manager is not None and wf_tracer_key is not None:
+ try:
+ graph_spec = wf_graph_tracer_manager.close_tracer(wf_tracer_key)
+ if graph_spec is not None and pipe_output is not None:
+ pipe_output.graph_spec = graph_spec
+ except Exception as tracer_exc:
+ workflow_log.warning(f"Failed to close per-workflow tracer: {tracer_exc}")
+
+ # Flush trace events and clean up event log
+ if event_log is not None:
+ # Drain buffered events and flush to the real backend via activity
+ buffered_events = event_log.drain()
+ if buffered_events:
+ try:
+ await workflow.execute_activity(
+ act_flush_trace_events,
+ arg=FlushTraceEventsArg(events=buffered_events),
+ start_to_close_timeout=timedelta(seconds=30),
+ retry_policy=RetryPolicy(maximum_attempts=3),
+ )
+ except Exception as flush_exc:
+ workflow_log.warning(f"Failed to flush trace events: {flush_exc}")
+
+ try:
+ event_log.close()
+ except Exception: # noqa: S110
+ pass
+
+ # Clear stale event log state from the report delegate
+ if wf_tracer_key is not None:
+ get_report_delegate().clear_event_log(context_key=wf_tracer_key)
+
+ if wf_library_id is not None:
+ try:
+ get_library_manager().teardown(library_id=wf_library_id)
+ finally:
+ teardown_current_library()
+
+ # Dehydrate PipeOutput for Temporal transit: serialize WorkingMemory to
+ # raw dict so the parent's data converter can deserialize without needing
+ # dynamic concept classes in its ClassRegistry.
+ assert pipe_output is not None
+ pipe_output = pipe_output.prepare_for_temporal(library_crate=library_crate)
+
+ workflow_log.debug("Workflow complete")
+ return pipe_output
diff --git a/pipelex/temporal/tprl_pipe/wf_pipe_run.py b/pipelex/temporal/tprl_pipe/wf_pipe_run.py
new file mode 100644
index 000000000..311fc59a0
--- /dev/null
+++ b/pipelex/temporal/tprl_pipe/wf_pipe_run.py
@@ -0,0 +1,114 @@
+from datetime import timedelta
+
+from temporalio import workflow
+from temporalio.common import RetryPolicy
+from temporalio.exceptions import ActivityError, ChildWorkflowError
+from typing_extensions import override
+
+with workflow.unsafe.imports_passed_through():
+ from pipelex.core.pipes.pipe_output import PipeOutput
+ from pipelex.pipe_run.delivery_assignment import DeliveryStatus
+ from pipelex.temporal.log_temporal import workflow_log
+ from pipelex.temporal.tprl.workflow_caller import WorkflowClass
+ from pipelex.temporal.tprl_pipe.act_assemble_graph import AssembleGraphArg, act_assemble_graph
+ from pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg, act_deliver
+ from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg
+ from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+
+
[email protected](name="wf_pipe_run")
+class WfPipeRun(WorkflowClass[PipeRunArg, PipeOutput]):
+ """Parent workflow that orchestrates pipe execution + delivery.
+
+ 1. Runs WfPipeRouter as a child workflow to execute the pipe
+ 2. Runs a delivery activity — always runs, even on failure (to notify the completion Lambda)
+ """
+
+ @override
+ @workflow.run
+ async def run(self, workflow_arg: PipeRunArg) -> PipeOutput:
+ workflow_log.debug("WfPipeRun start")
+
+ pipe_job = workflow_arg.pipe_job
+ delivery_assignment = workflow_arg.delivery_assignment
+ pipeline_run_id: str = pipe_job.job_metadata.pipeline_run_id
+
+ # Step 1: Execute the pipe via child workflow
+ workflow_log.debug(f"Starting child WfPipeRouter for pipe '{pipe_job.pipe.code}'")
+ status: DeliveryStatus = DeliveryStatus.COMPLETED
+ pipe_output: PipeOutput | None = None
+ execution_error: ChildWorkflowError | None = None
+
+ try:
+ pipe_output = await workflow.execute_child_workflow(
+ WfPipeRouter.run,
+ arg=pipe_job,
+ id=f"{workflow.info().workflow_id}-pipe-router",
+ )
+ workflow_log.debug("WfPipeRouter completed successfully")
+ except ChildWorkflowError as exc:
+ status = DeliveryStatus.FAILED
+ execution_error = exc
+ workflow_log.error(f"WfPipeRouter failed: {exc}")
+
+ # Step 2: Assemble full graph from trace events (cross-worker)
+ # Runs as an activity because DynamoDB reads are I/O forbidden in workflows.
+ if pipe_output is not None:
+ try:
+ graph_spec = await workflow.execute_activity(
+ act_assemble_graph,
+ arg=AssembleGraphArg(
+ pipeline_run_id=pipeline_run_id,
+ domain_code=pipe_job.pipe.domain_code,
+ main_pipe_code=pipe_job.pipe.code,
+ ),
+ start_to_close_timeout=timedelta(seconds=30),
+ retry_policy=RetryPolicy(maximum_attempts=3),
+ )
+ if graph_spec is not None:
+ pipe_output.graph_spec = graph_spec
+ except ActivityError as graph_exc:
+ workflow_log.warning(f"Graph assembly failed, continuing with delivery: {graph_exc}")
+ pipe_output.graph_assembly_error = str(graph_exc)
+
+ # Step 3: Run delivery activity if requested — notifies the completion
+ # Lambda of success or failure when a delivery_assignment was provided.
+ # No assignment → no delivery (matches PipeRun direct-mode semantics).
+ delivery_error: ActivityError | None = None
+ if delivery_assignment is not None:
+ workflow_log.debug(f"Running delivery: pipeline_run_id={pipeline_run_id}, status={status}")
+ activity_arg = DeliveryActivityArg(
+ user_id=pipe_job.job_metadata.user_id,
+ pipeline_run_id=pipeline_run_id,
+ delivery_assignment=delivery_assignment,
+ status=status,
+ )
+ # Include pipe_output only on success
+ if pipe_output is not None:
+ activity_arg = activity_arg.model_copy(update={"pipe_output": pipe_output})
+
+ try:
+ await workflow.execute_activity(
+ act_deliver,
+ arg=activity_arg,
+ start_to_close_timeout=timedelta(seconds=60),
+ retry_policy=RetryPolicy(maximum_attempts=3),
+ )
+ workflow_log.debug("Delivery completed")
+ except ActivityError as activity_error:
+ # Capture but do not raise yet: when both router and delivery fail
+ # (e.g., pipe failure + webhook outage) we must preserve the original
+ # execution_error for failure attribution, matching direct-mode PipeRun.
+ delivery_error = activity_error
+ workflow_log.error(f"Delivery activity failed: {activity_error}")
+
+ # Re-raise: prefer the original execution error so failure attribution stays
+ # correct when delivery also fails. Fall back to the delivery error otherwise.
+ if execution_error is not None:
+ raise execution_error
+ if delivery_error is not None:
+ raise delivery_error
+
+ assert pipe_output is not None
+ workflow_log.debug("WfPipeRun complete")
+ return pipe_output
diff --git a/pipelex/temporal/worker_cli.py b/pipelex/temporal/worker_cli.py
new file mode 100644
index 000000000..48dfd8c20
--- /dev/null
+++ b/pipelex/temporal/worker_cli.py
@@ -0,0 +1,86 @@
+"""python -m pipelex.temporal.worker_cli
+python -m pipelex.temporal.worker_cli --is-unit-testing
+python -m pipelex.temporal.worker_cli --is-not-sandboxed
+python -m pipelex.temporal.worker_cli --task-queue my_task_queue
+python -m pipelex.temporal.worker_cli --scope router
+"""
+
+import asyncio
+import os
+from typing import Annotated
+
+import typer
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.pipelex import Pipelex
+from pipelex.system.runtime import RunMode, runtime_manager
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.tools.misc.toml_utils import load_toml_from_path
+
+app = typer.Typer()
+
+
+async def run_worker(
+ project: str | None = None,
+ is_not_sandboxed: bool = False,
+ is_unit_testing: bool = False,
+ task_queue: str | None = None,
+ scope_name: str | None = None,
+):
+ if project is None:
+ log.info(f"Starting worker for current project '{project}', from {os.path.relpath(__file__)}")
+ else:
+ log.info(f"Starting worker for chosen project '{project}', from {os.path.relpath(__file__)}")
+ await get_task_manager().run_worker(
+ is_not_sandboxed=is_not_sandboxed,
+ is_unit_testing=is_unit_testing,
+ task_queue=task_queue,
+ scope_name=scope_name,
+ )
+
+
+def configure(
+ project: Annotated[str | None, typer.Argument(help="The project name if you don't want to get it from pyproject.toml")] = None,
+ is_not_sandboxed: Annotated[bool, typer.Option(help="Flag to run without sandbox")] = False,
+ is_unit_testing: Annotated[bool, typer.Option(help="Flag to indicate if running unit tests")] = False,
+ task_queue: Annotated[str | None, typer.Option(help="The task queue to use")] = None,
+ scope: Annotated[str | None, typer.Option(help="Worker scope name from [temporal.worker_scopes.scopes] (defaults to default_scope)")] = None,
+):
+ if is_unit_testing:
+ runtime_manager.set_run_mode(RunMode.UNIT_TEST)
+ if project is None:
+ pyproject = load_toml_from_path(path="pyproject.toml")
+ project = pyproject.get("project", {}).get("name") or pyproject.get("tool", {}).get("poetry", {}).get("name")
+ if not project:
+ msg = "Project name not found in pyproject.toml"
+ raise ValueError(msg)
+
+ Pipelex.make(temporal_enabled=True)
+
+ # Load base library from PIPELEXPATH at worker startup.
+ # This generates dynamic concept classes and registers them with Kajson (fixing deserialization)
+ # and populates the pipe library (fixing get_required_pipe() for controllers).
+ from pipelex.hub import get_library_manager, resolve_library_dirs, set_current_library # noqa: PLC0415
+
+ library_manager = get_library_manager()
+ library_manager.open_library(library_id="worker_base")
+ set_current_library(library_id="worker_base")
+ effective_dirs, source_label = resolve_library_dirs(library_dirs=None)
+ if effective_dirs:
+ log.info(f"Worker loading base library from {len(effective_dirs)} directory(ies) ({source_label})")
+ library_manager.load_libraries(library_id="worker_base", library_dirs=effective_dirs)
+ else:
+ log.info("No library directories configured for worker (PIPELEXPATH not set)")
+
+ if not get_config().temporal.is_enabled:
+ log.warning("temporal.is_enabled is false in config, but forcing it on for worker mode")
+ updated_temporal = get_config().temporal.model_copy(update={"is_enabled": True})
+ get_config().temporal = updated_temporal
+
+ asyncio.run(run_worker(project, is_not_sandboxed, is_unit_testing, task_queue, scope))
+
+
+if __name__ == "__main__":
+ app()
diff --git a/pipelex/temporal/wrapper/__init__.py b/pipelex/temporal/wrapper/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/temporal/wrapper/activity.py b/pipelex/temporal/wrapper/activity.py
new file mode 100644
index 000000000..59520abe8
--- /dev/null
+++ b/pipelex/temporal/wrapper/activity.py
@@ -0,0 +1,29 @@
+from datetime import timedelta
+from typing import Awaitable, Callable, ParamSpec, TypeVar
+
+from pydantic import BaseModel
+from temporalio import workflow
+
+from pipelex.config import get_config
+
+T = TypeVar("T")
+P = ParamSpec("P")
+B = TypeVar("B", bound=BaseModel)
+
+
+def start_tprl_activity(
+ activity: Callable[[B], Awaitable[T]],
+ workflow_arg: B,
+) -> Awaitable[T]:
+ worker_config = get_config().temporal.worker_config
+ max_timeout = timedelta(hours=24)
+
+ return workflow.start_activity( # pyright: ignore[reportUnknownMemberType, reportAssignmentType]
+ activity=activity,
+ arg=workflow_arg,
+ retry_policy=worker_config.retry_policy,
+ schedule_to_close_timeout=max_timeout,
+ schedule_to_start_timeout=max_timeout,
+ start_to_close_timeout=max_timeout,
+ heartbeat_timeout=max_timeout,
+ )
diff --git a/pipelex/test_extras/shared_pytest_plugins.py b/pipelex/test_extras/shared_pytest_plugins.py
index 5ff47c7ea..99d5c4bf7 100644
--- a/pipelex/test_extras/shared_pytest_plugins.py
+++ b/pipelex/test_extras/shared_pytest_plugins.py
@@ -10,6 +10,14 @@ from pipelex.pipe_run.pipe_run_params import PipeRunMode
from pipelex.system.environment import is_env_var_set, is_env_var_truthy, set_env
from pipelex.system.runtime import CODEX_CLOUD_ENV_VAR_KEY, RunMode, runtime_manager
from pipelex.tools.misc.placeholder import make_placeholder_value, value_is_placeholder
+from pipelex.types import StrEnum
+
+
+class ClassRegistryMode(StrEnum):
+ SHARED = "shared"
+ ISOLATED = "isolated"
+ BOTH = "both"
+
# List of environment variables that may need placeholders in CI
ENV_VAR_KEYS_WHICH_MAY_NEED_PLACEHOLDERS_IN_CI = [
@@ -73,6 +81,13 @@ def pytest_addoption(parser: Parser):
help="Disable inference for this test session. Uses mock content generator, "
"skips gateway terms check, and auto-skips tests marked with @pytest.mark.inference.",
)
+ parser.addoption(
+ "--class-registry",
+ default=ClassRegistryMode.BOTH,
+ choices=list(ClassRegistryMode),
+ help="Class registry mode: 'both' runs tests in shared and isolated modes (default), "
+ "'shared' leaks dynamic classes to global, 'isolated' scopes them to the library registry",
+ )
def pytest_configure(config: Config) -> None:
@@ -124,7 +139,7 @@ def pytest_configure(config: Config) -> None:
pytest.exit("Service terms not accepted - run 'pipelex init agreement' first", returncode=1)
[email protected](scope="session")
def pipe_run_mode(request: FixtureRequest) -> PipeRunMode:
# Force dry mode when inference is disabled
if request.config.getoption("--disable-inference", default=False):
diff --git a/pipelex/tools/misc/http_utils.py b/pipelex/tools/misc/http_utils.py
index 4377ffbc4..612a67cef 100644
--- a/pipelex/tools/misc/http_utils.py
+++ b/pipelex/tools/misc/http_utils.py
@@ -10,6 +10,10 @@ URL_MAX_LENGTH = 2048
# URI schemes that are handled internally and should not be validated
_SKIP_VALIDATION_PREFIXES = ("data:", "pipelex-storage://")
+# HTTP status codes where HEAD rejection is a known server misconfiguration
+# (CDNs, signed URLs, auth-gated endpoints) and a GET fallback is justified.
+_HEAD_REJECTED_CODES = {403, 405}
+
def get_user_agent() -> str:
version = get_package_version()
@@ -48,8 +52,8 @@ def _validate_http_url(url: str) -> None:
headers = {"User-Agent": user_agent}
try:
response = httpx.head(url, timeout=10, follow_redirects=True, headers=headers)
- if response.status_code == 405:
- # Server doesn't support HEAD — fall back to streaming GET (read only status, not body)
+ if response.status_code in _HEAD_REJECTED_CODES:
+ log.verbose(f"HEAD request to '{url}' returned {response.status_code}, falling back to streaming GET")
with httpx.stream("GET", url, timeout=10, follow_redirects=True, headers=headers) as stream_response:
stream_response.raise_for_status()
else:
diff --git a/pipelex/tools/storage/storage_config.py b/pipelex/tools/storage/storage_config.py
index c95531b60..9f3a20e28 100644
--- a/pipelex/tools/storage/storage_config.py
+++ b/pipelex/tools/storage/storage_config.py
@@ -99,9 +99,9 @@ class StorageGcpConfig(ConfigModel):
raise StorageConfigError(msg)
-class StorageConfig(ConfigModel):
- is_fetch_remote_content_enabled: bool
- is_upload_local_content_enabled: bool
+class StorageProviderConfig(ConfigModel):
+ """Provider-selection config shared by asset storage and payload codec storage."""
+
method: StorageMethod = Field(strict=False)
local: StorageLocalConfig | None = None
in_memory: StorageInMemoryConfig | None = None
@@ -109,7 +109,7 @@ class StorageConfig(ConfigModel):
gcp: StorageGcpConfig | None = None
@model_validator(mode="after")
- def validate_storage_config(self) -> Self:
+ def validate_storage_provider_config(self) -> Self:
match self.method:
case StorageMethod.LOCAL:
if not self.local:
@@ -159,3 +159,8 @@ class StorageConfig(ConfigModel):
msg = "gcp config is required to access uri_format"
raise StorageConfigError(msg)
return self.gcp.uri_format
+
+
+class StorageConfig(StorageProviderConfig):
+ is_fetch_remote_content_enabled: bool
+ is_upload_local_content_enabled: bool
diff --git a/pipelex/tools/storage/storage_provider_factory.py b/pipelex/tools/storage/storage_provider_factory.py
index 4e3d93d02..917d19c13 100644
--- a/pipelex/tools/storage/storage_provider_factory.py
+++ b/pipelex/tools/storage/storage_provider_factory.py
@@ -7,15 +7,15 @@ from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider
from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
from pipelex.tools.storage.local_storage_provider import LocalStorageProvider
from pipelex.tools.storage.s3_storage_provider import S3StorageProvider
-from pipelex.tools.storage.storage_config import StorageConfig, StorageMethod
+from pipelex.tools.storage.storage_config import StorageMethod, StorageProviderConfig
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
-def make_storage_provider_from_config(storage_config: StorageConfig) -> StorageProviderAbstract:
+def make_storage_provider_from_config(storage_provider_config: StorageProviderConfig) -> StorageProviderAbstract:
"""Create a storage provider based on the provided storage configuration.
Args:
- storage_config: The storage configuration specifying method and provider-specific settings.
+ storage_provider_config: The storage configuration specifying method and provider-specific settings.
Returns:
A configured storage provider instance.
@@ -23,34 +23,34 @@ def make_storage_provider_from_config(storage_config: StorageConfig) -> StorageP
Raises:
PipelexConfigError: If required provider-specific config is missing.
"""
- match storage_config.method:
+ match storage_provider_config.method:
case StorageMethod.LOCAL:
- log.verbose(f"Using local storage at: {storage_config.storage_path}")
- return LocalStorageProvider(root_path=Path(storage_config.storage_path))
+ log.verbose(f"Using local storage at: {storage_provider_config.storage_path}")
+ return LocalStorageProvider(root_path=Path(storage_provider_config.storage_path))
case StorageMethod.IN_MEMORY:
log.verbose("Using in-memory storage")
return InMemoryStorageProvider()
case StorageMethod.S3:
- if storage_config.s3 is None:
+ if storage_provider_config.s3 is None:
msg = "S3 config is required when method is s3"
raise StorageConfigError(msg)
- storage_config.s3.lazy_validate()
- log.verbose(f"Using S3 storage: bucket={storage_config.s3.bucket_name}, region={storage_config.s3.region}")
+ storage_provider_config.s3.lazy_validate()
+ log.verbose(f"Using S3 storage: bucket={storage_provider_config.s3.bucket_name}, region={storage_provider_config.s3.region}")
return S3StorageProvider(
- bucket_name=storage_config.s3.bucket_name,
- region=storage_config.s3.region,
- signed_urls_lifespan=storage_config.s3.signed_urls_lifespan,
+ bucket_name=storage_provider_config.s3.bucket_name,
+ region=storage_provider_config.s3.region,
+ signed_urls_lifespan=storage_provider_config.s3.signed_urls_lifespan,
)
case StorageMethod.GCP:
- if storage_config.gcp is None:
+ if storage_provider_config.gcp is None:
msg = "GCP config is required when method is gcp"
raise StorageConfigError(msg)
- storage_config.gcp.lazy_validate()
- log.verbose(f"Using GCP storage: bucket={storage_config.gcp.bucket_name}, project={storage_config.gcp.project_id}")
+ storage_provider_config.gcp.lazy_validate()
+ log.verbose(f"Using GCP storage: bucket={storage_provider_config.gcp.bucket_name}, project={storage_provider_config.gcp.project_id}")
return GcpStorageProvider(
- bucket_name=storage_config.gcp.bucket_name,
- project_id=storage_config.gcp.project_id,
+ bucket_name=storage_provider_config.gcp.bucket_name,
+ project_id=storage_provider_config.gcp.project_id,
credentials_file_path=get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH"),
- signed_urls_lifespan=storage_config.gcp.signed_urls_lifespan,
+ signed_urls_lifespan=storage_provider_config.gcp.signed_urls_lifespan,
)
diff --git a/pipelex/tools/typing/structure_printer.py b/pipelex/tools/typing/structure_printer.py
index c7e2489ff..ac45e775d 100644
--- a/pipelex/tools/typing/structure_printer.py
+++ b/pipelex/tools/typing/structure_printer.py
@@ -2,9 +2,9 @@ import types
from enum import Enum
from typing import Any, Union, get_args, get_origin, get_type_hints
-from kajson.kajson_manager import KajsonManager
from pydantic import BaseModel
+from pipelex.hub import get_class_registry
from pipelex.types import StrEnum
@@ -23,7 +23,7 @@ def _build_type_hints_namespace() -> dict[str, Any]:
"""
# Get all registered classes from the class registry
# The root attribute is a dict[str, type] but the type checker doesn't know this
- class_registry = KajsonManager.get_class_registry()
+ class_registry = get_class_registry()
root_dict: dict[str, Any] = getattr(class_registry, "root", {})
return dict(root_dict)
diff --git a/pipelex/tracing/__init__.py b/pipelex/tracing/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/tracing/buffering_event_log.py b/pipelex/tracing/buffering_event_log.py
new file mode 100644
index 000000000..95fd0c642
--- /dev/null
+++ b/pipelex/tracing/buffering_event_log.py
@@ -0,0 +1,55 @@
+"""In-memory buffering event log for use inside Temporal workflows.
+
+Collects trace events in a list with zero I/O. Events are flushed to the
+real backend (DynamoDB) via a Temporal activity after pipe execution.
+"""
+
+from typing_extensions import override
+
+from pipelex.tracing.event_log_protocol import EventLogProtocol
+from pipelex.tracing.trace_events import TraceEvent
+
+
+class BufferingEventLog(EventLogProtocol):
+ """Event log that buffers events in-memory for later flush via activity.
+
+ Use inside Temporal workflows where synchronous I/O is forbidden.
+ After pipe execution, call drain() to get the buffered events, then
+ flush them to the real backend via act_flush_trace_events.
+ """
+
+ def __init__(self) -> None:
+ self._buffer: list[TraceEvent] = []
+ self._sequence: int = 0
+
+ @override
+ def next_sequence(self) -> int:
+ """Return the next sequence number. Shared by all emitters."""
+ seq = self._sequence
+ self._sequence += 1
+ return seq
+
+ @override
+ def emit(self, event: TraceEvent) -> None:
+ """Append event to in-memory buffer. No I/O."""
+ self._buffer.append(event)
+
+ def drain(self) -> list[TraceEvent]:
+ """Return and clear all buffered events."""
+ events = list(self._buffer)
+ self._buffer.clear()
+ return events
+
+ @override
+ def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:
+ """Return buffered events matching the pipeline_run_id."""
+ return [event for event in self._buffer if event.pipeline_run_id == pipeline_run_id]
+
+ @override
+ def close(self) -> None:
+ """No-op."""
+
+ @override
+ def cleanup(self, pipeline_run_id: str) -> None:
+ """Remove buffered events for the given pipeline_run_id."""
+ self._buffer = [event for event in self._buffer if event.pipeline_run_id != pipeline_run_id]
diff --git a/pipelex/tracing/dynamodb_event_log.py b/pipelex/tracing/dynamodb_event_log.py
new file mode 100644
index 000000000..f95d81440
--- /dev/null
+++ b/pipelex/tracing/dynamodb_event_log.py
@@ -0,0 +1,150 @@
+"""DynamoDB-backed event log implementation.
+
+Stores trace events in a DynamoDB table with the schema:
+ PK: PIPELINE_RUN#{pipeline_run_id}
+ SK: EVENT#{workflow_id}#{sequence:010d}
+ payload: dict (full event via model_dump)
+
+Compatible with the pipelex-api-infra TraceEventDynamoDBAdapter schema.
+Requires: pip install "pipelex[dynamodb]"
+"""
+
+from typing import Any
+
+from pydantic import TypeAdapter, ValidationError
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.system.exceptions import MissingDependencyError
+from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
+from pipelex.tracing.event_log_protocol import EventLogProtocol
+from pipelex.tracing.trace_events import AnyTraceEvent, TraceEvent
+
+try:
+ import boto3
+ from boto3.dynamodb.conditions import Key as DynamoKey
+except ImportError:
+ boto3 = None # type: ignore[assignment]
+ DynamoKey = None # type: ignore[assignment, misc]
+
+_any_trace_event_adapter: TypeAdapter[TraceEvent] = TypeAdapter(AnyTraceEvent)
+
+
+class DynamoDBEventLog(EventLogProtocol):
+ """Event log backed by AWS DynamoDB.
+
+ Write path: PutItem per event, synchronous. DynamoDB's PK+SK uniqueness
+ provides natural deduplication for Temporal replay re-emissions.
+ Read path: Query on PK, returns events sorted by SK which encodes
+ (workflow_id, sequence) — matching the NDJSON ordering contract.
+
+ WARNING: Do not call emit() from inside a Temporal workflow body — the
+ synchronous boto3 HTTP call will block the workflow thread and trigger
+ the deadlock detector. Use BufferingEventLog + act_flush_trace_events instead.
+ """
+
+ def __init__(self, table_name: str, region: str) -> None:
+ if boto3 is None:
+ lib_name = "boto3"
+ lib_extra_name = "dynamodb"
+ msg = "boto3 is required for the DynamoDB event log backend."
+ raise MissingDependencyError(lib_name, lib_extra_name, msg)
+
+ self._table_name = table_name
+ self._region = region
+ self._sequence: int = 0
+ dynamodb: Any = boto3.resource("dynamodb", region_name=self._region) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
+ self._table: Any = dynamodb.Table(self._table_name) # pyright: ignore[reportUnknownMemberType]
+
+ @override
+ def next_sequence(self) -> int:
+ """Return the next sequence number. Shared by all emitters."""
+ seq = self._sequence
+ self._sequence += 1
+ return seq
+
+ @staticmethod
+ def _make_pk(pipeline_run_id: str) -> str:
+ return f"PIPELINE_RUN#{pipeline_run_id}"
+
+ @staticmethod
+ def _make_sk(workflow_id: str, sequence: int) -> str:
+ return f"EVENT#{workflow_id}#{sequence:010d}"
+
+ def _key_condition(self, pipeline_run_id: str) -> Any:
+ """Build a KeyConditionExpression for querying by pipeline_run_id."""
+ return DynamoKey("PK").eq(self._make_pk(pipeline_run_id)) # pyright: ignore[reportOptionalCall]
+
+ @override
+ def emit(self, event: TraceEvent) -> None:
+ """Write a single event to DynamoDB. Synchronous and idempotent."""
+ item: dict[str, Any] = {
+ "PK": self._make_pk(event.pipeline_run_id),
+ "SK": self._make_sk(event.workflow_id, event.sequence),
+ "pipeline_run_id": event.pipeline_run_id,
+ "workflow_id": event.workflow_id,
+ "sequence": event.sequence,
+ "payload": event.model_dump_json(),
+ }
+ self._table.put_item(Item=item)
+
+ @override
+ def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:
+ """Read all events for a pipeline run from DynamoDB."""
+ items: list[dict[str, Any]] = []
+
+ response = self._table.query(
+ KeyConditionExpression=self._key_condition(pipeline_run_id),
+ ScanIndexForward=True,
+ )
+ items.extend(response.get("Items", []))
+
+ while "LastEvaluatedKey" in response:
+ response = self._table.query(
+ KeyConditionExpression=self._key_condition(pipeline_run_id),
+ ScanIndexForward=True,
+ ExclusiveStartKey=response["LastEvaluatedKey"],
+ )
+ items.extend(response.get("Items", []))
+
+ events: list[TraceEvent] = []
+ for item in items:
+ payload = item.get("payload")
+ if payload is None:
+ log.warning(f"Skipping DynamoDB item with missing payload: PK={item.get('PK')}, SK={item.get('SK')}")
+ continue
+ try:
+ event = _any_trace_event_adapter.validate_json(payload)
+ except ValidationError as exc:
+ log.warning(f"Skipping unparseable DynamoDB item PK={item.get('PK')} SK={item.get('SK')}: {format_pydantic_validation_error(exc)}")
+ continue
+ events.append(event)
+
+ return events
+
+ @override
+ def close(self) -> None:
+ """No-op: boto3 resources are reusable across calls."""
+
+ @override
+ def cleanup(self, pipeline_run_id: str) -> None:
+ """Delete all events for a pipeline run. In practice TTL handles this."""
+ response = self._table.query(
+ KeyConditionExpression=self._key_condition(pipeline_run_id),
+ ProjectionExpression="PK, SK",
+ )
+ items: list[dict[str, Any]] = response.get("Items", [])
+ while "LastEvaluatedKey" in response:
+ response = self._table.query(
+ KeyConditionExpression=self._key_condition(pipeline_run_id),
+ ProjectionExpression="PK, SK",
+ ExclusiveStartKey=response["LastEvaluatedKey"],
+ )
+ items.extend(response.get("Items", []))
+
+ if not items:
+ return
+
+ with self._table.batch_writer() as writer:
+ for item in items:
+ writer.delete_item(Key={"PK": item["PK"], "SK": item["SK"]})
diff --git a/pipelex/tracing/event_log_factory.py b/pipelex/tracing/event_log_factory.py
new file mode 100644
index 000000000..b9f145a2b
--- /dev/null
+++ b/pipelex/tracing/event_log_factory.py
@@ -0,0 +1,33 @@
+"""Factory for creating event log backends from configuration."""
+
+from pipelex.base_exceptions import PipelexConfigError
+from pipelex.system.configuration.configs import TracingBackend, TracingConfig
+from pipelex.tracing.dynamodb_event_log import DynamoDBEventLog
+from pipelex.tracing.event_log_protocol import EventLogProtocol
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+
+
+def make_event_log(tracing_config: TracingConfig) -> EventLogProtocol:
+ """Create an event log backend from tracing configuration."""
+ match tracing_config.backend:
+ case TracingBackend.NDJSON:
+ if tracing_config.ndjson is None:
+ msg = "ndjson config is required when backend is 'ndjson'"
+ raise PipelexConfigError(msg)
+ return NdjsonEventLog(traces_dir=tracing_config.ndjson.traces_dir)
+ case TracingBackend.DYNAMODB:
+ if tracing_config.dynamodb is None:
+ msg = "dynamodb config is required when backend is 'dynamodb'"
+ raise PipelexConfigError(msg)
+ return DynamoDBEventLog(
+ table_name=tracing_config.dynamodb.table_name,
+ region=tracing_config.dynamodb.region,
+ )
+ case TracingBackend.TEMPORAL_DYNAMODB:
+ if tracing_config.temporal_dynamodb is None:
+ msg = "temporal_dynamodb config is required when backend is 'temporal_dynamodb'"
+ raise PipelexConfigError(msg)
+ return DynamoDBEventLog(
+ table_name=tracing_config.temporal_dynamodb.table_name,
+ region=tracing_config.temporal_dynamodb.region,
+ )
diff --git a/pipelex/tracing/event_log_protocol.py b/pipelex/tracing/event_log_protocol.py
new file mode 100644
index 000000000..fa70c90a4
--- /dev/null
+++ b/pipelex/tracing/event_log_protocol.py
@@ -0,0 +1,62 @@
+"""Protocol for trace event storage backends.
+
+Defines the interface that all event log implementations must satisfy.
+The NDJSON backend is the local implementation; cloud backends (e.g. DynamoDB)
+will implement the same protocol later.
+"""
+
+from typing import Protocol
+
+from pipelex.tracing.trace_events import TraceEvent
+
+
+class EventLogProtocol(Protocol):
+ """Storage backend for trace events.
+
+ Implementations must guarantee that emit() is synchronous and durable:
+ when emit() returns, the event must be persisted. This invariant ensures
+ all events from a child workflow are flushed before it returns to its parent.
+ """
+
+ def next_sequence(self) -> int:
+ """Return the next monotonically increasing sequence number.
+
+ All emitters sharing the same event log must use this method to
+ obtain sequence numbers, ensuring events never collide on the
+ (workflow_id, sequence) key regardless of storage backend.
+ """
+ ...
+
+ # TODO: wip - make it async
+ def emit(self, event: TraceEvent) -> None:
+ """Append a single event to the log.
+
+ Must be synchronous with an explicit flush. The event is durable
+ when this method returns.
+ """
+ ...
+
+ # TODO: wip - make it async
+ def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:
+ """Read all events for a pipeline run.
+
+ Returns events deduplicated by (workflow_id, event_type, sequence) and sorted
+ by (workflow_id, sequence) for deterministic ordering.
+ Returns an empty list if no events exist for the given run.
+ """
+ ...
+
+ def close(self) -> None:
+ """Close all held resources (file handles, connections).
+
+ Idempotent: safe to call multiple times. Does not delete data.
+ """
+ ...
+
+ def cleanup(self, pipeline_run_id: str) -> None:
+ """Remove all stored data for a pipeline run.
+
+ Releases any held resources (file handles, connections) and deletes
+ the underlying storage for the given run.
+ """
+ ...
diff --git a/pipelex/tracing/graphspec_assembler.py b/pipelex/tracing/graphspec_assembler.py
new file mode 100644
index 000000000..4e1a6eeb3
--- /dev/null
+++ b/pipelex/tracing/graphspec_assembler.py
@@ -0,0 +1,465 @@
+"""Reconstructs a GraphSpec from a flat list of trace events.
+
+This is the core algorithm for distributed tracing — equivalent to
+GraphTracer.teardown() but working from serialized events instead of
+in-memory state. Used after Temporal execution to assemble cross-worker
+graphs from NDJSON event files.
+"""
+
+from collections.abc import Sequence
+from datetime import datetime, timezone
+from typing import Any
+
+from pipelex import log
+from pipelex.graph.graphspec import (
+ EdgeKind,
+ EdgeSpec,
+ ErrorSpec,
+ GraphSpec,
+ IOSpec,
+ NodeIOSpec,
+ NodeKind,
+ NodeSpec,
+ NodeStatus,
+ PipelineRef,
+ TimingSpec,
+)
+from pipelex.tracing.trace_events import (
+ BatchAggregateEvent,
+ BatchItemEvent,
+ ControllerOutputEvent,
+ EdgeEvent,
+ ExecutionDataEvent,
+ ParallelCombineEvent,
+ PipeEndErrorEvent,
+ PipeEndSuccessEvent,
+ PipeStartEvent,
+ TraceEvent,
+ UsageReportEvent,
+)
+
+
+class _AssemblerNodeData:
+ """Internal mutable data structure for building a node from events.
+
+ Mirrors _MutableNodeData from graph_tracer.py.
+ """
+
+ def __init__(
+ self,
+ node_id: str,
+ pipe_code: str,
+ pipe_type: str,
+ node_kind: NodeKind,
+ started_at: datetime,
+ parent_node_id: str | None,
+ input_specs: list[IOSpec] | None = None,
+ ) -> None:
+ self.node_id = node_id
+ self.pipe_code = pipe_code
+ self.pipe_type = pipe_type
+ self.node_kind = node_kind
+ self.started_at = started_at
+ self.parent_node_id = parent_node_id
+ self.ended_at: datetime | None = None
+ self.status: NodeStatus = NodeStatus.RUNNING
+ self.metrics: dict[str, float] = {}
+ self.error: ErrorSpec | None = None
+ self.input_specs: list[IOSpec] = input_specs or []
+ self.output_specs: list[IOSpec] = []
+ self.execution_data: dict[str, Any] = {}
+
+ def to_node_spec(self) -> NodeSpec:
+ """Convert to immutable NodeSpec."""
+ assert self.started_at is not None
+ assert self.ended_at is not None
+ timing = TimingSpec(
+ started_at=self.started_at,
+ ended_at=self.ended_at,
+ )
+
+ node_io = NodeIOSpec(
+ inputs=self.input_specs,
+ outputs=list(self.output_specs),
+ )
+
+ return NodeSpec(
+ node_id=self.node_id,
+ kind=self.node_kind,
+ pipe_code=self.pipe_code,
+ pipe_type=self.pipe_type,
+ status=self.status,
+ timing=timing,
+ node_io=node_io,
+ error=self.error,
+ metrics=self.metrics,
+ execution_data=self.execution_data,
+ )
+
+
+class GraphSpecAssembler:
+ """Reconstructs a GraphSpec from trace events.
+
+ Two-pass algorithm:
+ - Pass 1: Build nodes, producer map, and collect metadata from all events.
+ - Pass 2: Generate DATA, BATCH_ITEM, BATCH_AGGREGATE, PARALLEL_COMBINE edges
+ using the complete producer map.
+ """
+
+ @staticmethod
+ def assemble(
+ events: Sequence[TraceEvent],
+ graph_id: str,
+ pipeline_ref: PipelineRef | None = None,
+ ) -> GraphSpec:
+ """Assemble a GraphSpec from a flat list of trace events.
+
+ Args:
+ events: Flat list of trace events, pre-sorted by (workflow_id, sequence).
+ graph_id: The graph identifier for the assembled GraphSpec.
+ pipeline_ref: Optional pipeline reference metadata.
+
+ Returns:
+ A complete GraphSpec with nodes and edges.
+ """
+ assembler = _AssemblerState(graph_id=graph_id, pipeline_ref=pipeline_ref or PipelineRef())
+ assembler.pass_one(events)
+ assembler.pass_two()
+ return assembler.build_graph_spec()
+
+
+class _AssemblerState:
+ """Mutable state for a single assembler invocation."""
+
+ def __init__(self, graph_id: str, pipeline_ref: PipelineRef) -> None:
+ self._graph_id = graph_id
+ self._pipeline_ref = pipeline_ref
+ self._earliest_timestamp: datetime | None = None
+
+ # Pass 1 accumulation
+ self._nodes: dict[str, _AssemblerNodeData] = {}
+ self._explicit_edges: list[EdgeSpec] = []
+ self._stuff_producer_map: dict[str, str] = {}
+ self._batch_item_map: dict[str, tuple[str | None, list[tuple[str, int]]]] = {}
+ self._batch_aggregate_map: dict[str, tuple[str | None, list[tuple[str, int]]]] = {}
+ self._parallel_combine_map: dict[str, tuple[str, list[tuple[str, str]]]] = {}
+
+ # Registries (accumulated from events, deduplicated)
+ self._pipe_registry: dict[str, dict[str, Any]] = {}
+ self._concept_registry: dict[str, dict[str, Any]] = {}
+
+ # Edge ID counter for assembler-generated edges
+ self._edge_sequence: int = 0
+ # Edges generated in Pass 2 (DATA, BATCH_ITEM, etc.)
+ self._generated_edges: list[EdgeSpec] = []
+
+ def pass_one(self, events: Sequence[TraceEvent]) -> None:
+ """Pass 1: Build nodes, producer map, and collect metadata."""
+ for event in events:
+ self._track_earliest_timestamp(event)
+
+ if isinstance(event, PipeStartEvent):
+ self._handle_pipe_start(event)
+ elif isinstance(event, PipeEndSuccessEvent):
+ self._handle_pipe_end_success(event)
+ elif isinstance(event, PipeEndErrorEvent):
+ self._handle_pipe_end_error(event)
+ elif isinstance(event, EdgeEvent):
+ self._handle_edge_event(event)
+ elif isinstance(event, ControllerOutputEvent):
+ self._handle_controller_output(event)
+ elif isinstance(event, BatchItemEvent):
+ self._handle_batch_item(event)
+ elif isinstance(event, BatchAggregateEvent):
+ self._handle_batch_aggregate(event)
+ elif isinstance(event, ParallelCombineEvent):
+ self._handle_parallel_combine(event)
+ elif isinstance(event, ExecutionDataEvent):
+ self._handle_execution_data(event)
+ elif isinstance(event, UsageReportEvent):
+ pass # Handled by UsageAggregator
+ else:
+ log.warning(f"Unknown event type: {type(event).__name__}")
+
+ # Mark any still-running nodes as CANCELED
+ self._mark_canceled_nodes()
+
+ def pass_two(self) -> None:
+ """Pass 2: Generate edges using the complete producer map."""
+ self._generate_data_edges()
+ self._generate_batch_item_edges()
+ self._generate_batch_aggregate_edges()
+ self._generate_parallel_combine_edges()
+
+ def build_graph_spec(self) -> GraphSpec:
+ """Build the final GraphSpec from accumulated state."""
+ nodes = [node_data.to_node_spec() for node_data in self._nodes.values()]
+ all_edges = self._explicit_edges + self._get_generated_edges()
+
+ return GraphSpec(
+ graph_id=self._graph_id,
+ created_at=self._earliest_timestamp or datetime.now(timezone.utc),
+ pipeline_ref=self._pipeline_ref,
+ nodes=nodes,
+ edges=all_edges,
+ pipe_registry=dict(self._pipe_registry),
+ concept_registry=dict(self._concept_registry),
+ )
+
+ # ------------------------------------------------------------------
+ # Pass 1 handlers
+ # ------------------------------------------------------------------
+
+ def _track_earliest_timestamp(self, event: TraceEvent) -> None:
+ if self._earliest_timestamp is None or event.timestamp < self._earliest_timestamp:
+ self._earliest_timestamp = event.timestamp
+
+ def _handle_pipe_start(self, event: PipeStartEvent) -> None:
+ node_data = _AssemblerNodeData(
+ node_id=event.node_id,
+ pipe_code=event.pipe_code,
+ pipe_type=event.pipe_type,
+ node_kind=event.node_kind,
+ started_at=event.timestamp,
+ parent_node_id=event.parent_node_id,
+ input_specs=list(event.input_specs),
+ )
+ self._nodes[event.node_id] = node_data
+
+ # Accumulate pipe and concept registry data (deduplicated)
+ if event.pipe_data:
+ pipe_ref = f"{event.pipe_data.get('domain_code', '')}.{event.pipe_data.get('code', '')}"
+ if pipe_ref not in self._pipe_registry:
+ self._pipe_registry[pipe_ref] = event.pipe_data
+ for concept_item in event.concept_data:
+ concept_ref = f"{concept_item.get('domain_code', '')}.{concept_item.get('code', '')}"
+ if concept_ref not in self._concept_registry:
+ self._concept_registry[concept_ref] = concept_item
+
+ def _handle_pipe_end_success(self, event: PipeEndSuccessEvent) -> None:
+ node_data = self._nodes.get(event.node_id)
+ if node_data is None:
+ log.warning(f"PipeEndSuccessEvent for unknown node: {event.node_id}")
+ return
+
+ node_data.ended_at = event.ended_at
+ node_data.status = NodeStatus.SUCCEEDED
+ if event.metrics:
+ node_data.metrics = event.metrics
+
+ # Accumulate output concept data (deduplicated)
+ if event.output_concept_data:
+ concept_ref = f"{event.output_concept_data.get('domain_code', '')}.{event.output_concept_data.get('code', '')}"
+ if concept_ref not in self._concept_registry:
+ self._concept_registry[concept_ref] = event.output_concept_data
+
+ # Pass-through detection (mirrors graph_tracer.py:476-488)
+ if event.output_spec is not None:
+ input_digests = {spec.digest for spec in node_data.input_specs if spec.digest is not None}
+ if event.output_spec.digest in input_digests:
+ # Pass-through: don't register as output or producer
+ pass
+ else:
+ node_data.output_specs.append(event.output_spec)
+ if event.output_spec.digest:
+ self._stuff_producer_map[event.output_spec.digest] = event.node_id
+
+ def _handle_pipe_end_error(self, event: PipeEndErrorEvent) -> None:
+ node_data = self._nodes.get(event.node_id)
+ if node_data is None:
+ log.warning(f"PipeEndErrorEvent for unknown node: {event.node_id}")
+ return
+
+ node_data.ended_at = event.ended_at
+ node_data.status = NodeStatus.FAILED
+ node_data.error = event.error
+
+ def _handle_edge_event(self, event: EdgeEvent) -> None:
+ # DATA, BATCH_ITEM, BATCH_AGGREGATE, PARALLEL_COMBINE are regenerated
+ # in pass 2 with full cross-worker visibility — skip to avoid duplicates.
+ match event.edge_kind:
+ case EdgeKind.CONTAINS | EdgeKind.SELECTED_OUTCOME | EdgeKind.CONTROL:
+ edge = EdgeSpec(
+ edge_id=event.edge_id,
+ source=event.source_node_id,
+ target=event.target_node_id,
+ kind=event.edge_kind,
+ label=event.label,
+ source_stuff_digest=event.source_stuff_digest,
+ target_stuff_digest=event.target_stuff_digest,
+ )
+ self._explicit_edges.append(edge)
+ case EdgeKind.DATA | EdgeKind.BATCH_ITEM | EdgeKind.BATCH_AGGREGATE | EdgeKind.PARALLEL_COMBINE:
+ pass # Regenerated in pass 2
+
+ def _handle_controller_output(self, event: ControllerOutputEvent) -> None:
+ node_data = self._nodes.get(event.node_id)
+ if node_data is None:
+ log.warning(f"ControllerOutputEvent for unknown node: {event.node_id}")
+ return
+
+ node_data.output_specs.append(event.output_spec)
+ if event.output_spec.digest:
+ self._stuff_producer_map[event.output_spec.digest] = event.node_id
+
+ def _handle_execution_data(self, event: ExecutionDataEvent) -> None:
+ node_data = self._nodes.get(event.node_id)
+ if node_data is None:
+ log.warning(f"ExecutionDataEvent for unknown node: {event.node_id}")
+ return
+ node_data.execution_data.update(event.execution_data)
+
+ def _handle_batch_item(self, event: BatchItemEvent) -> None:
+ if event.list_stuff_code not in self._batch_item_map:
+ self._batch_item_map[event.list_stuff_code] = (event.batch_controller_node_id, [])
+ _existing_controller_id, item_list = self._batch_item_map[event.list_stuff_code]
+ item_list.append((event.item_stuff_code, event.item_index))
+
+ def _handle_batch_aggregate(self, event: BatchAggregateEvent) -> None:
+ if event.output_list_stuff_code not in self._batch_aggregate_map:
+ self._batch_aggregate_map[event.output_list_stuff_code] = (event.batch_controller_node_id, [])
+ _existing_controller_id, item_list = self._batch_aggregate_map[event.output_list_stuff_code]
+ item_list.append((event.item_stuff_code, event.item_index))
+
+ def _handle_parallel_combine(self, event: ParallelCombineEvent) -> None:
+ # branch_producer_node_ids already snapshotted at emit time
+ self._parallel_combine_map[event.combined_stuff_code] = (
+ event.parallel_controller_node_id,
+ list(event.branch_producer_node_ids),
+ )
+
+ def _mark_canceled_nodes(self) -> None:
+ """Mark any still-RUNNING nodes as CANCELED (mirrors graph_tracer.py:160-164)."""
+ for node_data in self._nodes.values():
+ if node_data.status == NodeStatus.RUNNING:
+ node_data.status = NodeStatus.CANCELED
+ node_data.ended_at = datetime.now(timezone.utc)
+
+ # ------------------------------------------------------------------
+ # Pass 2 edge generation
+ # ------------------------------------------------------------------
+
+ def _make_edge_id(self) -> str:
+ edge_id = f"{self._graph_id}:asm_edge_{self._edge_sequence}"
+ self._edge_sequence += 1
+ return edge_id
+
+ def _get_generated_edges(self) -> list[EdgeSpec]:
+ """Return all edges generated in Pass 2 (stored via _add_generated_edge)."""
+ return self._generated_edges
+
+ def _add_generated_edge(
+ self,
+ source_node_id: str,
+ target_node_id: str,
+ edge_kind: EdgeKind,
+ label: str | None = None,
+ source_stuff_digest: str | None = None,
+ target_stuff_digest: str | None = None,
+ ) -> None:
+ edge = EdgeSpec(
+ edge_id=self._make_edge_id(),
+ source=source_node_id,
+ target=target_node_id,
+ kind=edge_kind,
+ label=label,
+ source_stuff_digest=source_stuff_digest,
+ target_stuff_digest=target_stuff_digest,
+ )
+ self._generated_edges.append(edge)
+
+ def _generate_data_edges(self) -> None:
+ """Generate DATA edges by correlating input digests with producer nodes.
+
+ Mirrors GraphTracer._generate_data_edges() (graph_tracer.py:199-222).
+ """
+ for consumer_node_id, node_data in self._nodes.items():
+ for input_spec in node_data.input_specs:
+ if input_spec.digest is None:
+ continue
+ producer_node_id = self._stuff_producer_map.get(input_spec.digest)
+ if producer_node_id is None:
+ continue
+ if producer_node_id == consumer_node_id:
+ continue
+ self._add_generated_edge(
+ source_node_id=producer_node_id,
+ target_node_id=consumer_node_id,
+ edge_kind=EdgeKind.DATA,
+ label=input_spec.name,
+ )
+
+ def _generate_batch_item_edges(self) -> None:
+ """Generate BATCH_ITEM edges for batch fan-out.
+
+ Mirrors GraphTracer._generate_batch_item_edges() (graph_tracer.py:224-267).
+ """
+ for list_stuff_code, (batch_controller_node_id, item_entries) in self._batch_item_map.items():
+ list_producer_node_id = self._stuff_producer_map.get(list_stuff_code)
+ source_node_id = batch_controller_node_id or list_producer_node_id
+
+ if not source_node_id:
+ # Legacy fallback: scan all nodes for a consumer of the list
+ for node_id, node_data in self._nodes.items():
+ for input_spec in node_data.input_specs:
+ if input_spec.digest == list_stuff_code:
+ source_node_id = node_id
+ break
+ if source_node_id:
+ break
+
+ if not source_node_id:
+ continue
+
+ for item_stuff_code, item_index in item_entries:
+ for consumer_node_id, node_data in self._nodes.items():
+ for input_spec in node_data.input_specs:
+ if input_spec.digest == item_stuff_code:
+ if source_node_id != consumer_node_id:
+ self._add_generated_edge(
+ source_node_id=source_node_id,
+ target_node_id=consumer_node_id,
+ edge_kind=EdgeKind.BATCH_ITEM,
+ label=f"[{item_index}]",
+ source_stuff_digest=list_stuff_code,
+ target_stuff_digest=item_stuff_code,
+ )
+ break
+
+ def _generate_batch_aggregate_edges(self) -> None:
+ """Generate BATCH_AGGREGATE edges from item producers to output list.
+
+ Mirrors GraphTracer._generate_batch_aggregate_edges() (graph_tracer.py:269-301).
+ """
+ for output_list_stuff_code, (batch_controller_node_id, item_entries) in self._batch_aggregate_map.items():
+ target_node_id = batch_controller_node_id or self._stuff_producer_map.get(output_list_stuff_code)
+ if not target_node_id:
+ continue
+
+ for item_stuff_code, item_index in item_entries:
+ item_producer_id = self._stuff_producer_map.get(item_stuff_code)
+ if item_producer_id and item_producer_id != target_node_id:
+ self._add_generated_edge(
+ source_node_id=item_producer_id,
+ target_node_id=target_node_id,
+ edge_kind=EdgeKind.BATCH_AGGREGATE,
+ label=f"[{item_index}]",
+ source_stuff_digest=item_stuff_code,
+ target_stuff_digest=output_list_stuff_code,
+ )
+
+ def _generate_parallel_combine_edges(self) -> None:
+ """Generate PARALLEL_COMBINE edges from branch producers to controller.
+
+ Mirrors GraphTracer._generate_parallel_combine_edges() (graph_tracer.py:303-321).
+ """
+ for combined_stuff_code, (parallel_controller_node_id, branch_entries) in self._parallel_combine_map.items():
+ for branch_stuff_code, branch_producer_id in branch_entries:
+ if branch_producer_id != parallel_controller_node_id:
+ self._add_generated_edge(
+ source_node_id=branch_producer_id,
+ target_node_id=parallel_controller_node_id,
+ edge_kind=EdgeKind.PARALLEL_COMBINE,
+ source_stuff_digest=branch_stuff_code,
+ target_stuff_digest=combined_stuff_code,
+ )
diff --git a/pipelex/tracing/in_memory_event_log.py b/pipelex/tracing/in_memory_event_log.py
new file mode 100644
index 000000000..639104e16
--- /dev/null
+++ b/pipelex/tracing/in_memory_event_log.py
@@ -0,0 +1,59 @@
+"""In-memory event log implementation for unit tests and direct mode."""
+
+from typing_extensions import override
+
+from pipelex.tracing.event_log_protocol import EventLogProtocol
+from pipelex.tracing.trace_events import TraceEvent
+
+
+class InMemoryEventLog(EventLogProtocol):
+ """Event log that stores events in a plain list.
+
+ Useful for unit tests and optional use in direct mode (single-process).
+ Provides the same deduplication and ordering guarantees as NdjsonEventLog.
+ """
+
+ def __init__(self) -> None:
+ self._events: list[TraceEvent] = []
+ self._sequence: int = 0
+
+ @override
+ def next_sequence(self) -> int:
+ """Return the next sequence number. Shared by all emitters."""
+ seq = self._sequence
+ self._sequence += 1
+ return seq
+
+ @override
+ def emit(self, event: TraceEvent) -> None:
+ """Append event to the in-memory list."""
+ self._events.append(event)
+
+ @override
+ def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:
+ """Return events for a pipeline run, deduplicated and sorted.
+
+ Deduplicates by (workflow_id, sequence), keeping the first occurrence.
+ Sorts by (workflow_id, sequence) for deterministic ordering.
+ """
+ filtered = [evt for evt in self._events if evt.pipeline_run_id == pipeline_run_id]
+
+ seen: set[tuple[str, str, int]] = set()
+ deduped: list[TraceEvent] = []
+ for event in filtered:
+ dedup_key = (event.workflow_id, type(event).__name__, event.sequence)
+ if dedup_key not in seen:
+ seen.add(dedup_key)
+ deduped.append(event)
+
+ deduped.sort(key=lambda evt: (evt.workflow_id, evt.sequence))
+ return deduped
+
+ @override
+ def close(self) -> None:
+ """No-op for in-memory implementation."""
+
+ @override
+ def cleanup(self, pipeline_run_id: str) -> None:
+ """Remove all events for the given pipeline run."""
+ self._events = [evt for evt in self._events if evt.pipeline_run_id != pipeline_run_id]
diff --git a/pipelex/tracing/ndjson_event_log.py b/pipelex/tracing/ndjson_event_log.py
new file mode 100644
index 000000000..00deb7e82
--- /dev/null
+++ b/pipelex/tracing/ndjson_event_log.py
@@ -0,0 +1,142 @@
+"""NDJSON file-based event log implementation.
+
+One JSON event per line, one file per workflow, organized by pipeline run:
+ {traces_dir}/{pipeline_run_id}/wf_{workflow_id}.ndjson
+"""
+
+import json
+import os
+import shutil
+from pathlib import Path
+from typing import IO
+
+from pydantic import TypeAdapter, ValidationError
+from typing_extensions import override
+
+from pipelex import log
+from pipelex.tracing.event_log_protocol import EventLogProtocol
+from pipelex.tracing.trace_events import AnyTraceEvent, TraceEvent
+
+_any_trace_event_adapter: TypeAdapter[TraceEvent] = TypeAdapter(AnyTraceEvent)
+
+
+class NdjsonEventLog(EventLogProtocol):
+ """Event log backed by NDJSON files on the local filesystem.
+
+ Write path: appends one JSON line per event, flushed immediately.
+ Read path: globs all .ndjson files in the run directory, parses,
+ deduplicates by (workflow_id, sequence), and sorts deterministically.
+ """
+
+ def __init__(self, traces_dir: str) -> None:
+ self._traces_dir = traces_dir
+ self._file_handles: dict[tuple[str, str], IO[str]] = {}
+ self._sequence: int = 0
+
+ @override
+ def next_sequence(self) -> int:
+ """Return the next sequence number. Shared by all emitters."""
+ seq = self._sequence
+ self._sequence += 1
+ return seq
+
+ # ------------------------------------------------------------------
+ # Write
+ # ------------------------------------------------------------------
+
+ @override
+ def emit(self, event: TraceEvent) -> None:
+ """Append event as a JSON line, flush immediately.
+
+ Creates the run directory on first write. Caches file handles
+ to avoid repeated open() calls for high-frequency events.
+ """
+ cache_key = (event.pipeline_run_id, event.workflow_id)
+ handle = self._file_handles.get(cache_key)
+
+ if handle is None:
+ run_dir = os.path.join(self._traces_dir, event.pipeline_run_id)
+ os.makedirs(run_dir, exist_ok=True)
+ file_path = os.path.join(run_dir, f"wf_{event.workflow_id}.ndjson")
+ handle = open(file_path, "a", encoding="utf-8") # noqa: SIM115
+ self._file_handles[cache_key] = handle
+
+ handle.write(event.model_dump_json() + "\n")
+ handle.flush()
+
+ # ------------------------------------------------------------------
+ # Read
+ # ------------------------------------------------------------------
+
+ @override
+ def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:
+ """Read all events for a pipeline run from NDJSON files.
+
+ Deduplicates by (workflow_id, sequence) to handle Temporal replay
+ re-emission. Sorts by (workflow_id, sequence) for deterministic ordering.
+ Corrupt lines (truncated JSON from crash-mid-write) are skipped with
+ a warning log.
+ """
+ run_dir = os.path.join(self._traces_dir, pipeline_run_id)
+ if not os.path.isdir(run_dir):
+ return []
+
+ ndjson_files = sorted(Path(run_dir).glob("*.ndjson"))
+
+ seen: set[tuple[str, str, int]] = set()
+ events: list[TraceEvent] = []
+
+ for ndjson_path in ndjson_files:
+ file_path = str(ndjson_path)
+ with open(file_path, encoding="utf-8") as fhandle:
+ for line_number, raw_line in enumerate(fhandle, start=1):
+ stripped = raw_line.strip()
+ if not stripped:
+ continue
+ try:
+ event = _any_trace_event_adapter.validate_json(stripped)
+ except (ValidationError, json.JSONDecodeError) as exc:
+ log.warning(f"Skipping corrupt line in {file_path}:{line_number} — {exc}")
+ continue
+
+ dedup_key = (event.workflow_id, type(event).__name__, event.sequence)
+ if dedup_key not in seen:
+ seen.add(dedup_key)
+ events.append(event)
+
+ # TODO: causal ordering — sorting by (workflow_id, sequence) groups by lexicographic
+ # workflow ID, not execution order. In parent/child workflow topologies this can cause
+ # incorrect producer map overwrites in GraphSpecAssembler. Consider timestamp-based
+ # or topology-aware ordering.
+ events.sort(key=lambda evt: (evt.workflow_id, evt.sequence))
+ return events
+
+ # ------------------------------------------------------------------
+ # Cleanup
+ # ------------------------------------------------------------------
+
+ @override
+ def cleanup(self, pipeline_run_id: str) -> None:
+ """Close cached file handles and remove the run directory."""
+ keys_to_remove = [key for key in self._file_handles if key[0] == pipeline_run_id]
+ for key in keys_to_remove:
+ self._file_handles[key].close()
+ del self._file_handles[key]
+
+ run_dir = os.path.join(self._traces_dir, pipeline_run_id)
+ if os.path.isdir(run_dir):
+ shutil.rmtree(run_dir)
+
+ @override
+ def close(self) -> None:
+ """Close all cached file handles."""
+ for handle in self._file_handles.values():
+ handle.close()
+ self._file_handles.clear()
+
+ def __del__(self) -> None:
+ try:
+ self.close()
+ except Exception: # noqa: S110
+ # Safety net during interpreter shutdown — logging may not be available
+ pass
diff --git a/pipelex/tracing/trace_events.py b/pipelex/tracing/trace_events.py
new file mode 100644
index 000000000..0445db9d3
--- /dev/null
+++ b/pipelex/tracing/trace_events.py
@@ -0,0 +1,177 @@
+"""Trace event models for distributed pipeline tracing.
+
+Defines serializable event types that capture the same information as
+GraphTracer's in-memory accumulation, but as immutable records suitable
+for NDJSON file storage and cross-worker graph assembly.
+"""
+
+from datetime import datetime
+from typing import Annotated, Any, Literal
+
+from pydantic import BaseModel, Field
+
+from pipelex.graph.graphspec import EdgeKind, ErrorSpec, IOSpec, NodeKind
+from pipelex.reporting.reporting_types import AnyTokensUsage
+from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
+from pipelex.types import StrEnum
+
+
+class TraceEventKind(StrEnum):
+ """Discriminator for trace event types."""
+
+ PIPE_START = "pipe_start"
+ PIPE_END_SUCCESS = "pipe_end_success"
+ PIPE_END_ERROR = "pipe_end_error"
+ EDGE = "edge"
+ CONTROLLER_OUTPUT = "controller_output"
+ BATCH_ITEM = "batch_item"
+ BATCH_AGGREGATE = "batch_aggregate"
+ PARALLEL_COMBINE = "parallel_combine"
+ EXECUTION_DATA = "execution_data"
+ USAGE_REPORT = "usage_report"
+
+
+class TraceEvent(BaseModel):
+ """Base model for all trace events.
+
+ Shared fields:
+ - pipeline_run_id: identifies the pipeline execution
+ - workflow_id: Temporal workflow ID or "direct" for in-process mode
+ - event_kind: discriminator for the event subclass (defined on each subclass as Literal)
+ - timestamp: UTC, for display/debugging only
+ - sequence: per-writer monotonic counter, for ordering and deduplication
+ """
+
+ pipeline_run_id: str
+ workflow_id: str
+ timestamp: datetime
+ sequence: int
+
+
+# ---------------------------------------------------------------------------
+# Event subclasses
+# ---------------------------------------------------------------------------
+
+
+class PipeStartEvent(TraceEvent):
+ """Emitted when a pipe begins execution."""
+
+ event_kind: Literal[TraceEventKind.PIPE_START] = TraceEventKind.PIPE_START
+ node_id: str
+ parent_node_id: str | None = None
+ pipe_code: str
+ pipe_type: str
+ node_kind: NodeKind
+ input_specs: list[IOSpec] = Field(default_factory=empty_list_factory_of(IOSpec))
+ pipe_data: dict[str, Any] = Field(default_factory=dict)
+ concept_data: list[dict[str, Any]] = Field(default_factory=empty_list_factory_of(dict))
+
+
+class PipeEndSuccessEvent(TraceEvent):
+ """Emitted when a pipe completes successfully."""
+
+ event_kind: Literal[TraceEventKind.PIPE_END_SUCCESS] = TraceEventKind.PIPE_END_SUCCESS
+ node_id: str
+ ended_at: datetime
+ output_spec: IOSpec | None = None
+ metrics: dict[str, float] = Field(default_factory=dict)
+ output_concept_data: dict[str, Any] = Field(default_factory=dict)
+
+
+class PipeEndErrorEvent(TraceEvent):
+ """Emitted when a pipe fails."""
+
+ event_kind: Literal[TraceEventKind.PIPE_END_ERROR] = TraceEventKind.PIPE_END_ERROR
+ node_id: str
+ ended_at: datetime
+ error: ErrorSpec
+
+
+class EdgeEvent(TraceEvent):
+ """Emitted when an edge is added (CONTAINS, SELECTED_OUTCOME, etc.)."""
+
+ event_kind: Literal[TraceEventKind.EDGE] = TraceEventKind.EDGE
+ edge_id: str
+ source_node_id: str
+ target_node_id: str
+ edge_kind: EdgeKind
+ label: str | None = None
+ source_stuff_digest: str | None = None
+ target_stuff_digest: str | None = None
+
+
+class ControllerOutputEvent(TraceEvent):
+ """Emitted when a controller registers an additional output."""
+
+ event_kind: Literal[TraceEventKind.CONTROLLER_OUTPUT] = TraceEventKind.CONTROLLER_OUTPUT
+ node_id: str
+ output_spec: IOSpec
+
+
+class BatchItemEvent(TraceEvent):
+ """Emitted when a batch item extraction is registered."""
+
+ event_kind: Literal[TraceEventKind.BATCH_ITEM] = TraceEventKind.BATCH_ITEM
+ list_stuff_code: str
+ item_stuff_code: str
+ item_index: int
+ batch_controller_node_id: str
+
+
+class BatchAggregateEvent(TraceEvent):
+ """Emitted when a batch aggregation is registered."""
+
+ event_kind: Literal[TraceEventKind.BATCH_AGGREGATE] = TraceEventKind.BATCH_AGGREGATE
+ output_list_stuff_code: str
+ item_stuff_code: str
+ item_index: int
+ batch_controller_node_id: str
+
+
+class ParallelCombineEvent(TraceEvent):
+ """Emitted when a parallel combine is registered.
+
+ branch_producer_node_ids contains (branch_stuff_code, producer_node_id) pairs
+ snapshotted from _stuff_producer_map before register_controller_output overrides it.
+ """
+
+ event_kind: Literal[TraceEventKind.PARALLEL_COMBINE] = TraceEventKind.PARALLEL_COMBINE
+ combined_stuff_code: str
+ branch_stuff_codes: list[str]
+ parallel_controller_node_id: str
+ branch_producer_node_ids: list[tuple[str, str]]
+
+
+class ExecutionDataEvent(TraceEvent):
+ """Emitted when a pipe registers execution metadata (rendered prompts, resolved models, etc.)."""
+
+ event_kind: Literal[TraceEventKind.EXECUTION_DATA] = TraceEventKind.EXECUTION_DATA
+ node_id: str
+ execution_data: dict[str, Any] = Field(default_factory=dict)
+
+
+class UsageReportEvent(TraceEvent):
+ """Emitted when an inference job reports token usage."""
+
+ event_kind: Literal[TraceEventKind.USAGE_REPORT] = TraceEventKind.USAGE_REPORT
+ node_id: str
+ tokens_usage: AnyTokensUsage
+
+
+# ---------------------------------------------------------------------------
+# Discriminated union for deserialization
+# ---------------------------------------------------------------------------
+
+AnyTraceEvent = Annotated[
+ PipeStartEvent
+ | PipeEndSuccessEvent
+ | PipeEndErrorEvent
+ | EdgeEvent
+ | ControllerOutputEvent
+ | BatchItemEvent
+ | BatchAggregateEvent
+ | ParallelCombineEvent
+ | ExecutionDataEvent
+ | UsageReportEvent,
+ Field(discriminator="event_kind"),
+]
diff --git a/pipelex/tracing/usage_aggregator.py b/pipelex/tracing/usage_aggregator.py
new file mode 100644
index 000000000..5f04ee3d6
--- /dev/null
+++ b/pipelex/tracing/usage_aggregator.py
@@ -0,0 +1,22 @@
+"""Aggregates usage report events from a trace event stream."""
+
+from collections.abc import Sequence
+
+from pipelex.reporting.reporting_types import AnyTokensUsage
+from pipelex.tracing.trace_events import TraceEvent, UsageReportEvent
+
+
+class UsageAggregator:
+ """Extracts token usage records from a flat list of trace events."""
+
+ @staticmethod
+ def aggregate(events: Sequence[TraceEvent]) -> list[AnyTokensUsage]:
+ """Collect all UsageReportEvent token usage records, preserving order.
+
+ Args:
+ events: Flat list of trace events (any types mixed).
+
+ Returns:
+ List of token usage records from UsageReportEvent entries only.
+ """
+ return [evt.tokens_usage for evt in events if isinstance(evt, UsageReportEvent)]
diff --git a/pipelex/urls.py b/pipelex/urls.py
index 14c4c3778..a6402bda7 100644
--- a/pipelex/urls.py
+++ b/pipelex/urls.py
@@ -32,3 +32,11 @@ class URLs:
png_example_3 = "https://pipelex-pytest-assets.s3.eu-west-3.amazonaws.com/png_example_3.png"
txt_example = "https://pipelex-pytest-assets.s3.eu-west-3.amazonaws.com/txt_example.txt"
+
+ openai_billing = "https://platform.openai.com/account/billing"
+ anthropic_billing = "https://platform.claude.com/settings/billing"
+ google_billing = "https://console.cloud.google.com/billing"
+ mistral_billing = "https://admin.mistral.ai/organization/billing"
+ aws_billing = "https://console.aws.amazon.com/billing"
+ linkup_billing = "https://app.linkup.so/organization/billing"
+ fal_billing = "https://fal.ai/dashboard/usage-billing/billing"
diff --git a/pyproject.toml b/pyproject.toml
index 77125f570..1d90e6acd 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -21,12 +21,13 @@ classifiers = [
dependencies = [
"aiofiles>=23.2.1",
"backports.strenum>=1.3.0 ; python_version < '3.11'",
+ "datamodel-code-generator>=0.55.0",
"filetype>=1.2.0",
"httpx>=0.23.0,<1.0.0",
- "instructor>=1.8.3,!=1.11.*,!=1.12.*", # 1.11.x caused typing errors with mypy
+ "instructor>=1.8.3,!=1.11.*,!=1.12.*", # 1.11.x caused typing errors with mypy
"jinja2>=3.1.4",
"json2html>=1.3.0",
- "kajson==0.3.1",
+ "kajson @ git+https://github.com/Pipelex/kajson.git@fix/enum-source-registry",
"markdown>=3.6",
"mthds>=0.3.0",
"networkx>=3.4.2",
@@ -43,7 +44,7 @@ dependencies = [
"pydantic>=2.10.6,<3.0.0",
"python-dotenv>=1.0.1",
"PyYAML>=6.0.2",
- "reportlab>=4.0,<5", # for generating synthetic PDFs
+ "reportlab>=4.0,<5", # for generating synthetic PDFs
"rich>=13.8.1",
"semantic-version>=2.10.0",
"shortuuid>=1.0.13",
@@ -84,7 +85,9 @@ google-genai = [
huggingface = ["huggingface_hub>=0.23,<1.0.0"]
linkup = ["linkup-sdk>=0.12.0"]
mistralai = ["mistralai>=1.12.0"]
+dynamodb = ["boto3>=1.34.131"]
s3 = ["boto3>=1.34.131", "aioboto3>=13.4.0"]
+temporal = ["temporalio==1.23.0", "aiohttp>=3.9.0"]
docs = [
"mike>=2.1.3",
"mkdocs>=1.6.1",
@@ -98,7 +101,7 @@ docs = [
dev = [
"boto3-stubs>=1.35.24",
"cryptography>=46.0.7",
- "moto[s3]>=5.0.0",
+ "moto[dynamodb,s3]>=5.0.0",
"mypy==1.19.1",
"pipelex-tools>=0.3.2",
"pyright==1.1.408",
@@ -108,6 +111,7 @@ dev = [
"pytest-cov>=6.1.1",
"pytest-mock>=3.14.0",
"pytest-sugar>=1.0.0",
+ "pytest-timeout>=2.3.1",
"pytest-xdist>= 3.6.1",
"ruff==0.14.13",
"types-aioboto3[bedrock,bedrock-runtime]>=13.4.0",
@@ -269,6 +273,7 @@ markers = [
"codex_disabled: tests that should not run in Codex",
"dry_runnable: tests that can be run in dry-run mode",
"pipelex_api: tests that require access to the Pipelex API",
+ "temporal: tests that require a Temporal server",
]
[tool.coverage.run]
diff --git a/tests/conftest.py b/tests/conftest.py
index d0249b6ad..c48204fce 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,5 +1,6 @@
from collections.abc import Callable, Generator
from pathlib import Path
+from typing import TYPE_CHECKING
import pytest
import shortuuid
@@ -18,9 +19,11 @@ from pipelex.system.pipelex_service.pipelex_service_config import (
from pipelex.system.pipelex_service.remote_config import RemoteConfig
from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher
from pipelex.system.runtime import IntegrationMode, runtime_manager
-from pipelex.system.telemetry.telemetry_manager import TelemetryManager
from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract
+if TYPE_CHECKING:
+ from pipelex.system.telemetry.telemetry_manager import TelemetryManager
+
pytest_plugins = [
"pipelex.test_extras.shared_pytest_plugins",
]
@@ -54,7 +57,7 @@ def _cached_load_pipelex_service_config(config_dir: Path) -> PipelexServiceConfi
return _pipelex_service_config_cache[config_dir]
-def _fast_telemetry_teardown(self: TelemetryManager) -> None:
+def _fast_telemetry_teardown(self: "TelemetryManager") -> None:
"""Skip expensive OTel/PostHog shutdown during tests (~0.49s per call).
Preserves exception capture cleanup (restores sys.excepthook) and
@@ -89,6 +92,8 @@ def cache_configs_for_session(session_mocker: MockerFixture):
_cached_load_pipelex_service_config,
)
# Skip expensive telemetry shutdown (OTel + PostHog flush) during tests
+ from pipelex.system.telemetry.telemetry_manager import TelemetryManager # noqa: PLC0415
+
session_mocker.patch.object(TelemetryManager, "teardown", _fast_telemetry_teardown)
diff --git a/tests/integration/pipelex/cogt/test_content_generator.py b/tests/integration/pipelex/cogt/test_content_generator.py
index c6f6b4da3..bb17dd3ce 100644
--- a/tests/integration/pipelex/cogt/test_content_generator.py
+++ b/tests/integration/pipelex/cogt/test_content_generator.py
@@ -114,12 +114,13 @@ class TestContentGenerator:
pretty_print(image, title=f"ContentGenerator: Image generated by '{img_gen_handle}' for '{positive_text}'")
assert isinstance(image, ImageContent)
- async def test_make_templated_text(self, content_generator: ContentGeneratorProtocol):
+ async def test_make_templated_text(self, job_metadata: JobMetadata, content_generator: ContentGeneratorProtocol):
context = {
"the_answer": "elementary, my dear Watson",
}
jinja2_text: str = await content_generator.make_templated_text(
+ job_metadata=job_metadata,
context=context,
template="The answer is: {{ the_answer }}",
)
diff --git a/tests/integration/pipelex/cogt/test_llm_generate_security.py b/tests/integration/pipelex/cogt/test_llm_generate_security.py
new file mode 100644
index 000000000..91562733b
--- /dev/null
+++ b/tests/integration/pipelex/cogt/test_llm_generate_security.py
@@ -0,0 +1,66 @@
+"""Integration tests at the llm_gen_object / llm_gen_object_list boundary: prove the
+security check fires BEFORE any LLM call is made, so a malicious schema never even
+touches the inference path. The LLM worker is mocked via pytest-mock — no inference
+marker, runs in CI.
+"""
+
+from typing import Any
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.content_generation.assignment_models import LLMAssignment, ObjectAssignment
+from pipelex.cogt.content_generation.exceptions import UnsafeSchemaError
+from pipelex.cogt.content_generation.llm_generate import llm_gen_object, llm_gen_object_list
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.pipeline.job_metadata import JobMetadata
+
+
+def _make_stub_llm_assignment() -> LLMAssignment:
+ return LLMAssignment(
+ job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),
+ llm_setting=LLMSetting(model="test-model", temperature=0.7),
+ llm_prompt=LLMPrompt(user_text="test prompt"),
+ )
+
+
+def _malicious_schema() -> dict[str, Any]:
+ return {
+ "title": "Innocent",
+ "type": "object",
+ "properties": {"hit": {"$ref": "#/$defs/Run"}},
+ "$defs": {"Run": {"x-python-import": {"module": "subprocess", "name": "run"}}},
+ }
+
+
+class TestLLMGenerateSecurity:
+ @pytest.mark.asyncio
+ async def test_llm_gen_object_rejects_unsafe_schema_before_calling_worker(self, mocker: MockerFixture) -> None:
+ """A malicious ObjectAssignment must raise UnsafeSchemaError before the LLM is invoked.
+
+ This is the critical contract: a tampered schema must never reach the LLM call
+ path, let alone consume inference budget.
+ """
+ mock_get_worker = mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker")
+ assignment = ObjectAssignment(
+ object_class_name="Innocent",
+ object_class_schema=_malicious_schema(),
+ llm_assignment_for_object=_make_stub_llm_assignment(),
+ )
+ with pytest.raises(UnsafeSchemaError):
+ await llm_gen_object(assignment)
+ mock_get_worker.return_value.gen_object.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_llm_gen_object_list_rejects_unsafe_schema_before_calling_worker(self, mocker: MockerFixture) -> None:
+ """Same contract for the list variant."""
+ mock_get_worker = mocker.patch("pipelex.cogt.content_generation.llm_generate.get_llm_worker")
+ assignment = ObjectAssignment(
+ object_class_name="Innocent",
+ object_class_schema=_malicious_schema(),
+ llm_assignment_for_object=_make_stub_llm_assignment(),
+ )
+ with pytest.raises(UnsafeSchemaError):
+ await llm_gen_object_list(assignment)
+ mock_get_worker.return_value.gen_object.assert_not_called()
diff --git a/tests/integration/pipelex/fixtures/generator_fixtures.py b/tests/integration/pipelex/fixtures/generator_fixtures.py
index 125af854e..5f6d53d27 100644
--- a/tests/integration/pipelex/fixtures/generator_fixtures.py
+++ b/tests/integration/pipelex/fixtures/generator_fixtures.py
@@ -20,38 +20,38 @@ def generated_content_factory(tmp_path: Path) -> GeneratedContentFactory:
Applies test overrides for bucket names and signed URL lifespans.
"""
- storage_config = get_config().pipelex.storage_config
- match storage_config.method:
+ storage_provider_config = get_config().pipelex.storage_config
+ match storage_provider_config.method:
case StorageMethod.S3:
- assert storage_config.s3 is not None
- patched_s3_config = storage_config.s3.model_copy(
+ assert storage_provider_config.s3 is not None
+ patched_s3_config = storage_provider_config.s3.model_copy(
update={
"bucket_name": S3_TEST_BUCKET,
"signed_urls_lifespan_seconds": TEST_SIGNED_URLS_LIFESPAN,
}
)
- patched_storage_config = storage_config.model_copy(update={"s3": patched_s3_config})
+ patched_provider_config = storage_provider_config.model_copy(update={"s3": patched_s3_config})
case StorageMethod.GCP:
- assert storage_config.gcp is not None
- patched_gcp_config = storage_config.gcp.model_copy(
+ assert storage_provider_config.gcp is not None
+ patched_gcp_config = storage_provider_config.gcp.model_copy(
update={
"bucket_name": GCP_TEST_BUCKET,
"signed_urls_lifespan_seconds": TEST_SIGNED_URLS_LIFESPAN,
}
)
- patched_storage_config = storage_config.model_copy(update={"gcp": patched_gcp_config})
+ patched_provider_config = storage_provider_config.model_copy(update={"gcp": patched_gcp_config})
case StorageMethod.LOCAL:
- assert storage_config.local is not None
- patched_local_config = storage_config.local.model_copy(
+ assert storage_provider_config.local is not None
+ patched_local_config = storage_provider_config.local.model_copy(
update={
"local_storage_path": tmp_path,
}
)
- patched_storage_config = storage_config.model_copy(update={"local": patched_local_config})
+ patched_provider_config = storage_provider_config.model_copy(update={"local": patched_local_config})
case StorageMethod.IN_MEMORY:
- patched_storage_config = storage_config
+ patched_provider_config = storage_provider_config
- storage_provider = make_storage_provider_from_config(patched_storage_config)
+ storage_provider = make_storage_provider_from_config(patched_provider_config)
return GeneratedContentFactory(storage_provider=storage_provider)
diff --git a/tests/integration/pipelex/fixtures/pipe_job_helpers.py b/tests/integration/pipelex/fixtures/pipe_job_helpers.py
new file mode 100644
index 000000000..fb0b10f3b
--- /dev/null
+++ b/tests/integration/pipelex/fixtures/pipe_job_helpers.py
@@ -0,0 +1,109 @@
+import uuid
+from collections.abc import Callable, Generator
+from pathlib import Path
+
+from pipelex.core.interpreter.interpreter import PipelexInterpreter
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_abstract import PipeAbstract
+from pipelex.hub import get_library_manager, get_report_delegate, get_required_pipe, set_current_library, teardown_current_library
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.pipeline.pipeline_models import SpecialPipelineId
+from pipelex.system.telemetry.otel_constants import OTelConstants
+
+PIPE_JOB_HELPERS_PIPELINE_RUN_ID_PREFIX = "pipe_job_helpers_test"
+
+
+def build_pipe_job(
+ pipe: PipeAbstract,
+ library_crate: LibraryCrate | None,
+ pipe_run_mode: PipeRunMode = PipeRunMode.DRY,
+ pipeline_run_id: str | None = None,
+) -> PipeJob:
+ """Build a PipeJob with empty working memory and default metadata."""
+ resolved_pipeline_run_id = SpecialPipelineId.DRY_RUN_UNTITLED if pipe_run_mode.is_dry else (pipeline_run_id or _make_unique_pipeline_run_id())
+ return PipeJobFactory.make_pipe_job(
+ pipe=pipe,
+ pipe_run_params=PipeRunParamsFactory.make_run_params(pipe_run_mode=pipe_run_mode),
+ job_metadata=JobMetadata(user_id=OTelConstants.DEFAULT_USER_ID, pipeline_run_id=resolved_pipeline_run_id),
+ working_memory=WorkingMemoryFactory.make_empty(),
+ library_crate=library_crate,
+ )
+
+
+def _make_unique_pipeline_run_id() -> str:
+ return f"{PIPE_JOB_HELPERS_PIPELINE_RUN_ID_PREFIX}_{uuid.uuid4().hex[:8]}"
+
+
+def pipe_job_from_library(
+ load_fn: Callable[[str], None],
+ pipe_code: str,
+ pipe_run_mode: PipeRunMode = PipeRunMode.DRY,
+ isolated_registry: bool = False,
+) -> Generator[PipeJob, None, None]:
+ """Shared fixture skeleton: open library, load via load_fn, build PipeJob, yield, teardown.
+
+ Args:
+ load_fn: Callable that loads content into the library.
+ pipe_code: The pipe code to look up after loading.
+ pipe_run_mode: Dry, mock, or live execution mode.
+ isolated_registry: When True, gives the library a scoped ClassRegistry so dynamic
+ concept classes are registered there instead of the global KajsonManager
+ registry. This simulates a clean worker process where the global registry
+ has no dynamic classes, forcing the deferred hydration path.
+ """
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+ if isolated_registry:
+ from kajson.class_registry import ClassRegistry # noqa: PLC0415
+ from kajson.kajson_manager import KajsonManager # noqa: PLC0415
+
+ # Pre-seed from global so core framework classes (PipeSequenceFactory, etc.)
+ # are available. Dynamic concept classes added during load_fn() will go to
+ # this scoped registry instead of the global, keeping the global clean.
+ global_registry = KajsonManager.get_class_registry()
+ scoped_registry = ClassRegistry()
+ if isinstance(global_registry, ClassRegistry):
+ scoped_registry.register_classes_dict(dict(global_registry.root))
+ library.set_class_registry(scoped_registry)
+ set_current_library(library_id=library_id)
+
+ # Open a reporting registry for live mode (required by the reporting manager)
+ needs_registry = not pipe_run_mode.is_dry
+ pipeline_run_id = _make_unique_pipeline_run_id()
+ if needs_registry:
+ get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)
+
+ try:
+ load_fn(library_id)
+
+ pipe = get_required_pipe(pipe_code=pipe_code)
+ library_crate = library_manager.get_crate(library_id=library_id)
+ pipe_job = build_pipe_job(pipe=pipe, library_crate=library_crate, pipe_run_mode=pipe_run_mode, pipeline_run_id=pipeline_run_id)
+
+ yield pipe_job
+ finally:
+ if needs_registry:
+ get_report_delegate().close_registry(pipeline_run_id=pipeline_run_id)
+ library_manager.teardown(library_id=library_id)
+ teardown_current_library()
+
+
+def pipe_job_from_bundle(
+ bundle_file: str,
+ pipe_code: str,
+ pipe_run_mode: PipeRunMode = PipeRunMode.DRY,
+ isolated_registry: bool = False,
+) -> Generator[PipeJob, None, None]:
+ """Build a PipeJob by loading a bundle from an mthds file."""
+
+ def _load(library_id: str) -> None:
+ mthds_content = Path(bundle_file).read_text(encoding="utf-8")
+ blueprint = PipelexInterpreter.make_pipelex_bundle_blueprint(mthds_content=mthds_content)
+ get_library_manager().load_from_blueprints(library_id=library_id, blueprints=[blueprint])
+
+ yield from pipe_job_from_library(_load, pipe_code=pipe_code, pipe_run_mode=pipe_run_mode, isolated_registry=isolated_registry)
diff --git a/tests/integration/pipelex/libraries/test_load_from_crate.py b/tests/integration/pipelex/libraries/test_load_from_crate.py
new file mode 100644
index 000000000..a0ad212e9
--- /dev/null
+++ b/tests/integration/pipelex/libraries/test_load_from_crate.py
@@ -0,0 +1,107 @@
+import tempfile
+from collections.abc import Callable
+from pathlib import Path
+
+from pipelex.core.interpreter.interpreter import PipelexInterpreter
+from pipelex.hub import get_library_manager
+from pipelex.libraries.library_crate_factory import LibraryCrateFactory
+
+SCORING_MTHDS = """\
+domain = "scoring"
+description = "Scoring domain"
+system_prompt = "You are a scoring assistant."
+main_pipe = "compute_score"
+
+[concept]
+ScoreResult = "A scoring result"
+
+[pipe.compute_score]
+type = "PipeLLM"
+description = "Compute a score"
+inputs = { data = "Text" }
+output = "ScoreResult"
+model = "$quick-reasoning"
+prompt = "Compute score from @data"
+"""
+
+ANALYTICS_MTHDS = """\
+domain = "analytics"
+description = "Analytics domain"
+
+[concept]
+AnalyticsResult = "An analytics result"
+
+[pipe.analyze]
+type = "PipeLLM"
+description = "Analyze data"
+inputs = { data = "Text" }
+output = "AnalyticsResult"
+model = "$quick-reasoning"
+prompt = "Analyze @data"
+"""
+
+
+class TestLoadFromCrate:
+ """Integration: crate path produces identical library content to blueprint path."""
+
+ def test_crate_equivalence(self, load_empty_library: Callable[[], str]):
+ """load_from_crate produces the same concept_refs, pipe_refs, and domain codes
+ as load_from_blueprints for the same input bundles.
+ """
+ library_manager = get_library_manager()
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ scoring_path = Path(tmp_dir) / "scoring.mthds"
+ analytics_path = Path(tmp_dir) / "analytics.mthds"
+ scoring_path.write_text(SCORING_MTHDS, encoding="utf-8")
+ analytics_path.write_text(ANALYTICS_MTHDS, encoding="utf-8")
+
+ # Parse blueprints
+ blueprints = [
+ PipelexInterpreter.make_pipelex_bundle_blueprint(bundle_path=scoring_path),
+ PipelexInterpreter.make_pipelex_bundle_blueprint(bundle_path=analytics_path),
+ ]
+
+ # Path A: load_from_blueprints (existing path)
+ lib_id_a = load_empty_library()
+ library_manager.load_from_blueprints(library_id=lib_id_a, blueprints=blueprints)
+ library_a = library_manager.get_library(library_id=lib_id_a)
+
+ # Collect refs from library A
+ concept_refs_a = set(library_a.concept_library.root.keys())
+ pipe_refs_a = set(library_a.pipe_library.root.keys())
+ domain_codes_a = set(library_a.domain_library.root.keys())
+
+ # Tear down library A
+ library_manager.teardown(library_id=lib_id_a)
+
+ # Path B: build crate then load_from_crate
+ crate = LibraryCrateFactory.make_from_blueprints(blueprints=blueprints)
+
+ lib_id_b = load_empty_library()
+ library_manager.load_from_crate(library_id=lib_id_b, crate=crate)
+ library_b = library_manager.get_library(library_id=lib_id_b)
+
+ # Collect refs from library B
+ concept_refs_b = set(library_b.concept_library.root.keys())
+ pipe_refs_b = set(library_b.pipe_library.root.keys())
+ domain_codes_b = set(library_b.domain_library.root.keys())
+
+ # Assert equivalence
+ assert concept_refs_a == concept_refs_b, f"Concept refs differ: {concept_refs_a} vs {concept_refs_b}"
+ assert pipe_refs_a == pipe_refs_b, f"Pipe refs differ: {pipe_refs_a} vs {pipe_refs_b}"
+ assert domain_codes_a == domain_codes_b, f"Domain codes differ: {domain_codes_a} vs {domain_codes_b}"
+
+ def test_crate_preserves_main_pipe(self):
+ """LibraryCrate preserves main_pipe from bundle domain metadata."""
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ scoring_path = Path(tmp_dir) / "scoring.mthds"
+ scoring_path.write_text(SCORING_MTHDS, encoding="utf-8")
+
+ blueprints = [
+ PipelexInterpreter.make_pipelex_bundle_blueprint(bundle_path=scoring_path),
+ ]
+
+ crate = LibraryCrateFactory.make_from_blueprints(blueprints=blueprints)
+ assert "scoring" in crate.domains
+ assert crate.domains["scoring"].main_pipe == "compute_score"
diff --git a/tests/integration/pipelex/temporal/conftest.py b/tests/integration/pipelex/temporal/conftest.py
new file mode 100644
index 000000000..0495f4b00
--- /dev/null
+++ b/tests/integration/pipelex/temporal/conftest.py
@@ -0,0 +1,161 @@
+from typing import AsyncGenerator, Generator, cast
+
+import pytest
+import pytest_asyncio
+from pytest import FixtureRequest, Parser
+from temporalio.client import Client as TemporalClient
+from temporalio.testing import WorkflowEnvironment
+
+from pipelex.pipelex import Pipelex
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+from pipelex.temporal.tasks import Tasks
+from pipelex.temporal.temporal_connect import connect_to_temporal_selected_server
+from pipelex.temporal.temporal_data_converter import data_converter
+from pipelex.temporal.temporal_hub import temporal_hub
+from pipelex.temporal.temporal_task_manager import TemporalTaskManager
+from pipelex.test_extras.shared_pytest_plugins import ClassRegistryMode
+
+TEMPORAL_SERVER_NONE = "none"
+TEMPORAL_SERVER_TIME_SKIPPING = "time-skipping"
+
+
+def pytest_addoption(parser: Parser) -> None:
+ parser.addoption(
+ "--temporal-server",
+ default=TEMPORAL_SERVER_NONE,
+ help="Which temporal server to use ('none' for in-process, 'time-skipping', or a profile name from temporal_server_configs)",
+ )
+
+
[email protected](scope="module", autouse=True)
+def reset_pipelex_config_fixture() -> Generator[None, None, None]:
+ """Override root conftest's fixture for Temporal tests.
+
+ Always initializes Pipelex per module so that logging and config are available
+ for boot_temporal and other fixtures, regardless of --temporal-server mode.
+ """
+ Pipelex.make(
+ integration_mode=IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST,
+ )
+ yield
+ Pipelex.teardown_if_needed()
+
+
[email protected](scope="module", autouse=True)
+def boot_temporal(reset_pipelex_config_fixture: None) -> Generator[None, None, None]: # noqa: ARG001
+ """Boot the temporal layer for temporal tests.
+
+ Depends on reset_pipelex_config_fixture to ensure Pipelex is initialized
+ before we access logging or config. Creates a TemporalTaskManager, populates
+ the task catalog, and registers it on the temporal_hub so that
+ get_task_manager() works.
+ """
+ manager = TemporalTaskManager()
+ temporal_hub.set_task_manager(manager)
+ manager.complement_catalog(
+ extra_catalog=Tasks.TASK_PACKS,
+ extra_workflows=[],
+ extra_activities=[],
+ )
+ manager.setup()
+
+ # Install Temporal-aware pipe routers and content generator so that:
+ # 1. Sub-pipes dispatch as child workflows (not inline in the sandbox)
+ # 2. Inference calls (LLM, img_gen, etc.) dispatch as activities (not inline)
+ # This mirrors what a full Temporal-enabled Pipelex.make() would set up.
+ from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory # noqa: PLC0415
+ from pipelex.hub import get_pipelex_hub, get_storage_provider # noqa: PLC0415
+ from pipelex.temporal.tprl_content_generation.content_generator_child_factory import ContentGeneratorChildFactory # noqa: PLC0415
+ from pipelex.temporal.tprl_pipe.temporal_pipe_router import make_temporal_pipe_router # noqa: PLC0415
+
+ pipelex_hub = get_pipelex_hub()
+ pipelex_hub.set_pipe_router(make_temporal_pipe_router())
+
+ generated_content_factory = GeneratedContentFactory(storage_provider=get_storage_provider())
+ content_generator_child = ContentGeneratorChildFactory.make_content_generator_child(
+ generated_content_factory=generated_content_factory,
+ )
+ pipelex_hub.set_content_generator(content_generator_child)
+
+ yield
+ manager.teardown()
+ temporal_hub.reset()
+
+ # Clear cached inference workers and SDK instances between modules.
+ # When session owns Pipelex (--temporal-server <profile>), the InferenceManager
+ # persists across modules. Each module's test class uses its own event loop
+ # (loop_scope="class"), so cached workers hold httpx connections bound to the
+ # previous module's (now-closed) event loop. Clearing forces fresh workers/clients
+ # on the next module's event loop.
+ from pipelex.hub import get_inference_manager, get_plugin_manager # noqa: PLC0415
+
+ get_inference_manager().teardown()
+ get_plugin_manager().plugin_sdk_registry.teardown()
+
+
+@pytest_asyncio.fixture(scope="session") # pyright: ignore[reportUntypedFunctionDecorator, reportUnknownMemberType]
+async def env(request: FixtureRequest) -> AsyncGenerator[WorkflowEnvironment, None]:
+ """Temporal test environment, shared across all temporal tests.
+
+ Uses an in-process server by default (--temporal-server none).
+ Pass a profile name from temporal_server_configs to connect to a real server.
+
+ For real server modes, Pipelex is temporarily bootstrapped here to read
+ connection config (session fixtures run before module-scoped ones), then
+ torn down so reset_pipelex_config_fixture can re-initialize per module.
+ The gRPC TemporalClient stays alive independently of the Pipelex lifecycle.
+ """
+ server_option: str = cast("str", request.config.getoption("--temporal-server"))
+ workflow_env: WorkflowEnvironment
+ if server_option == TEMPORAL_SERVER_NONE:
+ workflow_env = await WorkflowEnvironment.start_local(data_converter=data_converter) # pyright: ignore[reportUnknownMemberType]
+ elif server_option == TEMPORAL_SERVER_TIME_SKIPPING:
+ workflow_env = await WorkflowEnvironment.start_time_skipping(data_converter=data_converter)
+ else:
+ # Bootstrap Pipelex temporarily to read server connection config.
+ # If a module-scoped fixture already initialized Pipelex (happens when
+ # env is lazily triggered after reset_pipelex_config_fixture), skip init
+ # and just read the config from the existing instance.
+
+ # TODO: Pipelex.make() is heavy — it initializes inference, registries, etc.
+ # We only need the config here. Design a lightweight boot path (e.g.
+ # Pipelex.load_config_only()) that parses pipelex.toml without full init.
+ needs_teardown = False
+ if Pipelex.get_optional_instance() is None:
+ Pipelex.make(
+ integration_mode=IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST,
+ )
+ needs_teardown = True
+ temporal_client = await connect_to_temporal_selected_server(selected_server_config=server_option)
+ if needs_teardown:
+ Pipelex.teardown_if_needed()
+ workflow_env = WorkflowEnvironment.from_client(temporal_client)
+ yield workflow_env
+ await workflow_env.shutdown()
+
+
+@pytest_asyncio.fixture # pyright: ignore[reportUntypedFunctionDecorator, reportUnknownMemberType]
+async def temporal_client(env: WorkflowEnvironment) -> TemporalClient: # noqa: RUF029
+ """Temporal client connected to the test server."""
+ return env.client
+
+
+def pytest_generate_tests(metafunc: pytest.Metafunc) -> None:
+ """Auto-parametrize is_class_registry_isolated based on --class-registry.
+
+ By default (``--class-registry both``), every test class that requests
+ ``is_class_registry_isolated`` runs twice: once with classes in the global
+ registry (shared) and once scoped to the library (isolated). Pass
+ ``--class-registry shared`` or ``--class-registry isolated`` to force a
+ single mode (useful for debugging).
+ """
+ if "is_class_registry_isolated" not in metafunc.fixturenames:
+ return
+ mode = ClassRegistryMode(metafunc.config.getoption("--class-registry"))
+ match mode:
+ case ClassRegistryMode.BOTH:
+ metafunc.parametrize("is_class_registry_isolated", [False, True], ids=["shared", "isolated"], scope="class")
+ case ClassRegistryMode.ISOLATED:
+ metafunc.parametrize("is_class_registry_isolated", [True], ids=["isolated"], scope="class")
+ case ClassRegistryMode.SHARED:
+ metafunc.parametrize("is_class_registry_isolated", [False], ids=["shared"], scope="class")
diff --git a/tests/integration/pipelex/temporal/content_generation/conftest.py b/tests/integration/pipelex/temporal/content_generation/conftest.py
new file mode 100644
index 000000000..076cbe6c4
--- /dev/null
+++ b/tests/integration/pipelex/temporal/content_generation/conftest.py
@@ -0,0 +1,89 @@
+from collections.abc import Generator
+from typing import AsyncGenerator, cast
+
+import pytest
+import pytest_asyncio
+from pytest import FixtureRequest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry
+from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.config import get_config
+from pipelex.hub import get_class_registry, get_report_delegate
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment
+from pipelex.temporal.test_extras.temporal_registry_test_models import Person
+from pipelex.temporal.tprl_content_generation.content_generator_child import ContentGeneratorChild
+from pipelex.temporal.tprl_content_generation.content_generator_child_factory import ContentGeneratorChildFactory
+from pipelex.temporal.tprl_content_generation.content_generator_top_factory import ContentGeneratorTopFactory
+from tests.integration.pipelex.temporal.test_utils import rprint
+
+
[email protected](autouse=True)
+def enable_temporal_config():
+ """Enable temporal in config for content_generation tests that go through WorkflowExecutor."""
+ config = get_config()
+ original_enabled = config.temporal.is_enabled
+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})
+ yield
+ config.temporal = config.temporal.model_copy(update={"is_enabled": original_enabled})
+
+
[email protected](autouse=True)
+def register_test_temporal_models():
+ # Code to run before each test
+ rprint("\n[magenta]Temporal classes registration[/magenta]")
+ get_class_registry().register_class(class_type=Person)
+ yield
+ # Code to run after each test
+ rprint("\n[magenta]Temporal classes unregistration[/magenta]")
+ get_class_registry().unregister_class(class_type=Person)
+
+
+@pytest_asyncio.fixture # pyright: ignore[reportUntypedFunctionDecorator, reportUnknownMemberType]
+async def top_crafter( # noqa: RUF029
+ pipe_run_mode: PipeRunMode, temporal_client: TemporalClient, generated_content_factory: GeneratedContentFactory
+) -> AsyncGenerator[ContentGeneratorProtocol, None]:
+ rprint("\n[magenta]TopCrafter setup[/magenta]")
+ crafter: ContentGeneratorProtocol
+ if pipe_run_mode.is_dry:
+ crafter = ContentGeneratorDry()
+ else:
+ crafter = ContentGeneratorTopFactory.make_content_generator_top(
+ generated_content_factory=generated_content_factory,
+ worker_environment=TemporalWorkerEnvironment.INTERNAL,
+ temporal_client=temporal_client,
+ )
+ yield crafter
+ rprint("\n[magenta]TopCrafter teardown[/magenta]")
+
+
[email protected] # pyright: ignore[reportUntypedFunctionDecorator, reportUnknownMemberType]
+def child_crafter(generated_content_factory: GeneratedContentFactory) -> Generator[ContentGeneratorChild, None, None]:
+ # Code to run before each test
+ rprint("\n[magenta]ChildCrafter setup[/magenta]")
+ crafter = ContentGeneratorChildFactory.make_content_generator_child(
+ generated_content_factory=generated_content_factory,
+ )
+ # Return it for use in tests
+ yield crafter
+ # Code to run after each test
+ rprint("\n[magenta]ChildCrafter teardown[/magenta]")
+
+
+def tprl_job_metadata(request: FixtureRequest) -> Generator[JobMetadata, None, None]:
+ """Provide a JobMetadata instance with an open reporting registry.
+
+ Uses the test function name as pipeline_run_id, matching the convention
+ used by TestTprlCrafterTop tests.
+ """
+ pipeline_run_id = cast("str", request.node.originalname) # pyright: ignore[reportUnknownMemberType]
+ get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)
+ yield JobMetadata(
+ user_id="test",
+ pipeline_run_id=pipeline_run_id,
+ )
+ get_report_delegate().close_registry(pipeline_run_id=pipeline_run_id)
diff --git a/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_child.py b/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_child.py
new file mode 100644
index 000000000..43b0b1d39
--- /dev/null
+++ b/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_child.py
@@ -0,0 +1,33 @@
+import uuid
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.test_extras.temporal_test_tasks import TEMPORAL_TEST_ACTIVITIES, TEMPORAL_TEST_WORKFLOWS
+from pipelex.temporal.test_extras.wf_test_content_generator_child import WfTestContentGeneratorChild
+
+
[email protected](loop_scope="class")
+class TestTprlContentGeneratorChild:
+ async def test_tprl_content_generator_child(self, temporal_client: TemporalClient, tprl_job_metadata: JobMetadata, pipe_run_mode: PipeRunMode):
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ test_workflows=TEMPORAL_TEST_WORKFLOWS,
+ test_activities=TEMPORAL_TEST_ACTIVITIES,
+ ):
+ await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfTestContentGeneratorChild.run,
+ arg=pipe_run_mode.is_dry,
+ id=tprl_job_metadata.pipeline_run_id,
+ task_queue=task_queue,
+ )
diff --git a/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_top.py b/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_top.py
new file mode 100644
index 000000000..4024f6f81
--- /dev/null
+++ b/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_top.py
@@ -0,0 +1,231 @@
+import logging
+
+import pytest
+from temporalio.client import WorkflowFailureError
+
+from pipelex import pretty_print
+from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol
+from pipelex.cogt.extract.extract_input import ExtractInput
+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
+from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.core.stuffs.page_content import PageContent
+from pipelex.hub import get_model_deck
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.temporal.test_extras.temporal_registry_test_models import Person
+from tests.integration.pipelex.temporal.test_data import PipeTestCases
+
+USER_TEXT_FOR_BASE = """
+Write a detailed description of a woman's clothing in the style of a 19th-century novel.
+Keep it short: 3 sentences max
+"""
+
+USER_TEXT_FOR_SINGLE_PERSON = "name: John, age: 30, job: bank teller"
+USER_TEXT_FOR_SINGLE_PERSON_TEXT_THEN_OBJECT = """
+Imagine a female character that decides to become a cop once reaching middle age.
+Present this character in a couple of very short sentences.
+Be sure to include the character's full name, age and job.
+"""
+MULTIPLE_USER_TEXTS_FOR_PEOPLE = [
+ "name: Bob, age: 25, job: banker",
+ "name: Maria, age: 35, job: consultant",
+ "name: SLartiblfastikur, age: 30, job: fizzy buzzer",
+ "name: Alice, age: 40, job: developer",
+ "name: Tom, age: 45, job: TV presenter",
+ "name: Jerry, age: 50, job: nurse",
+]
+USER_TEXTS_FOR_PEOPLE_STR = "\n".join(MULTIPLE_USER_TEXTS_FOR_PEOPLE)
+USER_TEXT_FOR_MULTIPLE_PEOPLE_TEXT_THEN_OBJECT = """
+Imagine the 4 main characters for a sitcom in Paris.
+Present each character in one very short sentence.
+Be sure to include each character's full name, age and job.
+"""
+
+USER_TEXT_FOR_HAIKU = """
+Write a haiku about the meaning of life
+"""
+
+# silence a warning that comes from deep down in temporalio's pydantic converter
+pytestmark = pytest.mark.filterwarnings("ignore:The `parse_obj` method is deprecated")
+
+
[email protected](loop_scope="class")
+class TestTprlCrafterTop:
+ @pytest.mark.llm
+ @pytest.mark.inference
+ @pytest.mark.dry_runnable
+ async def test_tprl_make_llm_text_only(self, tprl_job_metadata: JobMetadata, top_crafter: ContentGeneratorProtocol):
+ llm_setting_main = get_model_deck().get_llm_setting(llm_choice="$testing-text")
+
+ text: str = await top_crafter.make_llm_text(
+ job_metadata=tprl_job_metadata,
+ llm_prompt_for_text=LLMPrompt(user_text=USER_TEXT_FOR_BASE),
+ llm_setting_main=llm_setting_main,
+ )
+ pretty_print(text, title="make_llm_text")
+
+ assert isinstance(text, str)
+
+ @pytest.mark.llm
+ @pytest.mark.inference
+ @pytest.mark.dry_runnable
+ async def test_tprl_make_object_direct(self, tprl_job_metadata: JobMetadata, top_crafter: ContentGeneratorProtocol):
+ llm_setting_for_object = get_model_deck().get_llm_setting(llm_choice="$testing-structured")
+
+ person_direct: Person = await top_crafter.make_object_direct(
+ job_metadata=tprl_job_metadata,
+ object_class=Person,
+ llm_prompt_for_object=LLMPrompt(user_text=USER_TEXT_FOR_SINGLE_PERSON),
+ llm_setting_for_object=llm_setting_for_object,
+ )
+ pretty_print(person_direct, title="make_object_direct")
+
+ assert isinstance(person_direct, Person)
+
+ @pytest.mark.llm
+ @pytest.mark.inference
+ @pytest.mark.dry_runnable
+ async def test_tprl_make_text_then_object(self, tprl_job_metadata: JobMetadata, top_crafter: ContentGeneratorProtocol):
+ llm_setting_main = get_model_deck().get_llm_setting(llm_choice="$testing-text")
+ llm_setting_for_object = get_model_deck().get_llm_setting(llm_choice="$testing-structured")
+
+ person_tto: Person = await top_crafter.make_text_then_object(
+ job_metadata=tprl_job_metadata,
+ object_class=Person,
+ llm_prompt_for_text=LLMPrompt(user_text=USER_TEXT_FOR_SINGLE_PERSON_TEXT_THEN_OBJECT),
+ llm_setting_main=llm_setting_main,
+ llm_setting_for_object=llm_setting_for_object,
+ )
+ pretty_print(person_tto, title="make_text_then_object")
+
+ assert isinstance(person_tto, Person)
+
+ @pytest.mark.llm
+ @pytest.mark.inference
+ @pytest.mark.dry_runnable
+ async def test_tprl_make_object_list_direct(self, tprl_job_metadata: JobMetadata, top_crafter: ContentGeneratorProtocol):
+ llm_setting_for_object = get_model_deck().get_llm_setting(llm_choice="$testing-structured")
+
+ person_list_direct: list[Person] = await top_crafter.make_object_list_direct(
+ job_metadata=tprl_job_metadata,
+ object_class=Person,
+ llm_prompt_for_object_list=LLMPrompt(user_text=USER_TEXTS_FOR_PEOPLE_STR),
+ llm_setting_for_object_list=llm_setting_for_object,
+ )
+ pretty_print(person_list_direct, title="make_object_list_direct")
+
+ assert isinstance(person_list_direct, list)
+ assert all(isinstance(person, Person) for person in person_list_direct)
+
+ @pytest.mark.llm
+ @pytest.mark.inference
+ @pytest.mark.dry_runnable
+ async def test_tprl_make_text_then_object_list(self, tprl_job_metadata: JobMetadata, top_crafter: ContentGeneratorProtocol):
+ llm_setting_main = get_model_deck().get_llm_setting(llm_choice="$testing-text")
+ llm_setting_for_object = get_model_deck().get_llm_setting(llm_choice="$testing-structured")
+
+ person_list_tto: list[Person] = await top_crafter.make_text_then_object_list(
+ job_metadata=tprl_job_metadata,
+ object_class=Person,
+ llm_prompt_for_text=LLMPrompt(user_text=USER_TEXT_FOR_MULTIPLE_PEOPLE_TEXT_THEN_OBJECT),
+ llm_setting_main=llm_setting_main,
+ llm_setting_for_object_list=llm_setting_for_object,
+ )
+ pretty_print(person_list_tto, title="make_text_then_object_list")
+
+ assert isinstance(person_list_tto, list)
+ assert all(isinstance(person, Person) for person in person_list_tto)
+
+ @pytest.mark.img_gen
+ @pytest.mark.inference
+ @pytest.mark.dry_runnable
+ async def test_tprl_craft_image(self, tprl_job_metadata: JobMetadata, top_crafter: ContentGeneratorProtocol):
+ image: ImageContent = await top_crafter.make_single_image(
+ job_metadata=tprl_job_metadata,
+ img_gen_handle="@default-small ",
+ img_gen_prompt=ImgGenPrompt(
+ positive_text="A dog with sunglasses coding on a laptop",
+ ),
+ )
+ pretty_print(image, title="craft_image")
+ assert isinstance(image, ImageContent)
+
+ async def test_tprl_jinja2_text(self, job_metadata: JobMetadata, pipe_run_mode: PipeRunMode, top_crafter: ContentGeneratorProtocol):
+ context = {
+ "the_answer": "elementary, my dear Watson",
+ }
+
+ jinja2_text: str = await top_crafter.make_templated_text(
+ job_metadata=job_metadata,
+ context=context,
+ template="The answer is: {{ the_answer }}",
+ )
+ pretty_print(jinja2_text, title="jinja2_text")
+ assert isinstance(jinja2_text, str)
+ if pipe_run_mode.is_dry:
+ assert "the_answer" in jinja2_text
+ else:
+ assert jinja2_text == "The answer is: elementary, my dear Watson"
+
+ @pytest.mark.extract
+ @pytest.mark.inference
+ @pytest.mark.dry_runnable
+ async def test_tprl_extract(self, tprl_job_metadata: JobMetadata, top_crafter: ContentGeneratorProtocol):
+ extract_output: list[PageContent] = await top_crafter.make_extract_pages(
+ job_metadata=tprl_job_metadata,
+ extract_handle="azure-document-intelligence",
+ extract_input=ExtractInput(image_uri=PipeTestCases.IMG_EXPENSE_REPORT_1),
+ extract_job_params=ExtractJobParams.make_default_extract_job_params(),
+ extract_job_config=ExtractJobConfig(),
+ )
+ pretty_print(extract_output, title="extract_pages")
+ assert isinstance(extract_output, list)
+ assert all(isinstance(page, PageContent) for page in extract_output)
+
+ @pytest.mark.llm
+ @pytest.mark.inference
+ async def test_tprl_make_llm_text_with_error(
+ self, tprl_job_metadata: JobMetadata, top_crafter: ContentGeneratorProtocol, capfd: pytest.CaptureFixture[str]
+ ):
+ """Test that a bad model handle causes a WorkflowFailureError.
+
+ capfd captures Temporal Rust core WARN logs (fd-level stderr)
+ that are otherwise an unreadable blob during expected workflow failures.
+ """
+ _ = capfd # side-effect fixture: captures fd-level stderr from Temporal Rust core
+ bad_handle_to_test_failure = "bad_handle_to_test_failure"
+ llm_setting_main = LLMSetting(model=bad_handle_to_test_failure, temperature=0.5, max_tokens=100)
+ # Filter out Temporal's "Completing activity as failed" traceback for this expected failure
+
+ class _ActivityFailedFilter(logging.Filter):
+ def filter(self, record: logging.LogRecord) -> bool: # pyright: ignore[reportImplicitOverride]
+ return "Completing activity as failed" not in record.getMessage()
+
+ activity_failed_filter = _ActivityFailedFilter()
+ activity_logger = logging.getLogger("temporalio.activity")
+ activity_logger.addFilter(activity_failed_filter)
+ try:
+ with pytest.raises(WorkflowFailureError) as excinfo:
+ await top_crafter.make_llm_text(
+ job_metadata=tprl_job_metadata,
+ llm_prompt_for_text=LLMPrompt(user_text=USER_TEXT_FOR_BASE),
+ llm_setting_main=llm_setting_main,
+ )
+ finally:
+ activity_logger.removeFilter(activity_failed_filter)
+ workflow_failure_error = excinfo.value
+ assert str(workflow_failure_error) == "Workflow execution failed"
+ pretty_print(f"Caught expected error: {workflow_failure_error}, caused by {workflow_failure_error.cause}")
+ cause_str = str(workflow_failure_error.cause)
+ assert any(
+ expected in cause_str
+ for expected in [
+ f"CogtManagerWorkerSetupError: No worker has been setup for '{bad_handle_to_test_failure}'",
+ f"ConfigNotFoundError: LLM Engine blueprint for llm_handle '{bad_handle_to_test_failure}' not found in deck's engine blueprints",
+ f"ModelNotFoundError: Model handle '{bad_handle_to_test_failure}' was not found in the model deck",
+ ]
+ )
diff --git a/tests/integration/pipelex/temporal/content_generation/test_tprl_make_content_generator.py b/tests/integration/pipelex/temporal/content_generation/test_tprl_make_content_generator.py
new file mode 100644
index 000000000..d98b69468
--- /dev/null
+++ b/tests/integration/pipelex/temporal/content_generation/test_tprl_make_content_generator.py
@@ -0,0 +1,20 @@
+import pytest
+
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.temporal.tprl_content_generation.content_generator_child_factory import ContentGeneratorChildFactory
+from pipelex.temporal.tprl_content_generation.content_generator_top_factory import ContentGeneratorTopFactory
+
+
+class TestTprlMakeContentGenerator:
+ def test_tprl_make_content_generator_top(self, generated_content_factory: GeneratedContentFactory):
+ crafter = ContentGeneratorTopFactory.make_content_generator_top(
+ generated_content_factory=generated_content_factory,
+ )
+ assert crafter
+
+ def test_tprl_make_content_generator_child(self, generated_content_factory: GeneratedContentFactory):
+ crafter = ContentGeneratorChildFactory.make_content_generator_child(
+ generated_content_factory=generated_content_factory,
+ )
+ assert crafter
diff --git a/tests/integration/pipelex/temporal/data_converter/conftest.py b/tests/integration/pipelex/temporal/data_converter/conftest.py
new file mode 100644
index 000000000..eea1aae7d
--- /dev/null
+++ b/tests/integration/pipelex/temporal/data_converter/conftest.py
@@ -0,0 +1,17 @@
+import pytest
+
+from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter
+
+
+def payload_converter() -> BaseModelPayloadConverter:
+ return BaseModelPayloadConverter()
+
+
+class CraftingTestCases:
+ USER_TEXT_FOR_BASE = """
+ Write a detailed description of a woman's clothing in the style of a 19th-century novel.
+ Keep it short: 3 sentences max
+ """
+
+ USER_TEXT_FOR_SINGLE_PERSON = "name: John, age: 30, job: bank teller"
diff --git a/tests/integration/pipelex/temporal/data_converter/test_data_conv_craft.py b/tests/integration/pipelex/temporal/data_converter/test_data_conv_craft.py
new file mode 100644
index 000000000..154af281e
--- /dev/null
+++ b/tests/integration/pipelex/temporal/data_converter/test_data_conv_craft.py
@@ -0,0 +1,60 @@
+import pytest
+
+from pipelex import pretty_print
+from pipelex.cogt.content_generation.assignment_models import LLMAssignment, ObjectAssignment
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.hub import get_model_deck
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter
+from pipelex.temporal.test_extras.temporal_registry_test_models import Person
+
+from .conftest import CraftingTestCases
+
+
+class TestDataConverterForCrafting:
+ def test_data_converter_for_make_llm_text(
+ self,
+ payload_converter: BaseModelPayloadConverter,
+ ):
+ user_text = CraftingTestCases.USER_TEXT_FOR_BASE
+ llm_setting = get_model_deck().get_llm_setting(llm_choice="$testing-text")
+ llm_prompt_for_text = LLMPrompt(user_text=user_text)
+ llm_assignment = LLMAssignment(
+ job_metadata=JobMetadata(user_id="test", pipeline_run_id="test"),
+ llm_setting=llm_setting,
+ llm_prompt=llm_prompt_for_text,
+ )
+ pretty_print(llm_assignment, title="llm_assignment")
+ payload = payload_converter.to_payload(llm_assignment)
+ pretty_print(payload, title="payload")
+ assert payload
+ restored: LLMAssignment = payload_converter.from_payload(payload, type_hint=LLMAssignment)
+ pretty_print(restored, title="restored LLMAssignment")
+ assert restored
+ assert llm_assignment == restored
+
+ def test_data_converter_for_make_object_direct(
+ self,
+ payload_converter: BaseModelPayloadConverter,
+ ):
+ user_text = CraftingTestCases.USER_TEXT_FOR_SINGLE_PERSON
+ llm_setting_for_object = get_model_deck().get_llm_setting(llm_choice="$testing-structured")
+ llm_prompt_for_object = LLMPrompt(user_text=user_text)
+ llm_assignment_for_object = LLMAssignment(
+ job_metadata=JobMetadata(user_id="test", pipeline_run_id="test"),
+ llm_setting=llm_setting_for_object,
+ llm_prompt=llm_prompt_for_object,
+ )
+ object_assignment = ObjectAssignment.make_for_class(
+ object_class=Person,
+ llm_assignment=llm_assignment_for_object,
+ )
+ pretty_print(object_assignment, title="object_assignment")
+ payload = payload_converter.to_payload(object_assignment)
+ pretty_print(payload, title="payload")
+ assert payload
+ restored: ObjectAssignment = payload_converter.from_payload(payload, type_hint=ObjectAssignment)
+ pretty_print(restored, title="restored ObjectAssignment")
+ assert restored
+ assert object_assignment == restored
diff --git a/tests/integration/pipelex/temporal/data_converter/test_data_conv_pydantic.py b/tests/integration/pipelex/temporal/data_converter/test_data_conv_pydantic.py
new file mode 100644
index 000000000..9ecc129c1
--- /dev/null
+++ b/tests/integration/pipelex/temporal/data_converter/test_data_conv_pydantic.py
@@ -0,0 +1,28 @@
+from typing import Any
+
+import pytest
+from pydantic import RootModel
+
+from pipelex import pretty_print
+from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter
+
+
+class MyRootModel(RootModel[dict[str, Any]]):
+ pass
+
+
+class TestDataConverterForRootModel:
+ def test_data_converter_for_root_model_with_dict(
+ self,
+ payload_converter: BaseModelPayloadConverter,
+ ):
+ my_root_model_1 = MyRootModel(root={"a": 1, "b": 2})
+ pretty_print(my_root_model_1, title="my_root_model_1")
+ payload = payload_converter.to_payload(my_root_model_1)
+ pretty_print(payload, title="payload")
+ assert payload
+ restored: MyRootModel = payload_converter.from_payload(payload, type_hint=MyRootModel)
+ pretty_print(restored, title="restored MyRootModel")
+ assert restored
+ assert my_root_model_1 == restored
diff --git a/tests/integration/pipelex/temporal/library_crate/conflict_concept_alpha.mthds b/tests/integration/pipelex/temporal/library_crate/conflict_concept_alpha.mthds
new file mode 100644
index 000000000..5143d125d
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/conflict_concept_alpha.mthds
@@ -0,0 +1,38 @@
+domain = "conflict_alpha"
+description = "PipeSequence with concept 'Result' having score+label fields (conflicts with beta's Result)"
+
+[concept.Result]
+description = "A scored result with label"
+
+[concept.Result.structure]
+score = { type = "integer", required = true, description = "Numeric score" }
+label = { type = "text", required = true, description = "Category label" }
+
+[pipe]
+[pipe.alpha_pipeline]
+type = "PipeSequence"
+description = "Two-step pipeline producing a scored Result then summarizing it"
+output = "Text"
+steps = [
+ { pipe = "alpha_generate", result = "alpha_result" },
+ { pipe = "alpha_summarize", result = "alpha_summary" },
+]
+
+[pipe.alpha_generate]
+type = "PipeLLM"
+description = "Generate a scored result"
+output = "Result"
+model = "$testing-text"
+prompt = "Generate a result with score=42 and label='excellent'. Respond with exactly those values."
+
+[pipe.alpha_summarize]
+type = "PipeLLM"
+description = "Summarize the scored result"
+inputs = { alpha_result = "Result" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Summarize this result:
+Score: @alpha_result.score
+Label: @alpha_result.label
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/conflict_concept_beta.mthds b/tests/integration/pipelex/temporal/library_crate/conflict_concept_beta.mthds
new file mode 100644
index 000000000..2d653dfa8
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/conflict_concept_beta.mthds
@@ -0,0 +1,40 @@
+domain = "conflict_beta"
+description = "PipeSequence with concept 'Result' having value+confidence+is_valid fields (conflicts with alpha's Result)"
+
+[concept.Result]
+description = "A confidence-rated result with validity"
+
+[concept.Result.structure]
+value = { type = "text", required = true, description = "The result value" }
+confidence = { type = "number", required = true, description = "Confidence score 0-1" }
+is_valid = { type = "text", required = true, description = "Whether the result is valid" }
+
+[pipe]
+[pipe.beta_pipeline]
+type = "PipeSequence"
+description = "Two-step pipeline producing a confidence-rated Result then summarizing it"
+output = "Text"
+steps = [
+ { pipe = "beta_generate", result = "beta_result" },
+ { pipe = "beta_summarize", result = "beta_summary" },
+]
+
+[pipe.beta_generate]
+type = "PipeLLM"
+description = "Generate a confidence-rated result"
+output = "Result"
+model = "$testing-text"
+prompt = "Generate a result with value='positive', confidence=0.95, is_valid='true'. Respond with exactly those values."
+
+[pipe.beta_summarize]
+type = "PipeLLM"
+description = "Summarize the confidence-rated result"
+inputs = { beta_result = "Result" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Summarize this result:
+Value: @beta_result.value
+Confidence: @beta_result.confidence
+Valid: @beta_result.is_valid
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/conflict_pipe_alpha.mthds b/tests/integration/pipelex/temporal/library_crate/conflict_pipe_alpha.mthds
new file mode 100644
index 000000000..cdb8fcc89
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/conflict_pipe_alpha.mthds
@@ -0,0 +1,31 @@
+domain = "pipe_conflict_alpha"
+description = "PipeSequence with alpha_shared_step PipeLLM about colors — tests per-workflow pipe_ref isolation"
+
+[pipe]
+[pipe.pipe_alpha_pipeline]
+type = "PipeSequence"
+description = "Two-step pipeline about colors"
+output = "Text"
+steps = [
+ { pipe = "alpha_shared_step", result = "step_result" },
+ { pipe = "alpha_finalize", result = "final_result" },
+]
+
+[pipe.alpha_shared_step]
+type = "PipeLLM"
+description = "Generate text about colors"
+output = "Text"
+model = "$testing-text"
+prompt = "List three primary colors in a single sentence."
+
+[pipe.alpha_finalize]
+type = "PipeLLM"
+description = "Finalize the color text"
+inputs = { step_result = "Text" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Rewrite this text in uppercase:
+
+@step_result.text
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/conflict_pipe_beta.mthds b/tests/integration/pipelex/temporal/library_crate/conflict_pipe_beta.mthds
new file mode 100644
index 000000000..945e06f71
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/conflict_pipe_beta.mthds
@@ -0,0 +1,31 @@
+domain = "pipe_conflict_beta"
+description = "PipeSequence with beta_shared_step PipeLLM about animals — tests per-workflow pipe_ref isolation"
+
+[pipe]
+[pipe.pipe_beta_pipeline]
+type = "PipeSequence"
+description = "Two-step pipeline about animals"
+output = "Text"
+steps = [
+ { pipe = "beta_shared_step", result = "step_result" },
+ { pipe = "beta_finalize", result = "final_result" },
+]
+
+[pipe.beta_shared_step]
+type = "PipeLLM"
+description = "Generate text about animals"
+output = "Text"
+model = "$testing-text"
+prompt = "List three farm animals in a single sentence."
+
+[pipe.beta_finalize]
+type = "PipeLLM"
+description = "Finalize the animal text"
+inputs = { step_result = "Text" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Rewrite this text in a formal tone:
+
+@step_result.text
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/conftest.py b/tests/integration/pipelex/temporal/library_crate/conftest.py
new file mode 100644
index 000000000..bb46bf5c0
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/conftest.py
@@ -0,0 +1,185 @@
+from collections.abc import Generator
+from pathlib import Path
+
+import pytest
+
+from pipelex.hub import get_library_manager
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle, pipe_job_from_library
+from tests.integration.pipelex.temporal.test_data import (
+ CombinedPipelineTemporalTestData,
+ ConflictConceptAlphaTestData,
+ ConflictConceptBetaTestData,
+ ConflictPipeAlphaTestData,
+ ConflictPipeBetaTestData,
+ LibraryCrateTestData,
+ MultiConceptAlphaTestData,
+ MultiConceptBetaTestData,
+ PipeBatchTemporalTestData,
+ PipeComposeTemporalTestData,
+ PipeConditionTemporalTestData,
+ PipeParallelTemporalTestData,
+)
+
+# --- Existing fixtures (Phase 2) ---
+
+
[email protected](scope="class")
+def pipe_job_from_library_dirs(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """Build a PipeJob by loading a specific bundle file (simulates PIPELEXPATH-style loading).
+
+ Uses library_file_paths instead of library_dirs to avoid picking up unrelated
+ .mthds bundles (conflict/multi test bundles) that share the same directory.
+ """
+
+ def _load(library_id: str) -> None:
+ get_library_manager().load_libraries(
+ library_id=library_id,
+ library_file_paths=[Path(LibraryCrateTestData.BUNDLE_FILE)],
+ )
+
+ yield from pipe_job_from_library(_load, pipe_code=LibraryCrateTestData.PIPE_CODE, isolated_registry=is_class_registry_isolated)
+
+
[email protected](scope="class")
+def pipe_job_from_mthds_content(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """Build a PipeJob by loading the test bundle from mthds_content string."""
+ yield from pipe_job_from_bundle(
+ bundle_file=LibraryCrateTestData.BUNDLE_FILE,
+ pipe_code=LibraryCrateTestData.PIPE_CODE,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
+# --- Conflict concept fixtures (Scenario 1) ---
+
+
[email protected](scope="class")
+def alpha_concept_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with concept Result(score, label)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=ConflictConceptAlphaTestData.BUNDLE_FILE,
+ pipe_code=ConflictConceptAlphaTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](scope="class")
+def beta_concept_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with concept Result(value, confidence, is_valid)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=ConflictConceptBetaTestData.BUNDLE_FILE,
+ pipe_code=ConflictConceptBetaTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
+# --- Conflict pipe fixtures (Scenario 2) ---
+
+
[email protected](scope="class")
+def alpha_pipe_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with pipe 'shared_step' about colors."""
+ yield from pipe_job_from_bundle(
+ bundle_file=ConflictPipeAlphaTestData.BUNDLE_FILE,
+ pipe_code=ConflictPipeAlphaTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](scope="class")
+def beta_pipe_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with pipe 'shared_step' about animals."""
+ yield from pipe_job_from_bundle(
+ bundle_file=ConflictPipeBetaTestData.BUNDLE_FILE,
+ pipe_code=ConflictPipeBetaTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
+# --- Multi-concept fixtures (Scenario 3) ---
+
+
[email protected](scope="class")
+def multi_alpha_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with Profile(name, age) + Summary(headline, body)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=MultiConceptAlphaTestData.BUNDLE_FILE,
+ pipe_code=MultiConceptAlphaTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](scope="class")
+def multi_beta_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with Profile(title, department, level) + Summary(content)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=MultiConceptBetaTestData.BUNDLE_FILE,
+ pipe_code=MultiConceptBetaTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
+# --- Controller type coverage fixtures ---
+
+
[email protected](scope="class")
+def condition_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with PipeCondition inside a PipeSequence."""
+ yield from pipe_job_from_bundle(
+ bundle_file=PipeConditionTemporalTestData.BUNDLE_FILE,
+ pipe_code=PipeConditionTemporalTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](scope="class")
+def parallel_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with PipeParallel inside a PipeSequence."""
+ yield from pipe_job_from_bundle(
+ bundle_file=PipeParallelTemporalTestData.BUNDLE_FILE,
+ pipe_code=PipeParallelTemporalTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](scope="class")
+def batch_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with PipeBatch inside a PipeSequence."""
+ yield from pipe_job_from_bundle(
+ bundle_file=PipeBatchTemporalTestData.BUNDLE_FILE,
+ pipe_code=PipeBatchTemporalTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](scope="class")
+def compose_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with PipeCompose operator and dynamic Report concept."""
+ yield from pipe_job_from_bundle(
+ bundle_file=PipeComposeTemporalTestData.BUNDLE_FILE,
+ pipe_code=PipeComposeTemporalTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](scope="class")
+def combined_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob with PipeParallel + PipeCondition in a single PipeSequence."""
+ yield from pipe_job_from_bundle(
+ bundle_file=CombinedPipelineTemporalTestData.BUNDLE_FILE,
+ pipe_code=CombinedPipelineTemporalTestData.PIPE_CODE,
+ pipe_run_mode=pipe_run_mode,
+ isolated_registry=is_class_registry_isolated,
+ )
diff --git a/tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds b/tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds
new file mode 100644
index 000000000..faa018fdc
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds
@@ -0,0 +1,38 @@
+domain = "dynamic_concept_test"
+description = "PipeSequence with dynamic concept (inline structure) for deferred hydration tests"
+
+[concept.Greeting]
+description = "A greeting with message and language"
+
+[concept.Greeting.structure]
+message = { type = "text", required = true, description = "The greeting message" }
+language = { type = "text", required = true, description = "The language of the greeting" }
+
+[pipe]
+[pipe.dynamic_greeting_sequence]
+type = "PipeSequence"
+description = "Two-step pipeline producing a dynamic Greeting concept"
+output = "Text"
+steps = [
+ { pipe = "generate_greeting", result = "greeting_result" },
+ { pipe = "summarize_greeting", result = "summary_result" },
+]
+
+[pipe.generate_greeting]
+type = "PipeLLM"
+description = "Generate a greeting in a specific language"
+output = "Greeting"
+model = "$testing-text"
+prompt = "Generate a greeting in French. Respond with exactly: message='Bonjour le monde', language='French'"
+
+[pipe.summarize_greeting]
+type = "PipeLLM"
+description = "Summarize the greeting"
+inputs = { greeting_result = "Greeting" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Summarize this greeting:
+Message: @greeting_result.message
+Language: @greeting_result.language
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/helpers.py b/tests/integration/pipelex/temporal/library_crate/helpers.py
new file mode 100644
index 000000000..cb4e698e4
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/helpers.py
@@ -0,0 +1,72 @@
+"""Shared test helpers for Temporal LibraryCrate integration tests."""
+
+import uuid
+from typing import Any
+
+from temporalio.client import Client as TemporalClient
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+
+
+def rehydrate_pipe_output(pipe_output: PipeOutput, pipe_job: PipeJob | None = None) -> PipeOutput:
+ """Rehydrate a deferred PipeOutput received from a Temporal workflow.
+
+ Thin wrapper around the production helper `rehydrate_pipe_output_with_crate`
+ so the integration tests share the exact same scoped-registry plumbing
+ that production submitters use.
+ """
+ library_crate = pipe_job.library_crate if pipe_job is not None else None
+ return rehydrate_pipe_output_with_crate(pipe_output, library_crate)
+
+
+async def execute_workflow(
+ pipe_job: PipeJob,
+ temporal_client: TemporalClient,
+ task_queue: str,
+ **kwargs: Any,
+) -> PipeOutput:
+ """Execute WfPipeRouter and return PipeOutput, rehydrating if needed."""
+ workflow_id = str(uuid.uuid4())
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=pipe_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ **kwargs,
+ )
+ return rehydrate_pipe_output(pipe_output, pipe_job)
+
+
+def assert_stuff_names(pipe_output: PipeOutput, expected_names: list[str]) -> None:
+ """Assert that all expected stuffs exist in the output with non-None content."""
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+ for stuff_name in expected_names:
+ assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"
+ stuff = working_memory.get_stuff(stuff_name)
+ assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"
+
+
+def assert_text_stuff_names(pipe_output: PipeOutput, expected_names: list[str]) -> None:
+ """Assert that all expected stuffs exist and contain TextContent."""
+ assert_stuff_names(pipe_output, expected_names)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+ for stuff_name in expected_names:
+ stuff = working_memory.get_stuff(stuff_name)
+ assert isinstance(stuff.content, TextContent), f"Expected TextContent for '{stuff_name}', got {type(stuff.content).__name__}"
+
+
+def assert_structured_fields(pipe_output: PipeOutput, stuff_name: str, expected_fields: list[str]) -> None:
+ """Assert that a stuff's content is StructuredContent with the expected fields."""
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+ stuff = working_memory.get_stuff(stuff_name)
+ assert isinstance(stuff.content, StructuredContent), f"Expected StructuredContent for '{stuff_name}', got {type(stuff.content).__name__}"
+ for field in expected_fields:
+ assert hasattr(stuff.content, field), f"StructuredContent for '{stuff_name}' missing field '{field}'"
diff --git a/tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds b/tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds
new file mode 100644
index 000000000..c2f4544fc
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds
@@ -0,0 +1,47 @@
+domain = "large_payload_test"
+description = "Multi-step pipeline that accumulates large WorkingMemory for codec stress testing"
+
+[pipe.large_payload_sequence]
+type = "PipeSequence"
+description = "Three-step pipeline accumulating verbose text output"
+output = "Text"
+steps = [
+ { pipe = "verbose_step_one", result = "step_one_result" },
+ { pipe = "verbose_step_two", result = "step_two_result" },
+ { pipe = "verbose_step_three", result = "step_three_result" },
+]
+
+[pipe.verbose_step_one]
+type = "PipeLLM"
+description = "First step: generate a detailed story"
+output = "Text"
+model = "$testing-text"
+prompt = "Write a very detailed, multi-paragraph story about a space adventure. Include character names, planet descriptions, and dialogue."
+
+[pipe.verbose_step_two]
+type = "PipeLLM"
+description = "Second step: expand with analysis"
+inputs = { step_one_result = "Text" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Analyze the following story in great detail. Cover themes, character development, plot structure, and writing style.
+
+@step_one_result.text
+"""
+
+[pipe.verbose_step_three]
+type = "PipeLLM"
+description = "Third step: create a comprehensive summary"
+inputs = { step_one_result = "Text", step_two_result = "Text" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Based on this story and analysis, write a comprehensive literary review.
+
+Story:
+@step_one_result.text
+
+Analysis:
+@step_two_result.text
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/multi_concept_alpha.mthds b/tests/integration/pipelex/temporal/library_crate/multi_concept_alpha.mthds
new file mode 100644
index 000000000..dc78cd768
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/multi_concept_alpha.mthds
@@ -0,0 +1,60 @@
+domain = "multi_alpha"
+description = "PipeSequence with Profile(name,age) and Summary(headline,body) — conflicts with beta's Profile and Summary"
+
+[concept.Profile]
+description = "A personal profile with name and age"
+
+[concept.Profile.structure]
+name = { type = "text", required = true, description = "Person's name" }
+age = { type = "integer", required = true, description = "Person's age" }
+
+[concept.Summary]
+description = "A summary with headline and body"
+
+[concept.Summary.structure]
+headline = { type = "text", required = true, description = "Summary headline" }
+body = { type = "text", required = true, description = "Summary body text" }
+
+[pipe]
+[pipe.multi_alpha_pipeline]
+type = "PipeSequence"
+description = "Three-step pipeline: generate personal Profile, then Summary, then multi_alpha_finalize as Text"
+output = "Text"
+steps = [
+ { pipe = "multi_alpha_generate_profile", result = "profile_result" },
+ { pipe = "multi_alpha_generate_summary", result = "summary_result" },
+ { pipe = "multi_alpha_finalize", result = "final_result" },
+]
+
+[pipe.multi_alpha_generate_profile]
+type = "PipeLLM"
+description = "Generate a personal profile"
+output = "Profile"
+model = "$testing-text"
+prompt = "Generate a profile with name='Alice' and age=30. Respond with exactly those values."
+
+[pipe.multi_alpha_generate_summary]
+type = "PipeLLM"
+description = "Summarize the profile"
+inputs = { profile_result = "Profile" }
+output = "Summary"
+model = "$testing-text"
+prompt = """
+Create a summary for this person:
+Name: @profile_result.name
+Age: @profile_result.age
+
+Respond with a headline and body.
+"""
+
+[pipe.multi_alpha_finalize]
+type = "PipeLLM"
+description = "Finalize as plain text"
+inputs = { summary_result = "Summary" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Format this summary as plain text:
+Headline: @summary_result.headline
+Body: @summary_result.body
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/multi_concept_beta.mthds b/tests/integration/pipelex/temporal/library_crate/multi_concept_beta.mthds
new file mode 100644
index 000000000..da3f45a0c
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/multi_concept_beta.mthds
@@ -0,0 +1,58 @@
+domain = "multi_beta"
+description = "PipeSequence with Profile(title,department,level) and Summary(content) — conflicts with alpha's Profile and Summary"
+
+[concept.Profile]
+description = "A professional profile with title, department, and level"
+
+[concept.Profile.structure]
+title = { type = "text", required = true, description = "Job title" }
+department = { type = "text", required = true, description = "Department name" }
+level = { type = "integer", required = true, description = "Seniority level" }
+
+[concept.Summary]
+description = "A compact summary with just content"
+
+[concept.Summary.structure]
+content = { type = "text", required = true, description = "Summary content" }
+
+[pipe]
+[pipe.multi_beta_pipeline]
+type = "PipeSequence"
+description = "Three-step pipeline: generate professional Profile, then Summary, then multi_beta_finalize as Text"
+output = "Text"
+steps = [
+ { pipe = "multi_beta_generate_profile", result = "profile_result" },
+ { pipe = "multi_beta_generate_summary", result = "summary_result" },
+ { pipe = "multi_beta_finalize", result = "final_result" },
+]
+
+[pipe.multi_beta_generate_profile]
+type = "PipeLLM"
+description = "Generate a professional profile"
+output = "Profile"
+model = "$testing-text"
+prompt = "Generate a profile with title='Engineer', department='Platform', level=3. Respond with exactly those values."
+
+[pipe.multi_beta_generate_summary]
+type = "PipeLLM"
+description = "Summarize the professional profile"
+inputs = { profile_result = "Profile" }
+output = "Summary"
+model = "$testing-text"
+prompt = """
+Create a one-line summary for this employee:
+Title: @profile_result.title
+Department: @profile_result.department
+Level: @profile_result.level
+"""
+
+[pipe.multi_beta_finalize]
+type = "PipeLLM"
+description = "Finalize as plain text"
+inputs = { summary_result = "Summary" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Format this summary as plain text:
+@summary_result.content
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds b/tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds
new file mode 100644
index 000000000..330cc640a
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds
@@ -0,0 +1,31 @@
+domain = "native_text_test"
+description = "PipeSequence using only native Text concepts (no dynamic classes)"
+
+[pipe]
+[pipe.native_text_sequence]
+type = "PipeSequence"
+description = "Two-step text pipeline using native Text concepts"
+output = "Text"
+steps = [
+ { pipe = "step_one", result = "step_one_result" },
+ { pipe = "step_two", result = "step_two_result" },
+]
+
+[pipe.step_one]
+type = "PipeLLM"
+description = "First step: generate text"
+output = "Text"
+model = "$testing-text"
+prompt = "Write a single sentence about the color blue."
+
+[pipe.step_two]
+type = "PipeLLM"
+description = "Second step: transform text"
+inputs = { step_one_result = "Text" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Rewrite the following text in a formal tone:
+
+@step_one_result.text
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds
new file mode 100644
index 000000000..0c805be4d
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds
@@ -0,0 +1,40 @@
+domain = "temporal_batch_test"
+description = "PipeBatch within a PipeSequence for Temporal fan-out child workflow dispatch testing"
+
+[concept]
+Topic = "A subject for content generation"
+TopicNote = "A short note about a topic"
+
+[pipe]
+[pipe.temporal_batch_sequence]
+type = "PipeSequence"
+description = "Generate topics then process each via batch"
+output = "TopicNote[]"
+steps = [
+ { pipe = "temporal_generate_topics", result = "topics" },
+ { pipe = "batch_temporal_describe_topics", result = "notes" },
+]
+
+[pipe.temporal_generate_topics]
+type = "PipeLLM"
+description = "Generate a list of topics"
+output = "Topic[3]"
+model = "$testing-text"
+prompt = "List exactly 3 topics: science, history, music"
+
+[pipe.batch_temporal_describe_topics]
+type = "PipeBatch"
+description = "Describe each topic via fan-out child workflows"
+inputs = { topics = "Topic[]" }
+output = "TopicNote[]"
+branch_pipe_code = "temporal_describe_topic"
+input_list_name = "topics"
+input_item_name = "topic"
+
+[pipe.temporal_describe_topic]
+type = "PipeLLM"
+description = "Write a short note about a topic"
+inputs = { topic = "Topic" }
+output = "TopicNote"
+model = "$testing-text"
+prompt = "Write one sentence about: @topic"
diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_combined.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_combined.mthds
new file mode 100644
index 000000000..db8f473c3
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/temporal_combined.mthds
@@ -0,0 +1,76 @@
+domain = "temporal_combined_test"
+description = "Combined PipeParallel + PipeCondition in a PipeSequence for nested Temporal dispatch testing"
+
+[concept.QualityReport]
+description = "A quality assessment report"
+
+[concept.QualityReport.structure]
+assessment = { type = "text", required = true, description = "Quality assessment" }
+confidence = { type = "text", required = true, description = "Confidence level" }
+
+[concept]
+PartContent = "Content generated by a parallel branch"
+
+[pipe]
+[pipe.temporal_combined_pipeline]
+type = "PipeSequence"
+description = "Combines parallel and condition dispatch patterns"
+output = "QualityReport"
+steps = [
+ { pipe = "parallel_generate", result = "parallel_done" },
+ { pipe = "quality_gate", result = "gate_result" },
+ { pipe = "produce_report", result = "final_report" },
+]
+
+[pipe.parallel_generate]
+type = "PipeParallel"
+description = "Generate two pieces of content in parallel"
+output = "Text"
+add_each_output = true
+branches = [
+ { pipe = "generate_part_a", result = "part_a" },
+ { pipe = "generate_part_b", result = "part_b" },
+]
+
+[pipe.generate_part_a]
+type = "PipeLLM"
+description = "Generate part A"
+output = "PartContent"
+model = "$testing-text"
+prompt = "Output exactly: Part A content"
+
+[pipe.generate_part_b]
+type = "PipeLLM"
+description = "Generate part B"
+output = "PartContent"
+model = "$testing-text"
+prompt = "Output exactly: Part B content"
+
+[pipe.quality_gate]
+type = "PipeCondition"
+description = "Quality check on generated content"
+output = "Text"
+expression_template = "pass"
+default_outcome = "handle_failure"
+
+[pipe.quality_gate.outcomes]
+pass = "continue"
+
+[pipe.handle_failure]
+type = "PipeLLM"
+description = "Handle quality check failure"
+output = "Text"
+model = "$testing-text"
+prompt = "Output exactly: quality_failed"
+
+[pipe.produce_report]
+type = "PipeLLM"
+description = "Produce the final quality report"
+inputs = { part_a = "PartContent", part_b = "PartContent" }
+output = "QualityReport"
+model = "$testing-text"
+prompt = """
+Create a quality report from:
+Part A: @part_a
+Part B: @part_b
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_compose.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_compose.mthds
new file mode 100644
index 000000000..dddd53046
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/temporal_compose.mthds
@@ -0,0 +1,44 @@
+domain = "temporal_compose_test"
+description = "PipeCompose operator within a PipeSequence for Temporal testing with deferred hydration"
+
+[concept.Report]
+description = "A simple report with title and body"
+
+[concept.Report.structure]
+title = { type = "text", required = true, description = "Report title" }
+body = { type = "text", required = true, description = "Report body" }
+
+[pipe]
+[pipe.temporal_compose_sequence]
+type = "PipeSequence"
+description = "Generate title and body then compose into a Report"
+output = "Report"
+steps = [
+ { pipe = "generate_title", result = "title_text" },
+ { pipe = "generate_body", result = "body_text" },
+ { pipe = "compose_report", result = "final_report" },
+]
+
+[pipe.generate_title]
+type = "PipeLLM"
+description = "Generate a report title"
+output = "Text"
+model = "$testing-text"
+prompt = "Output exactly: Quarterly Review"
+
+[pipe.generate_body]
+type = "PipeLLM"
+description = "Generate a report body"
+output = "Text"
+model = "$testing-text"
+prompt = "Output exactly: Revenue increased by 15 percent."
+
+[pipe.compose_report]
+type = "PipeCompose"
+description = "Compose title and body into a structured Report"
+inputs = { title_text = "Text", body_text = "Text" }
+output = "Report"
+
+[pipe.compose_report.construct]
+title = { from = "title_text.text" }
+body = { from = "body_text.text" }
diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_condition.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_condition.mthds
new file mode 100644
index 000000000..977804f92
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/temporal_condition.mthds
@@ -0,0 +1,55 @@
+domain = "temporal_condition_test"
+description = "PipeCondition within a PipeSequence for Temporal child workflow dispatch testing"
+
+[concept]
+CategoryLabel = "A category label for routing"
+
+[pipe]
+[pipe.temporal_condition_sequence]
+type = "PipeSequence"
+description = "Sequence that generates a category then routes via condition"
+output = "Text"
+steps = [
+ { pipe = "generate_category", result = "category_text" },
+ { pipe = "route_by_category", result = "routed_result" },
+]
+
+[pipe.generate_category]
+type = "PipeLLM"
+description = "Generate a category label"
+output = "CategoryLabel"
+model = "$testing-text"
+prompt = "Output exactly one word: alpha"
+
+[pipe.route_by_category]
+type = "PipeCondition"
+description = "Route based on category text content"
+inputs = { category_text = "CategoryLabel" }
+output = "Text"
+expression_template = "{{ category_text.text }}"
+default_outcome = "handle_default"
+
+[pipe.route_by_category.outcomes]
+alpha = "handle_alpha"
+beta = "handle_beta"
+
+[pipe.handle_alpha]
+type = "PipeLLM"
+description = "Handle alpha category"
+output = "Text"
+model = "$testing-text"
+prompt = "Output exactly: handled_alpha"
+
+[pipe.handle_beta]
+type = "PipeLLM"
+description = "Handle beta category"
+output = "Text"
+model = "$testing-text"
+prompt = "Output exactly: handled_beta"
+
+[pipe.handle_default]
+type = "PipeLLM"
+description = "Handle default category"
+output = "Text"
+model = "$testing-text"
+prompt = "Output exactly: handled_default"
diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds
new file mode 100644
index 000000000..c84ee14c5
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds
@@ -0,0 +1,52 @@
+domain = "temporal_parallel_test"
+description = "PipeParallel within a PipeSequence for Temporal concurrent child workflow dispatch testing"
+
+[concept]
+ToneAnalysis = "Analysis of text tone"
+LengthAnalysis = "Analysis of text length"
+
+[pipe]
+[pipe.temporal_parallel_sequence]
+type = "PipeSequence"
+description = "Sequence wrapping a parallel dispatch then a summary"
+output = "Text"
+steps = [
+ { pipe = "analyze_in_parallel", result = "parallel_done" },
+ { pipe = "summarize_results", result = "summary" },
+]
+
+[pipe.analyze_in_parallel]
+type = "PipeParallel"
+description = "Two branches running concurrently as separate child workflows"
+output = "Text"
+add_each_output = true
+branches = [
+ { pipe = "branch_tone", result = "tone_result" },
+ { pipe = "branch_length", result = "length_result" },
+]
+
+[pipe.branch_tone]
+type = "PipeLLM"
+description = "Analyze tone"
+output = "ToneAnalysis"
+model = "$testing-text"
+prompt = "Output exactly: formal_tone"
+
+[pipe.branch_length]
+type = "PipeLLM"
+description = "Analyze length"
+output = "LengthAnalysis"
+model = "$testing-text"
+prompt = "Output exactly: medium_length"
+
+[pipe.summarize_results]
+type = "PipeLLM"
+description = "Summarize parallel results"
+inputs = { tone_result = "ToneAnalysis", length_result = "LengthAnalysis" }
+output = "Text"
+model = "$testing-text"
+prompt = """
+Combine these results:
+Tone: @tone_result
+Length: @length_result
+"""
diff --git a/tests/integration/pipelex/temporal/library_crate/test_distributed_scopes.py b/tests/integration/pipelex/temporal/library_crate/test_distributed_scopes.py
new file mode 100644
index 000000000..2b85464a6
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_distributed_scopes.py
@@ -0,0 +1,87 @@
+"""Two scoped workers on the same task queue: router (workflow-only) + runner (activity-only).
+
+Validates that the worker scopes mechanism resolves correctly and that two scoped
+workers can coexist on the same task queue (Temporal SDK rejects overlapping task
+types — workflow-only + activity-only is the only viable split on one queue).
+
+NOTE: this in-process test cannot reproduce the cross-process registry-propagation
+bug described in TODOS.md, because both workers share the same Python process and
+the same global `KajsonManager`. The deterministic cross-process regression lives
+in the `/temporal-e2e-validate` skill, which spawns separate worker processes via
+the `--scope router`/`--scope runner` CLI flags.
+"""
+
+import uuid
+from collections.abc import Generator
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.config import get_config
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData
+
+
[email protected](scope="class")
+def pipe_job_with_dynamic_concept(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ yield from pipe_job_from_bundle(
+ bundle_file=DeferredHydrationTestData.BUNDLE_FILE,
+ pipe_code=DeferredHydrationTestData.PIPE_CODE,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](loop_scope="class")
+class TestDistributedScopes:
+ async def test_dynamic_concept_across_router_and_runner_workers(
+ self,
+ pipe_job_with_dynamic_concept: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Two scoped workers on the same queue: WfPipeRun must land on a different
+ worker than WfPipeRouter. Dynamic concept must still resolve end-to-end.
+ """
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ worker_scopes = get_config().temporal.worker_scopes
+ router_scope = worker_scopes.scopes["router"]
+ runner_scope = worker_scopes.scopes["runner"]
+
+ async with (
+ get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ scope=router_scope,
+ ),
+ get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ scope=runner_scope,
+ ),
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=pipe_job_with_dynamic_concept,
+ id=workflow_id,
+ task_queue=task_queue,
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ rehydrate_pipe_output(pipe_output, pipe_job=pipe_job_with_dynamic_concept)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+
+ greeting_stuff = working_memory.get_stuff("greeting_result")
+ assert isinstance(greeting_stuff.content, StructuredContent)
+ assert hasattr(greeting_stuff.content, "message")
+ assert hasattr(greeting_stuff.content, "language")
diff --git a/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.py b/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.py
new file mode 100644
index 000000000..d4098a393
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.py
@@ -0,0 +1,205 @@
+"""Integration test for submitter-side rehydration without bundle pre-loaded.
+
+Validates that `rehydrate_pipe_output_with_crate` succeeds against an empty
+submitter-side global registry by spinning up its own per-call scoped library
+from the `LibraryCrate` carried by the `PipeJob`.
+
+This is the production guarantee for remote API clients that built the
+`PipeJob` from a serialized `LibraryCrate` and never loaded the bundle into
+their global `KajsonManager` registry.
+"""
+
+import uuid
+from collections.abc import Generator
+
+import pytest
+from kajson.kajson_manager import KajsonManager
+from temporalio.client import Client as TemporalClient
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.hub import get_current_library, set_current_library, teardown_current_library
+from pipelex.pipe_run.exceptions import PipeJobError
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.hydration import hydrate_working_memory
+from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData
+
+GREETING_CLASS_NAME = f"{DeferredHydrationTestData.DOMAIN}__Greeting"
+
+
+def _current_library_id_or_none() -> str | None:
+ """Read current_library without raising when none is set."""
+ try:
+ return get_current_library()
+ except RuntimeError:
+ return None
+
+
[email protected](scope="class")
+def pipe_job_isolated_dynamic_concept() -> Generator[PipeJob, None, None]:
+ """PipeJob built from the dynamic-concept bundle with a scoped (isolated) ClassRegistry.
+
+ Forces ``isolated_registry=True`` so the dynamic Greeting class lands in
+ the fixture's scoped registry and the global ``KajsonManager`` registry
+ stays clean — the precondition for the "submitter without bundle
+ pre-loaded" scenario this module exercises.
+ """
+ yield from pipe_job_from_bundle(
+ bundle_file=DeferredHydrationTestData.BUNDLE_FILE,
+ pipe_code=DeferredHydrationTestData.PIPE_CODE,
+ isolated_registry=True,
+ )
+
+
[email protected](loop_scope="class")
+class TestSubmitterWithoutBundleLoaded:
+ async def _dispatch_and_get_deferred_pipe_output(
+ self,
+ pipe_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> PipeOutput:
+ """Run the workflow and return the PipeOutput as-received (with working_memory_raw populated)."""
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=pipe_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ assert pipe_output.working_memory_raw is not None, "Worker should have returned a deferred PipeOutput (working_memory_raw populated)"
+ return pipe_output
+
+ async def test_helper_rehydrates_with_clean_global_registry(
+ self,
+ pipe_job_isolated_dynamic_concept: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """rehydrate_pipe_output_with_crate succeeds when the submitter never loaded the bundle.
+
+ The fixture runs in isolated-registry mode, so the dynamic Greeting class
+ lives only in the fixture's scoped library, never in the global registry.
+ We additionally clear the current_library ContextVar before calling the
+ helper so the active class registry falls back to the (clean) global —
+ which simulates a remote submitter that received the PipeJob+crate over
+ the wire and never touched a Pipelex library at all.
+ """
+ pipe_output = await self._dispatch_and_get_deferred_pipe_output(
+ pipe_job_isolated_dynamic_concept,
+ temporal_client,
+ )
+
+ global_registry = KajsonManager.get_class_registry()
+ assert global_registry.get_class(name=GREETING_CLASS_NAME) is None, (
+ f"Precondition violated: '{GREETING_CLASS_NAME}' is in the global KajsonManager registry; "
+ "the test cannot prove the submitter-without-bundle path."
+ )
+
+ previous_library_id = _current_library_id_or_none()
+ teardown_current_library()
+ try:
+ rehydrated = rehydrate_pipe_output_with_crate(
+ pipe_output,
+ library_crate=pipe_job_isolated_dynamic_concept.library_crate,
+ )
+
+ assert rehydrated is pipe_output, "Helper should mutate in place and return the same instance"
+ assert rehydrated.working_memory_raw is None, "working_memory_raw should be cleared after rehydration"
+
+ greeting_stuff = rehydrated.working_memory.get_stuff("greeting_result")
+ assert isinstance(greeting_stuff.content, StructuredContent), (
+ f"Expected StructuredContent for 'greeting_result' after rehydration, got {type(greeting_stuff.content).__name__}"
+ )
+ assert hasattr(greeting_stuff.content, "message"), "Hydrated Greeting should have 'message' field"
+ assert hasattr(greeting_stuff.content, "language"), "Hydrated Greeting should have 'language' field"
+
+ assert global_registry.get_class(name=GREETING_CLASS_NAME) is None, (
+ f"Helper leaked '{GREETING_CLASS_NAME}' into the global KajsonManager registry"
+ )
+ assert _current_library_id_or_none() is None, "Helper should have restored the previous (None) current_library after teardown"
+ finally:
+ if previous_library_id is not None:
+ set_current_library(library_id=previous_library_id)
+
+ async def test_no_crate_path_fails_without_bundle(
+ self,
+ pipe_job_isolated_dynamic_concept: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Counterpart: rehydrating without the crate fails when the submitter has no bundle.
+
+ Demonstrates the exact silent-failure surface the helper closes — a
+ plain `hydrate_working_memory` call against a clean global registry
+ cannot resolve the dynamic Greeting class. The new helper avoids this
+ by opening a per-call scoped library from the crate.
+ """
+ pipe_output = await self._dispatch_and_get_deferred_pipe_output(
+ pipe_job_isolated_dynamic_concept,
+ temporal_client,
+ )
+ assert pipe_output.working_memory_raw is not None
+
+ previous_library_id = _current_library_id_or_none()
+ teardown_current_library()
+ try:
+ with pytest.raises(PipeJobError):
+ hydrate_working_memory(pipe_output.working_memory_raw)
+ finally:
+ if previous_library_id is not None:
+ set_current_library(library_id=previous_library_id)
+
+ async def test_repeated_rehydrations_do_not_leak(
+ self,
+ pipe_job_isolated_dynamic_concept: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Repeated back-to-back submitter rehydrations leave no residual state.
+
+ Dispatches one workflow, then invokes the helper twice in sequence on
+ independent copies of the deferred PipeOutput. Catches per-call
+ scoped-library leaks (open libraries piling up, dynamic classes
+ bleeding into the global registry, current_library not restored).
+ Sequential rather than `asyncio.gather`-parallel because the helper
+ is synchronous — gathering it would only run sequentially anyway.
+ """
+ pipe_output_template = await self._dispatch_and_get_deferred_pipe_output(
+ pipe_job_isolated_dynamic_concept,
+ temporal_client,
+ )
+ raw_payload = pipe_output_template.working_memory_raw
+ assert raw_payload is not None
+
+ global_registry = KajsonManager.get_class_registry()
+ assert global_registry.get_class(name=GREETING_CLASS_NAME) is None
+
+ previous_library_id = _current_library_id_or_none()
+ teardown_current_library()
+ try:
+ crate = pipe_job_isolated_dynamic_concept.library_crate
+ for _index in range(2):
+ pipe_output = pipe_output_template.model_copy(update={"working_memory_raw": dict(raw_payload)})
+ rehydrated = rehydrate_pipe_output_with_crate(pipe_output, library_crate=crate)
+ greeting_stuff = rehydrated.working_memory.get_stuff("greeting_result")
+ assert isinstance(greeting_stuff.content, StructuredContent)
+ assert hasattr(greeting_stuff.content, "message")
+ assert hasattr(greeting_stuff.content, "language")
+ assert _current_library_id_or_none() is None, "Helper should leave current_library back at None after each call"
+ assert global_registry.get_class(name=GREETING_CLASS_NAME) is None, (
+ f"Iteration leaked '{GREETING_CLASS_NAME}' into the global KajsonManager registry"
+ )
+ finally:
+ if previous_library_id is not None:
+ set_current_library(library_id=previous_library_id)
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_combined_pipeline.py b/tests/integration/pipelex/temporal/library_crate/test_wf_combined_pipeline.py
new file mode 100644
index 000000000..2ae9f64dd
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_combined_pipeline.py
@@ -0,0 +1,79 @@
+"""Integration tests for combined controller types through Temporal workflows.
+
+Validates that PipeParallel + PipeCondition nested within a single PipeSequence execute
+correctly on Temporal workers via LibraryCrate propagation. This exercises the deepest
+child workflow dispatch chain: sequence → parallel children → condition child → outcome child.
+Also exercises Phase 3 deferred hydration via the QualityReport inline structure concept.
+"""
+
+import uuid
+from datetime import timedelta
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.test_data import CombinedPipelineTemporalTestData
+
+
[email protected](loop_scope="class")
+class TestWfCombinedPipeline:
+ @staticmethod
+ def _assert_crate_structure(pipe_job: PipeJob) -> None:
+ crate = pipe_job.library_crate
+ assert crate is not None, "library_crate should be set on PipeJob"
+ assert crate.fingerprint, "fingerprint should be non-empty"
+
+ assert len(crate.pipes) == len(CombinedPipelineTemporalTestData.EXPECTED_PIPE_REFS), (
+ f"Expected {len(CombinedPipelineTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"
+ )
+ for pipe_ref in CombinedPipelineTemporalTestData.EXPECTED_PIPE_REFS:
+ assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"
+
+ async def test_crate_contains_combined_pipes(self, combined_job: PipeJob):
+ """Verify the crate has all pipe_refs from parallel, condition, and LLM pipes."""
+ self._assert_crate_structure(combined_job)
+
+ async def test_combined_pipeline_via_temporal(
+ self,
+ combined_job: PipeJob,
+ temporal_client: TemporalClient,
+ ):
+ """Nested parallel + condition dispatch produces a structured QualityReport on the worker."""
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=combined_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ execution_timeout=timedelta(seconds=30),
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ rehydrate_pipe_output(pipe_output)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+
+ for stuff_name in CombinedPipelineTemporalTestData.EXPECTED_STUFF_NAMES:
+ assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"
+
+ # Verify QualityReport has expected structured fields
+ report_stuff = working_memory.get_stuff("final_report")
+ assert isinstance(report_stuff.content, StructuredContent), (
+ f"Expected StructuredContent for final_report, got {type(report_stuff.content).__name__}"
+ )
+ for field_name in CombinedPipelineTemporalTestData.EXPECTED_REPORT_FIELDS:
+ assert hasattr(report_stuff.content, field_name), f"QualityReport missing field '{field_name}' — deferred hydration may have failed"
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_concept_isolation.py b/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_concept_isolation.py
new file mode 100644
index 000000000..c6d02547d
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_concept_isolation.py
@@ -0,0 +1,134 @@
+"""Concurrent concept isolation tests for Temporal workflows.
+
+Validates that two workflows running simultaneously on the same worker can each
+define a concept called 'Result' with incompatible structures, and the per-workflow
+ClassRegistry scoping (via ContextVar) keeps them isolated.
+
+Without proper scoping, the bare class name 'Result' would collide in the registry,
+causing one workflow to deserialize with the wrong field structure — a silent data
+corruption bug.
+
+┌──────────────────────────────────────────────────────────────┐
+│ Temporal Test Server (in-process) │
+├──────────────────────────────────────────────────────────────┤
+│ ┌──────────────── Worker (shared) ────────────────────────┐ │
+│ │ ┌─ Workflow A ─────────────┐ ┌─ Workflow B ─────────┐ │ │
+│ │ │ concept "Result": │ │ concept "Result": │ │ │
+│ │ │ score: int │ │ value: text │ │ │
+│ │ │ label: text │ │ confidence: number │ │ │
+│ │ │ │ │ is_valid: text │ │ │
+│ │ │ ClassRegistry A (scoped) │ │ ClassRegistry B │ │ │
+│ │ └──────────────────────────┘ └───────────────────────┘ │ │
+│ └─────────────────────────────────────────────────────────┘ │
+└──────────────────────────────────────────────────────────────┘
+"""
+
+import asyncio
+import uuid
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex import log
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.temporal_hub import get_task_manager
+from tests.integration.pipelex.temporal.library_crate.helpers import (
+ assert_structured_fields,
+ assert_stuff_names,
+ assert_text_stuff_names,
+ execute_workflow,
+)
+from tests.integration.pipelex.temporal.test_data import ConflictConceptAlphaTestData, ConflictConceptBetaTestData
+
+
[email protected](loop_scope="class")
+class TestWfConcurrentConceptIsolation:
+ async def test_alpha_solo(
+ self,
+ pipe_run_mode: PipeRunMode,
+ alpha_concept_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Alpha pipeline alone: Result should have score and label fields."""
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ output = await execute_workflow(alpha_concept_job, temporal_client, task_queue)
+
+ assert_stuff_names(output, ConflictConceptAlphaTestData.EXPECTED_STUFF_NAMES)
+ assert_structured_fields(output, "alpha_result", ConflictConceptAlphaTestData.EXPECTED_RESULT_FIELDS)
+ log.info(f"Alpha solo passed (mode={pipe_run_mode})")
+
+ async def test_beta_solo(
+ self,
+ pipe_run_mode: PipeRunMode,
+ beta_concept_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Beta pipeline alone: Result should have value, confidence, and is_valid fields."""
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ output = await execute_workflow(beta_concept_job, temporal_client, task_queue)
+
+ assert_stuff_names(output, ConflictConceptBetaTestData.EXPECTED_STUFF_NAMES)
+ assert_structured_fields(output, "beta_result", ConflictConceptBetaTestData.EXPECTED_RESULT_FIELDS)
+ log.info(f"Beta solo passed (mode={pipe_run_mode})")
+
+ async def test_concurrent_different_results(
+ self,
+ pipe_run_mode: PipeRunMode,
+ alpha_concept_job: PipeJob,
+ beta_concept_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Both pipelines run simultaneously on the same worker.
+
+ Each defines concept 'Result' with different fields. Without per-workflow
+ ClassRegistry scoping, one would overwrite the other in the registry.
+ """
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ alpha_output, beta_output = await asyncio.gather(
+ execute_workflow(alpha_concept_job, temporal_client, task_queue),
+ execute_workflow(beta_concept_job, temporal_client, task_queue),
+ )
+
+ # Alpha: Result has score + label
+ assert_stuff_names(alpha_output, ConflictConceptAlphaTestData.EXPECTED_STUFF_NAMES)
+ assert_structured_fields(alpha_output, "alpha_result", ConflictConceptAlphaTestData.EXPECTED_RESULT_FIELDS)
+
+ # Beta: Result has value + confidence + is_valid
+ assert_stuff_names(beta_output, ConflictConceptBetaTestData.EXPECTED_STUFF_NAMES)
+ assert_structured_fields(beta_output, "beta_result", ConflictConceptBetaTestData.EXPECTED_RESULT_FIELDS)
+
+ # Verify summary outputs are TextContent
+ assert_text_stuff_names(alpha_output, ["alpha_summary"])
+ assert_text_stuff_names(beta_output, ["beta_summary"])
+
+ log.info(f"Concurrent concept isolation passed (mode={pipe_run_mode})")
+
+ async def test_repeated_concurrent(
+ self,
+ pipe_run_mode: PipeRunMode,
+ alpha_concept_job: PipeJob,
+ beta_concept_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Run the concurrent test 5 times to catch intermittent ContextVar race conditions."""
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ for round_index in range(5):
+ alpha_output, beta_output = await asyncio.gather(
+ execute_workflow(alpha_concept_job, temporal_client, task_queue),
+ execute_workflow(beta_concept_job, temporal_client, task_queue),
+ )
+
+ assert_structured_fields(alpha_output, "alpha_result", ConflictConceptAlphaTestData.EXPECTED_RESULT_FIELDS)
+ assert_structured_fields(beta_output, "beta_result", ConflictConceptBetaTestData.EXPECTED_RESULT_FIELDS)
+ log.verbose(f"Round {round_index + 1}/5 passed")
+
+ log.info(f"Repeated concurrent concept isolation passed (mode={pipe_run_mode})")
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_pipe_isolation.py b/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_pipe_isolation.py
new file mode 100644
index 000000000..98058d133
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_pipe_isolation.py
@@ -0,0 +1,115 @@
+"""Concurrent pipe isolation tests for Temporal workflows.
+
+Validates that two workflows running simultaneously on the same worker each load
+their own LibraryCrate and resolve pipe_refs through per-workflow library scoping
+(via ContextVar). Alpha's pipes (about colors) and beta's pipes (about animals)
+live in separate domains and are resolved independently on each workflow.
+
+┌──────────────────────────────────────────────────────────────┐
+│ Temporal Test Server (in-process) │
+├──────────────────────────────────────────────────────────────┤
+│ ┌──────────────── Worker (shared) ────────────────────────┐ │
+│ │ ┌─ Workflow A ─────────────┐ ┌─ Workflow B ──────────┐ │ │
+│ │ │ domain: pipe_conflict_a │ │ domain: pipe_conflict_b│ │ │
+│ │ │ alpha_shared_step: │ │ beta_shared_step: │ │ │
+│ │ │ prompt: about colors │ │ prompt: about animals│ │
+│ │ │ │ │ │ │ │
+│ │ │ Library A (ContextVar) │ │ Library B (ContextVar) │ │ │
+│ │ └──────────────────────────┘ └────────────────────────┘ │ │
+│ └─────────────────────────────────────────────────────────┘ │
+└──────────────────────────────────────────────────────────────┘
+"""
+
+import asyncio
+import uuid
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex import log
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.temporal_hub import get_task_manager
+from tests.integration.pipelex.temporal.library_crate.helpers import assert_text_stuff_names, execute_workflow
+from tests.integration.pipelex.temporal.test_data import ConflictPipeAlphaTestData, ConflictPipeBetaTestData
+
+
[email protected](loop_scope="class")
+class TestWfConcurrentPipeIsolation:
+ async def test_pipe_alpha_solo(
+ self,
+ pipe_run_mode: PipeRunMode,
+ alpha_pipe_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Alpha pipeline alone: shared_step should generate text about colors."""
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ output = await execute_workflow(alpha_pipe_job, temporal_client, task_queue)
+
+ assert_text_stuff_names(output, ConflictPipeAlphaTestData.EXPECTED_STUFF_NAMES)
+ log.info(f"Pipe alpha solo passed (mode={pipe_run_mode})")
+
+ async def test_pipe_beta_solo(
+ self,
+ pipe_run_mode: PipeRunMode,
+ beta_pipe_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Beta pipeline alone: shared_step should generate text about animals."""
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ output = await execute_workflow(beta_pipe_job, temporal_client, task_queue)
+
+ assert_text_stuff_names(output, ConflictPipeBetaTestData.EXPECTED_STUFF_NAMES)
+ log.info(f"Pipe beta solo passed (mode={pipe_run_mode})")
+
+ async def test_concurrent_pipe_resolution(
+ self,
+ pipe_run_mode: PipeRunMode,
+ alpha_pipe_job: PipeJob,
+ beta_pipe_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Both pipelines run simultaneously on the same worker.
+
+ Each defines pipe 'shared_step' with a different prompt. Without per-workflow
+ library scoping, get_required_pipe('shared_step') could resolve the wrong one.
+ """
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ alpha_output, beta_output = await asyncio.gather(
+ execute_workflow(alpha_pipe_job, temporal_client, task_queue),
+ execute_workflow(beta_pipe_job, temporal_client, task_queue),
+ )
+
+ assert_text_stuff_names(alpha_output, ConflictPipeAlphaTestData.EXPECTED_STUFF_NAMES)
+ assert_text_stuff_names(beta_output, ConflictPipeBetaTestData.EXPECTED_STUFF_NAMES)
+
+ log.info(f"Concurrent pipe isolation passed (mode={pipe_run_mode})")
+
+ async def test_repeated_concurrent_pipe_resolution(
+ self,
+ pipe_run_mode: PipeRunMode,
+ alpha_pipe_job: PipeJob,
+ beta_pipe_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Run the concurrent test 5 times to catch intermittent race conditions."""
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ for round_index in range(5):
+ alpha_output, beta_output = await asyncio.gather(
+ execute_workflow(alpha_pipe_job, temporal_client, task_queue),
+ execute_workflow(beta_pipe_job, temporal_client, task_queue),
+ )
+
+ assert_text_stuff_names(alpha_output, ConflictPipeAlphaTestData.EXPECTED_STUFF_NAMES)
+ assert_text_stuff_names(beta_output, ConflictPipeBetaTestData.EXPECTED_STUFF_NAMES)
+ log.verbose(f"Round {round_index + 1}/5 passed")
+
+ log.info(f"Repeated concurrent pipe isolation passed (mode={pipe_run_mode})")
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py b/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py
new file mode 100644
index 000000000..a04f3574e
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py
@@ -0,0 +1,162 @@
+"""Integration tests for deferred WorkingMemory hydration through Temporal workflows.
+
+Validates Phase 3: PipeSequence controllers with dynamic concept classes execute
+on Temporal workers. The WorkingMemory is serialized as a raw dict before dispatch,
+then hydrated on the worker after load_from_crate() registers the dynamic classes.
+"""
+
+import uuid
+from collections.abc import Generator
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.stuff_content_factory import StuffContentFactory
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_required_concept
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData
+
+
[email protected](scope="class")
+def pipe_job_with_dynamic_concept(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """Build a PipeJob from the dynamic concept test bundle (mthds_content with inline structure)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=DeferredHydrationTestData.BUNDLE_FILE,
+ pipe_code=DeferredHydrationTestData.PIPE_CODE,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](loop_scope="class")
+class TestWfDeferredHydration:
+ async def test_crate_contains_dynamic_concept(self, pipe_job_with_dynamic_concept: PipeJob) -> None:
+ """The crate should contain the Greeting concept with its structure."""
+ crate = pipe_job_with_dynamic_concept.library_crate
+ assert crate is not None
+
+ assert len(crate.pipes) == len(DeferredHydrationTestData.EXPECTED_PIPE_REFS)
+ for pipe_ref in DeferredHydrationTestData.EXPECTED_PIPE_REFS:
+ assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"
+
+ # Verify the Greeting concept with structure is in the crate
+ greeting_ref = f"{DeferredHydrationTestData.DOMAIN}.Greeting"
+ assert greeting_ref in crate.concepts, f"Expected concept '{greeting_ref}' not found in crate"
+
+ async def test_dynamic_concept_sequence_via_temporal(
+ self,
+ pipe_job_with_dynamic_concept: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Full round-trip: dynamic concept classes are registered on the worker via deferred hydration."""
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=pipe_job_with_dynamic_concept,
+ id=workflow_id,
+ task_queue=task_queue,
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ rehydrate_pipe_output(pipe_output)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+
+ for stuff_name in DeferredHydrationTestData.EXPECTED_STUFF_NAMES:
+ assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"
+ stuff = working_memory.get_stuff(stuff_name)
+ assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"
+
+ # Verify greeting_result has StructuredContent with the right fields
+ greeting_stuff = working_memory.get_stuff("greeting_result")
+ assert isinstance(greeting_stuff.content, StructuredContent), f"Expected StructuredContent, got {type(greeting_stuff.content).__name__}"
+ assert hasattr(greeting_stuff.content, "message"), "Greeting content should have 'message' field"
+ assert hasattr(greeting_stuff.content, "language"), "Greeting content should have 'language' field"
+
+ # Verify summary_result is TextContent
+ summary_stuff = working_memory.get_stuff("summary_result")
+ assert isinstance(summary_stuff.content, TextContent)
+
+ async def test_input_wm_deferred_hydration_round_trip(
+ self,
+ pipe_job_with_dynamic_concept: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Exercises the full deferred hydration path: input WM with dynamic Greeting concept
+ is serialized via prepare_for_temporal(), sent through Temporal, and correctly
+ hydrated on the worker before the pipe sequence runs.
+ """
+ # Build a Stuff with the dynamic Greeting concept content
+ greeting_concept = get_required_concept(concept_ref=f"{DeferredHydrationTestData.DOMAIN}.Greeting")
+ greeting_content = StuffContentFactory.make_stuff_content_from_concept_required(
+ concept=greeting_concept,
+ value={"message": "Bonjour le monde", "language": "French"},
+ )
+ greeting_stuff = Stuff(
+ stuff_code="test_input",
+ stuff_name="greeting_result",
+ concept=greeting_concept,
+ content=greeting_content,
+ )
+
+ # Create a WorkingMemory with the dynamic-concept Stuff pre-populated
+ input_wm = WorkingMemory()
+ input_wm.root["greeting_result"] = greeting_stuff
+
+ # Copy the pipe job with pre-populated WM, then prepare for Temporal dispatch
+ pipe_job_with_input = pipe_job_with_dynamic_concept.model_copy(update={"working_memory": input_wm})
+ prepared_job = pipe_job_with_input.prepare_for_temporal()
+
+ # Verify prepare_for_temporal converted WM to raw dict
+ assert prepared_job.working_memory is None, "prepare_for_temporal should set working_memory to None"
+ assert prepared_job.working_memory_raw is not None, "prepare_for_temporal should populate working_memory_raw"
+ assert "greeting_result" in prepared_job.working_memory_raw.get("root", {}), "Raw WM should contain greeting_result"
+
+ # Dispatch through Temporal — the worker will hydrate working_memory_raw back to typed WM
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=prepared_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ rehydrate_pipe_output(pipe_output)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+
+ # The greeting_result should still exist (was in input WM, and also produced by the first pipe step)
+ assert working_memory.is_stuff_exists("greeting_result"), "greeting_result should be in output WM"
+ greeting_stuff_out = working_memory.get_stuff("greeting_result")
+ assert isinstance(greeting_stuff_out.content, StructuredContent), (
+ f"Expected StructuredContent after hydration, got {type(greeting_stuff_out.content).__name__}"
+ )
+ assert hasattr(greeting_stuff_out.content, "message"), "Hydrated Greeting should have 'message' field"
+ assert hasattr(greeting_stuff_out.content, "language"), "Hydrated Greeting should have 'language' field"
+
+ # summary_result should also exist (produced by the second pipe step)
+ assert working_memory.is_stuff_exists("summary_result"), "summary_result should be in output WM"
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_library_crate.py b/tests/integration/pipelex/temporal/library_crate/test_wf_library_crate.py
new file mode 100644
index 000000000..7e8211727
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_library_crate.py
@@ -0,0 +1,115 @@
+"""Integration tests for LibraryCrate propagation through Temporal workflows.
+
+Validates Phase 2 of the master plan: PipeSequence controllers execute on Temporal workers
+because the LibraryCrate ships inside the PipeJob and gets loaded on the worker.
+Uses a bundle with only native Text concepts to avoid the Layer 1 Kajson issue (Phase 3).
+"""
+
+import uuid
+from datetime import timedelta
+
+import pytest
+from temporalio.client import Client as TemporalClient
+from temporalio.client import WorkflowFailureError
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.test_data import LibraryCrateTestData
+
+
[email protected](loop_scope="class")
+class TestWfLibraryCrate:
+ @staticmethod
+ def _assert_crate_structure(pipe_job: PipeJob) -> None:
+ crate = pipe_job.library_crate
+ assert crate is not None, "library_crate should be set on PipeJob"
+ assert crate.fingerprint, "fingerprint should be non-empty"
+
+ assert len(crate.pipes) == len(LibraryCrateTestData.EXPECTED_PIPE_REFS), (
+ f"Expected {len(LibraryCrateTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"
+ )
+ for pipe_ref in LibraryCrateTestData.EXPECTED_PIPE_REFS:
+ assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"
+
+ json_str = crate.model_dump_json()
+ assert len(json_str) > 0
+ roundtripped = crate.model_validate_json(json_str)
+ assert roundtripped.fingerprint == crate.fingerprint
+
+ async def _execute_workflow_and_assert(self, pipe_job: PipeJob, temporal_client: TemporalClient) -> None:
+ """Execute WfPipeRouter on an inline worker and assert PipeOutput is returned."""
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=pipe_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ rehydrate_pipe_output(pipe_output)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+
+ for stuff_name in LibraryCrateTestData.EXPECTED_STUFF_NAMES:
+ assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"
+ stuff = working_memory.get_stuff(stuff_name)
+ assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"
+
+ async def test_crate_structure_from_dirs(self, pipe_job_from_library_dirs: PipeJob):
+ self._assert_crate_structure(pipe_job_from_library_dirs)
+
+ async def test_crate_structure_from_mthds_content(self, pipe_job_from_mthds_content: PipeJob):
+ self._assert_crate_structure(pipe_job_from_mthds_content)
+
+ async def test_pipe_sequence_via_temporal_dry_run(
+ self,
+ pipe_job_from_library_dirs: PipeJob,
+ temporal_client: TemporalClient,
+ ):
+ """Proves get_required_pipe() works on the worker for child pipes via LibraryCrate."""
+ await self._execute_workflow_and_assert(pipe_job_from_library_dirs, temporal_client)
+
+ async def test_pipe_sequence_via_temporal_from_mthds_content(
+ self,
+ pipe_job_from_mthds_content: PipeJob,
+ temporal_client: TemporalClient,
+ ):
+ """Same as above but the crate was built from an inline mthds_content string."""
+ await self._execute_workflow_and_assert(pipe_job_from_mthds_content, temporal_client)
+
+ async def test_missing_crate_fails_pipe_resolution(
+ self,
+ pipe_job_from_mthds_content: PipeJob,
+ temporal_client: TemporalClient,
+ ):
+ """A PipeJob without a crate should fail: child pipes can't resolve on worker."""
+ pipe_job_without_crate = pipe_job_from_mthds_content.model_copy(update={"library_crate": None})
+
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ with pytest.raises(WorkflowFailureError):
+ await temporal_client.execute_workflow(
+ workflow=WfPipeRouter.run,
+ arg=pipe_job_without_crate,
+ id=workflow_id,
+ task_queue=task_queue,
+ execution_timeout=timedelta(seconds=10),
+ )
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_multi_concept_isolation.py b/tests/integration/pipelex/temporal/library_crate/test_wf_multi_concept_isolation.py
new file mode 100644
index 000000000..2f679f061
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_multi_concept_isolation.py
@@ -0,0 +1,153 @@
+"""Multi-concept isolation stress tests for Temporal workflows.
+
+Validates the worst-case scenario: two workflows running simultaneously on the same
+worker, each defining MULTIPLE dynamic concepts with the SAME names but incompatible
+structures. This exercises the full ClassRegistry scoping path under maximum pressure.
+
+Both workflows define 'Profile' and 'Summary' concepts but with completely different
+field structures. If the per-workflow ClassRegistry scoping (via ContextVar) fails,
+one workflow's Profile class could be used to deserialize the other's — silently
+returning wrong data without any error.
+
+┌──────────────────────────────────────────────────────────────┐
+│ Temporal Test Server (in-process) │
+├──────────────────────────────────────────────────────────────┤
+│ ┌──────────────── Worker (shared) ────────────────────────┐ │
+│ │ ┌─ Workflow A ─────────────┐ ┌─ Workflow B ─────────┐ │ │
+│ │ │ "Profile": │ │ "Profile": │ │ │
+│ │ │ name: text │ │ title: text │ │ │
+│ │ │ age: integer │ │ department: text │ │ │
+│ │ │ │ │ level: integer │ │ │
+│ │ │ "Summary": │ │ "Summary": │ │ │
+│ │ │ headline: text │ │ content: text │ │ │
+│ │ │ body: text │ │ │ │ │
+│ │ └──────────────────────────┘ └───────────────────────┘ │ │
+│ └─────────────────────────────────────────────────────────┘ │
+└──────────────────────────────────────────────────────────────┘
+"""
+
+import asyncio
+import uuid
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex import log
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.temporal_hub import get_task_manager
+from tests.integration.pipelex.temporal.library_crate.helpers import (
+ assert_structured_fields,
+ assert_stuff_names,
+ assert_text_stuff_names,
+ execute_workflow,
+)
+from tests.integration.pipelex.temporal.test_data import MultiConceptAlphaTestData, MultiConceptBetaTestData
+
+
+def _assert_alpha_output(output: PipeOutput) -> None:
+ """Assert alpha pipeline produced Profile(name, age) + Summary(headline, body) + Text."""
+ assert_stuff_names(output, MultiConceptAlphaTestData.EXPECTED_STUFF_NAMES)
+ assert_structured_fields(output, "profile_result", MultiConceptAlphaTestData.EXPECTED_PROFILE_FIELDS)
+ assert_structured_fields(output, "summary_result", MultiConceptAlphaTestData.EXPECTED_SUMMARY_FIELDS)
+ assert_text_stuff_names(output, ["final_result"])
+
+
+def _assert_beta_output(output: PipeOutput) -> None:
+ """Assert beta pipeline produced Profile(title, department, level) + Summary(content) + Text."""
+ assert_stuff_names(output, MultiConceptBetaTestData.EXPECTED_STUFF_NAMES)
+ assert_structured_fields(output, "profile_result", MultiConceptBetaTestData.EXPECTED_PROFILE_FIELDS)
+ assert_structured_fields(output, "summary_result", MultiConceptBetaTestData.EXPECTED_SUMMARY_FIELDS)
+ assert_text_stuff_names(output, ["final_result"])
+
+
[email protected](loop_scope="class")
+class TestWfMultiConceptIsolation:
+ async def test_multi_alpha_solo(
+ self,
+ pipe_run_mode: PipeRunMode,
+ multi_alpha_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Alpha pipeline alone: Profile(name, age) + Summary(headline, body)."""
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ output = await execute_workflow(multi_alpha_job, temporal_client, task_queue)
+
+ _assert_alpha_output(output)
+ log.info(f"Multi-concept alpha solo passed (mode={pipe_run_mode})")
+
+ async def test_multi_beta_solo(
+ self,
+ pipe_run_mode: PipeRunMode,
+ multi_beta_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Beta pipeline alone: Profile(title, department, level) + Summary(content)."""
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ output = await execute_workflow(multi_beta_job, temporal_client, task_queue)
+
+ _assert_beta_output(output)
+ log.info(f"Multi-concept beta solo passed (mode={pipe_run_mode})")
+
+ async def test_concurrent_multi_concept(
+ self,
+ pipe_run_mode: PipeRunMode,
+ multi_alpha_job: PipeJob,
+ multi_beta_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Both pipelines run simultaneously with conflicting Profile and Summary classes.
+
+ This is the worst-case scenario: two dynamic classes with the same name ('Profile')
+ and two more ('Summary') coexist on the same worker. Without scoping, alpha's
+ Profile(name, age) could be used to deserialize beta's Profile(title, department, level).
+ """
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ alpha_output, beta_output = await asyncio.gather(
+ execute_workflow(multi_alpha_job, temporal_client, task_queue),
+ execute_workflow(multi_beta_job, temporal_client, task_queue),
+ )
+
+ _assert_alpha_output(alpha_output)
+ _assert_beta_output(beta_output)
+
+ log.info(f"Concurrent multi-concept isolation passed (mode={pipe_run_mode})")
+
+ async def test_high_concurrency_multi_concept(
+ self,
+ pipe_run_mode: PipeRunMode,
+ multi_alpha_job: PipeJob,
+ multi_beta_job: PipeJob,
+ temporal_client: TemporalClient,
+ ) -> None:
+ """Run 3 pairs concurrently (6 workflows) to stress ContextVar scoping.
+
+ Exercises the scenario where many workflow coroutines are interleaved on the
+ same worker, maximizing the chance of ContextVar leaking between them.
+ """
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ results = await asyncio.gather(
+ execute_workflow(multi_alpha_job, temporal_client, task_queue),
+ execute_workflow(multi_beta_job, temporal_client, task_queue),
+ execute_workflow(multi_alpha_job, temporal_client, task_queue),
+ execute_workflow(multi_beta_job, temporal_client, task_queue),
+ execute_workflow(multi_alpha_job, temporal_client, task_queue),
+ execute_workflow(multi_beta_job, temporal_client, task_queue),
+ )
+
+ # results[0], [2], [4] are alpha; [1], [3], [5] are beta
+ for index in (0, 2, 4):
+ _assert_alpha_output(results[index])
+ for index in (1, 3, 5):
+ _assert_beta_output(results[index])
+
+ log.info(f"High concurrency multi-concept isolation passed (mode={pipe_run_mode})")
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_batch.py b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_batch.py
new file mode 100644
index 000000000..69ddd5d60
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_batch.py
@@ -0,0 +1,70 @@
+"""Integration tests for PipeBatch dispatch through Temporal workflows.
+
+Validates that PipeBatch controllers execute on Temporal workers via LibraryCrate propagation.
+Each batch item is dispatched as a separate child workflow through TemporalPipeRouter (fan-out).
+"""
+
+import uuid
+from datetime import timedelta
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.test_data import PipeBatchTemporalTestData
+
+
[email protected](loop_scope="class")
+class TestWfPipeBatch:
+ @staticmethod
+ def _assert_crate_structure(pipe_job: PipeJob) -> None:
+ crate = pipe_job.library_crate
+ assert crate is not None, "library_crate should be set on PipeJob"
+ assert crate.fingerprint, "fingerprint should be non-empty"
+
+ assert len(crate.pipes) == len(PipeBatchTemporalTestData.EXPECTED_PIPE_REFS), (
+ f"Expected {len(PipeBatchTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"
+ )
+ for pipe_ref in PipeBatchTemporalTestData.EXPECTED_PIPE_REFS:
+ assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"
+
+ async def test_crate_contains_batch_pipes(self, batch_job: PipeJob):
+ """Verify the crate has all pipe_refs including the batch branch pipe."""
+ self._assert_crate_structure(batch_job)
+
+ async def test_batch_sequence_via_temporal(
+ self,
+ batch_job: PipeJob,
+ temporal_client: TemporalClient,
+ ):
+ """PipeBatch dispatches one child workflow per list item (fan-out) on the worker."""
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=batch_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ execution_timeout=timedelta(seconds=30),
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ rehydrate_pipe_output(pipe_output)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+
+ for stuff_name in PipeBatchTemporalTestData.EXPECTED_STUFF_NAMES:
+ assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"
+ stuff = working_memory.get_stuff(stuff_name)
+ assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_compose.py b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_compose.py
new file mode 100644
index 000000000..2a7dcf4bd
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_compose.py
@@ -0,0 +1,82 @@
+"""Integration tests for PipeCompose operator through Temporal workflows.
+
+Validates that PipeCompose operators execute on Temporal workers via LibraryCrate propagation.
+Also exercises Phase 3 deferred hydration — the Report concept has an inline structure that
+requires dynamic class registration on the worker before PipeCompose can construct it.
+"""
+
+import uuid
+from datetime import timedelta
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.test_data import PipeComposeTemporalTestData
+
+
[email protected](loop_scope="class")
+class TestWfPipeCompose:
+ @staticmethod
+ def _assert_crate_structure(pipe_job: PipeJob) -> None:
+ crate = pipe_job.library_crate
+ assert crate is not None, "library_crate should be set on PipeJob"
+ assert crate.fingerprint, "fingerprint should be non-empty"
+
+ assert len(crate.pipes) == len(PipeComposeTemporalTestData.EXPECTED_PIPE_REFS), (
+ f"Expected {len(PipeComposeTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"
+ )
+ for pipe_ref in PipeComposeTemporalTestData.EXPECTED_PIPE_REFS:
+ assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"
+
+ async def test_crate_contains_compose_pipes(self, compose_job: PipeJob):
+ """Verify the crate has all pipe_refs and the Report concept."""
+ self._assert_crate_structure(compose_job)
+ crate = compose_job.library_crate
+ assert crate is not None
+ report_ref = f"{PipeComposeTemporalTestData.DOMAIN}.Report"
+ assert report_ref in crate.concepts, f"Expected concept_ref '{report_ref}' not found in crate"
+
+ async def test_compose_sequence_via_temporal(
+ self,
+ compose_job: PipeJob,
+ temporal_client: TemporalClient,
+ ):
+ """PipeCompose constructs a structured Report on the worker after deferred hydration."""
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=compose_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ execution_timeout=timedelta(seconds=30),
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ rehydrate_pipe_output(pipe_output)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+
+ for stuff_name in PipeComposeTemporalTestData.EXPECTED_STUFF_NAMES:
+ assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"
+
+ # Verify the Report has the expected structured fields (deferred hydration worked)
+ report_stuff = working_memory.get_stuff("final_report")
+ assert isinstance(report_stuff.content, StructuredContent), (
+ f"Expected StructuredContent for final_report, got {type(report_stuff.content).__name__}"
+ )
+ for field_name in PipeComposeTemporalTestData.EXPECTED_REPORT_FIELDS:
+ assert hasattr(report_stuff.content, field_name), f"Report missing field '{field_name}' — deferred hydration may have failed"
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_condition.py b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_condition.py
new file mode 100644
index 000000000..5def39404
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_condition.py
@@ -0,0 +1,71 @@
+"""Integration tests for PipeCondition dispatch through Temporal workflows.
+
+Validates that PipeCondition controllers execute on Temporal workers via LibraryCrate propagation.
+The condition routes to different outcome pipes based on an expression — each outcome is a
+child workflow dispatched through TemporalPipeRouter.
+"""
+
+import uuid
+from datetime import timedelta
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.test_data import PipeConditionTemporalTestData
+
+
[email protected](loop_scope="class")
+class TestWfPipeCondition:
+ @staticmethod
+ def _assert_crate_structure(pipe_job: PipeJob) -> None:
+ crate = pipe_job.library_crate
+ assert crate is not None, "library_crate should be set on PipeJob"
+ assert crate.fingerprint, "fingerprint should be non-empty"
+
+ assert len(crate.pipes) == len(PipeConditionTemporalTestData.EXPECTED_PIPE_REFS), (
+ f"Expected {len(PipeConditionTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"
+ )
+ for pipe_ref in PipeConditionTemporalTestData.EXPECTED_PIPE_REFS:
+ assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"
+
+ async def test_crate_contains_condition_pipes(self, condition_job: PipeJob):
+ """Verify the crate has all pipe_refs including condition outcomes."""
+ self._assert_crate_structure(condition_job)
+
+ async def test_condition_sequence_via_temporal(
+ self,
+ condition_job: PipeJob,
+ temporal_client: TemporalClient,
+ ):
+ """PipeCondition dispatches the selected outcome as a child workflow on the worker."""
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=condition_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ execution_timeout=timedelta(seconds=30),
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ rehydrate_pipe_output(pipe_output)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+
+ for stuff_name in PipeConditionTemporalTestData.EXPECTED_STUFF_NAMES:
+ assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"
+ stuff = working_memory.get_stuff(stuff_name)
+ assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"
diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_parallel.py b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_parallel.py
new file mode 100644
index 000000000..b078c0dd6
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_parallel.py
@@ -0,0 +1,68 @@
+"""Integration tests for PipeParallel dispatch through Temporal workflows.
+
+Validates that PipeParallel controllers execute on Temporal workers via LibraryCrate propagation.
+Each parallel branch is dispatched as a concurrent child workflow through TemporalPipeRouter.
+"""
+
+import uuid
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.test_data import PipeParallelTemporalTestData
+
+
[email protected](loop_scope="class")
+class TestWfPipeParallel:
+ @staticmethod
+ def _assert_crate_structure(pipe_job: PipeJob) -> None:
+ crate = pipe_job.library_crate
+ assert crate is not None, "library_crate should be set on PipeJob"
+ assert crate.fingerprint, "fingerprint should be non-empty"
+
+ assert len(crate.pipes) == len(PipeParallelTemporalTestData.EXPECTED_PIPE_REFS), (
+ f"Expected {len(PipeParallelTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"
+ )
+ for pipe_ref in PipeParallelTemporalTestData.EXPECTED_PIPE_REFS:
+ assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"
+
+ async def test_crate_contains_parallel_pipes(self, parallel_job: PipeJob):
+ """Verify the crate has all pipe_refs including parallel branches."""
+ self._assert_crate_structure(parallel_job)
+
+ async def test_parallel_sequence_via_temporal(
+ self,
+ parallel_job: PipeJob,
+ temporal_client: TemporalClient,
+ ):
+ """PipeParallel dispatches branches as concurrent child workflows on the worker."""
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=parallel_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ )
+
+ assert isinstance(pipe_output, PipeOutput)
+ rehydrate_pipe_output(pipe_output)
+ working_memory = pipe_output.working_memory
+ assert working_memory is not None
+
+ for stuff_name in PipeParallelTemporalTestData.EXPECTED_STUFF_NAMES:
+ assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"
+ stuff = working_memory.get_stuff(stuff_name)
+ assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"
diff --git a/tests/integration/pipelex/temporal/test_data.py b/tests/integration/pipelex/temporal/test_data.py
new file mode 100644
index 000000000..84cff61ec
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_data.py
@@ -0,0 +1,468 @@
+from typing import ClassVar
+
+from pipelex.core.concepts.concept_factory import ConceptFactory
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.stuffs.image_content import ImageContent
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.core.stuffs.text_content import TextContent
+
+
+class PipeTestCases:
+ SYSTEM_PROMPT = "You are a pirate, you always talk like a pirate."
+ USER_PROMPT = "In 3 sentences, tell me about the sea."
+ USER_TEXT_TRICKY_1 = """
+ When my son was 7 he was 3ft tall. When he was 8 he was 4ft tall. When he was 9 he was 5ft tall.
+ How tall do you think he was when he was 12? and at 15?
+ """
+
+ # Create simple Stuff objects
+ SIMPLE_STUFF = StuffFactory.make_stuff(
+ name="text",
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),
+ content=TextContent(text="Describe a t-shirt in 2 sentences"),
+ )
+
+ IMG_EXPENSE_REPORT_1 = "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/invoices/invoice_1.png"
+ IMG_FASHION_PHOTO_1 = "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/fashion/fashion_photo_1.jpg"
+ SIMPLE_STUFF_TEXT = StuffFactory.make_stuff(
+ name="text",
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),
+ content=TextContent(text="Describe a t-shirt in 2 sentences"),
+ )
+ SIMPLE_STUFF_IMAGE = StuffFactory.make_stuff(
+ name="image",
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),
+ content=ImageContent(url=IMG_FASHION_PHOTO_1),
+ )
+
+ BATCH_TEST: ClassVar[list[tuple[str, Stuff, str, str]]] = [ # pipe_code, stuff, input_list_stuff_name, input_item_stuff_name
+ (
+ "batch_test",
+ StuffFactory.make_stuff(
+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),
+ name="colors",
+ content=ListContent(
+ items=[
+ TextContent(text="blue"),
+ TextContent(text="red"),
+ TextContent(text="green"),
+ ]
+ ),
+ ),
+ "colors",
+ "color",
+ ),
+ ]
+ STUFF_AND_PIPE: ClassVar[list[tuple[str, Stuff, str]]] = [ # topic, stuff, pipe_code
+ # TODO: fix testing implict concept
+ # (
+ # "Process Simple Text",
+ # SIMPLE_STUFF,
+ # "test_implicit_concept",
+ # ),
+ # (
+ # "Process Simple Text",
+ # SIMPLE_STUFF_TEXT,
+ # "simple_llm_test_from_text",
+ # ),
+ (
+ "Process Simple Image",
+ SIMPLE_STUFF_IMAGE,
+ "simple_llm_test_from_image",
+ ),
+ ]
+
+ NO_INPUT: ClassVar[list[tuple[str, str]]] = [ # topic, pipe
+ (
+ "Test with no input",
+ "test_no_input",
+ ),
+ (
+ "Imagine nature products of different colors",
+ "imagine_nature_product_list",
+ ),
+ (
+ "Create characters",
+ "create_characters",
+ ),
+ ]
+
+
+class LibraryCrateTestData:
+ """Test constants for LibraryCrate integration tests.
+
+ Uses a PipeSequence bundle with only native Text concepts (no dynamic classes)
+ to avoid the Layer 1 Kajson deserialization issue (Phase 3). This still fully
+ tests Layer 2 (pipe resolution via get_required_pipe on the worker).
+ """
+
+ BUNDLE_DIR: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate"
+ BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds"
+ PIPE_CODE: ClassVar[str] = "native_text_sequence"
+ DOMAIN: ClassVar[str] = "native_text_test"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "native_text_test.native_text_sequence",
+ "native_text_test.step_one",
+ "native_text_test.step_two",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "step_one_result",
+ "step_two_result",
+ ]
+
+
+class DeferredHydrationTestData:
+ """Test constants for Phase 3 deferred hydration integration tests.
+
+ Uses a PipeSequence bundle with a custom concept (Greeting) that has an inline
+ structure — this triggers dynamic class generation and exercises the full
+ deferred hydration + scoped ClassRegistry path.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds"
+ PIPE_CODE: ClassVar[str] = "dynamic_greeting_sequence"
+ DOMAIN: ClassVar[str] = "dynamic_concept_test"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "dynamic_concept_test.dynamic_greeting_sequence",
+ "dynamic_concept_test.generate_greeting",
+ "dynamic_concept_test.summarize_greeting",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "greeting_result",
+ "summary_result",
+ ]
+
+
+_CRATE_DIR: str = "tests/integration/pipelex/temporal/library_crate"
+
+
+class ConflictConceptAlphaTestData:
+ """Concept 'Result' with score (integer) + label (text).
+
+ Paired with ConflictConceptBetaTestData which defines a different 'Result'.
+ Tests that per-workflow ClassRegistry scoping keeps them isolated.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/conflict_concept_alpha.mthds"
+ PIPE_CODE: ClassVar[str] = "alpha_pipeline"
+ DOMAIN: ClassVar[str] = "conflict_alpha"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "conflict_alpha.alpha_pipeline",
+ "conflict_alpha.alpha_generate",
+ "conflict_alpha.alpha_summarize",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "alpha_result",
+ "alpha_summary",
+ ]
+
+ EXPECTED_RESULT_FIELDS: ClassVar[list[str]] = ["score", "label"]
+
+
+class ConflictConceptBetaTestData:
+ """Concept 'Result' with value (text) + confidence (number) + is_valid (text).
+
+ Paired with ConflictConceptAlphaTestData which defines a different 'Result'.
+ Tests that per-workflow ClassRegistry scoping keeps them isolated.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/conflict_concept_beta.mthds"
+ PIPE_CODE: ClassVar[str] = "beta_pipeline"
+ DOMAIN: ClassVar[str] = "conflict_beta"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "conflict_beta.beta_pipeline",
+ "conflict_beta.beta_generate",
+ "conflict_beta.beta_summarize",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "beta_result",
+ "beta_summary",
+ ]
+
+ EXPECTED_RESULT_FIELDS: ClassVar[list[str]] = ["value", "confidence", "is_valid"]
+
+
+class ConflictPipeAlphaTestData:
+ """Pipe 'alpha_shared_step' as PipeLLM about colors.
+
+ Paired with ConflictPipeBetaTestData. Tests that per-workflow library scoping
+ via ContextVar resolves the correct pipe_ref for each workflow.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/conflict_pipe_alpha.mthds"
+ PIPE_CODE: ClassVar[str] = "pipe_alpha_pipeline"
+ DOMAIN: ClassVar[str] = "pipe_conflict_alpha"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "pipe_conflict_alpha.pipe_alpha_pipeline",
+ "pipe_conflict_alpha.alpha_shared_step",
+ "pipe_conflict_alpha.alpha_finalize",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "step_result",
+ "final_result",
+ ]
+
+
+class ConflictPipeBetaTestData:
+ """Pipe 'beta_shared_step' as PipeLLM about animals.
+
+ Paired with ConflictPipeAlphaTestData. Tests that per-workflow library scoping
+ via ContextVar resolves the correct pipe_ref for each workflow.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/conflict_pipe_beta.mthds"
+ PIPE_CODE: ClassVar[str] = "pipe_beta_pipeline"
+ DOMAIN: ClassVar[str] = "pipe_conflict_beta"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "pipe_conflict_beta.pipe_beta_pipeline",
+ "pipe_conflict_beta.beta_shared_step",
+ "pipe_conflict_beta.beta_finalize",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "step_result",
+ "final_result",
+ ]
+
+
+class MultiConceptAlphaTestData:
+ """Profile(name, age) + Summary(headline, body).
+
+ Paired with MultiConceptBetaTestData which defines different Profile and Summary.
+ Tests worst-case: multiple same-named dynamic classes across concurrent workflows.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/multi_concept_alpha.mthds"
+ PIPE_CODE: ClassVar[str] = "multi_alpha_pipeline"
+ DOMAIN: ClassVar[str] = "multi_alpha"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "multi_alpha.multi_alpha_pipeline",
+ "multi_alpha.multi_alpha_generate_profile",
+ "multi_alpha.multi_alpha_generate_summary",
+ "multi_alpha.multi_alpha_finalize",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "profile_result",
+ "summary_result",
+ "final_result",
+ ]
+
+ EXPECTED_PROFILE_FIELDS: ClassVar[list[str]] = ["name", "age"]
+ EXPECTED_SUMMARY_FIELDS: ClassVar[list[str]] = ["headline", "body"]
+
+
+class MultiConceptBetaTestData:
+ """Profile(title, department, level) + Summary(content).
+
+ Paired with MultiConceptAlphaTestData which defines different Profile and Summary.
+ Tests worst-case: multiple same-named dynamic classes across concurrent workflows.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/multi_concept_beta.mthds"
+ PIPE_CODE: ClassVar[str] = "multi_beta_pipeline"
+ DOMAIN: ClassVar[str] = "multi_beta"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "multi_beta.multi_beta_pipeline",
+ "multi_beta.multi_beta_generate_profile",
+ "multi_beta.multi_beta_generate_summary",
+ "multi_beta.multi_beta_finalize",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "profile_result",
+ "summary_result",
+ "final_result",
+ ]
+
+ EXPECTED_PROFILE_FIELDS: ClassVar[list[str]] = ["title", "department", "level"]
+ EXPECTED_SUMMARY_FIELDS: ClassVar[list[str]] = ["content"]
+
+
+class PipeConditionTemporalTestData:
+ """PipeCondition within a PipeSequence for Temporal child workflow dispatch testing."""
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_condition.mthds"
+ PIPE_CODE: ClassVar[str] = "temporal_condition_sequence"
+ DOMAIN: ClassVar[str] = "temporal_condition_test"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "temporal_condition_test.temporal_condition_sequence",
+ "temporal_condition_test.generate_category",
+ "temporal_condition_test.route_by_category",
+ "temporal_condition_test.handle_alpha",
+ "temporal_condition_test.handle_beta",
+ "temporal_condition_test.handle_default",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "category_text",
+ "routed_result",
+ ]
+
+
+class PipeParallelTemporalTestData:
+ """PipeParallel within a PipeSequence for Temporal concurrent child workflow dispatch testing."""
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_parallel.mthds"
+ PIPE_CODE: ClassVar[str] = "temporal_parallel_sequence"
+ DOMAIN: ClassVar[str] = "temporal_parallel_test"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "temporal_parallel_test.temporal_parallel_sequence",
+ "temporal_parallel_test.analyze_in_parallel",
+ "temporal_parallel_test.branch_tone",
+ "temporal_parallel_test.branch_length",
+ "temporal_parallel_test.summarize_results",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "tone_result",
+ "length_result",
+ "summary",
+ ]
+
+
+class PipeBatchTemporalTestData:
+ """PipeBatch within a PipeSequence for Temporal fan-out child workflow dispatch testing."""
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_batch.mthds"
+ PIPE_CODE: ClassVar[str] = "temporal_batch_sequence"
+ DOMAIN: ClassVar[str] = "temporal_batch_test"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "temporal_batch_test.temporal_batch_sequence",
+ "temporal_batch_test.temporal_generate_topics",
+ "temporal_batch_test.batch_temporal_describe_topics",
+ "temporal_batch_test.temporal_describe_topic",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "topics",
+ "notes",
+ ]
+
+
+class PipeComposeTemporalTestData:
+ """PipeCompose operator within a PipeSequence with deferred hydration of Report concept."""
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_compose.mthds"
+ PIPE_CODE: ClassVar[str] = "temporal_compose_sequence"
+ DOMAIN: ClassVar[str] = "temporal_compose_test"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "temporal_compose_test.temporal_compose_sequence",
+ "temporal_compose_test.generate_title",
+ "temporal_compose_test.generate_body",
+ "temporal_compose_test.compose_report",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "title_text",
+ "body_text",
+ "final_report",
+ ]
+
+ EXPECTED_REPORT_FIELDS: ClassVar[list[str]] = ["title", "body"]
+
+
+class CombinedPipelineTemporalTestData:
+ """Combined PipeParallel + PipeCondition in a PipeSequence for nested Temporal dispatch testing."""
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_combined.mthds"
+ PIPE_CODE: ClassVar[str] = "temporal_combined_pipeline"
+ DOMAIN: ClassVar[str] = "temporal_combined_test"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "temporal_combined_test.temporal_combined_pipeline",
+ "temporal_combined_test.parallel_generate",
+ "temporal_combined_test.generate_part_a",
+ "temporal_combined_test.generate_part_b",
+ "temporal_combined_test.quality_gate",
+ "temporal_combined_test.handle_failure",
+ "temporal_combined_test.produce_report",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "part_a",
+ "part_b",
+ "final_report",
+ ]
+
+ EXPECTED_REPORT_FIELDS: ClassVar[list[str]] = ["assessment", "confidence"]
+
+
+class PayloadCodecPipelineTestData:
+ """Test constants for StoragePayloadCodec integration tests with real pipe execution.
+
+ Reuses existing bundles to prove the codec is transparent — same pipelines
+ work unchanged when payloads are offloaded to storage.
+ """
+
+ # Reuse native_text_sequence for single/multi-step codec tests
+ NATIVE_BUNDLE_FILE: ClassVar[str] = LibraryCrateTestData.BUNDLE_FILE
+ NATIVE_PIPE_CODE: ClassVar[str] = LibraryCrateTestData.PIPE_CODE
+ NATIVE_EXPECTED_STUFF_NAMES: ClassVar[list[str]] = LibraryCrateTestData.EXPECTED_STUFF_NAMES
+
+ # Reuse dynamic_concept_sequence for dynamic concept + codec test
+ DYNAMIC_BUNDLE_FILE: ClassVar[str] = DeferredHydrationTestData.BUNDLE_FILE
+ DYNAMIC_PIPE_CODE: ClassVar[str] = DeferredHydrationTestData.PIPE_CODE
+ DYNAMIC_EXPECTED_STUFF_NAMES: ClassVar[list[str]] = DeferredHydrationTestData.EXPECTED_STUFF_NAMES
+
+ # Codec config
+ SIZE_THRESHOLD: ClassVar[int] = 1024
+ STORAGE_PREFIX: ClassVar[str] = "test-codec-pipeline/"
+
+
+class LargePayloadTestData:
+ """Test constants for large payload stress testing through Temporal with codec."""
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/large_payload_sequence.mthds"
+ PIPE_CODE: ClassVar[str] = "large_payload_sequence"
+ DOMAIN: ClassVar[str] = "large_payload_test"
+
+ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [
+ "large_payload_test.large_payload_sequence",
+ "large_payload_test.verbose_step_one",
+ "large_payload_test.verbose_step_two",
+ "large_payload_test.verbose_step_three",
+ ]
+
+ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [
+ "step_one_result",
+ "step_two_result",
+ "step_three_result",
+ ]
+
+ SIZE_THRESHOLD: ClassVar[int] = 1024
+ STORAGE_PREFIX: ClassVar[str] = "test-large-payload/"
+
+
+class PipeOcrTestCases:
+ PIPE_OCR_IMAGE_TEST_CASES: ClassVar[list[str]] = [
+ # LOCAL
+ "tests/data/documents/solar_system.png",
+ # REMOTE
+ "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/documents/solar_system.png",
+ ]
+ PIPE_OCR_PDF_TEST_CASES: ClassVar[list[str]] = [
+ # LOCAL
+ "tests/data/documents/solar_system.pdf",
+ # REMOTE
+ "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/documents/solar_system.pdf",
+ ]
diff --git a/tests/integration/pipelex/temporal/test_payload_codec_pipeline.py b/tests/integration/pipelex/temporal/test_payload_codec_pipeline.py
new file mode 100644
index 000000000..a2dfa2ebd
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_payload_codec_pipeline.py
@@ -0,0 +1,231 @@
+"""Integration tests for StoragePayloadCodec with real Pipelex pipe execution.
+
+Validates that the codec is transparent to real pipe workflows — WorkingMemory
+survives encode→storage→decode across workflow/activity hops when payloads are
+offloaded to external storage. Tests single-step, multi-step, and dynamic concept
+pipelines through the full WfPipeRouter path.
+"""
+
+from __future__ import annotations
+
+import uuid
+from typing import TYPE_CHECKING, AsyncGenerator, ClassVar
+
+import pytest
+import pytest_asyncio
+from temporalio.testing import WorkflowEnvironment
+
+from pipelex.pipelex import Pipelex
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+from pipelex.temporal.codec.storage_payload_codec import StoragePayloadCodec
+from pipelex.temporal.tasks import Tasks
+from pipelex.temporal.temporal_data_converter import make_data_converter
+from pipelex.temporal.temporal_hub import get_task_manager, temporal_hub
+from pipelex.temporal.temporal_task_manager import TemporalTaskManager
+from pipelex.tools.storage.local_storage_provider import LocalStorageProvider
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.library_crate.helpers import assert_stuff_names, assert_text_stuff_names, execute_workflow
+from tests.integration.pipelex.temporal.test_data import (
+ LargePayloadTestData,
+ PayloadCodecPipelineTestData,
+)
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+ from pathlib import Path
+
+ from temporalio.client import Client as TemporalClient
+
+ from pipelex.pipe_run.pipe_job import PipeJob
+
+# ---------------------------------------------------------------------------
+# Override autouse fixtures from parent conftest — this test module manages
+# its own codec-enabled WorkflowEnvironment while still using the full
+# TemporalTaskManager infrastructure for real pipe execution.
+# ---------------------------------------------------------------------------
+
+
[email protected](scope="module", autouse=True)
+def reset_pipelex_config_fixture() -> Generator[None, None, None]:
+ """Initialize Pipelex for logging/config."""
+ Pipelex.make(
+ integration_mode=IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST,
+ )
+ yield
+ Pipelex.teardown_if_needed()
+
+
[email protected](scope="module", autouse=True)
+def boot_temporal(reset_pipelex_config_fixture: None) -> Generator[None, None, None]: # noqa: ARG001
+ """Boot the Temporal layer with pipe routers and content generator."""
+ manager = TemporalTaskManager()
+ temporal_hub.set_task_manager(manager)
+ manager.complement_catalog(
+ extra_catalog=Tasks.TASK_PACKS,
+ extra_workflows=[],
+ extra_activities=[],
+ )
+ manager.setup()
+
+ from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory # noqa: PLC0415
+ from pipelex.hub import get_pipelex_hub, get_storage_provider # noqa: PLC0415
+ from pipelex.temporal.tprl_content_generation.content_generator_child_factory import ContentGeneratorChildFactory # noqa: PLC0415
+ from pipelex.temporal.tprl_pipe.temporal_pipe_router import make_temporal_pipe_router # noqa: PLC0415
+
+ pipelex_hub = get_pipelex_hub()
+ pipelex_hub.set_pipe_router(make_temporal_pipe_router())
+
+ generated_content_factory = GeneratedContentFactory(storage_provider=get_storage_provider())
+ content_generator_child = ContentGeneratorChildFactory.make_content_generator_child(
+ generated_content_factory=generated_content_factory,
+ )
+ pipelex_hub.set_content_generator(content_generator_child)
+
+ yield
+ manager.teardown()
+ temporal_hub.reset()
+
+ from pipelex.hub import get_inference_manager, get_plugin_manager # noqa: PLC0415
+
+ get_inference_manager().teardown()
+ get_plugin_manager().plugin_sdk_registry.teardown()
+
+
+# ---------------------------------------------------------------------------
+# Codec-enabled WorkflowEnvironment
+# ---------------------------------------------------------------------------
+
+
+class CodecEnvData:
+ SIZE_THRESHOLD: ClassVar[int] = PayloadCodecPipelineTestData.SIZE_THRESHOLD
+ STORAGE_PREFIX: ClassVar[str] = PayloadCodecPipelineTestData.STORAGE_PREFIX
+
+
+@pytest_asyncio.fixture(scope="class") # pyright: ignore[reportUntypedFunctionDecorator, reportUnknownMemberType]
+async def codec_env(tmp_path_factory: pytest.TempPathFactory) -> AsyncGenerator[tuple[WorkflowEnvironment, Path], None]:
+ """Create a WorkflowEnvironment with StoragePayloadCodec enabled."""
+ storage_root: Path = tmp_path_factory.mktemp("payload-codec-pipeline")
+ storage_provider = LocalStorageProvider(root_path=storage_root)
+ codec = StoragePayloadCodec(
+ storage_provider=storage_provider,
+ size_threshold=CodecEnvData.SIZE_THRESHOLD,
+ storage_prefix=CodecEnvData.STORAGE_PREFIX,
+ )
+ converter = make_data_converter(payload_codec=codec)
+ async with await WorkflowEnvironment.start_local(data_converter=converter) as env: # pyright: ignore[reportUnknownMemberType]
+ yield env, storage_root
+
+
+# ---------------------------------------------------------------------------
+# Pipe job fixtures
+# ---------------------------------------------------------------------------
+
+
[email protected](scope="class")
+def native_pipe_job() -> Generator[PipeJob, None, None]:
+ """PipeJob for the native text sequence (2-step, no dynamic concepts)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=PayloadCodecPipelineTestData.NATIVE_BUNDLE_FILE,
+ pipe_code=PayloadCodecPipelineTestData.NATIVE_PIPE_CODE,
+ )
+
+
[email protected](scope="class")
+def dynamic_pipe_job() -> Generator[PipeJob, None, None]:
+ """PipeJob for the dynamic concept sequence (Greeting concept created at runtime)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=PayloadCodecPipelineTestData.DYNAMIC_BUNDLE_FILE,
+ pipe_code=PayloadCodecPipelineTestData.DYNAMIC_PIPE_CODE,
+ )
+
+
[email protected](scope="class")
+def large_payload_pipe_job() -> Generator[PipeJob, None, None]:
+ """PipeJob for the large payload stress test (3-step verbose sequence)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=LargePayloadTestData.BUNDLE_FILE,
+ pipe_code=LargePayloadTestData.PIPE_CODE,
+ )
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _count_stored_files(storage_root: Path) -> int:
+ """Count files in the storage root (excluding directories)."""
+ return len([path for path in storage_root.rglob("*") if path.is_file()])
+
+
+# ---------------------------------------------------------------------------
+# Tests
+# ---------------------------------------------------------------------------
+
+
[email protected](loop_scope="class")
+class TestPayloadCodecPipeline:
+ """Real pipe execution through Temporal with StoragePayloadCodec enabled."""
+
+ async def test_native_sequence_with_codec(
+ self,
+ codec_env: tuple[WorkflowEnvironment, Path],
+ native_pipe_job: PipeJob,
+ ) -> None:
+ """A 2-step PipeSequence with native Text concepts round-trips through the codec."""
+ env, storage_root = codec_env
+ temporal_client: TemporalClient = env.client
+ task_queue = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output = await execute_workflow(native_pipe_job, temporal_client, task_queue)
+
+ assert_text_stuff_names(pipe_output, PayloadCodecPipelineTestData.NATIVE_EXPECTED_STUFF_NAMES)
+ assert _count_stored_files(storage_root) > 0, "Codec should have offloaded payloads to storage"
+
+ async def test_dynamic_concept_with_codec(
+ self,
+ codec_env: tuple[WorkflowEnvironment, Path],
+ dynamic_pipe_job: PipeJob,
+ ) -> None:
+ """Dynamic concept (Greeting) + codec: the most demanding combination."""
+ env, storage_root = codec_env
+ temporal_client: TemporalClient = env.client
+ task_queue = str(uuid.uuid4())
+ files_before = _count_stored_files(storage_root)
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output = await execute_workflow(dynamic_pipe_job, temporal_client, task_queue)
+
+ assert_stuff_names(pipe_output, PayloadCodecPipelineTestData.DYNAMIC_EXPECTED_STUFF_NAMES)
+ assert _count_stored_files(storage_root) > files_before, "Codec should have stored additional payloads for dynamic concept pipeline"
+
+ async def test_large_payload_multi_step_with_codec(
+ self,
+ codec_env: tuple[WorkflowEnvironment, Path],
+ large_payload_pipe_job: PipeJob,
+ ) -> None:
+ """3-step pipeline accumulating verbose output — codec stress test."""
+ env, storage_root = codec_env
+ temporal_client: TemporalClient = env.client
+ task_queue = str(uuid.uuid4())
+ files_before = _count_stored_files(storage_root)
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output = await execute_workflow(large_payload_pipe_job, temporal_client, task_queue)
+
+ assert_text_stuff_names(pipe_output, LargePayloadTestData.EXPECTED_STUFF_NAMES)
+ assert _count_stored_files(storage_root) > files_before, "Codec should have stored payloads for large multi-step pipeline"
diff --git a/tests/integration/pipelex/temporal/test_payload_codec_roundtrip.py b/tests/integration/pipelex/temporal/test_payload_codec_roundtrip.py
new file mode 100644
index 000000000..42885944f
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_payload_codec_roundtrip.py
@@ -0,0 +1,151 @@
+"""Integration test for StoragePayloadCodec round-trip through Temporal.
+
+Verifies that large payloads are transparently offloaded to external storage
+and reconstructed on the other side of a workflow → activity → return chain.
+"""
+
+from __future__ import annotations
+
+import uuid
+from datetime import timedelta
+from typing import TYPE_CHECKING, ClassVar
+
+import pytest
+from pydantic import BaseModel
+from temporalio import activity, workflow
+from temporalio.testing import WorkflowEnvironment
+from temporalio.worker import UnsandboxedWorkflowRunner, Worker
+
+from pipelex.pipelex import Pipelex
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+from pipelex.temporal.codec.storage_payload_codec import StoragePayloadCodec
+from pipelex.temporal.temporal_data_converter import make_data_converter
+from pipelex.tools.storage.local_storage_provider import LocalStorageProvider
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+ from pathlib import Path
+
+ from temporalio.client import Client as TemporalClient
+
+# ---------------------------------------------------------------------------
+# Override autouse fixtures from parent conftest — this test needs Pipelex
+# initialized (for logging/config) but not the full TemporalTaskManager.
+# ---------------------------------------------------------------------------
+
+
[email protected](scope="module", autouse=True)
+def reset_pipelex_config_fixture() -> Generator[None, None, None]:
+ """Initialize Pipelex for logging/config without TemporalTaskManager."""
+ Pipelex.make(
+ integration_mode=IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST,
+ )
+ yield
+ Pipelex.teardown_if_needed()
+
+
[email protected](scope="module", autouse=True)
+def boot_temporal(reset_pipelex_config_fixture: None) -> None:
+ """No-op override: this test manages its own Worker and client."""
+
+
+# ---------------------------------------------------------------------------
+# Test model, activity, and workflow
+# ---------------------------------------------------------------------------
+
+
+class LargePayloadModel(BaseModel):
+ """Simple model carrying a large string payload."""
+
+ label: str
+ data: str
+
+
[email protected](name="act_echo_large_payload")
+async def act_echo_large_payload(payload: LargePayloadModel) -> LargePayloadModel: # noqa: RUF029
+ """Activity that returns its input unchanged — exercises decode then encode."""
+ return payload
+
+
[email protected](name="wf_echo_large_payload")
+class WfEchoLargePayload:
+ """Workflow that forwards a large payload through an activity and returns it."""
+
+ @workflow.run
+ async def run(self, payload: LargePayloadModel) -> LargePayloadModel:
+ return await workflow.execute_activity(
+ act_echo_large_payload,
+ payload,
+ start_to_close_timeout=timedelta(seconds=30),
+ )
+
+
+# ---------------------------------------------------------------------------
+# Test data
+# ---------------------------------------------------------------------------
+
+
+class TestPayloadCodecRoundTripTestData:
+ # 1 KB threshold — low enough that our test payload triggers offloading
+ SIZE_THRESHOLD: ClassVar[int] = 1024
+ STORAGE_PREFIX: ClassVar[str] = "test-payloads/"
+ # Generate a payload well above the threshold
+ LARGE_DATA: ClassVar[str] = "X" * 10_000
+ LABEL: ClassVar[str] = "roundtrip-test"
+
+
+# ---------------------------------------------------------------------------
+# Tests
+# ---------------------------------------------------------------------------
+
+
[email protected](loop_scope="class")
+class TestPayloadCodecRoundTrip:
+ """End-to-end test: client encode → storage → worker decode → activity → worker encode → storage → client decode."""
+
+ async def test_large_payload_survives_temporal_roundtrip(self, tmp_path: Path) -> None:
+ """A payload larger than the codec threshold round-trips through Temporal identically."""
+ # Arrange: build a codec-enabled environment
+ storage_root = tmp_path / "payload-store"
+ storage_root.mkdir()
+ storage_provider = LocalStorageProvider(root_path=storage_root)
+ codec = StoragePayloadCodec(
+ storage_provider=storage_provider,
+ size_threshold=TestPayloadCodecRoundTripTestData.SIZE_THRESHOLD,
+ storage_prefix=TestPayloadCodecRoundTripTestData.STORAGE_PREFIX,
+ )
+ converter = make_data_converter(payload_codec=codec)
+
+ input_payload = LargePayloadModel(
+ label=TestPayloadCodecRoundTripTestData.LABEL,
+ data=TestPayloadCodecRoundTripTestData.LARGE_DATA,
+ )
+
+ task_queue = str(uuid.uuid4())
+
+ async with await WorkflowEnvironment.start_local(data_converter=converter) as env: # pyright: ignore[reportUnknownMemberType]
+ temporal_client: TemporalClient = env.client
+ async with Worker(
+ temporal_client,
+ task_queue=task_queue,
+ workflows=[WfEchoLargePayload],
+ activities=[act_echo_large_payload],
+ workflow_runner=UnsandboxedWorkflowRunner(),
+ ):
+ result: LargePayloadModel = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ WfEchoLargePayload.run,
+ input_payload,
+ id="wf-payload-codec-roundtrip",
+ task_queue=task_queue,
+ )
+
+ # Assert: output matches input exactly
+ assert isinstance(result, LargePayloadModel)
+ assert result.label == input_payload.label
+ assert result.data == input_payload.data
+ assert result == input_payload
+
+ # Assert: storage was actually used (files were written)
+ stored_files = list(storage_root.rglob("*"))
+ stored_files = [file_path for file_path in stored_files if file_path.is_file()]
+ assert len(stored_files) > 0, "Codec should have offloaded payloads to storage"
diff --git a/tests/integration/pipelex/temporal/test_tasks.py b/tests/integration/pipelex/temporal/test_tasks.py
new file mode 100644
index 000000000..f4656dd8a
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_tasks.py
@@ -0,0 +1,30 @@
+import pytest
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.temporal.temporal_hub import get_task_manager
+
+
+class TestTasks:
+ def test_default_scope_is_known(self):
+ worker_scopes = get_config().temporal.worker_scopes
+ assert worker_scopes.default_scope in worker_scopes.scopes
+
+ def test_full_scope_packs_present(self):
+ packs = get_task_manager().task_packs()
+ log.verbose(packs, title="Task packs")
+ full_scope = get_config().temporal.worker_scopes.scopes["full"]
+ for required_task_pack in full_scope.required_tasks_packs:
+ assert required_task_pack in packs, f"Required task pack '{required_task_pack}' not found in packs"
+
+ def test_router_and_runner_names_resolve(self):
+ workflows, activities = get_task_manager().workflows_and_activities_str()
+ log.verbose(workflows, title="Workflows")
+ log.verbose(activities, title="Activities")
+ for scope_name in ("router", "runner"):
+ scope = get_config().temporal.worker_scopes.scopes[scope_name]
+ for required_workflow in scope.required_workflows:
+ assert required_workflow in workflows, f"Required workflow '{required_workflow}' not found in workflows"
+ for required_activity in scope.required_activities:
+ assert required_activity in activities, f"Required activity '{required_activity}' not found in activities"
diff --git a/tests/integration/pipelex/temporal/test_utils.py b/tests/integration/pipelex/temporal/test_utils.py
new file mode 100644
index 000000000..6782101e4
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_utils.py
@@ -0,0 +1,3 @@
+from rich import print as rprint
+
+__all__ = ["rprint"]
diff --git a/tests/integration/pipelex/temporal/tracing/conftest.py b/tests/integration/pipelex/temporal/tracing/conftest.py
new file mode 100644
index 000000000..a3dd60339
--- /dev/null
+++ b/tests/integration/pipelex/temporal/tracing/conftest.py
@@ -0,0 +1,100 @@
+"""Fixtures for Temporal graph tracing integration tests.
+
+Provides tracing-enabled PipeJob fixtures and a config fixture
+that enables event tracing for the test class.
+"""
+
+from collections.abc import Generator
+from pathlib import Path
+
+import pytest
+
+from pipelex.config import get_config
+from pipelex.graph.graph_tracer_manager import GraphTracerManager
+from pipelex.pipe_run.pipe_job import PipeJob
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.tracing.test_data import (
+ BatchTracingTestData,
+ ParallelTracingTestData,
+ SequenceTracingTestData,
+)
+
+_TRACING_DIR = Path(__file__).parent.resolve()
+
+
+def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:
+ """Mark every test under this directory with gha_disabled.
+
+ The hook receives all session-wide items (not just items below this
+ conftest), so filter by file path before adding the marker.
+
+ TODO: Tests under tests/integration/pipelex/temporal/tracing/ hang
+ reliably in CI under pytest-xdist parallelism (worker timeouts at
+ 180s on py3.11+). Pass locally and serially. Root cause is concurrent
+ PipeBatch/PipeParallel/PipeSequence + WorkflowEnvironment.start_local
+ contention under load. Re-enable once the underlying flake is fixed.
+ """
+ skip_marker = pytest.mark.gha_disabled
+ for item in items:
+ item_path = Path(str(item.path)).resolve()
+ if _TRACING_DIR in item_path.parents:
+ item.add_marker(skip_marker)
+
+
[email protected](scope="class")
+def tracing_tmp_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
+ """Per-class temp directory for NDJSON trace files."""
+ return tmp_path_factory.mktemp("traces")
+
+
[email protected](scope="class", autouse=True)
+def enable_tracing(tracing_tmp_dir: Path) -> Generator[None, None, None]:
+ """Enable tracing config and point traces_dir to the temp directory.
+
+ Restores original config values and cleans up GraphTracerManager on teardown.
+ """
+ tracing_config = get_config().pipelex.tracing_config
+ original_enabled = tracing_config.is_enabled
+ ndjson_config = tracing_config.ndjson
+ original_dir = ndjson_config.traces_dir if ndjson_config else ""
+
+ tracing_config.is_enabled = True
+ if ndjson_config:
+ ndjson_config.traces_dir = str(tracing_tmp_dir)
+
+ yield
+
+ tracing_config.is_enabled = original_enabled
+ if ndjson_config:
+ ndjson_config.traces_dir = original_dir
+ GraphTracerManager.clear_instance()
+
+
[email protected](scope="class")
+def sequence_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob for native_text_sequence (graph_context injected per-execution by execute_and_assemble)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=SequenceTracingTestData.BUNDLE_FILE,
+ pipe_code=SequenceTracingTestData.PIPE_CODE,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](scope="class")
+def parallel_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob for temporal_parallel (graph_context injected per-execution by execute_and_assemble)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=ParallelTracingTestData.BUNDLE_FILE,
+ pipe_code=ParallelTracingTestData.PIPE_CODE,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+
[email protected](scope="class")
+def batch_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """PipeJob for temporal_batch (graph_context injected per-execution by execute_and_assemble)."""
+ yield from pipe_job_from_bundle(
+ bundle_file=BatchTracingTestData.BUNDLE_FILE,
+ pipe_code=BatchTracingTestData.PIPE_CODE,
+ isolated_registry=is_class_registry_isolated,
+ )
diff --git a/tests/integration/pipelex/temporal/tracing/helpers.py b/tests/integration/pipelex/temporal/tracing/helpers.py
new file mode 100644
index 000000000..b85bb33ec
--- /dev/null
+++ b/tests/integration/pipelex/temporal/tracing/helpers.py
@@ -0,0 +1,135 @@
+"""Shared test helpers for Temporal graph tracing integration tests."""
+
+import uuid
+from collections import defaultdict
+from pathlib import Path
+
+from temporalio.client import Client as TemporalClient
+
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.graph.graph_config import DataInclusionConfig
+from pipelex.graph.graph_context import GraphContext
+from pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, NodeStatus, PipelineRef
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+
+
+def inject_graph_context(pipe_job: PipeJob, pipeline_run_id: str) -> PipeJob:
+ """Deep-copy a PipeJob and inject a GraphContext onto its JobMetadata.
+
+ Also overrides pipeline_run_id (must not be the dry-run sentinel,
+ since NdjsonEventLog uses it as a directory name).
+ """
+ graph_context = GraphContext(
+ graph_id=pipeline_run_id,
+ parent_node_id=None,
+ node_sequence=0,
+ data_inclusion=DataInclusionConfig(
+ pipe_and_concept_registry=True,
+ stuff_json_content=False,
+ stuff_text_content=False,
+ stuff_html_content=False,
+ error_stack_traces=False,
+ ),
+ )
+ new_metadata = pipe_job.job_metadata.model_copy(
+ update={
+ "graph_context": graph_context,
+ "pipeline_run_id": pipeline_run_id,
+ },
+ )
+ return pipe_job.model_copy(update={"job_metadata": new_metadata})
+
+
+class TracingResult:
+ """Result of a traced workflow execution."""
+
+ def __init__(self, pipe_output: PipeOutput, graph_spec: GraphSpec | None, pipeline_run_id: str) -> None:
+ self.pipe_output = pipe_output
+ self.graph_spec = graph_spec
+ self.pipeline_run_id = pipeline_run_id
+
+
+async def execute_and_assemble(
+ pipe_job: PipeJob,
+ temporal_client: TemporalClient,
+ traces_dir: str,
+) -> TracingResult:
+ """Execute a workflow on Temporal and assemble a GraphSpec from emitted events.
+
+ Each call uses a unique pipeline_run_id to isolate NDJSON files across calls.
+
+ Returns:
+ TracingResult with pipe_output, graph_spec, and pipeline_run_id.
+ """
+ # Give each execution its own pipeline_run_id to avoid event accumulation
+ execution_run_id = f"tracing_exec_{uuid.uuid4().hex[:12]}"
+ execution_job = inject_graph_context(pipe_job, execution_run_id)
+
+ task_queue = str(uuid.uuid4())
+ workflow_id = str(uuid.uuid4())
+
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=execution_job,
+ id=workflow_id,
+ task_queue=task_queue,
+ )
+
+ rehydrate_pipe_output(pipe_output)
+
+ # Read events and assemble GraphSpec
+ event_log = NdjsonEventLog(traces_dir=traces_dir)
+ try:
+ events = event_log.read_events(execution_run_id)
+ finally:
+ event_log.close()
+
+ if not events:
+ return TracingResult(pipe_output=pipe_output, graph_spec=None, pipeline_run_id=execution_run_id)
+
+ domain = pipe_job.pipe.domain_code
+ main_pipe = pipe_job.pipe.code
+ graph_spec = GraphSpecAssembler.assemble(
+ events=events,
+ graph_id=execution_run_id,
+ pipeline_ref=PipelineRef(domain=domain, main_pipe=main_pipe),
+ )
+ return TracingResult(pipe_output=pipe_output, graph_spec=graph_spec, pipeline_run_id=execution_run_id)
+
+
+def assert_all_nodes_terminal(graph_spec: GraphSpec) -> None:
+ """Assert that no nodes are left in RUNNING or SCHEDULED state."""
+ terminal_statuses = {NodeStatus.SUCCEEDED, NodeStatus.FAILED, NodeStatus.CANCELED, NodeStatus.SKIPPED}
+ for node in graph_spec.nodes:
+ assert node.status in terminal_statuses, f"Node '{node.node_id}' (pipe_code={node.pipe_code}) has non-terminal status: {node.status}"
+
+
+def edges_by_kind(graph_spec: GraphSpec) -> dict[EdgeKind, list[EdgeSpec]]:
+ """Group edges by their EdgeKind for targeted assertions."""
+ result: dict[EdgeKind, list[EdgeSpec]] = defaultdict(list)
+ for edge in graph_spec.edges:
+ result[edge.kind].append(edge)
+ return dict(result)
+
+
+def ndjson_files_for_run(traces_dir: str, pipeline_run_id: str) -> list[Path]:
+ """List all NDJSON files generated for a pipeline run."""
+ run_dir = Path(traces_dir) / pipeline_run_id
+ if not run_dir.exists():
+ return []
+ return sorted(run_dir.glob("*.ndjson"))
+
+
+def node_pipe_codes(graph_spec: GraphSpec) -> set[str | None]:
+ """Extract the set of pipe_codes from all nodes in a GraphSpec."""
+ return {node.pipe_code for node in graph_spec.nodes}
diff --git a/tests/integration/pipelex/temporal/tracing/test_data.py b/tests/integration/pipelex/temporal/tracing/test_data.py
new file mode 100644
index 000000000..ad89a606b
--- /dev/null
+++ b/tests/integration/pipelex/temporal/tracing/test_data.py
@@ -0,0 +1,66 @@
+"""Test data constants for Temporal graph tracing integration tests.
+
+Expected graph structures for each bundle used in tracing tests.
+"""
+
+from typing import ClassVar
+
+_CRATE_DIR = "tests/integration/pipelex/temporal/library_crate"
+
+
+class SequenceTracingTestData:
+ """Expected graph structure for native_text_sequence.mthds.
+
+ Bundle: PipeSequence with 2 PipeLLM steps (step_one → step_two).
+ Nodes: sequence controller + step_one + step_two.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/native_text_sequence.mthds"
+ PIPE_CODE: ClassVar[str] = "native_text_sequence"
+ DOMAIN: ClassVar[str] = "native_text_test"
+
+ EXPECTED_PIPE_CODES: ClassVar[set[str | None]] = {
+ "native_text_sequence",
+ "step_one",
+ "step_two",
+ }
+ EXPECTED_NODE_COUNT: ClassVar[int] = 3
+ EXPECTED_CONTAINS_EDGE_COUNT: ClassVar[int] = 2
+ MIN_DATA_EDGES: ClassVar[int] = 1
+
+
+class ParallelTracingTestData:
+ """Expected graph structure for temporal_parallel.mthds.
+
+ Bundle: PipeSequence wrapping PipeParallel (2 branches) + summarize step.
+ Nodes: outer sequence + analyze_in_parallel + branch_tone + branch_length + summarize_results.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_parallel.mthds"
+ PIPE_CODE: ClassVar[str] = "temporal_parallel_sequence"
+ DOMAIN: ClassVar[str] = "temporal_parallel_test"
+
+ EXPECTED_PIPE_CODES: ClassVar[set[str | None]] = {
+ "temporal_parallel_sequence",
+ "analyze_in_parallel",
+ "branch_tone",
+ "branch_length",
+ "summarize_results",
+ }
+ MIN_NODE_COUNT: ClassVar[int] = 5
+
+
+class BatchTracingTestData:
+ """Expected graph structure for temporal_batch.mthds.
+
+ Bundle: PipeSequence wrapping PipeBatch (fan-out to per-item child workflows).
+ Nodes: outer sequence + generate_topics + batch controller + per-item processors.
+ """
+
+ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_batch.mthds"
+ PIPE_CODE: ClassVar[str] = "temporal_batch_sequence"
+ DOMAIN: ClassVar[str] = "temporal_batch_test"
+
+ MIN_NODE_COUNT: ClassVar[int] = 4
+ MIN_BATCH_ITEM_EDGES: ClassVar[int] = 1
+ MIN_BATCH_AGGREGATE_EDGES: ClassVar[int] = 1
diff --git a/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_batch.py b/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_batch.py
new file mode 100644
index 000000000..c5d2dc0aa
--- /dev/null
+++ b/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_batch.py
@@ -0,0 +1,91 @@
+"""Integration tests for PipeBatch graph tracing through Temporal workflows.
+
+Validates that PipeBatch controllers running as fan-out child workflows
+produce BATCH_ITEM and BATCH_AGGREGATE edges in the assembled GraphSpec.
+"""
+
+from pathlib import Path
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.graph.graphspec import EdgeKind, GraphSpec, NodeStatus
+from pipelex.pipe_run.pipe_job import PipeJob
+from tests.integration.pipelex.temporal.tracing.helpers import (
+ TracingResult,
+ assert_all_nodes_terminal,
+ edges_by_kind,
+ execute_and_assemble,
+)
+from tests.integration.pipelex.temporal.tracing.test_data import BatchTracingTestData
+
+
[email protected](loop_scope="class")
+class TestWfGraphTracingBatch:
+ """PipeBatch produces BATCH_ITEM and BATCH_AGGREGATE edges in GraphSpec via Temporal tracing."""
+
+ async def _get_result(
+ self,
+ batch_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ) -> tuple[TracingResult, GraphSpec]:
+ """Execute workflow and return result with assembled GraphSpec, asserting it's not None."""
+ result = await execute_and_assemble(
+ pipe_job=batch_tracing_job,
+ temporal_client=temporal_client,
+ traces_dir=str(tracing_tmp_dir),
+ )
+ assert result.graph_spec is not None, "GraphSpec should be assembled from trace events"
+ return result, result.graph_spec
+
+ async def test_batch_all_nodes_succeeded(
+ self,
+ batch_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """All nodes in the batch graph are SUCCEEDED."""
+ _result, graph_spec = await self._get_result(batch_tracing_job, temporal_client, tracing_tmp_dir)
+ for node in graph_spec.nodes:
+ assert node.status == NodeStatus.SUCCEEDED, f"Node '{node.pipe_code}' should be SUCCEEDED, got {node.status}"
+ assert_all_nodes_terminal(graph_spec)
+
+ async def test_batch_node_count(
+ self,
+ batch_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """Graph has at least the minimum expected number of nodes."""
+ _result, graph_spec = await self._get_result(batch_tracing_job, temporal_client, tracing_tmp_dir)
+ assert len(graph_spec.nodes) >= BatchTracingTestData.MIN_NODE_COUNT
+
+ async def test_batch_item_edges(
+ self,
+ batch_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """BATCH_ITEM edges present for fan-out from list to item processors."""
+ _result, graph_spec = await self._get_result(batch_tracing_job, temporal_client, tracing_tmp_dir)
+ edge_groups = edges_by_kind(graph_spec)
+ item_edges = edge_groups.get(EdgeKind.BATCH_ITEM, [])
+ assert len(item_edges) >= BatchTracingTestData.MIN_BATCH_ITEM_EDGES, (
+ f"Expected at least {BatchTracingTestData.MIN_BATCH_ITEM_EDGES} BATCH_ITEM edges, got {len(item_edges)}"
+ )
+
+ async def test_batch_aggregate_edges(
+ self,
+ batch_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """BATCH_AGGREGATE edges present for fan-in from item processors back to controller."""
+ _result, graph_spec = await self._get_result(batch_tracing_job, temporal_client, tracing_tmp_dir)
+ edge_groups = edges_by_kind(graph_spec)
+ aggregate_edges = edge_groups.get(EdgeKind.BATCH_AGGREGATE, [])
+ assert len(aggregate_edges) >= BatchTracingTestData.MIN_BATCH_AGGREGATE_EDGES, (
+ f"Expected at least {BatchTracingTestData.MIN_BATCH_AGGREGATE_EDGES} BATCH_AGGREGATE edges, got {len(aggregate_edges)}"
+ )
diff --git a/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_parallel.py b/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_parallel.py
new file mode 100644
index 000000000..0f2a770a8
--- /dev/null
+++ b/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_parallel.py
@@ -0,0 +1,118 @@
+"""Integration tests for PipeParallel graph tracing through Temporal workflows.
+
+Validates that PipeParallel controllers running as concurrent child workflows
+produce correct graph structure in the assembled GraphSpec.
+"""
+
+from pathlib import Path
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.graph.graphspec import EdgeKind, GraphSpec, NodeStatus
+from pipelex.pipe_run.pipe_job import PipeJob
+from tests.integration.pipelex.temporal.tracing.helpers import (
+ TracingResult,
+ assert_all_nodes_terminal,
+ edges_by_kind,
+ execute_and_assemble,
+ ndjson_files_for_run,
+ node_pipe_codes,
+)
+from tests.integration.pipelex.temporal.tracing.test_data import ParallelTracingTestData
+
+
[email protected](loop_scope="class")
+class TestWfGraphTracingParallel:
+ """PipeParallel produces correct graph structure via Temporal tracing."""
+
+ async def _get_result(
+ self,
+ parallel_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ) -> tuple[TracingResult, GraphSpec]:
+ """Execute workflow and return result with assembled GraphSpec, asserting it's not None."""
+ result = await execute_and_assemble(
+ pipe_job=parallel_tracing_job,
+ temporal_client=temporal_client,
+ traces_dir=str(tracing_tmp_dir),
+ )
+ assert result.graph_spec is not None, "GraphSpec should be assembled from trace events"
+ return result, result.graph_spec
+
+ async def test_parallel_produces_multiple_ndjson_files(
+ self,
+ parallel_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """Parallel branches spawn child workflows, producing multiple NDJSON files."""
+ result = await execute_and_assemble(
+ pipe_job=parallel_tracing_job,
+ temporal_client=temporal_client,
+ traces_dir=str(tracing_tmp_dir),
+ )
+ ndjson_files = ndjson_files_for_run(str(tracing_tmp_dir), result.pipeline_run_id)
+ # Parent workflow + child workflows (at least 2 for the 2 branches)
+ assert len(ndjson_files) >= 2, f"Expected multiple NDJSON files (parent + child workflows), got {len(ndjson_files)}"
+
+ async def test_parallel_all_nodes_succeeded(
+ self,
+ parallel_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """All nodes in the parallel graph are SUCCEEDED."""
+ _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)
+ for node in graph_spec.nodes:
+ assert node.status == NodeStatus.SUCCEEDED, f"Node '{node.pipe_code}' should be SUCCEEDED, got {node.status}"
+ assert_all_nodes_terminal(graph_spec)
+
+ async def test_parallel_pipe_codes(
+ self,
+ parallel_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """All expected pipe_codes are present in the assembled graph."""
+ _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)
+ actual_codes = node_pipe_codes(graph_spec)
+ assert ParallelTracingTestData.EXPECTED_PIPE_CODES.issubset(actual_codes), (
+ f"Missing pipe_codes: {ParallelTracingTestData.EXPECTED_PIPE_CODES - actual_codes}"
+ )
+
+ async def test_parallel_node_count(
+ self,
+ parallel_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """Graph has at least the minimum expected number of nodes."""
+ _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)
+ assert len(graph_spec.nodes) >= ParallelTracingTestData.MIN_NODE_COUNT
+
+ async def test_parallel_contains_edges(
+ self,
+ parallel_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """CONTAINS edges from controllers to their children are present."""
+ _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)
+ edge_groups = edges_by_kind(graph_spec)
+ contains_edges = edge_groups.get(EdgeKind.CONTAINS, [])
+ assert len(contains_edges) >= 2, "At least 2 CONTAINS edges expected (branches inside parallel)"
+
+ async def test_parallel_data_edges(
+ self,
+ parallel_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """DATA edges connect branch outputs to downstream consumers."""
+ _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)
+ edge_groups = edges_by_kind(graph_spec)
+ data_edges = edge_groups.get(EdgeKind.DATA, [])
+ assert len(data_edges) >= 1, "At least 1 DATA edge expected (branch output → summarize step)"
diff --git a/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_sequence.py b/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_sequence.py
new file mode 100644
index 000000000..7225b2ec0
--- /dev/null
+++ b/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_sequence.py
@@ -0,0 +1,117 @@
+"""Integration tests for PipeSequence graph tracing through Temporal workflows.
+
+Validates that a PipeSequence (2 PipeLLM steps) running on a Temporal worker
+produces correct NDJSON trace events that assemble into a valid GraphSpec
+with CONTAINS and DATA edges.
+"""
+
+from pathlib import Path
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.graph.graphspec import EdgeKind, GraphSpec, NodeStatus
+from pipelex.pipe_run.pipe_job import PipeJob
+from tests.integration.pipelex.temporal.tracing.helpers import (
+ TracingResult,
+ assert_all_nodes_terminal,
+ edges_by_kind,
+ execute_and_assemble,
+ ndjson_files_for_run,
+ node_pipe_codes,
+)
+from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData
+
+
[email protected](loop_scope="class")
+class TestWfGraphTracingSequence:
+ """PipeSequence produces correct GraphSpec via NDJSON event tracing on Temporal."""
+
+ async def _get_result(
+ self,
+ sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ) -> tuple[TracingResult, GraphSpec]:
+ """Execute workflow and return result with assembled GraphSpec, asserting it's not None."""
+ result = await execute_and_assemble(
+ pipe_job=sequence_tracing_job,
+ temporal_client=temporal_client,
+ traces_dir=str(tracing_tmp_dir),
+ )
+ assert result.graph_spec is not None, "GraphSpec should be assembled from trace events"
+ return result, result.graph_spec
+
+ async def test_sequence_produces_ndjson_files(
+ self,
+ sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """Workflow execution writes NDJSON event files to the traces directory."""
+ result = await execute_and_assemble(
+ pipe_job=sequence_tracing_job,
+ temporal_client=temporal_client,
+ traces_dir=str(tracing_tmp_dir),
+ )
+ ndjson_files = ndjson_files_for_run(str(tracing_tmp_dir), result.pipeline_run_id)
+ assert len(ndjson_files) >= 1, "At least one NDJSON file should be written"
+
+ async def test_sequence_assembles_valid_graph(
+ self,
+ sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """Assembled GraphSpec has expected node count and matching graph_id."""
+ result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)
+ assert graph_spec.graph_id == result.pipeline_run_id
+ assert len(graph_spec.nodes) == SequenceTracingTestData.EXPECTED_NODE_COUNT
+
+ async def test_sequence_all_nodes_succeeded(
+ self,
+ sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """All nodes in the sequence graph have SUCCEEDED status."""
+ _result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)
+ for node in graph_spec.nodes:
+ assert node.status == NodeStatus.SUCCEEDED, f"Node '{node.pipe_code}' should be SUCCEEDED, got {node.status}"
+ assert_all_nodes_terminal(graph_spec)
+
+ async def test_sequence_pipe_codes(
+ self,
+ sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """All expected pipe_codes are present in the assembled graph nodes."""
+ _result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)
+ actual_codes = node_pipe_codes(graph_spec)
+ assert actual_codes == SequenceTracingTestData.EXPECTED_PIPE_CODES
+
+ async def test_sequence_contains_edges(
+ self,
+ sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """CONTAINS edges from sequence controller to each child step."""
+ _result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)
+ edge_groups = edges_by_kind(graph_spec)
+ contains_edges = edge_groups.get(EdgeKind.CONTAINS, [])
+ assert len(contains_edges) == SequenceTracingTestData.EXPECTED_CONTAINS_EDGE_COUNT
+
+ async def test_sequence_data_edges(
+ self,
+ sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ ):
+ """DATA edges connect step outputs to step inputs."""
+ _result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)
+ edge_groups = edges_by_kind(graph_spec)
+ data_edges = edge_groups.get(EdgeKind.DATA, [])
+ assert len(data_edges) >= SequenceTracingTestData.MIN_DATA_EDGES
diff --git a/tests/integration/pipelex/temporal/workflows/conftest.py b/tests/integration/pipelex/temporal/workflows/conftest.py
new file mode 100644
index 000000000..56a7a2514
--- /dev/null
+++ b/tests/integration/pipelex/temporal/workflows/conftest.py
@@ -0,0 +1,19 @@
+import uuid
+from collections.abc import Generator
+
+import pytest
+
+from pipelex.hub import get_report_delegate
+
+
+def workflow_run_id() -> Generator[str, None, None]:
+ """Provide a workflow run ID with an open reporting registry.
+
+ Opens a registry for the run ID before the test and closes it after,
+ mirroring the job_metadata fixture pattern.
+ """
+ run_id = str(uuid.uuid4())
+ get_report_delegate().open_registry(pipeline_run_id=run_id)
+ yield run_id
+ get_report_delegate().close_registry(pipeline_run_id=run_id)
diff --git a/tests/integration/pipelex/temporal/workflows/test_wf_child_crafter.py b/tests/integration/pipelex/temporal/workflows/test_wf_child_crafter.py
new file mode 100644
index 000000000..1a89e90e5
--- /dev/null
+++ b/tests/integration/pipelex/temporal/workflows/test_wf_child_crafter.py
@@ -0,0 +1,32 @@
+import uuid
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.test_extras.temporal_test_tasks import TEMPORAL_TEST_ACTIVITIES, TEMPORAL_TEST_WORKFLOWS
+from pipelex.temporal.test_extras.wf_test_content_generator_child import WfTestContentGeneratorChild
+
+
[email protected](loop_scope="class")
+class TestWfChildCrafter:
+ async def test_wf_child_craft(self, temporal_client: TemporalClient, workflow_run_id: str, pipe_run_mode: PipeRunMode):
+ task_queue = str(uuid.uuid4())
+ async with get_task_manager().make_worker(
+ temporal_client,
+ task_queue=task_queue,
+ is_not_sandboxed=True,
+ test_workflows=TEMPORAL_TEST_WORKFLOWS,
+ test_activities=TEMPORAL_TEST_ACTIVITIES,
+ ):
+ await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfTestContentGeneratorChild.run,
+ arg=pipe_run_mode.is_dry,
+ id=workflow_run_id,
+ task_queue=task_queue,
+ )
diff --git a/tests/integration/pipelex/temporal/workflows/test_wf_gen_text.py b/tests/integration/pipelex/temporal/workflows/test_wf_gen_text.py
new file mode 100644
index 000000000..720b9cecc
--- /dev/null
+++ b/tests/integration/pipelex/temporal/workflows/test_wf_gen_text.py
@@ -0,0 +1,59 @@
+# import uuid
+
+# import pytest
+# from pipelex import pretty_print
+# from pipelex.cogt.content_generation.assignment_models import LLMAssignment
+# from pipelex.cogt.llm.llm_prompt import LLMPrompt
+# from pipelex.hub import get_llm_deck
+# from pipelex.pipeline.job_metadata import JobMetadata
+# from temporalio import activity
+# from temporalio.client import Client as TemporalClient
+
+# from pipelex.temporal.temporal_hub import get_task_manager
+# from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text
+# from pipelex.temporal.tprl_content_generation.wf_make_llm_text import WfMakeLLMText
+
+# USER_TEXT_FOR_BASE = """
+# Write a detailed description of a woman's clothing in the style of a 19th-century novel.
+# Keep it short: 3 sentences max
+# """
+
+
+# @activity.defn(name="act_llm_gen_text")
+# async def act_llm_gen_text_mocked(llm_assignment: LLMAssignment) -> str:
+# return f"Mocked text from {llm_assignment.desc}"
+
+
+# # silence a warning that comes from deep down in temporalio's pydantic converter
+# pytestmark = pytest.mark.filterwarnings("ignore:The `parse_obj` method is deprecated", "ignore:The `dict` method is deprecated")
+
+# TODO: fix and restore this test
+
+# @pytest.mark.asyncio(loop_scope="class")
+# class TestAsyncCogtLLMGenTextAndObject:
+# async def test_wf_make_llm_text(self, temporal_client: TemporalClient):
+# llm_setting = get_llm_deck().get_llm_setting(llm_setting_or_preset_id="llm_for_testing_gen_text")
+# llm_prompt_for_text = LLMPrompt(user_text=USER_TEXT_FOR_BASE)
+
+# llm_assignment = LLMAssignment(
+# job_metadata=JobMetadata(),
+# llm_setting=llm_setting,
+# llm_prompt=llm_prompt_for_text,
+# )
+
+# task_queue = str(uuid.uuid4())
+# workflow_id = str(uuid.uuid4())
+# async with get_task_manager().make_worker(
+# temporal_client,
+# task_queue=task_queue,
+# substitute_activities={act_llm_gen_text: act_llm_gen_text_mocked},
+# ):
+# crafted_text = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+# WfMakeLLMText.run,
+# arg=llm_assignment,
+# id=workflow_id,
+# task_queue=task_queue,
+# )
+# pretty_print(crafted_text, title="make_llm_text")
+# assert isinstance(crafted_text, str)
+# assert crafted_text
diff --git a/tests/integration/pipelex/temporal/workflows/test_wf_jinja2.py b/tests/integration/pipelex/temporal/workflows/test_wf_jinja2.py
new file mode 100644
index 000000000..d510e0c42
--- /dev/null
+++ b/tests/integration/pipelex/temporal/workflows/test_wf_jinja2.py
@@ -0,0 +1,37 @@
+# import uuid
+
+# import pytest
+# from pipelex import pretty_print
+# from pipelex.cogt.content_generation.assignment_models import Jinja2Assignment
+# from temporalio.client import Client as TemporalClient
+
+# from pipelex.temporal.temporal_hub import get_task_manager
+# from pipelex.temporal.tprl_content_generation.wf_make_jinja2_text import WfMakeJinja2Text
+
+# TODO: fix and restore this test
+
+# @pytest.mark.asyncio(loop_scope="class")
+# class TestWfJinja2:
+# async def test_wf_jinja2_text(self, temporal_client: TemporalClient):
+# task_queue = str(uuid.uuid4())
+# workflow_id = str(uuid.uuid4())
+# input_content = {
+# # "jinja2": "The answer is: {{ the_answer }}",
+# "the_answer": "elementary, my dear Watson",
+# }
+# jinja2_assignment = Jinja2Assignment(
+# context=input_content,
+# jinja2="The answer is: {{ the_answer }}",
+# )
+# async with get_task_manager().make_worker(
+# temporal_client,
+# task_queue=task_queue,
+# ):
+# crafted_text = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+# WfMakeJinja2Text.run,
+# arg=jinja2_assignment,
+# id=workflow_id,
+# task_queue=task_queue,
+# )
+# pretty_print(crafted_text, title="make_llm_text")
+# assert crafted_text == "The answer is: elementary, my dear Watson"
diff --git a/tests/integration/pipelex/test_inline_structure_concepts.py b/tests/integration/pipelex/test_inline_structure_concepts.py
index 5fb75da21..e2e6ec428 100644
--- a/tests/integration/pipelex/test_inline_structure_concepts.py
+++ b/tests/integration/pipelex/test_inline_structure_concepts.py
@@ -57,15 +57,15 @@ class TestInlineStructureConcepts:
assert concept.domain_code == "test_domain"
assert concept.code == "TestFeatureAnalysis"
assert concept.description == "Analysis of a photo's visual content"
- assert concept.structure_class_name == "TestFeatureAnalysis"
+ assert concept.structure_class_name == "test_domain__TestFeatureAnalysis"
# Verify the generated class is registered and accessible
- assert get_class_registry().has_class("TestFeatureAnalysis")
- generated_class = get_class_registry().get_required_subclass("TestFeatureAnalysis", StructuredContent)
+ assert get_class_registry().has_class("test_domain__TestFeatureAnalysis")
+ generated_class = get_class_registry().get_required_subclass("test_domain__TestFeatureAnalysis", StructuredContent)
# Verify class structure
assert issubclass(generated_class, StructuredContent)
- assert generated_class.__name__ == "TestFeatureAnalysis"
+ assert generated_class.__name__ == "test_domain__TestFeatureAnalysis"
# Test instantiation of generated class
instance = generated_class(
@@ -112,7 +112,7 @@ class TestInlineStructureConcepts:
)
# Should default to TextContent since TestAutoDetect is not a registered class
- assert concept.structure_class_name == "TestAutoDetect"
+ assert concept.structure_class_name == "test_domain__TestAutoDetect"
assert concept.refines == "native.Text"
def test_inline_structure_with_complex_types(self):
@@ -152,10 +152,10 @@ class TestInlineStructureConcepts:
# Verify concept creation
assert concept.code == "ComplexDocument"
assert concept.domain_code == "test_domain"
- assert concept.structure_class_name == "ComplexDocument"
+ assert concept.structure_class_name == "test_domain__ComplexDocument"
# Verify the generated class works
- generated_class = get_class_registry().get_required_subclass("ComplexDocument", StructuredContent)
+ generated_class = get_class_registry().get_required_subclass("test_domain__ComplexDocument", StructuredContent)
# Test instantiation with complex types
instance = generated_class(
@@ -216,12 +216,12 @@ not a registered subclass of StuffContent",
)
# Both should be created successfully
- assert concept1.structure_class_name == "Person"
- assert concept2.structure_class_name == "Product"
+ assert concept1.structure_class_name == "test_domain__Person"
+ assert concept2.structure_class_name == "test_domain__Product"
# Both classes should be registered and independent
- person_class = get_class_registry().get_required_subclass("Person", StructuredContent)
- product_class = get_class_registry().get_required_subclass("Product", StructuredContent)
+ person_class = get_class_registry().get_required_subclass("test_domain__Person", StructuredContent)
+ product_class = get_class_registry().get_required_subclass("test_domain__Product", StructuredContent)
assert person_class != product_class
@@ -268,11 +268,11 @@ not a registered subclass of StuffContent",
# Verify concept properties
assert concept.domain_code == "test_domain"
assert concept.code == "AnimalInfo"
- assert concept.structure_class_name == "AnimalInfo"
+ assert concept.structure_class_name == "test_domain__AnimalInfo"
# Verify the generated class is registered and accessible
- assert get_class_registry().has_class("AnimalInfo")
- generated_class = get_class_registry().get_required_subclass("AnimalInfo", StructuredContent)
+ assert get_class_registry().has_class("test_domain__AnimalInfo")
+ generated_class = get_class_registry().get_required_subclass("test_domain__AnimalInfo", StructuredContent)
# Test instantiation with quoted data
instance = generated_class(
diff --git a/tests/unit/pipelex/cli/test_agent_output.py b/tests/unit/pipelex/cli/test_agent_output.py
index 78d022f4a..1923daf95 100644
--- a/tests/unit/pipelex/cli/test_agent_output.py
+++ b/tests/unit/pipelex/cli/test_agent_output.py
@@ -8,13 +8,16 @@ import json
import pytest
import typer
+from pipelex.base_exceptions import PipelexError
from pipelex.cli.agent_cli.commands.agent_output import (
+ AGENT_ERROR_DOMAINS,
AGENT_ERROR_HINTS,
_build_error_source, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
agent_error,
agent_success,
extract_validation_errors,
)
+from pipelex.cogt.exceptions import CogtError, InferenceBackendCredentialsError, InferenceBackendCredentialsErrorType, InferenceErrorCategory
from pipelex.core.bundles.exceptions import PipelexBundleBlueprintValidationErrorData
from pipelex.core.exceptions import PipeFactoryErrorData, PipesAndConceptValidationErrorData
from pipelex.core.pipes.exceptions import PipeFactoryErrorType, PipeValidationErrorType
@@ -260,3 +263,111 @@ class TestAgentOutput:
assert "error_source" in parsed
assert len(parsed["error_source"]) == 1
assert "no traceback" in parsed["error_source"][0]
+
+ # -------------------------------------------------------------------------
+ # to_error_report() integration tests
+ # -------------------------------------------------------------------------
+
+ def test_agent_error_uses_report_hint_from_cogt_error(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """agent_error should use user_action from to_error_report() as the hint field."""
+ cause = CogtError("inference failed", user_action="Check your API key and try again")
+ with pytest.raises(typer.Exit):
+ agent_error("inference failed", "CogtError", cause=cause)
+
+ parsed = json.loads(capsys.readouterr().err)
+ assert parsed["hint"] == "Check your API key and try again"
+
+ def test_agent_error_uses_report_retryable_from_cogt_error(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """agent_error should set retryable=True when error_category is TRANSIENT."""
+ cause = CogtError("rate limited", error_category=InferenceErrorCategory.TRANSIENT)
+ with pytest.raises(typer.Exit):
+ agent_error("rate limited", "CogtError", cause=cause)
+
+ parsed = json.loads(capsys.readouterr().err)
+ assert parsed["retryable"] is True
+
+ def test_agent_error_uses_report_error_category(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """agent_error should include error_category from to_error_report()."""
+ cause = CogtError("bad config", error_category=InferenceErrorCategory.CONFIGURATION)
+ with pytest.raises(typer.Exit):
+ agent_error("bad config", "CogtError", cause=cause)
+
+ parsed = json.loads(capsys.readouterr().err)
+ assert parsed["error_category"] == "configuration"
+
+ def test_agent_error_falls_back_to_lookup_for_non_pipelex_error(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """agent_error should use lookup dicts when cause is not a PipelexError."""
+ cause = FileNotFoundError("missing.mthds")
+ with pytest.raises(typer.Exit):
+ agent_error("file not found", "FileNotFoundError", cause=cause)
+
+ parsed = json.loads(capsys.readouterr().err)
+ assert parsed["hint"] == AGENT_ERROR_HINTS["FileNotFoundError"]
+ assert "error_category" not in parsed
+
+ def test_agent_error_falls_back_to_lookup_when_report_fields_none(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """agent_error should fall back to lookup when PipelexError has no category/user_action."""
+ cause = PipelexError("something failed")
+ with pytest.raises(typer.Exit):
+ agent_error("something failed", "ValidateBundleError", cause=cause)
+
+ parsed = json.loads(capsys.readouterr().err)
+ # hint should come from AGENT_ERROR_HINTS since PipelexError has no user_action
+ assert parsed["hint"] == AGENT_ERROR_HINTS["ValidateBundleError"]
+
+ def test_agent_error_report_hint_overrides_lookup(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """When cause has user_action, it should override the lookup dict hint."""
+ # Use an error_type that exists in AGENT_ERROR_HINTS
+ cause = CogtError("model not found", user_action="Use pipelex-agent models to list available models")
+ with pytest.raises(typer.Exit):
+ agent_error("model not found", "ModelChoiceNotFoundError", cause=cause)
+
+ parsed = json.loads(capsys.readouterr().err)
+ assert parsed["hint"] == "Use pipelex-agent models to list available models"
+ assert parsed["hint"] != AGENT_ERROR_HINTS["ModelChoiceNotFoundError"]
+
+ def test_agent_error_extra_still_overrides_report(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """Explicit **extra kwargs should override report-derived fields."""
+ cause = CogtError("failed", user_action="from report")
+ with pytest.raises(typer.Exit):
+ agent_error("failed", "CogtError", cause=cause, hint="custom override")
+
+ parsed = json.loads(capsys.readouterr().err)
+ assert parsed["hint"] == "custom override"
+
+ def test_agent_error_includes_model_and_provider_from_report(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """agent_error should include model and provider from to_error_report()."""
+ cause = InferenceBackendCredentialsError(
+ credentials_error_type=InferenceBackendCredentialsErrorType.VAR_NOT_FOUND,
+ backend_name="openai",
+ message="OPENAI_API_KEY not set",
+ key_name="OPENAI_API_KEY",
+ )
+ with pytest.raises(typer.Exit):
+ agent_error("OPENAI_API_KEY not set", "InferenceBackendCredentialsError", cause=cause)
+
+ parsed = json.loads(capsys.readouterr().err)
+ assert parsed["provider"] == "openai"
+
+ def test_agent_error_non_retryable_not_in_output(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """Non-retryable errors should not include retryable field in JSON output."""
+ cause = CogtError("bad config", error_category=InferenceErrorCategory.CONFIGURATION)
+ with pytest.raises(typer.Exit):
+ agent_error("bad config", "CogtError", cause=cause)
+
+ parsed = json.loads(capsys.readouterr().err)
+ assert "retryable" not in parsed
+
+ def test_agent_error_error_domain_and_category_coexist(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """error_domain (from lookup) and error_category (from report) should both appear."""
+ # Use an error_type that is in AGENT_ERROR_DOMAINS
+ error_type = "ModelChoiceNotFoundError"
+ assert error_type in AGENT_ERROR_DOMAINS, "precondition: error_type must be in AGENT_ERROR_DOMAINS"
+
+ cause = CogtError("model not found", error_category=InferenceErrorCategory.CONFIGURATION)
+ with pytest.raises(typer.Exit):
+ agent_error("model not found", error_type, cause=cause)
+
+ parsed = json.loads(capsys.readouterr().err)
+ assert parsed["error_domain"] == AGENT_ERROR_DOMAINS[error_type]
+ assert parsed["error_category"] == "configuration"
diff --git a/tests/unit/pipelex/cli/test_check_rules_sync_cmd.py b/tests/unit/pipelex/cli/test_check_rules_sync_cmd.py
new file mode 100644
index 000000000..99bce85f5
--- /dev/null
+++ b/tests/unit/pipelex/cli/test_check_rules_sync_cmd.py
@@ -0,0 +1,109 @@
+"""Unit tests for the check_rules_sync_cmd helpers."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+
+from pipelex.cli.dev_cli.commands.check_rules_sync_cmd import (
+ _DEFAULT_TARGETS, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+ _get_preferred_targets_from_toml, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+ check_rules_sync_cmd,
+)
+from pipelex.system.configuration.configs import AgentTarget
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+ from pytest_mock import MockerFixture
+
+
+class TestGetPreferredTargetsFromToml:
+ def test_default_includes_claude_and_agents(self) -> None:
+ """Fallback must match pipelex.toml default of ['claude', 'agents']."""
+ assert _DEFAULT_TARGETS == [AgentTarget.CLAUDE, AgentTarget.AGENTS]
+
+ def test_returns_default_when_toml_missing(self, mocker: MockerFixture, tmp_path: Path) -> None:
+ mocker.patch("pipelex.cli.dev_cli.commands.check_rules_sync_cmd.Path", return_value=tmp_path / "missing.toml")
+ result = _get_preferred_targets_from_toml()
+ assert result == _DEFAULT_TARGETS
+
+ @pytest.mark.parametrize(
+ ("toml_body", "expected"),
+ [
+ (
+ '[pipelex.kit_config]\npreferred_agent_targets = ["claude", "agents"]\n',
+ [AgentTarget.CLAUDE, AgentTarget.AGENTS],
+ ),
+ (
+ '[pipelex.kit_config]\npreferred_agent_targets = ["cursor"]\n',
+ [AgentTarget.CURSOR],
+ ),
+ # Empty list must NOT silently disable the sync check —
+ # it should fall back to defaults.
+ ("[pipelex.kit_config]\npreferred_agent_targets = []\n", _DEFAULT_TARGETS),
+ # Unknown value triggers ValueError → fallback.
+ ('[pipelex.kit_config]\npreferred_agent_targets = ["unknown"]\n', _DEFAULT_TARGETS),
+ # Missing key triggers KeyError → fallback.
+ ("[pipelex.kit_config]\n", _DEFAULT_TARGETS),
+ ],
+ )
+ def test_parses_toml_targets_with_safe_fallbacks(
+ self,
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+ toml_body: str,
+ expected: list[AgentTarget],
+ ) -> None:
+ pipelex_dir = tmp_path / "pipelex"
+ pipelex_dir.mkdir()
+ (pipelex_dir / "pipelex.toml").write_text(toml_body, encoding="utf-8")
+ monkeypatch.chdir(tmp_path)
+ result = _get_preferred_targets_from_toml()
+ assert result == expected
+
+
+class TestCheckRulesSyncCmdCursorExclusivity:
+ """The CURSOR shortcut must require exclusivity, not membership.
+
+ A mixed list like ['cursor', 'claude'] should NOT skip the sync check —
+ it would silently bypass verification of file-based targets.
+ """
+
+ def test_cursor_only_skips(self, mocker: MockerFixture) -> None:
+ mocker.patch(
+ "pipelex.cli.dev_cli.commands.check_rules_sync_cmd._get_preferred_targets_from_toml",
+ return_value=[AgentTarget.CURSOR],
+ )
+ load_index_mock = mocker.patch("pipelex.cli.dev_cli.commands.check_rules_sync_cmd.load_index")
+ check_rules_sync_cmd(show_diff=False, quiet=True)
+ load_index_mock.assert_called_once()
+
+ def test_mixed_cursor_with_claude_does_not_skip(self, mocker: MockerFixture) -> None:
+ """If the TOML somehow contains a mixed list, the cursor branch must NOT trigger.
+
+ The pydantic validator forbids the mix, but this command bypasses pydantic.
+ """
+ mocker.patch(
+ "pipelex.cli.dev_cli.commands.check_rules_sync_cmd._get_preferred_targets_from_toml",
+ return_value=[AgentTarget.CURSOR, AgentTarget.CLAUDE],
+ )
+ # Use a fake index with a CLAUDE target that points to a path we control.
+ fake_target = mocker.MagicMock()
+ fake_target.path = "/nonexistent/CLAUDE.md"
+ fake_target.sets = None
+ fake_index = mocker.MagicMock()
+ fake_index.agent_rules.targets = {AgentTarget.CLAUDE: fake_target}
+ mocker.patch(
+ "pipelex.cli.dev_cli.commands.check_rules_sync_cmd.load_index",
+ return_value=fake_index,
+ )
+ mocker.patch(
+ "pipelex.cli.dev_cli.commands.check_rules_sync_cmd.build_merged_rules",
+ return_value="",
+ )
+ # Mixed cursor + claude is invalid → command should exit(1), not silently skip.
+ with pytest.raises(SystemExit) as exc_info:
+ check_rules_sync_cmd(show_diff=False, quiet=True)
+ assert exc_info.value.code == 1
diff --git a/tests/unit/pipelex/cli/test_error_handlers.py b/tests/unit/pipelex/cli/test_error_handlers.py
new file mode 100644
index 000000000..d53305e7f
--- /dev/null
+++ b/tests/unit/pipelex/cli/test_error_handlers.py
@@ -0,0 +1,109 @@
+"""Unit tests for the main CLI error handlers."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+import typer
+
+from pipelex.cli.error_handlers import (
+ ErrorContext,
+ handle_inference_setup_required_error,
+ handle_model_availability_error,
+ handle_model_choice_error,
+ handle_model_deck_preset_error,
+ handle_validate_bundle_error,
+)
+from pipelex.cogt.exceptions import ModelDeckPresetValidatonError
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError
+from pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.pipeline.validate_bundle import ValidateBundleError
+from pipelex.system.pipelex_service.exceptions import InferenceSetupRequiredError
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+
+class TestErrorHandlers:
+ """Tests for Rich error handler functions."""
+
+ def test_handle_model_choice_error_exits_and_calls_to_error_report(self, mocker: MockerFixture) -> None:
+ """handle_model_choice_error should call to_error_report() and exit with code 1."""
+ mocker.patch("pipelex.cli.error_handlers.get_console")
+ exc = PipeOperatorModelChoiceError(
+ message="model 'gpt-5' not found",
+ pipe_type="llm_text",
+ pipe_code="my_pipe",
+ model_type=ModelType.LLM,
+ model_choice="gpt-5",
+ )
+ spy = mocker.spy(exc, "to_error_report")
+
+ with pytest.raises(typer.Exit) as exc_info:
+ handle_model_choice_error(exc, context=ErrorContext.PIPE_RUN)
+
+ assert exc_info.value.exit_code == 1
+ spy.assert_called_once()
+
+ def test_handle_model_availability_error_exits_and_calls_to_error_report(self, mocker: MockerFixture) -> None:
+ """handle_model_availability_error should call to_error_report() and exit with code 1."""
+ mocker.patch("pipelex.cli.error_handlers.get_console")
+ exc = PipeOperatorModelAvailabilityError(
+ message="model not available",
+ run_mode=PipeRunMode.LIVE,
+ pipe_type="llm_text",
+ pipe_code="my_pipe",
+ pipe_stack=["my_pipe"],
+ model_handle="gpt-4o",
+ fallback_list=["gpt-4o-mini"],
+ )
+ spy = mocker.spy(exc, "to_error_report")
+
+ with pytest.raises(typer.Exit) as exc_info:
+ handle_model_availability_error(exc, context=ErrorContext.PIPE_RUN)
+
+ assert exc_info.value.exit_code == 1
+ spy.assert_called_once()
+
+ def test_handle_model_deck_preset_error_exits_and_calls_to_error_report(self, mocker: MockerFixture) -> None:
+ """handle_model_deck_preset_error should call to_error_report() and exit with code 1."""
+ mocker.patch("pipelex.cli.error_handlers.get_console")
+ exc = ModelDeckPresetValidatonError(
+ message="preset references unavailable model",
+ model_type=ModelType.LLM,
+ preset_id="default",
+ model_handle="gpt-5",
+ enabled_backends={"openai"},
+ )
+ spy = mocker.spy(exc, "to_error_report")
+
+ with pytest.raises(typer.Exit) as exc_info:
+ handle_model_deck_preset_error(exc, context=ErrorContext.VALIDATION)
+
+ assert exc_info.value.exit_code == 1
+ spy.assert_called_once()
+
+ def test_handle_validate_bundle_error_exits_and_calls_to_error_report(self, mocker: MockerFixture) -> None:
+ """handle_validate_bundle_error should call to_error_report() and exit with code 1."""
+ mocker.patch("pipelex.cli.error_handlers.get_console")
+ exc = ValidateBundleError(message="validation failed")
+ spy = mocker.spy(exc, "to_error_report")
+
+ with pytest.raises(typer.Exit) as exc_info:
+ handle_validate_bundle_error(exc)
+
+ assert exc_info.value.exit_code == 1
+ spy.assert_called_once()
+
+ def test_handle_inference_setup_required_error_exits(self, mocker: MockerFixture) -> None:
+ """handle_inference_setup_required_error should exit with code 1."""
+ mocker.patch("pipelex.cli.error_handlers.get_console")
+ exc = InferenceSetupRequiredError()
+
+ with pytest.raises(typer.Exit) as exc_info:
+ handle_inference_setup_required_error(exc)
+
+ assert exc_info.value.exit_code == 1
diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py
new file mode 100644
index 000000000..075966c7a
--- /dev/null
+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_schema.py
@@ -0,0 +1,78 @@
+"""Unit tests for object_class_schema field on ObjectAssignment and TextThenObjectAssignment."""
+
+from pydantic import BaseModel
+
+from pipelex.cogt.content_generation.assignment_models import (
+ LLMAssignment,
+ LLMAssignmentFactory,
+ ObjectAssignment,
+ TextThenObjectAssignment,
+)
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_prompt_template import LLMPromptTemplate
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.pipeline.job_metadata import JobMetadata
+
+
+class SampleOutputModel(BaseModel):
+ name: str
+ value: int
+
+
+def _make_stub_job_metadata() -> JobMetadata:
+ return JobMetadata(
+ user_id="test-user",
+ pipeline_run_id="test-run",
+ )
+
+
+def _make_stub_llm_assignment() -> LLMAssignment:
+ return LLMAssignment(
+ job_metadata=_make_stub_job_metadata(),
+ llm_setting=LLMSetting(model="test-model", temperature=0.7),
+ llm_prompt=LLMPrompt(user_text="test prompt"),
+ )
+
+
+def _make_stub_llm_assignment_factory() -> LLMAssignmentFactory:
+ return LLMAssignmentFactory(
+ job_metadata=_make_stub_job_metadata(),
+ llm_setting=LLMSetting(model="test-model", temperature=0.7),
+ llm_prompt_factory=LLMPromptTemplate.make_for_structuring_from_preliminary_text(),
+ )
+
+
+class TestAssignmentModelsSchema:
+ def test_make_for_class_captures_schema(self) -> None:
+ """ObjectAssignment.make_for_class() captures the JSON schema."""
+ assignment = ObjectAssignment.make_for_class(
+ object_class=SampleOutputModel,
+ llm_assignment=_make_stub_llm_assignment(),
+ )
+ assert assignment.object_class_name == "SampleOutputModel"
+ assert assignment.object_class_schema is not None
+ assert assignment.object_class_schema["title"] == "SampleOutputModel"
+ assert "name" in assignment.object_class_schema["properties"]
+ assert "value" in assignment.object_class_schema["properties"]
+
+ def test_object_assignment_json_roundtrip(self) -> None:
+ """ObjectAssignment with schema survives JSON round-trip."""
+ assignment = ObjectAssignment.make_for_class(
+ object_class=SampleOutputModel,
+ llm_assignment=_make_stub_llm_assignment(),
+ )
+ json_str = assignment.model_dump_json()
+ restored = ObjectAssignment.model_validate_json(json_str)
+ assert restored.object_class_schema == assignment.object_class_schema
+ assert restored.object_class_name == "SampleOutputModel"
+
+ def test_text_then_object_assignment_carries_schema(self) -> None:
+ """TextThenObjectAssignment can carry an object_class_schema."""
+ schema = SampleOutputModel.model_json_schema()
+ assignment = TextThenObjectAssignment(
+ object_class_name="SampleOutputModel",
+ object_class_schema=schema,
+ llm_assignment_for_text=_make_stub_llm_assignment(),
+ llm_assignment_factory_to_object=_make_stub_llm_assignment_factory(),
+ )
+ assert assignment.object_class_schema["title"] == "SampleOutputModel"
diff --git a/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py
new file mode 100644
index 000000000..3c161a7f4
--- /dev/null
+++ b/tests/unit/pipelex/cogt/content_generation/test_assignment_models_security.py
@@ -0,0 +1,103 @@
+"""Wire-format perimeter tests: prove that x-python-* schema extensions survive
+serialization through ObjectAssignment / TextThenObjectAssignment, so the codegen
+boundary in SchemaToModelFactory is the only line of defense (and works).
+"""
+
+from typing import Any
+
+import pytest
+
+from pipelex.cogt.content_generation.assignment_models import (
+ LLMAssignment,
+ LLMAssignmentFactory,
+ ObjectAssignment,
+ TextThenObjectAssignment,
+)
+from pipelex.cogt.content_generation.exceptions import UnsafeSchemaError
+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_prompt_template import LLMPromptTemplate
+from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.pipeline.job_metadata import JobMetadata
+
+
+def _make_stub_job_metadata() -> JobMetadata:
+ return JobMetadata(
+ user_id="test-user",
+ pipeline_run_id="test-run",
+ )
+
+
+def _make_stub_llm_assignment() -> LLMAssignment:
+ return LLMAssignment(
+ job_metadata=_make_stub_job_metadata(),
+ llm_setting=LLMSetting(model="test-model", temperature=0.7),
+ llm_prompt=LLMPrompt(user_text="test prompt"),
+ )
+
+
+def _make_stub_llm_assignment_factory() -> LLMAssignmentFactory:
+ return LLMAssignmentFactory(
+ job_metadata=_make_stub_job_metadata(),
+ llm_setting=LLMSetting(model="test-model", temperature=0.7),
+ llm_prompt_factory=LLMPromptTemplate.make_for_structuring_from_preliminary_text(),
+ )
+
+
+def _malicious_schema() -> dict[str, Any]:
+ return {
+ "title": "Innocent",
+ "type": "object",
+ "properties": {"hit": {"$ref": "#/$defs/Run"}},
+ "$defs": {"Run": {"x-python-import": {"module": "subprocess", "name": "run"}}},
+ }
+
+
+class TestAssignmentModelsSecurity:
+ def test_object_assignment_preserves_x_python_import_through_json_roundtrip(self) -> None:
+ """A malicious x-python-import in object_class_schema survives the wire format unchanged.
+
+ The dict[str, Any] field is intentionally permissive — rejection happens at the
+ codegen boundary (SchemaToModelFactory), not at deserialization.
+ """
+ assignment = ObjectAssignment(
+ object_class_name="Innocent",
+ object_class_schema=_malicious_schema(),
+ llm_assignment_for_object=_make_stub_llm_assignment(),
+ )
+ json_str = assignment.model_dump_json()
+ restored = ObjectAssignment.model_validate_json(json_str)
+ assert restored.object_class_schema == _malicious_schema()
+ assert restored.object_class_schema["$defs"]["Run"]["x-python-import"] == {
+ "module": "subprocess",
+ "name": "run",
+ }
+
+ def test_text_then_object_assignment_carries_unsafe_schema_unchanged(self) -> None:
+ """TextThenObjectAssignment also carries x-python-* extensions verbatim — same threat surface."""
+ assignment = TextThenObjectAssignment(
+ object_class_name="Innocent",
+ object_class_schema=_malicious_schema(),
+ llm_assignment_for_text=_make_stub_llm_assignment(),
+ llm_assignment_factory_to_object=_make_stub_llm_assignment_factory(),
+ )
+ assert assignment.object_class_schema["$defs"]["Run"]["x-python-import"] == {
+ "module": "subprocess",
+ "name": "run",
+ }
+
+ def test_codegen_rejects_post_roundtrip_unsafe_schema(self) -> None:
+ """Closes the loop: a payload that survives the wire crossing is rejected at codegen.
+
+ This is the actual contract: the wire format is permissive on purpose, the
+ boundary in SchemaToModelFactory.make_from_json_schema is the chokepoint.
+ """
+ assignment = ObjectAssignment(
+ object_class_name="Innocent",
+ object_class_schema=_malicious_schema(),
+ llm_assignment_for_object=_make_stub_llm_assignment(),
+ )
+ restored = ObjectAssignment.model_validate_json(assignment.model_dump_json())
+ with pytest.raises(UnsafeSchemaError) as exc_info:
+ SchemaToModelFactory.make_from_json_schema(restored.object_class_schema, restored.object_class_name)
+ assert "x-python-import" in str(exc_info.value)
diff --git a/tests/unit/pipelex/cogt/content_generation/test_model_dump_subclass.py b/tests/unit/pipelex/cogt/content_generation/test_model_dump_subclass.py
new file mode 100644
index 000000000..1295a1c06
--- /dev/null
+++ b/tests/unit/pipelex/cogt/content_generation/test_model_dump_subclass.py
@@ -0,0 +1,50 @@
+"""Regression guard: model_dump(serialize_as_any=True) roundtrip for schema-reconstructed models.
+
+content_generator.py uses `object_class.model_validate(raw_obj.model_dump(serialize_as_any=True))`
+where raw_obj is a dynamically-reconstructed BaseModel from SchemaToModelFactory. This test verifies
+the roundtrip preserves all data, including nested models.
+"""
+
+from pydantic import BaseModel, Field
+
+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
+
+
+class InnerDetail(BaseModel):
+ label: str = Field(description="A label")
+ value: int = Field(description="A numeric value")
+
+
+class OuterModel(BaseModel):
+ title: str = Field(description="The title")
+ detail: InnerDetail = Field(description="Nested detail object")
+
+
+class TestModelDumpSubclass:
+ def test_reconstructed_flat_model_roundtrip(self) -> None:
+ """Flat reconstructed model roundtrips through model_dump(serialize_as_any=True) → model_validate."""
+ schema = InnerDetail.model_json_schema()
+ reconstructed_class = SchemaToModelFactory.make_from_json_schema(schema, "InnerDetail")
+
+ raw_obj: BaseModel = reconstructed_class(label="score", value=42)
+
+ dumped = raw_obj.model_dump(serialize_as_any=True)
+ restored = InnerDetail.model_validate(dumped)
+ assert restored.label == "score"
+ assert restored.value == 42
+
+ def test_reconstructed_nested_model_roundtrip(self) -> None:
+ """Nested reconstructed model roundtrips through model_dump(serialize_as_any=True) → model_validate."""
+ schema = OuterModel.model_json_schema()
+ reconstructed_class = SchemaToModelFactory.make_from_json_schema(schema, "OuterModel")
+
+ raw_obj: BaseModel = reconstructed_class(
+ title="Test",
+ detail={"label": "score", "value": 42},
+ )
+
+ dumped = raw_obj.model_dump(serialize_as_any=True)
+ restored = OuterModel.model_validate(dumped)
+ assert restored.title == "Test"
+ assert restored.detail.label == "score"
+ assert restored.detail.value == 42
diff --git a/tests/unit/pipelex/cogt/content_generation/test_schema_to_model.py b/tests/unit/pipelex/cogt/content_generation/test_schema_to_model.py
new file mode 100644
index 000000000..cc4afb1cc
--- /dev/null
+++ b/tests/unit/pipelex/cogt/content_generation/test_schema_to_model.py
@@ -0,0 +1,350 @@
+"""Unit tests for SchemaToModelFactory: reconstructing BaseModel classes from JSON schemas."""
+
+import threading
+import uuid
+from typing import Any
+
+import pytest
+from pydantic import BaseModel, Field
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.content_generation.exceptions import UnsafeSchemaError
+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
+
+
+class SimpleModel(BaseModel):
+ name: str = Field(description="The name")
+ age: int = Field(description="The age")
+
+
+class Address(BaseModel):
+ street: str
+ city: str
+
+
+class PersonWithAddress(BaseModel):
+ name: str
+ address: Address
+
+
+def _benign_object_schema() -> dict[str, Any]:
+ return {"title": "Innocent", "type": "object", "properties": {"x": {"type": "integer"}}}
+
+
+class TestSchemaToModel:
+ def test_simple_model_reconstruction(self) -> None:
+ """A simple model can be reconstructed from its JSON schema."""
+ schema = SimpleModel.model_json_schema()
+ result_class = SchemaToModelFactory.make_from_json_schema(schema, "SimpleModel")
+
+ assert result_class.__name__ == "SimpleModel"
+ assert "name" in result_class.model_fields
+ assert "age" in result_class.model_fields
+
+ def test_reconstructed_model_can_validate(self) -> None:
+ """The reconstructed model can validate data."""
+ schema = SimpleModel.model_json_schema()
+ result_class = SchemaToModelFactory.make_from_json_schema(schema, "SimpleModel")
+
+ instance = result_class(name="Alice", age=30)
+ assert instance.name == "Alice" # type: ignore[attr-defined]
+ assert instance.age == 30 # type: ignore[attr-defined]
+
+ def test_nested_model_reconstruction(self) -> None:
+ """A model with nested BaseModel fields (producing $defs) can be reconstructed."""
+ schema = PersonWithAddress.model_json_schema()
+ result_class = SchemaToModelFactory.make_from_json_schema(schema, "PersonWithAddress")
+
+ instance = result_class(
+ name="Bob",
+ address={"street": "123 Main", "city": "NYC"},
+ )
+ assert instance.name == "Bob" # type: ignore[attr-defined]
+ assert instance.address.city == "NYC" # type: ignore[attr-defined]
+
+ def test_kajson_class_source_attached(self) -> None:
+ """The reconstructed class has __kajson_class_source__ with the generated Python source."""
+ schema = SimpleModel.model_json_schema()
+ result_class = SchemaToModelFactory.make_from_json_schema(schema, "SimpleModel")
+
+ source = getattr(result_class, "__kajson_class_source__", None)
+ assert source is not None
+ assert "class SimpleModel" in source
+ assert "BaseModel" in source
+
+ def test_caching_returns_same_class(self) -> None:
+ """Calling with the same schema returns the same class object (cached)."""
+ schema = SimpleModel.model_json_schema()
+ class_1 = SchemaToModelFactory.make_from_json_schema(schema, "SimpleModel")
+ class_2 = SchemaToModelFactory.make_from_json_schema(schema, "SimpleModel")
+
+ assert class_1 is class_2
+
+ def test_different_schemas_different_classes(self) -> None:
+ """Different schemas produce different classes even if class names differ."""
+ schema_simple = SimpleModel.model_json_schema()
+ schema_nested = PersonWithAddress.model_json_schema()
+
+ class_simple = SchemaToModelFactory.make_from_json_schema(schema_simple, "SimpleModel")
+ class_nested = SchemaToModelFactory.make_from_json_schema(schema_nested, "PersonWithAddress")
+
+ assert class_simple is not class_nested
+
+ def test_json_roundtrip_with_reconstructed_class(self) -> None:
+ """An instance of the reconstructed class survives JSON round-trip."""
+ schema = SimpleModel.model_json_schema()
+ result_class = SchemaToModelFactory.make_from_json_schema(schema, "SimpleModel")
+
+ instance = result_class(name="Charlie", age=25)
+ json_str = instance.model_dump_json()
+ restored = result_class.model_validate_json(json_str)
+ assert restored.name == "Charlie" # type: ignore[attr-defined]
+ assert restored.age == 25 # type: ignore[attr-defined]
+
+ def test_normalized_class_name_lookup(self) -> None:
+ """A class_name with underscores/double-underscores resolves via PascalCase normalization."""
+ schema = SimpleModel.model_json_schema()
+ # Override the title to simulate a dynamic concept code like "my_namespace__Greeting"
+ schema["title"] = "my_namespace__Greeting"
+ result_class = SchemaToModelFactory.make_from_json_schema(schema, "my_namespace__Greeting")
+
+ assert "name" in result_class.model_fields
+ assert "age" in result_class.model_fields
+ instance = result_class(name="Ada", age=40)
+ assert instance.name == "Ada" # type: ignore[attr-defined]
+
+ def test_concurrent_first_miss_generates_source_only_once(self, mocker: MockerFixture) -> None:
+ """N threads racing on the same uncached schema must trigger codegen exactly once.
+
+ The cache is class-level state shared across tests, so we use a unique title per
+ run to guarantee a cold-cache start. Without serialization, every thread that
+ observes the empty cache pays the full _generate_source_from_schema cost.
+ """
+ schema = SimpleModel.model_json_schema()
+ unique_title = f"ConcurrentMiss_{uuid.uuid4().hex}"
+ schema["title"] = unique_title
+
+ spy = mocker.spy(SchemaToModelFactory, "_generate_source_from_schema")
+
+ thread_count = 8
+ barrier = threading.Barrier(thread_count)
+ results: list[type[BaseModel]] = []
+ results_lock = threading.Lock()
+
+ def worker() -> None:
+ barrier.wait()
+ result = SchemaToModelFactory.make_from_json_schema(schema, unique_title)
+ with results_lock:
+ results.append(result)
+
+ threads = [threading.Thread(target=worker) for _ in range(thread_count)]
+ for thread in threads:
+ thread.start()
+ for thread in threads:
+ thread.join()
+
+ assert len(results) == thread_count
+ first_class = results[0]
+ for result_class in results[1:]:
+ assert result_class is first_class
+ assert spy.call_count == 1
+
+ def test_cache_bounded_size_evicts_lru(self, mocker: MockerFixture) -> None:
+ """The schema cache caps at _SCHEMA_CACHE_MAX_SIZE and evicts least-recently-used entries.
+
+ Lowers the bound to 3, inserts 4 distinct schemas, and asserts: (1) cache size never
+ exceeds the bound, (2) the oldest schema was evicted (re-requesting it triggers
+ fresh codegen). Saves and restores the class-level cache to keep the test isolated.
+ """
+ # Snapshot then clear class-level state so this test is hermetic.
+ saved_cache = SchemaToModelFactory._schema_cache.copy() # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ SchemaToModelFactory._schema_cache.clear() # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ mocker.patch.object(SchemaToModelFactory, "_SCHEMA_CACHE_MAX_SIZE", 3)
+ spy = mocker.spy(SchemaToModelFactory, "_generate_source_from_schema")
+
+ try:
+ schemas: list[tuple[str, dict[str, Any]]] = []
+ for index in range(4):
+ schema = SimpleModel.model_json_schema()
+ title = f"BoundedCacheTest_{uuid.uuid4().hex}_{index}"
+ schema["title"] = title
+ schemas.append((title, schema))
+ SchemaToModelFactory.make_from_json_schema(schema, title)
+ assert len(SchemaToModelFactory._schema_cache) <= 3 # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert spy.call_count == 4
+ assert len(SchemaToModelFactory._schema_cache) == 3 # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ # Oldest schema (index 0) must have been evicted: re-requesting it triggers fresh codegen.
+ oldest_title, oldest_schema = schemas[0]
+ SchemaToModelFactory.make_from_json_schema(oldest_schema, oldest_title)
+ assert spy.call_count == 5
+
+ # Most-recently-used schema (index 3) must still be cached: no fresh codegen.
+ recent_title, recent_schema = schemas[3]
+ SchemaToModelFactory.make_from_json_schema(recent_schema, recent_title)
+ assert spy.call_count == 5
+ finally:
+ SchemaToModelFactory._schema_cache.clear() # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ SchemaToModelFactory._schema_cache.update(saved_cache) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ def test_exec_blocks_dangerous_builtins(self) -> None:
+ """The restricted exec namespace blocks open(), eval(), exec(), and compile()."""
+ malicious_source = "from pydantic import BaseModel\nclass Innocent(BaseModel):\n name: str = 'ok'\nleaked = open('/etc/passwd')\n"
+ with pytest.raises(NameError, match="open"):
+ SchemaToModelFactory._exec_and_extract_class(malicious_source, "Innocent") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ # ---- Layer 1: schema sanitization (x-python-* extensions are rejected) ----
+
+ def test_x_python_import_at_top_level_is_rejected(self) -> None:
+ """A schema with x-python-import at the top level is rejected before reaching codegen."""
+ schema = _benign_object_schema()
+ schema["x-python-import"] = {"module": "subprocess", "name": "run"}
+ with pytest.raises(UnsafeSchemaError) as exc_info:
+ SchemaToModelFactory.make_from_json_schema(schema, "Innocent")
+ assert "x-python-import" in str(exc_info.value)
+
+ def test_x_python_import_in_defs_is_rejected(self) -> None:
+ """Realistic exploit shape: x-python-import nested under $defs is rejected.
+
+ This is the actual attack vector — datamodel-code-generator emits
+ `from subprocess import run` from this payload with zero sanitization.
+ """
+ schema: dict[str, Any] = {
+ "title": "Innocent",
+ "type": "object",
+ "properties": {"hit": {"$ref": "#/$defs/Run"}},
+ "$defs": {"Run": {"x-python-import": {"module": "subprocess", "name": "run"}}},
+ }
+ with pytest.raises(UnsafeSchemaError) as exc_info:
+ SchemaToModelFactory.make_from_json_schema(schema, "Innocent")
+ assert "x-python-import" in str(exc_info.value)
+
+ def test_x_python_type_is_rejected(self) -> None:
+ """Sibling extension x-python-type must also be rejected."""
+ schema = _benign_object_schema()
+ schema["properties"]["x"]["x-python-type"] = "subprocess.Popen"
+ with pytest.raises(UnsafeSchemaError) as exc_info:
+ SchemaToModelFactory.make_from_json_schema(schema, "Innocent")
+ assert "x-python-type" in str(exc_info.value)
+
+ @pytest.mark.parametrize(
+ "extension_key",
+ [
+ "x-python-import",
+ "x-python-type",
+ "x-python-class-name",
+ "x-python-fancy-future-feature",
+ ],
+ )
+ def test_any_x_python_extension_is_rejected(self, extension_key: str) -> None:
+ """Future-proofing: any x-python-* extension in the schema is rejected."""
+ schema = _benign_object_schema()
+ schema[extension_key] = "anything"
+ with pytest.raises(UnsafeSchemaError) as exc_info:
+ SchemaToModelFactory.make_from_json_schema(schema, "Innocent")
+ assert extension_key in str(exc_info.value)
+
+ def test_deeply_nested_x_python_import_is_rejected(self) -> None:
+ """The walker must recurse into deeply nested structures (arrays + $defs + properties)."""
+ schema: dict[str, Any] = {
+ "title": "Outer",
+ "type": "object",
+ "properties": {
+ "items": {
+ "type": "array",
+ "items": {"$ref": "#/$defs/Inner"},
+ }
+ },
+ "$defs": {
+ "Inner": {
+ "type": "object",
+ "properties": {
+ "evil": {"x-python-import": {"module": "subprocess", "name": "run"}},
+ },
+ },
+ },
+ }
+ with pytest.raises(UnsafeSchemaError) as exc_info:
+ SchemaToModelFactory.make_from_json_schema(schema, "Outer")
+ assert "x-python-import" in str(exc_info.value)
+
+ def test_unsafe_schema_error_names_the_offending_path(self) -> None:
+ """The error message points to where the violation was found, for diagnostic logs."""
+ schema: dict[str, Any] = {
+ "title": "Innocent",
+ "type": "object",
+ "properties": {"hit": {"$ref": "#/$defs/Run"}},
+ "$defs": {"Run": {"x-python-import": {"module": "subprocess", "name": "run"}}},
+ }
+ with pytest.raises(UnsafeSchemaError) as exc_info:
+ SchemaToModelFactory.make_from_json_schema(schema, "Innocent")
+ message = str(exc_info.value)
+ assert "$defs" in message
+ assert "Run" in message
+
+ def test_multiple_violations_are_all_reported(self) -> None:
+ """All violations in a single schema are surfaced — incident response needs the full picture."""
+ schema: dict[str, Any] = {
+ "title": "Innocent",
+ "type": "object",
+ "properties": {
+ "a": {"x-python-import": {"module": "subprocess", "name": "run"}},
+ "b": {"x-python-type": "os.system"},
+ },
+ }
+ with pytest.raises(UnsafeSchemaError) as exc_info:
+ SchemaToModelFactory.make_from_json_schema(schema, "Innocent")
+ message = str(exc_info.value)
+ assert "x-python-import" in message
+ assert "x-python-type" in message
+
+ # ---- Layer 2: __import__ allowlist (defense-in-depth at the exec boundary) ----
+
+ def test_exec_blocks_import_subprocess(self) -> None:
+ """Even bypassing Layer 1, the restricted __import__ blocks `import subprocess`."""
+ malicious_source = "from pydantic import BaseModel\nimport subprocess\nclass Innocent(BaseModel):\n name: str = 'ok'\n"
+ with pytest.raises(ImportError, match="subprocess"):
+ SchemaToModelFactory._exec_and_extract_class(malicious_source, "Innocent") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ def test_exec_blocks_direct_dunder_import_call(self) -> None:
+ """A direct __import__('subprocess') call is also blocked."""
+ malicious_source = "from pydantic import BaseModel\nclass Innocent(BaseModel):\n name: str = 'ok'\n_leak = __import__('subprocess')\n"
+ with pytest.raises(ImportError, match="subprocess"):
+ SchemaToModelFactory._exec_and_extract_class(malicious_source, "Innocent") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ @pytest.mark.parametrize(
+ ("import_line", "expected_module"),
+ [
+ ("from os import system", "os"),
+ ("from socket import socket", "socket"),
+ ("import shutil", "shutil"),
+ ("from urllib.request import urlopen", "urllib"),
+ ],
+ )
+ def test_exec_blocks_dangerous_imports(self, import_line: str, expected_module: str) -> None:
+ """A range of dangerous stdlib imports is blocked by the allowlist."""
+ malicious_source = f"from pydantic import BaseModel\n{import_line}\nclass Innocent(BaseModel):\n name: str = 'ok'\n"
+ with pytest.raises(ImportError, match=expected_module):
+ SchemaToModelFactory._exec_and_extract_class(malicious_source, "Innocent") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ @pytest.mark.parametrize(
+ "import_line",
+ [
+ "from pydantic import BaseModel",
+ "from typing import Optional",
+ "from typing_extensions import override",
+ "from datetime import datetime",
+ "from decimal import Decimal",
+ "from uuid import UUID",
+ "from enum import Enum",
+ "from __future__ import annotations",
+ "from collections.abc import Mapping",
+ "import re",
+ ],
+ )
+ def test_exec_allowlisted_imports_succeed(self, import_line: str) -> None:
+ """Allowlisted imports required by datamodel-code-generator output continue to work."""
+ source = f"{import_line}\nfrom pydantic import BaseModel\nclass Innocent(BaseModel):\n name: str = 'ok'\n"
+ result_class = SchemaToModelFactory._exec_and_extract_class(source, "Innocent") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ assert result_class.__name__ == "Innocent"
diff --git a/tests/unit/pipelex/cogt/test_data.py b/tests/unit/pipelex/cogt/test_data.py
new file mode 100644
index 000000000..97c9ef2a7
--- /dev/null
+++ b/tests/unit/pipelex/cogt/test_data.py
@@ -0,0 +1,13 @@
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class ExceptionTestData:
+ # (topic, category, expected_is_retryable)
+ CATEGORY_RETRYABLE_CASES: ClassVar[list[tuple[str, InferenceErrorCategory, bool]]] = [
+ ("transient_is_retryable", InferenceErrorCategory.TRANSIENT, True),
+ ("configuration_not_retryable", InferenceErrorCategory.CONFIGURATION, False),
+ ("content_not_retryable", InferenceErrorCategory.CONTENT, False),
+ ("capacity_not_retryable", InferenceErrorCategory.CAPACITY, False),
+ ]
diff --git a/tests/unit/pipelex/cogt/test_error_classification.py b/tests/unit/pipelex/cogt/test_error_classification.py
new file mode 100644
index 000000000..22ffbdb99
--- /dev/null
+++ b/tests/unit/pipelex/cogt/test_error_classification.py
@@ -0,0 +1,169 @@
+"""Tests for the error classification helpers used by inference workers."""
+
+from typing import ClassVar
+
+import pytest
+
+from pipelex.cogt.inference.error_classification import (
+ is_content_policy_violation,
+ is_quota_exhaustion_anthropic,
+ is_quota_exhaustion_aws,
+ is_quota_exhaustion_gateway,
+ is_quota_exhaustion_google,
+ is_quota_exhaustion_mistral,
+ is_quota_exhaustion_openai,
+)
+
+
+class _TestCases:
+ # (topic, error_message, expected_result)
+ OPENAI_QUOTA_CASES: ClassVar[list[tuple[str, str, bool]]] = [
+ ("insufficient_quota", "Error: insufficient_quota - you have exceeded your billing limit", True),
+ ("exceeded_quota", "You exceeded your current quota, please check your plan", True),
+ ("generic_rate_limit", "Rate limit exceeded. Please retry after 20s", False),
+ ("empty_message", "", False),
+ ("unrelated_error", "Invalid API key provided", False),
+ ("case_insensitive", "Insufficient_Quota error occurred", True),
+ ]
+
+ ANTHROPIC_QUOTA_CASES: ClassVar[list[tuple[str, str, bool]]] = [
+ ("quota_in_message", "Your account quota has been exceeded", True),
+ ("billing_limit", "You have reached your billing limit", True),
+ ("billing_issue", "There is a billing issue with your account", True),
+ ("credit_balance", "Your credit balance is too low to make this request", True),
+ ("out_of_credits", "Account is out of credits", True),
+ ("insufficient_credit", "Insufficient credit on this API key", True),
+ ("generic_rate_limit", "rate_limit_error: Number of request tokens has exceeded your per-minute limit", False),
+ ("empty_message", "", False),
+ ("unrelated_error", "Invalid API key", False),
+ ("billing_word_alone_not_matched", "Please check your billing dashboard", False),
+ ("credit_card_declined_not_matched", "Your credit card was declined, please update payment method", False),
+ ("credit_account_not_matched", "We will credit your account within 24 hours", False),
+ ("no_credit_check_not_matched", "No credit check required", False),
+ ("case_insensitive", "Account Billing Issue detected", True),
+ ]
+
+ GOOGLE_QUOTA_CASES: ClassVar[list[tuple[str, str, bool]]] = [
+ ("quota_exceeded", "Quota exceeded for aiplatform.googleapis.com", True),
+ ("resource_exhausted", "Resource has been exhausted (e.g. check quota)", True),
+ ("billing_quota_exceeded", "Billing quota exceeded for project foo", True),
+ ("billing_limit_reached", "Billing limit reached on this account", True),
+ ("billing_exceeded", "Billing exceeded for the current period", True),
+ ("billing_account_not_active", "Billing account is not active", True),
+ ("billing_account_disabled", "billing account is disabled for project", True),
+ ("generic_rate_limit", "Rate limit exceeded per minute", False),
+ ("empty_message", "", False),
+ ("unrelated_error", "Model not found", False),
+ ("billing_setup_not_quota", "billing project not configured", False),
+ ("billing_enable_not_quota", "Please enable billing for this project to use this API", False),
+ ("billing_api_not_enabled_not_quota", "billing API not enabled", False),
+ ]
+
+ # (topic, error_message, status_code, expected_result)
+ MISTRAL_QUOTA_CASES: ClassVar[list[tuple[str, str, int, bool]]] = [
+ ("payment_required_402", "Payment required", 402, True),
+ ("quota_429", "Quota exceeded for your account", 429, True),
+ ("billing_429", "Billing limit reached", 429, True),
+ ("generic_rate_limit_429", "Rate limit exceeded. Please retry after 20s", 429, False),
+ ("server_error_500", "Internal server error", 500, False),
+ ("not_found_404", "Model not found", 404, False),
+ ("payment_required_402_any_message", "anything at all", 402, True),
+ ]
+
+ AWS_QUOTA_CASES: ClassVar[list[tuple[str, str, bool]]] = [
+ ("quota_exceeded", "Service quota exceeded for this account", True),
+ ("limit_exceeded", "Token limit exceeded for model", True),
+ ("generic_throttle", "Rate exceeded", False),
+ ("empty_message", "", False),
+ ("unrelated_error", "Access denied", False),
+ ]
+
+ # (topic, error_message, status_code, expected_result)
+ GATEWAY_QUOTA_CASES: ClassVar[list[tuple[str, str, int, bool]]] = [
+ ("payment_required_402", "Payment required", 402, True),
+ ("quota_429", "Your quota has been exhausted", 429, True),
+ ("billing_429", "Billing limit exceeded", 429, True),
+ ("insufficient_credits_429", "Insufficient credits", 429, True),
+ ("insufficient_quota_429", "insufficient_quota for this account", 429, True),
+ ("insufficient_funds_429", "Insufficient funds in your account", 429, True),
+ ("insufficient_balance_429", "Insufficient balance to complete request", 429, True),
+ ("credits_exhausted_429", "Credits exhausted on this key", 429, True),
+ ("insufficient_permissions_429_not_quota", "Insufficient permissions to use this model", 429, False),
+ ("insufficient_role_429_not_quota", "Insufficient role for the requested action", 429, False),
+ ("insufficient_access_429_not_quota", "Insufficient access privileges", 429, False),
+ ("generic_rate_limit_429", "Rate limit exceeded. Please retry", 429, False),
+ ("server_error_500", "Internal server error", 500, False),
+ ("payment_required_402_any_message", "random text", 402, True),
+ ]
+
+ CONTENT_POLICY_CASES: ClassVar[list[tuple[str, str, bool]]] = [
+ ("content_policy", "Your request was rejected due to content_policy_violation", True),
+ ("blocked_by_safety", "The response was blocked by safety systems", True),
+ ("content_filter", "content_filter triggered for this request", True),
+ ("safety_system", "Rejected by safety system", True),
+ ("safety_filter", "Blocked by safety filter", True),
+ ("generic_bad_request", "Invalid parameter: temperature must be between 0 and 2", False),
+ ("empty_message", "", False),
+ ("safety_word_alone_not_matched", "Please ensure thread safety when using this", False),
+ ("case_insensitive", "Content_Policy violation detected", True),
+ ]
+
+
+class TestErrorClassification:
+ """Tests for quota detection and content policy classification helpers."""
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "expected"),
+ _TestCases.OPENAI_QUOTA_CASES,
+ )
+ def test_is_quota_exhaustion_openai(self, _topic: str, error_message: str, expected: bool) -> None:
+ """Discriminate OpenAI quota exhaustion from generic rate limiting."""
+ assert is_quota_exhaustion_openai(error_message) == expected
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "expected"),
+ _TestCases.ANTHROPIC_QUOTA_CASES,
+ )
+ def test_is_quota_exhaustion_anthropic(self, _topic: str, error_message: str, expected: bool) -> None:
+ """Discriminate Anthropic quota exhaustion from generic rate limiting."""
+ assert is_quota_exhaustion_anthropic(error_message) == expected
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "expected"),
+ _TestCases.CONTENT_POLICY_CASES,
+ )
+ def test_is_content_policy_violation(self, _topic: str, error_message: str, expected: bool) -> None:
+ """Detect content policy and safety filter violations."""
+ assert is_content_policy_violation(error_message) == expected
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "expected"),
+ _TestCases.GOOGLE_QUOTA_CASES,
+ )
+ def test_is_quota_exhaustion_google(self, _topic: str, error_message: str, expected: bool) -> None:
+ """Discriminate Google quota exhaustion from generic rate limiting."""
+ assert is_quota_exhaustion_google(error_message) == expected
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "status_code", "expected"),
+ _TestCases.MISTRAL_QUOTA_CASES,
+ )
+ def test_is_quota_exhaustion_mistral(self, _topic: str, error_message: str, status_code: int, expected: bool) -> None:
+ """Discriminate Mistral quota exhaustion from generic rate limiting."""
+ assert is_quota_exhaustion_mistral(error_message, status_code) == expected
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "expected"),
+ _TestCases.AWS_QUOTA_CASES,
+ )
+ def test_is_quota_exhaustion_aws(self, _topic: str, error_message: str, expected: bool) -> None:
+ """Discriminate AWS quota exhaustion from generic throttling."""
+ assert is_quota_exhaustion_aws(error_message) == expected
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "status_code", "expected"),
+ _TestCases.GATEWAY_QUOTA_CASES,
+ )
+ def test_is_quota_exhaustion_gateway(self, _topic: str, error_message: str, status_code: int, expected: bool) -> None:
+ """Discriminate Gateway quota exhaustion from generic rate limiting."""
+ assert is_quota_exhaustion_gateway(error_message, status_code) == expected
diff --git a/tests/unit/pipelex/cogt/test_exceptions.py b/tests/unit/pipelex/cogt/test_exceptions.py
new file mode 100644
index 000000000..e312a87cd
--- /dev/null
+++ b/tests/unit/pipelex/cogt/test_exceptions.py
@@ -0,0 +1,181 @@
+import json
+from dataclasses import FrozenInstanceError
+
+import pytest
+
+from pipelex.base_exceptions import ErrorReport, PipelexError
+from pipelex.cogt.exceptions import (
+ CogtError,
+ InferenceBackendCredentialsError,
+ InferenceBackendCredentialsErrorType,
+ InferenceErrorCategory,
+ LLMCapabilityError,
+ LLMCompletionError,
+ LLMModelNotFoundError,
+)
+from tests.unit.pipelex.cogt.test_data import ExceptionTestData
+
+
+class TestErrorCategoryInfrastructure:
+ """Tests for the InferenceErrorCategory enum, CogtError fields, and to_error_report()."""
+
+ # --- InferenceErrorCategory enum ---
+
+ @pytest.mark.parametrize(
+ ("_topic", "category", "expected_retryable"),
+ ExceptionTestData.CATEGORY_RETRYABLE_CASES,
+ )
+ def test_is_retryable(
+ self,
+ _topic: str,
+ category: InferenceErrorCategory,
+ expected_retryable: bool,
+ ) -> None:
+ """is_retryable returns True only for TRANSIENT."""
+ assert category.is_retryable == expected_retryable
+
+ # --- error_category field ---
+
+ def test_cogt_error_default_category_is_none(self) -> None:
+ """CogtError with no category argument defaults to None."""
+ err = CogtError("something broke")
+ assert err.error_category is None
+
+ def test_cogt_error_instance_category_override(self) -> None:
+ """CogtError accepts error_category at construction time."""
+ err = CogtError("transient issue", error_category=InferenceErrorCategory.TRANSIENT)
+ assert err.error_category is InferenceErrorCategory.TRANSIENT
+
+ def test_subclass_class_level_category(self) -> None:
+ """Subclasses with class-level category inherit it without passing to __init__."""
+ err = LLMCapabilityError("model does not support vision")
+ assert err.error_category is InferenceErrorCategory.CONFIGURATION
+
+ def test_subclass_with_custom_init_preserves_category(self) -> None:
+ """Subclasses with custom __init__ still get their class-level category."""
+ err = LLMModelNotFoundError(message="not found", model_handle="gpt-4")
+ assert err.error_category is InferenceErrorCategory.CONFIGURATION
+ assert err.model_handle == "gpt-4"
+
+ def test_instance_override_beats_class_default(self) -> None:
+ """Per-instance category overrides the class-level default."""
+ err = LLMCompletionError("timeout", error_category=InferenceErrorCategory.TRANSIENT)
+ assert err.error_category is InferenceErrorCategory.TRANSIENT
+
+ # --- user_action field ---
+
+ def test_user_action_default_none(self) -> None:
+ """user_action defaults to None."""
+ err = CogtError("something broke")
+ assert err.user_action is None
+
+ def test_user_action_instance_set(self) -> None:
+ """user_action can be set at construction."""
+ err = CogtError("bad config", user_action="Check your API key")
+ assert err.user_action == "Check your API key"
+
+ def test_user_action_class_level(self) -> None:
+ """Subclass with class-level user_action inherits it."""
+ err = InferenceBackendCredentialsError(
+ credentials_error_type=InferenceBackendCredentialsErrorType.VAR_NOT_FOUND,
+ backend_name="openai",
+ message="key missing",
+ key_name="OPENAI_API_KEY",
+ )
+ assert err.user_action == "Check that the required API key environment variable is set"
+
+ # --- to_error_report() ---
+
+ def test_to_error_report_pipelex_error_minimal(self) -> None:
+ """PipelexError.to_error_report returns an ErrorReport with error_type and message."""
+ err = PipelexError("basic error")
+ report = err.to_error_report()
+ assert report.error_type == "PipelexError"
+ assert report.message == "basic error"
+ assert report.error_category is None
+ assert report.retryable is None
+
+ def test_to_error_report_cogt_error_no_category(self) -> None:
+ """CogtError report has None for optional fields when not set."""
+ err = CogtError("plain error")
+ report = err.to_error_report()
+ assert report.error_type == "CogtError"
+ assert report.message == "plain error"
+ assert report.error_category is None
+ assert report.retryable is None
+ assert report.user_action is None
+ assert report.model is None
+ assert report.provider is None
+
+ def test_to_error_report_with_category(self) -> None:
+ """CogtError report includes category and retryable when set."""
+ err = CogtError("oops", error_category=InferenceErrorCategory.CONTENT)
+ report = err.to_error_report()
+ assert report.error_category == "content"
+ assert report.retryable is False
+
+ def test_to_error_report_with_all_cogt_fields(self) -> None:
+ """Report includes user_action when set."""
+ err = CogtError(
+ "oops",
+ error_category=InferenceErrorCategory.TRANSIENT,
+ user_action="Retry in a moment",
+ )
+ report = err.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
+ assert report.user_action == "Retry in a moment"
+
+ def test_to_error_report_with_model_handle(self) -> None:
+ """Report includes model field for errors with model_handle."""
+ err = LLMModelNotFoundError(message="missing", model_handle="gpt-4o")
+ report = err.to_error_report()
+ assert report.model == "gpt-4o"
+ assert report.error_category == "configuration"
+
+ def test_to_error_report_with_provider(self) -> None:
+ """Report includes provider field for errors with backend_name."""
+ err = InferenceBackendCredentialsError(
+ credentials_error_type=InferenceBackendCredentialsErrorType.VAR_NOT_FOUND,
+ backend_name="openai",
+ message="key missing",
+ key_name="OPENAI_API_KEY",
+ )
+ report = err.to_error_report()
+ assert report.provider == "openai"
+ assert report.error_category == "configuration"
+
+ def test_to_error_report_to_dict_omits_none(self) -> None:
+ """to_dict() omits None-valued fields."""
+ err = CogtError("plain error")
+ report_dict = err.to_error_report().to_dict()
+ assert "error_category" not in report_dict
+ assert "retryable" not in report_dict
+ assert "user_action" not in report_dict
+ assert "model" not in report_dict
+ assert "provider" not in report_dict
+ assert report_dict == {"error_type": "CogtError", "message": "plain error"}
+
+ def test_to_error_report_to_dict_json_serializable(self) -> None:
+ """to_dict() output can be serialized to JSON."""
+ err = LLMModelNotFoundError(message="not found", model_handle="gpt-4o")
+ report_dict = err.to_error_report().to_dict()
+ serialized = json.dumps(report_dict)
+ deserialized = json.loads(serialized)
+ assert deserialized == report_dict
+
+ def test_error_report_is_frozen(self) -> None:
+ """ErrorReport is immutable."""
+ report = ErrorReport(error_type="CogtError", message="test")
+ with pytest.raises(FrozenInstanceError):
+ report.message = "mutated" # type: ignore[misc]
+
+ def test_from_exc_chain_preserved(self) -> None:
+ """Exception cause chain is preserved with from exc."""
+ original = ValueError("original error")
+ msg = "wrapped"
+ with pytest.raises(CogtError) as exc_info:
+ raise CogtError(msg, error_category=InferenceErrorCategory.TRANSIENT) from original
+ assert exc_info.value.__cause__ is original
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
diff --git a/tests/unit/pipelex/core/concepts/concept_factory/data.py b/tests/unit/pipelex/core/concepts/concept_factory/data.py
index b986379bb..d248de96f 100644
--- a/tests/unit/pipelex/core/concepts/concept_factory/data.py
+++ b/tests/unit/pipelex/core/concepts/concept_factory/data.py
@@ -102,7 +102,7 @@ class TestCases:
domain_code="my_domain",
code="DomainConcept",
description="A concept from same domain",
- structure_class_name="DomainConcept",
+ structure_class_name="my_domain__DomainConcept",
refines="native.Text",
),
),
@@ -128,7 +128,7 @@ class TestCases:
domain_code="my_domain",
code="PersonConcept",
description="A person with structured data",
- structure_class_name="PersonConcept",
+ structure_class_name="my_domain__PersonConcept",
refines=None,
),
),
@@ -142,7 +142,7 @@ class TestCases:
domain_code="my_domain",
code="ConceptWithRefines",
description="A concept with refines",
- structure_class_name="ConceptWithRefines",
+ structure_class_name="my_domain__ConceptWithRefines",
refines="native.Image",
),
),
diff --git a/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_concept_refs.py b/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_concept_refs.py
index b09815666..48fe0ff20 100644
--- a/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_concept_refs.py
+++ b/tests/unit/pipelex/core/concepts/structure_generation/test_structure_generator_concept_refs.py
@@ -61,7 +61,7 @@ class TestStructureGeneratorConceptRefs:
class Invoice(StructuredContent):
"""Generated Invoice class"""
- customer: "Customer" = Field(..., description="The customer for this invoice")
+ customer: "myapp__Customer" = Field(..., description="The customer for this invoice")
total: float = Field(..., description="Invoice total")
'''
)
@@ -96,7 +96,7 @@ class Invoice(StructuredContent):
class Invoice(StructuredContent):
"""Generated Invoice class"""
- line_items: List["LineItem"] = Field(..., description="List of line items")
+ line_items: List["myapp__LineItem"] = Field(..., description="List of line items")
total: float = Field(..., description="Invoice total")
'''
)
@@ -130,7 +130,7 @@ class Invoice(StructuredContent):
class CategoryNode(StructuredContent):
"""Generated CategoryNode class"""
- parent: Optional["Category"] = Field(default=None, description="Optional parent reference")
+ parent: Optional["myapp__Category"] = Field(default=None, description="Optional parent reference")
name: str = Field(..., description="Category name")
'''
)
@@ -160,7 +160,7 @@ class CategoryNode(StructuredContent):
class Parent(StructuredContent):
"""Generated Parent class"""
- children: Optional[List["Category"]] = Field(default=None, description="Optional child categories")
+ children: Optional[List["myapp__Category"]] = Field(default=None, description="Optional child categories")
'''
)
assert generated_code == expected_code
@@ -195,8 +195,8 @@ class Parent(StructuredContent):
class Order(StructuredContent):
"""Generated Order class"""
- customer: "Customer" = Field(..., description="Customer from CRM domain")
- products: List["Product"] = Field(..., description="Products from inventory domain")
+ customer: "crm__Customer" = Field(..., description="Customer from CRM domain")
+ products: List["inventory__Product"] = Field(..., description="Products from inventory domain")
'''
)
assert generated_code == expected_code
@@ -224,7 +224,7 @@ class Order(StructuredContent):
class Container(StructuredContent):
"""Generated Container class"""
- item: "SomeItem" = Field(..., description="An item reference")
+ item: "unknown_domain__SomeItem" = Field(..., description="An item reference")
'''
)
assert generated_code == expected_code
@@ -284,9 +284,9 @@ class ComplexEntity(StructuredContent):
id: int = Field(..., description="Unique identifier")
name: str = Field(..., description="Name")
- owner: Optional["User"] = Field(default=None, description="Owner reference")
+ owner: Optional["myapp__User"] = Field(default=None, description="Owner reference")
tags: Optional[List[str]] = Field(default=None, description="Tags")
- related_items: Optional[List["Item"]] = Field(default=None, description="Related items")
+ related_items: Optional[List["myapp__Item"]] = Field(default=None, description="Related items")
metadata: Optional[Dict[str, str]] = Field(default=None, description="Metadata")
'''
)
diff --git a/tests/unit/pipelex/core/pipes/test_execution_data_coverage.py b/tests/unit/pipelex/core/pipes/test_execution_data_coverage.py
new file mode 100644
index 000000000..0834cf9f7
--- /dev/null
+++ b/tests/unit/pipelex/core/pipes/test_execution_data_coverage.py
@@ -0,0 +1,115 @@
+"""Every concrete Pipe must call _register_execution_data in both live and dry run.
+
+Prevents regressions where a new pipe is added (or an existing one is refactored)
+and forgets to emit execution data. Without this, the graph tracer has no runtime
+info for that pipe and the sidepanel shows empty fields.
+
+The check reads the source of each pipe's run entry point and any `self.method(...)`
+it calls (transitively), then looks for `self._register_execution_data(`. Static,
+fast, no fixture setup.
+"""
+
+import importlib
+import inspect
+import pkgutil
+import re
+
+import pytest
+
+from pipelex import pipe_controllers, pipe_operators
+from pipelex.core.pipes.pipe_abstract import PipeAbstract
+
+REGISTER_CALL = "self._register_execution_data("
+SELF_CALL_RE = re.compile(r"self\.(\w+)\(")
+
+LIVE_RUN_METHOD_NAMES = ("_live_run_operator_pipe", "_live_run_controller_pipe")
+DRY_RUN_METHOD_NAMES = ("_dry_run_operator_pipe", "_dry_run_controller_pipe")
+
+
+def _discover_pipe_classes() -> list[type[PipeAbstract]]:
+ """Import every pipe module and return concrete leaf PipeAbstract subclasses."""
+ for package in (pipe_operators, pipe_controllers):
+ for module_info in pkgutil.walk_packages(package.__path__, prefix=f"{package.__name__}."):
+ importlib.import_module(module_info.name)
+
+ seen: set[type[PipeAbstract]] = set()
+
+ def walk(cls: type[PipeAbstract]) -> None:
+ for sub in cls.__subclasses__():
+ if sub not in seen:
+ seen.add(sub)
+ walk(sub)
+
+ walk(PipeAbstract) # type: ignore[type-abstract]
+
+ leaves = [cls for cls in seen if not cls.__module__.endswith((".pipe_operator", ".pipe_controller", ".pipe_abstract"))]
+ return sorted(leaves, key=lambda c: c.__name__)
+
+
+def _collect_reachable_source(cls: type[PipeAbstract], method_name: str) -> str:
+ """Return the source of `method_name` concatenated with the source of every
+ `self.<method>(...)` it calls, transitively. Handles cycles via a visited set.
+
+ This captures the common pattern where an entry point dispatches to helpers:
+ e.g. PipeCompose._live_run_operator_pipe → _run_template_mode → _register_execution_data.
+ """
+ visited: set[str] = set()
+ stack: list[str] = [method_name]
+ chunks: list[str] = []
+
+ while stack:
+ name = stack.pop()
+ if name in visited:
+ continue
+ visited.add(name)
+
+ method = getattr(cls, name, None)
+ if method is None:
+ continue
+ try:
+ source = inspect.getsource(method)
+ except (OSError, TypeError):
+ continue
+ chunks.append(source)
+
+ for match in SELF_CALL_RE.finditer(source):
+ callee = match.group(1)
+ if callee not in visited:
+ stack.append(callee)
+
+ return "\n".join(chunks)
+
+
+_PIPE_CLASSES = _discover_pipe_classes()
+
+
+class TestPipeExecutionDataCoverage:
+ """Every pipe must register execution data in both live and dry run."""
+
+ def test_pipe_classes_were_discovered(self) -> None:
+ assert _PIPE_CLASSES, "No Pipe subclasses discovered — check pipe_operators / pipe_controllers imports."
+
+ @pytest.mark.parametrize("pipe_class", _PIPE_CLASSES, ids=lambda c: c.__name__)
+ def test_live_run_calls_register_execution_data(self, pipe_class: type[PipeAbstract]) -> None:
+ entry = next((name for name in LIVE_RUN_METHOD_NAMES if getattr(pipe_class, name, None) is not None), None)
+ assert entry is not None, f"{pipe_class.__name__} must override one of {LIVE_RUN_METHOD_NAMES}"
+
+ source = _collect_reachable_source(pipe_class, entry)
+ assert REGISTER_CALL in source, (
+ f"{pipe_class.__name__}.{entry} does not call self._register_execution_data(...) "
+ f"(directly or via another self-method). Every pipe must register runtime data "
+ f"(rendered prompts, resolved models, composed fields, etc.) so the graph sidepanel "
+ f"has something to display."
+ )
+
+ @pytest.mark.parametrize("pipe_class", _PIPE_CLASSES, ids=lambda c: c.__name__)
+ def test_dry_run_calls_register_execution_data(self, pipe_class: type[PipeAbstract]) -> None:
+ entry = next((name for name in DRY_RUN_METHOD_NAMES if getattr(pipe_class, name, None) is not None), None)
+ assert entry is not None, f"{pipe_class.__name__} must override one of {DRY_RUN_METHOD_NAMES}"
+
+ source = _collect_reachable_source(pipe_class, entry)
+ assert REGISTER_CALL in source, (
+ f"{pipe_class.__name__}.{entry} does not call self._register_execution_data(...) "
+ f"(directly or via another self-method). Dry runs must also register execution data "
+ f"for complete dry-run graph visualization."
+ )
diff --git a/tests/unit/pipelex/core/pipes/test_pipe_output_prepare_for_temporal.py b/tests/unit/pipelex/core/pipes/test_pipe_output_prepare_for_temporal.py
new file mode 100644
index 000000000..9d0a6c2db
--- /dev/null
+++ b/tests/unit/pipelex/core/pipes/test_pipe_output_prepare_for_temporal.py
@@ -0,0 +1,63 @@
+from pytest_mock import MockerFixture
+
+from pipelex.core.concepts.concept import Concept
+from pipelex.core.domains.domain import SpecialDomain
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.text_content import TextContent
+
+
+def _make_populated_pipe_output() -> PipeOutput:
+ working_memory = WorkingMemory()
+ working_memory.root["greeting"] = Stuff(
+ stuff_code="test",
+ stuff_name="greeting",
+ concept=Concept(
+ code="Text",
+ domain_code=SpecialDomain.NATIVE,
+ description="Plain text",
+ structure_class_name="TextContent",
+ ),
+ content=TextContent(text="Hello!"),
+ )
+ return PipeOutput(working_memory=working_memory)
+
+
+class TestPipeOutputPrepareForTemporal:
+ def test_no_crate_returns_self_unchanged(self) -> None:
+ pipe_output = _make_populated_pipe_output()
+
+ result = pipe_output.prepare_for_temporal(library_crate=None)
+
+ assert result is pipe_output
+ assert result.working_memory_raw is None
+ assert "greeting" in result.working_memory.root
+
+ def test_with_crate_dehydrates(self, mocker: MockerFixture) -> None:
+ from pipelex.libraries.library_crate import LibraryCrate # noqa: PLC0415
+
+ fake_crate = mocker.MagicMock(spec=LibraryCrate)
+ pipe_output = _make_populated_pipe_output()
+
+ result = pipe_output.prepare_for_temporal(library_crate=fake_crate)
+
+ assert result is not pipe_output
+ assert result.working_memory_raw is not None
+ assert "greeting" in result.working_memory_raw.get("root", {})
+ # Original is unchanged.
+ assert pipe_output.working_memory_raw is None
+ assert "greeting" in pipe_output.working_memory.root
+ # The dehydrated copy has an empty WorkingMemory.
+ assert not result.working_memory.root
+
+ def test_with_crate_empty_wm_returns_self(self, mocker: MockerFixture) -> None:
+ from pipelex.libraries.library_crate import LibraryCrate # noqa: PLC0415
+
+ fake_crate = mocker.MagicMock(spec=LibraryCrate)
+ pipe_output = PipeOutput(working_memory=WorkingMemory())
+
+ result = pipe_output.prepare_for_temporal(library_crate=fake_crate)
+
+ assert result is pipe_output
+ assert result.working_memory_raw is None
diff --git a/tests/unit/pipelex/graph/test_graph_output_generation.py b/tests/unit/pipelex/graph/test_graph_output_generation.py
new file mode 100644
index 000000000..8c77b1ca3
--- /dev/null
+++ b/tests/unit/pipelex/graph/test_graph_output_generation.py
@@ -0,0 +1,192 @@
+"""Tests for graph output generation from an assembled GraphSpec.
+
+Validates that generate_graph_outputs and save_graph_outputs_to_dir produce
+the expected output files (GraphSpec JSON, Mermaid MMD, Mermaid HTML, ReactFlow HTML).
+"""
+
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+
+import pytest
+
+from pipelex.graph.graph_config import GraphConfig, GraphsInclusionConfig
+from pipelex.graph.graph_factory import generate_graph_outputs, save_graph_outputs_to_dir
+from pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, IOSpec, NodeIOSpec, NodeKind, NodeSpec, NodeStatus, PipelineRef, TimingSpec
+from tests.unit.pipelex.graph.conftest import make_graph_config
+
+_T0 = datetime(2025, 7, 1, 12, 0, 0, tzinfo=timezone.utc)
+
+
+def _make_sequence_graphspec() -> GraphSpec:
+ """Build a minimal sequence GraphSpec: parent + 2 children with CONTAINS and DATA edges."""
+ parent_id = "node_seq"
+ child_1_id = "node_step_one"
+ child_2_id = "node_step_two"
+
+ nodes = [
+ NodeSpec(
+ node_id=parent_id,
+ kind=NodeKind.CONTROLLER,
+ pipe_code="my_sequence",
+ pipe_type="PipeSequence",
+ status=NodeStatus.SUCCEEDED,
+ timing=TimingSpec(started_at=_T0, ended_at=_T0 + timedelta(seconds=10)),
+ ),
+ NodeSpec(
+ node_id=child_1_id,
+ kind=NodeKind.OPERATOR,
+ pipe_code="step_one",
+ pipe_type="PipeLLM",
+ status=NodeStatus.SUCCEEDED,
+ timing=TimingSpec(started_at=_T0 + timedelta(seconds=1), ended_at=_T0 + timedelta(seconds=4)),
+ node_io=NodeIOSpec(
+ outputs=[IOSpec(name="output_text", digest="digest_a")],
+ ),
+ ),
+ NodeSpec(
+ node_id=child_2_id,
+ kind=NodeKind.OPERATOR,
+ pipe_code="step_two",
+ pipe_type="PipeLLM",
+ status=NodeStatus.SUCCEEDED,
+ timing=TimingSpec(started_at=_T0 + timedelta(seconds=5), ended_at=_T0 + timedelta(seconds=9)),
+ node_io=NodeIOSpec(
+ inputs=[IOSpec(name="input_text", digest="digest_a")],
+ outputs=[IOSpec(name="output_text", digest="digest_b")],
+ ),
+ ),
+ ]
+ edges = [
+ EdgeSpec(
+ edge_id="edge_0",
+ source=parent_id,
+ target=child_1_id,
+ kind=EdgeKind.CONTAINS,
+ ),
+ EdgeSpec(
+ edge_id="edge_1",
+ source=parent_id,
+ target=child_2_id,
+ kind=EdgeKind.CONTAINS,
+ ),
+ EdgeSpec(
+ edge_id="edge_2",
+ source=child_1_id,
+ target=child_2_id,
+ kind=EdgeKind.DATA,
+ label="input_text",
+ ),
+ ]
+ return GraphSpec(
+ graph_id="output_test_graph",
+ created_at=_T0,
+ pipeline_ref=PipelineRef(domain="test_domain", main_pipe="my_sequence"),
+ nodes=nodes,
+ edges=edges,
+ )
+
+
+def _enable_all_outputs(graph_config: GraphConfig) -> GraphConfig:
+ """Return a copy of the config with all graph output types enabled."""
+ return graph_config.model_copy(
+ update={
+ "graphs_inclusion": GraphsInclusionConfig(
+ graphspec_json=True,
+ mermaidflow_mmd=True,
+ mermaidflow_html=True,
+ reactflow_html=True,
+ ),
+ },
+ )
+
+
[email protected](loop_scope="class")
+class TestGraphOutputGeneration:
+ """Verify that graph output generation produces valid content and files."""
+
+ async def test_generate_all_outputs(self) -> None:
+ """All four output types are generated when enabled."""
+ graph_spec = _make_sequence_graphspec()
+ config = _enable_all_outputs(make_graph_config())
+
+ outputs = await generate_graph_outputs(graph_spec, config, pipe_code="my_sequence")
+
+ assert outputs.graphspec_json is not None
+ assert "output_test_graph" in outputs.graphspec_json
+ assert "my_sequence" in outputs.graphspec_json
+
+ assert outputs.mermaidflow_mmd is not None
+ assert len(outputs.mermaidflow_mmd) > 0
+
+ assert outputs.mermaidflow_html is not None
+ assert "<!DOCTYPE html>" in outputs.mermaidflow_html or "<html" in outputs.mermaidflow_html
+
+ assert outputs.reactflow_html is not None
+ assert "<!DOCTYPE html>" in outputs.reactflow_html
+
+ async def test_save_outputs_to_dir(self, tmp_path: Path) -> None:
+ """save_graph_outputs_to_dir writes all output files to disk."""
+ graph_spec = _make_sequence_graphspec()
+ config = _enable_all_outputs(make_graph_config())
+
+ outputs = await generate_graph_outputs(graph_spec, config, pipe_code="my_sequence")
+ output_dir = tmp_path / "graph_output"
+ saved_files = save_graph_outputs_to_dir(outputs, output_dir)
+
+ assert output_dir.is_dir()
+ assert "graphspec_json" in saved_files
+ assert "mermaidflow_mmd" in saved_files
+ assert "mermaidflow_html" in saved_files
+ assert "reactflow_html" in saved_files
+
+ assert (output_dir / "graphspec.json").is_file()
+ assert (output_dir / "mermaidflow.mmd").is_file()
+ assert (output_dir / "mermaidflow.html").is_file()
+ assert (output_dir / "reactflow.html").is_file()
+
+ # Verify saved content is non-empty
+ for file_path in saved_files.values():
+ assert file_path.stat().st_size > 0
+
+ async def test_graphspec_json_only(self) -> None:
+ """Only GraphSpec JSON is generated when other outputs are disabled."""
+ graph_spec = _make_sequence_graphspec()
+ config = make_graph_config()
+ config = config.model_copy(
+ update={
+ "graphs_inclusion": GraphsInclusionConfig(
+ graphspec_json=True,
+ mermaidflow_mmd=False,
+ mermaidflow_html=False,
+ reactflow_html=False,
+ ),
+ },
+ )
+
+ outputs = await generate_graph_outputs(graph_spec, config, pipe_code="my_sequence")
+
+ assert outputs.graphspec_json is not None
+ assert outputs.mermaidflow_mmd is None
+ assert outputs.mermaidflow_html is None
+ assert outputs.reactflow_html is None
+
+ async def test_mermaid_mmd_contains_node_labels(self) -> None:
+ """Mermaid MMD output references the pipe codes from the graph."""
+ graph_spec = _make_sequence_graphspec()
+ config = make_graph_config()
+ config = config.model_copy(
+ update={
+ "graphs_inclusion": GraphsInclusionConfig(
+ graphspec_json=False,
+ mermaidflow_mmd=True,
+ mermaidflow_html=False,
+ reactflow_html=False,
+ ),
+ },
+ )
+
+ outputs = await generate_graph_outputs(graph_spec, config, pipe_code="my_sequence")
+
+ assert outputs.mermaidflow_mmd is not None
+ assert "step_one" in outputs.mermaidflow_mmd
+ assert "step_two" in outputs.mermaidflow_mmd
diff --git a/tests/unit/pipelex/graph/test_graph_tracer_event_emission.py b/tests/unit/pipelex/graph/test_graph_tracer_event_emission.py
new file mode 100644
index 000000000..d9810773f
--- /dev/null
+++ b/tests/unit/pipelex/graph/test_graph_tracer_event_emission.py
@@ -0,0 +1,546 @@
+"""Unit tests for GraphTracer event emission (dual-write mode).
+
+Validates that when an EventLogProtocol is provided, GraphTracer emits the
+correct trace events as a side effect alongside its existing in-memory accumulation.
+"""
+
+from datetime import datetime, timedelta, timezone
+
+from pipelex.graph.graph_tracer import GraphTracer
+from pipelex.graph.graphspec import EdgeKind, IOSpec, NodeKind, NodeStatus
+from pipelex.tracing.in_memory_event_log import InMemoryEventLog
+from pipelex.tracing.trace_events import (
+ BatchAggregateEvent,
+ BatchItemEvent,
+ ControllerOutputEvent,
+ EdgeEvent,
+ ParallelCombineEvent,
+ PipeEndErrorEvent,
+ PipeEndSuccessEvent,
+ PipeStartEvent,
+)
+from tests.unit.pipelex.graph.conftest import make_defaulted_data_inclusion_config
+
+
+class TestGraphTracerEventEmission:
+ """Tests for GraphTracer dual-write event emission."""
+
+ PIPELINE_RUN_ID = "run_evt_001"
+ WORKFLOW_ID = "wf_test_abc"
+ GRAPH_ID = "graph-emit-test"
+
+ def _make_tracer_with_event_log(
+ self,
+ workflow_id: str | None = None,
+ ) -> tuple[GraphTracer, InMemoryEventLog]:
+ """Create a GraphTracer configured with an InMemoryEventLog."""
+ event_log = InMemoryEventLog()
+ tracer = GraphTracer()
+ tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=workflow_id or self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+ return tracer, event_log
+
+ def test_pipe_start_emits_event(self) -> None:
+ """PipeStartEvent is emitted on on_pipe_start."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ node_id, _child_ctx = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="test_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ pipe_starts = [evt for evt in events if isinstance(evt, PipeStartEvent)]
+ assert len(pipe_starts) == 1
+
+ evt = pipe_starts[0]
+ assert evt.node_id == node_id
+ assert evt.pipe_code == "test_pipe"
+ assert evt.pipe_type == "PipeLLM"
+ assert evt.node_kind == NodeKind.OPERATOR
+ assert evt.parent_node_id is None
+ assert evt.workflow_id == self.WORKFLOW_ID
+ assert evt.pipeline_run_id == self.PIPELINE_RUN_ID
+
+ def test_pipe_start_with_parent_emits_contains_edge(self) -> None:
+ """CONTAINS EdgeEvent is emitted when parent_node_id is present."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ parent_id, child_ctx = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="parent_pipe",
+ pipe_type="PipeSequence",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ )
+
+ child_id, _ = tracer.on_pipe_start(
+ graph_context=child_ctx,
+ pipe_code="child_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(milliseconds=10),
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ edge_events = [evt for evt in events if isinstance(evt, EdgeEvent)]
+ contains_edges = [evt for evt in edge_events if evt.edge_kind == EdgeKind.CONTAINS]
+ assert len(contains_edges) == 1
+ assert contains_edges[0].source_node_id == parent_id
+ assert contains_edges[0].target_node_id == child_id
+
+ def test_pipe_end_success_emits_event(self) -> None:
+ """PipeEndSuccessEvent is emitted on on_pipe_end_success."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ node_id, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="test_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+
+ ended_at = started_at + timedelta(milliseconds=100)
+ output_spec = IOSpec(name="output", concept="Text", content_type="text/plain", digest="abc123")
+ tracer.on_pipe_end_success(
+ node_id=node_id,
+ ended_at=ended_at,
+ metrics={"tokens": 150.0},
+ output_spec=output_spec,
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ success_events = [evt for evt in events if isinstance(evt, PipeEndSuccessEvent)]
+ assert len(success_events) == 1
+
+ evt = success_events[0]
+ assert evt.node_id == node_id
+ assert evt.ended_at == ended_at
+ assert evt.metrics == {"tokens": 150.0}
+ assert evt.output_spec is not None
+ assert evt.output_spec.digest == "abc123"
+
+ def test_pipe_end_error_emits_event(self) -> None:
+ """PipeEndErrorEvent is emitted on on_pipe_end_error."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ node_id, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="test_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+
+ ended_at = started_at + timedelta(milliseconds=50)
+ tracer.on_pipe_end_error(
+ node_id=node_id,
+ ended_at=ended_at,
+ error_type="ValueError",
+ error_message="Something went wrong",
+ error_stack="traceback line 1\nline 2",
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ error_events = [evt for evt in events if isinstance(evt, PipeEndErrorEvent)]
+ assert len(error_events) == 1
+
+ evt = error_events[0]
+ assert evt.node_id == node_id
+ assert evt.ended_at == ended_at
+ assert evt.error.error_type == "ValueError"
+ assert evt.error.message == "Something went wrong"
+
+ def test_add_edge_emits_event(self) -> None:
+ """EdgeEvent is emitted on add_edge."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ node_a, _ctx_a = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="pipe_a",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+ node_b, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="pipe_b",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+
+ tracer.add_edge(
+ source_node_id=node_a,
+ target_node_id=node_b,
+ edge_kind=EdgeKind.DATA,
+ label="output",
+ source_stuff_digest="digest_a",
+ target_stuff_digest="digest_b",
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ edge_events = [evt for evt in events if isinstance(evt, EdgeEvent) and evt.edge_kind == EdgeKind.DATA]
+ assert len(edge_events) == 1
+
+ evt = edge_events[0]
+ assert evt.source_node_id == node_a
+ assert evt.target_node_id == node_b
+ assert evt.label == "output"
+ assert evt.source_stuff_digest == "digest_a"
+ assert evt.target_stuff_digest == "digest_b"
+
+ def test_register_controller_output_emits_event(self) -> None:
+ """ControllerOutputEvent is emitted on register_controller_output."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ node_id, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="parallel_pipe",
+ pipe_type="PipeParallel",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ )
+
+ output_spec = IOSpec(name="branch_output", concept="Text", content_type="text/plain", digest="branch_abc")
+ tracer.register_controller_output(node_id=node_id, output_spec=output_spec)
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ ctrl_events = [evt for evt in events if isinstance(evt, ControllerOutputEvent)]
+ assert len(ctrl_events) == 1
+
+ evt = ctrl_events[0]
+ assert evt.node_id == node_id
+ assert evt.output_spec.digest == "branch_abc"
+
+ def test_register_batch_item_extraction_emits_event(self) -> None:
+ """BatchItemEvent is emitted on register_batch_item_extraction."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ _context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ tracer.register_batch_item_extraction(
+ list_stuff_code="list_digest",
+ item_stuff_code="item_0_digest",
+ item_index=0,
+ batch_controller_node_id="graph-emit-test:node_0",
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ batch_events = [evt for evt in events if isinstance(evt, BatchItemEvent)]
+ assert len(batch_events) == 1
+
+ evt = batch_events[0]
+ assert evt.list_stuff_code == "list_digest"
+ assert evt.item_stuff_code == "item_0_digest"
+ assert evt.item_index == 0
+ assert evt.batch_controller_node_id == "graph-emit-test:node_0"
+
+ def test_register_batch_aggregation_emits_event(self) -> None:
+ """BatchAggregateEvent is emitted on register_batch_aggregation."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ _context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ tracer.register_batch_aggregation(
+ output_list_stuff_code="out_list_digest",
+ item_stuff_code="item_0_digest",
+ item_index=0,
+ batch_controller_node_id="graph-emit-test:node_1",
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ agg_events = [evt for evt in events if isinstance(evt, BatchAggregateEvent)]
+ assert len(agg_events) == 1
+
+ evt = agg_events[0]
+ assert evt.output_list_stuff_code == "out_list_digest"
+ assert evt.item_stuff_code == "item_0_digest"
+ assert evt.item_index == 0
+ assert evt.batch_controller_node_id == "graph-emit-test:node_1"
+
+ def test_register_parallel_combine_emits_event_with_snapshotted_producers(self) -> None:
+ """ParallelCombineEvent is emitted with correct snapshotted producer IDs."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+
+ # Create controller node
+ ctrl_id, ctrl_ctx = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="parallel_ctrl",
+ pipe_type="PipeParallel",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ )
+
+ # Create branch nodes and end them with outputs
+ branch_a_id, _ = tracer.on_pipe_start(
+ graph_context=ctrl_ctx,
+ pipe_code="branch_a",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+ tracer.on_pipe_end_success(
+ node_id=branch_a_id,
+ ended_at=started_at + timedelta(milliseconds=50),
+ output_spec=IOSpec(name="out_a", concept="Text", content_type="text/plain", digest="digest_a"),
+ )
+
+ branch_b_id, _ = tracer.on_pipe_start(
+ graph_context=ctrl_ctx,
+ pipe_code="branch_b",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+ tracer.on_pipe_end_success(
+ node_id=branch_b_id,
+ ended_at=started_at + timedelta(milliseconds=60),
+ output_spec=IOSpec(name="out_b", concept="Text", content_type="text/plain", digest="digest_b"),
+ )
+
+ # Register parallel combine — should snapshot producers
+ tracer.register_parallel_combine(
+ combined_stuff_code="combined_digest",
+ branch_stuff_codes=["digest_a", "digest_b"],
+ parallel_controller_node_id=ctrl_id,
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ combine_events = [evt for evt in events if isinstance(evt, ParallelCombineEvent)]
+ assert len(combine_events) == 1
+
+ evt = combine_events[0]
+ assert evt.combined_stuff_code == "combined_digest"
+ assert evt.parallel_controller_node_id == ctrl_id
+ assert evt.branch_stuff_codes == ["digest_a", "digest_b"]
+ # Snapshotted producers should be the branch nodes, not the controller
+ assert ("digest_a", branch_a_id) in evt.branch_producer_node_ids
+ assert ("digest_b", branch_b_id) in evt.branch_producer_node_ids
+
+ def test_no_event_log_works_as_before(self) -> None:
+ """GraphTracer without event_log (direct mode) works exactly as before."""
+ tracer = GraphTracer()
+ context = tracer.setup(
+ graph_id="direct-mode-test",
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ node_id, _child_ctx = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="test_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+
+ assert node_id == "direct-mode-test:node_0"
+
+ tracer.on_pipe_end_success(
+ node_id=node_id,
+ ended_at=started_at + timedelta(milliseconds=100),
+ )
+
+ graph_spec = tracer.teardown()
+ assert graph_spec is not None
+ assert len(graph_spec.nodes) == 1
+ assert graph_spec.nodes[0].status == NodeStatus.SUCCEEDED
+
+ def test_node_id_includes_workflow_id_in_temporal_mode(self) -> None:
+ """Node IDs include workflow_id segment when not in direct mode."""
+ tracer, event_log = self._make_tracer_with_event_log(workflow_id="wf_run_123")
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id="wf_run_123",
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ node_id, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="test_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+
+ assert "wf_run_123" in node_id
+ assert node_id == f"{self.GRAPH_ID}:wf_run_123:node_0"
+
+ def test_edge_id_includes_workflow_id_in_temporal_mode(self) -> None:
+ """Edge IDs include workflow_id segment when not in direct mode."""
+ tracer, event_log = self._make_tracer_with_event_log(workflow_id="wf_run_456")
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id="wf_run_456",
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ node_a, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="pipe_a",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+ node_b, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="pipe_b",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+
+ tracer.add_edge(
+ source_node_id=node_a,
+ target_node_id=node_b,
+ edge_kind=EdgeKind.DATA,
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ edge_events = [evt for evt in events if isinstance(evt, EdgeEvent) and evt.edge_kind == EdgeKind.DATA]
+ assert len(edge_events) == 1
+ assert "wf_run_456" in edge_events[0].edge_id
+
+ def test_event_sequence_is_monotonic(self) -> None:
+ """Events have monotonically increasing sequence numbers."""
+ tracer, event_log = self._make_tracer_with_event_log()
+
+ context = tracer.setup(
+ graph_id=self.GRAPH_ID,
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ for index in range(5):
+ node_id, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code=f"pipe_{index}",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(milliseconds=index),
+ )
+ tracer.on_pipe_end_success(
+ node_id=node_id,
+ ended_at=started_at + timedelta(milliseconds=index + 50),
+ )
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ sequences = [evt.sequence for evt in events]
+ assert sequences == sorted(sequences)
+ assert len(set(sequences)) == len(sequences) # All unique
+
+ def test_direct_mode_node_id_unchanged(self) -> None:
+ """In direct mode (no workflow_id), node ID format is unchanged."""
+ tracer = GraphTracer()
+ context = tracer.setup(
+ graph_id="direct-graph",
+ data_inclusion=make_defaulted_data_inclusion_config(),
+ )
+
+ started_at = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+ node_id, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="test_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at,
+ )
+
+ assert node_id == "direct-graph:node_0"
diff --git a/tests/unit/pipelex/graph/test_graphspec_roundtrip.py b/tests/unit/pipelex/graph/test_graphspec_roundtrip.py
index 75b3bdd9c..a41f11ccd 100644
--- a/tests/unit/pipelex/graph/test_graphspec_roundtrip.py
+++ b/tests/unit/pipelex/graph/test_graphspec_roundtrip.py
@@ -59,6 +59,8 @@ class TestGraphSpecRoundtrip:
assert restored.nodes[0].kind == NodeKind.OPERATOR
assert restored.nodes[0].status == NodeStatus.SUCCEEDED
assert len(restored.edges) == 0
+ assert '"format": "mthds"' in json_str
+ assert restored.meta["format"] == "mthds"
def test_roundtrip_complex_graph(self) -> None:
"""Test round-trip for a complex graph with multiple nodes, edges, and timing."""
@@ -161,7 +163,7 @@ class TestGraphSpecRoundtrip:
assert restored.edges[0].label == "step 1"
# Verify meta
- assert restored.meta == {"run_mode": "live"}
+ assert restored.meta == {"run_mode": "live", "format": "mthds"}
def test_roundtrip_with_error_spec(self) -> None:
"""Test round-trip for a graph with a failed node containing error info."""
diff --git a/tests/unit/pipelex/graph/test_graphspec_validation.py b/tests/unit/pipelex/graph/test_graphspec_validation.py
index f250cbfbd..788fa2151 100644
--- a/tests/unit/pipelex/graph/test_graphspec_validation.py
+++ b/tests/unit/pipelex/graph/test_graphspec_validation.py
@@ -308,6 +308,39 @@ class TestGraphSpecValidation:
# Should not raise
validate_graphspec(graph)
+ def test_meta_format_injected_when_missing(self) -> None:
+ graph = GraphSpec(
+ graph_id="test_graph",
+ created_at=ValidGraphData.CREATED_AT,
+ pipeline_ref=PipelineRef(),
+ nodes=[],
+ edges=[],
+ )
+ assert graph.meta["format"] == "mthds"
+
+ def test_meta_format_preserved_when_correct(self) -> None:
+ graph = GraphSpec(
+ graph_id="test_graph",
+ created_at=ValidGraphData.CREATED_AT,
+ pipeline_ref=PipelineRef(),
+ nodes=[],
+ edges=[],
+ meta={"format": "mthds", "extra": "value"},
+ )
+ assert graph.meta["format"] == "mthds"
+ assert graph.meta["extra"] == "value"
+
+ def test_meta_format_rejected_when_wrong(self) -> None:
+ with pytest.raises(ValidationError):
+ GraphSpec(
+ graph_id="test_graph",
+ created_at=ValidGraphData.CREATED_AT,
+ pipeline_ref=PipelineRef(),
+ nodes=[],
+ edges=[],
+ meta={"format": "other"},
+ )
+
def test_reject_extra_fields_on_load(self) -> None:
"""Test that extra/unknown fields in JSON are rejected."""
graph = GraphSpec(
diff --git a/tests/unit/pipelex/libraries/test_library_crate.py b/tests/unit/pipelex/libraries/test_library_crate.py
new file mode 100644
index 000000000..9bbf7a439
--- /dev/null
+++ b/tests/unit/pipelex/libraries/test_library_crate.py
@@ -0,0 +1,137 @@
+import pytest
+
+from pipelex.libraries.concept.exceptions import ConceptLibraryError
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.libraries.library_crate_factory import LibraryCrateFactory
+from pipelex.libraries.pipe.exceptions import PipeLibraryError
+from tests.unit.pipelex.libraries.test_library_crate_data import BlueprintSamples
+
+
+class TestLibraryCrate:
+ """Tests for LibraryCrate model and LibraryCrateFactory."""
+
+ def test_json_round_trip(self):
+ """LibraryCrate serializes to JSON and deserializes back identically."""
+ crate = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE],
+ )
+ json_str = crate.model_dump_json()
+ restored = LibraryCrate.model_validate_json(json_str)
+ assert restored.concepts == crate.concepts
+ assert restored.pipes == crate.pipes
+ assert restored.domains == crate.domains
+ assert restored.source_map == crate.source_map
+ assert restored.fingerprint == crate.fingerprint
+
+ def test_merge_same_domain(self):
+ """Two bundles for the same domain merge their concepts and pipes."""
+ crate = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE, BlueprintSamples.SCORING_EXTRA_BUNDLE],
+ )
+ assert "scoring.WeightedScore" in crate.concepts
+ assert "scoring.ScoreBreakdown" in crate.concepts
+ assert "scoring.compute_score" in crate.pipes
+ assert "scoring.explain_score" in crate.pipes
+ # Domain metadata: first-write-wins
+ assert "scoring" in crate.domains
+ assert crate.domains["scoring"].description == "Scoring domain"
+
+ def test_merge_across_domains(self):
+ """Bundles with different domains produce refs qualified with respective domains."""
+ crate = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE, BlueprintSamples.ANALYTICS_BUNDLE],
+ )
+ assert "scoring.WeightedScore" in crate.concepts
+ assert "analytics.Metric" in crate.concepts
+ assert "scoring.compute_score" in crate.pipes
+ assert "analytics.compute_metric" in crate.pipes
+ assert "scoring" in crate.domains
+ assert "analytics" in crate.domains
+
+ def test_string_concept_preserved(self):
+ """String-described concepts are preserved as strings (not converted to ConceptBlueprint)."""
+ crate = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.STRING_CONCEPT_BUNDLE],
+ )
+ concept = crate.concepts["simple.MyConcept"]
+ assert isinstance(concept, str)
+ assert concept == "A simple concept described as a string"
+
+ def test_concept_collision_cross_file_raises(self):
+ """Two bundles from different files declaring the same concept_ref raise ConceptLibraryError."""
+ with pytest.raises(ConceptLibraryError, match=r"two different bundle files"):
+ LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE, BlueprintSamples.SCORING_DUPLICATE_CONCEPT_BUNDLE],
+ )
+
+ def test_concept_collision_same_file_raises(self):
+ """Two bundles from the same file declaring the same concept_ref raise ConceptLibraryError."""
+ with pytest.raises(ConceptLibraryError, match=r"declared twice in the same bundle file"):
+ LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE, BlueprintSamples.SCORING_SAME_FILE_DUPLICATE_CONCEPT_BUNDLE],
+ )
+
+ def test_pipe_collision_cross_file_raises(self):
+ """Two bundles from different files declaring the same pipe_ref raise PipeLibraryError."""
+ with pytest.raises(PipeLibraryError, match=r"two different bundle files"):
+ LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE, BlueprintSamples.SCORING_DUPLICATE_PIPE_BUNDLE],
+ )
+
+ def test_pipe_collision_same_file_raises(self):
+ """Two bundles from the same file declaring the same pipe_ref raise PipeLibraryError."""
+ with pytest.raises(PipeLibraryError, match=r"declared twice in the same bundle file"):
+ LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE, BlueprintSamples.SCORING_SAME_FILE_DUPLICATE_PIPE_BUNDLE],
+ )
+
+ def test_fingerprint_determinism(self):
+ """Same blueprints produce the same fingerprint; different blueprints produce different ones."""
+ crate_a = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE],
+ )
+ crate_b = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE],
+ )
+ assert crate_a.fingerprint == crate_b.fingerprint
+ assert crate_a.fingerprint != ""
+
+ crate_c = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.ANALYTICS_BUNDLE],
+ )
+ assert crate_a.fingerprint != crate_c.fingerprint
+
+ def test_empty_blueprints(self):
+ """Bundle with no concepts and no pipes produces empty dicts and a valid fingerprint."""
+ crate = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.EMPTY_BUNDLE],
+ )
+ assert crate.concepts == {}
+ assert crate.pipes == {}
+ assert "empty_domain" in crate.domains
+ assert crate.fingerprint != ""
+
+ def test_source_map_populated(self):
+ """Source map tracks concept_ref and pipe_ref to their source file paths."""
+ crate = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.SCORING_BUNDLE],
+ )
+ assert crate.source_map["scoring.WeightedScore"] == "/fake/scoring.mthds"
+ assert crate.source_map["scoring.compute_score"] == "/fake/scoring.mthds"
+
+ def test_none_source_excluded_from_source_map(self):
+ """Bundles with source=None produce valid crates but no source_map entries."""
+ crate = LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.NONE_SOURCE_BUNDLE],
+ )
+ assert "nosource.Item" in crate.concepts
+ assert "nosource.process_item" in crate.pipes
+ assert "nosource.Item" not in crate.source_map
+ assert "nosource.process_item" not in crate.source_map
+
+ def test_concept_collision_none_source_raises(self):
+ """Two bundles with source=None declaring the same concept_ref raise ConceptLibraryError."""
+ with pytest.raises(ConceptLibraryError, match=r"two different bundle files"):
+ LibraryCrateFactory.make_from_blueprints(
+ blueprints=[BlueprintSamples.NONE_SOURCE_BUNDLE, BlueprintSamples.NONE_SOURCE_DUPLICATE_CONCEPT_BUNDLE],
+ )
diff --git a/tests/unit/pipelex/libraries/test_library_crate_accumulation.py b/tests/unit/pipelex/libraries/test_library_crate_accumulation.py
new file mode 100644
index 000000000..fc978000f
--- /dev/null
+++ b/tests/unit/pipelex/libraries/test_library_crate_accumulation.py
@@ -0,0 +1,152 @@
+"""Tests for LibraryManager blueprint accumulation, get_crate(), and fingerprint idempotency."""
+
+from typing import cast
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.hub import get_library_manager, set_current_library, teardown_current_library
+from pipelex.libraries.library import Library
+from pipelex.libraries.library_manager import LibraryManager
+from tests.unit.pipelex.libraries.test_library_crate_data import BlueprintSamples
+
+
+class TestLibraryCrateAccumulation:
+ """Tests for blueprint accumulation and get_crate() on LibraryManager."""
+
+ def test_get_crate_returns_none_for_unknown_library_id(self):
+ """get_crate() returns None for a library_id that was never opened."""
+ manager = LibraryManager()
+ result = manager.get_crate(library_id="nonexistent")
+ assert result is None
+
+ def test_get_crate_returns_none_when_no_blueprints_loaded(self):
+ """get_crate() returns None when a library exists but no blueprints were loaded."""
+ manager = LibraryManager()
+ manager.open_library(library_id="test-lib")
+ result = manager.get_crate(library_id="test-lib")
+ assert result is None
+
+ def test_get_crate_builds_crate_from_accumulated_blueprints(self):
+ """get_crate() builds a crate containing all concepts and pipes from loaded blueprints.
+
+ Uses the hub's LibraryManager + set_current_library so that load_from_blueprints()
+ can resolve concepts through the hub during pipe construction.
+ """
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ try:
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.SCORING_BUNDLE])
+
+ crate = library_manager.get_crate(library_id=library_id)
+ assert crate is not None
+ assert "scoring.WeightedScore" in crate.concepts
+ assert "scoring.compute_score" in crate.pipes
+ assert crate.fingerprint != ""
+ finally:
+ library_manager.teardown(library_id=library_id)
+ teardown_current_library()
+
+ def test_multiple_load_from_blueprints_all_included_in_crate(self):
+ """Blueprints from multiple load_from_blueprints() calls are all included in the crate."""
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ try:
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.SCORING_BUNDLE])
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.ANALYTICS_BUNDLE])
+
+ crate = library_manager.get_crate(library_id=library_id)
+ assert crate is not None
+ # From first load
+ assert "scoring.WeightedScore" in crate.concepts
+ assert "scoring.compute_score" in crate.pipes
+ # From second load
+ assert "analytics.Metric" in crate.concepts
+ assert "analytics.compute_metric" in crate.pipes
+ finally:
+ library_manager.teardown(library_id=library_id)
+ teardown_current_library()
+
+ def test_load_from_crate_idempotent_on_same_fingerprint(self):
+ """load_from_crate() returns empty list on second call with the same fingerprint."""
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ try:
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.SCORING_BUNDLE])
+
+ crate = library_manager.get_crate(library_id=library_id)
+ assert crate is not None
+
+ # Open a second library to load the crate into
+ second_library_id = "idempotency-test-lib"
+ library_manager.open_library(library_id=second_library_id)
+
+ # First load should return pipes
+ pipes_first = library_manager.load_from_crate(library_id=second_library_id, crate=crate)
+ assert len(pipes_first) > 0
+
+ # Second load with same fingerprint should be skipped (empty list)
+ pipes_second = library_manager.load_from_crate(library_id=second_library_id, crate=crate)
+ assert pipes_second == []
+
+ library_manager.teardown(library_id=second_library_id)
+ finally:
+ library_manager.teardown(library_id=library_id)
+ teardown_current_library()
+
+ def test_teardown_clears_blueprints_for_library_id(self):
+ """teardown(library_id) clears accumulated blueprints for that library_id."""
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ try:
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.SCORING_BUNDLE])
+
+ # Verify crate exists before teardown
+ crate = library_manager.get_crate(library_id=library_id)
+ assert crate is not None
+
+ # Teardown
+ library_manager.teardown(library_id=library_id)
+
+ # After teardown, get_crate should return None (library_id no longer known)
+ result = library_manager.get_crate(library_id=library_id)
+ assert result is None
+ finally:
+ teardown_current_library()
+
+ def test_failed_load_from_crate_does_not_cache_fingerprint(self, mocker: MockerFixture):
+ """A failed load_from_crate() must not cache the fingerprint, allowing retries."""
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ try:
+ library_manager.load_from_blueprints(library_id=library_id, blueprints=[BlueprintSamples.SCORING_BUNDLE])
+
+ crate = library_manager.get_crate(library_id=library_id)
+ assert crate is not None
+
+ # Open a target library to load the crate into
+ target_library_id = "failure-retry-test-lib"
+ library_manager.open_library(library_id=target_library_id)
+
+ try:
+ # Make validate_library raise to simulate a load failure (Library is a frozen
+ # Pydantic model, so we patch the class method rather than the instance)
+ mocker.patch.object(Library, "validate_library", side_effect=RuntimeError("simulated failure"))
+
+ with pytest.raises(RuntimeError, match="simulated failure"):
+ library_manager.load_from_crate(library_id=target_library_id, crate=crate)
+
+ # Fingerprint must NOT be cached — a retry should attempt loading again
+ concrete_manager = cast("LibraryManager", library_manager)
+ loaded_set = concrete_manager._loaded_fingerprints.get(target_library_id, set()) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ assert crate.fingerprint not in loaded_set
+ finally:
+ library_manager.teardown(library_id=target_library_id)
+ finally:
+ library_manager.teardown(library_id=library_id)
+ teardown_current_library()
diff --git a/tests/unit/pipelex/libraries/test_library_crate_data.py b/tests/unit/pipelex/libraries/test_library_crate_data.py
new file mode 100644
index 000000000..36b2cdae9
--- /dev/null
+++ b/tests/unit/pipelex/libraries/test_library_crate_data.py
@@ -0,0 +1,148 @@
+from typing import ClassVar
+
+from pipelex.core.bundles.pipelex_bundle_blueprint import PipelexBundleBlueprint
+from pipelex.core.concepts.concept_blueprint import ConceptBlueprint
+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint
+
+
+class BlueprintSamples:
+ """Sample PipelexBundleBlueprint objects for testing LibraryCrate."""
+
+ SCORING_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source="/fake/scoring.mthds",
+ domain="scoring",
+ description="Scoring domain",
+ system_prompt="You are a scoring assistant.",
+ concept={
+ "WeightedScore": ConceptBlueprint(description="A weighted score"),
+ },
+ pipe={
+ "compute_score": PipeLLMBlueprint(
+ description="Compute a weighted score",
+ output="WeightedScore",
+ prompt="Compute the score.",
+ ),
+ },
+ )
+
+ SCORING_EXTRA_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source="/fake/scoring_extra.mthds",
+ domain="scoring",
+ description="Scoring domain (extra)",
+ concept={
+ "ScoreBreakdown": ConceptBlueprint(description="A breakdown of a score"),
+ },
+ pipe={
+ "explain_score": PipeLLMBlueprint(
+ description="Explain a score breakdown",
+ output="ScoreBreakdown",
+ prompt="Explain the score.",
+ ),
+ },
+ )
+
+ ANALYTICS_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source="/fake/analytics.mthds",
+ domain="analytics",
+ description="Analytics domain",
+ concept={
+ "Metric": ConceptBlueprint(description="A metric"),
+ },
+ pipe={
+ "compute_metric": PipeLLMBlueprint(
+ description="Compute a metric",
+ output="Metric",
+ prompt="Compute the metric.",
+ ),
+ },
+ )
+
+ STRING_CONCEPT_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source="/fake/simple.mthds",
+ domain="simple",
+ description="Simple domain",
+ concept={
+ "MyConcept": "A simple concept described as a string",
+ },
+ )
+
+ EMPTY_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source="/fake/empty.mthds",
+ domain="empty_domain",
+ description="Empty domain",
+ )
+
+ # For collision tests: same domain + same concept code as SCORING_BUNDLE
+ SCORING_DUPLICATE_CONCEPT_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source="/fake/scoring_dup.mthds",
+ domain="scoring",
+ description="Scoring domain duplicate",
+ concept={
+ "WeightedScore": ConceptBlueprint(description="Duplicate weighted score"),
+ },
+ )
+
+ # For collision tests: same domain + same pipe code as SCORING_BUNDLE
+ SCORING_DUPLICATE_PIPE_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source="/fake/scoring_dup.mthds",
+ domain="scoring",
+ description="Scoring domain duplicate",
+ pipe={
+ "compute_score": PipeLLMBlueprint(
+ description="Duplicate compute score",
+ output="Text",
+ prompt="Duplicate.",
+ ),
+ },
+ )
+
+ # For same-file collision tests: same source as SCORING_BUNDLE + same concept
+ SCORING_SAME_FILE_DUPLICATE_CONCEPT_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source="/fake/scoring.mthds",
+ domain="scoring",
+ description="Scoring domain same-file duplicate",
+ concept={
+ "WeightedScore": ConceptBlueprint(description="Same-file duplicate weighted score"),
+ },
+ )
+
+ # For same-file collision tests: same source as SCORING_BUNDLE + same pipe
+ SCORING_SAME_FILE_DUPLICATE_PIPE_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source="/fake/scoring.mthds",
+ domain="scoring",
+ description="Scoring domain same-file duplicate",
+ pipe={
+ "compute_score": PipeLLMBlueprint(
+ description="Same-file duplicate compute score",
+ output="Text",
+ prompt="Same-file duplicate.",
+ ),
+ },
+ )
+
+ # For source=None edge case tests
+ NONE_SOURCE_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source=None,
+ domain="nosource",
+ description="Domain with no source file",
+ concept={
+ "Item": ConceptBlueprint(description="An item"),
+ },
+ pipe={
+ "process_item": PipeLLMBlueprint(
+ description="Process an item",
+ output="Item",
+ prompt="Process the item.",
+ ),
+ },
+ )
+
+ # For source=None collision tests: same domain + concept as NONE_SOURCE_BUNDLE
+ NONE_SOURCE_DUPLICATE_CONCEPT_BUNDLE: ClassVar[PipelexBundleBlueprint] = PipelexBundleBlueprint(
+ source=None,
+ domain="nosource",
+ description="Domain with no source file (duplicate)",
+ concept={
+ "Item": ConceptBlueprint(description="Duplicate item"),
+ },
+ )
diff --git a/tests/unit/pipelex/pipe_operators/pipe_compose/test_composer_field_resolutions.py b/tests/unit/pipelex/pipe_operators/pipe_compose/test_composer_field_resolutions.py
new file mode 100644
index 000000000..867327f00
--- /dev/null
+++ b/tests/unit/pipelex/pipe_operators/pipe_compose/test_composer_field_resolutions.py
@@ -0,0 +1,95 @@
+"""Unit tests for StructuredContentComposer.field_resolutions.
+
+Verifies that the per-field resolution record built during composition correctly
+captures how each field was built: method for every field, plus the rendered
+Jinja2 string for template fields. Nested fields record only their method.
+"""
+
+from typing import Callable
+
+import pytest
+from pydantic import Field
+
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.core.stuffs.stuff_factory import StuffFactory
+from pipelex.hub import get_native_concept
+from pipelex.pipe_operators.compose.construct_blueprint import ConstructBlueprint, ConstructFieldMethod
+from pipelex.pipe_operators.compose.structured_content_composer import StructuredContentComposer
+
+
+class HeadquartersForResolutions(StructuredContent):
+ """Inner address used as the target of the NESTED-method test field."""
+
+ street: str = Field(description="Street address rendered from a runtime param")
+ city: str = Field(description="City pulled from working memory")
+
+
+class CustomerForResolutions(StructuredContent):
+ """Source object placed in working memory for FROM_VAR / TEMPLATE resolution."""
+
+ customer_name: str = Field(description="Customer name")
+
+
+class ReportForResolutions(StructuredContent):
+ """Output class exercising all four ConstructFieldMethod variants in one composition."""
+
+ title: str = Field(description="Fixed literal")
+ customer_name: str = Field(description="Pulled from working memory")
+ summary: str = Field(description="Rendered Jinja2 template")
+ headquarters: HeadquartersForResolutions = Field(description="Nested sub-construct")
+
+
[email protected](loop_scope="class")
+class TestComposerFieldResolutions:
+ @pytest.fixture
+ def working_memory_with_inputs(self, load_empty_library: Callable[[], None]) -> WorkingMemory:
+ load_empty_library()
+ customer = CustomerForResolutions(customer_name="Acme Corp")
+ working_memory = WorkingMemoryFactory.make_from_single_stuff(
+ stuff=StuffFactory.make_stuff(
+ concept=get_native_concept(NativeConceptCode.TEXT),
+ content=customer,
+ name="customer",
+ ),
+ )
+ working_memory.add_new_stuff(
+ name="city",
+ stuff=StuffFactory.make_stuff(
+ concept=get_native_concept(NativeConceptCode.TEXT),
+ content=CustomerForResolutions(customer_name="Paris"),
+ name="city",
+ ),
+ )
+ return working_memory
+
+ async def test_field_resolutions_cover_all_methods(self, working_memory_with_inputs: WorkingMemory):
+ blueprint = ConstructBlueprint.make_from_raw(
+ {
+ "title": "Annual Report",
+ "customer_name": {"from": "customer.customer_name"},
+ "summary": {"template": "Hello $customer.customer_name"},
+ "headquarters": {
+ "street": {"template": "$_street_name"},
+ "city": {"from": "city.customer_name"},
+ },
+ }
+ )
+
+ composer = StructuredContentComposer(
+ construct_blueprint=blueprint,
+ working_memory=working_memory_with_inputs,
+ output_class=ReportForResolutions,
+ runtime_params={"_street_name": "123 Main St"},
+ )
+ await composer.compose()
+
+ resolutions = composer.field_resolutions
+ assert set(resolutions.keys()) == {"title", "customer_name", "summary", "headquarters"}
+
+ assert resolutions["title"] == {"method": ConstructFieldMethod.FIXED}
+ assert resolutions["customer_name"] == {"method": ConstructFieldMethod.FROM_VAR}
+ assert resolutions["summary"] == {"method": ConstructFieldMethod.TEMPLATE, "rendered": "Hello Acme Corp"}
+ assert resolutions["headquarters"] == {"method": ConstructFieldMethod.NESTED}
diff --git a/tests/unit/pipelex/pipe_run/test_delivery_assignment.py b/tests/unit/pipelex/pipe_run/test_delivery_assignment.py
new file mode 100644
index 000000000..7d857ab46
--- /dev/null
+++ b/tests/unit/pipelex/pipe_run/test_delivery_assignment.py
@@ -0,0 +1,66 @@
+from pipelex.pipe_run.delivery_assignment import (
+ DeliveryAssignment,
+ DeliveryStatus,
+ StorageTarget,
+ WebhookTarget,
+)
+
+
+class TestDeliveryAssignment:
+ def test_default_delivery_assignment(self) -> None:
+ assignment = DeliveryAssignment()
+ assert assignment.storage is None
+ assert assignment.webhooks == []
+
+ def test_storage_only(self) -> None:
+ assignment = DeliveryAssignment(storage=StorageTarget())
+ assert assignment.storage is not None
+ assert assignment.storage.key_prefix is None
+
+ def test_storage_with_prefix(self) -> None:
+ assignment = DeliveryAssignment(storage=StorageTarget(key_prefix="results/"))
+ assert assignment.storage is not None
+ assert assignment.storage.key_prefix == "results/"
+
+ def test_webhooks_only(self) -> None:
+ assignment = DeliveryAssignment(
+ webhooks=[WebhookTarget(url="https://example.com/callback")],
+ )
+ assert len(assignment.webhooks) == 1
+ assert assignment.webhooks[0].url == "https://example.com/callback"
+
+ def test_webhook_with_headers_and_payload(self) -> None:
+ webhook = WebhookTarget(
+ url="https://example.com/callback",
+ headers={"X-Custom": "value"},
+ payload={"extra_field": "extra_value"},
+ )
+ assert webhook.headers == {"X-Custom": "value"}
+ assert webhook.payload == {"extra_field": "extra_value"}
+
+ def test_storage_and_webhooks(self) -> None:
+ assignment = DeliveryAssignment(
+ storage=StorageTarget(key_prefix="output/"),
+ webhooks=[
+ WebhookTarget(url="https://a.com"),
+ WebhookTarget(url="https://b.com"),
+ ],
+ )
+ assert assignment.storage is not None
+ assert len(assignment.webhooks) == 2
+
+ def test_serialization_roundtrip(self) -> None:
+ assignment = DeliveryAssignment(
+ storage=StorageTarget(key_prefix="test/"),
+ webhooks=[WebhookTarget(url="https://example.com", payload={"key": "val"})],
+ )
+ dumped = assignment.model_dump()
+ restored = DeliveryAssignment(**dumped)
+ assert restored.storage is not None
+ assert restored.storage.key_prefix == "test/"
+ assert restored.webhooks[0].url == "https://example.com"
+ assert restored.webhooks[0].payload == {"key": "val"}
+
+ def test_delivery_status_values(self) -> None:
+ assert DeliveryStatus.COMPLETED == "COMPLETED"
+ assert DeliveryStatus.FAILED == "FAILED"
diff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
new file mode 100644
index 000000000..08f4f37e9
--- /dev/null
+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py
@@ -0,0 +1,275 @@
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.pipe_run.delivery_assignment import (
+ DeliveryAssignment,
+ DeliveryStatus,
+ StorageTarget,
+ WebhookTarget,
+)
+from pipelex.pipe_run.delivery_executor import DeliveryExecutor
+from pipelex.pipe_run.exceptions import StorageDeliveryError, WebhookDeliveryError
+
+
[email protected](loop_scope="class")
+class TestDeliveryExecutor:
+ async def test_execute_storage_only(self, mocker: MockerFixture) -> None:
+ mock_storage = mocker.AsyncMock()
+ mock_storage.store = mocker.AsyncMock(return_value="pipelex-storage://test-key")
+ mock_storage.public_url = mocker.Mock(return_value="file:///tmp/results/plr-123")
+ mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)
+
+ mock_output = mocker.MagicMock()
+ mock_output.working_memory_raw = None
+ mock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}
+ mock_output.working_memory.get_optional_main_stuff.return_value = None
+ mock_output.graph_spec = None
+
+ executor = DeliveryExecutor()
+ assignment = DeliveryAssignment(storage=StorageTarget())
+
+ await executor.execute(
+ pipe_output=mock_output,
+ user_id="test-user",
+ pipeline_run_id="plr-123",
+ delivery_assignment=assignment,
+ status=DeliveryStatus.COMPLETED,
+ )
+
+ mock_storage.store.assert_called()
+ stored_keys = [call.kwargs["key"] for call in mock_storage.store.call_args_list]
+ assert any("test-user/plr-123/working_memory.json" in key for key in stored_keys)
+
+ async def test_execute_webhook_only(self, mocker: MockerFixture) -> None:
+ mock_client = mocker.AsyncMock()
+ mock_response = mocker.MagicMock()
+ mock_response.raise_for_status = mocker.Mock()
+ mock_client.__aenter__ = mocker.AsyncMock(return_value=mock_client)
+ mock_client.__aexit__ = mocker.AsyncMock(return_value=False)
+ mock_client.post = mocker.AsyncMock(return_value=mock_response)
+ mocker.patch("pipelex.pipe_run.delivery_executor.httpx.AsyncClient", return_value=mock_client)
+
+ executor = DeliveryExecutor()
+ assignment = DeliveryAssignment(
+ webhooks=[WebhookTarget(url="https://example.com/callback")],
+ )
+
+ await executor.execute(
+ pipe_output=None,
+ user_id="test-user",
+ pipeline_run_id="plr-456",
+ delivery_assignment=assignment,
+ status=DeliveryStatus.COMPLETED,
+ )
+
+ mock_client.post.assert_called_once()
+ call_kwargs = mock_client.post.call_args
+ payload = call_kwargs.kwargs["json"]
+ assert payload["pipeline_run_id"] == "plr-456"
+ assert payload["status"] == "COMPLETED"
+ assert "result_url" not in payload
+
+ async def test_execute_webhook_with_custom_payload(self, mocker: MockerFixture) -> None:
+ mock_client = mocker.AsyncMock()
+ mock_response = mocker.MagicMock()
+ mock_response.raise_for_status = mocker.Mock()
+ mock_client.__aenter__ = mocker.AsyncMock(return_value=mock_client)
+ mock_client.__aexit__ = mocker.AsyncMock(return_value=False)
+ mock_client.post = mocker.AsyncMock(return_value=mock_response)
+ mocker.patch("pipelex.pipe_run.delivery_executor.httpx.AsyncClient", return_value=mock_client)
+
+ executor = DeliveryExecutor()
+ assignment = DeliveryAssignment(
+ webhooks=[WebhookTarget(url="https://example.com", payload={"custom": "data"})],
+ )
+
+ await executor.execute(
+ pipe_output=None,
+ user_id="test-user",
+ pipeline_run_id="plr-789",
+ delivery_assignment=assignment,
+ status=DeliveryStatus.FAILED,
+ )
+
+ payload = mock_client.post.call_args.kwargs["json"]
+ assert payload["status"] == "FAILED"
+ assert payload["custom"] == "data"
+
+ async def test_execute_no_storage_on_failure(self, mocker: MockerFixture) -> None:
+ """Storage should be skipped when pipe_output is None (failure case)."""
+ mock_storage = mocker.AsyncMock()
+ mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)
+
+ executor = DeliveryExecutor()
+ assignment = DeliveryAssignment(storage=StorageTarget())
+
+ await executor.execute(
+ pipe_output=None,
+ user_id="test-user",
+ pipeline_run_id="plr-fail",
+ delivery_assignment=assignment,
+ status=DeliveryStatus.FAILED,
+ )
+
+ mock_storage.store.assert_not_called()
+
+ async def test_execute_empty_assignment(self) -> None:
+ """Empty assignment should do nothing without errors."""
+ executor = DeliveryExecutor()
+ assignment = DeliveryAssignment()
+
+ await executor.execute(
+ pipe_output=None,
+ user_id="test-user",
+ pipeline_run_id="plr-noop",
+ delivery_assignment=assignment,
+ status=DeliveryStatus.COMPLETED,
+ )
+
+ async def test_storage_failure_raises(self, mocker: MockerFixture) -> None:
+ mock_storage = mocker.AsyncMock()
+ mock_storage.store = mocker.AsyncMock(side_effect=Exception("S3 down"))
+ mocker.patch("pipelex.pipe_run.delivery_executor.get_storage_provider", return_value=mock_storage)
+
+ mock_output = mocker.MagicMock()
+ mock_output.working_memory_raw = None
+ mock_output.working_memory.smart_dump.return_value = {}
+ mock_output.working_memory.get_optional_main_stuff.return_value = None
+ mock_output.graph_spec = None
+
+ executor = DeliveryExecutor()
+ assignment = DeliveryAssignment(storage=StorageTarget())
+
+ with pytest.raises(StorageDeliveryError):
+ await executor.execute(
+ pipe_output=mock_output,
+ user_id="test-user",
+ pipeline_run_id="plr-err",
+ delivery_assignment=assignment,
+ status=DeliveryStatus.COMPLETED,
+ )
+
+ async def test_try_local_hydrate_stuff_returns_typed_for_builtin(self) -> None:
+ from pipelex.core.stuffs.text_content import TextContent # noqa: PLC0415
+ from pipelex.hub import get_class_registry # noqa: PLC0415
+
+ registry = get_class_registry()
+ if not registry.has_class(name="TextContent"):
+ registry.register_class(TextContent)
+
+ stuff_raw = {
+ "stuff_code": "test",
+ "stuff_name": "greeting",
+ "concept": {
+ "code": "Text",
+ "domain_code": "native",
+ "description": "Plain text",
+ "structure_class_name": "TextContent",
+ },
+ "content": {"text": "Hello!"},
+ }
+
+ result = DeliveryExecutor.try_local_hydrate_stuff(stuff_raw)
+
+ assert result is not None
+ assert isinstance(result.content, TextContent)
+ assert result.content.text == "Hello!"
+
+ async def test_try_local_hydrate_stuff_returns_none_for_missing_class(self, mocker: MockerFixture) -> None:
+ from pipelex import log as pipelex_log # noqa: PLC0415
+
+ warn_spy = mocker.spy(pipelex_log, "warning")
+
+ stuff_raw = {
+ "stuff_code": "test",
+ "stuff_name": "x",
+ "concept": {
+ "code": "Greeting",
+ "domain_code": "dynamic_test",
+ "description": "Dynamic concept",
+ "structure_class_name": "dynamic_test__Greeting",
+ },
+ "content": {"message": "hi"},
+ }
+
+ result = DeliveryExecutor.try_local_hydrate_stuff(stuff_raw)
+
+ assert result is None
+ assert warn_spy.call_count == 1
+ assert "Local hydration failed" in str(warn_spy.call_args)
+
+ async def test_try_local_hydrate_stuff_returns_none_for_malformed_dict(self, mocker: MockerFixture) -> None:
+ from pipelex import log as pipelex_log # noqa: PLC0415
+
+ warn_spy = mocker.spy(pipelex_log, "warning")
+
+ stuff_raw: dict[str, object] = {"stuff_code": "test", "content": {"text": "x"}}
+
+ result = DeliveryExecutor.try_local_hydrate_stuff(stuff_raw)
+
+ assert result is None
+ assert warn_spy.call_count == 1
+
+ async def test_raw_fallback_html_escapes_special_chars(self, mocker: MockerFixture) -> None:
+ """Fallback HTML rendering must escape HTML-special chars to prevent XSS.
+
+ json.dumps does not escape <, >, or &, so a pipeline output containing
+ "</pre><script>..." would break out of the <pre> wrapper and execute
+ as live HTML when the stored result file is fetched.
+ """
+ mock_output = mocker.MagicMock()
+ mock_output.working_memory_raw = {
+ "root": {
+ "main_stuff": {
+ "stuff_code": "main_stuff",
+ "concept": {
+ "code": "Unknown",
+ "domain_code": "dynamic_test",
+ "description": "Dynamic concept not registered locally",
+ "structure_class_name": "dynamic_test__Unknown",
+ },
+ "content": {"payload": "</pre><script>alert(1)</script><pre>"},
+ }
+ },
+ "aliases": {},
+ }
+ mock_output.graph_spec = None
+
+ files = await DeliveryExecutor().generate_result_files(mock_output)
+
+ html_text = files["main_stuff.html"].data.decode("utf-8")
+ assert html_text.startswith("<pre>")
+ assert html_text.endswith("</pre>")
+ # The injected closing tag must be escaped, not present as live HTML inside the wrapper.
+ inner = html_text[len("<pre>") : -len("</pre>")]
+ assert "</pre>" not in inner
+ assert "<script>" not in inner
+ assert "&lt;/pre&gt;" in html_text
+ assert "&lt;script&gt;" in html_text
+
+ # JSON file must still contain the unescaped raw content (it's not HTML).
+ json_text = files["main_stuff.json"].data.decode("utf-8")
+ assert "</pre><script>alert(1)</script><pre>" in json_text
+
+ async def test_webhook_failure_raises(self, mocker: MockerFixture) -> None:
+ import httpx # noqa: PLC0415
+
+ mock_client = mocker.AsyncMock()
+ mock_client.__aenter__ = mocker.AsyncMock(return_value=mock_client)
+ mock_client.__aexit__ = mocker.AsyncMock(return_value=False)
+ mock_client.post = mocker.AsyncMock(side_effect=httpx.ConnectError("Connection refused"))
+ mocker.patch("pipelex.pipe_run.delivery_executor.httpx.AsyncClient", return_value=mock_client)
+
+ executor = DeliveryExecutor()
+ assignment = DeliveryAssignment(
+ webhooks=[WebhookTarget(url="https://down.example.com")],
+ )
+
+ with pytest.raises(WebhookDeliveryError):
+ await executor.execute(
+ pipe_output=None,
+ user_id="test-user",
+ pipeline_run_id="plr-err",
+ delivery_assignment=assignment,
+ status=DeliveryStatus.COMPLETED,
+ )
diff --git a/tests/unit/pipelex/pipe_run/test_graph_assembly.py b/tests/unit/pipelex/pipe_run/test_graph_assembly.py
new file mode 100644
index 000000000..1cab6375f
--- /dev/null
+++ b/tests/unit/pipelex/pipe_run/test_graph_assembly.py
@@ -0,0 +1,143 @@
+import json
+
+import pytest
+from pydantic import ValidationError
+from pytest_mock import MockerFixture
+
+from pipelex.base_exceptions import PipelexConfigError
+from pipelex.pipe_run.graph_assembly import assemble_graph_on_output
+from pipelex.system.exceptions import MissingDependencyError
+
+
+class TestAssembleGraphOnOutput:
+ def test_returns_when_tracing_disabled(self, mocker: MockerFixture) -> None:
+ """When tracing is disabled, the function returns without touching pipe_output."""
+ mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")
+ mock_config.return_value.pipelex.tracing_config.is_enabled = False
+
+ mock_make_event_log = mocker.patch("pipelex.pipe_run.graph_assembly.make_event_log")
+
+ mock_pipe_output = mocker.MagicMock()
+ original_graph_spec = mock_pipe_output.graph_spec
+
+ assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-disabled")
+
+ mock_make_event_log.assert_not_called()
+ assert mock_pipe_output.graph_spec is original_graph_spec
+
+ def test_swallows_oserror_with_warning(self, mocker: MockerFixture) -> None:
+ """OSError from event log read is caught; pipe_output.graph_spec is left unchanged."""
+ mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")
+ mock_config.return_value.pipelex.tracing_config.is_enabled = True
+
+ mock_event_log = mocker.MagicMock()
+ mock_event_log.read_events = mocker.MagicMock(side_effect=OSError("file vanished"))
+ mocker.patch(
+ "pipelex.pipe_run.graph_assembly.make_event_log",
+ return_value=mock_event_log,
+ )
+
+ mock_pipe_output = mocker.MagicMock()
+ original_graph_spec = mock_pipe_output.graph_spec
+
+ assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-oserror")
+
+ assert mock_pipe_output.graph_spec is original_graph_spec
+ mock_event_log.close.assert_called_once()
+
+ def test_swallows_pipelex_config_error(self, mocker: MockerFixture) -> None:
+ """PipelexConfigError from make_event_log is caught (broken tracing config doesn't fail the pipe)."""
+ mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")
+ mock_config.return_value.pipelex.tracing_config.is_enabled = True
+
+ mocker.patch(
+ "pipelex.pipe_run.graph_assembly.make_event_log",
+ side_effect=PipelexConfigError("ndjson section missing"),
+ )
+
+ mock_pipe_output = mocker.MagicMock()
+ original_graph_spec = mock_pipe_output.graph_spec
+
+ assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-config")
+
+ assert mock_pipe_output.graph_spec is original_graph_spec
+
+ def test_swallows_missing_dependency_error(self, mocker: MockerFixture) -> None:
+ """MissingDependencyError from make_event_log is caught (e.g., boto3 missing for DynamoDB)."""
+ mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")
+ mock_config.return_value.pipelex.tracing_config.is_enabled = True
+
+ mocker.patch(
+ "pipelex.pipe_run.graph_assembly.make_event_log",
+ side_effect=MissingDependencyError(dependency_name="boto3", extra_name="aws"),
+ )
+
+ mock_pipe_output = mocker.MagicMock()
+ original_graph_spec = mock_pipe_output.graph_spec
+
+ assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-missing-dep")
+
+ assert mock_pipe_output.graph_spec is original_graph_spec
+
+ def test_swallows_validation_error(self, mocker: MockerFixture) -> None:
+ """Pydantic ValidationError from GraphSpec construction is caught."""
+ mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")
+ mock_config.return_value.pipelex.tracing_config.is_enabled = True
+
+ mock_event_log = mocker.MagicMock()
+ mock_event_log.read_events = mocker.MagicMock(return_value=[mocker.MagicMock()])
+ mocker.patch(
+ "pipelex.pipe_run.graph_assembly.make_event_log",
+ return_value=mock_event_log,
+ )
+
+ validation_error = ValidationError.from_exception_data(title="GraphSpec", line_errors=[])
+ mocker.patch(
+ "pipelex.pipe_run.graph_assembly.GraphSpecAssembler.assemble",
+ side_effect=validation_error,
+ )
+
+ mock_pipe_output = mocker.MagicMock()
+ original_graph_spec = mock_pipe_output.graph_spec
+
+ assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-validation")
+
+ assert mock_pipe_output.graph_spec is original_graph_spec
+ mock_event_log.close.assert_called_once()
+
+ def test_swallows_json_decode_error(self, mocker: MockerFixture) -> None:
+ """json.JSONDecodeError from event log read is caught."""
+ mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")
+ mock_config.return_value.pipelex.tracing_config.is_enabled = True
+
+ mock_event_log = mocker.MagicMock()
+ mock_event_log.read_events = mocker.MagicMock(
+ side_effect=json.JSONDecodeError("corrupt", doc="{", pos=0),
+ )
+ mocker.patch(
+ "pipelex.pipe_run.graph_assembly.make_event_log",
+ return_value=mock_event_log,
+ )
+
+ mock_pipe_output = mocker.MagicMock()
+ original_graph_spec = mock_pipe_output.graph_spec
+
+ assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-jsondecode")
+
+ assert mock_pipe_output.graph_spec is original_graph_spec
+ mock_event_log.close.assert_called_once()
+
+ def test_propagates_unexpected_keyerror(self, mocker: MockerFixture) -> None:
+ """Programming bugs (KeyError) propagate — proves tightening from blanket except Exception."""
+ mock_config = mocker.patch("pipelex.pipe_run.graph_assembly.get_config")
+ mock_config.return_value.pipelex.tracing_config.is_enabled = True
+
+ mocker.patch(
+ "pipelex.pipe_run.graph_assembly.make_event_log",
+ side_effect=KeyError("unexpected_key"),
+ )
+
+ mock_pipe_output = mocker.MagicMock()
+
+ with pytest.raises(KeyError):
+ assemble_graph_on_output(pipe_output=mock_pipe_output, pipeline_run_id="plr-keyerror")
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py
new file mode 100644
index 000000000..49d211a96
--- /dev/null
+++ b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py
@@ -0,0 +1,116 @@
+from typing import TYPE_CHECKING, cast
+
+import pytest
+
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.pipe_run.exceptions import PipeJobError
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.pipeline.job_metadata import JobMetadata
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_abstract import PipeAbstract
+
+
+def _make_pipe_job(
+ working_memory: WorkingMemory | None = None,
+ library_crate: LibraryCrate | None = None,
+) -> PipeJob:
+ """Build a PipeJob using model_construct to bypass pipe validation (we're testing PipeJob methods, not pipe)."""
+ return PipeJob.model_construct(
+ pipe=cast("PipeAbstract", None),
+ working_memory=working_memory,
+ working_memory_raw=None,
+ pipe_run_params=PipeRunParamsFactory.make_run_params(),
+ job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),
+ output_name=None,
+ library_crate=library_crate,
+ )
+
+
+def _make_dummy_crate() -> LibraryCrate:
+ """Build a minimal LibraryCrate for testing."""
+ return LibraryCrate(concepts={}, pipes={}, domains={}, source_map={})
+
+
+class TestPipeJobHydration:
+ def test_prepare_for_temporal_moves_wm_to_raw(self) -> None:
+ """When library_crate is present, prepare_for_temporal() returns a copy with WM serialized to raw."""
+ working_memory = WorkingMemory()
+ crate = _make_dummy_crate()
+ pipe_job = _make_pipe_job(working_memory=working_memory, library_crate=crate)
+
+ result = pipe_job.prepare_for_temporal()
+
+ assert result.working_memory is None
+ assert result.working_memory_raw is not None
+ assert isinstance(result.working_memory_raw, dict)
+
+ def test_prepare_for_temporal_noop_without_crate(self) -> None:
+ """Without library_crate, prepare_for_temporal() returns self unchanged."""
+ working_memory = WorkingMemory()
+ pipe_job = _make_pipe_job(working_memory=working_memory, library_crate=None)
+
+ result = pipe_job.prepare_for_temporal()
+
+ assert result is pipe_job
+ assert result.working_memory is working_memory
+ assert result.working_memory_raw is None
+
+ def test_prepare_for_temporal_empty_wm(self) -> None:
+ """Empty WorkingMemory serializes to a dict with empty root."""
+ working_memory = WorkingMemory()
+ crate = _make_dummy_crate()
+ pipe_job = _make_pipe_job(working_memory=working_memory, library_crate=crate)
+
+ result = pipe_job.prepare_for_temporal()
+
+ assert result.working_memory_raw is not None
+ assert result.working_memory_raw["root"] == {}
+
+ def test_prepare_for_temporal_does_not_mutate_original(self) -> None:
+ """prepare_for_temporal() leaves the original PipeJob unchanged."""
+ working_memory = WorkingMemory()
+ crate = _make_dummy_crate()
+ pipe_job = _make_pipe_job(working_memory=working_memory, library_crate=crate)
+
+ _ = pipe_job.prepare_for_temporal()
+
+ assert pipe_job.working_memory is working_memory
+ assert pipe_job.working_memory_raw is None
+
+ def test_get_working_memory_from_typed(self) -> None:
+ """get_working_memory() returns the typed WorkingMemory when set."""
+ working_memory = WorkingMemory()
+ pipe_job = _make_pipe_job(working_memory=working_memory)
+
+ result = pipe_job.get_working_memory()
+ assert result is working_memory
+
+ def test_get_working_memory_from_raw_raises(self) -> None:
+ """get_working_memory() raises PipeJobError when only raw is set."""
+ pipe_job = _make_pipe_job(working_memory=None)
+ pipe_job.working_memory_raw = {"root": {}, "aliases": {}}
+
+ with pytest.raises(PipeJobError, match="raw form"):
+ pipe_job.get_working_memory()
+
+ def test_get_working_memory_both_none_returns_empty(self) -> None:
+ """get_working_memory() returns empty WorkingMemory when both are None."""
+ pipe_job = _make_pipe_job(working_memory=None)
+
+ result = pipe_job.get_working_memory()
+ assert isinstance(result, WorkingMemory)
+ assert len(result.root) == 0
+
+ def test_prepare_for_temporal_crate_without_wm(self) -> None:
+ """library_crate present but working_memory is None returns self unchanged."""
+ crate = _make_dummy_crate()
+ pipe_job = _make_pipe_job(working_memory=None, library_crate=crate)
+
+ result = pipe_job.prepare_for_temporal()
+
+ assert result is pipe_job
+ assert result.working_memory is None
+ assert result.working_memory_raw is None
diff --git a/tests/unit/pipelex/pipe_run/test_pipe_run.py b/tests/unit/pipelex/pipe_run/test_pipe_run.py
new file mode 100644
index 000000000..04fec5d47
--- /dev/null
+++ b/tests/unit/pipelex/pipe_run/test_pipe_run.py
@@ -0,0 +1,224 @@
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, StorageTarget
+from pipelex.pipe_run.exceptions import PipeRouterError, WebhookDeliveryError
+from pipelex.pipe_run.pipe_run import PipeRun
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+
+
[email protected](loop_scope="class")
+class TestPipeRun:
+ async def test_run_success_with_delivery(self, mocker: MockerFixture) -> None:
+ mock_output = mocker.MagicMock()
+ mock_router = mocker.AsyncMock()
+ mock_router.run = mocker.AsyncMock(return_value=mock_output)
+
+ mock_executor = mocker.patch(
+ "pipelex.pipe_run.pipe_run.DeliveryExecutor",
+ )
+ mock_executor_instance = mock_executor.return_value
+ mock_executor_instance.execute = mocker.AsyncMock()
+
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata.pipeline_run_id = "plr-123"
+
+ pipe_run = PipeRun(pipe_router=mock_router)
+ assignment = DeliveryAssignment(storage=StorageTarget())
+
+ result = await pipe_run.run(pipe_job=mock_job, delivery_assignment=assignment)
+
+ assert result == mock_output
+ mock_router.run.assert_called_once()
+ mock_executor_instance.execute.assert_called_once()
+
+ async def test_run_success_no_delivery_when_none(self, mocker: MockerFixture) -> None:
+ """When delivery_assignment is None, the delivery executor is not called."""
+ mock_output = mocker.MagicMock()
+ mock_router = mocker.AsyncMock()
+ mock_router.run = mocker.AsyncMock(return_value=mock_output)
+
+ mock_executor = mocker.patch(
+ "pipelex.pipe_run.pipe_run.DeliveryExecutor",
+ )
+ mock_executor_instance = mock_executor.return_value
+ mock_executor_instance.execute = mocker.AsyncMock()
+
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata.pipeline_run_id = "plr-no-delivery"
+
+ pipe_run = PipeRun(pipe_router=mock_router)
+
+ result = await pipe_run.run(pipe_job=mock_job)
+
+ assert result == mock_output
+ mock_executor_instance.execute.assert_not_called()
+
+ async def test_run_failure_delivers_failed_status(self, mocker: MockerFixture) -> None:
+ """On pipe execution failure, delivery runs with FAILED status, then error re-raises."""
+ mock_router = mocker.AsyncMock()
+ mock_router.run = mocker.AsyncMock(
+ side_effect=PipeRouterError(
+ message="Pipe failed",
+ run_mode=PipeRunMode.LIVE,
+ pipe_code="test_pipe",
+ output_name=None,
+ pipe_stack=[],
+ ),
+ )
+
+ mock_executor = mocker.patch(
+ "pipelex.pipe_run.pipe_run.DeliveryExecutor",
+ )
+ mock_executor_instance = mock_executor.return_value
+ mock_executor_instance.execute = mocker.AsyncMock()
+
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata.pipeline_run_id = "plr-fail"
+
+ pipe_run = PipeRun(pipe_router=mock_router)
+ assignment = DeliveryAssignment(storage=StorageTarget())
+
+ with pytest.raises(PipeRouterError):
+ await pipe_run.run(pipe_job=mock_job, delivery_assignment=assignment)
+
+ # Delivery should still have been called with FAILED status
+ mock_executor_instance.execute.assert_called_once()
+ call_kwargs = mock_executor_instance.execute.call_args.kwargs
+ assert call_kwargs["status"] == "FAILED"
+ assert call_kwargs["pipe_output"] is None
+
+ async def test_run_failure_delivers_before_raising(self, mocker: MockerFixture) -> None:
+ """Delivery runs BEFORE the error is re-raised."""
+ call_order: list[str] = []
+
+ mock_router = mocker.AsyncMock()
+
+ async def failing_run(*_args: object, **_kwargs: object) -> None: # noqa: RUF029
+ call_order.append("router.run")
+ raise PipeRouterError(
+ message="fail",
+ run_mode=PipeRunMode.LIVE,
+ pipe_code="p",
+ output_name=None,
+ pipe_stack=[],
+ )
+
+ mock_router.run = failing_run
+
+ mock_executor = mocker.patch(
+ "pipelex.pipe_run.pipe_run.DeliveryExecutor",
+ )
+
+ async def mock_execute(*_args: object, **_kwargs: object) -> None: # noqa: RUF029
+ call_order.append("delivery.execute")
+
+ mock_executor.return_value.execute = mock_execute
+
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata.pipeline_run_id = "plr-order"
+
+ pipe_run = PipeRun(pipe_router=mock_router)
+ assignment = DeliveryAssignment(storage=StorageTarget())
+
+ with pytest.raises(PipeRouterError):
+ await pipe_run.run(pipe_job=mock_job, delivery_assignment=assignment)
+
+ assert call_order == ["router.run", "delivery.execute"]
+
+ async def test_run_failure_then_tracer_close_oserror_raises_original_error(self, mocker: MockerFixture) -> None:
+ """When pipe fails AND close_tracer raises OSError, the original PipeRouterError is raised, not the OSError."""
+ original_error = PipeRouterError(
+ message="pipe blew up",
+ run_mode=PipeRunMode.LIVE,
+ pipe_code="test_pipe",
+ output_name=None,
+ pipe_stack=[],
+ )
+ mock_router = mocker.AsyncMock()
+ mock_router.run = mocker.AsyncMock(side_effect=original_error)
+
+ mock_tracer_manager = mocker.MagicMock()
+ mock_tracer_manager.close_tracer = mocker.MagicMock(side_effect=OSError("disk full"))
+ mocker.patch(
+ "pipelex.pipe_run.pipe_run.GraphTracerManager.get_instance",
+ return_value=mock_tracer_manager,
+ )
+
+ mock_executor = mocker.patch("pipelex.pipe_run.pipe_run.DeliveryExecutor")
+ mock_executor.return_value.execute = mocker.AsyncMock()
+
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata.pipeline_run_id = "plr-mask-tracer"
+
+ pipe_run = PipeRun(pipe_router=mock_router)
+
+ with pytest.raises(PipeRouterError) as exc_info:
+ await pipe_run.run(pipe_job=mock_job, delivery_assignment=DeliveryAssignment(storage=StorageTarget()))
+
+ assert exc_info.value is original_error
+ mock_tracer_manager.close_tracer.assert_called_once_with("plr-mask-tracer")
+
+ async def test_run_success_with_tracer_close_oserror_propagates_oserror(self, mocker: MockerFixture) -> None:
+ """When pipe succeeds but close_tracer raises OSError, OSError propagates (no prior error to preserve)."""
+ mock_output = mocker.MagicMock()
+ mock_router = mocker.AsyncMock()
+ mock_router.run = mocker.AsyncMock(return_value=mock_output)
+
+ mock_tracer_manager = mocker.MagicMock()
+ close_error = OSError("disk full")
+ mock_tracer_manager.close_tracer = mocker.MagicMock(side_effect=close_error)
+ mocker.patch(
+ "pipelex.pipe_run.pipe_run.GraphTracerManager.get_instance",
+ return_value=mock_tracer_manager,
+ )
+
+ mock_executor = mocker.patch("pipelex.pipe_run.pipe_run.DeliveryExecutor")
+ mock_executor.return_value.execute = mocker.AsyncMock()
+
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata.pipeline_run_id = "plr-tracer-only"
+
+ pipe_run = PipeRun(pipe_router=mock_router)
+
+ with pytest.raises(OSError, match="disk full") as exc_info:
+ await pipe_run.run(pipe_job=mock_job)
+
+ assert exc_info.value is close_error
+
+ async def test_run_failure_with_tracer_and_delivery_failures_raises_original(self, mocker: MockerFixture) -> None:
+ """Combined failure: pipe fails, tracer close fails, delivery fails — original PipeRouterError is raised."""
+ original_error = PipeRouterError(
+ message="pipe blew up",
+ run_mode=PipeRunMode.LIVE,
+ pipe_code="test_pipe",
+ output_name=None,
+ pipe_stack=[],
+ )
+ mock_router = mocker.AsyncMock()
+ mock_router.run = mocker.AsyncMock(side_effect=original_error)
+
+ mock_tracer_manager = mocker.MagicMock()
+ mock_tracer_manager.close_tracer = mocker.MagicMock(side_effect=OSError("disk full"))
+ mocker.patch(
+ "pipelex.pipe_run.pipe_run.GraphTracerManager.get_instance",
+ return_value=mock_tracer_manager,
+ )
+
+ mock_executor = mocker.patch("pipelex.pipe_run.pipe_run.DeliveryExecutor")
+ mock_executor.return_value.execute = mocker.AsyncMock(
+ side_effect=WebhookDeliveryError("webhook 503"),
+ )
+
+ mock_job = mocker.MagicMock()
+ mock_job.job_metadata.pipeline_run_id = "plr-triple-fail"
+
+ pipe_run = PipeRun(pipe_router=mock_router)
+ assignment = DeliveryAssignment(storage=StorageTarget())
+
+ with pytest.raises(PipeRouterError) as exc_info:
+ await pipe_run.run(pipe_job=mock_job, delivery_assignment=assignment)
+
+ assert exc_info.value is original_error
+ mock_tracer_manager.close_tracer.assert_called_once()
+ mock_executor.return_value.execute.assert_called_once()
diff --git a/tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_error_handling.py b/tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_error_handling.py
new file mode 100644
index 000000000..5b333ca89
--- /dev/null
+++ b/tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_error_handling.py
@@ -0,0 +1,225 @@
+"""Tests for Anthropic worker SDK exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import anthropic
+import httpx
+import pytest
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError
+from pipelex.plugins.anthropic.anthropic_exceptions import AnthropicCredentialsError
+from pipelex.plugins.anthropic.anthropic_llm_worker import AnthropicLLMWorker
+from tests.unit.pipelex.plugins.anthropic.test_data import AnthropicErrorHandlingTestData
+
+
+def _mock_httpx_response(status_code: int = 429) -> httpx.Response:
+ """Create a minimal httpx.Response for constructing SDK exceptions."""
+ request = httpx.Request("POST", "https://api.anthropic.com/v1/messages")
+ return httpx.Response(status_code=status_code, request=request)
+
+
+def _make_anthropic_rate_limit_error(message: str) -> anthropic.RateLimitError:
+ return anthropic.RateLimitError(message, response=_mock_httpx_response(429), body=None)
+
+
+def _make_anthropic_bad_request_error(message: str) -> anthropic.BadRequestError:
+ return anthropic.BadRequestError(message, response=_mock_httpx_response(400), body=None)
+
+
+def _make_anthropic_timeout_error() -> anthropic.APITimeoutError:
+ request = httpx.Request("POST", "https://api.anthropic.com/v1/messages")
+ return anthropic.APITimeoutError(request=request)
+
+
+def _make_anthropic_connection_error(message: str = "Connection error.") -> anthropic.APIConnectionError:
+ request = httpx.Request("POST", "https://api.anthropic.com/v1/messages")
+ return anthropic.APIConnectionError(message=message, request=request)
+
+
+def _make_anthropic_auth_error(message: str) -> anthropic.AuthenticationError:
+ return anthropic.AuthenticationError(message, response=_mock_httpx_response(401), body=None)
+
+
+def _make_anthropic_permission_denied_error(message: str) -> anthropic.PermissionDeniedError:
+ return anthropic.PermissionDeniedError(message, response=_mock_httpx_response(403), body=None)
+
+
+def _make_worker(mocker: MockerFixture) -> AnthropicLLMWorker:
+ """Create a minimal AnthropicLLMWorker with mocked internals."""
+ worker = object.__new__(AnthropicLLMWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "claude-sonnet-4-20250514"
+ mock_model.name = "claude-sonnet-4"
+ mock_model.thinking_mode = None
+ mock_model.max_tokens = 4096
+ worker.inference_model = mock_model
+ worker.default_max_tokens = 4096
+
+ # Mock the streaming client
+ mock_client = mocker.MagicMock()
+ mock_stream_context = mocker.MagicMock()
+ mock_stream_context.__aenter__ = mocker.AsyncMock()
+ mock_stream_context.__aexit__ = mocker.AsyncMock(return_value=False)
+ mock_client.messages.stream = mocker.MagicMock(return_value=mock_stream_context)
+ worker.anthropic_async_client = mock_client
+
+ return worker
+
+
+def _make_llm_job(mocker: MockerFixture) -> Any:
+ """Create a mock LLM job."""
+ job = mocker.MagicMock()
+ job.applied_job_params = None
+ job.job_params.temperature = 0.5
+ job.job_params.max_tokens = None
+ job.job_params.reasoning_effort = None
+ job.job_params.reasoning_budget = None
+ job.job_report.llm_tokens_usage = None
+ job.llm_prompt.system_text = "system"
+ return job
+
+
[email protected](loop_scope="class")
+class TestAnthropicWorkerErrorHandling:
+ """Tests for Anthropic worker SDK exception handling and error categorization."""
+
+ # ---- RateLimitError tests ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "expected_category", "expected_action_substring"),
+ AnthropicErrorHandlingTestData.RATE_LIMIT_CASES,
+ )
+ async def test_rate_limit(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_action_substring: str | None,
+ ) -> None:
+ """RateLimitError is caught and categorized correctly (TRANSIENT or CAPACITY)."""
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_rate_limit_error(error_message)
+ worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ if expected_action_substring:
+ assert exc_info.value.user_action is not None
+ assert expected_action_substring in exc_info.value.user_action.lower()
+
+ # ---- APITimeoutError test ----
+
+ async def test_timeout_is_transient(self, mocker: MockerFixture) -> None:
+ """APITimeoutError is caught and categorized as TRANSIENT."""
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_timeout_error()
+ worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.__cause__ is sdk_exc
+
+ # ---- BadRequestError tests with content policy detection ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "expected_category", "expected_action_substring"),
+ AnthropicErrorHandlingTestData.BAD_REQUEST_CASES,
+ )
+ async def test_bad_request(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_action_substring: str | None,
+ ) -> None:
+ """BadRequestError is caught with content policy detection."""
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_bad_request_error(error_message)
+ worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ if expected_action_substring:
+ assert exc_info.value.user_action is not None
+ assert expected_action_substring in exc_info.value.user_action.lower()
+
+ # ---- Existing exception categories ----
+
+ async def test_connection_error_is_transient(self, mocker: MockerFixture) -> None:
+ """APIConnectionError should be categorized as TRANSIENT."""
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_connection_error("Connection refused")
+ worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.__cause__ is sdk_exc
+
+ async def test_auth_error_is_configuration(self, mocker: MockerFixture) -> None:
+ """AuthenticationError raises AnthropicCredentialsError with CONFIGURATION category."""
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_auth_error("Invalid API key")
+ worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(AnthropicCredentialsError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CONFIGURATION
+ assert exc_info.value.__cause__ is sdk_exc
+
+ # ---- PermissionDeniedError tests ----
+
+ async def test_permission_denied_quota_is_capacity(self, mocker: MockerFixture) -> None:
+ """PermissionDeniedError with quota message is CAPACITY."""
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_permission_denied_error("Your account quota has been exceeded")
+ worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CAPACITY
+
+ async def test_permission_denied_generic_is_configuration(self, mocker: MockerFixture) -> None:
+ """PermissionDeniedError without quota message is CONFIGURATION."""
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_permission_denied_error("You do not have access to this resource")
+ worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CONFIGURATION
+
+ # ---- to_error_report() integration ----
+
+ async def test_error_report_includes_category(self, mocker: MockerFixture) -> None:
+ """to_error_report() includes error_category and retryable from the exception."""
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_rate_limit_error("Rate limit exceeded")
+ worker.anthropic_async_client.messages.stream.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
diff --git a/tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_object_error_handling.py b/tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_object_error_handling.py
new file mode 100644
index 000000000..e41c0476c
--- /dev/null
+++ b/tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_object_error_handling.py
@@ -0,0 +1,267 @@
+"""Tests for Anthropic worker structured-generation error handling.
+
+Verifies that ``_gen_object`` correctly unwraps ``InstructorRetryException`` to
+recover the underlying Anthropic SDK exception so transient/capacity/auth errors
+are categorized correctly instead of being flattened to ``CONTENT``.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import anthropic
+import httpx
+import pytest
+from instructor.core import FailedAttempt, InstructorRetryException
+from pydantic import BaseModel
+from tenacity import RetryError
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError
+from pipelex.plugins.anthropic.anthropic_exceptions import AnthropicCredentialsError
+from pipelex.plugins.anthropic.anthropic_llm_worker import AnthropicLLMWorker
+
+
+def _mock_httpx_response(status_code: int) -> httpx.Response:
+ request = httpx.Request("POST", "https://api.anthropic.com/v1/messages")
+ return httpx.Response(status_code=status_code, request=request)
+
+
+def _make_anthropic_rate_limit_error(message: str) -> anthropic.RateLimitError:
+ return anthropic.RateLimitError(message, response=_mock_httpx_response(429), body=None)
+
+
+def _make_anthropic_bad_request_error(message: str) -> anthropic.BadRequestError:
+ return anthropic.BadRequestError(message, response=_mock_httpx_response(400), body=None)
+
+
+def _make_anthropic_timeout_error() -> anthropic.APITimeoutError:
+ request = httpx.Request("POST", "https://api.anthropic.com/v1/messages")
+ return anthropic.APITimeoutError(request=request)
+
+
+def _make_anthropic_connection_error(message: str = "Connection refused") -> anthropic.APIConnectionError:
+ request = httpx.Request("POST", "https://api.anthropic.com/v1/messages")
+ return anthropic.APIConnectionError(message=message, request=request)
+
+
+def _make_anthropic_auth_error(message: str) -> anthropic.AuthenticationError:
+ return anthropic.AuthenticationError(message, response=_mock_httpx_response(401), body=None)
+
+
+def _make_anthropic_permission_denied_error(message: str) -> anthropic.PermissionDeniedError:
+ return anthropic.PermissionDeniedError(message, response=_mock_httpx_response(403), body=None)
+
+
+def _wrap_in_instructor_retry(sdk_exc: Exception, *, include_failed_attempts: bool = True) -> InstructorRetryException:
+ """Build an ``InstructorRetryException`` as instructor's retry loop would."""
+ failed_attempts: list[FailedAttempt] | None = [FailedAttempt(attempt_number=1, exception=sdk_exc)] if include_failed_attempts else None
+ return InstructorRetryException(
+ str(sdk_exc),
+ last_completion=None,
+ n_attempts=1,
+ total_usage=0,
+ create_kwargs={},
+ failed_attempts=failed_attempts,
+ )
+
+
+class _DummySchema(BaseModel):
+ value: str
+
+
+def _make_worker(mocker: MockerFixture) -> AnthropicLLMWorker:
+ worker = object.__new__(AnthropicLLMWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "claude-sonnet-4-20250514"
+ mock_model.name = "claude-sonnet-4"
+ mock_model.thinking_mode = None
+ mock_model.max_tokens = 4096
+ # listed_constraints is a list; the worker checks `in`, so an empty list is fine
+ mock_model.listed_constraints = []
+ worker.inference_model = mock_model
+ worker.default_max_tokens = 4096
+
+ instructor_client = mocker.MagicMock()
+ instructor_client.chat.completions.create_with_completion = mocker.AsyncMock()
+ worker.instructor_for_objects = instructor_client
+
+ return worker
+
+
+def _make_llm_job(mocker: MockerFixture) -> Any:
+ job = mocker.MagicMock()
+ job.applied_job_params = None
+ job.job_params.temperature = 0.5
+ job.job_params.max_tokens = None
+ job.job_params.reasoning_effort = None
+ job.job_params.reasoning_budget = None
+ job.job_config.max_retries = 1
+ job.job_report.llm_tokens_usage = None
+ return job
+
+
+def _patch_gen_object_dependencies(mocker: MockerFixture) -> None:
+ """Patch the module-level helpers _gen_object relies on."""
+ config_mock = mocker.MagicMock()
+ config_mock.cogt.llm_config.anthropic_config.structured_output_timeout_seconds = 1200
+ mocker.patch("pipelex.plugins.anthropic.anthropic_llm_worker.get_config", return_value=config_mock)
+ mocker.patch(
+ "pipelex.plugins.anthropic.anthropic_llm_worker.AnthropicFactory.make_simple_messages",
+ new=mocker.AsyncMock(return_value=[]),
+ )
+ mocker.patch(
+ "pipelex.plugins.anthropic.anthropic_llm_worker.AnthropicFactory.calculate_safe_max_tokens_for_timeout",
+ return_value=4096,
+ )
+
+
[email protected](loop_scope="class")
+class TestAnthropicWorkerObjectErrorHandling:
+ """``_gen_object`` must categorize SDK errors that ``instructor`` wraps in ``InstructorRetryException``."""
+
+ async def test_wrapped_rate_limit_is_transient(self, mocker: MockerFixture) -> None:
+ _patch_gen_object_dependencies(mocker)
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_rate_limit_error("Number of request tokens has exceeded your per-minute limit")
+ wrapped = _wrap_in_instructor_retry(sdk_exc)
+ worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=_make_llm_job(mocker), schema=_DummySchema) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.user_action is not None
+ assert "retry" in exc_info.value.user_action.lower()
+ assert exc_info.value.__cause__ is wrapped
+
+ async def test_wrapped_rate_limit_quota_is_capacity(self, mocker: MockerFixture) -> None:
+ _patch_gen_object_dependencies(mocker)
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_rate_limit_error("Your account quota has been exceeded")
+ wrapped = _wrap_in_instructor_retry(sdk_exc)
+ worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=_make_llm_job(mocker), schema=_DummySchema) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CAPACITY
+ assert exc_info.value.user_action is not None
+ assert "billing" in exc_info.value.user_action.lower()
+
+ async def test_wrapped_timeout_is_transient(self, mocker: MockerFixture) -> None:
+ _patch_gen_object_dependencies(mocker)
+ worker = _make_worker(mocker)
+ wrapped = _wrap_in_instructor_retry(_make_anthropic_timeout_error())
+ worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=_make_llm_job(mocker), schema=_DummySchema) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+
+ async def test_wrapped_bad_request_content_policy(self, mocker: MockerFixture) -> None:
+ _patch_gen_object_dependencies(mocker)
+ worker = _make_worker(mocker)
+ sdk_exc = _make_anthropic_bad_request_error("Your request was rejected due to content_policy_violation")
+ wrapped = _wrap_in_instructor_retry(sdk_exc)
+ worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=_make_llm_job(mocker), schema=_DummySchema) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CONTENT
+ assert exc_info.value.user_action is not None
+ assert "safety filters" in exc_info.value.user_action.lower()
+
+ async def test_wrapped_connection_error_is_transient(self, mocker: MockerFixture) -> None:
+ _patch_gen_object_dependencies(mocker)
+ worker = _make_worker(mocker)
+ wrapped = _wrap_in_instructor_retry(_make_anthropic_connection_error())
+ worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=_make_llm_job(mocker), schema=_DummySchema) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+
+ async def test_wrapped_auth_error_raises_credentials_error(self, mocker: MockerFixture) -> None:
+ _patch_gen_object_dependencies(mocker)
+ worker = _make_worker(mocker)
+ wrapped = _wrap_in_instructor_retry(_make_anthropic_auth_error("Invalid API key"))
+ worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(AnthropicCredentialsError) as exc_info:
+ await worker._gen_object(llm_job=_make_llm_job(mocker), schema=_DummySchema) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CONFIGURATION
+
+ async def test_wrapped_permission_quota_is_capacity(self, mocker: MockerFixture) -> None:
+ _patch_gen_object_dependencies(mocker)
+ worker = _make_worker(mocker)
+ wrapped = _wrap_in_instructor_retry(_make_anthropic_permission_denied_error("Your account quota has been exceeded"))
+ worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=_make_llm_job(mocker), schema=_DummySchema) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CAPACITY
+
+ async def test_unrecognized_underlying_falls_back_to_content(self, mocker: MockerFixture) -> None:
+ """A wrapped non-SDK exception (e.g. validation failure) keeps the CONTENT fallback."""
+ _patch_gen_object_dependencies(mocker)
+ worker = _make_worker(mocker)
+ wrapped = _wrap_in_instructor_retry(ValueError("Schema validation failed"))
+ worker.instructor_for_objects.chat.completions.create_with_completion.side_effect = wrapped # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=_make_llm_job(mocker), schema=_DummySchema) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CONTENT
+ assert exc_info.value.__cause__ is wrapped
+
+ async def test_extract_underlying_uses_cause_when_failed_attempts_missing(self, mocker: MockerFixture) -> None:
+ """When ``failed_attempts`` is None, the helper must fall back to walking ``__cause__``."""
+ sdk_exc = _make_anthropic_rate_limit_error("Rate limited")
+ retry_error = RetryError(last_attempt=mocker.MagicMock(_exception=sdk_exc))
+ wrapped = _wrap_in_instructor_retry(sdk_exc, include_failed_attempts=False)
+ wrapped.__cause__ = retry_error
+
+ recovered = AnthropicLLMWorker._extract_underlying_sdk_exception(instructor_exc=wrapped) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert recovered is sdk_exc
+
+ async def test_real_instructor_wraps_rate_limit_and_fix_unwraps_correctly(self, mocker: MockerFixture) -> None:
+ """End-to-end: drive the real instructor library with an SDK exception and
+ verify the worker still categorizes it as TRANSIENT.
+
+ This locks in two assumptions our unit tests rely on:
+ 1. instructor really does wrap ``RateLimitError`` in ``InstructorRetryException``
+ (if a future instructor version stops doing this, the bug disappears and
+ this test should be revisited).
+ 2. ``InstructorRetryException.failed_attempts[-1].exception`` is the original
+ SDK exception, so our extractor recovers the right object.
+ """
+ import instructor # noqa: PLC0415 # imported here to mirror runtime usage
+
+ _patch_gen_object_dependencies(mocker)
+ worker = _make_worker(mocker)
+
+ anthropic_client = anthropic.AsyncAnthropic(api_key="fake")
+ sdk_exc = _make_anthropic_rate_limit_error("Number of request tokens has exceeded your per-minute limit")
+ anthropic_client.messages.create = mocker.AsyncMock(side_effect=sdk_exc) # type: ignore[method-assign]
+ worker.instructor_for_objects = instructor.from_anthropic(anthropic_client)
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_object(llm_job=_make_llm_job(mocker), schema=_DummySchema) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ # The ``from`` chain should preserve the wrapper (not the raw SDK exc),
+ # so traceback shows the full instructor → tenacity → SDK story.
+ cause = exc_info.value.__cause__
+ assert isinstance(cause, InstructorRetryException)
+ assert cause.failed_attempts is not None
+ assert isinstance(cause.failed_attempts[-1].exception, anthropic.RateLimitError)
diff --git a/tests/unit/pipelex/plugins/anthropic/test_data.py b/tests/unit/pipelex/plugins/anthropic/test_data.py
new file mode 100644
index 000000000..95da1be19
--- /dev/null
+++ b/tests/unit/pipelex/plugins/anthropic/test_data.py
@@ -0,0 +1,48 @@
+"""Test data for Anthropic worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class AnthropicErrorHandlingTestData:
+ """Test cases for Anthropic worker SDK exception handling.
+
+ Each tuple: (topic, error_message, expected_category, expected_user_action_substring_or_none)
+ """
+
+ RATE_LIMIT_CASES: ClassVar[list[tuple[str, str, InferenceErrorCategory, str | None]]] = [
+ (
+ "generic_rate_limit",
+ "rate_limit_error: Number of request tokens has exceeded your per-minute limit",
+ InferenceErrorCategory.TRANSIENT,
+ "retry",
+ ),
+ (
+ "quota_exhaustion",
+ "Your account quota has been exceeded",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "billing_limit",
+ "You have reached your billing limit",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ ]
+
+ BAD_REQUEST_CASES: ClassVar[list[tuple[str, str, InferenceErrorCategory, str | None]]] = [
+ (
+ "generic_bad_request",
+ "Invalid parameter: max_tokens must be positive",
+ InferenceErrorCategory.CONTENT,
+ None,
+ ),
+ (
+ "content_policy",
+ "Your request was rejected due to content_policy_violation",
+ InferenceErrorCategory.CONTENT,
+ "safety filters",
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/azure_rest/test_azure_worker_error_handling.py b/tests/unit/pipelex/plugins/azure_rest/test_azure_worker_error_handling.py
new file mode 100644
index 000000000..afbde3925
--- /dev/null
+++ b/tests/unit/pipelex/plugins/azure_rest/test_azure_worker_error_handling.py
@@ -0,0 +1,185 @@
+"""Tests for Azure ImgGen worker HTTP exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import httpx
+import pytest
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import ImgGenGenerationError, InferenceErrorCategory
+from pipelex.plugins.azure_rest.azure_img_gen_worker import AzureImgGenWorker
+from tests.unit.pipelex.plugins.azure_rest.test_data import AzureErrorHandlingTestData
+
+
+def _make_httpx_status_error(status_code: int, text: str = "") -> httpx.HTTPStatusError:
+ """Create a minimal httpx.HTTPStatusError for testing."""
+ request = httpx.Request("POST", "https://test.azure.com/api")
+ response = httpx.Response(status_code=status_code, request=request, text=text)
+ return httpx.HTTPStatusError("error", request=request, response=response)
+
+
+def _make_azure_img_gen_worker(mocker: MockerFixture) -> AzureImgGenWorker:
+ """Create a minimal AzureImgGenWorker with mocked internals."""
+ worker = object.__new__(AzureImgGenWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-azure-model"
+ mock_model.model_id = "dall-e-3"
+ mock_model.name = "dall-e-3"
+ mock_model.rules = mocker.MagicMock()
+ worker.inference_model = mock_model
+ worker.api_key = "test-api-key"
+ worker.endpoint = "https://test.azure.com"
+ worker.api_version = "2024-02-01"
+ worker.plugin = mocker.MagicMock()
+ return worker
+
+
+def _make_img_gen_job(mocker: MockerFixture) -> Any:
+ """Create a mock ImgGen job."""
+ job = mocker.MagicMock()
+ job.img_gen_prompt.positive_text = "a sunset over mountains"
+ job.job_report.img_gen_tokens_usage = None
+ return job
+
+
[email protected](loop_scope="class")
+class TestAzureWorkerErrorHandling:
+ """Tests for Azure ImgGen worker HTTP exception handling and error categorization."""
+
+ @pytest.mark.parametrize(
+ ("_topic", "status_code", "response_text", "expected_category", "expected_message_substring"),
+ AzureErrorHandlingTestData.HTTP_STATUS_ERROR_CASES,
+ )
+ async def test_http_status_error_categorization(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ status_code: int,
+ response_text: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """HTTPStatusError is caught and categorized correctly based on status code."""
+ worker = _make_azure_img_gen_worker(mocker)
+ sdk_exc = _make_httpx_status_error(status_code, response_text)
+
+ mock_client = mocker.MagicMock()
+ mock_response = mocker.MagicMock()
+ mock_response.raise_for_status.side_effect = sdk_exc
+ mock_client.__aenter__ = mocker.AsyncMock(return_value=mock_client)
+ mock_client.__aexit__ = mocker.AsyncMock(return_value=False)
+ mock_client.post = mocker.AsyncMock(return_value=mock_response)
+
+ mocker.patch("pipelex.plugins.azure_rest.azure_img_gen_worker.httpx.AsyncClient", return_value=mock_client)
+ mocker.patch(
+ "pipelex.plugins.azure_rest.azure_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._gen_image_list(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ @pytest.mark.parametrize(
+ ("_topic", "expected_category", "expected_message_substring"),
+ AzureErrorHandlingTestData.CONNECT_ERROR_CASES,
+ )
+ async def test_connect_error_is_transient(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """ConnectError is caught and categorized as TRANSIENT."""
+ worker = _make_azure_img_gen_worker(mocker)
+ request = httpx.Request("POST", "https://test.azure.com/api")
+ sdk_exc = httpx.ConnectError("Connection refused", request=request)
+
+ mock_client = mocker.MagicMock()
+ mock_client.__aenter__ = mocker.AsyncMock(return_value=mock_client)
+ mock_client.__aexit__ = mocker.AsyncMock(return_value=False)
+ mock_client.post = mocker.AsyncMock(side_effect=sdk_exc)
+
+ mocker.patch("pipelex.plugins.azure_rest.azure_img_gen_worker.httpx.AsyncClient", return_value=mock_client)
+ mocker.patch(
+ "pipelex.plugins.azure_rest.azure_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._gen_image_list(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ @pytest.mark.parametrize(
+ ("_topic", "expected_category", "expected_message_substring"),
+ AzureErrorHandlingTestData.TIMEOUT_ERROR_CASES,
+ )
+ async def test_timeout_error_is_transient(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """TimeoutException is caught and categorized as TRANSIENT."""
+ worker = _make_azure_img_gen_worker(mocker)
+ sdk_exc = httpx.ReadTimeout("Request timed out")
+
+ mock_client = mocker.MagicMock()
+ mock_client.__aenter__ = mocker.AsyncMock(return_value=mock_client)
+ mock_client.__aexit__ = mocker.AsyncMock(return_value=False)
+ mock_client.post = mocker.AsyncMock(side_effect=sdk_exc)
+
+ mocker.patch("pipelex.plugins.azure_rest.azure_img_gen_worker.httpx.AsyncClient", return_value=mock_client)
+ mocker.patch(
+ "pipelex.plugins.azure_rest.azure_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._gen_image_list(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ async def test_error_report_includes_category(self, mocker: MockerFixture) -> None:
+ """to_error_report() includes error_category and retryable from the exception."""
+ worker = _make_azure_img_gen_worker(mocker)
+ sdk_exc = _make_httpx_status_error(429, "Rate limit exceeded")
+
+ mock_client = mocker.MagicMock()
+ mock_response = mocker.MagicMock()
+ mock_response.raise_for_status.side_effect = sdk_exc
+ mock_client.__aenter__ = mocker.AsyncMock(return_value=mock_client)
+ mock_client.__aexit__ = mocker.AsyncMock(return_value=False)
+ mock_client.post = mocker.AsyncMock(return_value=mock_response)
+
+ mocker.patch("pipelex.plugins.azure_rest.azure_img_gen_worker.httpx.AsyncClient", return_value=mock_client)
+ mocker.patch(
+ "pipelex.plugins.azure_rest.azure_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._gen_image_list(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
+ assert report.error_type == "ImgGenGenerationError"
diff --git a/tests/unit/pipelex/plugins/azure_rest/test_data.py b/tests/unit/pipelex/plugins/azure_rest/test_data.py
new file mode 100644
index 000000000..2249715ec
--- /dev/null
+++ b/tests/unit/pipelex/plugins/azure_rest/test_data.py
@@ -0,0 +1,73 @@
+"""Test data for Azure REST worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class AzureErrorHandlingTestData:
+ """Test cases for Azure ImgGen worker HTTP exception handling.
+
+ Each tuple: (topic, status_code, response_text, expected_category, expected_message_substring)
+ """
+
+ HTTP_STATUS_ERROR_CASES: ClassVar[list[tuple[str, int, str, InferenceErrorCategory, str]]] = [
+ (
+ "rate_limit_429",
+ 429,
+ "Rate limit exceeded",
+ InferenceErrorCategory.TRANSIENT,
+ "rate limit",
+ ),
+ (
+ "quota_exhausted_402",
+ 402,
+ "Quota exhausted for this subscription",
+ InferenceErrorCategory.CAPACITY,
+ "quota",
+ ),
+ (
+ "auth_error_401",
+ 401,
+ "Invalid subscription key",
+ InferenceErrorCategory.CONFIGURATION,
+ "authentication error",
+ ),
+ (
+ "forbidden_403",
+ 403,
+ "Access denied",
+ InferenceErrorCategory.CONFIGURATION,
+ "authentication error",
+ ),
+ (
+ "bad_request_400",
+ 400,
+ "Invalid parameter value",
+ InferenceErrorCategory.CONTENT,
+ "bad request",
+ ),
+ (
+ "server_error_500",
+ 500,
+ "Internal server error",
+ InferenceErrorCategory.TRANSIENT,
+ "server error",
+ ),
+ ]
+
+ CONNECT_ERROR_CASES: ClassVar[list[tuple[str, InferenceErrorCategory, str]]] = [
+ (
+ "connection_refused",
+ InferenceErrorCategory.TRANSIENT,
+ "connection error",
+ ),
+ ]
+
+ TIMEOUT_ERROR_CASES: ClassVar[list[tuple[str, InferenceErrorCategory, str]]] = [
+ (
+ "request_timeout",
+ InferenceErrorCategory.TRANSIENT,
+ "timed out",
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/bedrock/test_bedrock_worker_error_handling.py b/tests/unit/pipelex/plugins/bedrock/test_bedrock_worker_error_handling.py
new file mode 100644
index 000000000..73e239c35
--- /dev/null
+++ b/tests/unit/pipelex/plugins/bedrock/test_bedrock_worker_error_handling.py
@@ -0,0 +1,160 @@
+"""Tests for Bedrock worker ClientError exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import pytest
+from botocore.exceptions import ClientError
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError
+from pipelex.plugins.bedrock.bedrock_llm_worker import BedrockLLMWorker
+from tests.unit.pipelex.plugins.bedrock.test_data import BedrockLLMErrorHandlingTestData
+
+
+def _make_bedrock_client_error(error_code: str, error_message: str) -> ClientError:
+ """Create a botocore ClientError with the given error code and message."""
+ return ClientError(
+ error_response={"Error": {"Code": error_code, "Message": error_message}},
+ operation_name="Converse",
+ )
+
+
+def _make_bedrock_llm_worker(mocker: MockerFixture) -> BedrockLLMWorker:
+ """Create a minimal BedrockLLMWorker with mocked internals."""
+ worker = object.__new__(BedrockLLMWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "anthropic.claude-3"
+ mock_model.name = "anthropic.claude-3"
+ mock_model.thinking_mode = None
+ mock_model.max_tokens = 4096
+ worker.inference_model = mock_model
+ worker.default_max_tokens = 4096
+
+ mock_client = mocker.MagicMock()
+ mock_client.chat = mocker.AsyncMock()
+ worker.bedrock_client_for_text = mock_client
+
+ return worker
+
+
+def _make_llm_job(mocker: MockerFixture) -> Any:
+ """Create a mock LLM job."""
+ job = mocker.MagicMock()
+ job.applied_job_params = None
+ job.job_params.temperature = 0.5
+ job.job_params.max_tokens = None
+ job.job_params.reasoning_effort = None
+ job.job_params.reasoning_budget = None
+ job.job_report.llm_tokens_usage = None
+ job.llm_prompt.system_text = "system"
+ return job
+
+
[email protected](loop_scope="class")
+class TestBedrockWorkerErrorHandling:
+ """Tests for Bedrock worker ClientError exception handling and error categorization."""
+
+ # ---- ClientError tests ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_code", "error_message", "expected_category", "expected_action_substring"),
+ BedrockLLMErrorHandlingTestData.CLIENT_ERROR_CASES,
+ )
+ async def test_client_error_classification(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ error_code: str,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_action_substring: str | None,
+ ) -> None:
+ """ClientError is caught and categorized correctly by the Bedrock LLM worker."""
+ worker = _make_bedrock_llm_worker(mocker)
+ sdk_exc = _make_bedrock_client_error(error_code, error_message)
+ worker.bedrock_client_for_text.chat.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ # Patch BedrockFactory.make_simple_message to return a mock
+ mock_message = mocker.MagicMock()
+ mock_message.to_dict_list = mocker.MagicMock(return_value=[{"role": "user", "content": "test"}])
+ mocker.patch(
+ "pipelex.plugins.bedrock.bedrock_llm_worker.BedrockFactory.make_simple_message",
+ return_value=mock_message,
+ )
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ if expected_action_substring:
+ assert exc_info.value.user_action is not None
+ assert expected_action_substring in exc_info.value.user_action.lower()
+
+ # ---- to_error_report() integration ----
+
+ async def test_error_report_transient_is_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() for TRANSIENT Bedrock errors has retryable=True."""
+ worker = _make_bedrock_llm_worker(mocker)
+ sdk_exc = _make_bedrock_client_error("ThrottlingException", "Rate exceeded for model")
+ worker.bedrock_client_for_text.chat.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mock_message = mocker.MagicMock()
+ mock_message.to_dict_list = mocker.MagicMock(return_value=[{"role": "user", "content": "test"}])
+ mocker.patch(
+ "pipelex.plugins.bedrock.bedrock_llm_worker.BedrockFactory.make_simple_message",
+ return_value=mock_message,
+ )
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
+ assert report.error_type == "LLMCompletionError"
+
+ async def test_error_report_capacity_not_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() for CAPACITY Bedrock errors has retryable=False."""
+ worker = _make_bedrock_llm_worker(mocker)
+ sdk_exc = _make_bedrock_client_error("ServiceQuotaExceededException", "Service quota exceeded")
+ worker.bedrock_client_for_text.chat.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mock_message = mocker.MagicMock()
+ mock_message.to_dict_list = mocker.MagicMock(return_value=[{"role": "user", "content": "test"}])
+ mocker.patch(
+ "pipelex.plugins.bedrock.bedrock_llm_worker.BedrockFactory.make_simple_message",
+ return_value=mock_message,
+ )
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "capacity"
+ assert report.retryable is False
+
+ async def test_error_report_configuration_not_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() for CONFIGURATION Bedrock errors has retryable=False."""
+ worker = _make_bedrock_llm_worker(mocker)
+ sdk_exc = _make_bedrock_client_error("AccessDeniedException", "Access denied")
+ worker.bedrock_client_for_text.chat.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mock_message = mocker.MagicMock()
+ mock_message.to_dict_list = mocker.MagicMock(return_value=[{"role": "user", "content": "test"}])
+ mocker.patch(
+ "pipelex.plugins.bedrock.bedrock_llm_worker.BedrockFactory.make_simple_message",
+ return_value=mock_message,
+ )
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "configuration"
+ assert report.retryable is False
diff --git a/tests/unit/pipelex/plugins/bedrock/test_data.py b/tests/unit/pipelex/plugins/bedrock/test_data.py
new file mode 100644
index 000000000..e12bde14d
--- /dev/null
+++ b/tests/unit/pipelex/plugins/bedrock/test_data.py
@@ -0,0 +1,64 @@
+"""Test data for Bedrock worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class BedrockLLMErrorHandlingTestData:
+ """Test cases for Bedrock LLM worker ClientError exception handling.
+
+ Each tuple: (topic, error_code, error_message, expected_category, expected_user_action_substring_or_none)
+ """
+
+ CLIENT_ERROR_CASES: ClassVar[list[tuple[str, str, str, InferenceErrorCategory, str | None]]] = [
+ (
+ "throttling_generic",
+ "ThrottlingException",
+ "Rate exceeded for model",
+ InferenceErrorCategory.TRANSIENT,
+ "retry",
+ ),
+ (
+ "throttling_quota",
+ "ThrottlingException",
+ "Account quota limit exceeded for this model",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "service_quota_exceeded",
+ "ServiceQuotaExceededException",
+ "You have exceeded the service quota for this resource",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "access_denied",
+ "AccessDeniedException",
+ "User is not authorized to access this model",
+ InferenceErrorCategory.CONFIGURATION,
+ None,
+ ),
+ (
+ "validation_exception",
+ "ValidationException",
+ "Invalid parameter: max_tokens must be positive",
+ InferenceErrorCategory.CONTENT,
+ None,
+ ),
+ (
+ "model_not_ready",
+ "ModelNotReadyException",
+ "Model is not ready for inference",
+ InferenceErrorCategory.TRANSIENT,
+ None,
+ ),
+ (
+ "service_unavailable",
+ "ServiceUnavailableException",
+ "Service is temporarily unavailable",
+ InferenceErrorCategory.TRANSIENT,
+ None,
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/docling/test_data.py b/tests/unit/pipelex/plugins/docling/test_data.py
new file mode 100644
index 000000000..bd017d2b5
--- /dev/null
+++ b/tests/unit/pipelex/plugins/docling/test_data.py
@@ -0,0 +1,43 @@
+"""Test data for Docling worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class DoclingErrorHandlingTestData:
+ """Test cases for Docling extract worker exception handling.
+
+ Each tuple: (topic, exception_class, exception_message, expected_category, expected_message_substring)
+ """
+
+ EXTRACTION_ERROR_CASES: ClassVar[list[tuple[str, type[Exception], str, InferenceErrorCategory, str]]] = [
+ (
+ "file_not_found",
+ FileNotFoundError,
+ "No such file: /tmp/missing.pdf",
+ InferenceErrorCategory.CONFIGURATION,
+ "file not found",
+ ),
+ (
+ "invalid_format_value_error",
+ ValueError,
+ "Unsupported document format: .xyz",
+ InferenceErrorCategory.CONTENT,
+ "invalid document format",
+ ),
+ (
+ "runtime_conversion_failure",
+ RuntimeError,
+ "Docling engine crashed during conversion",
+ InferenceErrorCategory.CONTENT,
+ "conversion failed",
+ ),
+ (
+ "io_error_transient",
+ OSError,
+ "Disk read error on /tmp/doc.pdf",
+ InferenceErrorCategory.TRANSIENT,
+ "i/o error",
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/docling/test_docling_worker_error_handling.py b/tests/unit/pipelex/plugins/docling/test_docling_worker_error_handling.py
new file mode 100644
index 000000000..c07c16237
--- /dev/null
+++ b/tests/unit/pipelex/plugins/docling/test_docling_worker_error_handling.py
@@ -0,0 +1,120 @@
+"""Tests for Docling extract worker exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import ExtractJobFailureError, InferenceErrorCategory
+from pipelex.plugins.docling.docling_extract_worker import DoclingExtractWorker
+from pipelex.tools.uri.prepared_file import PreparedFileLocalPath
+from tests.unit.pipelex.plugins.docling.test_data import DoclingErrorHandlingTestData
+
+
+def _make_docling_extract_worker(mocker: MockerFixture) -> DoclingExtractWorker:
+ """Create a minimal DoclingExtractWorker with mocked internals."""
+ worker = object.__new__(DoclingExtractWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-docling-model"
+ mock_model.model_id = "docling"
+ mock_model.name = "docling"
+ worker.inference_model = mock_model
+
+ mock_sdk = mocker.MagicMock()
+ mock_sdk.document_converter = mocker.MagicMock()
+ worker.docling_sdk = mock_sdk
+
+ return worker
+
+
[email protected](loop_scope="class")
+class TestDoclingWorkerErrorHandling:
+ """Tests for Docling extract worker exception handling and error categorization."""
+
+ @pytest.mark.parametrize(
+ ("_topic", "exception_class", "exception_message", "expected_category", "expected_message_substring"),
+ DoclingErrorHandlingTestData.EXTRACTION_ERROR_CASES,
+ )
+ async def test_extraction_error_categorization(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ exception_class: type[Exception],
+ exception_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """Docling conversion errors are caught and categorized correctly."""
+ worker = _make_docling_extract_worker(mocker)
+ sdk_exc = exception_class(exception_message)
+
+ # Mock asyncio.to_thread to raise the exception (simulating docling conversion failure)
+ mocker.patch(
+ "pipelex.plugins.docling.docling_extract_worker.asyncio.to_thread",
+ side_effect=sdk_exc,
+ )
+ # Mock prepare_file_from_uri to return a local path that the match statement can handle
+ mock_prepared_file = PreparedFileLocalPath(path="/tmp/test.pdf") # noqa: S108
+ mocker.patch(
+ "pipelex.plugins.docling.docling_extract_worker.prepare_file_from_uri",
+ new_callable=mocker.AsyncMock,
+ return_value=mock_prepared_file,
+ )
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_from_source(source_uri="/tmp/test.pdf") # noqa: SLF001, S108 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ async def test_error_report_for_file_not_found(self, mocker: MockerFixture) -> None:
+ """to_error_report() for FileNotFoundError has CONFIGURATION category."""
+ worker = _make_docling_extract_worker(mocker)
+ sdk_exc = FileNotFoundError("No such file: /tmp/missing.pdf")
+
+ mocker.patch(
+ "pipelex.plugins.docling.docling_extract_worker.asyncio.to_thread",
+ side_effect=sdk_exc,
+ )
+ mock_prepared_file = PreparedFileLocalPath(path="/tmp/test.pdf") # noqa: S108
+ mocker.patch(
+ "pipelex.plugins.docling.docling_extract_worker.prepare_file_from_uri",
+ new_callable=mocker.AsyncMock,
+ return_value=mock_prepared_file,
+ )
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_from_source(source_uri="/tmp/test.pdf") # noqa: SLF001, S108 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "configuration"
+ assert report.retryable is False
+ assert report.error_type == "ExtractJobFailureError"
+
+ async def test_error_report_for_io_error(self, mocker: MockerFixture) -> None:
+ """to_error_report() for OSError has TRANSIENT category and is retryable."""
+ worker = _make_docling_extract_worker(mocker)
+ sdk_exc = OSError("Disk I/O failure")
+
+ mocker.patch(
+ "pipelex.plugins.docling.docling_extract_worker.asyncio.to_thread",
+ side_effect=sdk_exc,
+ )
+ mock_prepared_file = PreparedFileLocalPath(path="/tmp/test.pdf") # noqa: S108
+ mocker.patch(
+ "pipelex.plugins.docling.docling_extract_worker.prepare_file_from_uri",
+ new_callable=mocker.AsyncMock,
+ return_value=mock_prepared_file,
+ )
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_from_source(source_uri="/tmp/test.pdf") # noqa: SLF001, S108 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
diff --git a/tests/unit/pipelex/plugins/fal/test_data.py b/tests/unit/pipelex/plugins/fal/test_data.py
new file mode 100644
index 000000000..c5d202c31
--- /dev/null
+++ b/tests/unit/pipelex/plugins/fal/test_data.py
@@ -0,0 +1,81 @@
+"""Test data for FAL worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class FalErrorHandlingTestData:
+ """Test cases for FAL ImgGen worker SDK exception handling.
+
+ Each tuple: (topic, status_code, message, expected_category, expected_message_substring)
+ """
+
+ HTTP_ERROR_CASES: ClassVar[list[tuple[str, int, str, InferenceErrorCategory, str]]] = [
+ (
+ "rate_limit_429",
+ 429,
+ "Too many requests",
+ InferenceErrorCategory.TRANSIENT,
+ "rate limit",
+ ),
+ (
+ "quota_exhausted_402",
+ 402,
+ "Payment required - quota exhausted",
+ InferenceErrorCategory.CAPACITY,
+ "quota",
+ ),
+ (
+ "auth_error_401",
+ 401,
+ "Invalid API key",
+ InferenceErrorCategory.CONFIGURATION,
+ "authentication error",
+ ),
+ (
+ "forbidden_403",
+ 403,
+ "Access denied",
+ InferenceErrorCategory.CONFIGURATION,
+ "authentication error",
+ ),
+ (
+ "bad_request_400",
+ 400,
+ "Invalid parameter",
+ InferenceErrorCategory.CONTENT,
+ "bad request",
+ ),
+ (
+ "server_error_500",
+ 500,
+ "Internal server error",
+ InferenceErrorCategory.TRANSIENT,
+ "api error",
+ ),
+ ]
+
+ MISSING_CREDENTIALS_CASES: ClassVar[list[tuple[str, InferenceErrorCategory, str]]] = [
+ (
+ "missing_fal_key",
+ InferenceErrorCategory.CONFIGURATION,
+ "fal api key",
+ ),
+ ]
+
+ TIMEOUT_CASES: ClassVar[list[tuple[str, InferenceErrorCategory, str]]] = [
+ (
+ "request_timeout",
+ InferenceErrorCategory.TRANSIENT,
+ "timed out",
+ ),
+ ]
+
+ GENERIC_CLIENT_ERROR_CASES: ClassVar[list[tuple[str, InferenceErrorCategory, str]]] = [
+ (
+ "generic_fal_error",
+ InferenceErrorCategory.TRANSIENT,
+ "fal error",
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/fal/test_fal_worker_error_handling.py b/tests/unit/pipelex/plugins/fal/test_fal_worker_error_handling.py
new file mode 100644
index 000000000..39846da8b
--- /dev/null
+++ b/tests/unit/pipelex/plugins/fal/test_fal_worker_error_handling.py
@@ -0,0 +1,208 @@
+"""Tests for FAL ImgGen worker SDK exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import pytest
+from fal_client.auth import MissingCredentialsError
+from fal_client.client import FalClientError, FalClientHTTPError, FalClientTimeoutError
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import ImgGenGenerationError, InferenceErrorCategory
+from pipelex.plugins.fal.fal_img_gen_worker import FalImgGenWorker
+from tests.unit.pipelex.plugins.fal.test_data import FalErrorHandlingTestData
+
+
+def _make_fal_http_error(status_code: int, message: str) -> FalClientHTTPError:
+ """Create a FalClientHTTPError for testing."""
+ import httpx # noqa: PLC0415
+
+ request = httpx.Request("POST", "https://fal.ai/test")
+ response = httpx.Response(status_code=status_code, request=request, text=message)
+ return FalClientHTTPError(message=message, status_code=status_code, response_headers={}, response=response)
+
+
+def _make_fal_timeout_error() -> FalClientTimeoutError:
+ """Create a FalClientTimeoutError for testing."""
+ return FalClientTimeoutError(timeout=30.0)
+
+
+def _make_fal_client_error(message: str = "FAL client error") -> FalClientError:
+ """Create a FalClientError for testing."""
+ return FalClientError(message)
+
+
+def _make_missing_credentials_error() -> MissingCredentialsError:
+ """Create a minimal MissingCredentialsError for testing."""
+ return MissingCredentialsError()
+
+
+def _make_fal_img_gen_worker(mocker: MockerFixture) -> FalImgGenWorker:
+ """Create a minimal FalImgGenWorker with mocked internals."""
+ worker = object.__new__(FalImgGenWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-fal-model"
+ mock_model.model_id = "fal-ai/flux/dev"
+ mock_model.name = "flux-dev"
+ mock_model.rules = mocker.MagicMock()
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ mock_client.submit = mocker.AsyncMock()
+ worker.fal_async_client = mock_client
+
+ return worker
+
+
+def _make_img_gen_job(mocker: MockerFixture) -> Any:
+ """Create a mock ImgGen job."""
+ job = mocker.MagicMock()
+ job.img_gen_prompt.positive_text = "a sunset over mountains"
+ job.job_report.img_gen_tokens_usage = None
+ return job
+
+
[email protected](loop_scope="class")
+class TestFalWorkerErrorHandling:
+ """Tests for FAL ImgGen worker SDK exception handling and error categorization."""
+
+ @pytest.mark.parametrize(
+ ("_topic", "status_code", "message", "expected_category", "expected_message_substring"),
+ FalErrorHandlingTestData.HTTP_ERROR_CASES,
+ )
+ async def test_http_error_categorization(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ status_code: int,
+ message: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """FalClientHTTPError is caught and categorized correctly based on status code."""
+ worker = _make_fal_img_gen_worker(mocker)
+ sdk_exc = _make_fal_http_error(status_code, message)
+ worker.fal_async_client.submit.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mocker.patch(
+ "pipelex.plugins.fal.fal_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test", "model": "test-model-id"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._submit_and_get_result(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ @pytest.mark.parametrize(
+ ("_topic", "expected_category", "expected_message_substring"),
+ FalErrorHandlingTestData.MISSING_CREDENTIALS_CASES,
+ )
+ async def test_missing_credentials_is_configuration(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """MissingCredentialsError is caught and categorized as CONFIGURATION."""
+ worker = _make_fal_img_gen_worker(mocker)
+ sdk_exc = _make_missing_credentials_error()
+ worker.fal_async_client.submit.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mocker.patch(
+ "pipelex.plugins.fal.fal_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test", "model": "test-model-id"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._submit_and_get_result(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ @pytest.mark.parametrize(
+ ("_topic", "expected_category", "expected_message_substring"),
+ FalErrorHandlingTestData.TIMEOUT_CASES,
+ )
+ async def test_timeout_error_is_transient(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """FalClientTimeoutError is caught and categorized as TRANSIENT."""
+ worker = _make_fal_img_gen_worker(mocker)
+ sdk_exc = _make_fal_timeout_error()
+ worker.fal_async_client.submit.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mocker.patch(
+ "pipelex.plugins.fal.fal_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test", "model": "test-model-id"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._submit_and_get_result(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ @pytest.mark.parametrize(
+ ("_topic", "expected_category", "expected_message_substring"),
+ FalErrorHandlingTestData.GENERIC_CLIENT_ERROR_CASES,
+ )
+ async def test_generic_client_error_is_transient(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """FalClientError is caught and categorized as TRANSIENT."""
+ worker = _make_fal_img_gen_worker(mocker)
+ sdk_exc = _make_fal_client_error()
+ worker.fal_async_client.submit.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mocker.patch(
+ "pipelex.plugins.fal.fal_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test", "model": "test-model-id"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._submit_and_get_result(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ async def test_error_report_includes_category(self, mocker: MockerFixture) -> None:
+ """to_error_report() includes error_category and retryable from the exception."""
+ worker = _make_fal_img_gen_worker(mocker)
+ sdk_exc = _make_fal_http_error(402, "Payment required")
+ worker.fal_async_client.submit.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mocker.patch(
+ "pipelex.plugins.fal.fal_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test", "model": "test-model-id"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._submit_and_get_result(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "capacity"
+ assert report.retryable is False
+ assert report.error_type == "ImgGenGenerationError"
diff --git a/tests/unit/pipelex/plugins/gateway/test_data.py b/tests/unit/pipelex/plugins/gateway/test_data.py
new file mode 100644
index 000000000..d684e3792
--- /dev/null
+++ b/tests/unit/pipelex/plugins/gateway/test_data.py
@@ -0,0 +1,86 @@
+"""Test data for Gateway worker error handling and quota detection tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class GatewayQuotaDetectionTestData:
+ """Test cases for Portkey error classification across gateway workers.
+
+ CLASSIFY_CASES tuple: (topic, exception_type_name, status_code, error_message, expected_category)
+ The exception_type_name is used to select which portkey exception to construct.
+ """
+
+ CLASSIFY_CASES: ClassVar[list[tuple[str, str, int, str, InferenceErrorCategory]]] = [
+ (
+ "rate_limit_transient",
+ "RateLimitError",
+ 429,
+ "Rate limit exceeded. Please retry after 20s",
+ InferenceErrorCategory.TRANSIENT,
+ ),
+ (
+ "rate_limit_quota_keyword",
+ "RateLimitError",
+ 429,
+ "You have exceeded your quota allocation",
+ InferenceErrorCategory.CAPACITY,
+ ),
+ (
+ "rate_limit_billing_keyword",
+ "RateLimitError",
+ 429,
+ "Billing limit reached",
+ InferenceErrorCategory.CAPACITY,
+ ),
+ (
+ "payment_required_402",
+ "APIStatusError",
+ 402,
+ "Payment required",
+ InferenceErrorCategory.CAPACITY,
+ ),
+ (
+ "auth_error_401",
+ "AuthenticationError",
+ 401,
+ "Invalid API key",
+ InferenceErrorCategory.CONFIGURATION,
+ ),
+ (
+ "permission_denied_403",
+ "PermissionDeniedError",
+ 403,
+ "Access denied",
+ InferenceErrorCategory.CONFIGURATION,
+ ),
+ (
+ "bad_request_400",
+ "BadRequestError",
+ 400,
+ "Invalid parameters",
+ InferenceErrorCategory.CONTENT,
+ ),
+ (
+ "not_found_404",
+ "NotFoundError",
+ 404,
+ "Model not found",
+ InferenceErrorCategory.CONFIGURATION,
+ ),
+ (
+ "timeout_error",
+ "APITimeoutError",
+ 0,
+ "Request timed out",
+ InferenceErrorCategory.TRANSIENT,
+ ),
+ (
+ "connection_error",
+ "APIConnectionError",
+ 0,
+ "Connection refused",
+ InferenceErrorCategory.TRANSIENT,
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/gateway/test_gateway_quota_detection.py b/tests/unit/pipelex/plugins/gateway/test_gateway_quota_detection.py
new file mode 100644
index 000000000..f97aaacda
--- /dev/null
+++ b/tests/unit/pipelex/plugins/gateway/test_gateway_quota_detection.py
@@ -0,0 +1,244 @@
+"""Tests for Gateway worker Portkey error classification and quota detection."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import httpx
+import pytest
+from portkey_ai.api_resources import exceptions as portkey_exceptions
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import ExtractJobFailureError, ImgGenGenerationError, InferenceErrorCategory
+from pipelex.plugins.gateway.gateway_exceptions import GatewaySearchResponseError
+from pipelex.plugins.gateway.gateway_extract_worker import GatewayExtractWorker
+from pipelex.plugins.gateway.gateway_factory import GatewayFactory
+from pipelex.plugins.gateway.gateway_img_gen_worker import GatewayImgGenWorker
+from pipelex.plugins.gateway.gateway_protocols import GatewayExtractProtocol
+from pipelex.plugins.gateway.gateway_search_worker import GatewaySearchWorker
+from tests.unit.pipelex.plugins.gateway.test_data import GatewayQuotaDetectionTestData
+
+
+def _make_portkey_exception(exception_type_name: str, status_code: int, message: str) -> portkey_exceptions.APIError:
+ """Create a Portkey exception by type name for testing."""
+ request = httpx.Request("POST", "https://api.portkey.ai/v1/test")
+ response = httpx.Response(status_code=status_code, request=request)
+
+ exception_map: dict[str, type[portkey_exceptions.APIError]] = {
+ "RateLimitError": portkey_exceptions.RateLimitError,
+ "AuthenticationError": portkey_exceptions.AuthenticationError,
+ "PermissionDeniedError": portkey_exceptions.PermissionDeniedError,
+ "BadRequestError": portkey_exceptions.BadRequestError,
+ "NotFoundError": portkey_exceptions.NotFoundError,
+ "APIStatusError": portkey_exceptions.APIStatusError,
+ }
+
+ if exception_type_name == "APITimeoutError":
+ return portkey_exceptions.APITimeoutError(request=request)
+ if exception_type_name == "APIConnectionError":
+ return portkey_exceptions.APIConnectionError(message=message, request=request)
+
+ exc_class = exception_map.get(exception_type_name)
+ if exc_class is None:
+ msg = f"Unknown portkey exception type: {exception_type_name}"
+ raise ValueError(msg)
+ return exc_class(message, request=request, response=response, body=None) # type: ignore[call-arg] # pyright: ignore[reportCallIssue]
+
+
+def _make_gateway_extract_worker(mocker: MockerFixture) -> GatewayExtractWorker:
+ """Create a minimal GatewayExtractWorker with mocked internals."""
+ worker = object.__new__(GatewayExtractWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-gateway-extract"
+ mock_model.model_id = "mistral-doc-ai"
+ mock_model.name = "mistral-doc-ai"
+ mock_model.tag = "test-extract-tag"
+ mock_model.extra_headers = {}
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ worker.portkey_client = mock_client
+
+ mock_tenacity = mocker.MagicMock()
+ mock_tenacity.wait_multiplier = 1
+ mock_tenacity.wait_max = 10
+ mock_tenacity.wait_exp_base = 2
+ mock_tenacity.max_retries = 1
+ worker._tenacity_config = mock_tenacity # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ return worker
+
+
+def _make_gateway_img_gen_worker(mocker: MockerFixture) -> GatewayImgGenWorker:
+ """Create a minimal GatewayImgGenWorker with mocked internals."""
+ worker = object.__new__(GatewayImgGenWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-gateway-imggen"
+ mock_model.model_id = "gpt-image-1"
+ mock_model.name = "gpt-image-1"
+ mock_model.tag = "test-imggen-tag"
+ mock_model.extra_headers = {}
+ mock_model.rules = mocker.MagicMock()
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ worker.portkey_client = mock_client
+
+ return worker
+
+
+def _make_gateway_search_worker(mocker: MockerFixture) -> GatewaySearchWorker:
+ """Create a minimal GatewaySearchWorker with mocked internals."""
+ worker = object.__new__(GatewaySearchWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-gateway-search"
+ mock_model.model_id = "linkup/standard"
+ mock_model.name = "linkup-standard"
+ mock_model.tag = "test-search-tag"
+ mock_model.extra_headers = {}
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ worker.portkey_client = mock_client
+
+ mock_tenacity = mocker.MagicMock()
+ mock_tenacity.wait_multiplier = 1
+ mock_tenacity.wait_max = 10
+ mock_tenacity.wait_exp_base = 2
+ mock_tenacity.max_retries = 1
+ worker._tenacity_config = mock_tenacity # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ return worker
+
+
[email protected](loop_scope="class")
+class TestGatewayQuotaDetection:
+ """Tests for Portkey error classification across gateway workers."""
+
+ # ---- Classification method tests (direct unit tests) ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "exception_type_name", "status_code", "error_message", "expected_category"),
+ GatewayQuotaDetectionTestData.CLASSIFY_CASES,
+ )
+ async def test_classify_error_category(
+ self,
+ _topic: str,
+ exception_type_name: str,
+ status_code: int,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ ) -> None:
+ """GatewayFactory.classify_error_category returns correct category for Portkey errors."""
+ exc = _make_portkey_exception(exception_type_name, status_code, error_message)
+
+ result = GatewayFactory.classify_error_category(exc)
+
+ assert result is expected_category
+
+ # ---- Full flow tests: verify the exception propagates with category ----
+
+ async def test_imggen_worker_propagates_rate_limit_as_transient(self, mocker: MockerFixture) -> None:
+ """GatewayImgGenWorker raises ImgGenGenerationError with TRANSIENT for rate limit."""
+ worker = _make_gateway_img_gen_worker(mocker)
+ sdk_exc = _make_portkey_exception("RateLimitError", 429, "Rate limit exceeded")
+
+ mock_options = mocker.MagicMock()
+ mock_options.post = mocker.AsyncMock(side_effect=sdk_exc)
+ worker.portkey_client.with_options = mocker.MagicMock(return_value=mock_options) # type: ignore[method-assign]
+
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test"},
+ )
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_img_gen_worker.GatewayDeck.get_config_id",
+ return_value="test-config",
+ )
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_img_gen_worker.GatewayFactory.make_error_summary_from_portkey_error",
+ return_value="Rate limit exceeded",
+ )
+
+ img_gen_job = mocker.MagicMock()
+ img_gen_job.job_report.img_gen_tokens_usage = None
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._gen_image_list(img_gen_job=img_gen_job, nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.__cause__ is sdk_exc
+
+ async def test_search_worker_propagates_auth_error_as_configuration(self, mocker: MockerFixture) -> None:
+ """GatewaySearchWorker raises GatewaySearchResponseError with CONFIGURATION for auth error."""
+ worker = _make_gateway_search_worker(mocker)
+ sdk_exc = _make_portkey_exception("AuthenticationError", 401, "Invalid API key")
+
+ mock_options = mocker.MagicMock()
+ mock_options.post = mocker.AsyncMock(side_effect=sdk_exc)
+ worker.portkey_client.with_options = mocker.MagicMock(return_value=mock_options) # type: ignore[method-assign]
+
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_search_worker.GatewayDeck.get_config_id",
+ return_value="test-config",
+ )
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_search_worker.GatewayFactory.make_error_summary_from_portkey_error",
+ return_value="Invalid API key",
+ )
+
+ with pytest.raises(GatewaySearchResponseError) as exc_info:
+ await worker._call_relay(model="linkup/standard", content='{"query": "test"}') # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CONFIGURATION
+ assert exc_info.value.__cause__ is sdk_exc
+
+ async def test_extract_worker_propagates_quota_as_capacity(self, mocker: MockerFixture) -> None:
+ """GatewayExtractWorker raises ExtractJobFailureError with CAPACITY for quota exhaustion."""
+ worker = _make_gateway_extract_worker(mocker)
+ sdk_exc = _make_portkey_exception("RateLimitError", 429, "You have exceeded your quota allocation")
+
+ mock_options = mocker.MagicMock()
+ mock_options.post = mocker.AsyncMock(side_effect=sdk_exc)
+ worker.portkey_client.with_options = mocker.MagicMock(return_value=mock_options) # type: ignore[method-assign]
+
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_extract_worker.GatewayDeck.get_config_id",
+ return_value="test-config",
+ )
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_extract_worker.GatewayFactory.make_error_summary_from_portkey_error",
+ return_value="Quota exhausted",
+ )
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_extract_worker.GatewayFactory.make_extras",
+ return_value=({}, {"messages": [{"role": "user", "content": "test"}]}),
+ )
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_extract_worker.make_base64_url_from_any_uri",
+ new_callable=mocker.AsyncMock,
+ return_value="data:application/pdf;base64,dGVzdA==",
+ )
+
+ extract_job = mocker.MagicMock()
+ extract_job.extract_input.image_uri = None
+ extract_job.extract_input.document_uri = "/tmp/test.pdf" # noqa: S108
+ extract_job.job_params.max_nb_images = 0
+ extract_job.job_params.should_caption_images = False
+
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_extract_worker.GatewayExtractProtocol.make_from_model_handle",
+ return_value=GatewayExtractProtocol.MISTRAL_DOC_AI,
+ )
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_extract_worker.get_storage_provider",
+ )
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_document(extract_job=extract_job) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CAPACITY
+ assert exc_info.value.__cause__ is sdk_exc
diff --git a/tests/unit/pipelex/plugins/google/test_data.py b/tests/unit/pipelex/plugins/google/test_data.py
new file mode 100644
index 000000000..971d056ee
--- /dev/null
+++ b/tests/unit/pipelex/plugins/google/test_data.py
@@ -0,0 +1,85 @@
+"""Test data for Google worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class GoogleErrorHandlingTestData:
+ """Test cases for Google worker SDK exception handling.
+
+ Each tuple: (topic, status_code, error_message, expected_category, expected_user_action_substring_or_none)
+ """
+
+ CLIENT_ERROR_CASES: ClassVar[list[tuple[str, int, str, InferenceErrorCategory, str | None]]] = [
+ (
+ "not_found_404",
+ 404,
+ "Model gemini-99 not found",
+ InferenceErrorCategory.CONFIGURATION,
+ None,
+ ),
+ (
+ "auth_401",
+ 401,
+ "Request had invalid authentication credentials",
+ InferenceErrorCategory.CONFIGURATION,
+ None,
+ ),
+ (
+ "forbidden_403",
+ 403,
+ "Permission denied on resource project",
+ InferenceErrorCategory.CONFIGURATION,
+ None,
+ ),
+ (
+ "rate_limit_429_quota",
+ 429,
+ "Resource has been exhausted (e.g. check quota)",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "rate_limit_429_billing",
+ 429,
+ "Billing account is not active",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "rate_limit_429_generic",
+ 429,
+ "Too many requests, please slow down",
+ InferenceErrorCategory.TRANSIENT,
+ "retry",
+ ),
+ (
+ "bad_request_400_content_policy",
+ 400,
+ "Your request was rejected due to content_policy_violation",
+ InferenceErrorCategory.CONTENT,
+ "safety filters",
+ ),
+ (
+ "bad_request_400_safety",
+ 400,
+ "The response was blocked by safety systems",
+ InferenceErrorCategory.CONTENT,
+ "safety filters",
+ ),
+ (
+ "bad_request_400_generic",
+ 400,
+ "Invalid parameter: temperature must be between 0 and 2",
+ InferenceErrorCategory.CONTENT,
+ None,
+ ),
+ (
+ "other_client_error_fallback",
+ 422,
+ "Unprocessable entity",
+ InferenceErrorCategory.TRANSIENT,
+ None,
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/google/test_google_worker_error_handling.py b/tests/unit/pipelex/plugins/google/test_google_worker_error_handling.py
new file mode 100644
index 000000000..6dddd9b10
--- /dev/null
+++ b/tests/unit/pipelex/plugins/google/test_google_worker_error_handling.py
@@ -0,0 +1,229 @@
+"""Tests for Google worker SDK exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import pytest
+from google.genai import errors as genai_errors
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import (
+ ImgGenGenerationError,
+ InferenceErrorCategory,
+ LLMCompletionError,
+)
+from pipelex.plugins.google.google_img_gen_worker import GoogleImgGenWorker
+from pipelex.plugins.google.google_llm_worker import GoogleLLMWorker
+from tests.unit.pipelex.plugins.google.test_data import GoogleErrorHandlingTestData
+
+
+def _make_google_client_error(code: int, message: str) -> genai_errors.ClientError:
+ """Create a ClientError with a minimal response_json dict."""
+ response_json = {"message": message, "status": "ERROR"}
+ return genai_errors.ClientError(code, response_json, None)
+
+
+def _make_google_server_error(code: int, message: str) -> genai_errors.ServerError:
+ """Create a ServerError with a minimal response_json dict."""
+ response_json = {"message": message, "status": "ERROR"}
+ return genai_errors.ServerError(code, response_json, None)
+
+
+def _make_google_llm_worker(mocker: MockerFixture) -> GoogleLLMWorker:
+ """Create a minimal GoogleLLMWorker with mocked internals."""
+ worker = object.__new__(GoogleLLMWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "gemini-pro"
+ mock_model.name = "gemini-pro"
+ mock_model.thinking_mode = None
+ worker.inference_model = mock_model
+
+ mock_async_client = mocker.MagicMock()
+ mock_async_client.models.generate_content = mocker.AsyncMock()
+ worker.genai_async_client = mock_async_client
+
+ # Mock the factory so it doesn't try to parse mock prompt objects as real Pydantic types
+ mocker.patch(
+ "pipelex.plugins.google.google_llm_worker.GoogleFactory.prepare_user_contents",
+ new_callable=mocker.AsyncMock,
+ return_value=[],
+ )
+
+ return worker
+
+
+def _make_google_img_gen_worker(mocker: MockerFixture) -> GoogleImgGenWorker:
+ """Create a minimal GoogleImgGenWorker with mocked internals."""
+ worker = object.__new__(GoogleImgGenWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "imagen-3.0-generate-002"
+ mock_model.name = "imagen-3.0-generate-002"
+ worker.inference_model = mock_model
+
+ mock_async_client = mocker.MagicMock()
+ mock_async_client.models.generate_content = mocker.AsyncMock()
+ worker.genai_async_client = mock_async_client
+
+ # Mock the factory static methods so they don't fail before the API call
+ mocker.patch(
+ "pipelex.plugins.google.google_img_gen_worker.GoogleImgGenFactory.aspect_ratio_literal",
+ return_value="1:1",
+ )
+ mocker.patch(
+ "pipelex.plugins.google.google_img_gen_worker.GoogleImgGenFactory.dimensions_for_aspect_ratio_and_size",
+ return_value=(1024, 1024),
+ )
+
+ return worker
+
+
+def _make_llm_job(mocker: MockerFixture) -> Any:
+ """Create a mock LLM job."""
+ job = mocker.MagicMock()
+ job.applied_job_params = None
+ job.job_params.temperature = 0.5
+ job.job_params.max_tokens = None
+ job.job_params.seed = None
+ job.job_params.reasoning_effort = None
+ job.job_params.reasoning_budget = None
+ job.job_report.llm_tokens_usage = None
+ job.llm_prompt.system_text = "system"
+ return job
+
+
+def _make_img_gen_job(mocker: MockerFixture) -> Any:
+ """Create a mock ImgGen job."""
+ job = mocker.MagicMock()
+ job.job_params.aspect_ratio = None
+ job.job_params.output_format = None
+ job.job_params.is_moderated = True
+ job.job_params.background = None
+ job.job_params.quality = None
+ job.img_gen_prompt.positive_text = "a cute cat"
+ job.job_report.img_gen_tokens_usage = None
+ return job
+
+
[email protected](loop_scope="class")
+class TestGoogleWorkerErrorHandling:
+ """Tests for Google worker SDK exception handling and error categorization."""
+
+ # ---- LLM worker ClientError tests (parametrized) ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "status_code", "error_message", "expected_category", "expected_action_substring"),
+ GoogleErrorHandlingTestData.CLIENT_ERROR_CASES,
+ )
+ async def test_llm_client_error(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ status_code: int,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_action_substring: str | None,
+ ) -> None:
+ """ClientError is caught and categorized correctly by the LLM worker."""
+ worker = _make_google_llm_worker(mocker)
+ sdk_exc = _make_google_client_error(code=status_code, message=error_message)
+ worker.genai_async_client.models.generate_content.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ if expected_action_substring:
+ assert exc_info.value.user_action is not None
+ assert expected_action_substring in exc_info.value.user_action.lower()
+
+ # ---- LLM worker ServerError test ----
+
+ async def test_llm_server_error_has_transient_category(self, mocker: MockerFixture) -> None:
+ """ServerError should be categorized as TRANSIENT."""
+ worker = _make_google_llm_worker(mocker)
+ sdk_exc = _make_google_server_error(code=500, message="Internal server error")
+ worker.genai_async_client.models.generate_content.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.__cause__ is sdk_exc
+
+ # ---- ImgGen worker ClientError tests (parametrized) ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "status_code", "error_message", "expected_category", "expected_action_substring"),
+ GoogleErrorHandlingTestData.CLIENT_ERROR_CASES,
+ )
+ async def test_img_gen_client_error(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ status_code: int,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_action_substring: str | None,
+ ) -> None:
+ """ClientError is caught and categorized correctly by the ImgGen worker."""
+ worker = _make_google_img_gen_worker(mocker)
+ sdk_exc = _make_google_client_error(code=status_code, message=error_message)
+ worker.genai_async_client.models.generate_content.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._gen_image(img_gen_job=_make_img_gen_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ if expected_action_substring:
+ assert exc_info.value.user_action is not None
+ assert expected_action_substring in exc_info.value.user_action.lower()
+
+ # ---- ImgGen worker ServerError test ----
+
+ async def test_img_gen_server_error_has_transient_category(self, mocker: MockerFixture) -> None:
+ """ServerError should be categorized as TRANSIENT for ImgGen worker."""
+ worker = _make_google_img_gen_worker(mocker)
+ sdk_exc = _make_google_server_error(code=500, message="Internal server error")
+ worker.genai_async_client.models.generate_content.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._gen_image(img_gen_job=_make_img_gen_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.__cause__ is sdk_exc
+
+ # ---- to_error_report() integration ----
+
+ async def test_error_report_includes_category_and_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() includes error_category and retryable from the exception."""
+ worker = _make_google_llm_worker(mocker)
+ sdk_exc = _make_google_client_error(code=429, message="Too many requests")
+ worker.genai_async_client.models.generate_content.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
+ assert report.error_type == "LLMCompletionError"
+
+ async def test_error_report_capacity_not_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() for CAPACITY errors has retryable=False."""
+ worker = _make_google_llm_worker(mocker)
+ sdk_exc = _make_google_client_error(code=429, message="Resource has been exhausted (e.g. check quota)")
+ worker.genai_async_client.models.generate_content.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "capacity"
+ assert report.retryable is False
diff --git a/tests/unit/pipelex/plugins/huggingface/test_data.py b/tests/unit/pipelex/plugins/huggingface/test_data.py
new file mode 100644
index 000000000..83e6d871d
--- /dev/null
+++ b/tests/unit/pipelex/plugins/huggingface/test_data.py
@@ -0,0 +1,72 @@
+"""Test data for HuggingFace worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class HuggingFaceErrorHandlingTestData:
+ """Test cases for HuggingFace ImgGen worker SDK exception handling.
+
+ Each tuple: (topic, status_code_or_none, message, expected_category, expected_message_substring)
+ """
+
+ HF_HTTP_ERROR_CASES: ClassVar[list[tuple[str, int | None, str, InferenceErrorCategory, str]]] = [
+ (
+ "rate_limit_429",
+ 429,
+ "Rate limit exceeded",
+ InferenceErrorCategory.TRANSIENT,
+ "rate limit",
+ ),
+ (
+ "quota_exhausted_402",
+ 402,
+ "Quota exhausted",
+ InferenceErrorCategory.CAPACITY,
+ "quota",
+ ),
+ (
+ "auth_error_401",
+ 401,
+ "Invalid token",
+ InferenceErrorCategory.CONFIGURATION,
+ "authentication error",
+ ),
+ (
+ "forbidden_403",
+ 403,
+ "Access denied",
+ InferenceErrorCategory.CONFIGURATION,
+ "authentication error",
+ ),
+ (
+ "bad_request_400",
+ 400,
+ "Invalid parameters",
+ InferenceErrorCategory.CONTENT,
+ "bad request",
+ ),
+ (
+ "unknown_status_none",
+ None,
+ "Unknown error occurred",
+ InferenceErrorCategory.TRANSIENT,
+ "api error",
+ ),
+ (
+ "server_error_503",
+ 503,
+ "Service unavailable",
+ InferenceErrorCategory.TRANSIENT,
+ "api error",
+ ),
+ ]
+
+ TIMEOUT_CASES: ClassVar[list[tuple[str, InferenceErrorCategory, str]]] = [
+ (
+ "inference_timeout",
+ InferenceErrorCategory.TRANSIENT,
+ "timed out",
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/huggingface/test_hf_worker_error_handling.py b/tests/unit/pipelex/plugins/huggingface/test_hf_worker_error_handling.py
new file mode 100644
index 000000000..1b7dbdd2a
--- /dev/null
+++ b/tests/unit/pipelex/plugins/huggingface/test_hf_worker_error_handling.py
@@ -0,0 +1,143 @@
+"""Tests for HuggingFace ImgGen worker SDK exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import pytest
+from huggingface_hub.errors import HfHubHTTPError, InferenceTimeoutError
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import ImgGenGenerationError, InferenceErrorCategory
+from pipelex.plugins.huggingface.huggingface_img_gen_worker import HuggingFaceImgGenWorker
+from tests.unit.pipelex.plugins.huggingface.test_data import HuggingFaceErrorHandlingTestData
+
+
+def _make_hf_http_error(status_code: int | None, message: str) -> HfHubHTTPError:
+ """Create a minimal HfHubHTTPError for testing with optional status code."""
+ exc = HfHubHTTPError(message)
+ if status_code is not None:
+ # HfHubHTTPError stores the response on .response attribute
+ mock_response = type("MockResponse", (), {"status_code": status_code})()
+ exc.response = mock_response # type: ignore[attr-defined]
+ else:
+ exc.response = None # type: ignore[attr-defined]
+ return exc
+
+
+def _make_inference_timeout_error(message: str = "Inference timed out") -> InferenceTimeoutError:
+ """Create a minimal InferenceTimeoutError for testing."""
+ return InferenceTimeoutError(message)
+
+
+def _make_hf_img_gen_worker(mocker: MockerFixture) -> HuggingFaceImgGenWorker:
+ """Create a minimal HuggingFaceImgGenWorker with mocked internals."""
+ worker = object.__new__(HuggingFaceImgGenWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-hf-model"
+ mock_model.model_id = "stabilityai/stable-diffusion-xl-base-1.0"
+ mock_model.name = "sdxl-base"
+ mock_model.rules = mocker.MagicMock()
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ mock_client.text_to_image = mocker.AsyncMock()
+ worker.hf_async_client = mock_client
+
+ return worker
+
+
+def _make_img_gen_job(mocker: MockerFixture) -> Any:
+ """Create a mock ImgGen job."""
+ job = mocker.MagicMock()
+ job.img_gen_prompt.positive_text = "a sunset over mountains"
+ job.job_params.output_format = None
+ job.job_report.img_gen_tokens_usage = None
+ return job
+
+
[email protected](loop_scope="class")
+class TestHuggingFaceWorkerErrorHandling:
+ """Tests for HuggingFace ImgGen worker SDK exception handling and error categorization."""
+
+ @pytest.mark.parametrize(
+ ("_topic", "status_code", "message", "expected_category", "expected_message_substring"),
+ HuggingFaceErrorHandlingTestData.HF_HTTP_ERROR_CASES,
+ )
+ async def test_hf_http_error_categorization(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ status_code: int | None,
+ message: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """HfHubHTTPError is caught and categorized correctly based on status code."""
+ worker = _make_hf_img_gen_worker(mocker)
+ sdk_exc = _make_hf_http_error(status_code, message)
+ worker.hf_async_client.text_to_image.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mocker.patch(
+ "pipelex.plugins.huggingface.huggingface_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test", "model": "test-model-id"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._generate_single_image(img_gen_job=_make_img_gen_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ @pytest.mark.parametrize(
+ ("_topic", "expected_category", "expected_message_substring"),
+ HuggingFaceErrorHandlingTestData.TIMEOUT_CASES,
+ )
+ async def test_inference_timeout_is_transient(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """InferenceTimeoutError is caught and categorized as TRANSIENT."""
+ worker = _make_hf_img_gen_worker(mocker)
+ sdk_exc = _make_inference_timeout_error()
+ worker.hf_async_client.text_to_image.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mocker.patch(
+ "pipelex.plugins.huggingface.huggingface_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test", "model": "test-model-id"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._generate_single_image(img_gen_job=_make_img_gen_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ async def test_error_report_includes_category(self, mocker: MockerFixture) -> None:
+ """to_error_report() includes error_category and retryable from the exception."""
+ worker = _make_hf_img_gen_worker(mocker)
+ sdk_exc = _make_hf_http_error(401, "Invalid token")
+ worker.hf_async_client.text_to_image.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mocker.patch(
+ "pipelex.plugins.huggingface.huggingface_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "test", "model": "test-model-id"},
+ )
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._generate_single_image(img_gen_job=_make_img_gen_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "configuration"
+ assert report.retryable is False
+ assert report.error_type == "ImgGenGenerationError"
diff --git a/tests/unit/pipelex/plugins/linkup/test_data.py b/tests/unit/pipelex/plugins/linkup/test_data.py
new file mode 100644
index 000000000..e8a2bd95c
--- /dev/null
+++ b/tests/unit/pipelex/plugins/linkup/test_data.py
@@ -0,0 +1,157 @@
+"""Test data for Linkup worker error handling tests."""
+
+from typing import ClassVar
+
+from linkup import (
+ LinkupAuthenticationError,
+ LinkupFailedFetchError,
+ LinkupFetchResponseTooLargeError,
+ LinkupFetchUrlIsFileError,
+ LinkupInsufficientCreditError,
+ LinkupInvalidRequestError,
+ LinkupNoResultError,
+ LinkupTimeoutError,
+ LinkupTooManyRequestsError,
+ LinkupUnknownError,
+)
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class LinkupExtractErrorHandlingTestData:
+ """Test cases for Linkup extract worker exception handling.
+
+ Each tuple: (topic, exception_class, exception_message, expected_category, expected_message_substring)
+ """
+
+ EXTRACT_ERROR_CASES: ClassVar[list[tuple[str, type[Exception], str, InferenceErrorCategory, str]]] = [
+ (
+ "auth_error",
+ LinkupAuthenticationError,
+ "Invalid API key",
+ InferenceErrorCategory.CONFIGURATION,
+ "authentication",
+ ),
+ (
+ "insufficient_credit",
+ LinkupInsufficientCreditError,
+ "No credits remaining",
+ InferenceErrorCategory.CAPACITY,
+ "credits exhausted",
+ ),
+ (
+ "rate_limit",
+ LinkupTooManyRequestsError,
+ "Too many requests",
+ InferenceErrorCategory.TRANSIENT,
+ "rate limit",
+ ),
+ (
+ "timeout",
+ LinkupTimeoutError,
+ "Request timed out",
+ InferenceErrorCategory.TRANSIENT,
+ "timed out",
+ ),
+ (
+ "invalid_request",
+ LinkupInvalidRequestError,
+ "Bad URL format",
+ InferenceErrorCategory.CONTENT,
+ "invalid request",
+ ),
+ (
+ "response_too_large",
+ LinkupFetchResponseTooLargeError,
+ "Response exceeds size limit",
+ InferenceErrorCategory.CONTENT,
+ "fetch error",
+ ),
+ (
+ "url_is_file",
+ LinkupFetchUrlIsFileError,
+ "URL points to a file",
+ InferenceErrorCategory.CONTENT,
+ "fetch error",
+ ),
+ (
+ "failed_fetch",
+ LinkupFailedFetchError,
+ "Could not fetch URL",
+ InferenceErrorCategory.TRANSIENT,
+ "linkup error",
+ ),
+ (
+ "no_result",
+ LinkupNoResultError,
+ "No results found",
+ InferenceErrorCategory.TRANSIENT,
+ "linkup error",
+ ),
+ (
+ "unknown_error",
+ LinkupUnknownError,
+ "Something went wrong",
+ InferenceErrorCategory.TRANSIENT,
+ "linkup error",
+ ),
+ ]
+
+
+class LinkupSearchErrorHandlingTestData:
+ """Test cases for Linkup search worker exception handling.
+
+ Each tuple: (topic, exception_class, exception_message, expected_category, expected_message_substring)
+ """
+
+ SEARCH_ERROR_CASES: ClassVar[list[tuple[str, type[Exception], str, InferenceErrorCategory, str]]] = [
+ (
+ "auth_error",
+ LinkupAuthenticationError,
+ "Invalid API key",
+ InferenceErrorCategory.CONFIGURATION,
+ "authentication",
+ ),
+ (
+ "insufficient_credit",
+ LinkupInsufficientCreditError,
+ "No credits remaining",
+ InferenceErrorCategory.CAPACITY,
+ "credits exhausted",
+ ),
+ (
+ "rate_limit",
+ LinkupTooManyRequestsError,
+ "Too many requests",
+ InferenceErrorCategory.TRANSIENT,
+ "rate limit",
+ ),
+ (
+ "timeout",
+ LinkupTimeoutError,
+ "Request timed out",
+ InferenceErrorCategory.TRANSIENT,
+ "timed out",
+ ),
+ (
+ "invalid_request",
+ LinkupInvalidRequestError,
+ "Bad query format",
+ InferenceErrorCategory.CONTENT,
+ "invalid request",
+ ),
+ (
+ "no_result",
+ LinkupNoResultError,
+ "No results found",
+ InferenceErrorCategory.TRANSIENT,
+ "linkup error",
+ ),
+ (
+ "unknown_error",
+ LinkupUnknownError,
+ "Something went wrong",
+ InferenceErrorCategory.TRANSIENT,
+ "linkup error",
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/linkup/test_linkup_worker_error_handling.py b/tests/unit/pipelex/plugins/linkup/test_linkup_worker_error_handling.py
new file mode 100644
index 000000000..1ee476126
--- /dev/null
+++ b/tests/unit/pipelex/plugins/linkup/test_linkup_worker_error_handling.py
@@ -0,0 +1,168 @@
+"""Tests for Linkup extract and search worker exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import pytest
+from linkup import LinkupAuthenticationError, LinkupTooManyRequestsError
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import ExtractJobFailureError, InferenceErrorCategory, SearchJobFailureError
+from pipelex.plugins.linkup.linkup_extract_worker import LinkupExtractWorker
+from pipelex.plugins.linkup.linkup_search_worker import LinkupSearchWorker
+from tests.unit.pipelex.plugins.linkup.test_data import LinkupExtractErrorHandlingTestData, LinkupSearchErrorHandlingTestData
+
+
+def _make_linkup_extract_worker(mocker: MockerFixture) -> LinkupExtractWorker:
+ """Create a minimal LinkupExtractWorker with mocked internals."""
+ worker = object.__new__(LinkupExtractWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-linkup-extract"
+ mock_model.model_id = "linkup-fetch"
+ mock_model.name = "linkup-fetch"
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ mock_client.async_fetch = mocker.AsyncMock()
+ setattr(worker, "_linkup_client", mock_client) # noqa: B010
+
+ return worker
+
+
+def _make_linkup_search_worker(mocker: MockerFixture) -> LinkupSearchWorker:
+ """Create a minimal LinkupSearchWorker with mocked internals."""
+ worker = object.__new__(LinkupSearchWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-linkup-search"
+ mock_model.model_id = "linkup/standard"
+ mock_model.name = "linkup-standard"
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ mock_client.async_search = mocker.AsyncMock()
+ setattr(worker, "_linkup_client", mock_client) # noqa: B010
+
+ return worker
+
+
+def _get_linkup_client(worker: Any) -> Any:
+ """Access the _linkup_client attribute via object.__getattribute__ to avoid linter complaints."""
+ return worker._linkup_client # noqa: SLF001
+
+
+def _make_extract_job(mocker: MockerFixture) -> Any:
+ """Create a mock ExtractJob for Linkup extract worker."""
+ job = mocker.MagicMock()
+ job.extract_input.document_uri = "https://example.com/page"
+ job.extract_input.image_uri = None
+ job.job_params.max_nb_images = None
+ job.job_params.render_js = False
+ job.job_params.include_raw_html = False
+ job.job_report.extract_tokens_usage = None
+ return job
+
+
+def _make_search_job(mocker: MockerFixture) -> Any:
+ """Create a mock SearchJob for Linkup search worker."""
+ job = mocker.MagicMock()
+ job.query = "test search query"
+ job.job_params.search_setting.include_images = False
+ job.job_params.search_setting.include_inline_citations = False
+ job.job_params.search_setting.max_results = 10
+ job.job_params.include_domains = None
+ job.job_params.exclude_domains = None
+ job.job_params.from_date = None
+ job.job_params.to_date = None
+ job.job_report.search_tokens_usage = None
+ return job
+
+
[email protected](loop_scope="class")
+class TestLinkupWorkerErrorHandling:
+ """Tests for Linkup extract and search worker exception handling and error categorization."""
+
+ # ---- Extract worker tests ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "exception_class", "exception_message", "expected_category", "expected_message_substring"),
+ LinkupExtractErrorHandlingTestData.EXTRACT_ERROR_CASES,
+ )
+ async def test_extract_error_categorization(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ exception_class: type[Exception],
+ exception_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """Linkup extract errors are caught and categorized correctly."""
+ worker = _make_linkup_extract_worker(mocker)
+ sdk_exc = exception_class(exception_message)
+ _get_linkup_client(worker).async_fetch.side_effect = sdk_exc
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_pages(extract_job=_make_extract_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ # ---- Search worker tests ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "exception_class", "exception_message", "expected_category", "expected_message_substring"),
+ LinkupSearchErrorHandlingTestData.SEARCH_ERROR_CASES,
+ )
+ async def test_search_error_categorization(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ exception_class: type[Exception],
+ exception_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """Linkup search errors are caught and categorized correctly."""
+ worker = _make_linkup_search_worker(mocker)
+ sdk_exc = exception_class(exception_message)
+ _get_linkup_client(worker).async_search.side_effect = sdk_exc
+
+ with pytest.raises(SearchJobFailureError) as exc_info:
+ await worker._search_sourced_answer(search_job=_make_search_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ # ---- Error report tests ----
+
+ async def test_extract_error_report_for_auth(self, mocker: MockerFixture) -> None:
+ """to_error_report() for authentication error has CONFIGURATION category."""
+ worker = _make_linkup_extract_worker(mocker)
+ sdk_exc = LinkupAuthenticationError("Invalid API key")
+ _get_linkup_client(worker).async_fetch.side_effect = sdk_exc
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_pages(extract_job=_make_extract_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "configuration"
+ assert report.retryable is False
+ assert report.error_type == "ExtractJobFailureError"
+
+ async def test_search_error_report_for_rate_limit(self, mocker: MockerFixture) -> None:
+ """to_error_report() for rate limit error has TRANSIENT category and is retryable."""
+ worker = _make_linkup_search_worker(mocker)
+ sdk_exc = LinkupTooManyRequestsError("Too many requests")
+ _get_linkup_client(worker).async_search.side_effect = sdk_exc
+
+ with pytest.raises(SearchJobFailureError) as exc_info:
+ await worker._search_sourced_answer(search_job=_make_search_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
diff --git a/tests/unit/pipelex/plugins/mistral/test_data.py b/tests/unit/pipelex/plugins/mistral/test_data.py
new file mode 100644
index 000000000..f6d3e176c
--- /dev/null
+++ b/tests/unit/pipelex/plugins/mistral/test_data.py
@@ -0,0 +1,144 @@
+"""Test data for Mistral worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class MistralLLMErrorHandlingTestData:
+ """Test cases for Mistral LLM worker SDK exception handling.
+
+ Each tuple: (topic, status_code, error_message, expected_category, expected_user_action_substring_or_none)
+ """
+
+ SDK_ERROR_CASES: ClassVar[list[tuple[str, int, str, InferenceErrorCategory, str | None]]] = [
+ (
+ "payment_required_402",
+ 402,
+ "Payment required: insufficient credits",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "rate_limit_quota_429",
+ 429,
+ "Rate limit reached: your account quota has been exceeded",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "rate_limit_generic_429",
+ 429,
+ "Rate limit exceeded. Please retry after 20s",
+ InferenceErrorCategory.TRANSIENT,
+ "retry",
+ ),
+ (
+ "auth_error_401",
+ 401,
+ "Invalid API key provided",
+ InferenceErrorCategory.CONFIGURATION,
+ None,
+ ),
+ (
+ "forbidden_403",
+ 403,
+ "You do not have access to this resource",
+ InferenceErrorCategory.CONFIGURATION,
+ None,
+ ),
+ (
+ "model_not_found_404",
+ 404,
+ "Model mistral-unknown not found",
+ InferenceErrorCategory.CONFIGURATION,
+ None,
+ ),
+ (
+ "bad_request_content_policy_400",
+ 400,
+ "Your request was rejected due to content_policy_violation",
+ InferenceErrorCategory.CONTENT,
+ "safety filters",
+ ),
+ (
+ "bad_request_generic_400",
+ 400,
+ "Invalid parameter: temperature must be between 0 and 1",
+ InferenceErrorCategory.CONTENT,
+ None,
+ ),
+ (
+ "server_error_500",
+ 500,
+ "Internal server error",
+ InferenceErrorCategory.TRANSIENT,
+ None,
+ ),
+ ]
+
+
+class MistralExtractErrorHandlingTestData:
+ """Test cases for Mistral Extract worker SDK exception handling.
+
+ Each tuple: (topic, status_code, error_message, expected_category, expected_user_action_substring_or_none)
+ """
+
+ SDK_ERROR_CASES: ClassVar[list[tuple[str, int, str, InferenceErrorCategory, str | None]]] = [
+ (
+ "payment_required_402",
+ 402,
+ "Payment required: insufficient credits",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "rate_limit_quota_429",
+ 429,
+ "Rate limit reached: your account quota has been exceeded",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "rate_limit_generic_429",
+ 429,
+ "Rate limit exceeded. Please retry after 20s",
+ InferenceErrorCategory.TRANSIENT,
+ "retry",
+ ),
+ (
+ "auth_error_401",
+ 401,
+ "Invalid API key provided",
+ InferenceErrorCategory.CONFIGURATION,
+ None,
+ ),
+ (
+ "model_not_found_404",
+ 404,
+ "Model mistral-ocr-unknown not found",
+ InferenceErrorCategory.CONFIGURATION,
+ None,
+ ),
+ (
+ "bad_request_content_policy_400",
+ 400,
+ "Your request was rejected due to content_policy_violation",
+ InferenceErrorCategory.CONTENT,
+ "safety filters",
+ ),
+ (
+ "bad_request_generic_400",
+ 400,
+ "Invalid parameter: unsupported document format",
+ InferenceErrorCategory.CONTENT,
+ None,
+ ),
+ (
+ "server_error_503",
+ 503,
+ "Service temporarily unavailable",
+ InferenceErrorCategory.TRANSIENT,
+ None,
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/mistral/test_mistral_worker_error_handling.py b/tests/unit/pipelex/plugins/mistral/test_mistral_worker_error_handling.py
new file mode 100644
index 000000000..230e501d7
--- /dev/null
+++ b/tests/unit/pipelex/plugins/mistral/test_mistral_worker_error_handling.py
@@ -0,0 +1,204 @@
+"""Tests for Mistral worker SDK exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import httpx
+import pytest
+from mistralai import MistralError
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import (
+ ExtractJobFailureError,
+ InferenceErrorCategory,
+ LLMCompletionError,
+)
+from pipelex.plugins.mistral.mistral_extract_worker import MistralExtractWorker
+from pipelex.plugins.mistral.mistral_llm_worker import MistralLLMWorker
+from tests.unit.pipelex.plugins.mistral.test_data import (
+ MistralExtractErrorHandlingTestData,
+ MistralLLMErrorHandlingTestData,
+)
+
+
+def _mock_httpx_response(status_code: int) -> httpx.Response:
+ """Create a minimal httpx.Response for constructing MistralError."""
+ request = httpx.Request("POST", "https://api.mistral.ai/v1/test")
+ return httpx.Response(status_code=status_code, request=request)
+
+
+def _make_mistral_error(status_code: int, message: str) -> MistralError:
+ """Create a MistralError with the given status code and message."""
+ return MistralError(message, raw_response=_mock_httpx_response(status_code))
+
+
+def _make_mistral_llm_worker(mocker: MockerFixture) -> MistralLLMWorker:
+ """Create a minimal MistralLLMWorker with mocked internals."""
+ worker = object.__new__(MistralLLMWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "mistral-large"
+ mock_model.name = "mistral-large"
+ mock_model.thinking_mode = None
+ mock_model.max_tokens = 4096
+ worker.inference_model = mock_model
+ worker.default_max_tokens = 4096
+
+ mock_client = mocker.MagicMock()
+ mock_client.chat.complete_async = mocker.AsyncMock()
+ worker.mistral_client_for_text = mock_client
+
+ mock_factory = mocker.MagicMock()
+ mock_factory.make_simple_messages = mocker.AsyncMock(return_value=[])
+ worker.mistral_factory = mock_factory
+
+ return worker
+
+
+def _make_mistral_extract_worker(mocker: MockerFixture) -> MistralExtractWorker:
+ """Create a minimal MistralExtractWorker with mocked internals."""
+ worker = object.__new__(MistralExtractWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "mistral-ocr"
+ mock_model.name = "mistral-ocr"
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ mock_client.ocr.process_async = mocker.AsyncMock()
+ worker.mistral_client = mock_client
+
+ return worker
+
+
+def _make_llm_job(mocker: MockerFixture) -> Any:
+ """Create a mock LLM job."""
+ job = mocker.MagicMock()
+ job.applied_job_params = None
+ job.job_params.temperature = 0.5
+ job.job_params.max_tokens = None
+ job.job_params.seed = None
+ job.job_params.reasoning_effort = None
+ job.job_params.reasoning_budget = None
+ job.job_report.llm_tokens_usage = None
+ job.llm_prompt.system_text = "system"
+ return job
+
+
[email protected](loop_scope="class")
+class TestMistralWorkerErrorHandling:
+ """Tests for Mistral worker SDK exception handling and error categorization."""
+
+ # ---- MistralLLMWorker error handling ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "status_code", "error_message", "expected_category", "expected_action_substring"),
+ MistralLLMErrorHandlingTestData.SDK_ERROR_CASES,
+ )
+ async def test_llm_worker_error_classification(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ status_code: int,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_action_substring: str | None,
+ ) -> None:
+ """MistralError is caught and categorized correctly by the LLM worker."""
+ worker = _make_mistral_llm_worker(mocker)
+ sdk_exc = _make_mistral_error(status_code, error_message)
+ worker.mistral_client_for_text.chat.complete_async.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ if expected_action_substring:
+ assert exc_info.value.user_action is not None
+ assert expected_action_substring in exc_info.value.user_action.lower()
+
+ # ---- MistralExtractWorker error handling ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "status_code", "error_message", "expected_category", "expected_action_substring"),
+ MistralExtractErrorHandlingTestData.SDK_ERROR_CASES,
+ )
+ async def test_extract_worker_error_classification(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ status_code: int,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_action_substring: str | None,
+ ) -> None:
+ """MistralError is caught and categorized correctly by the Extract worker."""
+ worker = _make_mistral_extract_worker(mocker)
+ sdk_exc = _make_mistral_error(status_code, error_message)
+ worker.mistral_client.ocr.process_async.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ # Patch the static factory method that runs before the API call
+ mocker.patch(
+ "pipelex.plugins.mistral.mistral_extract_worker.MistralFactory.make_mistral_image_url_chunk_from_uri",
+ return_value={"type": "image_url", "image_url": "https://example.com/test.png"},
+ )
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_page_from_image(image_uri="https://example.com/test.png") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ if expected_action_substring:
+ assert exc_info.value.user_action is not None
+ assert expected_action_substring in exc_info.value.user_action.lower()
+
+ # ---- to_error_report() integration ----
+
+ async def test_llm_error_report_transient_is_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() for TRANSIENT LLM errors has retryable=True."""
+ worker = _make_mistral_llm_worker(mocker)
+ sdk_exc = _make_mistral_error(429, "Rate limit exceeded")
+ worker.mistral_client_for_text.chat.complete_async.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
+ assert report.error_type == "LLMCompletionError"
+
+ async def test_llm_error_report_capacity_not_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() for CAPACITY LLM errors has retryable=False."""
+ worker = _make_mistral_llm_worker(mocker)
+ sdk_exc = _make_mistral_error(402, "Payment required: insufficient credits")
+ worker.mistral_client_for_text.chat.complete_async.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "capacity"
+ assert report.retryable is False
+
+ async def test_extract_error_report_includes_category(self, mocker: MockerFixture) -> None:
+ """to_error_report() for Extract errors includes error_category."""
+ worker = _make_mistral_extract_worker(mocker)
+ sdk_exc = _make_mistral_error(401, "Invalid API key")
+ worker.mistral_client.ocr.process_async.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ mocker.patch(
+ "pipelex.plugins.mistral.mistral_extract_worker.MistralFactory.make_mistral_image_url_chunk_from_uri",
+ return_value={"type": "image_url", "image_url": "https://example.com/test.png"},
+ )
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_page_from_image(image_uri="https://example.com/test.png") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "configuration"
+ assert report.retryable is False
diff --git a/tests/unit/pipelex/plugins/openai/test_data.py b/tests/unit/pipelex/plugins/openai/test_data.py
new file mode 100644
index 000000000..0b938082a
--- /dev/null
+++ b/tests/unit/pipelex/plugins/openai/test_data.py
@@ -0,0 +1,62 @@
+"""Test data for OpenAI worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class OpenAIErrorHandlingTestData:
+ """Test cases for OpenAI worker SDK exception handling.
+
+ Each tuple: (topic, error_message, expected_category, expected_user_action_substring_or_none)
+ """
+
+ RATE_LIMIT_CASES: ClassVar[list[tuple[str, str, InferenceErrorCategory, str | None]]] = [
+ (
+ "generic_rate_limit",
+ "Rate limit exceeded. Please retry after 20s",
+ InferenceErrorCategory.TRANSIENT,
+ "retry",
+ ),
+ (
+ "quota_insufficient",
+ "Error: insufficient_quota - you have exceeded your billing limit",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ (
+ "quota_exceeded",
+ "You exceeded your current quota, please check your plan",
+ InferenceErrorCategory.CAPACITY,
+ "billing",
+ ),
+ ]
+
+ TIMEOUT_CASES: ClassVar[list[tuple[str, str, InferenceErrorCategory]]] = [
+ (
+ "api_timeout",
+ "Request timed out",
+ InferenceErrorCategory.TRANSIENT,
+ ),
+ ]
+
+ BAD_REQUEST_CASES: ClassVar[list[tuple[str, str, InferenceErrorCategory, str | None]]] = [
+ (
+ "generic_bad_request",
+ "Invalid parameter: temperature must be between 0 and 2",
+ InferenceErrorCategory.CONTENT,
+ None,
+ ),
+ (
+ "content_policy",
+ "Your request was rejected due to content_policy_violation",
+ InferenceErrorCategory.CONTENT,
+ "safety filters",
+ ),
+ (
+ "blocked_by_safety",
+ "The response was blocked by safety systems",
+ InferenceErrorCategory.CONTENT,
+ "safety filters",
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/openai/test_openai_worker_error_handling.py b/tests/unit/pipelex/plugins/openai/test_openai_worker_error_handling.py
new file mode 100644
index 000000000..05d997655
--- /dev/null
+++ b/tests/unit/pipelex/plugins/openai/test_openai_worker_error_handling.py
@@ -0,0 +1,352 @@
+"""Tests for OpenAI worker SDK exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import httpx
+import openai
+import pytest
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import (
+ ImgGenGenerationError,
+ InferenceErrorCategory,
+ LLMCompletionError,
+)
+from pipelex.plugins.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker
+from pipelex.plugins.openai.openai_img_gen_worker import OpenAIImgGenWorker
+from pipelex.plugins.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker
+from tests.unit.pipelex.plugins.openai.test_data import OpenAIErrorHandlingTestData
+
+
+def _mock_httpx_response(status_code: int = 429) -> httpx.Response:
+ """Create a minimal httpx.Response for constructing SDK exceptions."""
+ request = httpx.Request("POST", "https://api.openai.com/v1/test")
+ return httpx.Response(status_code=status_code, request=request)
+
+
+def _make_openai_rate_limit_error(message: str) -> openai.RateLimitError:
+ return openai.RateLimitError(message, response=_mock_httpx_response(429), body=None)
+
+
+def _make_openai_bad_request_error(message: str) -> openai.BadRequestError:
+ return openai.BadRequestError(message, response=_mock_httpx_response(400), body=None)
+
+
+def _make_openai_timeout_error() -> openai.APITimeoutError:
+ request = httpx.Request("POST", "https://api.openai.com/v1/test")
+ return openai.APITimeoutError(request=request)
+
+
+def _make_openai_connection_error(message: str = "Connection error.") -> openai.APIConnectionError:
+ request = httpx.Request("POST", "https://api.openai.com/v1/test")
+ return openai.APIConnectionError(message=message, request=request)
+
+
+def _make_openai_not_found_error(message: str) -> openai.NotFoundError:
+ return openai.NotFoundError(message, response=_mock_httpx_response(404), body=None)
+
+
+def _make_openai_auth_error(message: str) -> openai.AuthenticationError:
+ return openai.AuthenticationError(message, response=_mock_httpx_response(401), body=None)
+
+
+def _make_completions_llm_worker(mocker: MockerFixture) -> OpenAICompletionsLLMWorker:
+ """Create a minimal OpenAICompletionsLLMWorker with mocked internals."""
+ worker = object.__new__(OpenAICompletionsLLMWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "gpt-4o"
+ mock_model.name = "gpt-4o"
+ mock_model.thinking_mode = None
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ mock_client.chat.completions.create = mocker.AsyncMock()
+ worker.openai_client_for_text = mock_client
+
+ mock_factory = mocker.MagicMock()
+ mock_factory.make_simple_messages = mocker.AsyncMock(return_value=[])
+ mock_factory.make_extras = mocker.MagicMock(return_value=({}, {}))
+ worker.openai_completions_factory = mock_factory
+
+ return worker
+
+
+def _make_responses_llm_worker(mocker: MockerFixture) -> OpenAIResponsesLLMWorker:
+ """Create a minimal OpenAIResponsesLLMWorker with mocked internals."""
+ worker = object.__new__(OpenAIResponsesLLMWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "gpt-4o"
+ mock_model.name = "gpt-4o"
+ mock_model.thinking_mode = None
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ mock_client.responses.create = mocker.AsyncMock()
+ worker.openai_client_for_responses = mock_client
+
+ mock_factory = mocker.MagicMock()
+ mock_factory.make_input_items = mocker.AsyncMock(return_value=[])
+ mock_factory.make_extras = mocker.MagicMock(return_value=({}, {}))
+ worker.openai_responses_factory = mock_factory
+
+ return worker
+
+
+def _make_img_gen_worker(mocker: MockerFixture) -> OpenAIImgGenWorker:
+ """Create a minimal OpenAIImgGenWorker with mocked internals."""
+ worker = object.__new__(OpenAIImgGenWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "gpt-image-1"
+ mock_model.name = "gpt-image-1"
+ mock_model.rules = mocker.MagicMock()
+ worker.inference_model = mock_model
+
+ mock_client = mocker.MagicMock()
+ mock_client.images.generate = mocker.AsyncMock()
+ mock_client.images.edit = mocker.AsyncMock()
+ worker.openai_client = mock_client
+
+ mocker.patch(
+ "pipelex.plugins.openai.openai_img_gen_worker.ImgGenArgsFactory.make_args_for_model",
+ new_callable=mocker.AsyncMock,
+ return_value={"prompt": "a cute cat", "model": "gpt-image-1"},
+ )
+
+ return worker
+
+
+def _make_llm_job(mocker: MockerFixture) -> Any:
+ """Create a mock LLM job."""
+ job = mocker.MagicMock()
+ job.applied_job_params = None
+ job.job_params.temperature = 0.5
+ job.job_params.max_tokens = None
+ job.job_params.seed = None
+ job.job_params.reasoning_effort = None
+ job.job_params.reasoning_budget = None
+ job.job_report.llm_tokens_usage = None
+ job.llm_prompt.system_text = "system"
+ return job
+
+
+def _make_img_gen_job(mocker: MockerFixture) -> Any:
+ """Create a mock ImgGen job."""
+ job = mocker.MagicMock()
+ job.job_params.aspect_ratio = None
+ job.job_params.output_format = None
+ job.job_params.is_moderated = True
+ job.job_params.background = None
+ job.job_params.quality = None
+ job.img_gen_prompt.positive_text = "a cute cat"
+ job.job_report.img_gen_tokens_usage = None
+ return job
+
+
[email protected](loop_scope="class")
+class TestOpenAIWorkerErrorHandling:
+ """Tests for OpenAI worker SDK exception handling and error categorization."""
+
+ # ---- RateLimitError tests (completions LLM worker) ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "expected_category", "expected_action_substring"),
+ OpenAIErrorHandlingTestData.RATE_LIMIT_CASES,
+ )
+ async def test_completions_llm_rate_limit(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_action_substring: str | None,
+ ) -> None:
+ """RateLimitError is caught and categorized correctly (TRANSIENT or CAPACITY)."""
+ worker = _make_completions_llm_worker(mocker)
+ sdk_exc = _make_openai_rate_limit_error(error_message)
+ worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ if expected_action_substring:
+ assert exc_info.value.user_action is not None
+ assert expected_action_substring in exc_info.value.user_action.lower()
+
+ # ---- APITimeoutError tests ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "_error_message", "expected_category"),
+ OpenAIErrorHandlingTestData.TIMEOUT_CASES,
+ )
+ async def test_completions_llm_timeout(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ _error_message: str,
+ expected_category: InferenceErrorCategory,
+ ) -> None:
+ """APITimeoutError is caught and categorized as TRANSIENT."""
+ worker = _make_completions_llm_worker(mocker)
+ sdk_exc = _make_openai_timeout_error()
+ worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+
+ # ---- BadRequestError tests with content policy detection ----
+
+ @pytest.mark.parametrize(
+ ("_topic", "error_message", "expected_category", "expected_action_substring"),
+ OpenAIErrorHandlingTestData.BAD_REQUEST_CASES,
+ )
+ async def test_completions_llm_bad_request(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ error_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_action_substring: str | None,
+ ) -> None:
+ """BadRequestError is caught with content policy detection."""
+ worker = _make_completions_llm_worker(mocker)
+ sdk_exc = _make_openai_bad_request_error(error_message)
+ worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ if expected_action_substring:
+ assert exc_info.value.user_action is not None
+ assert expected_action_substring in exc_info.value.user_action.lower()
+
+ # ---- Existing exception categories ----
+
+ async def test_completions_llm_not_found_has_configuration_category(self, mocker: MockerFixture) -> None:
+ """NotFoundError should be categorized as CONFIGURATION."""
+ worker = _make_completions_llm_worker(mocker)
+ sdk_exc = _make_openai_not_found_error("model gpt-99 not found")
+ worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CONFIGURATION
+ assert exc_info.value.__cause__ is sdk_exc
+
+ async def test_completions_llm_connection_error_has_transient_category(self, mocker: MockerFixture) -> None:
+ """APIConnectionError should be categorized as TRANSIENT."""
+ worker = _make_completions_llm_worker(mocker)
+ sdk_exc = _make_openai_connection_error("Connection refused")
+ worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.__cause__ is sdk_exc
+
+ async def test_completions_llm_auth_error_has_configuration_category(self, mocker: MockerFixture) -> None:
+ """AuthenticationError should be categorized as CONFIGURATION."""
+ worker = _make_completions_llm_worker(mocker)
+ sdk_exc = _make_openai_auth_error("Invalid API key")
+ worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CONFIGURATION
+ assert exc_info.value.__cause__ is sdk_exc
+
+ # ---- Responses LLM worker ----
+
+ async def test_responses_llm_rate_limit_transient(self, mocker: MockerFixture) -> None:
+ """Responses worker: generic rate limit is TRANSIENT."""
+ worker = _make_responses_llm_worker(mocker)
+ sdk_exc = _make_openai_rate_limit_error("Rate limit exceeded")
+ worker.openai_client_for_responses.responses.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.__cause__ is sdk_exc
+
+ async def test_responses_llm_rate_limit_quota(self, mocker: MockerFixture) -> None:
+ """Responses worker: quota exhaustion rate limit is CAPACITY."""
+ worker = _make_responses_llm_worker(mocker)
+ sdk_exc = _make_openai_rate_limit_error("insufficient_quota")
+ worker.openai_client_for_responses.responses.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CAPACITY
+
+ # ---- ImgGen worker ----
+
+ async def test_img_gen_rate_limit_transient(self, mocker: MockerFixture) -> None:
+ """ImgGen worker: generic rate limit raises ImgGenGenerationError with TRANSIENT."""
+ worker = _make_img_gen_worker(mocker)
+ sdk_exc = _make_openai_rate_limit_error("Rate limit exceeded")
+ worker.openai_client.images.generate.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._gen_image_list(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.TRANSIENT
+ assert exc_info.value.__cause__ is sdk_exc
+
+ async def test_img_gen_auth_error(self, mocker: MockerFixture) -> None:
+ """ImgGen worker: AuthenticationError raises ImgGenGenerationError with CONFIGURATION."""
+ worker = _make_img_gen_worker(mocker)
+ sdk_exc = _make_openai_auth_error("Invalid API key")
+ worker.openai_client.images.generate.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(ImgGenGenerationError) as exc_info:
+ await worker._gen_image_list(img_gen_job=_make_img_gen_job(mocker), nb_images=1) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is InferenceErrorCategory.CONFIGURATION
+ assert exc_info.value.__cause__ is sdk_exc
+
+ # ---- to_error_report() integration ----
+
+ async def test_error_report_includes_category_and_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() includes error_category and retryable from the exception."""
+ worker = _make_completions_llm_worker(mocker)
+ sdk_exc = _make_openai_rate_limit_error("Rate limit exceeded")
+ worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
+ assert report.error_type == "LLMCompletionError"
+
+ async def test_error_report_capacity_not_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() for CAPACITY errors has retryable=False."""
+ worker = _make_completions_llm_worker(mocker)
+ sdk_exc = _make_openai_rate_limit_error("insufficient_quota")
+ worker.openai_client_for_text.chat.completions.create.side_effect = sdk_exc # type: ignore[attr-defined] # pyright: ignore[reportAttributeAccessIssue]
+
+ with pytest.raises(LLMCompletionError) as exc_info:
+ await worker._gen_text(llm_job=_make_llm_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "capacity"
+ assert report.retryable is False
diff --git a/tests/unit/pipelex/plugins/pypdfium2/test_data.py b/tests/unit/pipelex/plugins/pypdfium2/test_data.py
new file mode 100644
index 000000000..8b400567b
--- /dev/null
+++ b/tests/unit/pipelex/plugins/pypdfium2/test_data.py
@@ -0,0 +1,43 @@
+"""Test data for pypdfium2 worker error handling tests."""
+
+from typing import ClassVar
+
+from pipelex.cogt.exceptions import InferenceErrorCategory
+
+
+class Pypdfium2ErrorHandlingTestData:
+ """Test cases for pypdfium2 extract worker exception handling.
+
+ Each tuple: (topic, exception_class, exception_message, expected_category, expected_message_substring)
+ """
+
+ EXTRACTION_ERROR_CASES: ClassVar[list[tuple[str, type[Exception], str, InferenceErrorCategory, str]]] = [
+ (
+ "file_not_found",
+ FileNotFoundError,
+ "No such file: /tmp/missing.pdf",
+ InferenceErrorCategory.CONFIGURATION,
+ "file not found",
+ ),
+ (
+ "invalid_pdf_format",
+ ValueError,
+ "Not a valid PDF file",
+ InferenceErrorCategory.CONTENT,
+ "invalid pdf format",
+ ),
+ (
+ "runtime_extraction_failure",
+ RuntimeError,
+ "Failed to parse PDF structure",
+ InferenceErrorCategory.CONTENT,
+ "extraction failed",
+ ),
+ (
+ "io_error_transient",
+ OSError,
+ "Permission denied: /tmp/locked.pdf",
+ InferenceErrorCategory.TRANSIENT,
+ "i/o error",
+ ),
+ ]
diff --git a/tests/unit/pipelex/plugins/pypdfium2/test_pypdfium2_worker_error_handling.py b/tests/unit/pipelex/plugins/pypdfium2/test_pypdfium2_worker_error_handling.py
new file mode 100644
index 000000000..2df36173a
--- /dev/null
+++ b/tests/unit/pipelex/plugins/pypdfium2/test_pypdfium2_worker_error_handling.py
@@ -0,0 +1,128 @@
+"""Tests for pypdfium2 extract worker exception handling and error categorization."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import pytest
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.cogt.exceptions import ExtractJobFailureError, InferenceErrorCategory
+from pipelex.plugins.pypdfium2.pypdfium2_worker import Pypdfium2Worker
+from tests.unit.pipelex.plugins.pypdfium2.test_data import Pypdfium2ErrorHandlingTestData
+
+
+def _make_pypdfium2_worker(mocker: MockerFixture) -> Pypdfium2Worker:
+ """Create a minimal Pypdfium2Worker with mocked internals."""
+ worker = object.__new__(Pypdfium2Worker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-pypdfium2"
+ mock_model.model_id = "pypdfium2"
+ mock_model.name = "pypdfium2"
+ worker.inference_model = mock_model
+ return worker
+
+
+def _make_extract_job(mocker: MockerFixture) -> Any:
+ """Create a mock ExtractJob for pypdfium2 worker."""
+ job = mocker.MagicMock()
+ job.extract_input.image_uri = None
+ job.extract_input.document_uri = "/tmp/test.pdf" # noqa: S108
+ job.job_params.max_nb_images = 0 # No images, simplifies the test
+ job.job_report.extract_tokens_usage = None
+ return job
+
+
[email protected](loop_scope="class")
+class TestPypdfium2WorkerErrorHandling:
+ """Tests for pypdfium2 extract worker exception handling and error categorization."""
+
+ @pytest.mark.parametrize(
+ ("_topic", "exception_class", "exception_message", "expected_category", "expected_message_substring"),
+ Pypdfium2ErrorHandlingTestData.EXTRACTION_ERROR_CASES,
+ )
+ async def test_extraction_error_categorization(
+ self,
+ mocker: MockerFixture,
+ _topic: str,
+ exception_class: type[Exception],
+ exception_message: str,
+ expected_category: InferenceErrorCategory,
+ expected_message_substring: str,
+ ) -> None:
+ """PDF extraction errors are caught and categorized correctly."""
+ worker = _make_pypdfium2_worker(mocker)
+ sdk_exc = exception_class(exception_message)
+
+ # Mock _resolve_pdf_uri to return a simple path (bypasses URI resolution)
+ mocker.patch.object(
+ worker,
+ "_resolve_pdf_uri",
+ new_callable=mocker.AsyncMock,
+ return_value="/tmp/test.pdf", # noqa: S108
+ )
+
+ # Mock pypdfium2_renderer methods to raise the exception
+ mocker.patch(
+ "pipelex.plugins.pypdfium2.pypdfium2_worker.pypdfium2_renderer.extract_text_from_pdf_pages",
+ new_callable=mocker.AsyncMock,
+ side_effect=sdk_exc,
+ )
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_pages(extract_job=_make_extract_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert exc_info.value.error_category is expected_category
+ assert exc_info.value.__cause__ is sdk_exc
+ assert expected_message_substring in exc_info.value.args[0].lower()
+
+ async def test_error_report_for_file_not_found(self, mocker: MockerFixture) -> None:
+ """to_error_report() for FileNotFoundError has CONFIGURATION category."""
+ worker = _make_pypdfium2_worker(mocker)
+ sdk_exc = FileNotFoundError("No such file: /tmp/missing.pdf")
+
+ mocker.patch.object(
+ worker,
+ "_resolve_pdf_uri",
+ new_callable=mocker.AsyncMock,
+ return_value="/tmp/missing.pdf", # noqa: S108
+ )
+ mocker.patch(
+ "pipelex.plugins.pypdfium2.pypdfium2_worker.pypdfium2_renderer.extract_text_from_pdf_pages",
+ new_callable=mocker.AsyncMock,
+ side_effect=sdk_exc,
+ )
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_pages(extract_job=_make_extract_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "configuration"
+ assert report.retryable is False
+ assert report.error_type == "ExtractJobFailureError"
+
+ async def test_error_report_for_os_error_is_retryable(self, mocker: MockerFixture) -> None:
+ """to_error_report() for OSError has TRANSIENT category and is retryable."""
+ worker = _make_pypdfium2_worker(mocker)
+ sdk_exc = OSError("Disk read error")
+
+ mocker.patch.object(
+ worker,
+ "_resolve_pdf_uri",
+ new_callable=mocker.AsyncMock,
+ return_value="/tmp/test.pdf", # noqa: S108
+ )
+ mocker.patch(
+ "pipelex.plugins.pypdfium2.pypdfium2_worker.pypdfium2_renderer.extract_text_from_pdf_pages",
+ new_callable=mocker.AsyncMock,
+ side_effect=sdk_exc,
+ )
+
+ with pytest.raises(ExtractJobFailureError) as exc_info:
+ await worker._extract_pages(extract_job=_make_extract_job(mocker)) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ report = exc_info.value.to_error_report()
+ assert report.error_category == "transient"
+ assert report.retryable is True
diff --git a/tests/unit/pipelex/reporting/test_reporting_event_emission.py b/tests/unit/pipelex/reporting/test_reporting_event_emission.py
new file mode 100644
index 000000000..e8cdff7c7
--- /dev/null
+++ b/tests/unit/pipelex/reporting/test_reporting_event_emission.py
@@ -0,0 +1,306 @@
+"""Unit tests for ReportingManager event emission.
+
+Validates that when an EventLogProtocol is provided, ReportingManager emits
+UsageReportEvent alongside existing local accumulation, with per-context
+isolation for concurrent workflows.
+"""
+
+from datetime import datetime, timedelta, timezone
+
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.graph.graph_config import DataInclusionConfig
+from pipelex.graph.graph_context import GraphContext
+from pipelex.pipeline.job_metadata import JobMetadata, UnitJobId
+from pipelex.reporting.reporting_manager import ReportingManager
+from pipelex.tracing.in_memory_event_log import InMemoryEventLog
+from pipelex.tracing.trace_events import UsageReportEvent
+
+DATA_INCLUSION_OFF = DataInclusionConfig(
+ pipe_and_concept_registry=False,
+ stuff_json_content=False,
+ stuff_text_content=False,
+ stuff_html_content=False,
+ error_stack_traces=False,
+)
+
+
+def _make_test_llm_job(
+ pipeline_run_id: str,
+ graph_context: GraphContext | None = None,
+) -> LLMJob:
+ """Create a minimal LLMJob for testing event emission."""
+ now = datetime.now(timezone.utc)
+ job_metadata = JobMetadata(
+ user_id="test_user",
+ pipeline_run_id=pipeline_run_id,
+ graph_context=graph_context,
+ started_at=now,
+ completed_at=now + timedelta(seconds=1),
+ unit_job_id=UnitJobId.LLM_GEN_TEXT,
+ )
+ tokens_usage = LLMTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},
+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},
+ )
+ return LLMJob(
+ job_metadata=job_metadata,
+ llm_prompt=LLMPrompt(),
+ job_params=LLMJobParams(temperature=0.5),
+ job_config=LLMJobConfig(max_retries=1),
+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),
+ )
+
+
+def _make_graph_context(
+ graph_id: str,
+ parent_node_id: str | None = None,
+ tracer_key: str | None = None,
+ node_sequence: int = 0,
+) -> GraphContext:
+ """Create a GraphContext for testing with lookup_key resolution."""
+ return GraphContext(
+ graph_id=graph_id,
+ tracer_key=tracer_key,
+ parent_node_id=parent_node_id,
+ node_sequence=node_sequence,
+ data_inclusion=DATA_INCLUSION_OFF,
+ )
+
+
+class TestReportingEventEmission:
+ """Tests for ReportingManager usage event emission with per-context isolation."""
+
+ PIPELINE_RUN_ID = "run_rpt_001"
+ WORKFLOW_ID = "wf_rpt_abc"
+
+ def _make_reporting_manager_with_event_log(self) -> tuple[ReportingManager, InMemoryEventLog]:
+ """Create a ReportingManager configured with an event log.
+
+ The context_key is PIPELINE_RUN_ID, matching the lookup_key of GraphContexts
+ created with graph_id=PIPELINE_RUN_ID and no tracer_key.
+ """
+ event_log = InMemoryEventLog()
+ manager = ReportingManager()
+ manager.setup()
+ manager.set_event_log(
+ context_key=self.PIPELINE_RUN_ID,
+ event_log=event_log,
+ workflow_id=self.WORKFLOW_ID,
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ )
+ manager.open_registry(self.PIPELINE_RUN_ID)
+ return manager, event_log
+
+ # ------------------------------------------------------------------
+ # Existing tests (updated for context-based resolution)
+ # ------------------------------------------------------------------
+
+ def test_report_inference_job_emits_usage_event(self) -> None:
+ """UsageReportEvent is emitted when reporting an LLM inference job with event_log set."""
+ manager, event_log = self._make_reporting_manager_with_event_log()
+
+ graph_context = _make_graph_context(graph_id=self.PIPELINE_RUN_ID)
+ llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, graph_context=graph_context)
+ manager.report_inference_job(llm_job)
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert len(usage_events) == 1
+ assert usage_events[0].tokens_usage.nb_tokens_by_category[TokenCategory.INPUT] == 100
+ assert usage_events[0].node_id == "unknown" # No graph_context parent set
+
+ def test_report_inference_job_captures_node_id_from_graph_context(self) -> None:
+ """UsageReportEvent captures the parent node_id from graph context."""
+ manager, event_log = self._make_reporting_manager_with_event_log()
+
+ graph_context = _make_graph_context(
+ graph_id=self.PIPELINE_RUN_ID,
+ parent_node_id="test-graph:node_42",
+ node_sequence=43,
+ )
+
+ llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, graph_context=graph_context)
+ manager.report_inference_job(llm_job)
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert len(usage_events) == 1
+ assert usage_events[0].node_id == "test-graph:node_42"
+ assert usage_events[0].workflow_id == self.WORKFLOW_ID
+
+ # ------------------------------------------------------------------
+ # Registry tests (unchanged behavior)
+ # ------------------------------------------------------------------
+
+ def test_inject_tokens_usages_adds_to_registry(self) -> None:
+ """inject_tokens_usages adds externally-collected usage data to the pipeline's registry."""
+ manager = ReportingManager()
+ manager.setup()
+ manager.open_registry(self.PIPELINE_RUN_ID)
+
+ tokens_usage = LLMTokensUsage(
+ job_metadata=_make_test_llm_job(self.PIPELINE_RUN_ID).job_metadata,
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},
+ nb_tokens_by_category={TokenCategory.INPUT: 200, TokenCategory.OUTPUT: 100},
+ )
+
+ manager.inject_tokens_usages(
+ pipeline_run_id=self.PIPELINE_RUN_ID,
+ tokens_usages=[tokens_usage],
+ )
+
+ registry = manager._get_registry(self.PIPELINE_RUN_ID) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ usages = registry.get_current_tokens_usage()
+ assert len(usages) == 1
+ assert usages[0].nb_tokens_by_category[TokenCategory.INPUT] == 200
+
+ def test_inject_tokens_usages_auto_creates_registry(self) -> None:
+ """inject_tokens_usages auto-creates registry if it doesn't exist yet."""
+ manager = ReportingManager()
+ manager.setup()
+
+ tokens_usage = LLMTokensUsage(
+ job_metadata=_make_test_llm_job("new_run").job_metadata,
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 2.0},
+ nb_tokens_by_category={TokenCategory.INPUT: 50, TokenCategory.OUTPUT: 25},
+ )
+
+ manager.inject_tokens_usages(
+ pipeline_run_id="new_run",
+ tokens_usages=[tokens_usage],
+ )
+
+ registry = manager._get_registry("new_run") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ usages = registry.get_current_tokens_usage()
+ assert len(usages) == 1
+
+ def test_no_event_log_works_as_before(self) -> None:
+ """ReportingManager without event_log works exactly as before."""
+ manager = ReportingManager()
+ manager.setup()
+ manager.open_registry("direct_run")
+ manager.close_registry("direct_run")
+ manager.teardown()
+
+ # ------------------------------------------------------------------
+ # Per-context isolation regression tests
+ # ------------------------------------------------------------------
+
+ def test_concurrent_contexts_are_isolated(self) -> None:
+ """Two concurrent contexts emit events to their own event logs without interference."""
+ event_log_a = InMemoryEventLog()
+ event_log_b = InMemoryEventLog()
+ manager = ReportingManager()
+ manager.setup()
+
+ manager.set_event_log(context_key="wf_a", event_log=event_log_a, workflow_id="wf_a", pipeline_run_id="run_a")
+ manager.set_event_log(context_key="wf_b", event_log=event_log_b, workflow_id="wf_b", pipeline_run_id="run_b")
+ manager.open_registry("run_a")
+ manager.open_registry("run_b")
+
+ ctx_a = _make_graph_context(graph_id="run_a", tracer_key="wf_a")
+ ctx_b = _make_graph_context(graph_id="run_b", tracer_key="wf_b")
+
+ job_a = _make_test_llm_job("run_a", graph_context=ctx_a)
+ job_b = _make_test_llm_job("run_b", graph_context=ctx_b)
+
+ manager.report_inference_job(job_a)
+ manager.report_inference_job(job_b)
+
+ events_a = [evt for evt in event_log_a.read_events("run_a") if isinstance(evt, UsageReportEvent)]
+ events_b = [evt for evt in event_log_b.read_events("run_b") if isinstance(evt, UsageReportEvent)]
+ assert len(events_a) == 1
+ assert events_a[0].workflow_id == "wf_a"
+ assert events_a[0].pipeline_run_id == "run_a"
+ assert len(events_b) == 1
+ assert events_b[0].workflow_id == "wf_b"
+ assert events_b[0].pipeline_run_id == "run_b"
+
+ def test_clear_event_log_removes_only_target_context(self) -> None:
+ """Clearing one context does not affect another."""
+ event_log_a = InMemoryEventLog()
+ event_log_b = InMemoryEventLog()
+ manager = ReportingManager()
+ manager.setup()
+
+ manager.set_event_log(context_key="wf_a", event_log=event_log_a, workflow_id="wf_a", pipeline_run_id="run_a")
+ manager.set_event_log(context_key="wf_b", event_log=event_log_b, workflow_id="wf_b", pipeline_run_id="run_b")
+ manager.open_registry("run_a")
+ manager.open_registry("run_b")
+
+ # Clear context A before any emission
+ manager.clear_event_log(context_key="wf_a")
+
+ ctx_a = _make_graph_context(graph_id="run_a", tracer_key="wf_a")
+ ctx_b = _make_graph_context(graph_id="run_b", tracer_key="wf_b")
+
+ job_a = _make_test_llm_job("run_a", graph_context=ctx_a)
+ job_b = _make_test_llm_job("run_b", graph_context=ctx_b)
+
+ manager.report_inference_job(job_a) # Should NOT emit (context cleared)
+ manager.report_inference_job(job_b) # Should emit
+
+ events_a = [evt for evt in event_log_a.read_events("run_a") if isinstance(evt, UsageReportEvent)]
+ events_b = [evt for evt in event_log_b.read_events("run_b") if isinstance(evt, UsageReportEvent)]
+ assert len(events_a) == 0
+ assert len(events_b) == 1
+
+ def test_no_graph_context_skips_emission(self) -> None:
+ """Jobs without graph_context skip event emission even when event logs are configured."""
+ manager, event_log = self._make_reporting_manager_with_event_log()
+
+ llm_job = _make_test_llm_job(self.PIPELINE_RUN_ID, graph_context=None)
+ manager.report_inference_job(llm_job)
+
+ events = event_log.read_events(self.PIPELINE_RUN_ID)
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert len(usage_events) == 0
+
+ def test_clear_event_log_is_idempotent(self) -> None:
+ """Calling clear_event_log for a non-existent key does not raise."""
+ manager = ReportingManager()
+ manager.setup()
+ manager.clear_event_log(context_key="nonexistent") # Should not raise
+
+ def test_teardown_clears_all_contexts(self) -> None:
+ """teardown() removes all event log contexts."""
+ manager, _event_log = self._make_reporting_manager_with_event_log()
+ manager.teardown()
+ assert len(manager._event_log_contexts) == 0 # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ def test_sequence_counters_are_independent(self) -> None:
+ """Each context maintains its own monotonic sequence counter."""
+ event_log_a = InMemoryEventLog()
+ event_log_b = InMemoryEventLog()
+ manager = ReportingManager()
+ manager.setup()
+
+ manager.set_event_log(context_key="wf_a", event_log=event_log_a, workflow_id="wf_a", pipeline_run_id="run_a")
+ manager.set_event_log(context_key="wf_b", event_log=event_log_b, workflow_id="wf_b", pipeline_run_id="run_b")
+ manager.open_registry("run_a")
+ manager.open_registry("run_b")
+
+ ctx_a = _make_graph_context(graph_id="run_a", tracer_key="wf_a")
+ ctx_b = _make_graph_context(graph_id="run_b", tracer_key="wf_b")
+
+ # Emit 3 events for context A, 1 for context B
+ for _index in range(3):
+ manager.report_inference_job(_make_test_llm_job("run_a", graph_context=ctx_a))
+ manager.report_inference_job(_make_test_llm_job("run_b", graph_context=ctx_b))
+
+ events_a = [evt for evt in event_log_a.read_events("run_a") if isinstance(evt, UsageReportEvent)]
+ events_b = [evt for evt in event_log_b.read_events("run_b") if isinstance(evt, UsageReportEvent)]
+ assert [evt.sequence for evt in events_a] == [0, 1, 2]
+ assert [evt.sequence for evt in events_b] == [0]
diff --git a/tests/unit/pipelex/temporal/test_conditional_worker.py b/tests/unit/pipelex/temporal/test_conditional_worker.py
new file mode 100644
index 000000000..5231c4008
--- /dev/null
+++ b/tests/unit/pipelex/temporal/test_conditional_worker.py
@@ -0,0 +1,86 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+from unittest.mock import AsyncMock
+
+import pytest
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment
+from pipelex.temporal.tprl.conditional_worker import with_conditional_worker
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor
+
+
[email protected](loop_scope="class")
+class TestConditionalWorker:
+ """Verify that with_conditional_worker restores task_queue after execution."""
+
+ async def test_task_queue_not_mutated_after_multiple_calls(self, mocker: MockerFixture) -> None:
+ """Calling a decorated method multiple times must not grow the task_queue name."""
+ original_queue = "my-task-queue"
+
+ # Build a concrete WorkflowExecutor with INTERNAL worker environment
+ executor: WorkflowExecutor[Any, Any] = WorkflowExecutor(
+ task_queue=original_queue,
+ worker_environment=TemporalWorkerEnvironment.INTERNAL,
+ )
+
+ # Mock temporal_client() to return a fake client
+ mocker.patch.object(executor, "temporal_client", new_callable=AsyncMock)
+
+ # Mock get_task_manager().make_worker() to return an async context manager
+ mock_task_manager = mocker.MagicMock()
+ mock_worker_ctx = AsyncMock()
+ mock_worker_ctx.__aenter__ = AsyncMock(return_value=None)
+ mock_worker_ctx.__aexit__ = AsyncMock(return_value=False)
+ mock_task_manager.make_worker.return_value = mock_worker_ctx
+ mocker.patch(
+ "pipelex.temporal.tprl.conditional_worker.get_task_manager",
+ return_value=mock_task_manager,
+ )
+
+ # Define a decorated method
+ @with_conditional_worker
+ async def fake_execute_workflow(_self_arg: WorkflowExecutor[Any, Any]) -> str: # noqa: RUF029
+ return "done"
+
+ # Call multiple times on the same instance
+ for _index in range(5):
+ result = await fake_execute_workflow(executor)
+ assert result == "done"
+ assert executor.task_queue == original_queue, f"task_queue mutated after call: expected {original_queue!r}, got {executor.task_queue!r}"
+
+ async def test_task_queue_restored_after_exception(self, mocker: MockerFixture) -> None:
+ """task_queue must be restored even when the wrapped function raises."""
+ original_queue = "error-queue"
+
+ executor: WorkflowExecutor[Any, Any] = WorkflowExecutor(
+ task_queue=original_queue,
+ worker_environment=TemporalWorkerEnvironment.INTERNAL,
+ )
+
+ mocker.patch.object(executor, "temporal_client", new_callable=AsyncMock)
+
+ mock_task_manager = mocker.MagicMock()
+ mock_worker_ctx = AsyncMock()
+ mock_worker_ctx.__aenter__ = AsyncMock(return_value=None)
+ mock_worker_ctx.__aexit__ = AsyncMock(return_value=False)
+ mock_task_manager.make_worker.return_value = mock_worker_ctx
+ mocker.patch(
+ "pipelex.temporal.tprl.conditional_worker.get_task_manager",
+ return_value=mock_task_manager,
+ )
+
+ @with_conditional_worker
+ async def failing_execute_workflow(_self_arg: WorkflowExecutor[Any, Any]) -> str: # noqa: RUF029
+ msg = "workflow failed"
+ raise RuntimeError(msg)
+
+ with pytest.raises(RuntimeError, match="workflow failed"):
+ await failing_execute_workflow(executor)
+
+ assert executor.task_queue == original_queue, (
+ f"task_queue not restored after exception: expected {original_queue!r}, got {executor.task_queue!r}"
+ )
diff --git a/tests/unit/pipelex/temporal/test_full_catalog_collisions.py b/tests/unit/pipelex/temporal/test_full_catalog_collisions.py
new file mode 100644
index 000000000..ad55c7273
--- /dev/null
+++ b/tests/unit/pipelex/temporal/test_full_catalog_collisions.py
@@ -0,0 +1,128 @@
+from typing import Any
+
+import pytest
+
+from pipelex.temporal.exceptions import TemporalConfigError
+from pipelex.temporal.temporal_tasks import TaskPack, TemporalTasks
+from pipelex.temporal.temporal_types import ActivityType, WorkflowType
+
+
+def _make_workflow(name: str) -> WorkflowType:
+ return type(name, (), {})
+
+
+def _make_activity(name: str) -> ActivityType:
+ def _impl(payload: Any) -> Any:
+ return payload
+
+ _impl.__name__ = name
+ return _impl
+
+
+class TestFullCatalogCollisions:
+ def test_same_workflow_callable_in_multiple_packs_dedups(self):
+ workflow = _make_workflow("WfShared")
+ activity = _make_activity("act_shared")
+ tasks = TemporalTasks()
+ tasks.complement_catalog(
+ extra_catalog={
+ "pack_one": TaskPack(workflow_list=[workflow], activity_list=[activity]),
+ "pack_two": TaskPack(workflow_list=[workflow], activity_list=[activity]),
+ },
+ extra_workflows=[],
+ extra_activities=[],
+ )
+
+ workflows, activities = tasks.workflows_and_activities()
+
+ assert workflows == [workflow]
+ assert activities == [activity]
+
+ def test_same_callable_in_pack_and_extras_dedups(self):
+ workflow = _make_workflow("WfShared")
+ activity = _make_activity("act_shared")
+ tasks = TemporalTasks()
+ tasks.complement_catalog(
+ extra_catalog={
+ "pack_one": TaskPack(workflow_list=[workflow], activity_list=[activity]),
+ },
+ extra_workflows=[workflow],
+ extra_activities=[activity],
+ )
+
+ workflows, activities = tasks.workflows_and_activities()
+
+ assert workflows == [workflow]
+ assert activities == [activity]
+
+ def test_distinct_workflows_with_same_name_across_packs_raises(self):
+ workflow_one = _make_workflow("WfClash")
+ workflow_two = _make_workflow("WfClash")
+ assert workflow_one is not workflow_two
+
+ tasks = TemporalTasks()
+ tasks.complement_catalog(
+ extra_catalog={
+ "pack_one": TaskPack(workflow_list=[workflow_one], activity_list=[]),
+ "pack_two": TaskPack(workflow_list=[workflow_two], activity_list=[]),
+ },
+ extra_workflows=[],
+ extra_activities=[],
+ )
+
+ with pytest.raises(TemporalConfigError, match="Workflow name collision: 'WfClash'") as exc_info:
+ tasks.workflows_and_activities()
+ assert "task pack 'pack_two'" in str(exc_info.value)
+
+ def test_distinct_workflows_with_same_name_pack_and_extras_raises(self):
+ workflow_in_pack = _make_workflow("WfClash")
+ workflow_in_extras = _make_workflow("WfClash")
+
+ tasks = TemporalTasks()
+ tasks.complement_catalog(
+ extra_catalog={
+ "pack_one": TaskPack(workflow_list=[workflow_in_pack], activity_list=[]),
+ },
+ extra_workflows=[workflow_in_extras],
+ extra_activities=[],
+ )
+
+ with pytest.raises(TemporalConfigError, match="Workflow name collision: 'WfClash'") as exc_info:
+ tasks.workflows_and_activities()
+ assert "extra_workflows" in str(exc_info.value)
+
+ def test_distinct_activities_with_same_name_across_packs_raises(self):
+ activity_one = _make_activity("act_clash")
+ activity_two = _make_activity("act_clash")
+ assert activity_one is not activity_two
+
+ tasks = TemporalTasks()
+ tasks.complement_catalog(
+ extra_catalog={
+ "pack_one": TaskPack(workflow_list=[], activity_list=[activity_one]),
+ "pack_two": TaskPack(workflow_list=[], activity_list=[activity_two]),
+ },
+ extra_workflows=[],
+ extra_activities=[],
+ )
+
+ with pytest.raises(TemporalConfigError, match="Activity name collision: 'act_clash'") as exc_info:
+ tasks.workflows_and_activities()
+ assert "task pack 'pack_two'" in str(exc_info.value)
+
+ def test_distinct_activities_with_same_name_pack_and_extras_raises(self):
+ activity_in_pack = _make_activity("act_clash")
+ activity_in_extras = _make_activity("act_clash")
+
+ tasks = TemporalTasks()
+ tasks.complement_catalog(
+ extra_catalog={
+ "pack_one": TaskPack(workflow_list=[], activity_list=[activity_in_pack]),
+ },
+ extra_workflows=[],
+ extra_activities=[activity_in_extras],
+ )
+
+ with pytest.raises(TemporalConfigError, match="Activity name collision: 'act_clash'") as exc_info:
+ tasks.workflows_and_activities()
+ assert "extra_activities" in str(exc_info.value)
diff --git a/tests/unit/pipelex/temporal/test_storage_payload_codec.py b/tests/unit/pipelex/temporal/test_storage_payload_codec.py
new file mode 100644
index 000000000..f931cc8c0
--- /dev/null
+++ b/tests/unit/pipelex/temporal/test_storage_payload_codec.py
@@ -0,0 +1,276 @@
+from __future__ import annotations
+
+import pytest
+from temporalio.api.common.v1 import Payload
+
+from pipelex.temporal.codec.storage_payload_codec import STORAGE_REF_ENCODING, StoragePayloadCodec
+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
+
+TEST_THRESHOLD = 1024
+TEST_PREFIX = "test-payloads/"
+
+
+def _make_payload(size: int) -> Payload:
+ """Build a Payload with the given data size."""
+ return Payload(
+ metadata={"encoding": b"binary/plain"},
+ data=b"x" * size,
+ )
+
+
[email protected](loop_scope="class")
+class TestStoragePayloadCodec:
+ """Unit tests for StoragePayloadCodec encode/decode behavior."""
+
+ @pytest.fixture
+ def storage(self) -> InMemoryStorageProvider:
+ return InMemoryStorageProvider()
+
+ @pytest.fixture
+ def codec(self, storage: InMemoryStorageProvider) -> StoragePayloadCodec:
+ return StoragePayloadCodec(
+ storage_provider=storage,
+ size_threshold=TEST_THRESHOLD,
+ storage_prefix=TEST_PREFIX,
+ )
+
+ async def test_below_threshold_passes_through(
+ self,
+ codec: StoragePayloadCodec,
+ ) -> None:
+ """Payloads smaller than the threshold pass through encode/decode unchanged."""
+ small_payload = _make_payload(TEST_THRESHOLD - 100)
+ original_bytes = small_payload.SerializeToString()
+
+ encoded = await codec.encode([small_payload])
+
+ assert len(encoded) == 1
+ assert encoded[0].SerializeToString() == original_bytes
+
+ decoded = await codec.decode(encoded)
+
+ assert len(decoded) == 1
+ assert decoded[0].SerializeToString() == original_bytes
+
+ async def test_above_threshold_offloads_to_storage(
+ self,
+ codec: StoragePayloadCodec,
+ storage: InMemoryStorageProvider,
+ ) -> None:
+ """Payloads larger than the threshold are offloaded to storage and reconstructed on decode."""
+ large_payload = _make_payload(TEST_THRESHOLD + 100)
+ original_bytes = large_payload.SerializeToString()
+
+ encoded = await codec.encode([large_payload])
+
+ assert len(encoded) == 1
+ ref_payload = encoded[0]
+
+ # The reference payload must carry the storage-ref encoding marker
+ assert ref_payload.metadata.get("encoding") == STORAGE_REF_ENCODING
+
+ # The reference payload data is a URI pointing to the stored blob
+ uri = ref_payload.data.decode()
+ assert uri.startswith("pipelex-storage://")
+
+ # The storage provider must contain the original serialized payload
+ assert len(storage.root) == 1
+
+ # Decode must reconstruct the original payload exactly
+ decoded = await codec.decode(encoded)
+
+ assert len(decoded) == 1
+ assert decoded[0].SerializeToString() == original_bytes
+
+ async def test_content_addressed_deduplication(
+ self,
+ codec: StoragePayloadCodec,
+ storage: InMemoryStorageProvider,
+ ) -> None:
+ """Encoding the same large payload twice produces a single storage entry."""
+ large_payload = _make_payload(TEST_THRESHOLD + 50)
+
+ await codec.encode([large_payload])
+ await codec.encode([large_payload])
+
+ assert len(storage.root) == 1
+
+ async def test_mixed_payloads(
+ self,
+ codec: StoragePayloadCodec,
+ storage: InMemoryStorageProvider,
+ ) -> None:
+ """In a batch, small payloads pass through and large payloads are offloaded."""
+ small = _make_payload(10)
+ large = _make_payload(TEST_THRESHOLD + 200)
+ small_bytes = small.SerializeToString()
+ large_bytes = large.SerializeToString()
+
+ encoded = await codec.encode([small, large])
+
+ assert len(encoded) == 2
+
+ # First payload (small) passed through unchanged
+ assert encoded[0].SerializeToString() == small_bytes
+
+ # Second payload (large) is a storage reference
+ assert encoded[1].metadata.get("encoding") == STORAGE_REF_ENCODING
+
+ # Only the large payload was stored
+ assert len(storage.root) == 1
+
+ # Full round-trip preserves both payloads
+ decoded = await codec.decode(encoded)
+
+ assert len(decoded) == 2
+ assert decoded[0].SerializeToString() == small_bytes
+ assert decoded[1].SerializeToString() == large_bytes
+
+ async def test_exactly_at_threshold_is_offloaded(
+ self,
+ storage: InMemoryStorageProvider,
+ ) -> None:
+ """A payload whose serialized size equals the threshold is offloaded, not passed through."""
+ payload = _make_payload(500)
+ serialized_size = len(payload.SerializeToString())
+
+ codec = StoragePayloadCodec(
+ storage_provider=storage,
+ size_threshold=serialized_size,
+ storage_prefix=TEST_PREFIX,
+ )
+
+ encoded = await codec.encode([payload])
+
+ assert len(encoded) == 1
+ assert encoded[0].metadata.get("encoding") == STORAGE_REF_ENCODING
+ assert len(storage.root) == 1
+
+ @pytest.mark.parametrize(
+ ("topic", "size"),
+ [
+ ("small", 10),
+ ("at_threshold", TEST_THRESHOLD),
+ ("above_threshold", TEST_THRESHOLD + 500),
+ ],
+ )
+ async def test_round_trip_fidelity(
+ self,
+ codec: StoragePayloadCodec,
+ topic: str,
+ size: int,
+ ) -> None:
+ """decode(encode(payloads)) returns identical payloads for various sizes."""
+ payload = _make_payload(size)
+ original_bytes = payload.SerializeToString()
+
+ encoded = await codec.encode([payload])
+ decoded = await codec.decode(encoded)
+
+ assert len(decoded) == 1
+ assert decoded[0].SerializeToString() == original_bytes, f"Round-trip failed for {topic} (size={size})"
+
+ # -------------------------------------------------------------------
+ # _extract_job_routing tests
+ # -------------------------------------------------------------------
+
+ async def test_storage_key_includes_routing_when_available(
+ self,
+ codec: StoragePayloadCodec,
+ ) -> None:
+ """When job_metadata is present, storage key includes user_id and pipeline_run_id."""
+ routed_payload = Payload(
+ metadata={"encoding": b"json/plain"},
+ data=b'{"job_metadata": {"user_id": "user42", "pipeline_run_id": "run99"}, "big": "' + b"x" * (TEST_THRESHOLD + 500) + b'"}',
+ )
+
+ encoded = await codec.encode([routed_payload])
+ assert len(encoded) == 1
+ uri = encoded[0].data.decode()
+ assert "user42" in uri
+ assert "run99" in uri
+
+ @pytest.mark.parametrize(
+ ("topic", "payload_data"),
+ [
+ (
+ "non_json_encoding_no_routing",
+ b"x" * (TEST_THRESHOLD + 500),
+ ),
+ (
+ "json_missing_job_metadata_no_routing",
+ b'{"other_key": "value", "pad": "' + b"x" * (TEST_THRESHOLD + 500) + b'"}',
+ ),
+ (
+ "json_non_dict_job_metadata_no_routing",
+ b'{"job_metadata": "not a dict", "pad": "' + b"x" * (TEST_THRESHOLD + 500) + b'"}',
+ ),
+ ],
+ )
+ async def test_storage_key_falls_back_to_flat_when_no_routing(
+ self,
+ storage: InMemoryStorageProvider,
+ topic: str,
+ payload_data: bytes,
+ ) -> None:
+ """Payloads without valid job_metadata use a flat storage key (prefix + hash only)."""
+ codec = StoragePayloadCodec(
+ storage_provider=storage,
+ size_threshold=TEST_THRESHOLD,
+ storage_prefix=TEST_PREFIX,
+ )
+ payload = Payload(
+ metadata={"encoding": b"json/plain" if topic.startswith("json_") else b"binary/plain"},
+ data=payload_data,
+ )
+
+ encoded = await codec.encode([payload])
+ assert len(encoded) == 1
+ uri = encoded[0].data.decode()
+ # Flat keys have no slashes after the prefix (just prefix + hash)
+ key_after_scheme = uri.split("://", 1)[-1]
+ segments = key_after_scheme.split("/")
+ # prefix is "test-payloads/" so we expect: test-payloads/{hash} = 2 segments
+ assert len(segments) == 2, f"Expected flat key for {topic}, got: {key_after_scheme}"
+
+ # -------------------------------------------------------------------
+ # _sanitize_path_segment tests
+ # -------------------------------------------------------------------
+
+ @pytest.mark.parametrize(
+ ("topic", "user_id", "pipeline_run_id", "expected_user", "expected_run"),
+ [
+ ("path_traversal_dotdot", "../../../etc", "run1", "_________etc", "run1"),
+ ("path_traversal_in_run", "user1", "../../secret", "user1", "______secret"),
+ ("slashes_stripped", "user/sub/path", "run/id", "user_sub_path", "run_id"),
+ ("special_chars", "[email protected]", "run id+test", "user_evil_com", "run_id_test"),
+ ("clean_ids_unchanged", "user42", "run-99_abc", "user42", "run-99_abc"),
+ ],
+ )
+ async def test_storage_key_sanitizes_routing_segments(
+ self,
+ codec: StoragePayloadCodec,
+ topic: str,
+ user_id: str,
+ pipeline_run_id: str,
+ expected_user: str,
+ expected_run: str,
+ ) -> None:
+ """Routing segments are sanitized to prevent path traversal and special char issues."""
+ metadata_json = (
+ f'{{"job_metadata": {{"user_id": "{user_id}", "pipeline_run_id": "{pipeline_run_id}"}}, "big": "' + "x" * (TEST_THRESHOLD + 500) + '"}'
+ )
+ routed_payload = Payload(
+ metadata={"encoding": b"json/plain"},
+ data=metadata_json.encode(),
+ )
+
+ encoded = await codec.encode([routed_payload])
+ assert len(encoded) == 1
+ uri = encoded[0].data.decode()
+ key_after_scheme = uri.split("://", 1)[-1]
+ # Expect: test-payloads/{safe_user}/{safe_run}/{hash} = 4 segments
+ segments = key_after_scheme.split("/")
+ assert len(segments) == 4, f"Expected routed key for {topic}, got: {key_after_scheme}"
+ assert segments[1] == expected_user, f"User segment mismatch for {topic}"
+ assert segments[2] == expected_run, f"Run segment mismatch for {topic}"
diff --git a/tests/unit/pipelex/temporal/test_worker_scope_resolution.py b/tests/unit/pipelex/temporal/test_worker_scope_resolution.py
new file mode 100644
index 000000000..60a596f45
--- /dev/null
+++ b/tests/unit/pipelex/temporal/test_worker_scope_resolution.py
@@ -0,0 +1,151 @@
+from typing import Any
+
+import pytest
+
+from pipelex.temporal.config_temporal import WorkerScope
+from pipelex.temporal.exceptions import WorkerScopeConfigError
+from pipelex.temporal.temporal_tasks import TaskPack, TemporalTasks
+
+
+class _WfAlpha:
+ pass
+
+
+class _WfBeta:
+ pass
+
+
+class _WfGamma:
+ pass
+
+
+def _act_one(_payload: Any) -> Any:
+ return _payload
+
+
+def _act_two(_payload: Any) -> Any:
+ return _payload
+
+
+def _act_three(_payload: Any) -> Any:
+ return _payload
+
+
+def _make_tasks() -> TemporalTasks:
+ tasks = TemporalTasks()
+ tasks.complement_catalog(
+ extra_catalog={
+ "pack_one": TaskPack(workflow_list=[_WfAlpha, _WfBeta], activity_list=[_act_one, _act_two]),
+ "pack_two": TaskPack(workflow_list=[_WfGamma], activity_list=[_act_three]),
+ },
+ extra_workflows=[],
+ extra_activities=[],
+ )
+ return tasks
+
+
+def _scope(
+ required_tasks_packs: list[str] | None = None,
+ required_workflows: list[str] | None = None,
+ required_activities: list[str] | None = None,
+ excluded_workflows: list[str] | None = None,
+ excluded_activities: list[str] | None = None,
+ disable_all_workflows: bool = False,
+ disable_all_activities: bool = False,
+) -> WorkerScope:
+ return WorkerScope(
+ required_tasks_packs=required_tasks_packs or [],
+ required_workflows=required_workflows or [],
+ required_activities=required_activities or [],
+ excluded_workflows=excluded_workflows or [],
+ excluded_activities=excluded_activities or [],
+ disable_all_workflows=disable_all_workflows,
+ disable_all_activities=disable_all_activities,
+ )
+
+
+class TestWorkerScopeResolution:
+ def test_no_scope_returns_full_catalog(self):
+ tasks = _make_tasks()
+ workflows, activities = tasks.workflows_and_activities()
+ assert set(workflows) == {_WfAlpha, _WfBeta, _WfGamma}
+ assert set(activities) == {_act_one, _act_two, _act_three}
+
+ def test_pack_only_scope(self):
+ tasks = _make_tasks()
+ workflows, activities = tasks.workflows_and_activities(scope=_scope(required_tasks_packs=["pack_one"]))
+ assert set(workflows) == {_WfAlpha, _WfBeta}
+ assert set(activities) == {_act_one, _act_two}
+
+ def test_required_workflows_and_activities_only(self):
+ tasks = _make_tasks()
+ workflows, activities = tasks.workflows_and_activities(
+ scope=_scope(required_workflows=["_WfGamma"], required_activities=["_act_three"]),
+ )
+ assert set(workflows) == {_WfGamma}
+ assert set(activities) == {_act_three}
+
+ def test_pack_plus_required(self):
+ tasks = _make_tasks()
+ workflows, activities = tasks.workflows_and_activities(
+ scope=_scope(required_tasks_packs=["pack_one"], required_workflows=["_WfGamma"]),
+ )
+ assert set(workflows) == {_WfAlpha, _WfBeta, _WfGamma}
+ assert set(activities) == {_act_one, _act_two}
+
+ def test_exclusions_apply_after_inclusions(self):
+ tasks = _make_tasks()
+ workflows, activities = tasks.workflows_and_activities(
+ scope=_scope(
+ required_tasks_packs=["pack_one", "pack_two"],
+ excluded_workflows=["_WfBeta"],
+ excluded_activities=["_act_one"],
+ ),
+ )
+ assert set(workflows) == {_WfAlpha, _WfGamma}
+ assert set(activities) == {_act_two, _act_three}
+
+ def test_unknown_pack_raises(self):
+ tasks = _make_tasks()
+ with pytest.raises(WorkerScopeConfigError, match="Unknown task pack"):
+ tasks.workflows_and_activities(scope=_scope(required_tasks_packs=["pack_missing"]))
+
+ def test_unknown_required_workflow_raises(self):
+ tasks = _make_tasks()
+ with pytest.raises(WorkerScopeConfigError, match="required_workflows"):
+ tasks.workflows_and_activities(scope=_scope(required_workflows=["WfUnknown"]))
+
+ def test_unknown_required_activity_raises(self):
+ tasks = _make_tasks()
+ with pytest.raises(WorkerScopeConfigError, match="required_activities"):
+ tasks.workflows_and_activities(scope=_scope(required_activities=["act_unknown"]))
+
+ def test_unknown_excluded_workflow_raises(self):
+ tasks = _make_tasks()
+ with pytest.raises(WorkerScopeConfigError, match="excluded_workflows"):
+ tasks.workflows_and_activities(
+ scope=_scope(required_tasks_packs=["pack_one"], excluded_workflows=["WfUnknown"]),
+ )
+
+ def test_unknown_excluded_activity_raises(self):
+ tasks = _make_tasks()
+ with pytest.raises(WorkerScopeConfigError, match="excluded_activities"):
+ tasks.workflows_and_activities(
+ scope=_scope(required_tasks_packs=["pack_one"], excluded_activities=["act_unknown"]),
+ )
+
+ def test_empty_resolved_scope_raises(self):
+ tasks = _make_tasks()
+ with pytest.raises(WorkerScopeConfigError, match="empty set"):
+ tasks.workflows_and_activities(scope=_scope())
+
+ def test_full_excluded_to_empty_raises(self):
+ tasks = _make_tasks()
+ with pytest.raises(WorkerScopeConfigError, match="empty set"):
+ tasks.workflows_and_activities(
+ scope=_scope(
+ required_tasks_packs=["pack_one", "pack_two"],
+ excluded_workflows=["_WfAlpha", "_WfBeta", "_WfGamma"],
+ excluded_activities=["_act_one", "_act_two", "_act_three"],
+ ),
+ )
diff --git a/tests/unit/pipelex/temporal/tprl_pipe/test_hydration.py b/tests/unit/pipelex/temporal/tprl_pipe/test_hydration.py
new file mode 100644
index 000000000..aab62fbf1
--- /dev/null
+++ b/tests/unit/pipelex/temporal/tprl_pipe/test_hydration.py
@@ -0,0 +1,157 @@
+from typing import cast
+
+import pytest
+
+from pipelex.core.concepts.concept import Concept
+from pipelex.core.domains.domain import SpecialDomain
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.list_content import ListContent
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_class_registry
+from pipelex.pipe_run.exceptions import PipeJobError
+from pipelex.temporal.tprl_pipe.hydration import hydrate_working_memory
+
+
+def _make_text_concept() -> Concept:
+ """Build a native Text concept for testing."""
+ return Concept(
+ code="Text",
+ domain_code=SpecialDomain.NATIVE,
+ description="Plain text",
+ structure_class_name="TextContent",
+ )
+
+
+def _make_text_stuff(name: str, text: str) -> Stuff:
+ """Build a simple text Stuff."""
+ return Stuff(
+ stuff_code="test",
+ stuff_name=name,
+ concept=_make_text_concept(),
+ content=TextContent(text=text),
+ )
+
+
+class TestHydrateWorkingMemory:
+ @pytest.fixture(autouse=True)
+ def _register_text_content(self) -> None:
+ """Ensure TextContent is registered in the ClassRegistry for hydration tests."""
+ registry = get_class_registry()
+ if not registry.has_class(name="TextContent"):
+ registry.register_class(TextContent)
+
+ def test_hydrate_with_native_text(self) -> None:
+ """A raw dict containing TextContent stuff hydrates to typed TextContent."""
+ working_memory = WorkingMemory()
+ working_memory.root["greeting"] = _make_text_stuff("greeting", "Hello, world!")
+
+ raw = working_memory.dump_for_temporal()
+ hydrated = hydrate_working_memory(raw)
+
+ assert "greeting" in hydrated.root
+ stuff = hydrated.root["greeting"]
+ assert isinstance(stuff.content, TextContent)
+ assert stuff.content.text == "Hello, world!"
+ assert stuff.stuff_name == "greeting"
+
+ def test_hydrate_empty(self) -> None:
+ """An empty WorkingMemory raw dict hydrates to empty WorkingMemory."""
+ working_memory = WorkingMemory()
+ raw = working_memory.dump_for_temporal()
+
+ hydrated = hydrate_working_memory(raw)
+
+ assert len(hydrated.root) == 0
+ assert len(hydrated.aliases) == 0
+
+ def test_hydrate_preserves_aliases(self) -> None:
+ """Aliases survive the dump/hydrate round-trip."""
+ working_memory = WorkingMemory()
+ working_memory.root["greeting"] = _make_text_stuff("greeting", "Hello!")
+ working_memory.aliases["main_stuff"] = "greeting"
+
+ raw = working_memory.dump_for_temporal()
+ hydrated = hydrate_working_memory(raw)
+
+ assert hydrated.aliases == {"main_stuff": "greeting"}
+ assert "greeting" in hydrated.root
+
+ def test_hydrate_list_content_round_trip(self) -> None:
+ """ListContent survives dump_for_temporal/hydrate round-trip as a plain list."""
+ working_memory = WorkingMemory()
+ list_stuff = Stuff(
+ stuff_code="test",
+ stuff_name="colors",
+ concept=_make_text_concept(),
+ content=ListContent(
+ items=[
+ TextContent(text="blue"),
+ TextContent(text="red"),
+ TextContent(text="green"),
+ ]
+ ),
+ )
+ working_memory.root["colors"] = list_stuff
+
+ raw = working_memory.dump_for_temporal()
+
+ # Verify the Temporal format uses a plain list, not {"items": [...]}
+ assert isinstance(raw["root"]["colors"]["content"], list)
+
+ hydrated = hydrate_working_memory(raw)
+
+ assert "colors" in hydrated.root
+ stuff = hydrated.root["colors"]
+ content = stuff.content
+ assert isinstance(content, ListContent)
+ list_content = cast("ListContent[TextContent]", content)
+ assert len(list_content.items) == 3
+ assert all(isinstance(item, TextContent) for item in list_content.items)
+ assert list_content.items[0].text == "blue"
+ assert list_content.items[1].text == "red"
+ assert list_content.items[2].text == "green"
+
+ def test_hydrate_raises_on_missing_registry_class(self) -> None:
+ """Hydration raises PipeJobError when the concept's structure_class_name is not in the registry."""
+ raw = {
+ "root": {
+ "bad_stuff": {
+ "stuff_code": "test",
+ "stuff_name": "bad_stuff",
+ "concept": {
+ "code": "NonExistent",
+ "domain_code": "native",
+ "description": "Missing class",
+ "structure_class_name": "NonExistentContent",
+ },
+ "content": {"text": "hello"},
+ },
+ },
+ "aliases": {},
+ }
+
+ with pytest.raises(PipeJobError, match="bad_stuff"):
+ hydrate_working_memory(raw)
+
+ def test_hydrate_raises_on_validation_error(self) -> None:
+ """Hydration raises PipeJobError when content doesn't match the expected schema."""
+ raw = {
+ "root": {
+ "invalid_stuff": {
+ "stuff_code": "test",
+ "stuff_name": "invalid_stuff",
+ "concept": {
+ "code": "Text",
+ "domain_code": "native",
+ "description": "Plain text",
+ "structure_class_name": "TextContent",
+ },
+ "content": {"completely_wrong_field": 42},
+ },
+ },
+ "aliases": {},
+ }
+
+ with pytest.raises(PipeJobError, match="invalid_stuff"):
+ hydrate_working_memory(raw)
diff --git a/tests/unit/pipelex/temporal/tprl_pipe/test_submitter_hydration.py b/tests/unit/pipelex/temporal/tprl_pipe/test_submitter_hydration.py
new file mode 100644
index 000000000..2af304a6c
--- /dev/null
+++ b/tests/unit/pipelex/temporal/tprl_pipe/test_submitter_hydration.py
@@ -0,0 +1,120 @@
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.core.concepts.concept import Concept
+from pipelex.core.domains.domain import SpecialDomain
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.hub import get_class_registry, get_library_manager
+from pipelex.pipe_run.exceptions import PipeJobError
+from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
+
+
+def _text_concept() -> Concept:
+ return Concept(
+ code="Text",
+ domain_code=SpecialDomain.NATIVE,
+ description="Plain text",
+ structure_class_name="TextContent",
+ )
+
+
+def _pipe_output_with_text(name: str, text: str) -> PipeOutput:
+ working_memory = WorkingMemory()
+ working_memory.root[name] = Stuff(
+ stuff_code="test",
+ stuff_name=name,
+ concept=_text_concept(),
+ content=TextContent(text=text),
+ )
+ raw = working_memory.dump_for_temporal()
+ return PipeOutput(working_memory=WorkingMemory(), working_memory_raw=raw)
+
+
+class TestRehydratePipeOutputWithCrate:
+ @pytest.fixture(autouse=True)
+ def _register_text_content(self) -> None:
+ registry = get_class_registry()
+ if not registry.has_class(name="TextContent"):
+ registry.register_class(TextContent)
+
+ def test_returns_same_instance_when_raw_is_none(self, mocker: MockerFixture) -> None:
+ pipe_output = PipeOutput(working_memory=WorkingMemory(), working_memory_raw=None)
+ open_spy = mocker.spy(get_library_manager(), "open_library")
+
+ result = rehydrate_pipe_output_with_crate(pipe_output, library_crate=None)
+
+ assert result is pipe_output
+ assert result.working_memory_raw is None
+ assert open_spy.call_count == 0
+
+ def test_no_crate_uses_active_registry(self) -> None:
+ """With library_crate=None and only built-in concept, hydration uses the active registry."""
+ pipe_output = _pipe_output_with_text("greeting", "Hello, world!")
+
+ result = rehydrate_pipe_output_with_crate(pipe_output, library_crate=None)
+
+ assert result is pipe_output
+ assert pipe_output.working_memory_raw is None
+ stuff = pipe_output.working_memory.root["greeting"]
+ assert isinstance(stuff.content, TextContent)
+ assert stuff.content.text == "Hello, world!"
+
+ def test_no_crate_propagates_pipejob_error_for_unknown_class(self) -> None:
+ """Without a crate, an unknown structure_class_name raises (existing failure mode)."""
+ raw = {
+ "root": {
+ "bad_stuff": {
+ "stuff_code": "test",
+ "stuff_name": "bad_stuff",
+ "concept": {
+ "code": "NonExistent",
+ "domain_code": "native",
+ "description": "missing",
+ "structure_class_name": "AbsolutelyNonExistentClass",
+ },
+ "content": {"text": "x"},
+ },
+ },
+ "aliases": {},
+ }
+ pipe_output = PipeOutput(working_memory=WorkingMemory(), working_memory_raw=raw)
+
+ with pytest.raises(PipeJobError):
+ rehydrate_pipe_output_with_crate(pipe_output, library_crate=None)
+
+ def test_teardown_runs_on_hydrate_failure_with_crate(self, mocker: MockerFixture) -> None:
+ """If hydration raises while a crate scope is open, the scoped library is still torn down."""
+ # Mock load_from_crate to be a no-op so we don't need a real crate.
+ # The raw dict will trigger a hydrate failure via unknown class.
+ from pipelex.libraries.library_crate import LibraryCrate # noqa: PLC0415
+
+ fake_crate = mocker.MagicMock(spec=LibraryCrate)
+ library_manager = get_library_manager()
+ teardown_spy = mocker.spy(library_manager, "teardown")
+ mocker.patch.object(library_manager, "load_from_crate", return_value=None)
+
+ raw = {
+ "root": {
+ "bad_stuff": {
+ "stuff_code": "test",
+ "stuff_name": "bad_stuff",
+ "concept": {
+ "code": "NonExistent",
+ "domain_code": "native",
+ "description": "missing",
+ "structure_class_name": "AbsolutelyNonExistentClass",
+ },
+ "content": {"text": "x"},
+ },
+ },
+ "aliases": {},
+ }
+ pipe_output = PipeOutput(working_memory=WorkingMemory(), working_memory_raw=raw)
+
+ with pytest.raises(PipeJobError):
+ rehydrate_pipe_output_with_crate(pipe_output, library_crate=fake_crate)
+
+ assert teardown_spy.call_count == 1, "Scoped rehydration library should be torn down on failure"
diff --git a/tests/unit/pipelex/temporal/tprl_pipe/test_validate_as_known_class.py b/tests/unit/pipelex/temporal/tprl_pipe/test_validate_as_known_class.py
new file mode 100644
index 000000000..cf5713ea8
--- /dev/null
+++ b/tests/unit/pipelex/temporal/tprl_pipe/test_validate_as_known_class.py
@@ -0,0 +1,66 @@
+from pydantic import ConfigDict
+
+from pipelex.core.stuffs.stuff_content import StuffContent
+from pipelex.core.stuffs.text_content import TextContent
+from pipelex.temporal.tprl_pipe.hydration import _validate_as_known_class # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+
+
+class _StuffContentAllowExtra(StuffContent):
+ # extra="allow" lets us observe whether nested subclass-specific data
+ # made it into the dump that the function feeds into model_validate.
+ model_config = ConfigDict(extra="allow")
+
+
+class _SpecificMeta(_StuffContentAllowExtra):
+ flavor: str
+
+
+class _OuterContent(StuffContent):
+ meta: _StuffContentAllowExtra
+
+
+class _RebuiltOuterContent(StuffContent):
+ # Same shape as _OuterContent but a distinct Python class object — stands in
+ # for the "rebuilt after re-exec" class identity in the cross-exec scenario.
+ meta: _StuffContentAllowExtra
+
+
+class TestValidateAsKnownClass:
+ def test_returns_instance_when_class_identity_matches(self) -> None:
+ """When type(raw_item) is item_class, return the instance as-is (no round-trip)."""
+ instance = TextContent(text="hello")
+ result = _validate_as_known_class(TextContent, instance)
+ assert result is instance
+
+ def test_validates_dict_input(self) -> None:
+ """When raw_item is a dict, validate it through model_validate."""
+ result = _validate_as_known_class(TextContent, {"text": "hello"})
+ assert isinstance(result, TextContent)
+ assert result.text == "hello"
+
+ def test_cross_exec_round_trip_basic(self) -> None:
+ """Cross-exec instance (different class identity, same shape) round-trips correctly."""
+ original = _OuterContent(meta=_StuffContentAllowExtra())
+ result = _validate_as_known_class(_RebuiltOuterContent, original)
+ assert type(result) is _RebuiltOuterContent
+
+ def test_cross_exec_preserves_nested_subclass_data(self) -> None:
+ """Cross-exec round-trip preserves subclass-specific data in nested base-typed fields.
+
+ The nested ``meta`` field is annotated as ``_StuffContentAllowExtra`` but holds a
+ ``_SpecificMeta`` instance carrying an extra ``flavor`` field. ``smart_dump``
+ uses ``serialize_as_any=True`` so the dump reflects the runtime type and keeps
+ ``flavor``. Plain ``model_dump`` would dump using the field annotation and drop
+ ``flavor`` before validation ever sees it — so this assertion would fail if the
+ implementation regressed back to ``model_dump()``.
+ """
+ instance = _OuterContent(meta=_SpecificMeta(flavor="vanilla"))
+ result = _validate_as_known_class(_RebuiltOuterContent, instance)
+
+ assert type(result) is _RebuiltOuterContent
+ extras = result.meta.model_extra or {}
+ assert extras.get("flavor") == "vanilla", (
+ f"Nested subclass field 'flavor' was lost in the cross-exec round-trip — "
+ f"likely a regression from smart_dump() back to model_dump() without "
+ f"serialize_as_any=True. result.meta.model_extra={result.meta.model_extra}"
+ )
diff --git a/tests/unit/pipelex/test_hub_class_registry.py b/tests/unit/pipelex/test_hub_class_registry.py
new file mode 100644
index 000000000..9a896b229
--- /dev/null
+++ b/tests/unit/pipelex/test_hub_class_registry.py
@@ -0,0 +1,83 @@
+from kajson.class_registry import ClassRegistry
+from kajson.kajson_manager import KajsonManager
+from pydantic import BaseModel
+
+from pipelex.hub import (
+ get_class_registry,
+ get_library_manager,
+ set_current_library,
+ teardown_current_library,
+)
+from pipelex.libraries.library_factory import LibraryFactory
+
+
+class ScopedModel(BaseModel):
+ scoped_field: str
+
+
+class TestHubClassRegistry:
+ def test_get_class_registry_without_library_id_returns_global(self) -> None:
+ """Without library_id set, hub returns the global KajsonManager registry."""
+ result = get_class_registry()
+ assert result is KajsonManager.get_class_registry()
+
+ def test_get_class_registry_with_library_id_returns_library_registry(self) -> None:
+ """When _library_id is set and the library has a ClassRegistry, hub returns it."""
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+
+ workflow_registry = ClassRegistry()
+ workflow_registry.register_class(ScopedModel)
+ library.set_class_registry(workflow_registry)
+
+ set_current_library(library_id=library_id)
+ try:
+ result = get_class_registry()
+ assert result is workflow_registry
+ assert result.has_class(name="ScopedModel")
+ finally:
+ teardown_current_library()
+ library_manager.teardown(library_id=library_id)
+
+ def test_get_class_registry_library_without_registry_falls_back_to_global(self) -> None:
+ """_library_id set but library has no _class_registry -> falls back to global."""
+ library_manager = get_library_manager()
+ library_id, _library = library_manager.open_library()
+ # Don't set a class_registry on the library
+
+ set_current_library(library_id=library_id)
+ try:
+ result = get_class_registry()
+ assert result is KajsonManager.get_class_registry()
+ finally:
+ teardown_current_library()
+ library_manager.teardown(library_id=library_id)
+
+ def test_library_class_registry_not_in_model_dump(self) -> None:
+ """Library._class_registry (PrivateAttr) is not included in model_dump() output."""
+ library = LibraryFactory.make_empty()
+ registry = ClassRegistry()
+ library.set_class_registry(registry)
+
+ dump = library.model_dump()
+ assert "_class_registry" not in dump
+ assert "class_registry" not in dump
+
+ def test_library_class_registry_gc_with_teardown(self) -> None:
+ """When a library is torn down, its ClassRegistry is released."""
+ library_manager = get_library_manager()
+ library_id, library = library_manager.open_library()
+
+ workflow_registry = ClassRegistry()
+ workflow_registry.register_class(ScopedModel)
+ library.set_class_registry(workflow_registry)
+
+ set_current_library(library_id=library_id)
+ assert get_class_registry().has_class(name="ScopedModel")
+
+ teardown_current_library()
+ library_manager.teardown(library_id=library_id)
+
+ # After teardown, hub falls back to global (which doesn't have ScopedModel)
+ global_registry = get_class_registry()
+ assert not global_registry.has_class(name="ScopedModel")
diff --git a/tests/unit/pipelex/tools/misc/test_http_utils.py b/tests/unit/pipelex/tools/misc/test_http_utils.py
index 169bcb8d0..3ed0a82d8 100644
--- a/tests/unit/pipelex/tools/misc/test_http_utils.py
+++ b/tests/unit/pipelex/tools/misc/test_http_utils.py
@@ -23,6 +23,45 @@ def _raise_status_error(status_code: int) -> Callable[..., NoReturn]:
class TestValidateHttpUrl:
+ """Tests for HTTP URL validation HEAD/GET fallback logic and warn-only contract."""
+
+ def test_head_success_does_not_fall_back_to_get(self, mocker: MockerFixture) -> None:
+ """When HEAD returns 200, no GET request is made."""
+ mock_head_response = mocker.MagicMock()
+ mock_head_response.status_code = 200
+ mock_head_response.raise_for_status = mocker.MagicMock()
+ mocker.patch("pipelex.tools.misc.http_utils.httpx.head", return_value=mock_head_response)
+ mock_stream = mocker.patch("pipelex.tools.misc.http_utils.httpx.stream")
+
+ validate_url_resource_exists("https://example.com/file.png")
+
+ mock_head_response.raise_for_status.assert_called_once()
+ mock_stream.assert_not_called()
+
+ @pytest.mark.parametrize(
+ "status_code",
+ [
+ pytest.param(403, id="forbidden"),
+ pytest.param(405, id="method-not-allowed"),
+ ],
+ )
+ def test_head_rejection_falls_back_to_get(self, mocker: MockerFixture, status_code: int) -> None:
+ """When HEAD returns a rejection code (403, 405), a streaming GET is attempted."""
+ mock_head_response = mocker.MagicMock()
+ mock_head_response.status_code = status_code
+
+ mocker.patch("pipelex.tools.misc.http_utils.httpx.head", return_value=mock_head_response)
+
+ mock_get_response = mocker.MagicMock()
+ mock_get_response.raise_for_status = mocker.MagicMock()
+ mock_get_response.__enter__ = mocker.MagicMock(return_value=mock_get_response)
+ mock_get_response.__exit__ = mocker.MagicMock(return_value=False)
+ mocker.patch("pipelex.tools.misc.http_utils.httpx.stream", return_value=mock_get_response)
+
+ validate_url_resource_exists("https://example.com/file.png")
+
+ mock_get_response.raise_for_status.assert_called_once()
+
@pytest.mark.parametrize("status_code", [401, 403, 429])
def test_bot_block_status_codes_are_debug_only(
self,
diff --git a/tests/unit/pipelex/tools/test_class_registry_utils.py b/tests/unit/pipelex/tools/test_class_registry_utils.py
index c536b6bd3..67587be92 100644
--- a/tests/unit/pipelex/tools/test_class_registry_utils.py
+++ b/tests/unit/pipelex/tools/test_class_registry_utils.py
@@ -28,7 +28,7 @@ class TestClassRegistryUtilsUnit:
# Mock the global class registry
mock_registry = mocker.MagicMock()
mocker.patch(
- "pipelex.system.registries.class_registry_utils.KajsonManager.get_class_registry",
+ "pipelex.system.registries.class_registry_utils.get_class_registry",
return_value=mock_registry,
)
diff --git a/tests/unit/pipelex/tracing/conftest.py b/tests/unit/pipelex/tracing/conftest.py
new file mode 100644
index 000000000..a8e3d9604
--- /dev/null
+++ b/tests/unit/pipelex/tracing/conftest.py
@@ -0,0 +1,42 @@
+"""Shared fixtures for tracing tests."""
+
+from datetime import datetime, timezone
+
+from pipelex.graph.graphspec import EdgeKind, NodeKind
+from pipelex.tracing.trace_events import EdgeEvent, PipeStartEvent
+
+
+def make_trace_event(
+ workflow_id: str = "wf_abc",
+ sequence: int = 0,
+ pipeline_run_id: str = "run_001",
+) -> PipeStartEvent:
+ """Create a minimal PipeStartEvent for testing."""
+ return PipeStartEvent(
+ pipeline_run_id=pipeline_run_id,
+ workflow_id=workflow_id,
+ timestamp=datetime(2025, 6, 15, 10, 30, 0, tzinfo=timezone.utc),
+ sequence=sequence,
+ node_id=f"graph_1:{workflow_id}:node_{sequence}",
+ pipe_code="test_pipe",
+ pipe_type="PipeLLMGenText",
+ node_kind=NodeKind.OPERATOR,
+ )
+
+
+def make_edge_event(
+ workflow_id: str = "wf_abc",
+ sequence: int = 0,
+ pipeline_run_id: str = "run_001",
+) -> EdgeEvent:
+ """Create a minimal EdgeEvent for testing."""
+ return EdgeEvent(
+ pipeline_run_id=pipeline_run_id,
+ workflow_id=workflow_id,
+ timestamp=datetime(2025, 6, 15, 10, 30, 0, tzinfo=timezone.utc),
+ sequence=sequence,
+ edge_id=f"graph_1:{workflow_id}:edge_{sequence}",
+ source_node_id=f"graph_1:{workflow_id}:node_0",
+ target_node_id=f"graph_1:{workflow_id}:node_1",
+ edge_kind=EdgeKind.CONTAINS,
+ )
diff --git a/tests/unit/pipelex/tracing/test_assembler_equivalence.py b/tests/unit/pipelex/tracing/test_assembler_equivalence.py
new file mode 100644
index 000000000..7a3c5a191
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_assembler_equivalence.py
@@ -0,0 +1,484 @@
+"""Equivalence tests: compare GraphTracer teardown output with GraphSpecAssembler output.
+
+Runs identical tracing scenarios through both paths:
+- Path A: GraphTracer in direct mode (no event log) → teardown() → GraphSpec
+- Path B: GraphTracer with InMemoryEventLog → read events → GraphSpecAssembler.assemble() → GraphSpec
+
+The two GraphSpecs must be structurally identical (same nodes, edges, statuses, data flow),
+allowing for normalized IDs (workflow_id segment stripped) and ignored TimingSpec.
+"""
+
+import re
+from datetime import datetime, timedelta, timezone
+from typing import Any, Callable
+
+import pytest
+
+from pipelex.graph.graph_context import GraphContext
+from pipelex.graph.graph_tracer import GraphTracer
+from pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, IOSpec, NodeKind, NodeSpec
+from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
+from pipelex.tracing.in_memory_event_log import InMemoryEventLog
+from tests.unit.pipelex.graph.conftest import make_defaulted_data_inclusion_config
+
+# ---------------------------------------------------------------------------
+# Shared constants
+# ---------------------------------------------------------------------------
+
+_GRAPH_ID = "equiv_graph"
+_PIPELINE_RUN_ID = "equiv_run_001"
+_WORKFLOW_ID = "wf_equiv"
+_T0 = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+
+# ---------------------------------------------------------------------------
+# Structural comparison helpers
+# ---------------------------------------------------------------------------
+
+# Pattern to strip the workflow_id segment from node/edge IDs:
+# "graph_id:wf_xxx:node_0" → "graph_id:node_0"
+_WF_SEGMENT_RE = re.compile(r":" + re.escape(_WORKFLOW_ID) + r":")
+
+
+def _normalize_id(identifier: str) -> str:
+ """Strip the workflow_id segment from a node or edge ID."""
+ return _WF_SEGMENT_RE.sub(":", identifier)
+
+
+def _normalize_node(node: NodeSpec) -> dict[str, Any]:
+ """Extract structurally comparable fields from a NodeSpec, ignoring timing."""
+ return {
+ "node_id": _normalize_id(node.node_id),
+ "kind": node.kind,
+ "pipe_code": node.pipe_code,
+ "pipe_type": node.pipe_type,
+ "status": node.status,
+ "error_type": node.error.error_type if node.error else None,
+ "error_message": node.error.message if node.error else None,
+ "metrics": node.metrics,
+ "inputs": sorted(
+ [(spec.name, spec.digest) for spec in node.node_io.inputs],
+ key=lambda pair: (pair[0] or "", pair[1] or ""),
+ ),
+ "outputs": sorted(
+ [(spec.name, spec.digest) for spec in node.node_io.outputs],
+ key=lambda pair: (pair[0] or "", pair[1] or ""),
+ ),
+ }
+
+
+def _normalize_edge(edge: EdgeSpec) -> tuple[str, str, EdgeKind, str | None]:
+ """Extract structurally comparable fields from an EdgeSpec."""
+ return (
+ _normalize_id(edge.source),
+ _normalize_id(edge.target),
+ edge.kind,
+ edge.label,
+ )
+
+
+def _assert_graphs_equivalent(direct_spec: GraphSpec, assembled_spec: GraphSpec) -> None:
+ """Assert two GraphSpecs are structurally equivalent.
+
+ Compares nodes and edges with normalized IDs and without timing information.
+ """
+ # Compare nodes by normalized ID
+ direct_normalized = [_normalize_node(node) for node in direct_spec.nodes]
+ direct_nodes = {normalized["node_id"]: normalized for normalized in direct_normalized}
+ assembled_normalized = [_normalize_node(node) for node in assembled_spec.nodes]
+ assembled_nodes = {normalized["node_id"]: normalized for normalized in assembled_normalized}
+
+ assert set(direct_nodes.keys()) == set(assembled_nodes.keys()), (
+ f"Node ID mismatch.\n"
+ f"Direct only: {set(direct_nodes.keys()) - set(assembled_nodes.keys())}\n"
+ f"Assembled only: {set(assembled_nodes.keys()) - set(direct_nodes.keys())}"
+ )
+
+ for node_id, direct_node in direct_nodes.items():
+ assert direct_node == assembled_nodes[node_id], f"Node {node_id} differs.\nDirect: {direct_node}\nAssembled: {assembled_nodes[node_id]}"
+
+ # Compare edges as sets (order doesn't matter)
+ direct_edges = {_normalize_edge(edge) for edge in direct_spec.edges}
+ assembled_edges = {_normalize_edge(edge) for edge in assembled_spec.edges}
+
+ assert direct_edges == assembled_edges, (
+ f"Edge mismatch.\nDirect only: {direct_edges - assembled_edges}\nAssembled only: {assembled_edges - direct_edges}"
+ )
+
+
+# ---------------------------------------------------------------------------
+# Scenario runners: execute the same scenario through both paths
+# ---------------------------------------------------------------------------
+
+# A scenario is a callable that takes a GraphTracer and its initial GraphContext,
+# and exercises a tracing scenario on it.
+ScenarioFn = Callable[[GraphTracer, GraphContext], None]
+
+
+def _run_both_paths(scenario: ScenarioFn) -> tuple[GraphSpec, GraphSpec]:
+ """Run a scenario through both direct and event-log paths, return both GraphSpecs."""
+ data_inclusion = make_defaulted_data_inclusion_config()
+
+ # Path A: direct mode (no event log)
+ tracer_direct = GraphTracer()
+ ctx_direct = tracer_direct.setup(graph_id=_GRAPH_ID, data_inclusion=data_inclusion)
+ scenario(tracer_direct, ctx_direct)
+ direct_spec = tracer_direct.teardown()
+ assert direct_spec is not None
+
+ # Path B: event log mode
+ event_log = InMemoryEventLog()
+ tracer_event = GraphTracer()
+ ctx_event = tracer_event.setup(
+ graph_id=_GRAPH_ID,
+ data_inclusion=data_inclusion,
+ event_log=event_log,
+ workflow_id=_WORKFLOW_ID,
+ pipeline_run_id=_PIPELINE_RUN_ID,
+ )
+ scenario(tracer_event, ctx_event)
+ # Don't call teardown on the event-log tracer — we want to assemble from events
+ # But we do need to let the tracer flush remaining state
+ tracer_event.teardown()
+
+ events = event_log.read_events(_PIPELINE_RUN_ID)
+ assembled_spec = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)
+
+ return direct_spec, assembled_spec
+
+
+# ---------------------------------------------------------------------------
+# Scenarios
+# ---------------------------------------------------------------------------
+
+
+def _scenario_simple_sequence(tracer: GraphTracer, context: GraphContext) -> None:
+ """3 child pipes in sequence under a parent controller."""
+ started_at = _T0
+
+ # Parent controller
+ parent_id, child_ctx = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="sequence",
+ pipe_type="PipeSequence",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ )
+
+ # Child 1: produces digest_a
+ child1_id, _ = tracer.on_pipe_start(
+ graph_context=child_ctx,
+ pipe_code="gen_text",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=1),
+ )
+ tracer.on_pipe_end_success(
+ node_id=child1_id,
+ ended_at=started_at + timedelta(seconds=2),
+ output_spec=IOSpec(name="output_text", digest="digest_a"),
+ )
+
+ # Child 2: consumes digest_a, produces digest_b
+ child2_id, _ = tracer.on_pipe_start(
+ graph_context=child_ctx,
+ pipe_code="refine",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=3),
+ input_specs=[IOSpec(name="input_text", digest="digest_a")],
+ )
+ tracer.on_pipe_end_success(
+ node_id=child2_id,
+ ended_at=started_at + timedelta(seconds=4),
+ output_spec=IOSpec(name="refined_text", digest="digest_b"),
+ )
+
+ # Child 3: consumes digest_b
+ child3_id, _ = tracer.on_pipe_start(
+ graph_context=child_ctx,
+ pipe_code="format",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=5),
+ input_specs=[IOSpec(name="input_text", digest="digest_b")],
+ )
+ tracer.on_pipe_end_success(
+ node_id=child3_id,
+ ended_at=started_at + timedelta(seconds=6),
+ output_spec=IOSpec(name="formatted_text", digest="digest_c"),
+ )
+
+ # Parent ends
+ tracer.on_pipe_end_success(
+ node_id=parent_id,
+ ended_at=started_at + timedelta(seconds=7),
+ )
+
+
+def _scenario_parallel_branches(tracer: GraphTracer, context: GraphContext) -> None:
+ """Controller with 2 branch pipes producing PARALLEL_COMBINE edges."""
+ started_at = _T0
+
+ ctrl_id, ctrl_ctx = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="parallel",
+ pipe_type="PipeParallel",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ )
+
+ # Branch A
+ branch_a_id, _ = tracer.on_pipe_start(
+ graph_context=ctrl_ctx,
+ pipe_code="branch_a",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=1),
+ )
+ tracer.on_pipe_end_success(
+ node_id=branch_a_id,
+ ended_at=started_at + timedelta(seconds=2),
+ output_spec=IOSpec(name="out_a", digest="digest_branch_a"),
+ )
+
+ # Branch B
+ branch_b_id, _ = tracer.on_pipe_start(
+ graph_context=ctrl_ctx,
+ pipe_code="branch_b",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=1),
+ )
+ tracer.on_pipe_end_success(
+ node_id=branch_b_id,
+ ended_at=started_at + timedelta(seconds=3),
+ output_spec=IOSpec(name="out_b", digest="digest_branch_b"),
+ )
+
+ # Register parallel combine (must happen before register_controller_output)
+ tracer.register_parallel_combine(
+ combined_stuff_code="digest_combined",
+ branch_stuff_codes=["digest_branch_a", "digest_branch_b"],
+ parallel_controller_node_id=ctrl_id,
+ )
+
+ # Register controller outputs (overrides producer map)
+ tracer.register_controller_output(
+ node_id=ctrl_id,
+ output_spec=IOSpec(name="combined_a", digest="digest_branch_a"),
+ )
+ tracer.register_controller_output(
+ node_id=ctrl_id,
+ output_spec=IOSpec(name="combined_b", digest="digest_branch_b"),
+ )
+
+ # Controller ends
+ tracer.on_pipe_end_success(
+ node_id=ctrl_id,
+ ended_at=started_at + timedelta(seconds=4),
+ output_spec=IOSpec(name="combined_out", digest="digest_combined"),
+ )
+
+
+def _scenario_batch_fan_out_fan_in(tracer: GraphTracer, context: GraphContext) -> None:
+ """Batch controller with item extraction and aggregation."""
+ started_at = _T0
+
+ ctrl_id, ctrl_ctx = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="batch",
+ pipe_type="PipeBatch",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ input_specs=[IOSpec(name="input_list", digest="digest_list")],
+ )
+
+ # Register batch item extractions
+ tracer.register_batch_item_extraction(
+ list_stuff_code="digest_list",
+ item_stuff_code="digest_item_0",
+ item_index=0,
+ batch_controller_node_id=ctrl_id,
+ )
+ tracer.register_batch_item_extraction(
+ list_stuff_code="digest_list",
+ item_stuff_code="digest_item_1",
+ item_index=1,
+ batch_controller_node_id=ctrl_id,
+ )
+
+ # Branch 0: processes item 0
+ branch0_id, _ = tracer.on_pipe_start(
+ graph_context=ctrl_ctx,
+ pipe_code="process",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=1),
+ input_specs=[IOSpec(name="item", digest="digest_item_0")],
+ )
+ tracer.on_pipe_end_success(
+ node_id=branch0_id,
+ ended_at=started_at + timedelta(seconds=2),
+ output_spec=IOSpec(name="result", digest="digest_result_0"),
+ )
+
+ # Branch 1: processes item 1
+ branch1_id, _ = tracer.on_pipe_start(
+ graph_context=ctrl_ctx,
+ pipe_code="process",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=1),
+ input_specs=[IOSpec(name="item", digest="digest_item_1")],
+ )
+ tracer.on_pipe_end_success(
+ node_id=branch1_id,
+ ended_at=started_at + timedelta(seconds=3),
+ output_spec=IOSpec(name="result", digest="digest_result_1"),
+ )
+
+ # Register batch aggregations
+ tracer.register_batch_aggregation(
+ output_list_stuff_code="digest_output_list",
+ item_stuff_code="digest_result_0",
+ item_index=0,
+ batch_controller_node_id=ctrl_id,
+ )
+ tracer.register_batch_aggregation(
+ output_list_stuff_code="digest_output_list",
+ item_stuff_code="digest_result_1",
+ item_index=1,
+ batch_controller_node_id=ctrl_id,
+ )
+
+ # Controller ends
+ tracer.on_pipe_end_success(
+ node_id=ctrl_id,
+ ended_at=started_at + timedelta(seconds=4),
+ output_spec=IOSpec(name="output_list", digest="digest_output_list"),
+ )
+
+
+def _scenario_partial_failure(tracer: GraphTracer, context: GraphContext) -> None:
+ """Child pipe fails, parent has no end event → FAILED + CANCELED nodes."""
+ started_at = _T0
+
+ _parent_id, child_ctx = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="sequence",
+ pipe_type="PipeSequence",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ )
+
+ child_id, _ = tracer.on_pipe_start(
+ graph_context=child_ctx,
+ pipe_code="gen_text",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=1),
+ )
+
+ tracer.on_pipe_end_error(
+ node_id=child_id,
+ ended_at=started_at + timedelta(seconds=2),
+ error_type="LLMError",
+ error_message="Model returned an error",
+ )
+ # Parent has no end event → will be marked CANCELED
+
+
+def _scenario_pass_through(tracer: GraphTracer, context: GraphContext) -> None:
+ """Pass-through output (same digest as input) is not registered as producer."""
+ started_at = _T0
+
+ # Pass-through pipe: input digest_a → output digest_a (same)
+ pt_id, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="passthrough",
+ pipe_type="PipeParallel",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ input_specs=[IOSpec(name="main_input", digest="digest_a")],
+ )
+ tracer.on_pipe_end_success(
+ node_id=pt_id,
+ ended_at=started_at + timedelta(seconds=1),
+ output_spec=IOSpec(name="main_output", digest="digest_a"), # Same digest = pass-through
+ )
+
+ # Downstream pipe consumes digest_a
+ down_id, _ = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="consumer",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=2),
+ input_specs=[IOSpec(name="input", digest="digest_a")],
+ )
+ tracer.on_pipe_end_success(
+ node_id=down_id,
+ ended_at=started_at + timedelta(seconds=3),
+ )
+
+
+def _scenario_condition_selected_outcome(tracer: GraphTracer, context: GraphContext) -> None:
+ """Condition pipe with SELECTED_OUTCOME edge."""
+ started_at = _T0
+
+ cond_id, cond_ctx = tracer.on_pipe_start(
+ graph_context=context,
+ pipe_code="check",
+ pipe_type="PipeCondition",
+ node_kind=NodeKind.CONTROLLER,
+ started_at=started_at,
+ )
+
+ outcome_id, _ = tracer.on_pipe_start(
+ graph_context=cond_ctx,
+ pipe_code="branch_true",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ started_at=started_at + timedelta(seconds=1),
+ )
+
+ tracer.add_edge(
+ source_node_id=cond_id,
+ target_node_id=outcome_id,
+ edge_kind=EdgeKind.SELECTED_OUTCOME,
+ label="true",
+ )
+
+ tracer.on_pipe_end_success(
+ node_id=outcome_id,
+ ended_at=started_at + timedelta(seconds=2),
+ )
+ tracer.on_pipe_end_success(
+ node_id=cond_id,
+ ended_at=started_at + timedelta(seconds=3),
+ )
+
+
+# ---------------------------------------------------------------------------
+# Tests
+# ---------------------------------------------------------------------------
+
+
+class TestAssemblerEquivalence:
+ """Equivalence tests comparing GraphTracer teardown with GraphSpecAssembler."""
+
+ @pytest.mark.parametrize(
+ "scenario_fn",
+ [
+ pytest.param(_scenario_simple_sequence, id="simple_sequence"),
+ pytest.param(_scenario_parallel_branches, id="parallel_branches"),
+ pytest.param(_scenario_batch_fan_out_fan_in, id="batch_fan_out_fan_in"),
+ pytest.param(_scenario_partial_failure, id="partial_failure"),
+ pytest.param(_scenario_pass_through, id="pass_through"),
+ pytest.param(_scenario_condition_selected_outcome, id="condition_selected_outcome"),
+ ],
+ )
+ def test_equivalence(self, scenario_fn: ScenarioFn) -> None:
+ """GraphTracer teardown and GraphSpecAssembler produce structurally identical GraphSpecs."""
+ direct_spec, assembled_spec = _run_both_paths(scenario_fn)
+ _assert_graphs_equivalent(direct_spec, assembled_spec)
diff --git a/tests/unit/pipelex/tracing/test_graphspec_assembler.py b/tests/unit/pipelex/tracing/test_graphspec_assembler.py
new file mode 100644
index 000000000..d5cbe6f2a
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_graphspec_assembler.py
@@ -0,0 +1,644 @@
+"""Tests for GraphSpecAssembler."""
+
+from datetime import datetime, timedelta, timezone
+from typing import Any
+
+from pipelex.graph.graphspec import EdgeKind, ErrorSpec, IOSpec, NodeKind, NodeStatus
+from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
+from pipelex.tracing.trace_events import (
+ BatchAggregateEvent,
+ BatchItemEvent,
+ ControllerOutputEvent,
+ EdgeEvent,
+ ParallelCombineEvent,
+ PipeEndErrorEvent,
+ PipeEndSuccessEvent,
+ PipeStartEvent,
+)
+
+# ---------------------------------------------------------------------------
+# Shared test constants and helpers
+# ---------------------------------------------------------------------------
+
+_GRAPH_ID = "test_graph"
+_PIPELINE_RUN_ID = "run_001"
+_WF_A = "wf_a"
+_WF_B = "wf_b"
+_T0 = datetime(2025, 6, 15, 10, 0, 0, tzinfo=timezone.utc)
+
+
+def _time_at(seconds: int) -> datetime:
+ """Create a timestamp offset from T0."""
+ return _T0 + timedelta(seconds=seconds)
+
+
+def _node_id(workflow_id: str, index: int) -> str:
+ return f"{_GRAPH_ID}:{workflow_id}:node_{index}"
+
+
+def _edge_id(workflow_id: str, index: int) -> str:
+ return f"{_GRAPH_ID}:{workflow_id}:edge_{index}"
+
+
+def _base(workflow_id: str, sequence: int, timestamp_offset: int = 0) -> dict[str, Any]:
+ """Base fields for event construction."""
+ return {
+ "pipeline_run_id": _PIPELINE_RUN_ID,
+ "workflow_id": workflow_id,
+ "timestamp": _time_at(timestamp_offset or sequence),
+ "sequence": sequence,
+ }
+
+
+def _io_spec(name: str, digest: str) -> IOSpec:
+ return IOSpec(name=name, digest=digest)
+
+
+# ---------------------------------------------------------------------------
+# Tests
+# ---------------------------------------------------------------------------
+
+
+class TestGraphSpecAssembler:
+ """Tests for GraphSpecAssembler.assemble()."""
+
+ def test_empty_events(self) -> None:
+ """Empty event list produces empty GraphSpec."""
+ result = GraphSpecAssembler.assemble(events=[], graph_id=_GRAPH_ID)
+ assert result.graph_id == _GRAPH_ID
+ assert result.nodes == []
+ assert result.edges == []
+
+ def test_simple_sequence(self) -> None:
+ """3 child pipes in sequence under a parent.
+
+ parent contains child_1, child_2, child_3.
+ child_1 produces digest_a, child_2 consumes digest_a and produces digest_b,
+ child_3 consumes digest_b.
+ Expected: 4 SUCCEEDED nodes, 3 CONTAINS edges, 2 DATA edges.
+ """
+ parent = _node_id(_WF_A, 0)
+ child_1 = _node_id(_WF_A, 1)
+ child_2 = _node_id(_WF_A, 2)
+ child_3 = _node_id(_WF_A, 3)
+
+ events = [
+ # Parent starts
+ PipeStartEvent(
+ **_base(_WF_A, 0),
+ node_id=parent,
+ pipe_code="sequence",
+ pipe_type="PipeSequence",
+ node_kind=NodeKind.CONTROLLER,
+ ),
+ # Child 1 starts
+ PipeStartEvent(
+ **_base(_WF_A, 1),
+ node_id=child_1,
+ parent_node_id=parent,
+ pipe_code="gen_text",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ ),
+ # CONTAINS edge parent → child_1
+ EdgeEvent(
+ **_base(_WF_A, 2),
+ edge_id=_edge_id(_WF_A, 0),
+ source_node_id=parent,
+ target_node_id=child_1,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ # Child 1 ends with output digest_a
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 3),
+ node_id=child_1,
+ ended_at=_time_at(3),
+ output_spec=_io_spec("output_text", "digest_a"),
+ ),
+ # Child 2 starts with input digest_a
+ PipeStartEvent(
+ **_base(_WF_A, 4),
+ node_id=child_2,
+ parent_node_id=parent,
+ pipe_code="refine",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[_io_spec("input_text", "digest_a")],
+ ),
+ # CONTAINS edge parent → child_2
+ EdgeEvent(
+ **_base(_WF_A, 5),
+ edge_id=_edge_id(_WF_A, 1),
+ source_node_id=parent,
+ target_node_id=child_2,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ # Child 2 ends with output digest_b
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 6),
+ node_id=child_2,
+ ended_at=_time_at(6),
+ output_spec=_io_spec("refined_text", "digest_b"),
+ ),
+ # Child 3 starts with input digest_b
+ PipeStartEvent(
+ **_base(_WF_A, 7),
+ node_id=child_3,
+ parent_node_id=parent,
+ pipe_code="format",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[_io_spec("input_text", "digest_b")],
+ ),
+ # CONTAINS edge parent → child_3
+ EdgeEvent(
+ **_base(_WF_A, 8),
+ edge_id=_edge_id(_WF_A, 2),
+ source_node_id=parent,
+ target_node_id=child_3,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ # Child 3 ends
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 9),
+ node_id=child_3,
+ ended_at=_time_at(9),
+ output_spec=_io_spec("formatted_text", "digest_c"),
+ ),
+ # Parent ends
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 10),
+ node_id=parent,
+ ended_at=_time_at(10),
+ ),
+ ]
+
+ result = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)
+
+ # 4 nodes, all SUCCEEDED
+ assert len(result.nodes) == 4
+ for node in result.nodes:
+ assert node.status == NodeStatus.SUCCEEDED
+
+ # 3 CONTAINS edges + 2 DATA edges = 5 total
+ contains_edges = [edge for edge in result.edges if edge.kind == EdgeKind.CONTAINS]
+ data_edges = [edge for edge in result.edges if edge.kind == EdgeKind.DATA]
+
+ assert len(contains_edges) == 3
+ assert len(data_edges) == 2
+
+ # DATA edges: child_1 → child_2 (digest_a), child_2 → child_3 (digest_b)
+ data_edge_pairs = {(edge.source, edge.target) for edge in data_edges}
+ assert (child_1, child_2) in data_edge_pairs
+ assert (child_2, child_3) in data_edge_pairs
+
+ def test_parallel_branches(self) -> None:
+ """Controller + 2 branch pipes → PARALLEL_COMBINE edges."""
+ controller = _node_id(_WF_A, 0)
+ branch_1 = _node_id(_WF_A, 1)
+ branch_2 = _node_id(_WF_A, 2)
+
+ events = [
+ PipeStartEvent(
+ **_base(_WF_A, 0),
+ node_id=controller,
+ pipe_code="parallel",
+ pipe_type="PipeParallel",
+ node_kind=NodeKind.CONTROLLER,
+ ),
+ PipeStartEvent(
+ **_base(_WF_A, 1),
+ node_id=branch_1,
+ parent_node_id=controller,
+ pipe_code="branch_a",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ ),
+ EdgeEvent(
+ **_base(_WF_A, 2),
+ edge_id=_edge_id(_WF_A, 0),
+ source_node_id=controller,
+ target_node_id=branch_1,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 3),
+ node_id=branch_1,
+ ended_at=_time_at(3),
+ output_spec=_io_spec("branch_out", "digest_branch_1"),
+ ),
+ PipeStartEvent(
+ **_base(_WF_A, 4),
+ node_id=branch_2,
+ parent_node_id=controller,
+ pipe_code="branch_b",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ ),
+ EdgeEvent(
+ **_base(_WF_A, 5),
+ edge_id=_edge_id(_WF_A, 1),
+ source_node_id=controller,
+ target_node_id=branch_2,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 6),
+ node_id=branch_2,
+ ended_at=_time_at(6),
+ output_spec=_io_spec("branch_out", "digest_branch_2"),
+ ),
+ # ParallelCombine with snapshotted branch producers
+ ParallelCombineEvent(
+ **_base(_WF_A, 7),
+ combined_stuff_code="digest_combined",
+ branch_stuff_codes=["digest_branch_1", "digest_branch_2"],
+ parallel_controller_node_id=controller,
+ branch_producer_node_ids=[
+ ("digest_branch_1", branch_1),
+ ("digest_branch_2", branch_2),
+ ],
+ ),
+ # Controller registers combined output (overrides producer map)
+ ControllerOutputEvent(
+ **_base(_WF_A, 8),
+ node_id=controller,
+ output_spec=_io_spec("combined_out", "digest_combined"),
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 9),
+ node_id=controller,
+ ended_at=_time_at(9),
+ ),
+ ]
+
+ result = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)
+
+ parallel_edges = [edge for edge in result.edges if edge.kind == EdgeKind.PARALLEL_COMBINE]
+ assert len(parallel_edges) == 2
+
+ parallel_pairs = {(edge.source, edge.target) for edge in parallel_edges}
+ assert (branch_1, controller) in parallel_pairs
+ assert (branch_2, controller) in parallel_pairs
+
+ def test_batch_fan_out_fan_in(self) -> None:
+ """Batch controller + 2 items → BATCH_ITEM + BATCH_AGGREGATE edges."""
+ controller = _node_id(_WF_A, 0)
+ branch_0 = _node_id(_WF_A, 1)
+ branch_1 = _node_id(_WF_A, 2)
+
+ events = [
+ PipeStartEvent(
+ **_base(_WF_A, 0),
+ node_id=controller,
+ pipe_code="batch",
+ pipe_type="PipeBatch",
+ node_kind=NodeKind.CONTROLLER,
+ input_specs=[_io_spec("input_list", "digest_list")],
+ ),
+ # Register batch item extractions
+ BatchItemEvent(
+ **_base(_WF_A, 1),
+ list_stuff_code="digest_list",
+ item_stuff_code="digest_item_0",
+ item_index=0,
+ batch_controller_node_id=controller,
+ ),
+ BatchItemEvent(
+ **_base(_WF_A, 2),
+ list_stuff_code="digest_list",
+ item_stuff_code="digest_item_1",
+ item_index=1,
+ batch_controller_node_id=controller,
+ ),
+ # Branch 0 processes item 0
+ PipeStartEvent(
+ **_base(_WF_A, 3),
+ node_id=branch_0,
+ parent_node_id=controller,
+ pipe_code="process",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[_io_spec("item", "digest_item_0")],
+ ),
+ EdgeEvent(
+ **_base(_WF_A, 4),
+ edge_id=_edge_id(_WF_A, 0),
+ source_node_id=controller,
+ target_node_id=branch_0,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 5),
+ node_id=branch_0,
+ ended_at=_time_at(5),
+ output_spec=_io_spec("result", "digest_result_0"),
+ ),
+ # Branch 1 processes item 1
+ PipeStartEvent(
+ **_base(_WF_A, 6),
+ node_id=branch_1,
+ parent_node_id=controller,
+ pipe_code="process",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[_io_spec("item", "digest_item_1")],
+ ),
+ EdgeEvent(
+ **_base(_WF_A, 7),
+ edge_id=_edge_id(_WF_A, 1),
+ source_node_id=controller,
+ target_node_id=branch_1,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 8),
+ node_id=branch_1,
+ ended_at=_time_at(8),
+ output_spec=_io_spec("result", "digest_result_1"),
+ ),
+ # Register batch aggregations
+ BatchAggregateEvent(
+ **_base(_WF_A, 9),
+ output_list_stuff_code="digest_output_list",
+ item_stuff_code="digest_result_0",
+ item_index=0,
+ batch_controller_node_id=controller,
+ ),
+ BatchAggregateEvent(
+ **_base(_WF_A, 10),
+ output_list_stuff_code="digest_output_list",
+ item_stuff_code="digest_result_1",
+ item_index=1,
+ batch_controller_node_id=controller,
+ ),
+ # Controller ends
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 11),
+ node_id=controller,
+ ended_at=_time_at(11),
+ output_spec=_io_spec("output_list", "digest_output_list"),
+ ),
+ ]
+
+ result = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)
+
+ batch_item_edges = [edge for edge in result.edges if edge.kind == EdgeKind.BATCH_ITEM]
+ batch_agg_edges = [edge for edge in result.edges if edge.kind == EdgeKind.BATCH_AGGREGATE]
+
+ # 2 BATCH_ITEM edges: controller → branch_0, controller → branch_1
+ assert len(batch_item_edges) == 2
+ batch_item_pairs = {(edge.source, edge.target) for edge in batch_item_edges}
+ assert (controller, branch_0) in batch_item_pairs
+ assert (controller, branch_1) in batch_item_pairs
+
+ # 2 BATCH_AGGREGATE edges: branch_0 → controller, branch_1 → controller
+ assert len(batch_agg_edges) == 2
+ batch_agg_pairs = {(edge.source, edge.target) for edge in batch_agg_edges}
+ assert (branch_0, controller) in batch_agg_pairs
+ assert (branch_1, controller) in batch_agg_pairs
+
+ def test_condition_selected_outcome(self) -> None:
+ """Condition pipe with SELECTED_OUTCOME edge."""
+ condition = _node_id(_WF_A, 0)
+ outcome = _node_id(_WF_A, 1)
+
+ events = [
+ PipeStartEvent(
+ **_base(_WF_A, 0),
+ node_id=condition,
+ pipe_code="check",
+ pipe_type="PipeCondition",
+ node_kind=NodeKind.CONTROLLER,
+ ),
+ PipeStartEvent(
+ **_base(_WF_A, 1),
+ node_id=outcome,
+ parent_node_id=condition,
+ pipe_code="branch_true",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ ),
+ EdgeEvent(
+ **_base(_WF_A, 2),
+ edge_id=_edge_id(_WF_A, 0),
+ source_node_id=condition,
+ target_node_id=outcome,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ EdgeEvent(
+ **_base(_WF_A, 3),
+ edge_id=_edge_id(_WF_A, 1),
+ source_node_id=condition,
+ target_node_id=outcome,
+ edge_kind=EdgeKind.SELECTED_OUTCOME,
+ label="true",
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 4),
+ node_id=outcome,
+ ended_at=_time_at(4),
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 5),
+ node_id=condition,
+ ended_at=_time_at(5),
+ ),
+ ]
+
+ result = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)
+
+ selected_edges = [edge for edge in result.edges if edge.kind == EdgeKind.SELECTED_OUTCOME]
+ assert len(selected_edges) == 1
+ assert selected_edges[0].source == condition
+ assert selected_edges[0].target == outcome
+ assert selected_edges[0].label == "true"
+
+ def test_partial_failure(self) -> None:
+ """Child fails with error, parent has no end event → FAILED + CANCELED."""
+ parent = _node_id(_WF_A, 0)
+ child = _node_id(_WF_A, 1)
+
+ events = [
+ PipeStartEvent(
+ **_base(_WF_A, 0),
+ node_id=parent,
+ pipe_code="sequence",
+ pipe_type="PipeSequence",
+ node_kind=NodeKind.CONTROLLER,
+ ),
+ PipeStartEvent(
+ **_base(_WF_A, 1),
+ node_id=child,
+ parent_node_id=parent,
+ pipe_code="gen_text",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ ),
+ EdgeEvent(
+ **_base(_WF_A, 2),
+ edge_id=_edge_id(_WF_A, 0),
+ source_node_id=parent,
+ target_node_id=child,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ PipeEndErrorEvent(
+ **_base(_WF_A, 3),
+ node_id=child,
+ ended_at=_time_at(3),
+ error=ErrorSpec(
+ error_type="LLMError",
+ message="Model returned an error",
+ ),
+ ),
+ # No end event for parent
+ ]
+
+ result = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)
+
+ nodes_by_id = {node.node_id: node for node in result.nodes}
+ assert nodes_by_id[child].status == NodeStatus.FAILED
+ child_error = nodes_by_id[child].error
+ assert child_error is not None
+ assert child_error.error_type == "LLMError"
+ assert nodes_by_id[parent].status == NodeStatus.CANCELED
+
+ def test_multiple_workflows(self) -> None:
+ """Events from two workflows assemble into one graph."""
+ node_a0 = _node_id(_WF_A, 0)
+ node_b0 = _node_id(_WF_B, 0)
+
+ events = [
+ # wf_a events (sorted first alphabetically)
+ PipeStartEvent(
+ **_base(_WF_A, 0),
+ node_id=node_a0,
+ pipe_code="pipe_a",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 1),
+ node_id=node_a0,
+ ended_at=_time_at(1),
+ output_spec=_io_spec("output", "digest_from_a"),
+ ),
+ # wf_b events (sorted second)
+ PipeStartEvent(
+ **_base(_WF_B, 0),
+ node_id=node_b0,
+ pipe_code="pipe_b",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[_io_spec("input", "digest_from_a")],
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_B, 1),
+ node_id=node_b0,
+ ended_at=_time_at(3),
+ ),
+ ]
+
+ result = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)
+
+ assert len(result.nodes) == 2
+ data_edges = [edge for edge in result.edges if edge.kind == EdgeKind.DATA]
+ assert len(data_edges) == 1
+ assert data_edges[0].source == node_a0
+ assert data_edges[0].target == node_b0
+
+ def test_cross_workflow_producer_map(self) -> None:
+ """Producer in alphabetically-later workflow, consumer in earlier.
+
+ wf_a consumes digest_x (events come first in sorted order).
+ wf_b produces digest_x (events come second).
+ Two-pass algorithm builds complete producer map in Pass 1
+ before generating DATA edges in Pass 2.
+ """
+ consumer = _node_id(_WF_A, 0)
+ producer = _node_id(_WF_B, 0)
+
+ events = [
+ # wf_a: consumer (sorted first)
+ PipeStartEvent(
+ **_base(_WF_A, 0),
+ node_id=consumer,
+ pipe_code="consumer_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[_io_spec("input", "digest_x")],
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 1),
+ node_id=consumer,
+ ended_at=_time_at(5),
+ ),
+ # wf_b: producer (sorted second)
+ PipeStartEvent(
+ **_base(_WF_B, 0),
+ node_id=producer,
+ pipe_code="producer_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_B, 1),
+ node_id=producer,
+ ended_at=_time_at(3),
+ output_spec=_io_spec("output", "digest_x"),
+ ),
+ ]
+
+ result = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)
+
+ data_edges = [edge for edge in result.edges if edge.kind == EdgeKind.DATA]
+ assert len(data_edges) == 1
+ assert data_edges[0].source == producer
+ assert data_edges[0].target == consumer
+
+ def test_pass_through_detection(self) -> None:
+ """Output digest matching input digest is not registered as producer."""
+ passthrough = _node_id(_WF_A, 0)
+ downstream = _node_id(_WF_A, 1)
+
+ events = [
+ # Pass-through pipe: input digest_a → output digest_a (same)
+ PipeStartEvent(
+ **_base(_WF_A, 0),
+ node_id=passthrough,
+ pipe_code="passthrough",
+ pipe_type="PipeParallel",
+ node_kind=NodeKind.CONTROLLER,
+ input_specs=[_io_spec("main_input", "digest_a")],
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 1),
+ node_id=passthrough,
+ ended_at=_time_at(1),
+ output_spec=_io_spec("main_output", "digest_a"), # Same digest = pass-through
+ ),
+ # Downstream pipe consumes digest_a
+ PipeStartEvent(
+ **_base(_WF_A, 2),
+ node_id=downstream,
+ pipe_code="consumer",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[_io_spec("input", "digest_a")],
+ ),
+ PipeEndSuccessEvent(
+ **_base(_WF_A, 3),
+ node_id=downstream,
+ ended_at=_time_at(3),
+ ),
+ ]
+
+ result = GraphSpecAssembler.assemble(events=events, graph_id=_GRAPH_ID)
+
+ # No DATA edge from passthrough → downstream because passthrough didn't register as producer
+ data_edges = [edge for edge in result.edges if edge.kind == EdgeKind.DATA]
+ assert len(data_edges) == 0
+
+ # Verify passthrough node has no output_specs (pass-through was skipped)
+ passthrough_node = next(node for node in result.nodes if node.node_id == passthrough)
+ assert len(passthrough_node.node_io.outputs) == 0
diff --git a/tests/unit/pipelex/tracing/test_in_memory_event_log.py b/tests/unit/pipelex/tracing/test_in_memory_event_log.py
new file mode 100644
index 000000000..34d120d24
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_in_memory_event_log.py
@@ -0,0 +1,71 @@
+"""Tests for InMemoryEventLog."""
+
+from pipelex.tracing.in_memory_event_log import InMemoryEventLog
+from tests.unit.pipelex.tracing.conftest import make_trace_event
+
+
+class TestInMemoryEventLog:
+ """Tests for the in-memory event log backend."""
+
+ def test_emit_and_read_back(self) -> None:
+ """Events survive an emit/read cycle with correct order and content."""
+ event_log = InMemoryEventLog()
+ events = [make_trace_event(sequence=idx) for idx in range(3)]
+ for evt in events:
+ event_log.emit(evt)
+
+ result = event_log.read_events("run_001")
+
+ assert len(result) == 3
+ for idx, read_event in enumerate(result):
+ assert read_event.sequence == idx
+ assert read_event.workflow_id == "wf_abc"
+
+ def test_deduplication(self) -> None:
+ """Duplicate (workflow_id, sequence) pairs are deduplicated, keeping first."""
+ event_log = InMemoryEventLog()
+ event_a = make_trace_event(sequence=0)
+ event_b = make_trace_event(sequence=0)
+ event_log.emit(event_a)
+ event_log.emit(event_b)
+
+ result = event_log.read_events("run_001")
+
+ assert len(result) == 1
+ assert result[0] is event_a
+
+ def test_cleanup_removes_events(self) -> None:
+ """Cleanup removes events for one run but preserves others."""
+ event_log = InMemoryEventLog()
+ event_log.emit(make_trace_event(pipeline_run_id="run_001"))
+ event_log.emit(make_trace_event(pipeline_run_id="run_002"))
+
+ event_log.cleanup("run_001")
+
+ assert len(event_log.read_events("run_001")) == 0
+ assert len(event_log.read_events("run_002")) == 1
+
+ def test_read_nonexistent_run(self) -> None:
+ """Reading a nonexistent pipeline_run_id returns an empty list."""
+ event_log = InMemoryEventLog()
+
+ result = event_log.read_events("nonexistent_run")
+
+ assert result == []
+
+ def test_multiple_workflows_sorted(self) -> None:
+ """Events from multiple workflows are returned sorted by (workflow_id, sequence)."""
+ event_log = InMemoryEventLog()
+ event_log.emit(make_trace_event(workflow_id="wf_zzz", sequence=0))
+ event_log.emit(make_trace_event(workflow_id="wf_aaa", sequence=1))
+ event_log.emit(make_trace_event(workflow_id="wf_aaa", sequence=0))
+
+ result = event_log.read_events("run_001")
+
+ assert len(result) == 3
+ assert result[0].workflow_id == "wf_aaa"
+ assert result[0].sequence == 0
+ assert result[1].workflow_id == "wf_aaa"
+ assert result[1].sequence == 1
+ assert result[2].workflow_id == "wf_zzz"
+ assert result[2].sequence == 0
diff --git a/tests/unit/pipelex/tracing/test_ndjson_dedup_assembly.py b/tests/unit/pipelex/tracing/test_ndjson_dedup_assembly.py
new file mode 100644
index 000000000..479257837
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_ndjson_dedup_assembly.py
@@ -0,0 +1,168 @@
+"""Test that NDJSON deduplication produces correct graphs when Temporal replays events.
+
+Validates the end-to-end path: write duplicate events → read with dedup → assemble → correct GraphSpec.
+"""
+
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from pipelex.graph.graphspec import EdgeKind, IOSpec, NodeKind, NodeStatus
+from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from pipelex.tracing.trace_events import (
+ EdgeEvent,
+ PipeEndSuccessEvent,
+ PipeStartEvent,
+ TraceEvent,
+)
+
+_GRAPH_ID = "dedup_test"
+_PIPELINE_RUN_ID = "dedup_run_001"
+_WF_ID = "wf_replay"
+_T0 = datetime(2025, 7, 1, 12, 0, 0, tzinfo=timezone.utc)
+
+
+def _time_at(seconds: int) -> datetime:
+ return _T0 + timedelta(seconds=seconds)
+
+
+def _base(sequence: int) -> dict[str, Any]:
+ return {
+ "pipeline_run_id": _PIPELINE_RUN_ID,
+ "workflow_id": _WF_ID,
+ "timestamp": _time_at(sequence),
+ "sequence": sequence,
+ }
+
+
+def _make_sequence_events() -> list[TraceEvent]:
+ """Create events for a simple sequence: parent + 2 children with DATA flow."""
+ parent = f"{_GRAPH_ID}:{_WF_ID}:node_0"
+ child_1 = f"{_GRAPH_ID}:{_WF_ID}:node_1"
+ child_2 = f"{_GRAPH_ID}:{_WF_ID}:node_2"
+
+ return [
+ PipeStartEvent(
+ **_base(0),
+ node_id=parent,
+ pipe_code="sequence",
+ pipe_type="PipeSequence",
+ node_kind=NodeKind.CONTROLLER,
+ ),
+ PipeStartEvent(
+ **_base(1),
+ node_id=child_1,
+ parent_node_id=parent,
+ pipe_code="step_one",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ ),
+ EdgeEvent(
+ **_base(2),
+ edge_id=f"{_GRAPH_ID}:{_WF_ID}:edge_0",
+ source_node_id=parent,
+ target_node_id=child_1,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ PipeEndSuccessEvent(
+ **_base(3),
+ node_id=child_1,
+ ended_at=_time_at(3),
+ output_spec=IOSpec(name="output", digest="digest_a"),
+ ),
+ PipeStartEvent(
+ **_base(4),
+ node_id=child_2,
+ parent_node_id=parent,
+ pipe_code="step_two",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[IOSpec(name="input", digest="digest_a")],
+ ),
+ EdgeEvent(
+ **_base(5),
+ edge_id=f"{_GRAPH_ID}:{_WF_ID}:edge_1",
+ source_node_id=parent,
+ target_node_id=child_2,
+ edge_kind=EdgeKind.CONTAINS,
+ ),
+ PipeEndSuccessEvent(
+ **_base(6),
+ node_id=child_2,
+ ended_at=_time_at(6),
+ output_spec=IOSpec(name="output", digest="digest_b"),
+ ),
+ PipeEndSuccessEvent(
+ **_base(7),
+ node_id=parent,
+ ended_at=_time_at(7),
+ ),
+ ]
+
+
+class TestNdjsonDedupAssembly:
+ """Verify that duplicate events from Temporal replay are deduplicated and assemble correctly."""
+
+ def test_duplicated_events_produce_correct_graph(self, tmp_path: Path) -> None:
+ """Write every event twice (simulating Temporal replay), read, assemble — graph is correct."""
+ events = _make_sequence_events()
+
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ for event in events:
+ event_log.emit(event)
+ event_log.emit(event) # Replay duplicate
+ event_log.close()
+
+ read_events = event_log.read_events(_PIPELINE_RUN_ID)
+ assert len(read_events) == len(events), "Dedup should halve the event count"
+
+ graph = GraphSpecAssembler.assemble(events=read_events, graph_id=_GRAPH_ID)
+
+ assert len(graph.nodes) == 3
+ for node in graph.nodes:
+ assert node.status == NodeStatus.SUCCEEDED
+
+ contains_edges = [edge for edge in graph.edges if edge.kind == EdgeKind.CONTAINS]
+ data_edges = [edge for edge in graph.edges if edge.kind == EdgeKind.DATA]
+ assert len(contains_edges) == 2
+ assert len(data_edges) == 1
+
+ def test_partial_replay_still_correct(self, tmp_path: Path) -> None:
+ """Only some events are replayed — graph is still correct with no duplicates."""
+ events = _make_sequence_events()
+
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ for index_event, event in enumerate(events):
+ event_log.emit(event)
+ if index_event % 2 == 0:
+ event_log.emit(event) # Replay only even-indexed events
+ event_log.close()
+
+ read_events = event_log.read_events(_PIPELINE_RUN_ID)
+ assert len(read_events) == len(events)
+
+ graph = GraphSpecAssembler.assemble(events=read_events, graph_id=_GRAPH_ID)
+
+ assert len(graph.nodes) == 3
+ for node in graph.nodes:
+ assert node.status == NodeStatus.SUCCEEDED
+
+ @pytest.mark.parametrize("replay_count", [3, 5])
+ def test_multiple_replays_still_correct(self, tmp_path: Path, replay_count: int) -> None:
+ """Events replayed N times still assemble into a single correct graph."""
+ events = _make_sequence_events()
+
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ for _round in range(replay_count):
+ for event in events:
+ event_log.emit(event)
+ event_log.close()
+
+ read_events = event_log.read_events(_PIPELINE_RUN_ID)
+ assert len(read_events) == len(events)
+
+ graph = GraphSpecAssembler.assemble(events=read_events, graph_id=_GRAPH_ID)
+ assert len(graph.nodes) == 3
diff --git a/tests/unit/pipelex/tracing/test_ndjson_event_log.py b/tests/unit/pipelex/tracing/test_ndjson_event_log.py
new file mode 100644
index 000000000..0b871cec8
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_ndjson_event_log.py
@@ -0,0 +1,180 @@
+"""Tests for NdjsonEventLog."""
+
+import logging
+import multiprocessing
+from pathlib import Path
+
+import pytest
+
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from pipelex.tracing.trace_events import EdgeEvent, PipeStartEvent
+from tests.unit.pipelex.tracing.conftest import make_edge_event, make_trace_event
+
+
+def _worker_fn(traces_dir: str, pipeline_run_id: str, workflow_id: str, num_events: int) -> None:
+ """Write events from a subprocess. Must be module-level for macOS spawn."""
+ event_log = NdjsonEventLog(traces_dir=traces_dir)
+ for index_event in range(num_events):
+ event = make_trace_event(
+ workflow_id=workflow_id,
+ sequence=index_event,
+ pipeline_run_id=pipeline_run_id,
+ )
+ event_log.emit(event)
+ event_log.close()
+
+
+class TestNdjsonEventLog:
+ """Tests for the NDJSON file-based event log backend."""
+
+ def test_emit_and_read_back(self, tmp_path: Path) -> None:
+ """Events survive an emit/read cycle with correct order and content."""
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ events = [make_trace_event(sequence=idx) for idx in range(3)]
+ for evt in events:
+ event_log.emit(evt)
+
+ result = event_log.read_events("run_001")
+
+ assert len(result) == 3
+ for idx, read_event in enumerate(result):
+ assert read_event.sequence == idx
+ assert read_event.workflow_id == "wf_abc"
+ assert isinstance(read_event, PipeStartEvent)
+ assert read_event.pipe_code == "test_pipe"
+
+ def test_deduplication(self, tmp_path: Path) -> None:
+ """Duplicate (workflow_id, event_kind, sequence) triples are deduplicated."""
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ event_log.emit(make_trace_event(sequence=0))
+ event_log.emit(make_trace_event(sequence=0))
+
+ result = event_log.read_events("run_001")
+
+ assert len(result) == 1
+
+ def test_different_event_kinds_not_deduped(self, tmp_path: Path) -> None:
+ """Events with same (workflow_id, sequence) but different event_kind are NOT deduped."""
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ event_log.emit(make_trace_event(workflow_id="wf_abc", sequence=0))
+ event_log.emit(make_edge_event(workflow_id="wf_abc", sequence=0))
+
+ result = event_log.read_events("run_001")
+
+ assert len(result) == 2
+ event_kinds = {type(evt) for evt in result}
+ assert PipeStartEvent in event_kinds
+ assert EdgeEvent in event_kinds
+
+ def test_multiple_workflows(self, tmp_path: Path) -> None:
+ """Events from multiple workflows are returned sorted by (workflow_id, sequence)."""
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ event_log.emit(make_trace_event(workflow_id="wf_zzz", sequence=0))
+ event_log.emit(make_trace_event(workflow_id="wf_aaa", sequence=1))
+ event_log.emit(make_trace_event(workflow_id="wf_aaa", sequence=0))
+ event_log.emit(make_trace_event(workflow_id="wf_mmm", sequence=0))
+
+ result = event_log.read_events("run_001")
+
+ assert len(result) == 4
+ assert result[0].workflow_id == "wf_aaa"
+ assert result[0].sequence == 0
+ assert result[1].workflow_id == "wf_aaa"
+ assert result[1].sequence == 1
+ assert result[2].workflow_id == "wf_mmm"
+ assert result[3].workflow_id == "wf_zzz"
+
+ def test_separate_files_per_workflow(self, tmp_path: Path) -> None:
+ """Each workflow_id gets its own NDJSON file."""
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ event_log.emit(make_trace_event(workflow_id="wf_alpha", sequence=0))
+ event_log.emit(make_trace_event(workflow_id="wf_beta", sequence=0))
+
+ run_dir = tmp_path / "run_001"
+ ndjson_files = sorted(run_dir.glob("*.ndjson"))
+ assert len(ndjson_files) == 2
+ assert ndjson_files[0].name == "wf_wf_alpha.ndjson"
+ assert ndjson_files[1].name == "wf_wf_beta.ndjson"
+
+ def test_cleanup_removes_directory(self, tmp_path: Path) -> None:
+ """Cleanup removes the pipeline run directory and closes file handles."""
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ event_log.emit(make_trace_event())
+
+ run_dir = tmp_path / "run_001"
+ assert run_dir.is_dir()
+
+ event_log.cleanup("run_001")
+
+ assert not run_dir.exists()
+
+ def test_read_nonexistent_run(self, tmp_path: Path) -> None:
+ """Reading a nonexistent pipeline_run_id returns an empty list."""
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+
+ result = event_log.read_events("nonexistent_run")
+
+ assert result == []
+
+ def test_corrupt_line_skipped(self, tmp_path: Path, caplog: pytest.LogCaptureFixture) -> None:
+ """Corrupt NDJSON lines are skipped with a warning."""
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ event_log.emit(make_trace_event(sequence=0))
+ event_log.emit(make_trace_event(sequence=1))
+ event_log.close()
+
+ ndjson_file = tmp_path / "run_001" / "wf_wf_abc.ndjson"
+ with open(ndjson_file, "a", encoding="utf-8") as fhandle:
+ fhandle.write("not valid json\n")
+ fhandle.write('{"event_kind": "pipe_start", "bad": true}\n')
+
+ with caplog.at_level(logging.WARNING):
+ result = event_log.read_events("run_001")
+
+ assert len(result) == 2
+ assert any("corrupt" in record.message.lower() or "skipping" in record.message.lower() for record in caplog.records)
+
+ def test_multiprocess_concurrent_writes(self, tmp_path: Path) -> None:
+ """Multiple processes writing to different workflow files produces no corruption."""
+ traces_dir = str(tmp_path)
+ pipeline_run_id = "run_001"
+ events_per_worker = 10
+ workflow_ids = ["wf_worker_0", "wf_worker_1", "wf_worker_2"]
+
+ processes: list[multiprocessing.Process] = []
+ for workflow_id in workflow_ids:
+ proc = multiprocessing.Process(
+ target=_worker_fn,
+ args=(traces_dir, pipeline_run_id, workflow_id, events_per_worker),
+ )
+ processes.append(proc)
+
+ for proc in processes:
+ proc.start()
+ for proc in processes:
+ proc.join(timeout=30)
+
+ for proc in processes:
+ assert proc.exitcode == 0, f"Worker process exited with code {proc.exitcode}"
+
+ reader = NdjsonEventLog(traces_dir=traces_dir)
+ result = reader.read_events(pipeline_run_id)
+
+ assert len(result) == len(workflow_ids) * events_per_worker
+
+ ndjson_files = list((tmp_path / pipeline_run_id).glob("*.ndjson"))
+ assert len(ndjson_files) == len(workflow_ids)
+
+ def test_emit_after_cleanup_creates_fresh(self, tmp_path: Path) -> None:
+ """Emitting after cleanup creates a fresh directory and file."""
+ event_log = NdjsonEventLog(traces_dir=str(tmp_path))
+ event_log.emit(make_trace_event(sequence=0))
+ event_log.cleanup("run_001")
+
+ assert not (tmp_path / "run_001").exists()
+
+ event_log.emit(make_trace_event(sequence=1))
+ result = event_log.read_events("run_001")
+
+ assert len(result) == 1
+ assert result[0].sequence == 1
diff --git a/tests/unit/pipelex/tracing/test_trace_events.py b/tests/unit/pipelex/tracing/test_trace_events.py
new file mode 100644
index 000000000..878b48f1e
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_trace_events.py
@@ -0,0 +1,347 @@
+"""Serialization round-trip tests for trace event models."""
+
+from datetime import datetime, timezone
+from typing import Any, ClassVar
+
+import pytest
+from pydantic import TypeAdapter
+
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.graph.graphspec import EdgeKind, ErrorSpec, IOSpec, NodeKind
+from pipelex.pipeline.job_metadata import JobCategory, JobMetadata, UnitJobId
+from pipelex.tracing.trace_events import (
+ AnyTraceEvent,
+ BatchAggregateEvent,
+ BatchItemEvent,
+ ControllerOutputEvent,
+ EdgeEvent,
+ ExecutionDataEvent,
+ ParallelCombineEvent,
+ PipeEndErrorEvent,
+ PipeEndSuccessEvent,
+ PipeStartEvent,
+ TraceEvent,
+ TraceEventKind,
+ UsageReportEvent,
+)
+
+
+class _Shared:
+ """Shared test data for trace event tests."""
+
+ PIPELINE_RUN_ID = "run_abc123"
+ WORKFLOW_ID = "wf_pipe_router_xyz"
+ TIMESTAMP = datetime(2025, 6, 15, 10, 30, 0, tzinfo=timezone.utc)
+ NODE_ID = "graph_1:wf_pipe_router_xyz:node_0"
+ PARENT_NODE_ID = "graph_1:wf_pipe_router_xyz:node_1"
+
+ @staticmethod
+ def make_base_fields(sequence: int = 0) -> dict[str, Any]:
+ return {
+ "pipeline_run_id": _Shared.PIPELINE_RUN_ID,
+ "workflow_id": _Shared.WORKFLOW_ID,
+ "timestamp": _Shared.TIMESTAMP,
+ "sequence": sequence,
+ }
+
+ @staticmethod
+ def make_io_spec(name: str = "input_text", digest: str = "abc123") -> IOSpec:
+ return IOSpec(
+ name=name,
+ concept="TextContent",
+ content_type="text/plain",
+ preview="Hello world",
+ size=11,
+ digest=digest,
+ )
+
+ @staticmethod
+ def make_error_spec() -> ErrorSpec:
+ return ErrorSpec(
+ error_type="LLMError",
+ message="Model returned an error",
+ stack="Traceback (most recent call last):\n File ...",
+ )
+
+ @staticmethod
+ def make_job_metadata() -> JobMetadata:
+ return JobMetadata(
+ user_id="user_test",
+ pipeline_run_id=_Shared.PIPELINE_RUN_ID,
+ pipe_code="test_pipe",
+ unit_job_id=UnitJobId.LLM_GEN_TEXT,
+ job_category=JobCategory.LLM_JOB,
+ )
+
+ @staticmethod
+ def make_llm_tokens_usage() -> LLMTokensUsage:
+ return LLMTokensUsage(
+ job_metadata=_Shared.make_job_metadata(),
+ inference_model_name="claude-sonnet",
+ unit_costs={CostCategory.INPUT: 0.003, CostCategory.OUTPUT: 0.015},
+ inference_model_id="claude-sonnet-4-20250514",
+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},
+ )
+
+
+# Type adapter for the discriminated union
+_any_trace_event_adapter: TypeAdapter[TraceEvent] = TypeAdapter(AnyTraceEvent)
+
+
+class TestTraceEvents:
+ """Serialization round-trip and discriminated union tests for trace events."""
+
+ EVENTS_AND_KINDS: ClassVar[list[tuple[str, TraceEventKind, dict[str, Any]]]] = [
+ (
+ "pipe_start",
+ TraceEventKind.PIPE_START,
+ {
+ "node_id": _Shared.NODE_ID,
+ "parent_node_id": _Shared.PARENT_NODE_ID,
+ "pipe_code": "gen_summary",
+ "pipe_type": "PipeLLM",
+ "node_kind": NodeKind.OPERATOR,
+ "input_specs": [_Shared.make_io_spec()],
+ },
+ ),
+ (
+ "pipe_end_success",
+ TraceEventKind.PIPE_END_SUCCESS,
+ {
+ "node_id": _Shared.NODE_ID,
+ "ended_at": _Shared.TIMESTAMP,
+ "output_spec": _Shared.make_io_spec(name="output_text", digest="def456"),
+ "metrics": {"tokens": 150.0},
+ },
+ ),
+ (
+ "pipe_end_error",
+ TraceEventKind.PIPE_END_ERROR,
+ {
+ "node_id": _Shared.NODE_ID,
+ "ended_at": _Shared.TIMESTAMP,
+ "error": _Shared.make_error_spec(),
+ },
+ ),
+ (
+ "edge",
+ TraceEventKind.EDGE,
+ {
+ "edge_id": "graph_1:wf_pipe_router_xyz:edge_0",
+ "source_node_id": _Shared.NODE_ID,
+ "target_node_id": _Shared.PARENT_NODE_ID,
+ "edge_kind": EdgeKind.CONTAINS,
+ "label": "child",
+ "source_stuff_digest": "abc123",
+ "target_stuff_digest": "def456",
+ },
+ ),
+ (
+ "controller_output",
+ TraceEventKind.CONTROLLER_OUTPUT,
+ {
+ "node_id": _Shared.NODE_ID,
+ "output_spec": _Shared.make_io_spec(name="ctrl_output", digest="ghi789"),
+ },
+ ),
+ (
+ "batch_item",
+ TraceEventKind.BATCH_ITEM,
+ {
+ "list_stuff_code": "list_digest_001",
+ "item_stuff_code": "item_digest_001",
+ "item_index": 0,
+ "batch_controller_node_id": _Shared.PARENT_NODE_ID,
+ },
+ ),
+ (
+ "batch_aggregate",
+ TraceEventKind.BATCH_AGGREGATE,
+ {
+ "output_list_stuff_code": "agg_digest_001",
+ "item_stuff_code": "item_digest_002",
+ "item_index": 1,
+ "batch_controller_node_id": _Shared.PARENT_NODE_ID,
+ },
+ ),
+ (
+ "parallel_combine",
+ TraceEventKind.PARALLEL_COMBINE,
+ {
+ "combined_stuff_code": "combined_001",
+ "branch_stuff_codes": ["branch_a", "branch_b"],
+ "parallel_controller_node_id": _Shared.PARENT_NODE_ID,
+ "branch_producer_node_ids": [("branch_a", "graph_1:wf_a:node_0"), ("branch_b", "graph_1:wf_b:node_0")],
+ },
+ ),
+ (
+ "execution_data",
+ TraceEventKind.EXECUTION_DATA,
+ {
+ "node_id": _Shared.NODE_ID,
+ "execution_data": {"rendered_prompt": "hello", "resolved_model": "claude-sonnet"},
+ },
+ ),
+ (
+ "usage_report",
+ TraceEventKind.USAGE_REPORT,
+ {
+ "node_id": _Shared.NODE_ID,
+ "tokens_usage": _Shared.make_llm_tokens_usage(),
+ },
+ ),
+ ]
+
+ EVENT_CLASSES: ClassVar[dict[TraceEventKind, type[TraceEvent]]] = {
+ TraceEventKind.PIPE_START: PipeStartEvent,
+ TraceEventKind.PIPE_END_SUCCESS: PipeEndSuccessEvent,
+ TraceEventKind.PIPE_END_ERROR: PipeEndErrorEvent,
+ TraceEventKind.EDGE: EdgeEvent,
+ TraceEventKind.CONTROLLER_OUTPUT: ControllerOutputEvent,
+ TraceEventKind.BATCH_ITEM: BatchItemEvent,
+ TraceEventKind.BATCH_AGGREGATE: BatchAggregateEvent,
+ TraceEventKind.PARALLEL_COMBINE: ParallelCombineEvent,
+ TraceEventKind.EXECUTION_DATA: ExecutionDataEvent,
+ TraceEventKind.USAGE_REPORT: UsageReportEvent,
+ }
+
+ @pytest.mark.parametrize(
+ ("event_kind", "extra_fields"),
+ [(entry[1], entry[2]) for entry in EVENTS_AND_KINDS],
+ ids=[entry[0] for entry in EVENTS_AND_KINDS],
+ )
+ def test_round_trip_per_event_type(self, event_kind: TraceEventKind, extra_fields: dict[str, Any]) -> None:
+ """Each event type survives JSON serialization round-trip."""
+ event_cls = self.EVENT_CLASSES[event_kind]
+ event = event_cls(**_Shared.make_base_fields(), **extra_fields)
+
+ json_str = event.model_dump_json()
+ restored = event_cls.model_validate_json(json_str)
+
+ assert restored == event
+ assert restored.model_dump()["event_kind"] == event_kind
+
+ @pytest.mark.parametrize(
+ ("event_kind", "extra_fields"),
+ [(entry[1], entry[2]) for entry in EVENTS_AND_KINDS],
+ ids=[entry[0] for entry in EVENTS_AND_KINDS],
+ )
+ def test_discriminated_union_deserialization(self, event_kind: TraceEventKind, extra_fields: dict[str, Any]) -> None:
+ """AnyTraceEvent discriminated union resolves to the correct subclass."""
+ event_cls = self.EVENT_CLASSES[event_kind]
+ event = event_cls(**_Shared.make_base_fields(), **extra_fields)
+
+ json_str = event.model_dump_json()
+ restored = _any_trace_event_adapter.validate_json(json_str)
+
+ assert type(restored) is event_cls
+ assert restored.model_dump()["event_kind"] == event_kind
+
+ def test_pipe_start_with_empty_inputs(self) -> None:
+ """PipeStartEvent works with no input specs."""
+ event = PipeStartEvent(
+ **_Shared.make_base_fields(),
+ node_id=_Shared.NODE_ID,
+ pipe_code="noop_pipe",
+ pipe_type="PipeNoop",
+ node_kind=NodeKind.OPERATOR,
+ )
+
+ json_str = event.model_dump_json()
+ restored = PipeStartEvent.model_validate_json(json_str)
+
+ assert restored.input_specs == []
+ assert restored.parent_node_id is None
+
+ def test_pipe_end_success_with_no_output(self) -> None:
+ """PipeEndSuccessEvent works with no output spec."""
+ event = PipeEndSuccessEvent(
+ **_Shared.make_base_fields(),
+ node_id=_Shared.NODE_ID,
+ ended_at=_Shared.TIMESTAMP,
+ )
+
+ json_str = event.model_dump_json()
+ restored = PipeEndSuccessEvent.model_validate_json(json_str)
+
+ assert restored.output_spec is None
+ assert restored.metrics == {}
+
+ def test_io_spec_survives_round_trip_within_event(self) -> None:
+ """IOSpec fields (digest, preview, concept) are preserved through event serialization."""
+ io_spec = IOSpec(
+ name="complex_stuff",
+ concept="StructuredData",
+ content_type="application/json",
+ preview='{"key": "value"}',
+ size=16,
+ digest="sha256_abc",
+ extra={"custom_field": "custom_value"},
+ )
+ event = PipeStartEvent(
+ **_Shared.make_base_fields(),
+ node_id=_Shared.NODE_ID,
+ pipe_code="test_pipe",
+ pipe_type="PipeLLM",
+ node_kind=NodeKind.OPERATOR,
+ input_specs=[io_spec],
+ )
+
+ json_str = event.model_dump_json()
+ restored = PipeStartEvent.model_validate_json(json_str)
+
+ restored_io = restored.input_specs[0]
+ assert restored_io.name == "complex_stuff"
+ assert restored_io.concept == "StructuredData"
+ assert restored_io.digest == "sha256_abc"
+ assert restored_io.extra == {"custom_field": "custom_value"}
+
+ def test_error_spec_survives_round_trip_within_event(self) -> None:
+ """ErrorSpec fields are preserved through event serialization."""
+ error = ErrorSpec(
+ error_type="ValidationError",
+ message="Field 'name' is required",
+ stack="long stack trace here...",
+ )
+ event = PipeEndErrorEvent(
+ **_Shared.make_base_fields(),
+ node_id=_Shared.NODE_ID,
+ ended_at=_Shared.TIMESTAMP,
+ error=error,
+ )
+
+ json_str = event.model_dump_json()
+ restored = PipeEndErrorEvent.model_validate_json(json_str)
+
+ assert restored.error.error_type == "ValidationError"
+ assert restored.error.message == "Field 'name' is required"
+ assert restored.error.stack == "long stack trace here..."
+
+ def test_usage_report_with_llm_tokens_round_trip(self) -> None:
+ """UsageReportEvent with LLMTokensUsage survives round-trip via AnyTokensUsage discriminator."""
+ tokens_usage = _Shared.make_llm_tokens_usage()
+ event = UsageReportEvent(
+ **_Shared.make_base_fields(),
+ node_id=_Shared.NODE_ID,
+ tokens_usage=tokens_usage,
+ )
+
+ json_str = event.model_dump_json()
+ restored = _any_trace_event_adapter.validate_json(json_str)
+
+ assert isinstance(restored, UsageReportEvent)
+ assert isinstance(restored.tokens_usage, LLMTokensUsage)
+ assert restored.tokens_usage.inference_model_name == "claude-sonnet"
+ assert restored.tokens_usage.nb_tokens_by_category[TokenCategory.INPUT] == 100
+ assert restored.tokens_usage.nb_tokens_by_category[TokenCategory.OUTPUT] == 50
+
+ def test_all_event_kinds_covered(self) -> None:
+ """Every TraceEventKind has a corresponding test case and event class."""
+ tested_kinds = {entry[1] for entry in self.EVENTS_AND_KINDS}
+ all_kinds = set(TraceEventKind)
+ assert tested_kinds == all_kinds, f"Missing test cases for: {all_kinds - tested_kinds}"
+
+ mapped_kinds = set(self.EVENT_CLASSES.keys())
+ assert mapped_kinds == all_kinds, f"Missing class mappings for: {all_kinds - mapped_kinds}"
diff --git a/tests/unit/pipelex/tracing/test_usage_aggregator.py b/tests/unit/pipelex/tracing/test_usage_aggregator.py
new file mode 100644
index 000000000..e749e4715
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_usage_aggregator.py
@@ -0,0 +1,106 @@
+"""Tests for UsageAggregator."""
+
+from datetime import datetime, timezone
+from typing import Any
+
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.graph.graphspec import NodeKind
+from pipelex.pipeline.job_metadata import JobCategory, JobMetadata, UnitJobId
+from pipelex.tracing.trace_events import PipeStartEvent, TraceEvent, UsageReportEvent
+from pipelex.tracing.usage_aggregator import UsageAggregator
+
+_TIMESTAMP = datetime(2025, 6, 15, 10, 30, 0, tzinfo=timezone.utc)
+_PIPELINE_RUN_ID = "run_001"
+_WORKFLOW_ID = "wf_abc"
+
+
+def _make_llm_tokens_usage(model_name: str = "claude-sonnet") -> LLMTokensUsage:
+ return LLMTokensUsage(
+ job_metadata=JobMetadata(
+ user_id="user_test",
+ pipeline_run_id=_PIPELINE_RUN_ID,
+ pipe_code="test_pipe",
+ unit_job_id=UnitJobId.LLM_GEN_TEXT,
+ job_category=JobCategory.LLM_JOB,
+ ),
+ inference_model_name=model_name,
+ unit_costs={CostCategory.INPUT: 0.003, CostCategory.OUTPUT: 0.015},
+ inference_model_id="claude-sonnet-4-20250514",
+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 50},
+ )
+
+
+def _make_base_fields(sequence: int = 0) -> dict[str, Any]:
+ return {
+ "pipeline_run_id": _PIPELINE_RUN_ID,
+ "workflow_id": _WORKFLOW_ID,
+ "timestamp": _TIMESTAMP,
+ "sequence": sequence,
+ }
+
+
+def _make_usage_event(sequence: int, model_name: str = "claude-sonnet") -> UsageReportEvent:
+ return UsageReportEvent(
+ **_make_base_fields(sequence),
+ node_id=f"graph_1:{_WORKFLOW_ID}:node_{sequence}",
+ tokens_usage=_make_llm_tokens_usage(model_name),
+ )
+
+
+def _make_pipe_start_event(sequence: int) -> PipeStartEvent:
+ return PipeStartEvent(
+ **_make_base_fields(sequence),
+ node_id=f"graph_1:{_WORKFLOW_ID}:node_{sequence}",
+ pipe_code="test_pipe",
+ pipe_type="PipeLLMGenText",
+ node_kind=NodeKind.OPERATOR,
+ )
+
+
+class TestUsageAggregator:
+ """Tests for UsageAggregator.aggregate()."""
+
+ def test_collects_all_usage_events(self) -> None:
+ """Mixed event types — only UsageReportEvent tokens extracted."""
+ events: list[TraceEvent] = [
+ _make_pipe_start_event(0),
+ _make_usage_event(1, model_name="claude-sonnet"),
+ _make_pipe_start_event(2),
+ _make_usage_event(3, model_name="gpt-4"),
+ ]
+
+ result = UsageAggregator.aggregate(events)
+
+ assert len(result) == 2
+ assert result[0].inference_model_name == "claude-sonnet"
+ assert result[1].inference_model_name == "gpt-4"
+
+ def test_empty_events_returns_empty_list(self) -> None:
+ """No events produces empty list."""
+ result = UsageAggregator.aggregate([])
+ assert result == []
+
+ def test_ignores_non_usage_events(self) -> None:
+ """Only non-usage events → empty result."""
+ events: list[TraceEvent] = [
+ _make_pipe_start_event(0),
+ _make_pipe_start_event(1),
+ ]
+
+ result = UsageAggregator.aggregate(events)
+ assert result == []
+
+ def test_preserves_order(self) -> None:
+ """Multiple usage events returned in input order."""
+ events: list[TraceEvent] = [
+ _make_usage_event(0, model_name="model_a"),
+ _make_usage_event(1, model_name="model_b"),
+ _make_usage_event(2, model_name="model_c"),
+ ]
+
+ result = UsageAggregator.aggregate(events)
+
+ assert len(result) == 3
+ assert [usage.inference_model_name for usage in result] == ["model_a", "model_b", "model_c"]
diff --git a/uv.lock b/uv.lock
index acd9f5a0d..4539a3f18 100644
--- a/uv.lock
+++ b/uv.lock
@@ -289,6 +289,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/da/42/e921fccf5015463e32a3cf6ee7f980a6ed0f395ceeaa45060b61d86486c2/anyio-4.13.0-py3-none-any.whl", hash = "sha256:08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708", size = 114353, upload-time = "2026-03-24T12:59:08.246Z" },
]
+[[package]]
+name = "argcomplete"
+version = "3.6.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/38/61/0b9ae6399dd4a58d8c1b1dc5a27d6f2808023d0b5dd3104bb99f45a33ff6/argcomplete-3.6.3.tar.gz", hash = "sha256:62e8ed4fd6a45864acc8235409461b72c9a28ee785a2011cc5eb78318786c89c", size = 73754, upload-time = "2025-10-20T03:33:34.741Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/74/f5/9373290775639cb67a2fce7f629a1c240dce9f12fe927bc32b2736e16dfc/argcomplete-3.6.3-py3-none-any.whl", hash = "sha256:f5007b3a600ccac5d25bbce33089211dfd49eab4a7718da3f10e3082525a92ce", size = 43846, upload-time = "2025-10-20T03:33:33.021Z" },
+]
+
[[package]]
name = "astroid"
version = "4.0.4"
@@ -382,6 +391,50 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/1a/39/47f9197bdd44df24d67ac8893641e16f386c984a0619ef2ee4c51fbbc019/beautifulsoup4-4.14.3-py3-none-any.whl", hash = "sha256:0918bfe44902e6ad8d57732ba310582e98da931428d231a5ecb9e7c703a735bb", size = 107721, upload-time = "2025-11-30T15:08:24.087Z" },
]
+[[package]]
+name = "black"
+version = "26.3.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "mypy-extensions" },
+ { name = "packaging" },
+ { name = "pathspec" },
+ { name = "platformdirs" },
+ { name = "pytokens" },
+ { name = "tomli", marker = "python_full_version < '3.11'" },
+ { name = "typing-extensions", marker = "python_full_version < '3.11'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e1/c5/61175d618685d42b005847464b8fb4743a67b1b8fdb75e50e5a96c31a27a/black-26.3.1.tar.gz", hash = "sha256:2c50f5063a9641c7eed7795014ba37b0f5fa227f3d408b968936e24bc0566b07", size = 666155, upload-time = "2026-03-12T03:36:03.593Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/32/a8/11170031095655d36ebc6664fe0897866f6023892396900eec0e8fdc4299/black-26.3.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:86a8b5035fce64f5dcd1b794cf8ec4d31fe458cf6ce3986a30deb434df82a1d2", size = 1866562, upload-time = "2026-03-12T03:39:58.639Z" },
+ { url = "https://files.pythonhosted.org/packages/69/ce/9e7548d719c3248c6c2abfd555d11169457cbd584d98d179111338423790/black-26.3.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:5602bdb96d52d2d0672f24f6ffe5218795736dd34807fd0fd55ccd6bf206168b", size = 1703623, upload-time = "2026-03-12T03:40:00.347Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/0a/8d17d1a9c06f88d3d030d0b1d4373c1551146e252afe4547ed601c0e697f/black-26.3.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6c54a4a82e291a1fee5137371ab488866b7c86a3305af4026bdd4dc78642e1ac", size = 1768388, upload-time = "2026-03-12T03:40:01.765Z" },
+ { url = "https://files.pythonhosted.org/packages/52/79/c1ee726e221c863cde5164f925bacf183dfdf0397d4e3f94889439b947b4/black-26.3.1-cp310-cp310-win_amd64.whl", hash = "sha256:6e131579c243c98f35bce64a7e08e87fb2d610544754675d4a0e73a070a5aa3a", size = 1412969, upload-time = "2026-03-12T03:40:03.252Z" },
+ { url = "https://files.pythonhosted.org/packages/73/a5/15c01d613f5756f68ed8f6d4ec0a1e24b82b18889fa71affd3d1f7fad058/black-26.3.1-cp310-cp310-win_arm64.whl", hash = "sha256:5ed0ca58586c8d9a487352a96b15272b7fa55d139fc8496b519e78023a8dab0a", size = 1220345, upload-time = "2026-03-12T03:40:04.892Z" },
+ { url = "https://files.pythonhosted.org/packages/17/57/5f11c92861f9c92eb9dddf515530bc2d06db843e44bdcf1c83c1427824bc/black-26.3.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:28ef38aee69e4b12fda8dba75e21f9b4f979b490c8ac0baa7cb505369ac9e1ff", size = 1851987, upload-time = "2026-03-12T03:40:06.248Z" },
+ { url = "https://files.pythonhosted.org/packages/54/aa/340a1463660bf6831f9e39646bf774086dbd8ca7fc3cded9d59bbdf4ad0a/black-26.3.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:bf9bf162ed91a26f1adba8efda0b573bc6924ec1408a52cc6f82cb73ec2b142c", size = 1689499, upload-time = "2026-03-12T03:40:07.642Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/01/b726c93d717d72733da031d2de10b92c9fa4c8d0c67e8a8a372076579279/black-26.3.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:474c27574d6d7037c1bc875a81d9be0a9a4f9ee95e62800dab3cfaadbf75acd5", size = 1754369, upload-time = "2026-03-12T03:40:09.279Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/09/61e91881ca291f150cfc9eb7ba19473c2e59df28859a11a88248b5cbbc4d/black-26.3.1-cp311-cp311-win_amd64.whl", hash = "sha256:5e9d0d86df21f2e1677cc4bd090cd0e446278bcbbe49bf3659c308c3e402843e", size = 1413613, upload-time = "2026-03-12T03:40:10.943Z" },
+ { url = "https://files.pythonhosted.org/packages/16/73/544f23891b22e7efe4d8f812371ab85b57f6a01b2fc45e3ba2e52ba985b8/black-26.3.1-cp311-cp311-win_arm64.whl", hash = "sha256:9a5e9f45e5d5e1c5b5c29b3bd4265dcc90e8b92cf4534520896ed77f791f4da5", size = 1219719, upload-time = "2026-03-12T03:40:12.597Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/f8/da5eae4fc75e78e6dceb60624e1b9662ab00d6b452996046dfa9b8a6025b/black-26.3.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b5e6f89631eb88a7302d416594a32faeee9fb8fb848290da9d0a5f2903519fc1", size = 1895920, upload-time = "2026-03-12T03:40:13.921Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/9f/04e6f26534da2e1629b2b48255c264cabf5eedc5141d04516d9d68a24111/black-26.3.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:41cd2012d35b47d589cb8a16faf8a32ef7a336f56356babd9fcf70939ad1897f", size = 1718499, upload-time = "2026-03-12T03:40:15.239Z" },
+ { url = "https://files.pythonhosted.org/packages/04/91/a5935b2a63e31b331060c4a9fdb5a6c725840858c599032a6f3aac94055f/black-26.3.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f76ff19ec5297dd8e66eb64deda23631e642c9393ab592826fd4bdc97a4bce7", size = 1794994, upload-time = "2026-03-12T03:40:17.124Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/0a/86e462cdd311a3c2a8ece708d22aba17d0b2a0d5348ca34b40cdcbea512e/black-26.3.1-cp312-cp312-win_amd64.whl", hash = "sha256:ddb113db38838eb9f043623ba274cfaf7d51d5b0c22ecb30afe58b1bb8322983", size = 1420867, upload-time = "2026-03-12T03:40:18.83Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/e5/22515a19cb7eaee3440325a6b0d95d2c0e88dd180cb011b12ae488e031d1/black-26.3.1-cp312-cp312-win_arm64.whl", hash = "sha256:dfdd51fc3e64ea4f35873d1b3fb25326773d55d2329ff8449139ebaad7357efb", size = 1230124, upload-time = "2026-03-12T03:40:20.425Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/77/5728052a3c0450c53d9bb3945c4c46b91baa62b2cafab6801411b6271e45/black-26.3.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:855822d90f884905362f602880ed8b5df1b7e3ee7d0db2502d4388a954cc8c54", size = 1895034, upload-time = "2026-03-12T03:40:21.813Z" },
+ { url = "https://files.pythonhosted.org/packages/52/73/7cae55fdfdfbe9d19e9a8d25d145018965fe2079fa908101c3733b0c55a0/black-26.3.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8a33d657f3276328ce00e4d37fe70361e1ec7614da5d7b6e78de5426cb56332f", size = 1718503, upload-time = "2026-03-12T03:40:23.666Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/87/af89ad449e8254fdbc74654e6467e3c9381b61472cc532ee350d28cfdafb/black-26.3.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f1cd08e99d2f9317292a311dfe578fd2a24b15dbce97792f9c4d752275c1fa56", size = 1793557, upload-time = "2026-03-12T03:40:25.497Z" },
+ { url = "https://files.pythonhosted.org/packages/43/10/d6c06a791d8124b843bf325ab4ac7d2f5b98731dff84d6064eafd687ded1/black-26.3.1-cp313-cp313-win_amd64.whl", hash = "sha256:c7e72339f841b5a237ff14f7d3880ddd0fc7f98a1199e8c4327f9a4f478c1839", size = 1422766, upload-time = "2026-03-12T03:40:27.14Z" },
+ { url = "https://files.pythonhosted.org/packages/59/4f/40a582c015f2d841ac24fed6390bd68f0fc896069ff3a886317959c9daf8/black-26.3.1-cp313-cp313-win_arm64.whl", hash = "sha256:afc622538b430aa4c8c853f7f63bc582b3b8030fd8c80b70fb5fa5b834e575c2", size = 1232140, upload-time = "2026-03-12T03:40:28.882Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/da/e36e27c9cebc1311b7579210df6f1c86e50f2d7143ae4fcf8a5017dc8809/black-26.3.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:2d6bfaf7fd0993b420bed691f20f9492d53ce9a2bcccea4b797d34e947318a78", size = 1889234, upload-time = "2026-03-12T03:40:30.964Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/7b/9871acf393f64a5fa33668c19350ca87177b181f44bb3d0c33b2d534f22c/black-26.3.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:f89f2ab047c76a9c03f78d0d66ca519e389519902fa27e7a91117ef7611c0568", size = 1720522, upload-time = "2026-03-12T03:40:32.346Z" },
+ { url = "https://files.pythonhosted.org/packages/03/87/e766c7f2e90c07fb7586cc787c9ae6462b1eedab390191f2b7fc7f6170a9/black-26.3.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b07fc0dab849d24a80a29cfab8d8a19187d1c4685d8a5e6385a5ce323c1f015f", size = 1787824, upload-time = "2026-03-12T03:40:33.636Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/94/2424338fb2d1875e9e83eed4c8e9c67f6905ec25afd826a911aea2b02535/black-26.3.1-cp314-cp314-win_amd64.whl", hash = "sha256:0126ae5b7c09957da2bdbd91a9ba1207453feada9e9fe51992848658c6c8e01c", size = 1445855, upload-time = "2026-03-12T03:40:35.442Z" },
+ { url = "https://files.pythonhosted.org/packages/86/43/0c3338bd928afb8ee7471f1a4eec3bdbe2245ccb4a646092a222e8669840/black-26.3.1-cp314-cp314-win_arm64.whl", hash = "sha256:92c0ec1f2cc149551a2b7b47efc32c866406b6891b0ee4625e95967c8f4acfb1", size = 1258109, upload-time = "2026-03-12T03:40:36.832Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/0d/52d98722666d6fc6c3dd4c76df339501d6efd40e0ff95e6186a7b7f0befd/black-26.3.1-py3-none-any.whl", hash = "sha256:2bd5aa94fc267d38bb21a70d7410a89f1a1d318841855f698746f8e7f51acd1b", size = 207542, upload-time = "2026-03-12T03:36:01.668Z" },
+]
+
[[package]]
name = "boto3"
version = "1.40.61"
@@ -925,6 +978,26 @@ nvtx = [
{ name = "nvidia-nvtx", marker = "(python_full_version < '3.11' and sys_platform == 'win32') or sys_platform == 'linux'" },
]
+[[package]]
+name = "datamodel-code-generator"
+version = "0.56.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "argcomplete" },
+ { name = "black" },
+ { name = "genson" },
+ { name = "inflect" },
+ { name = "isort" },
+ { name = "jinja2" },
+ { name = "pydantic" },
+ { name = "pyyaml" },
+ { name = "tomli", marker = "python_full_version < '3.11'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6f/9c/5379be8daf9ff2fbbb9a7efcf68c037b089e5b76f360c4f9ca730916e2ee/datamodel_code_generator-0.56.1.tar.gz", hash = "sha256:697abd90cc4eb2c65f130be79a83a24746c3f2d0e15e6eb9dbf17b96784449be", size = 840372, upload-time = "2026-04-16T17:09:53.537Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f7/cd/4a4a13f55069d88d8405aa325874b04c1b1dbc830f0d66c0e56cbd16aa12/datamodel_code_generator-0.56.1-py3-none-any.whl", hash = "sha256:cb2adc18a301f1a2cfcd56672037218eeaff0573669861c2b371eefe86753874", size = 256845, upload-time = "2026-04-16T17:09:51.81Z" },
+]
+
[[package]]
name = "defusedxml"
version = "0.7.1"
@@ -952,6 +1025,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" },
]
+[[package]]
+name = "docker"
+version = "7.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pywin32", marker = "sys_platform == 'win32'" },
+ { name = "requests" },
+ { name = "urllib3" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/91/9b/4a2ea29aeba62471211598dac5d96825bb49348fa07e906ea930394a83ce/docker-7.1.0.tar.gz", hash = "sha256:ad8c70e6e3f8926cb8a92619b832b4ea5299e2831c14284663184e200546fa6c", size = 117834, upload-time = "2024-05-23T11:13:57.216Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e3/26/57c6fb270950d476074c087527a558ccb6f4436657314bfb6cdf484114c4/docker-7.1.0-py3-none-any.whl", hash = "sha256:c96b93b7f0a746f9e77d325bcfb87422a3d8bd4f03136ae8a85b37f1898d5fc0", size = 147774, upload-time = "2024-05-23T11:13:55.01Z" },
+]
+
[[package]]
name = "docling"
version = "2.84.0"
@@ -1313,6 +1400,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d5/1f/5f4a3cd9e4440e9d9bc78ad0a91a1c8d46b4d429d5239ebe6793c9fe5c41/fsspec-2026.3.0-py3-none-any.whl", hash = "sha256:d2ceafaad1b3457968ed14efa28798162f1638dbb5d2a6868a2db002a5ee39a4", size = 202595, upload-time = "2026-03-27T19:11:13.595Z" },
]
+[[package]]
+name = "genson"
+version = "1.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c5/cf/2303c8ad276dcf5ee2ad6cf69c4338fd86ef0f471a5207b069adf7a393cf/genson-1.3.0.tar.gz", hash = "sha256:e02db9ac2e3fd29e65b5286f7135762e2cd8a986537c075b06fc5f1517308e37", size = 34919, upload-time = "2024-05-15T22:08:49.123Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f8/5c/e226de133afd8bb267ec27eead9ae3d784b95b39a287ed404caab39a5f50/genson-1.3.0-py3-none-any.whl", hash = "sha256:468feccd00274cc7e4c09e84b08704270ba8d95232aa280f65b986139cec67f7", size = 21470, upload-time = "2024-05-15T22:08:47.056Z" },
+]
+
[[package]]
name = "ghp-import"
version = "2.1.0"
@@ -1600,6 +1696,19 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/fa/5e/f8e9a1d23b9c20a551a8a02ea3637b4642e22c2626e3a13a9a29cdea99eb/importlib_metadata-8.7.1-py3-none-any.whl", hash = "sha256:5a1f80bf1daa489495071efbb095d75a634cf28a8bc299581244063b53176151", size = 27865, upload-time = "2025-12-21T10:00:18.329Z" },
]
+[[package]]
+name = "inflect"
+version = "7.5.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "more-itertools" },
+ { name = "typeguard" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/78/c6/943357d44a21fd995723d07ccaddd78023eace03c1846049a2645d4324a3/inflect-7.5.0.tar.gz", hash = "sha256:faf19801c3742ed5a05a8ce388e0d8fe1a07f8d095c82201eb904f5d27ad571f", size = 73751, upload-time = "2024-12-28T17:11:18.897Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8a/eb/427ed2b20a38a4ee29f24dbe4ae2dafab198674fe9a85e3d6adf9e5f5f41/inflect-7.5.0-py3-none-any.whl", hash = "sha256:2aea70e5e70c35d8350b8097396ec155ffd68def678c7ff97f51aa69c1d92344", size = 35197, upload-time = "2024-12-28T17:11:15.931Z" },
+]
+
[[package]]
name = "iniconfig"
version = "2.3.0"
@@ -1829,15 +1938,11 @@ wheels = [
[[package]]
name = "kajson"
-version = "0.3.1"
-source = { registry = "https://pypi.org/simple" }
+version = "0.4.2"
+source = { git = "https://github.com/Pipelex/kajson.git?rev=fix%2Fenum-source-registry#4e80ebf729e22f4f3a6a4fac846a024e38fc2fd0" }
dependencies = [
{ name = "pydantic" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/b8/51/dc0529d2b678003e783cc9e79b0ca16db1ca1035935a5623deb822b33b8f/kajson-0.3.1.tar.gz", hash = "sha256:a170883f04a9c5e72a3831ce65720324b078483e0360e6f249400e1c4b8d9ffa", size = 20275, upload-time = "2025-07-11T08:26:13.243Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/9d/3b/3529c470a4cc1051abe2e4cd6a8a714b14eea26a8486bb3e1cd19a5c60d9/kajson-0.3.1-py3-none-any.whl", hash = "sha256:c129b5dd66f757ed8077c49803ea290f91ba6f5699717eb97b65d960bcc83ac4", size = 26721, upload-time = "2025-07-11T08:26:11.622Z" },
-]
[[package]]
name = "latex2mathml"
@@ -2363,6 +2468,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/c4/ec/38443b1f2a3821bbcb24e46cd8ba979154417794d54baf949fefde1c2146/mkdocs_redirects-1.2.2-py3-none-any.whl", hash = "sha256:7dbfa5647b79a3589da4401403d69494bd1f4ad03b9c15136720367e1f340ed5", size = 6142, upload-time = "2024-11-07T14:57:19.143Z" },
]
+[[package]]
+name = "more-itertools"
+version = "11.0.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/f7/139d22fef48ac78127d18e01d80cf1be40236ae489769d17f35c3d425293/more_itertools-11.0.2.tar.gz", hash = "sha256:392a9e1e362cbc106a2457d37cabf9b36e5e12efd4ebff1654630e76597df804", size = 144659, upload-time = "2026-04-09T15:01:33.297Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cb/98/6af411189d9413534c3eb691182bff1f5c6d44ed2f93f2edfe52a1bbceb8/more_itertools-11.0.2-py3-none-any.whl", hash = "sha256:6e35b35f818b01f691643c6c611bc0902f2e92b46c18fffa77ae1e7c46e912e4", size = 71939, upload-time = "2026-04-09T15:01:32.21Z" },
+]
+
[[package]]
name = "moto"
version = "5.1.22"
@@ -2384,6 +2498,10 @@ wheels = [
]
[package.optional-dependencies]
+dynamodb = [
+ { name = "docker" },
+ { name = "py-partiql-parser" },
+]
s3 = [
{ name = "py-partiql-parser" },
{ name = "pyyaml" },
@@ -2747,6 +2865,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" },
]
+[[package]]
+name = "nexus-rpc"
+version = "1.3.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2e/f2/d54f5c03d8f4672ccc0875787a385f53dcb61f98a8ae594b5620e85b9cb3/nexus_rpc-1.3.0.tar.gz", hash = "sha256:e56d3b57b60d707ce7a72f83f23f106b86eca1043aa658e44582ab5ff30ab9ad", size = 75650, upload-time = "2025-12-08T22:59:13.002Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d6/74/0afd841de3199c148146c1d43b4bfb5605b2f1dc4c9a9087fe395091ea5a/nexus_rpc-1.3.0-py3-none-any.whl", hash = "sha256:aee0707b4861b22d8124ecb3f27d62dafbe8777dc50c66c91e49c006f971b92d", size = 28873, upload-time = "2025-12-08T22:59:12.024Z" },
+]
+
[[package]]
name = "nodeenv"
version = "1.10.0"
@@ -3501,6 +3631,7 @@ source = { editable = "." }
dependencies = [
{ name = "aiofiles" },
{ name = "backports-strenum", marker = "python_full_version < '3.11'" },
+ { name = "datamodel-code-generator" },
{ name = "filetype" },
{ name = "httpx" },
{ name = "instructor" },
@@ -3545,7 +3676,7 @@ bedrock = [
dev = [
{ name = "boto3-stubs" },
{ name = "cryptography" },
- { name = "moto", extra = ["s3"] },
+ { name = "moto", extra = ["dynamodb", "s3"] },
{ name = "mypy" },
{ name = "pipelex-tools" },
{ name = "pylint" },
@@ -3555,6 +3686,7 @@ dev = [
{ name = "pytest-cov" },
{ name = "pytest-mock" },
{ name = "pytest-sugar" },
+ { name = "pytest-timeout" },
{ name = "pytest-xdist" },
{ name = "ruff" },
{ name = "types-aioboto3", extra = ["bedrock", "bedrock-runtime"] },
@@ -3576,6 +3708,9 @@ docs = [
{ name = "mkdocs-meta-manager" },
{ name = "mkdocs-redirects" },
]
+dynamodb = [
+ { name = "boto3" },
+]
fal = [
{ name = "fal-client" },
]
@@ -3605,21 +3740,28 @@ s3 = [
{ name = "aioboto3" },
{ name = "boto3" },
]
+temporal = [
+ { name = "aiohttp" },
+ { name = "temporalio" },
+]
[package.metadata]
requires-dist = [
{ name = "aioboto3", marker = "extra == 'bedrock'", specifier = ">=13.4.0" },
{ name = "aioboto3", marker = "extra == 's3'", specifier = ">=13.4.0" },
{ name = "aiofiles", specifier = ">=23.2.1" },
+ { name = "aiohttp", marker = "extra == 'temporal'", specifier = ">=3.9.0" },
{ name = "anthropic", marker = "extra == 'anthropic'", specifier = ">=0.78.0" },
{ name = "backports-strenum", marker = "python_full_version < '3.11'", specifier = ">=1.3.0" },
{ name = "boto3", marker = "extra == 'bedrock'", specifier = ">=1.34.131" },
+ { name = "boto3", marker = "extra == 'dynamodb'", specifier = ">=1.34.131" },
{ name = "boto3", marker = "extra == 's3'", specifier = ">=1.34.131" },
{ name = "boto3-stubs", marker = "extra == 'dev'", specifier = ">=1.35.24" },
{ name = "cryptography", marker = "extra == 'dev'", specifier = ">=46.0.7" },
{ name = "cryptography", marker = "extra == 'gcp-storage'", specifier = ">=46.0.7" },
{ name = "cryptography", marker = "extra == 'google'", specifier = ">=46.0.7" },
{ name = "cryptography", marker = "extra == 'google-genai'", specifier = ">=46.0.7" },
+ { name = "datamodel-code-generator", specifier = ">=0.55.0" },
{ name = "docling", marker = "extra == 'docling'", specifier = ">=2.64.0" },
{ name = "fal-client", marker = "extra == 'fal'", specifier = ">=0.4.1" },
{ name = "filetype", specifier = ">=1.2.0" },
@@ -3632,7 +3774,7 @@ requires-dist = [
{ name = "instructor", extras = ["google-genai"], marker = "extra == 'google-genai'" },
{ name = "jinja2", specifier = ">=3.1.4" },
{ name = "json2html", specifier = ">=1.3.0" },
- { name = "kajson", specifier = "==0.3.1" },
+ { name = "kajson", git = "https://github.com/Pipelex/kajson.git?rev=fix%2Fenum-source-registry" },
{ name = "linkup-sdk", marker = "extra == 'linkup'", specifier = ">=0.12.0" },
{ name = "lxml", marker = "extra == 'docling'", specifier = ">=6.1.0" },
{ name = "markdown", specifier = ">=3.6" },
@@ -3644,7 +3786,7 @@ requires-dist = [
{ name = "mkdocs-material", marker = "extra == 'docs'", specifier = ">=9.6.14" },
{ name = "mkdocs-meta-manager", marker = "extra == 'docs'", specifier = ">=1.1.0" },
{ name = "mkdocs-redirects", marker = "extra == 'docs'", specifier = "==1.2.2" },
- { name = "moto", extras = ["s3"], marker = "extra == 'dev'", specifier = ">=5.0.0" },
+ { name = "moto", extras = ["dynamodb", "s3"], marker = "extra == 'dev'", specifier = ">=5.0.0" },
{ name = "mthds", specifier = ">=0.3.0" },
{ name = "mypy", marker = "extra == 'dev'", specifier = "==1.19.1" },
{ name = "networkx", specifier = ">=3.4.2" },
@@ -3667,6 +3809,7 @@ requires-dist = [
{ name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=6.1.1" },
{ name = "pytest-mock", marker = "extra == 'dev'", specifier = ">=3.14.0" },
{ name = "pytest-sugar", marker = "extra == 'dev'", specifier = ">=1.0.0" },
+ { name = "pytest-timeout", marker = "extra == 'dev'", specifier = ">=2.3.1" },
{ name = "pytest-xdist", marker = "extra == 'dev'", specifier = ">=3.6.1" },
{ name = "python-dotenv", specifier = ">=1.0.1" },
{ name = "pyyaml", specifier = ">=6.0.2" },
@@ -3675,6 +3818,7 @@ requires-dist = [
{ name = "ruff", marker = "extra == 'dev'", specifier = "==0.14.13" },
{ name = "semantic-version", specifier = ">=2.10.0" },
{ name = "shortuuid", specifier = ">=1.0.13" },
+ { name = "temporalio", marker = "extra == 'temporal'", specifier = "==1.23.0" },
{ name = "tomli", specifier = ">=2.3.0" },
{ name = "tomlkit", specifier = ">=0.13.2" },
{ name = "typer", specifier = ">=0.16.0" },
@@ -3685,7 +3829,7 @@ requires-dist = [
{ name = "types-pyyaml", marker = "extra == 'dev'", specifier = ">=6.0.12.20250326" },
{ name = "typing-extensions", specifier = ">=4.13.2" },
]
-provides-extras = ["anthropic", "bedrock", "docling", "fal", "gcp-storage", "google", "google-genai", "huggingface", "linkup", "mistralai", "s3", "docs", "dev"]
+provides-extras = ["anthropic", "bedrock", "docling", "fal", "gcp-storage", "google", "google-genai", "huggingface", "linkup", "mistralai", "dynamodb", "s3", "temporal", "docs", "dev"]
[[package]]
name = "pipelex-tools"
@@ -4421,6 +4565,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/87/d5/81d38a91c1fdafb6711f053f5a9b92ff788013b19821257c2c38c1e132df/pytest_sugar-1.1.1-py3-none-any.whl", hash = "sha256:2f8319b907548d5b9d03a171515c1d43d2e38e32bd8182a1781eb20b43344cc8", size = 11440, upload-time = "2025-08-23T12:19:34.894Z" },
]
+[[package]]
+name = "pytest-timeout"
+version = "2.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pytest" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ac/82/4c9ecabab13363e72d880f2fb504c5f750433b2b6f16e99f4ec21ada284c/pytest_timeout-2.4.0.tar.gz", hash = "sha256:7e68e90b01f9eff71332b25001f85c75495fc4e3a836701876183c4bcfd0540a", size = 17973, upload-time = "2025-05-05T19:44:34.99Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fa/b6/3127540ecdf1464a00e5a01ee60a1b09175f6913f0644ac748494d9c4b21/pytest_timeout-2.4.0-py3-none-any.whl", hash = "sha256:c42667e5cdadb151aeb5b26d114aff6bdf5a907f176a007a30b940d3d865b5c2", size = 14382, upload-time = "2025-05-05T19:44:33.502Z" },
+]
+
[[package]]
name = "pytest-xdist"
version = "3.8.0"
@@ -4483,6 +4639,45 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d9/4f/00be2196329ebbff56ce564aa94efb0fbc828d00de250b1980de1a34ab49/python_pptx-1.0.2-py3-none-any.whl", hash = "sha256:160838e0b8565a8b1f67947675886e9fea18aa5e795db7ae531606d68e785cba", size = 472788, upload-time = "2024-08-07T17:33:28.192Z" },
]
+[[package]]
+name = "pytokens"
+version = "0.4.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b6/34/b4e015b99031667a7b960f888889c5bd34ef585c85e1cb56a594b92836ac/pytokens-0.4.1.tar.gz", hash = "sha256:292052fe80923aae2260c073f822ceba21f3872ced9a68bb7953b348e561179a", size = 23015, upload-time = "2026-01-30T01:03:45.924Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/42/24/f206113e05cb8ef51b3850e7ef88f20da6f4bf932190ceb48bd3da103e10/pytokens-0.4.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2a44ed93ea23415c54f3face3b65ef2b844d96aeb3455b8a69b3df6beab6acc5", size = 161522, upload-time = "2026-01-30T01:02:50.393Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/e9/06a6bf1b90c2ed81a9c7d2544232fe5d2891d1cd480e8a1809ca354a8eb2/pytokens-0.4.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:add8bf86b71a5d9fb5b89f023a80b791e04fba57960aa790cc6125f7f1d39dfe", size = 246945, upload-time = "2026-01-30T01:02:52.399Z" },
+ { url = "https://files.pythonhosted.org/packages/69/66/f6fb1007a4c3d8b682d5d65b7c1fb33257587a5f782647091e3408abe0b8/pytokens-0.4.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:670d286910b531c7b7e3c0b453fd8156f250adb140146d234a82219459b9640c", size = 259525, upload-time = "2026-01-30T01:02:53.737Z" },
+ { url = "https://files.pythonhosted.org/packages/04/92/086f89b4d622a18418bac74ab5db7f68cf0c21cf7cc92de6c7b919d76c88/pytokens-0.4.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:4e691d7f5186bd2842c14813f79f8884bb03f5995f0575272009982c5ac6c0f7", size = 262693, upload-time = "2026-01-30T01:02:54.871Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/7b/8b31c347cf94a3f900bdde750b2e9131575a61fdb620d3d3c75832262137/pytokens-0.4.1-cp310-cp310-win_amd64.whl", hash = "sha256:27b83ad28825978742beef057bfe406ad6ed524b2d28c252c5de7b4a6dd48fa2", size = 103567, upload-time = "2026-01-30T01:02:56.414Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/92/790ebe03f07b57e53b10884c329b9a1a308648fc083a6d4a39a10a28c8fc/pytokens-0.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d70e77c55ae8380c91c0c18dea05951482e263982911fc7410b1ffd1dadd3440", size = 160864, upload-time = "2026-01-30T01:02:57.882Z" },
+ { url = "https://files.pythonhosted.org/packages/13/25/a4f555281d975bfdd1eba731450e2fe3a95870274da73fb12c40aeae7625/pytokens-0.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a58d057208cb9075c144950d789511220b07636dd2e4708d5645d24de666bdc", size = 248565, upload-time = "2026-01-30T01:02:59.912Z" },
+ { url = "https://files.pythonhosted.org/packages/17/50/bc0394b4ad5b1601be22fa43652173d47e4c9efbf0044c62e9a59b747c56/pytokens-0.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b49750419d300e2b5a3813cf229d4e5a4c728dae470bcc89867a9ad6f25a722d", size = 260824, upload-time = "2026-01-30T01:03:01.471Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/54/3e04f9d92a4be4fc6c80016bc396b923d2a6933ae94b5f557c939c460ee0/pytokens-0.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:d9907d61f15bf7261d7e775bd5d7ee4d2930e04424bab1972591918497623a16", size = 264075, upload-time = "2026-01-30T01:03:04.143Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/1b/44b0326cb5470a4375f37988aea5d61b5cc52407143303015ebee94abfd6/pytokens-0.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:ee44d0f85b803321710f9239f335aafe16553b39106384cef8e6de40cb4ef2f6", size = 103323, upload-time = "2026-01-30T01:03:05.412Z" },
+ { url = "https://files.pythonhosted.org/packages/41/5d/e44573011401fb82e9d51e97f1290ceb377800fb4eed650b96f4753b499c/pytokens-0.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:140709331e846b728475786df8aeb27d24f48cbcf7bcd449f8de75cae7a45083", size = 160663, upload-time = "2026-01-30T01:03:06.473Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/e6/5bbc3019f8e6f21d09c41f8b8654536117e5e211a85d89212d59cbdab381/pytokens-0.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6d6c4268598f762bc8e91f5dbf2ab2f61f7b95bdc07953b602db879b3c8c18e1", size = 255626, upload-time = "2026-01-30T01:03:08.177Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/3c/2d5297d82286f6f3d92770289fd439956b201c0a4fc7e72efb9b2293758e/pytokens-0.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:24afde1f53d95348b5a0eb19488661147285ca4dd7ed752bbc3e1c6242a304d1", size = 269779, upload-time = "2026-01-30T01:03:09.756Z" },
+ { url = "https://files.pythonhosted.org/packages/20/01/7436e9ad693cebda0551203e0bf28f7669976c60ad07d6402098208476de/pytokens-0.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5ad948d085ed6c16413eb5fec6b3e02fa00dc29a2534f088d3302c47eb59adf9", size = 268076, upload-time = "2026-01-30T01:03:10.957Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/df/533c82a3c752ba13ae7ef238b7f8cdd272cf1475f03c63ac6cf3fcfb00b6/pytokens-0.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:3f901fe783e06e48e8cbdc82d631fca8f118333798193e026a50ce1b3757ea68", size = 103552, upload-time = "2026-01-30T01:03:12.066Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/dc/08b1a080372afda3cceb4f3c0a7ba2bde9d6a5241f1edb02a22a019ee147/pytokens-0.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8bdb9d0ce90cbf99c525e75a2fa415144fd570a1ba987380190e8b786bc6ef9b", size = 160720, upload-time = "2026-01-30T01:03:13.843Z" },
+ { url = "https://files.pythonhosted.org/packages/64/0c/41ea22205da480837a700e395507e6a24425151dfb7ead73343d6e2d7ffe/pytokens-0.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5502408cab1cb18e128570f8d598981c68a50d0cbd7c61312a90507cd3a1276f", size = 254204, upload-time = "2026-01-30T01:03:14.886Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/d2/afe5c7f8607018beb99971489dbb846508f1b8f351fcefc225fcf4b2adc0/pytokens-0.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:29d1d8fb1030af4d231789959f21821ab6325e463f0503a61d204343c9b355d1", size = 268423, upload-time = "2026-01-30T01:03:15.936Z" },
+ { url = "https://files.pythonhosted.org/packages/68/d4/00ffdbd370410c04e9591da9220a68dc1693ef7499173eb3e30d06e05ed1/pytokens-0.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:970b08dd6b86058b6dc07efe9e98414f5102974716232d10f32ff39701e841c4", size = 266859, upload-time = "2026-01-30T01:03:17.458Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/c9/c3161313b4ca0c601eeefabd3d3b576edaa9afdefd32da97210700e47652/pytokens-0.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:9bd7d7f544d362576be74f9d5901a22f317efc20046efe2034dced238cbbfe78", size = 103520, upload-time = "2026-01-30T01:03:18.652Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/a7/b470f672e6fc5fee0a01d9e75005a0e617e162381974213a945fcd274843/pytokens-0.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:4a14d5f5fc78ce85e426aa159489e2d5961acf0e47575e08f35584009178e321", size = 160821, upload-time = "2026-01-30T01:03:19.684Z" },
+ { url = "https://files.pythonhosted.org/packages/80/98/e83a36fe8d170c911f864bfded690d2542bfcfacb9c649d11a9e6eb9dc41/pytokens-0.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97f50fd18543be72da51dd505e2ed20d2228c74e0464e4262e4899797803d7fa", size = 254263, upload-time = "2026-01-30T01:03:20.834Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/95/70d7041273890f9f97a24234c00b746e8da86df462620194cef1d411ddeb/pytokens-0.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dc74c035f9bfca0255c1af77ddd2d6ae8419012805453e4b0e7513e17904545d", size = 268071, upload-time = "2026-01-30T01:03:21.888Z" },
+ { url = "https://files.pythonhosted.org/packages/da/79/76e6d09ae19c99404656d7db9c35dfd20f2086f3eb6ecb496b5b31163bad/pytokens-0.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f66a6bbe741bd431f6d741e617e0f39ec7257ca1f89089593479347cc4d13324", size = 271716, upload-time = "2026-01-30T01:03:23.633Z" },
+ { url = "https://files.pythonhosted.org/packages/79/37/482e55fa1602e0a7ff012661d8c946bafdc05e480ea5a32f4f7e336d4aa9/pytokens-0.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:b35d7e5ad269804f6697727702da3c517bb8a5228afa450ab0fa787732055fc9", size = 104539, upload-time = "2026-01-30T01:03:24.788Z" },
+ { url = "https://files.pythonhosted.org/packages/30/e8/20e7db907c23f3d63b0be3b8a4fd1927f6da2395f5bcc7f72242bb963dfe/pytokens-0.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:8fcb9ba3709ff77e77f1c7022ff11d13553f3c30299a9fe246a166903e9091eb", size = 168474, upload-time = "2026-01-30T01:03:26.428Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/81/88a95ee9fafdd8f5f3452107748fd04c24930d500b9aba9738f3ade642cc/pytokens-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:79fc6b8699564e1f9b521582c35435f1bd32dd06822322ec44afdeba666d8cb3", size = 290473, upload-time = "2026-01-30T01:03:27.415Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/35/3aa899645e29b6375b4aed9f8d21df219e7c958c4c186b465e42ee0a06bf/pytokens-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d31b97b3de0f61571a124a00ffe9a81fb9939146c122c11060725bd5aea79975", size = 303485, upload-time = "2026-01-30T01:03:28.558Z" },
+ { url = "https://files.pythonhosted.org/packages/52/a0/07907b6ff512674d9b201859f7d212298c44933633c946703a20c25e9d81/pytokens-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:967cf6e3fd4adf7de8fc73cd3043754ae79c36475c1c11d514fc72cf5490094a", size = 306698, upload-time = "2026-01-30T01:03:29.653Z" },
+ { url = "https://files.pythonhosted.org/packages/39/2a/cbbf9250020a4a8dd53ba83a46c097b69e5eb49dd14e708f496f548c6612/pytokens-0.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:584c80c24b078eec1e227079d56dc22ff755e0ba8654d8383b2c549107528918", size = 116287, upload-time = "2026-01-30T01:03:30.912Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/78/397db326746f0a342855b81216ae1f0a32965deccfd7c830a2dbc66d2483/pytokens-0.4.1-py3-none-any.whl", hash = "sha256:26cef14744a8385f35d0e095dc8b3a7583f6c953c2e3d269c7f82484bf5ad2de", size = 13729, upload-time = "2026-01-30T01:03:45.029Z" },
+]
+
[[package]]
name = "pytz"
version = "2026.1.post1"
@@ -5396,6 +5591,26 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl", hash = "sha256:f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3", size = 39814, upload-time = "2026-03-04T18:55:31.284Z" },
]
+[[package]]
+name = "temporalio"
+version = "1.23.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nexus-rpc" },
+ { name = "protobuf" },
+ { name = "python-dateutil", marker = "python_full_version < '3.11'" },
+ { name = "types-protobuf" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/67/48/ba7413e2fab8dcd277b9df00bafa572da24e9ca32de2f38d428dc3a2825c/temporalio-1.23.0.tar.gz", hash = "sha256:72750494b00eb73ded9db76195e3a9b53ff548780f73d878ec3f807ee3191410", size = 1933051, upload-time = "2026-02-18T17:48:22.353Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6f/71/26c8f21dca9092201b3b9cb7aff42460b4864b5999aa4c6a4343ac66f1fd/temporalio-1.23.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:6b69ac8d75f2d90e66f4edce4316f6a33badc4a30b22efc50e9eddaa9acdc216", size = 12311037, upload-time = "2026-02-18T17:47:47.628Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/47/43102816139f2d346680cb7cc1e53da5f6968355ac65b4d35d4edbfca896/temporalio-1.23.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:1bbbb2f9c3cdd09451565163f6d741e51f109694c49435d475fdfa42b597219d", size = 11821906, upload-time = "2026-02-18T17:47:55.314Z" },
+ { url = "https://files.pythonhosted.org/packages/00/b0/899ff28464a0e17adf17476bdfac8faf4ea41870358ff2d14737e43f9e66/temporalio-1.23.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cf6570e0ee696f99a38d855da4441a890c7187357c16505ed458ac9ef274ed70", size = 12063601, upload-time = "2026-02-18T17:48:03.994Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/17/b8c6d2ec3e113c6a788322513a5ff635bdd54b3791d092ed0e273467748a/temporalio-1.23.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b82d6cca54c9f376b50e941dd10d12f7fe5b692a314fb087be72cd2898646a79", size = 12394579, upload-time = "2026-02-18T17:48:11.65Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/b7/f9ef7fd5ee65aef7d59ab1e95cb1b45df2fe49c17e3aa4d650ae3322f015/temporalio-1.23.0-cp310-abi3-win_amd64.whl", hash = "sha256:43c3b99a46dd329761a256f3855710c4a5b322afc879785e468bdd0b94faace6", size = 12834494, upload-time = "2026-02-18T17:48:19.071Z" },
+]
+
[[package]]
name = "tenacity"
version = "9.1.4"
@@ -5760,6 +5975,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/f6/56/6113c23ff46c00aae423333eb58b3e60bdfe9179d542781955a5e1514cb3/triton-3.6.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:46bd1c1af4b6704e554cad2eeb3b0a6513a980d470ccfa63189737340c7746a7", size = 188397994, upload-time = "2026-01-20T16:01:14.236Z" },
]
+[[package]]
+name = "typeguard"
+version = "4.5.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2b/e8/66e25efcc18542d58706ce4e50415710593721aae26e794ab1dec34fb66f/typeguard-4.5.1.tar.gz", hash = "sha256:f6f8ecbbc819c9bc749983cc67c02391e16a9b43b8b27f15dc70ed7c4a007274", size = 80121, upload-time = "2026-02-19T16:09:03.392Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/91/88/b55b3117287a8540b76dbdd87733808d4d01c8067a3b339408c250bb3600/typeguard-4.5.1-py3-none-any.whl", hash = "sha256:44d2bf329d49a244110a090b55f5f91aa82d9a9834ebfd30bcc73651e4a8cc40", size = 36745, upload-time = "2026-02-19T16:09:01.6Z" },
+]
+
[[package]]
name = "typer"
version = "0.21.2"
@@ -5875,6 +6102,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/9b/60/57fe0ebec42315a70f34920c325a803f4c069f33025043f45ac4a4266faf/types_networkx-3.6.1.20260402-py3-none-any.whl", hash = "sha256:2ea0075288632bf777e877b3c2a0d1d6991844a08a0b4a72d23c513436d64038", size = 162536, upload-time = "2026-04-02T04:20:06.375Z" },
]
+[[package]]
+name = "types-protobuf"
+version = "7.34.1.20260503"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a0/31/87969cb3e62287bde7598b78b3c098d2873d54f5fb5a7cfbcaa73b8c965e/types_protobuf-7.34.1.20260503.tar.gz", hash = "sha256:effbc819aa17e02448dde99f089c6794662d66f4b2797e922f185ffe0b24e766", size = 68830, upload-time = "2026-05-03T05:19:50.739Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f9/67/a33fb18090a927794a5ee4b1a30730b528ace0dad6b18932540d21258184/types_protobuf-7.34.1.20260503-py3-none-any.whl", hash = "sha256:75fd66121d56785c91828b8bf7b511f39ba847f11e682573e41847f01e9cd1de", size = 86019, upload-time = "2026-05-03T05:19:49.486Z" },
+]
+
[[package]]
name = "types-pyyaml"
version = "6.0.12.20250915"
diff --git a/wip/02-master-plan.md b/wip/02-master-plan.md
new file mode 100644
index 000000000..583337bd9
--- /dev/null
+++ b/wip/02-master-plan.md
@@ -0,0 +1,107 @@
+# Master Plan v3 — Distributed Workers
+
+> **Status**: Live plan — drives the next round of distributed-execution work.
+> **Date**: 2026-05-04
+> **Predecessors**: [archive/00-master-plan.md](archive/00-master-plan.md) (Phases 0–5, shipped) · [archive/01-master-plan.md](archive/01-master-plan.md) (interim plan, now split into per-topic files).
+
+The big direction now is to actually run **activities on standalone Worker pools** — separate processes from the workflow Worker pool — so we can scale workflow orchestration and inference activity execution independently. Today's tracing / cost-reporting design was built for the single-bundle-per-Worker case and breaks the moment activities move off-process. That has to be fixed before anything else.
+
+---
+
+## Priority order
+
+| # | Item | Status | Owner file |
+|---|---|---|---|
+| **P0** | **Tracing & cost reporting across separate-process workers** | Not started — top priority | this file (problem statement) |
+| **P1** | **Cross-worker cost report assembly wiring** | Not started — depends on P0 design | this file (problem statement) |
+| **P2** | Phase 6a — Local cross-package dependencies in crate | Not started | [phase6a-local-cross-package-deps.md](phase6a-local-cross-package-deps.md) |
+| **P3** | Phase 6b — Remote dependencies from GitHub | Not started — needs P2 | [phase6b-remote-deps-from-github.md](phase6b-remote-deps-from-github.md) |
+
+What's already shipped on this front: see [tracing-cost-reporting-as-built.md](tracing-cost-reporting-as-built.md). It also enumerates the open issues (T1, T2, T3) that motivate P0 and P1.
+
+Sibling tracks (separate branches, not in this plan): worker error-handling Phases 4–7 (`error-handling-phase-{4,5,6,7}-*.md`), instructor-unwrap port (`instructor-unwrap-other-workers.md`).
+
+---
+
+## P0 — Tracing & cost reporting across separate-process workers
+
+### Why this is top priority
+
+The whole point of the distributed-execution work is to allow **activities on standalone Worker pools** (separate processes from the workflow Worker pool). The current tracing implementation cannot serve that topology — it relies on the activity sharing the workflow's process so the singleton `ReportingManager` already has `set_event_log` configured. Until P0 is solved, splitting activities off is a regression on observability and cost reporting, not a feature.
+
+### Problems with the current implementation
+
+(All concretely enumerated in [tracing-cost-reporting-as-built.md](tracing-cost-reporting-as-built.md). Summarized here so the priority is legible.)
+
+1. **Standalone activity workers lose usage events.** When an activity runs on a process that never executed `WfPipeRouter.run()`, that process's `ReportingManager._event_log_contexts` is empty for the workflow's `lookup_key`. `_emit_usage_event` returns silently; usage data is dropped. The `_get_registry` TODO at `reporting_manager.py:111-117` ("Auto-create registry for unknown pipeline IDs ... TODO: replace with proper distributed reporting system") confirms this is a known gap.
+2. **`BufferingEventLog` + `act_flush_trace_events` is workflow-scoped.** Both pieces are tied to the workflow lifecycle (buffer in workflow context, flush after pipe execution). They cannot serve a standalone activity Worker that has no enclosing workflow process.
+3. **`set_event_log` lives on a process-singleton `ReportingManager`.** The per-context `_event_log_contexts` dict pattern relies on the activity emitting from the same process that configured the event log. The moment activity execution moves off-process, the activity has no way to discover its event log without something flowing in via request data (`JobMetadata` / a new `TracingContext`) instead of via a process-local dict.
+
+### Design discussion (deferred)
+
+We're not designing the fix yet — that comes when we pick up P0. The two natural starting points are (a) the original Phase 4.5 Step 6 `TracingActivityInboundInterceptor` design (now in `archive/00-master-plan.md` and `archive/01-master-plan.md`), and (b) plumbing tracing config + workflow id through `JobMetadata` so each activity can construct its own event log directly. Either way the activity needs request-scoped tracing data, not process-scoped state.
+
+### Acceptance criteria (sketch — refine when planning)
+
+- [ ] Activities deployed on standalone Worker pools (separate process from workflow Workers) emit `UsageReportEvent`s that land in the same backend partition as the rest of the run.
+- [ ] No reliance on `WfPipeRouter` having executed in the activity's process.
+- [ ] No silent drops — if tracing is enabled, an activity that fails to emit raises or logs explicitly.
+- [ ] Direct mode and the current single-bundle Worker mode keep working unchanged.
+- [ ] Tests covering: (1) standalone activity Worker, (2) mixed worker pool, (3) tracing disabled, (4) backend = NDJSON, (5) backend = DynamoDB.
+
+---
+
+## P1 — Cross-worker cost report assembly wiring
+
+### Why this is second, not first
+
+Even if P0 ships, the events still need to be turned into a cost report. Today the read-back path exists for the **graph** (`assemble_graph_on_output` / `act_assemble_graph` → `GraphSpecAssembler`) but not for **usage**. The pieces all exist on the manager side:
+
+- `UsageAggregator.aggregate(events) → list[AnyTokensUsage]` (`pipelex/tracing/usage_aggregator.py`).
+- `ReportingManager.inject_tokens_usages(pipeline_run_id, tokens_usages)` (`reporting_manager.py:191`) — docstring: *"Used after assembling usage data from distributed trace events, so that generate_report() can produce a complete cost report across all workers."*
+- `ReportingManager.generate_report(pipeline_run_id)` (`reporting_manager.py:247`).
+
+Nothing in the runtime calls these. `generate_report()` has zero runtime callers; only integration tests invoke it. Captured events sit in the backend and never become a cost report.
+
+This is a small piece of plumbing once P0's design is settled — the same place that calls the graph assembler can call the usage aggregator and inject the result into the report delegate before `generate_report` runs.
+
+### Acceptance criteria (sketch)
+
+- [ ] Runtime path: read events for `pipeline_run_id` → `UsageAggregator.aggregate()` → `ReportingManager.inject_tokens_usages()` → `generate_report()`.
+- [ ] Wired in both direct mode (in `pipelex/pipeline/runner.py`) and Temporal mode (alongside or inside the existing graph-assembly hook).
+- [ ] Cross-worker case verified: parent workflow on Worker A, child workflow on Worker B, activity on Worker C — single end-of-run cost report contains usage from all three.
+- [ ] Replace / remove the `_get_registry` TODO at `reporting_manager.py:111-117` since the proper distributed path now exists.
+
+---
+
+## P2 — Phase 6a: Local cross-package dependencies in crate
+
+Detailed plan: [phase6a-local-cross-package-deps.md](phase6a-local-cross-package-deps.md).
+
+In one line: ship dependency blueprints inside the `LibraryCrate` so workers don't need the dependency packages on PIPELEXPATH. Prerequisite for P3.
+
+---
+
+## P3 — Phase 6b: Remote dependencies from GitHub
+
+Detailed plan: [phase6b-remote-deps-from-github.md](phase6b-remote-deps-from-github.md).
+
+In one line: extend the blueprint collector from P2 with a remote (GitHub) resolver so the crate can carry deps fetched from external addresses, making workers fully stateless.
+
+---
+
+## Dependencies between items
+
+```
+P0 (Tracing across separate workers)
+ │
+ ▼
+P1 (Cross-worker cost report assembly)
+
+P2 (Phase 6a — local cross-package deps)
+ │
+ ▼
+P3 (Phase 6b — remote deps from GitHub)
+```
+
+P0/P1 and P2/P3 are independent tracks; P0 blocks P1, P2 blocks P3.
diff --git a/wip/archive/00-master-plan.md b/wip/archive/00-master-plan.md
new file mode 100644
index 000000000..1d9c706f5
--- /dev/null
+++ b/wip/archive/00-master-plan.md
@@ -0,0 +1,409 @@
+# Master Plan v2: LibraryCrate & Distributed Execution — SHIPPED
+
+> **Status**: Archived (2026-05-04). All phases complete and merged.
+> **Date**: 2026-03-23 (originally drafted)
+> **Successor**: [../01-master-plan.md](../01-master-plan.md) (Phase 4.5 Step 6 + Phase 6 — next phases)
+> **Related**: [early-library-as-execution-context.md](early-library-as-execution-context.md), [phase0-pipe-namespace-fix.md](phase0-pipe-namespace-fix.md), [phase4-explicit-class-registry.md](phase4-explicit-class-registry.md), [phase5-payload-codec-strategy.md](phase5-payload-codec-strategy.md), [phase2-crate-propagation-rationale.md](phase2-crate-propagation-rationale.md)
+
+---
+
+## Goal
+
+Introduce the LibraryCrate as the universal intermediate representation for library loading, then leverage it to enable distributed execution (Temporal) with full library context. Get there incrementally with fast, testable results at each phase.
+
+---
+
+## Design Principles
+
+1. **Three-stage pipeline.** `Bundles (files) → LibraryCrate (data) → Library (live)`. The crate is the clean boundary between sourcing and execution.
+
+2. **Flat and ref-keyed.** The crate is two flat dicts: concepts keyed by `concept_ref`, pipes keyed by `pipe_ref`. Domain is a namespace prefix in the key, not a structural container. No source paths, no package provenance.
+
+3. **Same API for direct and distributed execution.** The LibraryCrate is not Temporal-specific. Direct mode builds and loads it in-process. Distributed mode serializes and ships it. The crate model is identical.
+
+4. **Transparency in dashboards.** Everything in PipeJob remains visible as structured JSON in Temporal dashboards. No opaque bytes.
+
+5. **TDD where it makes sense.** Each phase defines clear "done" criteria with specific tests. Write tests first for the new models and interfaces.
+
+---
+
+## Architecture Summary
+
+| Phase | Component | What it solves |
+|-------|-----------|----------------|
+| **0** | Pipe namespace fix (`pipe_ref`) | Prerequisite: pipes indexed by domain-qualified ref, symmetric with concepts |
+| **1** | LibraryCrate (direct mode) | Universal intermediate between bundles and live library |
+| **2** | LibraryCrate on Temporal | Library loading on workers (Layer 2: pipe resolution) |
+| **3** | Deferred WorkingMemory hydration | Dynamic class registration timing (Layer 1: Kajson deser) |
+| **4** | Explicit ClassRegistry | Library-owned registries, no singleton scoping in Kajson |
+| **4.5** | Distributed Tracing & Reporting | Cross-worker graph tracing and cost reporting via NDJSON event log |
+| **5** | StoragePayloadCodec | Payload size limits (2MB) for large libraries and WorkingMemory |
+| **6** | Cross-Package Dependencies in Crate | Dependency content included in crate; remote fetch from GitHub |
+
+---
+
+## Phase 0: Pipe Namespace Fix (`pipe_ref`)
+
+> **Status**: Complete — PR #780 (628e414d)
+
+**Goal**: Pipes are indexed by domain-qualified `pipe_ref` (`domain.pipe_code`), symmetric with how concepts use `concept_ref` (`domain.ConceptCode`). Multiple bundles can contribute to the same domain.
+
+**Why first**: The LibraryCrate organizes content by domain. If pipes can collide across domains, the crate can't reliably merge content. This fix is also valuable independently — it eliminates a latent bug.
+
+See [phase0-pipe-namespace-fix.md](phase0-pipe-namespace-fix.md) for the full technical spec.
+
+### Changes
+
+- Add `pipe_ref` property to `PipeAbstract` = `f"{domain_code}.{code}"`
+- Rekey `PipeLibrary` root dict from bare `code` to `pipe_ref`
+- Update `add_new_pipe()`, `get_optional_pipe()`, `get_required_pipe()` to use `pipe_ref` as primary key
+- Update `library_manager.load_from_blueprints()` pipe tracking and source mapping
+- Fix `DomainLibrary.add_domain()` to support additive merging (or make idempotent for same-domain re-adds)
+- Update all call sites that assume bare-code pipe indexing
+
+### Key files
+
+| File | Change |
+|------|--------|
+| `pipelex/core/pipes/pipe_abstract.py` | Add `pipe_ref` property |
+| `pipelex/libraries/pipe/pipe_library.py` | Rekey root dict by `pipe_ref`, update all lookup methods |
+| `pipelex/libraries/pipe/pipe_library_abstract.py` | Update interface signatures |
+| `pipelex/libraries/library_manager.py` | Update `pipe_source_in_this_load`, source map tracking |
+| `pipelex/libraries/domain/domain_library.py` | Support additive domain merging |
+| `pipelex/hub.py` | Update global pipe lookup helpers |
+| `pipelex/pipe_controllers/sub_pipe.py` | Update pipe resolution |
+| Multiple call sites | Audit all `get_required_pipe(pipe_code=...)` calls |
+
+### Done when
+
+- [x] All existing tests pass with `pipe_ref`-based indexing
+- [x] New unit test: two pipes with same code in different domains coexist in one `PipeLibrary`
+- [x] New unit test: domain-qualified lookup returns correct pipe when same code exists in multiple domains
+- [x] New unit test: bare-code lookup still works when unambiguous (single domain)
+- [x] New unit test: bare-code lookup raises when ambiguous (same code in multiple domains)
+- [x] New unit test: multiple bundles for the same domain can be loaded (domain merging)
+- [x] `make agent-check` passes
+- [x] `make agent-test` passes
+
+---
+
+## Phase 1: LibraryCrate (Direct Mode)
+
+> **Status**: Complete
+
+**Goal**: All library loading goes through `LibraryCrate` as an intermediate. The three-stage pipeline (`bundles → crate → library`) is the only path. Works in direct execution mode.
+
+### LibraryCrate model
+
+```python
+class LibraryCrate(BaseModel):
+ """Complete library content, ready to load into a live Library."""
+ concepts: dict[str, ConceptBlueprint | str] # concept_ref -> blueprint or description
+ pipes: dict[str, PipeBlueprintUnion] # pipe_ref -> blueprint
+ domains: dict[str, DomainBlueprint] # domain_code -> domain metadata
+ source_map: dict[str, str] # ref -> source file path
+ fingerprint: str # SHA256 of serialized content
+```
+
+Flat structure. Domain is implicit in the keys — `scoring.WeightedScore`, `scoring.compute_score`. No `DomainCrate` wrapper.
+
+### Changes
+
+- Define `LibraryCrate` model with serialization tests
+- `LibraryCrateFactory.make_from_blueprints(blueprints)`: for each bundle, qualify concept codes and pipe codes with the bundle's domain, add to flat dicts, compute fingerprint. Same ref appearing twice = error.
+- New `LibraryManager.load_from_crate(library_id, crate)`: extract domain codes from refs, create domains/concepts/pipes. This becomes the single entry point for loading.
+- Refactor `load_from_blueprints()`: internally builds a crate, then calls `load_from_crate()`. Callers don't change yet, but the path goes through the crate.
+- Update `pipeline_run_setup()` to route through the crate path.
+
+### Key files
+
+| File | Change |
+|------|--------|
+| `pipelex/libraries/library_crate.py` | **New** — `LibraryCrate` model |
+| `pipelex/libraries/library_crate_factory.py` | **New** — `make_from_blueprints()`, fingerprint computation |
+| `pipelex/libraries/library_manager.py` | Add `load_from_crate()`, refactor `load_from_blueprints()` to go through crate |
+| `pipelex/pipeline/pipeline_run_setup.py` | Route through crate |
+
+### Done when
+
+- [x] `LibraryCrate` model serializes and deserializes correctly (unit test: JSON round-trip)
+- [x] `make_from_blueprints()` merges correctly (unit test: two blueprints for same domain → both pipes/concepts in flat crate with qualified refs)
+- [x] `make_from_blueprints()` raises on concept/pipe code collision within a domain (unit test)
+- [x] `load_from_crate()` produces a valid library equivalent to `load_from_blueprints()` (integration test)
+- [x] All existing pipeline tests pass through the new `bundles → crate → library` path
+- [x] `make agent-check` passes
+- [x] `make agent-test` passes
+
+---
+
+## Phase 2: LibraryCrate on Temporal
+
+> **Status**: Complete
+
+**Goal**: Temporal workers receive and load a LibraryCrate. Pipe controllers (PipeSequence, PipeCondition, PipeBatch, PipeParallel) work on workers. This solves Layer 2 (pipe resolution).
+
+**What it does NOT solve yet**: Layer 1 (dynamic class deserialization) for concepts introduced by `mthds_content` that aren't in PIPELEXPATH. That's Phase 3.
+
+See [phase2-crate-propagation-rationale.md](phase2-crate-propagation-rationale.md) for the design rationale and rejected alternatives.
+
+### Design
+
+The `LibraryCrate` lives on `PipeJob` and propagates through the signature chain so that every child workflow receives it — each child workflow is an independent Temporal workflow that can land on any worker, so the crate must travel in the serialized `PipeJob`.
+
+Each `WfPipeRouter` loads the crate inline via `load_from_crate()`, idempotent via fingerprint.
+
+### Propagation chain
+
+```
+WfPipeRouter.run(pipe_job)
+ # 1. Load crate into this worker's library (idempotent via fingerprint)
+ # 2. Pass crate through the signature chain:
+ → pipe.run_pipe(..., library_crate=pipe_job.library_crate)
+ → _live_run_pipe(..., library_crate)
+ → [PipeController] _live_run_controller_pipe(..., library_crate)
+ → SubPipe.run_pipe(..., library_crate)
+ → PipeJobFactory.make_pipe_job(..., library_crate)
+ → PipeRouterChild → new WfPipeRouter(child_pipe_job)
+ # child_pipe_job.library_crate is set → cycle repeats
+```
+
+`library_crate` is an optional parameter (default `None`) at each level. Pipe operators receive it but ignore it — they have no child pipes.
+
+### Changes
+
+- Add `library_crate: LibraryCrate | None` field to `PipeJob`
+- In `PipeRouterTop`: after `pipeline_run_setup()` loads the library, build the flat crate from the blueprints that were loaded beyond PIPELEXPATH base. Attach to `PipeJob`.
+- In `WfPipeRouter.run()`: load the crate inline via `load_from_crate()`, pass to `pipe.run_pipe()`
+- Thread `library_crate` through the signature chain: `PipeAbstract.run_pipe()` → `_live_run_pipe()` → `PipeController._live_run_controller_pipe()` → `SubPipe.run_pipe()` → `PipeJobFactory.make_pipe_job()`
+- `PipeOperator._live_run_pipe()`: accept param, ignore it (no child pipes)
+
+### Key files
+
+| File | Change |
+|------|--------|
+| `pipelex/pipe_run/pipe_job.py` | Add `library_crate: LibraryCrate | None` field |
+| `pipelex/pipe_run/pipe_job_factory.py` | Accept and forward `library_crate` |
+| `pipelex/core/pipes/pipe_abstract.py` | Add optional `library_crate` param to `run_pipe()`, `_live_run_pipe()` |
+| `pipelex/pipe_operators/pipe_operator.py` | Add param to `_live_run_pipe()` (ignored) |
+| `pipelex/pipe_controllers/pipe_controller.py` | Add param to `_live_run_pipe()`, `_live_run_controller_pipe()` |
+| `pipelex/pipe_controllers/sequence/pipe_sequence.py` | Forward `library_crate` to `SubPipe.run_pipe()` |
+| `pipelex/pipe_controllers/batch/pipe_batch.py` | Forward `library_crate` to child pipe execution |
+| `pipelex/pipe_controllers/condition/pipe_condition.py` | Forward `library_crate` to child pipe execution |
+| `pipelex/pipe_controllers/parallel/pipe_parallel.py` | Forward `library_crate` to `SubPipe.run_pipe()` |
+| `pipelex/pipe_controllers/sub_pipe.py` | Add param, set on child `PipeJob` via factory |
+| `pipelex/pipe_run/pipe_router.py` | Pass `library_crate` from `PipeJob` to `run_pipe()` |
+| `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Load crate inline, pass `library_crate` to `run_pipe()` |
+| `pipelex/temporal/tprl_pipe/pipe_router_top.py` | Build `LibraryCrate` when dispatching |
+
+### Done when
+
+- [x] Integration test: PipeSequence through Temporal with PIPELEXPATH-based library only (no crate needed)
+- [x] Integration test: PipeSequence with `mthds_content` containing additional pipes (crate shipped, loaded on worker)
+- [x] `get_required_pipe()` works on worker for child pipes
+- [x] Crate is visible as structured JSON in Temporal dashboard
+- [x] `make agent-check` passes
+- [x] `make agent-test` passes
+
+---
+
+## Phase 3: Deferred WorkingMemory Hydration
+
+> **Status**: Complete
+
+**Goal**: Handle Layer 1 — when `mthds_content` introduces dynamic concept classes that the worker doesn't know about at deserialization time.
+
+**The problem**: Temporal auto-deserializes PipeJob via the Kajson data converter. If WorkingMemory contains Stuff objects with dynamic content classes (e.g., `RawText`), Kajson fails because those classes aren't registered on the worker yet. The crate loading can't happen until the workflow starts, but the workflow input must be deserialized first.
+
+**The solution**: WorkingMemory travels as a raw JSON dict when it contains dynamic concepts. After library setup registers the classes, the dict is hydrated into a typed `WorkingMemory`.
+
+### PipeJob changes
+
+```python
+class PipeJob(BaseModel):
+ pipe: PipeAbstract
+ working_memory: WorkingMemory | None = None # typed (direct mode, or no dynamic concepts)
+ working_memory_raw: dict[str, Any] | None = None # raw JSON (when deferred hydration needed)
+ pipe_run_params: PipeRunParams
+ job_metadata: JobMetadata
+ output_name: str | None = None
+ library_crate: LibraryCrate | None = None
+```
+
+### Key files
+
+| File | Change |
+|------|--------|
+| `pipelex/pipe_run/pipe_job.py` | Add `working_memory_raw: dict[str, Any] | None` field |
+| `pipelex/temporal/tprl_pipe/pipe_router_top.py` | Detect dynamic concepts, serialize WM to raw dict |
+| `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Hydration logic after library setup |
+| `pipelex/temporal/tprl_pipe/hydration.py` | **New** — `hydrate_working_memory()` utility |
+
+### Done when
+
+- [x] Integration test: PipeSequence with `mthds_content` containing a custom concept with inline structure (dynamic class)
+- [x] `working_memory_raw` hydrates correctly after library setup
+- [x] Stuff objects have correct typed content after hydration
+- [x] `working_memory_raw` is visible as plain JSON in Temporal dashboard
+- [x] `make agent-check` passes
+- [x] `make agent-test` passes
+
+---
+
+## Phase 4: Explicit ClassRegistry
+
+> **Status**: Complete
+
+**Goal**: Move ClassRegistry scoping out of Kajson (a serialization library) and into Pipelex's Library lifecycle. Fix two bugs that only manifest with separate Temporal worker processes: decoder bypass (dynamic classes with `__module__="builtins"`) and teardown clobber (non-stack-safe `finally` block).
+
+See [phase4-explicit-class-registry.md](phase4-explicit-class-registry.md) for the full design.
+
+### Design
+
+- **Kajson becomes a pure library**: `loads()`/`load()` accept an explicit `class_registry` parameter. ContextVar and `set_scoped_class_registry()` are removed from `KajsonManager`.
+- **Library owns its ClassRegistry**: `Library` gets a `PrivateAttr` `_class_registry`. Each workflow creates a plain `ClassRegistry` pre-seeded from global, attaches it to its Library. GC'd with the Library — no teardown bugs, no memory leaks.
+- **Reuse existing `_library_id` ContextVar**: `hub.get_class_registry()` reads the active library_id, gets that library's ClassRegistry. No new ContextVar, no CompositeClassRegistry.
+- **Explicit at the boundary**: `temporal_data_converter` passes the registry to `kajson.loads()`.
+- **Migrate ~20 callers**: `KajsonManager.get_class_registry()` → `hub.get_class_registry()` (mechanical).
+
+### Key files
+
+| File | Change |
+|------|--------|
+| `kajson/kajson.py` | Add `class_registry` param to `loads()`/`load()` |
+| `kajson/json_decoder.py` | Accept + use explicit registry in decoder |
+| `kajson/kajson_manager.py` | Remove ContextVar, simplify to global-only |
+| `pipelex/libraries/library.py` | PrivateAttr ClassRegistry on Library |
+| `pipelex/hub.py` | `get_class_registry()` reads from library |
+| `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Plain ClassRegistry, library-associated |
+| `pipelex/temporal/temporal_data_converter.py` | Pass explicit registry to `kajson.loads()` |
+| ~9 Pipelex files | Migrate `KajsonManager.get_class_registry()` → `hub.get_class_registry()` |
+
+### Done when
+
+- [x] `kajson.loads(data, class_registry=reg)` resolves dynamic classes with `__module__="builtins"` (new unit test)
+- [x] `KajsonManager` has no ContextVar, no `set_scoped_class_registry`
+- [x] `Library._class_registry` is a PrivateAttr, GC'd with Library
+- [x] All ~20 callers migrated from `KajsonManager.get_class_registry()` to `hub.get_class_registry()`
+- [x] Manual Temporal test: 3-process setup with `dynamic_concept_sequence.mthds` passes
+- [x] `make agent-check` passes (both repos)
+- [x] `make agent-test` passes (both repos)
+
+---
+
+## Phase 4.5: Distributed Tracing & Reporting (Local Version)
+
+> **Status**: Complete (Steps 0–5). Step 6 moved to [01-master-plan.md](01-master-plan.md).
+
+**Goal**: Add a file-based event log to GraphTracer so that graph tracing and cost reporting work across Temporal workers. The in-memory path is preserved unchanged for direct mode. Full implementation details in [phase4.5-distributed-tracing-implementation.md](phase4.5-distributed-tracing-implementation.md).
+
+**Design**: Dual-write in GraphTracer — when an `EventLogProtocol` is provided, events are emitted to NDJSON files alongside the existing in-memory accumulation. After all workflows complete, an assembler reads events from all files and reconstructs a cross-worker `GraphSpec`. No new tracer class, no call site changes.
+
+### Done when
+
+- [x] **Step 0 — Event Models**: `TraceEvent` hierarchy with discriminated union, serialization round-trips, `TokensUsage` models fixed to `Literal` discriminators
+- [x] **Step 1 — EventLogProtocol & NDJSON Backend**: `NdjsonEventLog` with emit/read/cleanup/dedup, `InMemoryEventLog` for tests, corrupt-line handling, multiprocess concurrent write safety
+- [x] **Step 2 — GraphSpec Assembler**: Two-pass assembly (nodes + producer map → edges), structural equivalence with current `GraphTracer` output, `UsageAggregator`
+- [x] **Step 3 — Emit Integration**: Dual-write in `GraphTracer` (optional `EventLogProtocol`), `ReportingManager` emits `UsageReportEvent`, node/edge IDs include `workflow_id` segment
+- [x] **Step 4 — Wire into Pipeline Lifecycle**: `TracingConfig`, `NdjsonEventLog` created in `WfPipeRouter` and `pipeline_run_setup`, assembly in `runner.py` finally block, direct mode unchanged
+- [x] **Step 5 — Temporal Integration Tests**: PipeSequence/PipeParallel/PipeBatch through Temporal produce correct GraphSpec, replay dedup, failure/graph-output tests
+- **Step 6 — Standalone Activity Tracing**: moved to [01-master-plan.md](01-master-plan.md)
+
+---
+
+## Phase 5: StoragePayloadCodec
+
+**Goal**: Remove the 2MB payload size limit for production workloads with large libraries or WorkingMemory containing images/documents.
+
+**Why last**: Phases 0-4 work for small-to-medium payloads. Phase 5 is needed when payloads grow beyond Temporal's limits.
+
+### StoragePayloadCodec
+
+Extends `temporalio.converter.PayloadCodec`. Operates at the wire boundary — transparent to all application code.
+
+```
+Application code StoragePayloadCodec Temporal Server
+----- ----- -----
+pass WorkingMemory (50MB) --> encode(): upload to storage, --> stores small ref
+ replace with storage key ref in Event History
+
+receive storage ref <-- decode(): download from storage, <-- reads ref from
+return WorkingMemory (50MB) reconstruct original payload Event History
+```
+
+- Content-addressed (SHA256 key) = natural deduplication
+- Uses Pipelex's existing `StorageProviderAbstract`
+- V1: `LocalStorageProvider` (same filesystem for dev)
+- Threshold: 1MB (configurable, well under 2MB hard limit)
+
+### Key files
+
+| File | Change |
+|------|--------|
+| `pipelex/temporal/storage_payload_codec.py` | **New** — StoragePayloadCodec class |
+| `pipelex/temporal/temporal_data_converter.py` | Add codec to DataConverter |
+| `pipelex/temporal/temporal_connect.py` | Pass codec-enabled converter |
+| `pipelex/temporal/worker_cli.py` | Pass codec-enabled converter |
+| `pipelex/temporal/config_temporal.py` | Add payload codec config |
+| `pipelex/pipelex.toml` | Add `[pipelex.temporal.payload_codec]` |
+
+### Done when
+
+- [x] Unit test: payloads above threshold are stored externally, below threshold pass through
+- [x] Unit test: content-addressed deduplication works
+- [x] Integration test: large WorkingMemory survives Temporal round-trip
+- [x] `make agent-check` passes
+- [x] `make agent-test` passes
+
+---
+
+## E2E Test Coverage
+
+> **Status**: Complete — all controllers passing
+
+**Goal**: Comprehensive Temporal regression test suite covering all pipe controller types (PipeSequence, PipeParallel, PipeCondition, PipeBatch) and non-LLM operators (PipeCompose). Validates that LibraryCrate propagation, deferred hydration, and per-workflow isolation work across all pipe dispatch patterns.
+
+### What was added
+
+5 new `.mthds` bundles in `tests/integration/pipelex/temporal/library_crate/`:
+
+| Bundle | Controller/Operator | Pattern |
+|--------|-------------------|---------|
+| `temporal_condition.mthds` | PipeCondition | Sequence → LLM → Condition → 3 outcome LLMs |
+| `temporal_parallel.mthds` | PipeParallel | Sequence → Parallel (2 branches) → LLM summary |
+| `temporal_batch.mthds` | PipeBatch | Sequence → LLM (list) → Batch fan-out → LLM per-item |
+| `temporal_compose.mthds` | PipeCompose | Sequence → 2 LLMs → Compose construct (+ dynamic Report concept) |
+| `temporal_combined.mthds` | Mixed | Sequence → Parallel → Condition → LLM (nested dispatch) |
+
+5 new test files with 10 tests (5 crate structure + 5 execution) — all passing.
+
+### StuffArtefact serialization issue — Resolved
+
+**Was**: PipeCondition and PipeCompose dispatched `WfMakeJinja2Text` (a `ContentGeneratorChild` Temporal sub-workflow) for expression/template evaluation. These sub-workflows serialized working memory through the Temporal data converter, and `StuffArtefact` (a plain Python object, not a Pydantic model) was not JSON-serializable. PipeBatch had a related hang.
+
+**Resolution**: Jinja2 templating no longer routes through a `ContentGeneratorChild` Temporal sub-workflow — it happens in-process. Working memory is never serialized across processes for templating, so `StuffArtefact` objects in dry-run mode are no longer a problem. No changes to `StuffArtefact` itself were needed.
+
+### Test summary
+
+| Controller | Crate structure | Execution (dry-run) | Execution (live) |
+|------------|:-:|:-:|:-:|
+| PipeSequence | pass | pass | pass |
+| PipeParallel | pass | pass | untested |
+| PipeCondition | pass | pass | untested |
+| PipeBatch | pass | pass | untested |
+| PipeCompose | pass | pass | untested |
+| Combined (Parallel+Condition) | pass | pass | untested |
+
+---
+
+**Next phases**: See [../01-master-plan.md](../01-master-plan.md) for Phase 4.5 Step 6 (Standalone Activity Tracing), Phase 6 (Cross-Package Dependencies), and Future Phases.
+
+---
+
+## Implementation Pattern Per Phase
+
+1. **Tests first** — Define new models, write serialization/unit tests before implementation
+2. **Model + core logic** — Implement models and core functions, pass unit tests
+3. **Integration** — Wire into existing plumbing, pass integration tests
+4. **`make agent-check`** — Lint + type check
+5. **`make agent-test`** — Full suite passes
+6. **Review "done" checklist** — All criteria met before moving to next phase
diff --git a/wip/archive/01-master-plan.md b/wip/archive/01-master-plan.md
new file mode 100644
index 000000000..fe094aa2b
--- /dev/null
+++ b/wip/archive/01-master-plan.md
@@ -0,0 +1,134 @@
+# Master Plan v2 — Next Phases (Distributed Execution)
+
+> **Status**: Live plan — drives the remaining LibraryCrate / distributed-execution work
+> **Last refreshed**: 2026-05-04
+> **Predecessor**: [archive/00-master-plan.md](archive/00-master-plan.md) — Phases 0–5 + E2E coverage, all shipped.
+
+This file tracks the **distributed execution** work that follows the LibraryCrate refactor. Worker error-handling work lives in its own files (see "Sibling work streams" below).
+
+---
+
+## What's left
+
+| Item | Status | Why we still need it |
+|---|---|---|
+| **Phase 6a — Local cross-package dependencies in crate** | Not started | Today the crate only ships the main package's blueprints; cross-package deps still resolve via PIPELEXPATH on the worker. Removing this is the prerequisite to truly stateless workers. |
+| **Phase 6b — Remote (GitHub) dependencies** | Not started | Builds on 6a; lets the crate carry deps fetched from remote addresses so workers don't need any package pre-installed. |
+
+## Recently shipped (since 00-master-plan)
+
+| Item | What actually shipped |
+|---|---|
+| **Phase 4.5 Step 6 — Cross-process tracing** | Shipped via `feature/dynamodb-tracer`, but with a different design than the originally proposed `TracingActivityInboundInterceptor`. See "Phase 4.5 Step 6 — as built" below. |
+
+Open architectural notes and known limitations live in [deferred-items.md](deferred-items.md). Long-term direction in [future-crate-first-architecture.md](future-crate-first-architecture.md). Tracing background (pre-implementation analysis — decisions superseded) in [distributed-tracing-and-reporting.md](distributed-tracing-and-reporting.md).
+
+---
+
+## Phase 4.5 Step 6 — as built
+
+The original Step 6 plan called for a `TracingActivityInboundInterceptor` to set up an NDJSON event log per standalone activity. The team picked a different design:
+
+- **Pluggable backends behind `EventLogProtocol`** — `pipelex/tracing/event_log_protocol.py`. Implementations:
+ - `NdjsonEventLog` — local file backend (`pipelex/tracing/ndjson_event_log.py`).
+ - `DynamoDBEventLog` — cloud backend (`pipelex/tracing/dynamodb_event_log.py`), schema compatible with `pipelex-api-infra`'s `TraceEventDynamoDBAdapter`. Available behind `pip install "pipelex[dynamodb]"`.
+ - `BufferingEventLog` — in-memory buffer used **inside** Temporal workflow code (`pipelex/tracing/buffering_event_log.py`). Synchronous I/O is forbidden in workflow context, so workflows buffer here and flush via an activity.
+ - `InMemoryEventLog` — for tests.
+ - Backend chosen at runtime via `make_event_log(tracing_config)` in `pipelex/tracing/event_log_factory.py` (`TracingBackend.NDJSON | DYNAMODB | TEMPORAL_DYNAMODB`).
+- **Workflow → activity flush, not interceptor.** `WfPipeRouter.run()` (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`) wires a `BufferingEventLog` into the per-workflow `GraphTracerManager` and into the `ReportingManager` via `set_event_log(context_key=workflow_id, ...)`. After pipe execution, buffered events are drained and persisted by `act_flush_trace_events` (`pipelex/temporal/tprl_pipe/act_flush_trace_events.py`), which runs the synchronous boto3 / file writes off the workflow thread.
+- **Per-context keying** — `ReportingManager` holds a dict of `_event_log_contexts` keyed by `graph_context.lookup_key`, so concurrent workflows don't trample each other.
+- **Direct mode unchanged** — `pipeline_run_setup.py:282` calls the same `set_event_log` path, so the same `EventLogProtocol` machinery runs in-process for non-Temporal execution.
+
+Trade-off vs the original interceptor design: this approach does not capture usage events emitted from activities running in a **separate worker process** that doesn't share the workflow's `ReportingManager`. In the current deployment shape (workflows + activities on the same Worker bundle) this is a non-issue. If we ever split activities onto standalone Workers, the interceptor pattern from the old plan becomes the natural extension — at that point reopen this as a follow-up.
+
+Pre-implementation analysis (proposed SQLite/Redis backends, decision gates, etc.) is in [distributed-tracing-and-reporting.md](distributed-tracing-and-reporting.md), kept as historical context.
+
+---
+
+## Phase 6: Cross-Package Dependencies in Crate
+
+> **Status**: Not started
+
+**Goal**: Include cross-package dependency content in the LibraryCrate so that Temporal workers can execute pipelines that reference concepts and pipes from other packages — without those packages being installed on the worker. Then extend to remote dependencies fetched from GitHub.
+
+See [future-crate-first-architecture.md](future-crate-first-architecture.md) for the full crate-first architectural vision and design rationale.
+
+### Why
+
+Today, cross-package dependencies (`alias->domain.ConceptCode`, `alias::domain.pipe_code`) resolve through child library lookups at loading time. The `LibraryCrate` shipped to Temporal workers does NOT include dependency content — only the main package's blueprints. This means workers must have all dependency packages pre-installed on PIPELEXPATH. Phase 6 removes this requirement: the crate becomes truly self-contained.
+
+### Phase 6a: Local Cross-Package Dependencies
+
+**Goal**: Dependency blueprints are included in the crate. Workers can execute pipelines with cross-package deps without having the dependency packages installed.
+
+**Key changes**:
+
+1. **Extract blueprint collector from `_load_single_dependency`**: Split the current method into two parts:
+ - **Collect**: resolve the dependency, parse its `.mthds` files into blueprints, determine exports
+ - **Load**: create child Library, load domains/concepts/pipes, register aliases
+ The collector produces blueprints that accumulate into `_blueprints[library_id]` alongside the main package's blueprints.
+
+2. **Resolve cross-package aliases in the flattened crate**: Cross-package refs use alias-based syntax (`alias->domain.ConceptCode`). In a flattened crate, there are no child libraries. Options:
+ - Resolve aliases during crate building (replace `alias->domain.ConceptCode` with `domain.ConceptCode` in all blueprints)
+ - Carry alias mappings in the crate (`aliases: dict[str, str]`) for runtime resolution
+ Decision TBD when starting implementation.
+
+3. **Update `load_from_crate()`**: Handle dependency content in the flat crate — register domains, concepts, and pipes from deps.
+
+**Done when**:
+
+- [ ] `_load_single_dependency` split into collect + load
+- [ ] Dependency blueprints accumulate into `_blueprints[library_id]`
+- [ ] `get_crate()` produces a crate that includes dependency content
+- [ ] Cross-package aliases resolved (either at crate build time or via alias map)
+- [ ] `load_from_crate()` handles dependency content correctly
+- [ ] Integration test: PipeSequence referencing a cross-package concept/pipe, executed on Temporal worker without the dependency package on PIPELEXPATH
+- [ ] `make agent-check` passes
+- [ ] `make agent-test` passes
+
+### Phase 6b: Remote Dependencies (GitHub)
+
+**Goal**: Dependencies can be fetched from remote addresses (e.g., `github.com/org/repo/package`). The crate becomes fully self-contained for cloud-native execution where workers are stateless.
+
+**Key changes**:
+
+1. **Remote resolution strategy**: Add a `REMOTE` resolution strategy to the blueprint collector extracted in 6a. Remote fetch clones/downloads the package from GitHub, parses its `.mthds` files into blueprints, and includes them in the collection.
+
+2. **Dependency address format**: Define the address format for remote deps (e.g., `github.com/org/repo@version/path/to/package`). The address goes in the package manifest.
+
+3. **Caching**: Cache fetched packages locally (content-addressed by address + version) to avoid redundant clones.
+
+4. **Transitive deps**: Remote packages may themselves have dependencies (local or remote). The collector recurses.
+
+**Done when**:
+
+- [ ] Remote fetch strategy implemented (git clone or archive download)
+- [ ] Dependency address format defined and parsed
+- [ ] Remote package blueprints included in crate
+- [ ] Local cache for fetched packages
+- [ ] Transitive remote deps resolved
+- [ ] Integration test: pipeline with remote dep from a GitHub repo, executed on Temporal worker
+- [ ] `make agent-check` passes
+- [ ] `make agent-test` passes
+
+---
+
+## Dependencies between remaining items
+
+```
+Phase 6a (Local cross-package deps) ← requires Phase 2's crate propagation (shipped)
+ │
+ ▼
+Phase 6b (Remote deps from GitHub) ← requires Phase 6a's blueprint collector
+```
+
+---
+
+## Sibling work streams (not in this plan)
+
+These tracks are tracked in their own files and ship on their own branches:
+
+| Stream | Files | Branch |
+|---|---|---|
+| Worker error-handling Phases 4–7 | `error-handling-phase-{4,5,6,7}-*.md`, `error-handling-phases-0-3-completed.md`, `worker-error-handling-review.md`, `error-handling-review.md` | `refactor/Inference-error-handling` |
+| Instructor-unwrap port to OpenAI/Mistral/Google workers | `instructor-unwrap-other-workers.md` | (open — Anthropic done, four workers remaining) |
diff --git a/wip/archive/activity_storage-plan.md b/wip/archive/activity_storage-plan.md
new file mode 100644
index 000000000..575288ac5
--- /dev/null
+++ b/wip/archive/activity_storage-plan.md
@@ -0,0 +1,123 @@
+# Activity-Level Storage — Execution Plan
+
+> Design: [wip/activity_storage.md](wip/activity_storage.md)
+> Branch: `fix/Temporal-Img`
+
+---
+
+## Phase 1: Image Generation (fixes the immediate bug) ✅
+
+### 1.1 Modify `act_img_gen_images` to include storage
+
+- [x] **File**: `pipelex/cogt/content_generation/img_gen_generate.py`
+ - Add `img_gen_single_image_and_store()` and `img_gen_image_list_and_store()` functions
+ - These call the existing generation logic, then use `GeneratedContentFactory` (with `get_storage_provider()`) to store and return `ImageContent`
+ - Keep the original `img_gen_single_image()` and `img_gen_image_list()` functions for the non-Temporal path
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/act_img_gen_generate.py`
+ - Change `act_img_gen_images` return type: `list[GeneratedImageRawDetails]` → `list[ImageContent]`
+ - Call the new `_and_store` functions instead of the raw generation functions
+ - The activity receives `ImgGenAssignment` (unchanged) which already has `job_metadata` with `user_id` and `pipeline_run_id`
+
+### 1.2 Update `WfMakeImages`
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/wf_make_images.py`
+ - Change return type: `list[GeneratedImageRawDetails]` → `list[ImageContent]`
+ - No logic changes — just the type annotation
+
+### 1.3 Update `ContentGeneratorChild` and `ContentGeneratorTop`
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/content_generator_child.py`
+ - `make_single_image()`: Gets `list[ImageContent]` from `WfMakeImages`, extracts the single item directly. Removed `make_image_content()` call
+ - `make_image_list()`: Gets `list[ImageContent]` from `WfMakeImages`, returns directly. Removed comprehension calling `make_image_content()`
+ - `make_image_content()`: Kept for protocol compliance but no longer called from image generation paths
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/content_generator_top.py`
+ - Same changes as `ContentGeneratorChild` — `make_single_image()` and `make_image_list()` now use `ImageContent` directly from workflow
+
+### 1.4 Update test workflow
+
+- [x] **File**: `pipelex/temporal/test_extras/wf_test_content_generator_child.py`
+ - Image generation is commented out (pre-existing TODO). No changes needed
+
+### 1.5 Lint + test
+
+- [x] Run `make agent-check` — 0 errors, 0 warnings
+- [x] Run `make agent-test` — all tests passed
+
+---
+
+## Phase 2: Extraction with images ✅
+
+### 2.1 Modify `act_extract_gen_extract_pages` to include storage
+
+- [x] **File**: `pipelex/cogt/content_generation/extract_generate.py`
+ - Added `extract_gen_pages_and_store()` that calls extraction, then uses `GeneratedContentFactory.make_page_contents()` to store extracted images and return `list[PageContent]`
+ - Kept original `extract_gen_pages()` for non-Temporal path
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/act_extract_generate.py`
+ - Changed return type: `ExtractOutput` → `list[PageContent]`
+ - Calls the new `_and_store` function
+
+### 2.2 Update `WfMakeExtract`
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/wf_make_extract.py`
+ - Changed return type: `ExtractOutput` → `list[PageContent]`
+
+### 2.3 Update `ContentGeneratorChild.make_extract_pages` and `ContentGeneratorTop`
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/content_generator_child.py`
+ - `make_extract_pages()`: Gets `list[PageContent]` from `WfMakeExtract`, returns directly. Removed `make_page_contents()` call
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/content_generator_top.py`
+ - Same changes as `ContentGeneratorChild`
+
+### 2.4 Lint + test
+
+- [x] Run `make agent-check` — 0 errors, 0 warnings
+- [x] Run `make agent-test` — all tests passed
+
+---
+
+## Phase 3: Page view rendering ✅
+
+### 3.1 Create new activity and workflow for page view rendering
+
+- [x] **File**: `pipelex/cogt/content_generation/assignment_models.py`
+ - Added `RenderPageViewsAssignment` model with `job_metadata`, `document_uri`, `page_views_dpi`
+
+- [x] **File** (new): `pipelex/temporal/tprl_content_generation/act_render_page_views.py`
+ - Created `act_render_page_views` activity
+ - Renders PDF pages via pypdfium2, stores images, returns `list[ImageContent]`
+
+- [x] **File** (new): `pipelex/temporal/tprl_content_generation/wf_render_page_views.py`
+ - Created `WfRenderPageViews` workflow wrapping the activity
+
+### 3.2 Update `ContentGeneratorChild.make_render_page_views` and `ContentGeneratorTop`
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/content_generator_child.py`
+ - `make_render_page_views()`: Delegates to `WfRenderPageViews` child workflow. Removed direct pypdfium2 rendering and `make_image_content` calls
+
+- [x] **File**: `pipelex/temporal/tprl_content_generation/content_generator_top.py`
+ - Same changes — delegates to `WfRenderPageViews` workflow
+
+### 3.3 Register in task catalog
+
+- [x] **File**: `pipelex/temporal/tasks.py`
+ - Added `WfRenderPageViews` to `CRAFTING` workflow list
+ - Added `act_render_page_views` to `CRAFTING` activity list
+
+### 3.4 Lint + test
+
+- [x] Run `make agent-check` — 0 errors, 0 warnings (ruff auto-removed unused imports)
+- [x] Run `make agent-test` — all tests passed
+
+---
+
+## Phase 4: Cleanup and full validation ✅
+
+- [x] Run `make agent-check` (full lint + type check) — 0 errors
+- [x] Run `make agent-test` (full test suite) — all passed
+- [ ] Verify no `PayloadSizeWarning` in worker logs (requires live Temporal test)
+- [ ] Verify no `NotImplementedError` crashes (requires live Temporal test)
+- [ ] Run `/temporal-e2e-validate` Mode 2 (3-process test) — includes image payload tiers
diff --git a/wip/archive/activity_storage.md b/wip/archive/activity_storage.md
new file mode 100644
index 000000000..e06125f76
--- /dev/null
+++ b/wip/archive/activity_storage.md
@@ -0,0 +1,207 @@
+# Activity-Level Storage: Fixing Non-Deterministic I/O in Temporal Workflows
+
+> **Status**: Design
+> **Date**: 2026-04-01
+> **Related**: [phase5-payload-codec-strategy.md](phase5-payload-codec-strategy.md), Temporal img pipeline bug (fix/Temporal-Img branch)
+> **Trigger**: `NotImplementedError` when running `test_image_out_in.mthds` through Temporal — aiohttp's connector cleanup calls `loop.is_closed()` on Temporal's sandboxed event loop
+
+---
+
+## 1. The Bug
+
+When running an image-generation pipeline through Temporal, the workflow crashes at the S3 storage step:
+
+```
+wf_pipe_router.py:127 → pipe_img_gen.py:214 → content_generator_child.py:372
+→ content_generator_child.py:306 → generated_content_factory.py:113
+→ s3_storage_provider.py:157 → aiohttp connector close → NotImplementedError
+```
+
+`ContentGeneratorChild.make_image_content()` calls `GeneratedContentFactory.make_image_content()` directly within the workflow context. The factory does S3 storage via aiobotocore/aiohttp. aiohttp's cleanup calls `loop.is_closed()` on Temporal's restricted event loop, which doesn't implement that method.
+
+**Root cause**: Non-deterministic I/O (S3 upload, HTTP fetch) executing inside a Temporal workflow instead of an activity.
+
+---
+
+## 2. Scope of the Problem
+
+Three methods in `ContentGeneratorChild` perform I/O directly in workflow context:
+
+| Method | Line | I/O performed | Called by |
+|--------|------|---------------|-----------|
+| `make_image_content` | 300-314 | S3 storage via factory | `make_single_image`, `make_image_list`, `make_render_page_views` |
+| `make_page_contents` | 317-326 | S3 storage via factory (for each page image) | `make_extract_pages` |
+| `make_render_page_views` | 462-490 | pypdfium2 PDF rendering + S3 storage via `make_image_content` | `make_extract_pages` |
+
+All other content generation methods (LLM text, objects, image generation, extraction, templating) correctly delegate to child workflows that run activities.
+
+---
+
+## 3. Why a Separate Storage Activity is Not the Right Fix
+
+The naive fix — create an activity that wraps the storage call — has two problems:
+
+1. **Payload size limit**: Temporal enforces a **2MB limit per individual payload** (inputs and outputs of both activities and workflows). The current `act_img_gen_images` already returns `list[GeneratedImageRawDetails]` containing base64 image data. We already see `PayloadSizeWarning` at 1.8MB. Larger images would hard-fail. A separate storage activity would need to receive the same large data as input — hitting the same limit.
+
+2. **Double history pollution**: The large image data would enter workflow history twice — once as the generation activity output, once as the storage activity input.
+
+The Temporal-recommended "Large Data Handling" pattern (see `phase5-payload-codec-strategy.md` section "Large Data Handling" from the Temporal developer skill): activities should read AND write large data internally, passing only small references through the workflow.
+
+---
+
+## 4. The Solution: Merge Storage into Generation Activities
+
+### 4.1 Core Principle
+
+Each content generation activity becomes responsible for the full lifecycle: **generate → store → return lightweight reference**. Large binary data never crosses a Temporal boundary.
+
+### 4.2 New Activity Signatures
+
+**Image generation** (currently returns large `GeneratedImageRawDetails`):
+
+```python
+# BEFORE
+async def act_img_gen_images(img_gen_assignment: ImgGenAssignment) -> list[GeneratedImageRawDetails]:
+ ... # generate, return raw base64 data
+
+# AFTER
+async def act_img_gen_images(img_gen_assignment: ImgGenAssignment) -> list[ImageContent]:
+ ... # generate, store to S3, return ImageContent with URLs
+```
+
+The activity internally:
+1. Generates the image (existing logic)
+2. Calls `GeneratedContentFactory.make_image_content()` to store and build `ImageContent`
+3. Returns `ImageContent` (only URLs, no binary data)
+
+**Extraction** (currently returns `ExtractOutput` which can contain large extracted images):
+
+```python
+# BEFORE
+async def act_extract_gen_extract_pages(extract_assignment: ExtractAssignment) -> ExtractOutput:
+ ... # extract, return raw extracted data with images
+
+# AFTER
+async def act_extract_gen_extract_pages(extract_assignment: ExtractAssignment) -> list[PageContent]:
+ ... # extract, store images, return PageContent with URLs
+```
+
+**Page view rendering** (currently done entirely in workflow context):
+
+```python
+# NEW
+async def act_render_page_views(render_assignment: RenderPageViewsAssignment) -> list[ImageContent]:
+ ... # render PDF pages, store images, return ImageContent with URLs
+```
+
+### 4.3 Workflow Changes
+
+**`WfMakeImages`**: Returns `list[ImageContent]` instead of `list[GeneratedImageRawDetails]`.
+
+**`WfMakeExtract`**: Returns `list[PageContent]` instead of `ExtractOutput`.
+
+**New `WfRenderPageViews`**: Wraps the new `act_render_page_views` activity.
+
+### 4.4 ContentGeneratorChild Changes
+
+`make_image_content` and `make_page_contents` become trivial or eliminated — the child workflows already return the final content types.
+
+| Method | Before | After |
+|--------|--------|-------|
+| `make_single_image` | Calls `WfMakeImages` (returns raw details) → `make_image_content` (S3 I/O in workflow) | Calls `WfMakeImages` (returns `ImageContent`) → done |
+| `make_image_list` | Same pattern as above | Same fix |
+| `make_image_content` | Direct factory call with S3 I/O | No longer needed for Temporal path — storage happens in activity |
+| `make_extract_pages` | Calls `WfMakeExtract` (returns `ExtractOutput`) → `make_page_contents` (S3 I/O in workflow) | Calls `WfMakeExtract` (returns `list[PageContent]`) → done |
+| `make_page_contents` | Direct factory call with S3 I/O | No longer needed for Temporal path |
+| `make_render_page_views` | pypdfium2 rendering + S3 I/O in workflow | Calls `WfRenderPageViews` → done |
+
+### 4.5 Non-Temporal Path Unchanged
+
+`ContentGenerator` (the non-Temporal implementation) continues to work as-is. The factory-level `make_image_content` and `make_page_contents` remain for the non-Temporal path. The change is only in the Temporal activities and `ContentGeneratorChild`.
+
+---
+
+## 5. New Assignment Models
+
+The activities need assignment models that carry enough context for storage:
+
+```python
+class ImgGenAssignment(BaseModel):
+ # existing fields...
+ job_metadata: JobMetadata # already has user_id, pipeline_run_id for storage keys
+ img_gen_handle: str
+ img_gen_prompt: ImgGenPrompt
+ img_gen_job_params: ImgGenJobParams
+ img_gen_job_config: ImgGenJobConfig
+ nb_images: int
+ # NO NEW FIELDS NEEDED — job_metadata already carries what storage needs
+```
+
+`ImgGenAssignment` already contains `job_metadata` (with `user_id` and `pipeline_run_id`) which the `GeneratedContentFactory` needs for building storage keys. No model changes required for image generation.
+
+For page view rendering, a new assignment model:
+
+```python
+class RenderPageViewsAssignment(BaseModel):
+ job_metadata: JobMetadata
+ document_uri: str
+ page_views_dpi: int
+```
+
+---
+
+## 6. How the Activity Gets the Storage Provider
+
+Activities run in the worker process, which has the full Pipelex runtime initialized. The activity can access storage via:
+
+```python
+from pipelex.hub import get_storage_provider
+
+storage_provider = get_storage_provider()
+generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)
+```
+
+This is already the pattern used by other worker-side code (`gateway_extract_worker.py`, `pypdfium2_worker.py`).
+
+---
+
+## 7. Impact on the `ContentGeneratorProtocol`
+
+The protocol defines `make_image_content` and `make_page_contents` as public methods. After this change:
+
+- **Non-Temporal** (`ContentGenerator`): Still uses them — no change.
+- **Temporal** (`ContentGeneratorChild`): These methods are no longer called from workflow context. They could be removed from the child, or kept as pass-throughs.
+
+**Decision**: Keep the protocol unchanged. The `ContentGeneratorChild` implementations of `make_image_content` and `make_page_contents` will still exist but won't be called from workflow code paths. They're kept for protocol compliance and in case they're called from non-workflow contexts in the future.
+
+---
+
+## 8. Relationship to PayloadCodec (Phase 5)
+
+The PayloadCodec (`phase5-payload-codec-strategy.md`) is the general solution for large payload offloading — it transparently offloads any payload exceeding a size threshold to external storage at the wire boundary.
+
+This design is complementary, not competing:
+
+| Concern | This design | PayloadCodec |
+|---------|-------------|--------------|
+| Non-deterministic I/O in workflow | **Fixes** — moves I/O to activities | Does not fix — codec doesn't change where I/O happens |
+| Large payloads through Temporal | **Fixes for images** — activities return URLs | **Fixes for everything** — transparent offloading |
+| Required regardless of codec? | **Yes** — even with a codec, S3 I/O cannot run in workflow context | N/A |
+
+Even after PayloadCodec is implemented, activities should still handle their own storage. The codec is a safety net for payloads that are unavoidably large (e.g., library contexts), not a substitute for proper activity design.
+
+---
+
+## 9. Risk Assessment
+
+| Risk | Likelihood | Mitigation |
+|------|-----------|------------|
+| Changing `WfMakeImages` return type breaks existing tests | High | Update test assertions; tests already use `InMemoryStorageProvider` |
+| Extract output loss of detail (returning `PageContent` instead of `ExtractOutput`) | Medium | Ensure all needed fields from `ExtractOutput` are captured in `PageContent` |
+| `make_render_page_views` in `ContentGeneratorChild` also uses pypdfium2 I/O | Known | Covered by the new `WfRenderPageViews` workflow |
+| Storage provider not initialized in worker | Low | Worker init already sets up full Pipelex runtime; other activities use `get_storage_provider()` |
diff --git a/wip/archive/early-library-as-execution-context.md b/wip/archive/early-library-as-execution-context.md
new file mode 100644
index 000000000..f081a0639
--- /dev/null
+++ b/wip/archive/early-library-as-execution-context.md
@@ -0,0 +1,199 @@
+# The LibraryCrate: From File-System Bundles to Execution Context
+
+> **Status**: Architecture vision
+> **Date**: 2026-03-23
+> **Related**: [00-master-plan.md](00-master-plan.md), [phase0-pipe-namespace-fix.md](phase0-pipe-namespace-fix.md)
+
+---
+
+## 1. The Three-Stage Pipeline
+
+Library loading follows a three-stage pipeline. Each stage has a distinct role and a clean boundary with the next.
+
+```
+Bundles (files) ──► LibraryCrate (data) ──► Library (live)
+```
+
+**Bundles** are file-system and packaging artifacts: `.mthds` files, method packages with manifests, remote dependencies. They are how methods are authored, versioned, and distributed. A single domain can be spread across multiple bundle files — for example, a `scoring` domain might have `scoring_core.mthds` and `scoring_advanced.mthds`.
+
+**LibraryCrate** is a flat, source-agnostic, serializable snapshot. It contains concept blueprints keyed by `concept_ref` and pipe blueprints keyed by `pipe_ref` — both fully-qualified with their domain prefix. It carries no memory of which file or package each blueprint came from. When multiple bundles contribute to the same domain, their blueprints simply land in the same flat dicts (strict merge — same ref appearing twice is an error).
+
+**Library** is the live runtime. Concepts are instantiated as `Concept` objects with generated structure classes. Pipes are instantiated as `PipeAbstract` subclass instances. The library is the execution truth — what pipe controllers query when resolving child pipes at runtime.
+
+### What about domains?
+
+Domains are a **namespace convention**, not a runtime container. A domain like `scoring` provides the prefix in `scoring.WeightedScore` (concept) and `scoring.compute_score` (pipe). Domains help organize methods at authoring time and prevent naming conflicts, but at runtime they carry no meaningful state. The `Domain` object in the library has only `code`, `description`, and an optional `system_prompt` — none of which are needed for pipe or concept resolution. The LibraryCrate doesn't need domain-level grouping; the namespace is already encoded in the refs.
+
+### Why a middle stage?
+
+Without the LibraryCrate, the system jumps directly from file-system artifacts to live objects. This creates two problems:
+
+1. **Serialization**: Live objects contain runtime state (generated classes, resolved references) that cannot be cleanly serialized. Blueprints are pure Pydantic models — they serialize trivially.
+
+2. **Decoupling**: The live library shouldn't know about files, packages, or remote repositories. The crate is the clean handoff point: everything upstream of it deals with sourcing and parsing; everything downstream deals with instantiation and execution.
+
+---
+
+## 2. What a Pipeline Execution Actually Needs
+
+When a pipeline runs, it needs access to **concepts and pipes** — not files.
+
+1. **It needs all the concepts and pipes in its dependency graph.** A pipe controller resolves child pipes, which resolve their children, and so on. Each pipe references concepts for its inputs and outputs. The full transitive closure of these references defines what the execution needs.
+
+2. **It does not need anything outside that dependency graph.** A library loaded from the file system might contain 50 pipes. A given pipeline execution might only touch 15 of them and a fraction of the concepts. The rest is dead weight.
+
+This is possible to determine statically because Pipelex's execution is deterministic — the dependency graph of a pipeline is known before execution begins. There are no dynamic pipe lookups or runtime-resolved references.
+
+### How dependencies are resolved today
+
+When a pipeline is executed via CLI or API:
+
+1. The caller provides a bundle (as `mthds_content` or by referencing a file) and optionally a `pipe_code` (defaults to the bundle's `main_pipe`).
+
+2. `pipeline_run_setup()` loads **all** bundles from the provided library directories (`PIPELEXPATH`, `-L` flag, `library_dirs` parameter) and from installed methods (`.mthds/methods/` directories). This is a brute-force approach — everything available gets loaded.
+
+3. If `mthds_content` was provided, its blueprint is loaded on top.
+
+4. After loading, the library is validated: every pipe's dependencies (child pipes for controllers, concepts for inputs/outputs) must exist in the loaded library. If something is missing, validation fails.
+
+This "load everything, validate after" approach works but is wasteful. It loads pipes and concepts that will never be used, and it requires that all transitive dependencies be available on the file system at load time.
+
+---
+
+## 3. The LibraryCrate
+
+The LibraryCrate is a self-contained, serializable representation of the concepts and pipes needed for execution, organized by domain.
+
+### Structure
+
+```python
+class LibraryCrate(BaseModel):
+ """Complete library content, ready to load into a live Library."""
+ concepts: dict[str, ConceptBlueprint] # concept_ref (domain.Code) -> blueprint
+ pipes: dict[str, PipeBlueprintUnion] # pipe_ref (domain.pipe_code) -> blueprint
+ fingerprint: str # SHA256 of serialized content
+```
+
+The crate is flat. No domain-level grouping — the domain is implicit in the keys (`scoring.WeightedScore`, `scoring.compute_score`). This mirrors how the live library already works for concepts (keyed by `concept_ref`), and how it will work for pipes once the `pipe_ref` fix lands.
+
+### Properties
+
+- **Flat and ref-keyed**: Two dicts, keyed by fully-qualified refs. Domain is a namespace prefix, not a structural container.
+- **Source-agnostic**: No file paths, no bundle identity, no package provenance. The crate doesn't know (or care) where its blueprints came from.
+- **Serializable**: Pure Pydantic models. JSON round-trip is trivial. This is what makes it suitable for both in-process use and wire transfer.
+- **Fingerprinted**: A SHA256 digest of the serialized content enables caching — if the same crate has already been loaded, skip the work.
+
+### How it's built
+
+```python
+# 1. Parse bundles from files
+blueprints: list[PipelexBundleBlueprint] = [
+ PipelexInterpreter.make_pipelex_bundle_blueprint(bundle_path=path)
+ for path in mthds_file_paths
+]
+
+# 2. Build crate: qualify refs with domain, merge into flat dicts
+crate = LibraryCrateFactory.make_from_blueprints(blueprints)
+
+# 3. Load into live library
+library_manager.load_from_crate(library_id, crate)
+```
+
+Each bundle declares a domain (e.g., `scoring`). During crate construction, concept codes become `concept_ref`s (`Score` → `scoring.Score`) and pipe codes become `pipe_ref`s (`compute_score` → `scoring.compute_score`). Multiple bundles — even for the same domain — simply add entries to the flat dicts. If the same ref appears twice, that's a conflict — error, not silent override.
+
+### What blueprints contain
+
+**ConceptBlueprint** (`core/concepts/concept_blueprint.py`): `description`, `structure`, `refines`. Clean Pydantic BaseModel.
+
+**PipeBlueprintUnion** (`core/bundles/pipelex_bundle_blueprint.py`): Discriminated union of 10 pipe blueprint types (`PipeFuncBlueprint`, `PipeLLMBlueprint`, `PipeSequenceBlueprint`, etc.). All are Pydantic models with fields for inputs, output, type-specific configuration (prompts, steps, branches, etc.).
+
+These are already the "data form" of pipes and concepts. They parse cleanly from `.mthds` files, they serialize cleanly to JSON, and the existing factory methods (`ConceptFactory.make_from_blueprint()`, `PipeFactory.make_from_blueprint()`) know how to instantiate live objects from them.
+
+---
+
+## 4. Namespace Prerequisites
+
+Before building the LibraryCrate, a fundamental asymmetry in the library must be fixed.
+
+**Concepts are properly namespaced.** The `ConceptLibrary` indexes concepts by `concept_ref` = `domain.ConceptCode` (e.g., `scoring.WeightedScore`). Two concepts with the same code in different domains coexist without conflict.
+
+**Pipes are NOT namespaced.** The `PipeLibrary` indexes pipes by bare `code` (e.g., `compute_score`, not `scoring.compute_score`). If two domains define a pipe with the same code, the second one collides with the first — the library raises an error as if it were a duplicate, even though they're in different domains.
+
+This must be fixed by introducing `pipe_ref` = `domain.pipe_code` on `PipeAbstract` and rekeying `PipeLibrary` by `pipe_ref`, symmetric with how concepts work. The domain library must also support additive merging (multiple bundles contributing to the same domain).
+
+Once `pipe_ref` is in place, the flat LibraryCrate keying becomes natural — both concepts and pipes use fully-qualified refs as dict keys, and the domain is just the prefix before the dot.
+
+See [phase0-pipe-namespace-fix.md](phase0-pipe-namespace-fix.md) for the detailed technical spec.
+
+---
+
+## 5. Dependency Resolution Roadmap
+
+Dependency resolution improves in phases, from brute-force to precise.
+
+### Current state: Load everything
+
+`pipeline_run_setup()` loads all bundles from all provided library paths. Validation checks that dependencies exist, but nothing is selective. The full library travels to execution even if only a fraction is needed.
+
+### Near-term: Full crate, validated
+
+The LibraryCrate contains everything that was loaded — the full set of domains, concepts, and pipes. This is equivalent to today's behavior but with the crate as the clean intermediate. No stripping yet.
+
+### Planned: Static dependency tree analysis (crate stripping)
+
+From the entry pipe, walk the transitive closure:
+- For each pipe controller, collect its `pipe_dependencies()` (child pipe codes)
+- For each pipe (controller or operator), collect its `concept_dependencies` (input/output concepts)
+- Recursively resolve until the full dependency graph is known
+
+Then strip the crate to only the domains, concepts, and pipes in the dependency graph. This reduces payload sizes for distributed execution and clarifies exactly what context is needed.
+
+This is feasible because execution is deterministic — the dependency graph is fully known at build time.
+
+### Future: Remote dependency resolution
+
+Resolve dependencies from GitHub method package addresses. When a bundle declares a dependency on `github:org/repo`, clone to a temp directory, parse the bundles, and merge into the crate. This enables running pipelines whose dependencies aren't pre-installed locally.
+
+---
+
+## 6. Generalization Across Execution Modes
+
+The LibraryCrate is **not** a Temporal-specific construct. It is the universal intermediate representation for all execution modes.
+
+**Direct execution** (in-process):
+```
+Bundles → LibraryCrate → Library (live, in same process)
+```
+
+**Distributed execution** (Temporal):
+```
+Bundles → LibraryCrate → serialize → send to worker → deserialize → Library (live, on worker)
+```
+
+The only difference is that distributed execution adds a serialize/deserialize step. The crate itself is identical.
+
+This means we build and validate the LibraryCrate in direct mode first. Once it works — once all existing tests pass through the `bundles → crate → library` path — distributed execution gets serialization for free because the crate is already a Pydantic model.
+
+---
+
+## 7. External Storage for Large Payloads
+
+When payloads grow beyond Temporal's 2MB limit (large libraries, WorkingMemory with images/documents), external storage becomes necessary.
+
+Each pipeline execution has a **pipeline run ID** that can serve as the storage key:
+
+- Before dispatching to Temporal, upload the crate and working memory to a storage backend
+- Workers fetch by run ID, load, execute, write back results
+- Uses the existing `StorageProviderAbstract` system
+
+This is a later optimization (Phase 4 in the master plan), not part of the core LibraryCrate design. The crate's serialization is a prerequisite — you can't upload what you can't serialize — but the storage mechanism is independent.
+
+---
+
+## 8. Open Questions
+
+- **Caching by fingerprint**: If the same LibraryCrate (same fingerprint) is used across multiple pipeline runs, can workers cache the loaded library and skip re-loading? This is especially valuable for Temporal workers processing many runs with the same library.
+
+- **Storage backend choice**: S3-compatible? Redis for short-lived data? The `StorageProviderAbstract` makes this pluggable, but the default choice matters for developer experience.
+
+- **Domain `system_prompt` migration**: The bundle-level `system_prompt` currently flows into the `Domain` object and is used as a fallback by PipeLLM pipes that don't define their own. When we flatten the crate (no domain-level data), this fallback must be inlined at pipe factory time — each PipeLLM gets its system_prompt resolved during construction, before entering the crate. This is a small change in `PipeLLMFactory` but needs to be done as part of Phase 0 or Phase 1.
diff --git a/wip/archive/early-temporal-library-fix-proposals.md b/wip/archive/early-temporal-library-fix-proposals.md
new file mode 100644
index 000000000..4fa07fc3c
--- /dev/null
+++ b/wip/archive/early-temporal-library-fix-proposals.md
@@ -0,0 +1,441 @@
+# Architecture Discussion: Fixing Library Unavailability in Distributed Execution (v2)
+
+> **Status**: Draft v2 — for team architecture review
+> **Date**: 2026-03-22
+> **Context**: See [Pipe Routing & Execution](../under-the-hood/pipe-routing-and-execution.md) for how direct and distributed execution work. See [temporal-worker-problem.md](../../.claude/skills/temporal-diagnose/references/temporal-worker-problem.md) for the detailed root cause analysis and error patterns.
+> **Changes from v1**: Relaxed C1 (breaking changes acceptable), replaced naive `mthds_content: str` model with proper multi-bundle library context analysis, proposed `LibraryContext` model for API evolution.
+
+---
+
+## 1. Problem Statement
+
+When pipe controllers (PipeSequence, PipeCondition, PipeBatch, PipeParallel) run via **distributed execution** (Temporal), they fail because the library is never loaded on the worker process.
+
+```
+API Process Temporal Worker
+───────────────────────────────────────── ──────────────────────────────
+pipeline_run_setup()
+ ├─ Loads library (library_manager) (empty here)
+ ├─ Generates dynamic concept classes (classes don't exist here)
+ ├─ Registers them with Kajson (Kajson registry incomplete here)
+ ├─ Resolves pipe by code (can't resolve here)
+ └─ Creates PipeJob (top-level pipe)
+ │
+ └─ PipeRouterTop sends PipeJob ──────► WfPipeRouter.run(pipe_job)
+ to Temporal ├─ Kajson deserializes PipeJob
+ │ └─ FAILS (Layer 1): unknown class
+ └─ pipe.run_pipe()
+ └─ get_required_pipe() FAILS (Layer 2)
+```
+
+### Layer 1 -- Deserialization Failure (hits first)
+
+The `PipeJob`'s `WorkingMemory` contains `Stuff` objects with dynamically-generated concept content classes (e.g., `RawText` inheriting from `TextContent`). These classes are created by `ConceptFactory` during library loading and registered with Kajson's class registry. On the worker, these classes don't exist -> `KajsonDecoderError: Class 'RawText' not found in module 'builtins'`.
+
+This is a **hard blocker**: Temporal's workflow instance cannot even deserialize the workflow input.
+
+### Layer 2 -- Pipe Resolution Failure (would hit after Layer 1 is fixed)
+
+Controllers call `get_required_pipe(child_pipe_code)` to resolve child pipes at runtime. This queries the `library_manager` singleton, which is empty on the worker -> pipe not found.
+
+### Why Tests Don't Catch It
+
+- Integration tests use `PipeRouter` (direct, in-process) -- library is shared
+- Temporal tests only cover leaf workflows (text gen, jinja2) that don't call `get_required_pipe()`
+- No test sends a pipe controller through `WfPipeRouter`
+
+### Key Insight
+
+Both layers are fixed by the same fundamental action: **loading the library on the worker**. Library loading generates and registers dynamic classes (fixing Layer 1) AND populates `library_manager` with pipe definitions (fixing Layer 2).
+
+---
+
+## 2. Constraints
+
+Any solution must respect these constraints:
+
+| # | Constraint | Rationale |
+|---|-----------|-----------|
+| C1 | **Minimize disruption to direct execution** | Additive changes (optional params, new methods) are welcome. Breaking changes are acceptable if justified -- discuss damage control before implementation, but retro-compatibility is not required. |
+| C2 | **Temporal replay safety** | Workflow code must be deterministic. Side effects (file I/O, library loading) belong in Activities, not workflow `run()` methods. Activities re-execute cleanly on replay. |
+| C3 | **Worker reuse** | A single worker processes jobs from different API calls, potentially with different library contexts. The solution must handle this without conflict. |
+| C4 | **Dynamic class generation is unavoidable** | Concepts declared in `.mthds` files create Python classes at library-load time. These must exist before Kajson deserialization. |
+| C5 | **Library context is multi-layered** | See section 2.1. The library is not a single string -- it's a multi-file, multi-package system with manifests, dependencies, and namespace isolation. Any solution must account for this. |
+| C6 | **Library loading is I/O-bound** | Reads files, parses TOML, generates classes, validates. Not suitable for workflow code (violates C2). |
+| C7 | **Temporal payload size** | Default limit is 2MB (configurable). Solutions that embed large payloads must account for this. |
+| C8 | **Reproduce from sources, not snapshots** | The solution should reproduce library state from source inputs (file contents, manifests, dependency metadata), not serialize the resolved `Library` object graph (which includes dependency libraries, class registries, and dynamically generated Python types). |
+
+### 2.1 Understanding the Library Data Model
+
+The current proposals v1 treated `mthds_content` as a single string that could carry "the library context." This is fundamentally incorrect. MTHDS is a **multi-bundle, multi-package** system:
+
+**A package** is a directory containing:
+
+- A `METHODS.toml` manifest (identity, exports, dependencies)
+- One or more `.mthds` bundle files (each declaring one domain)
+- Bundles within a package sharing the same domain code **merge** their namespaces
+
+**Cross-package dependencies** are declared in `METHODS.toml`:
+
+```toml
+[dependencies]
+scoring_lib = { address = "github.com/mthds/scoring-lib", version = "^0.5.0" }
+```
+
+Pipes reference cross-package pipes via alias syntax: `scoring_lib->scoring.compute_weighted_score`. Only exported pipes (declared in `[exports]`) are visible across package boundaries.
+
+**How library loading actually works** (`library_manager.load_libraries()`):
+
+1. Discovers all `.mthds` files from directory paths
+2. Walks up to find `METHODS.toml` manifests for each file
+3. Resolves dependencies (local path + address-based remote fetch)
+4. Enforces visibility/exports between packages
+5. Loads all blueprints into a `Library` with `dependency_libraries: dict[str, Library]`
+
+**The current `mthds_content` parameter** in `pipeline_run_setup()` only handles a single string -> single `PipelexBundleBlueprint`. This works for simple single-domain bundles but cannot represent multi-file packages with manifests and cross-package dependencies. This is a known limitation of the current API surface, not the target architecture.
+
+**Two loading paths exist today:**
+
+| Path | Input | Handles multi-file? | Handles deps? | Used by |
+|------|-------|---------------------|---------------|---------|
+| Directory-based | `library_dirs` (filesystem paths) | Yes | Yes | PIPELEXPATH, installed methods |
+| Content-based | `mthds_content` (single string) | No | No | API single-file bundles |
+
+The Temporal solution must work with both paths and must not bake in the assumption that all library context fits in a single string.
+
+---
+
+## 3. Solution Options
+
+### Option A: Worker Startup + Per-Workflow Library Context
+
+**Summary**: Workers load base libraries from `PIPELEXPATH` at startup (Tier 1). Per-workflow library context -- whether a single bundle or a full package -- travels with the workflow input and is loaded in an Activity before the main pipe executes (Tier 2, cached by content hash).
+
+**How it fixes each layer**:
+
+- **Layer 1**: Base library loading at worker startup generates and registers all base concept classes with Kajson. For additional content, a "library setup" Activity runs before the pipe, parsing the content and registering dynamic classes.
+- **Layer 2**: Base pipes populated at startup. Additional pipes loaded in the setup Activity. `get_required_pipe()` finds everything.
+
+**Tier 1 -- Base Library at Worker Startup (already implemented)**:
+
+`worker_cli.py` already loads base libraries after `Pipelex.make()`:
+
+```python
+# In configure(), after Pipelex.make(temporal_enabled=True):
+library_manager = get_library_manager()
+library_manager.open_library(library_id="worker_base")
+effective_dirs, _ = resolve_library_dirs(library_dirs=None)
+if effective_dirs:
+ library_manager.load_libraries(library_id="worker_base", library_dirs=effective_dirs)
+```
+
+This generates and registers all base concept classes with Kajson, and populates the pipe library. Fixes both layers for workflows that only use base pipes/concepts (the majority of cases).
+
+**Tier 2 -- Per-Workflow Library Context**:
+
+For workflows with additional library content beyond the base, a `LibraryContext` model travels with the workflow:
+
+```python
+class LibraryContext(BaseModel):
+ """Describes library content that must be loaded on the worker before execution."""
+ bundles: list[BundleContent] # One or more .mthds files
+ manifest_content: str | None # METHODS.toml content, if present
+ resolved_dependencies: list[ResolvedDependencyRef] | None # pre-resolved dep refs
+ fingerprint: str # SHA256 hash for caching
+```
+
+This replaces the naive `mthds_content: str | None` model. For the current single-file API case, it wraps the string in `bundles` with one entry. For future multi-file support, it carries the full package context.
+
+**Changes required**:
+
+- ~~`worker_cli.py`: Add library loading~~ (already done)
+- New `LibraryContext` model in `pipelex/temporal/`
+- `PipeRouterTop`: Build `LibraryContext` from the current library state when dispatching
+- New `act_library_setup.py`: Activity that loads `LibraryContext` into the library (cached by fingerprint)
+- `WfPipeRouter`: Call the setup Activity before `pipe.run_pipe()`
+
+**Impact on direct execution**: None. `pipeline_run_setup()` and `PipeRouter` unchanged.
+
+**Replay safety**: Library loading happens in Activities (replay-safe). The Activity is idempotent -- loading the same fingerprint twice is a no-op.
+
+**Critical issue**: Layer 1 has a timing problem. Temporal automatically deserializes the PipeJob (via the data converter) *before* the workflow code runs. If the PipeJob contains custom concept instances, deserialization fails before the setup Activity can register the classes. This means **Option A alone cannot fix Layer 1 for content beyond the base library**. See Option D for the solution.
+
+---
+
+### Option B: Flatten All Pipes into PipeJob (Self-Contained Job)
+
+**Summary**: At `pipeline_run_setup()` time, resolve the entire pipe tree and embed all referenced pipes directly into the PipeJob. The worker needs no library for pipe resolution.
+
+**How it fixes each layer**:
+
+- **Layer 1**: Does NOT fix. The `WorkingMemory` still contains `Stuff` objects with dynamic concept classes. Flattening pipes doesn't address deserialization of concept content classes.
+- **Layer 2**: Fixed. `get_required_pipe()` calls replaced by lookups into a `resolved_pipes` map on the PipeJob.
+
+**Changes required**:
+
+- `PipeJob`: Add `resolved_pipes: dict[str, PipeAbstract]`
+- All pipe controllers: Refactor to accept a pipe resolver instead of calling `get_required_pipe()` from the hub
+- `pipeline_run_setup()`: Recursively resolve all child pipes and attach to PipeJob
+
+**Impact on direct execution**: Significant. Pipe controllers currently call `get_required_pipe()` globally. Changing them to use a local resolver changes the interface everywhere, adding complexity even in direct mode. With relaxed C1, this is *allowed* but the cost is disproportionate.
+
+**Replay safety**: Excellent. Everything needed is in the workflow input.
+
+**Performance**: Larger PipeJob payloads. Redundant data if pipes are shared across workflows.
+
+**Complexity**: High. Requires refactoring all pipe controllers.
+
+**Verdict**: Fixes only Layer 2, high complexity. Not recommended as standalone approach. The key insight is: once the library is loaded on the worker (which is necessary for Layer 1 regardless), pipe resolution works identically to direct execution -- making the flattening effort unnecessary.
+
+---
+
+### Option C: Ship Full Library State per Workflow
+
+**Summary**: Serialize the entire library (domains, concepts, pipes, class definitions) and include it in every workflow invocation. The worker reconstructs the library from this payload.
+
+**How it fixes each layer**:
+
+- **Layer 1**: Include concept structure blueprints in the payload. A pre-deserialization step regenerates dynamic classes.
+- **Layer 2**: Full pipe library reconstructed from serialized state.
+
+**Important distinction**: There are two sub-approaches here:
+
+- **C-snapshot**: Serialize the resolved `Library` object graph. This is impractical -- `Library` includes `dependency_libraries`, dynamically generated Python types, Kajson registry state, etc. Violates C8.
+- **C-sources**: Ship the source inputs (file contents, manifests, dependency metadata) and reproduce the library on the worker. This is essentially what Option A Tier 2 does with `LibraryContext`.
+
+**Verdict for C-snapshot**: Over-engineered, poor performance, every workflow carries the entire library, risk of hitting payload size limits (C7), violates C8. Not recommended.
+
+**Verdict for C-sources**: This is Option A Tier 2 by another name. The right principle ("reproduce from source inputs") is incorporated into the recommended approach.
+
+---
+
+### Option D: Deferred Deserialization
+
+**Summary**: Instead of letting Temporal automatically deserialize the PipeJob, the workflow receives a raw payload (bytes or a lightweight wrapper). A library setup Activity runs first to load the library and register dynamic classes. Then the workflow manually deserializes the PipeJob via `kajson.loads()`.
+
+This is designed to solve Option A's critical timing issue with Layer 1.
+
+**How it fixes each layer**:
+
+- **Layer 1**: Library is loaded (including dynamic class registration) *before* deserialization, so all classes exist when `kajson.loads()` runs.
+- **Layer 2**: Same as Option A -- library is loaded.
+
+**When is this needed?** Only when the workflow introduces dynamic classes not already in the worker's base library. For the common case (PIPELEXPATH-based libraries loaded at worker startup), Temporal's normal deserialization works fine because all classes are registered at startup.
+
+**Changes required**:
+
+- `PipeRouterTop`: Serialize PipeJob to raw JSON bytes before dispatching
+- New envelope model:
+
+```python
+class TemporalPipeJobEnvelope(BaseModel):
+ """Lightweight wrapper that Temporal can deserialize without dynamic classes."""
+ library_context: LibraryContext | None # None = use worker's base library
+ library_dirs_fingerprint: str | None # Verify worker's base matches API's
+ raw_pipe_job_payload: bytes # Pre-serialized PipeJob JSON
+```
+
+- `WfPipeRouter`: Accept `TemporalPipeJobEnvelope` instead of `PipeJob`. Call setup Activity if `library_context` is present, then manually `kajson.loads()` the raw payload.
+- Temporal data converter: No changes -- the envelope is a simple model that deserializes without dynamic classes
+
+**Impact on direct execution**: None. Only touches Temporal layer code.
+
+**Replay safety**: Same as Option A -- Activity-based.
+
+**Performance**: Minimal overhead -- one extra serialization/deserialization step.
+
+**Complexity**: Moderate. The main conceptual shift is that the workflow input type changes from `PipeJob` to `TemporalPipeJobEnvelope`.
+
+---
+
+## 4. Key Dilemmas
+
+### 4.1 Where should library loading happen?
+
+| Location | Pros | Cons |
+|----------|------|------|
+| **Worker startup** (for base library) | Amortized cost, guaranteed availability, simple | Worker restart needed if base library changes. Only handles filesystem-available content. |
+| **Per-workflow Activity** (for additional content) | Guarantees correct version, handles per-request content | Adds latency per workflow (mitigated by caching) |
+| **Hybrid** (both) | Best of both worlds | Two loading paths to maintain |
+| **Shared filesystem** (workers mount same PIPELEXPATH) | Zero additional work -- Tier 1 handles everything | Requires deployment architecture where workers access same dirs |
+
+**Recommendation**: Hybrid, with shared filesystem as the primary deployment model. Base at startup, additional content per-workflow-with-caching. In practice, most deployments will share PIPELEXPATH, making Tier 1 sufficient for the vast majority of cases.
+
+### 4.2 Should PipeJob carry resolved pipes or just pipe codes?
+
+| Approach | Pros | Cons |
+|----------|------|------|
+| **Current** (resolved top-level only) | Minimal payload | Layer 2 breaks for children |
+| **All resolved** (flattened) | Self-contained, no library needed on worker | Bloats payload, forces refactoring of controllers |
+| **Codes only + worker-side resolution** | Cleanest -- worker resolves exactly like direct mode | Requires library on worker (but we need that for Layer 1 anyway) |
+
+**Recommendation**: Codes only + worker-side resolution. Once the library is loaded on the worker (which is necessary for Layer 1 regardless), pipe resolution works identically to direct execution. No changes to controllers or the PipeJob model.
+
+### 4.3 Dynamic class registration timing
+
+This is the crux of the Layer 1 problem:
+
+```
+Temporal automatically deserializes workflow input
+ +-- Uses data converter -> kajson.loads()
+ +-- Needs dynamic classes registered BEFORE this point
+ +-- But workflow code hasn't run yet -> can't load library first
+```
+
+Three approaches:
+
+| Approach | How | Trade-off |
+|----------|-----|-----------|
+| **Pre-load at worker level** | Load base library at worker startup | Works for base concepts only. Additional content's concepts aren't available. |
+| **Deferred deserialization** (Option D) | Send raw bytes, deserialize manually after library setup | Changes workflow input type. Small conceptual overhead. |
+| **Lazy class resolution in Kajson** | Modify Kajson decoder to load classes on demand | Invasive change to a shared library. Complex. |
+
+**Recommendation**: Pre-load at worker level for base concepts (covers the majority of cases) + deferred deserialization for additional content (Option D handles it cleanly).
+
+### 4.4 Caching strategy
+
+Caching is only relevant for Tier 2 (per-workflow library context). For Tier 1 (PIPELEXPATH), the library is loaded once at worker startup and persists for the worker's lifetime.
+
+| Strategy | Cache key | Scope | Trade-off |
+|----------|-----------|-------|-----------|
+| **No caching** | -- | Always reload | Simple but wasteful |
+| **Content-hash** | `SHA256(library_context)` | Per unique context | Handles same content from different API calls |
+| **Library-ID** | `library_id` parameter | Per library instance | Uses existing `library_manager` multi-library support |
+
+**Recommendation**: Content-hash caching for Tier 2 overlays. The `library_manager` already supports multiple concurrent libraries keyed by `library_id`. Map content hashes to library IDs. The Activity checks: "is this fingerprint already loaded?" -> skip if yes.
+
+### 4.5 What if different workflows need different library contexts?
+
+The `library_manager` already supports multiple concurrent libraries, and `set_current_library()` uses a `ContextVar` so different async tasks can have different current libraries. On the worker, each workflow's setup Activity would set the appropriate library ID before executing.
+
+**Risk**: Dynamic class names could collide if two different contexts define the same concept code with different structures. Mitigation: prefix class names with a content hash, or validate that same-named concepts have identical structures.
+
+**Assessment**: This scenario only arises with API-provided content (Tier 2). For filesystem-based libraries (Tier 1), all workers load the same content. Worth noting but not a design blocker.
+
+### 4.6 How should the API evolve for multi-file packages?
+
+The current `mthds_content: str | None` parameter in `pipeline_run_setup()` handles only a single `.mthds` file. Real packages are multi-file with manifests and dependencies. The API needs to evolve.
+
+**Proposed model -- `LibraryContext`**:
+
+```python
+class BundleContent(BaseModel):
+ """A single .mthds bundle file's content with its filename for domain identification."""
+ filename: str # e.g., "contracts.mthds"
+ content: str # The MTHDS TOML content
+
+class ResolvedDependencyRef(BaseModel):
+ """A pre-resolved dependency reference (address + version already resolved)."""
+ alias: str # The dependency alias in METHODS.toml
+ address: str # Resolved package address
+ version: str # Resolved version
+
+class LibraryContext(BaseModel):
+ """Complete library context that can be shipped to a worker."""
+ bundles: list[BundleContent] # One or more .mthds files
+ manifest_content: str | None = None # METHODS.toml content
+ resolved_dependencies: list[ResolvedDependencyRef] | None = None # Pre-resolved deps
+ fingerprint: str # Hash for caching
+```
+
+**How `pipeline_run_setup()` would consume it**:
+
+The existing `mthds_content: str | None` parameter could be supplemented or replaced with a `library_context: LibraryContext | None` parameter. For backward compatibility, `mthds_content` could be sugar for a single-bundle `LibraryContext`.
+
+**How it flows to Temporal**:
+
+The `TemporalPipeJobEnvelope` carries `library_context`. The worker's `act_library_setup` Activity:
+
+1. Checks `fingerprint` against cache -- skip if already loaded
+2. Parses each bundle in `bundles` into a `PipelexBundleBlueprint`
+3. If `manifest_content` is present, parses it and enforces visibility
+4. If `resolved_dependencies` is present, loads dependency packages from installed methods (or fetches by address)
+5. Calls `library_manager.load_from_blueprints()` with all blueprints
+
+**What about dependencies that need fetching?**
+
+For production, dependency packages should be pre-installed on the worker (via `mthds install` or shared filesystem). The `resolved_dependencies` field provides the addresses for the worker to locate them. If a dependency is not installed, the Activity can attempt to fetch it (same as `library_manager` does today for address-based deps), but this adds latency and requires network access from the worker.
+
+**Recommendation**: Start with the `LibraryContext` model. The initial implementation can focus on the single-bundle case (wrapping current `mthds_content`). Multi-bundle support is a clear extension path without architectural changes.
+
+---
+
+## 5. Recommendation
+
+**Recommended approach: Option A (base at startup) + Option D (deferred deserialization with `LibraryContext`)**
+
+### Tier 1 -- Base Library at Worker Startup (done)
+
+Already implemented in `worker_cli.py`. Workers load base libraries from `PIPELEXPATH` on startup. This fixes both layers for the majority of use cases where pipes and concepts come from filesystem-based packages.
+
+### Tier 2 -- Deferred Deserialization with LibraryContext
+
+For workflows with library content beyond the base:
+
+1. `PipeRouterTop` builds a `LibraryContext` from the current library state (capturing what was loaded beyond PIPELEXPATH)
+2. Serializes PipeJob to raw bytes
+3. Wraps both in a `TemporalPipeJobEnvelope`
+4. `WfPipeRouter` receives the envelope, runs `act_library_setup` Activity if `library_context` is present
+5. Manually deserializes PipeJob via `kajson.loads()` after library is loaded
+
+### Implementation Sequence
+
+1. **(Done)** Tier 1 -- base library loading in `worker_cli.py`
+2. **Integration test** -- Send a PipeSequence through Temporal with PIPELEXPATH-based library to validate Tier 1
+3. **Design `LibraryContext` model** -- Start with single-bundle wrapper for current `mthds_content` behavior
+4. **Implement deferred deserialization** (Option D) -- `TemporalPipeJobEnvelope`, `act_library_setup`, `WfPipeRouter` changes
+5. **Library fingerprint validation** -- Worker checks its base library matches API's expectation
+6. **Multi-bundle support** -- Extend `LibraryContext` to carry multiple bundles + manifest
+
+### Why Not the Others
+
+| Option | Why not |
+|--------|---------|
+| B (flatten pipes) | Doesn't fix Layer 1. High refactoring cost for marginal benefit -- once library is loaded (required for Layer 1 anyway), pipe resolution works for free. |
+| C (ship library snapshot) | Serializing resolved `Library` is impractical. The right version of this idea (reproduce from sources) is what `LibraryContext` does. |
+
+---
+
+## 6. Appendix
+
+### PipeJob Model
+
+```python
+class PipeJob(BaseModel):
+ pipe: PipeAbstract # The resolved pipe object
+ working_memory: WorkingMemory # Runtime data store
+ pipe_run_params: PipeRunParams # Run mode, multiplicity, pipe stack
+ job_metadata: JobMetadata # Pipeline run ID, user ID, OTel context
+ output_name: str | None = None # Override output variable name
+```
+
+PipeJob stays unchanged -- no `mthds_content` or library context added to it. Library context travels in the `TemporalPipeJobEnvelope` wrapper, separate from the job itself.
+
+### get_required_pipe() Call Sites in Controllers
+
+| Controller | File | Usage |
+|------------|------|-------|
+| PipeSequence | `pipe_controllers/sequence/pipe_sequence.py` | Resolves each `sub_pipe.pipe_code` in the sequence |
+| PipeCondition | `pipe_controllers/condition/pipe_condition.py` | Resolves the selected branch pipe from `outcome_map` |
+| PipeBatch | `pipe_controllers/batch/pipe_batch.py` | Resolves `branch_pipe_code` for each batch item |
+| PipeParallel | `pipe_controllers/parallel/pipe_parallel.py` | Resolves each parallel branch pipe |
+| SubPipe | `pipe_controllers/sub_pipe.py` | Used during validation |
+
+### Key Code Paths
+
+| What | Where |
+|------|-------|
+| Library loading (API-side) | `pipelex/pipeline/pipeline_run_setup.py` |
+| Hub singleton + get_required_pipe | `pipelex/hub.py` |
+| Library manager | `pipelex/libraries/library_manager.py` |
+| Dynamic concept class generation | `pipelex/core/concepts/concept_factory.py` |
+| Structure generator (creates classes) | `pipelex/core/concepts/structure_generation/generator.py` |
+| Kajson class registration | `pipelex/pipelex.py` (CoreRegistryModels) + `concept_factory.py` (dynamic) |
+| Kajson data converter (Temporal serde) | `pipelex/temporal/temporal_data_converter.py` |
+| Workflow definition | `pipelex/temporal/tprl_pipe/wf_pipe_router.py` |
+| Router (Temporal) | `pipelex/temporal/tprl_pipe/pipe_router_top.py` |
+| Router (direct) | `pipelex/pipe_run/pipe_router.py` |
+| Worker startup (Tier 1 -- already implemented) | `pipelex/temporal/worker_cli.py` |
+| MTHDS package manifest spec | `../mthds/docs/spec/manifest-format.md` |
+| Namespace resolution spec | `../mthds/docs/spec/namespace-resolution.md` |
+| Package loading algorithm | `../mthds/docs/implementers/package-loading.md` |
diff --git a/wip/archive/kajson-decoder-class-registry.md b/wip/archive/kajson-decoder-class-registry.md
new file mode 100644
index 000000000..c262e0c9e
--- /dev/null
+++ b/wip/archive/kajson-decoder-class-registry.md
@@ -0,0 +1,63 @@
+# Kajson Decoder: ClassRegistry Fallback Not Reached for Dynamic Classes
+
+## Problem
+
+The Kajson JSON decoder (`kajson/json_decoder.py:140-153`) resolves classes in three steps:
+
+1. **Line 140-150**: If `module_name` is in `sys.modules`, use `getattr(sys.modules[module_name], class_name)`. If not found, raise `KajsonDecoderError`.
+2. **Line 151-153**: Else, try `KajsonManager.get_class_registry().get_class(name=class_name)`.
+3. **Line 155+**: Else, try `importlib.import_module(module_name)` and `getattr`.
+
+The ClassRegistry lookup (step 2) only runs when the module is NOT in `sys.modules`. Dynamic classes generated at runtime (e.g., `GreetingContent` from inline concept structures) are registered in the ClassRegistry with `__module__` set to `"builtins"`. Since `builtins` is always in `sys.modules`, the decoder takes step 1, does `getattr(builtins, "Greeting")`, fails, and raises at line 150 — **never reaching the ClassRegistry fallback at line 151**.
+
+## How It Manifests
+
+In a Temporal PipeSequence with dynamic concepts:
+
+1. Parent workflow dispatches a child workflow (e.g., `generate_greeting` PipeLLM)
+2. Child workflow executes successfully, produces `PipeOutput` with `Greeting` StructuredContent
+3. Child result is serialized by Kajson with `__class__: "Greeting"`, `__module__: "builtins"`
+4. Parent workflow receives the child result — Kajson decoder does `getattr(sys.modules["builtins"], "Greeting")` → `AttributeError` → `KajsonDecoderError` at line 150
+5. The ClassRegistry (line 151), which DOES have the `Greeting` class registered, is never consulted
+
+## Why In-Process Tests Pass
+
+The integration test runs the worker in-process (`is_not_sandboxed=True`). The test fixture pre-loads the blueprint globally, which registers the dynamic class in the ClassRegistry. Since the class was generated in the same process, `StructureGenerator` may also place it in a module that makes `getattr` succeed. With a **separate** worker process, the dynamic class is only in the ClassRegistry (loaded from the crate), not in any importable module attribute.
+
+## Verified via Manual Test
+
+Using `/temporal-diagnose` with a 3-process setup (Temporal server + separate worker + job submitter), the `dynamic_concept_sequence.mthds` bundle fails with:
+
+```
+KajsonDecoderError: Class 'Greeting' not found in module 'builtins'
+```
+
+The worker logs show the child workflow result deserialization failing at `_convert_payloads` in the Temporal SDK, which calls `kajson.loads()`.
+
+## Proposed Solution
+
+In `kajson/json_decoder.py`, when the module IS in `sys.modules` but `getattr` fails (line 149-150), fall back to the ClassRegistry before raising:
+
+```python
+# Current (line 149-150):
+else:
+ raise KajsonDecoderError(f"Class '{class_name}' not found in module '{module_name}'")
+
+# Proposed:
+else:
+ # Module is loaded but class not found as attribute — try ClassRegistry
+ # (handles dynamic classes registered at runtime, including scoped registries)
+ registered_class = KajsonManager.get_class_registry().get_class(name=class_name)
+ if registered_class is not None:
+ the_class = registered_class
+ else:
+ raise KajsonDecoderError(f"Class '{class_name}' not found in module '{module_name}' or ClassRegistry")
+```
+
+This preserves the existing priority (module attribute first, registry second) and only changes behavior when `getattr` fails for a non-generic class.
+
+## Dependencies
+
+- Scoped registry teardown (token-based, see `wip/scoped-registry-teardown.md`) should be implemented first, so the decoder uses the correct scoped registry in concurrent workflow scenarios
+- The Kajson decoder change should be tested with both global and scoped registry scenarios
+- The `hub.get_class_registry()` → `KajsonManager.get_class_registry()` change (already applied in `pipelex/hub.py`) is complementary — it fixes explicit lookups (e.g., `StuffContentFactory`) while this fix covers Kajson deserialization
diff --git a/wip/archive/phase0-pipe-namespace-fix.md b/wip/archive/phase0-pipe-namespace-fix.md
new file mode 100644
index 000000000..1befe4d18
--- /dev/null
+++ b/wip/archive/phase0-pipe-namespace-fix.md
@@ -0,0 +1,195 @@
+# Pipe Namespace Fix: Domain-Qualified `pipe_ref`
+
+> **Status**: Technical spec — Phase 0 of the master plan
+> **Date**: 2026-03-23
+> **Related**: [00-master-plan.md](00-master-plan.md), [archive/early-library-as-execution-context.md](archive/early-library-as-execution-context.md)
+
+---
+
+## 1. Problem
+
+Pipes and concepts have asymmetric namespace behavior in the library.
+
+**Concepts** are indexed by `concept_ref` = `domain.ConceptCode` (e.g., `scoring.WeightedScore`). Two concepts with the same code in different domains coexist without conflict. This is correct.
+
+**Pipes** are indexed by bare `code` (e.g., `compute_score`, not `scoring.compute_score`). If two domains define a pipe with the same code, the second collides with the first.
+
+### Where the asymmetry lives
+
+| Component | Concept key | Pipe key |
+|-----------|-------------|----------|
+| `ConceptLibrary.root` | `concept_ref` = `domain.Code` | — |
+| `PipeLibrary.root` | — | bare `code` |
+| `add_new_concept()` | checks `concept.concept_ref` | — |
+| `add_new_pipe()` | — | checks `pipe.code` |
+| `get_optional_concept()` | looks up by `concept_ref` | — |
+| `get_optional_pipe()` | looks up by bare `code`, domain validation is a fallback | — |
+
+### Code references
+
+- `PipeLibrary.add_new_pipe()` at `pipe_library.py:39-48`: keys by `pipe.code`
+- `PipeLibrary.get_optional_pipe()` at `pipe_library.py:56-82`: primary lookup by bare code, domain-qualified lookup is a fallback that still resolves to bare code internally
+- `ConceptLibrary.add_new_concept()` at `concept_library.py:81-85`: keys by `concept.concept_ref`
+- `library_manager.py:378`: `pipe_source_in_this_load` tracks by bare `pipe_code`
+- `library_manager.py:402`: `_pipe_source_map` tracks by bare `pipe_code`
+
+### Additional issue: Domain merging
+
+`DomainLibrary.add_domain()` at `domain_library.py:32-37` raises if the domain already exists. This prevents multiple bundles from contributing to the same domain — a legitimate use case (e.g., `scoring_core.mthds` and `scoring_advanced.mthds` both declaring `domain: scoring`).
+
+---
+
+## 2. The Fix
+
+### 2.1 Add `pipe_ref` to `PipeAbstract`
+
+```python
+# pipelex/core/pipes/pipe_abstract.py
+class PipeAbstract(ABC, BaseModel):
+ code: str
+ domain_code: str
+ # ...
+
+ @property
+ def pipe_ref(self) -> str:
+ """Domain-qualified pipe reference, e.g. 'scoring.compute_score'."""
+ return f"{self.domain_code}.{self.code}"
+```
+
+This mirrors `concept_ref` on `Concept` (which is `domain.ConceptCode`).
+
+### 2.2 Rekey `PipeLibrary`
+
+```python
+# pipelex/libraries/pipe/pipe_library.py
+PipeLibraryRoot = dict[str, PipeAbstract] # key changes from code to pipe_ref
+
+def add_new_pipe(self, pipe: PipeAbstract):
+ if pipe.pipe_ref in self.root:
+ msg = f"Pipe '{pipe.pipe_ref}' already exists in the library. ..."
+ raise PipeLibraryError(msg)
+ self.root[pipe.pipe_ref] = pipe
+```
+
+### 2.3 Update lookup methods
+
+`get_optional_pipe()` needs to handle three lookup forms:
+
+1. **Domain-qualified** (`scoring.compute_score`): Direct lookup by `pipe_ref` — this becomes the primary path
+2. **Bare code** (`compute_score`): Search across all domains. If exactly one match, return it. If multiple matches (same code in different domains), raise ambiguity error. If zero matches, return None.
+3. **Cross-package** (`alias->scoring.compute_score` or `alias->compute_score`): Existing aliased lookup, adjusted for `pipe_ref` keying
+
+```python
+def get_optional_pipe(self, pipe_code: str) -> PipeAbstract | None:
+ # 1. Direct lookup (works for pipe_ref and cross-package keys)
+ pipe = self.root.get(pipe_code)
+ if pipe is not None:
+ return pipe
+
+ # 2. Cross-package refs
+ if QualifiedRef.has_cross_package_prefix(pipe_code):
+ alias, remainder = QualifiedRef.split_cross_package_ref(pipe_code)
+ # Try domain-qualified remainder
+ pipe = self.root.get(f"{alias}->{remainder}")
+ if pipe is not None:
+ return pipe
+ # Try bare code remainder — search aliased entries
+ if "." not in remainder:
+ # ... search alias->domain.code entries
+ pass
+ return None
+
+ # 3. Bare code — search across domains
+ if "." not in pipe_code:
+ matches = [p for p in self.root.values() if p.code == pipe_code]
+ if len(matches) == 1:
+ return matches[0]
+ if len(matches) > 1:
+ domains = [p.domain_code for p in matches]
+ msg = f"Ambiguous pipe code '{pipe_code}' found in domains: {domains}. Use domain-qualified ref."
+ raise PipeLibraryError(msg)
+ return None
+
+ return None
+```
+
+### 2.4 Update `library_manager.py`
+
+In `load_from_blueprints()`:
+- `pipe_source_in_this_load` should key by `pipe_ref` (= `f"{blueprint.domain}.{pipe_code}"`)
+- `_pipe_source_map` should key by `pipe_ref`
+
+### 2.5 Fix domain merging
+
+Domains are namespace metadata — they carry `code`, `description`, and `system_prompt`. Since the LibraryCrate is flat (no domain-level grouping), domains exist only in the live `Library` as lightweight metadata objects extracted from the refs.
+
+`DomainLibrary.add_domain()` should be idempotent for re-adds of the same domain:
+
+```python
+def add_domain(self, domain: Domain):
+ domain_code = domain.code
+ if domain_code in self.root:
+ # Same domain from another bundle — idempotent (Domain is just namespace metadata)
+ return
+ self.root[domain_code] = domain
+```
+
+Since domains carry no meaningful runtime state beyond the namespace itself, simple idempotency is sufficient. The `system_prompt` field on Domain is a legacy convenience (PipeLLM fallback) that should be inlined at pipe factory time before crate construction.
+
+---
+
+## 3. Migration: Call Sites to Update
+
+Every place that passes a bare pipe code as a lookup key needs to be audited. Most will continue to work because the bare-code fallback handles unambiguous lookups. But some sites may need to pass domain-qualified refs.
+
+### Direct `get_required_pipe()` / `get_optional_pipe()` callers
+
+These are the 41 files identified in the investigation. Key categories:
+
+- **Pipe controllers** (`sub_pipe.py`, `pipe_sequence.py`, `pipe_batch.py`, `pipe_parallel.py`, `pipe_condition.py`): These call `get_required_pipe()` with the sub-pipe code from the blueprint. Currently bare codes. If the pipe is in the same domain, the bare code lookup will find it (unambiguous). If cross-domain, the blueprint already uses domain-qualified or cross-package refs.
+
+- **Pipeline setup** (`pipeline_run_setup.py`, `pipeline_manager.py`): Calls `get_required_pipe(pipe_code=pipe_code)` where `pipe_code` comes from user input or `main_pipe`. User input may be bare or qualified — the fallback handles both.
+
+- **Hub helpers** (`hub.py`): Thin wrappers, no change needed beyond forwarding.
+
+- **Temporal routers** (`pipe_router_top.py`, `pipe_router_child.py`): Same pattern as pipeline setup.
+
+- **CLI commands**: Various show/validate/run commands that resolve pipes by code. Should work via fallback.
+
+### `_pipe_source_map` consumers
+
+`library_manager.py` uses `_pipe_source_map` to track which file a pipe came from. Must rekey by `pipe_ref`. Check `get_pipe_source()` method.
+
+### Tests
+
+Existing test helpers that create pipes and add them to libraries will need to provide `domain_code` for proper `pipe_ref` generation. Check all test fixtures and factories.
+
+---
+
+## 4. Test Plan
+
+### Unit tests for `PipeLibrary`
+
+1. **Multi-domain coexistence**: Create two pipes with same code (`compute_score`) in different domains (`scoring`, `analytics`). Add both to same `PipeLibrary`. Both exist and are retrievable by `pipe_ref`.
+
+2. **Domain-qualified lookup**: Look up `scoring.compute_score` → returns scoring domain's pipe. Look up `analytics.compute_score` → returns analytics domain's pipe.
+
+3. **Bare-code unambiguous**: When only one pipe with code `compute_score` exists, bare lookup `compute_score` returns it.
+
+4. **Bare-code ambiguous**: When two domains both have `compute_score`, bare lookup `compute_score` raises `PipeLibraryError` with helpful message listing the domains.
+
+5. **Cross-package refs still work**: `alias->scoring.compute_score` lookup works with `pipe_ref`-keyed entries.
+
+### Unit tests for `DomainLibrary`
+
+6. **Domain idempotency**: Add domain `scoring` twice → no error, domain exists once.
+
+### Integration tests
+
+7. **Load multi-bundle same domain**: Create two `.mthds` files both declaring `domain: scoring` with different pipes. Load both. All pipes accessible. Domain exists once.
+
+8. **Full pipeline through multi-domain library**: A PipeSequence referencing pipes from multiple domains runs correctly.
+
+### Regression
+
+9. **All existing tests pass**: `make agent-test` green.
diff --git a/wip/archive/phase1-handoff.md b/wip/archive/phase1-handoff.md
new file mode 100644
index 000000000..bc9099e42
--- /dev/null
+++ b/wip/archive/phase1-handoff.md
@@ -0,0 +1,380 @@
+# LibraryCrate: Direct Execution Mode
+
+> **Status**: Ready for implementation
+> **Branch from**: `dev` (at or after commit `628e414d` — pipe_ref namespace fix)
+> **Date**: 2026-03-24
+
+---
+
+## 1. Motivation
+
+Library loading currently happens in a single monolithic step: `load_from_blueprints()` takes raw `PipelexBundleBlueprint` objects (the direct parse output of `.mthds` files) and produces a live `Library` with instantiated domains, concepts, and pipes. There is no stable intermediate representation between "files on disk" and "fully wired runtime objects."
+
+This creates three problems:
+
+1. **Tight coupling.** Parsing, merging, qualifying, and instantiating are interleaved. You cannot inspect the merged library content without instantiating every concept and pipe.
+2. **Testability.** Testing the merging/qualification logic requires running the full factory stack, including `KajsonManager`, forward-reference resolution, and class registry setup.
+3. **No caching boundary.** There is no fingerprint or content-addressable snapshot of what a library contains, so there is no way to detect "same content, already loaded" without re-running the full pipeline.
+
+The fix is a clean three-stage pipeline:
+
+```
+Bundles (files) → LibraryCrate (data) → Library (live)
+```
+
+The `LibraryCrate` is a pure-data Pydantic model holding the fully qualified, merged blueprints for all concepts and pipes. It is the single gateway between sourcing and instantiation.
+
+---
+
+## 2. Prerequisite: Pipe Namespace Fix (Complete)
+
+PR #780 (`628e414d`) refactored the pipe namespace so that `PipeLibrary` is keyed by `pipe_ref` (`domain.pipe_code`), symmetric with how `ConceptLibrary` uses `concept_ref` (`domain.ConceptCode`). This is required because the LibraryCrate uses qualified refs as dictionary keys.
+
+Key changes already on `dev`:
+- `PipeAbstract.pipe_ref` property: `f"{self.domain_code}.{self.code}"`
+- `PipeLibrary.root` keyed by `pipe_ref`
+- `DomainLibrary.add_domain()` is idempotent for multi-bundle domain merging
+- Duplicate concept detection across bundles in `_load_concepts_from_blueprints()`
+- Duplicate pipe detection across bundles in `load_from_blueprints()`
+
+---
+
+## 3. LibraryCrate Model
+
+### New file: `pipelex/libraries/library_crate.py`
+
+```python
+class LibraryCrate(BaseModel):
+ """Complete library content as qualified blueprints, ready to load into a live Library.
+
+ The crate is a flat, domain-agnostic snapshot of all concepts and pipes.
+ Domain is encoded in the dictionary keys (e.g. 'scoring.WeightedScore',
+ 'scoring.compute_score'), not in a structural container.
+ """
+
+ concepts: dict[str, ConceptBlueprint] = Field(default_factory=dict)
+ """concept_ref (domain.ConceptCode) -> ConceptBlueprint"""
+
+ pipes: dict[str, PipeBlueprintUnion] = Field(default_factory=dict)
+ """pipe_ref (domain.pipe_code) -> PipeBlueprintUnion"""
+
+ fingerprint: str = ""
+ """SHA-256 hex digest of the serialized concepts + pipes content."""
+```
+
+### Design decisions
+
+- **Flat dicts, not nested by domain.** Domain is a namespace prefix in the key. No `DomainCrate` wrapper — merging is trivial.
+- **Blueprints, not instantiated objects.** The crate holds `ConceptBlueprint` and `PipeBlueprintUnion` — the same types that come out of `PipelexBundleBlueprint`. No factory logic, no class registry interaction, no forward-reference resolution.
+- **String-described concepts are normalized.** When a bundle declares `Concept1 = "Description"` (a `str` value in `PipelexBundleBlueprint.concept`), the factory converts it to `ConceptBlueprint(description="Description")` before inserting into the crate. The crate always holds `ConceptBlueprint`, never bare strings.
+- **Fingerprint** is computed over the deterministic JSON serialization of `concepts` and `pipes` (excluding the fingerprint field itself). Enables downstream caching by content identity.
+- **No source paths, no package provenance.** The crate is a pure content snapshot. Source tracking (`_pipe_source_map`, `loaded_mthds_paths`) remains in `LibraryManager`, outside the crate.
+- **No domain metadata.** Domain `description` and `system_prompt` are NOT stored in the crate. They are passed separately via `domain_metadata` parameter to `load_from_crate()`. See Section 7.
+
+---
+
+## 4. LibraryCrateFactory
+
+### New file: `pipelex/libraries/library_crate_factory.py`
+
+```python
+class LibraryCrateFactory:
+ @classmethod
+ def make_from_blueprints(
+ cls,
+ blueprints: list[PipelexBundleBlueprint],
+ ) -> LibraryCrate:
+ """Build a LibraryCrate from parsed bundle blueprints.
+
+ For each bundle:
+ 1. Qualify concept codes with the bundle's domain → concept_ref keys
+ 2. Qualify pipe codes with the bundle's domain → pipe_ref keys
+ 3. Normalize string-described concepts to ConceptBlueprint
+ 4. Merge into flat dicts (same ref twice = error)
+ 5. Compute SHA-256 fingerprint
+ """
+```
+
+### Algorithm
+
+```
+concepts: dict[str, ConceptBlueprint] = {}
+pipes: dict[str, PipeBlueprintUnion] = {}
+
+for each blueprint in blueprints:
+ domain_code = blueprint.domain
+
+ if blueprint.concept is not None:
+ for concept_code, value in blueprint.concept.items():
+ concept_ref = f"{domain_code}.{concept_code}"
+ if concept_ref in concepts:
+ raise LibraryCrateError(f"Concept '{concept_ref}' declared twice")
+ if isinstance(value, str):
+ concepts[concept_ref] = ConceptBlueprint(description=value)
+ else:
+ concepts[concept_ref] = value
+
+ if blueprint.pipe is not None:
+ for pipe_code, pipe_blueprint in blueprint.pipe.items():
+ pipe_ref = f"{domain_code}.{pipe_code}"
+ if pipe_ref in pipes:
+ raise LibraryCrateError(f"Pipe '{pipe_ref}' declared twice")
+ pipes[pipe_ref] = pipe_blueprint
+
+fingerprint = _compute_fingerprint(concepts, pipes)
+return LibraryCrate(concepts=concepts, pipes=pipes, fingerprint=fingerprint)
+```
+
+### Fingerprint computation
+
+Use `model_dump_json()` for deterministic serialization with `sort_keys` to ensure order-independence. SHA-256 hex digest.
+
+### New file: `pipelex/libraries/library_crate_exceptions.py`
+
+```python
+class LibraryCrateError(Exception):
+ """Raised when building a LibraryCrate fails (e.g. ref collision)."""
+```
+
+---
+
+## 5. LibraryManager.load_from_crate()
+
+### New method on `LibraryManager` (`pipelex/libraries/library_manager.py`)
+
+```python
+def load_from_crate(
+ self,
+ library_id: str,
+ crate: LibraryCrate,
+ domain_metadata: dict[str, DomainBlueprint] | None = None,
+) -> list[PipeAbstract]:
+ """Load a LibraryCrate into a live Library.
+
+ Args:
+ library_id: The library to load into
+ crate: The LibraryCrate containing qualified blueprints
+ domain_metadata: Optional domain metadata (description, system_prompt)
+ keyed by domain_code. When absent, domains get empty descriptions.
+ """
+```
+
+### Algorithm
+
+```
+1. Extract unique domain codes from refs
+ domain_codes = set()
+ for ref in crate.concepts:
+ parsed = QualifiedRef.parse_concept_ref(ref)
+ domain_codes.add(parsed.domain_path)
+ for ref in crate.pipes:
+ parsed = QualifiedRef.parse_pipe_ref(ref)
+ domain_codes.add(parsed.domain_path)
+
+2. Create Domain objects
+ for domain_code in domain_codes:
+ if domain_metadata and domain_code in domain_metadata:
+ domain = DomainFactory.make_from_blueprint(domain_metadata[domain_code])
+ else:
+ domain = Domain(code=domain_code, description="")
+ library.domain_library.add_domain(domain)
+
+3. Load concepts (topological ordering, forward refs, cycle detection)
+ Build concept entries from crate.concepts (dict[concept_ref, ConceptBlueprint])
+ instead of iterating over PipelexBundleBlueprint objects.
+
+ For each concept_ref, concept_blueprint in crate.concepts.items():
+ parsed = QualifiedRef.parse_concept_ref(concept_ref)
+ domain_code = parsed.domain_path
+ concept_code = parsed.local_code
+ # Feed into topological sorter (same logic as _load_concepts_from_blueprints)
+
+ Load in topological order via ConceptFactory.make_from_blueprint()
+ library.concept_library.add_concepts(all_concepts)
+
+4. Resolve forward references
+ self._rebuild_models_with_forward_refs(all_concepts)
+
+5. Detect concept cycles
+ self._detect_concept_cycles(all_concepts)
+
+6. Load pipes
+ For each pipe_ref, pipe_blueprint in crate.pipes.items():
+ parsed = QualifiedRef.parse_pipe_ref(pipe_ref)
+ domain_code = parsed.domain_path
+ pipe_code = parsed.local_code
+
+ # Collect concept codes from same domain for validation
+ concept_codes_for_domain = [
+ QualifiedRef.parse_concept_ref(ref).local_code
+ for ref in crate.concepts
+ if ref.startswith(f"{domain_code}.")
+ ]
+
+ pipe = PipeFactory[PipeAbstract].make_from_blueprint(
+ domain_code=domain_code,
+ pipe_code=pipe_code,
+ blueprint=pipe_blueprint,
+ concept_codes_from_the_same_domain=concept_codes_for_domain,
+ )
+ library.pipe_library.add_pipes(all_pipes)
+
+7. Validate
+ library.validate_library()
+
+8. Return all_pipes
+```
+
+### Also add abstract method
+
+In `pipelex/libraries/library_manager_abstract.py`, add the `load_from_crate()` abstract method.
+
+---
+
+## 6. Refactor load_from_blueprints()
+
+The existing `load_from_blueprints()` becomes a two-step delegation:
+
+```python
+def load_from_blueprints(
+ self,
+ library_id: str,
+ blueprints: list[PipelexBundleBlueprint],
+) -> list[PipeAbstract]:
+ # Step 0: Load address-based dependencies (unchanged)
+ self._load_address_based_dependencies(library_id=library_id, blueprints=blueprints)
+
+ # Step 1: Build domain metadata from blueprints (first-write-wins per domain)
+ domain_metadata: dict[str, DomainBlueprint] = {}
+ for blueprint in blueprints:
+ if blueprint.domain not in domain_metadata:
+ domain_metadata[blueprint.domain] = DomainBlueprint(
+ source=blueprint.source,
+ code=blueprint.domain,
+ description=blueprint.description or "",
+ system_prompt=blueprint.system_prompt,
+ )
+
+ # Step 2: Build the crate
+ crate = LibraryCrateFactory.make_from_blueprints(blueprints=blueprints)
+
+ # Step 3: Load from crate
+ return self.load_from_crate(
+ library_id=library_id,
+ crate=crate,
+ domain_metadata=domain_metadata,
+ )
+```
+
+**Callers do not change.** `pipeline_run_setup()`, `_load_mthds_files_into_library()`, and the dependency loading code all call `load_from_blueprints()` and continue to work as before.
+
+**Note on `_pipe_source_map`**: The current `load_from_blueprints()` tracks which `.mthds` file each pipe came from. After refactoring, source tracking must be preserved. Two options:
+- (a) Build the source map in the refactored `load_from_blueprints()` after `load_from_crate()` returns, by iterating blueprints and matching pipe_refs.
+- (b) Pass source info as an additional parameter to `load_from_crate()`.
+
+Option (a) is simpler and keeps the crate source-agnostic.
+
+**`load_concepts_only_from_blueprints()`** can remain as-is for Phase 1 — it's used by LSP/validation tooling and doesn't need the crate abstraction yet.
+
+---
+
+## 7. system_prompt and Domain Metadata
+
+The `PipelexBundleBlueprint.system_prompt` field flows into `Domain.system_prompt` and is used as a fallback system prompt for PipeLLM pipes. There's already a TODO comment in `domain_library.py` (line 35-36) about inlining this at pipe factory time.
+
+**For this phase:**
+- Domain metadata (`description`, `system_prompt`) is NOT stored in the crate.
+- Domain metadata is passed as a separate `domain_metadata` parameter to `load_from_crate()`.
+- When `load_from_blueprints()` calls `load_from_crate()`, it extracts domain metadata from the blueprints and passes it along.
+- The existing `DomainLibrary.add_domain()` first-write-wins behavior is preserved.
+
+This keeps the crate focused on content. Inlining `system_prompt` into pipe blueprints is a separate optimization.
+
+---
+
+## 8. File Summary
+
+| File | Action | Description |
+|------|--------|-------------|
+| `pipelex/libraries/library_crate.py` | **New** | `LibraryCrate` model |
+| `pipelex/libraries/library_crate_factory.py` | **New** | `LibraryCrateFactory.make_from_blueprints()`, fingerprint |
+| `pipelex/libraries/library_crate_exceptions.py` | **New** | `LibraryCrateError` exception |
+| `pipelex/libraries/library_manager.py` | **Modify** | Add `load_from_crate()`, refactor `load_from_blueprints()` |
+| `pipelex/libraries/library_manager_abstract.py` | **Modify** | Add `load_from_crate()` abstract method |
+| `tests/unit/pipelex/libraries/test_library_crate.py` | **New** | Unit tests for model and factory |
+| `tests/unit/pipelex/libraries/test_library_crate_data.py` | **New** | Test data |
+| `tests/integration/pipelex/libraries/test_load_from_crate.py` | **New** | Integration: crate ↔ blueprints equivalence |
+
+---
+
+## 9. Test Plan
+
+### Unit tests: `tests/unit/pipelex/libraries/test_library_crate.py`
+
+One `TestLibraryCrate` class:
+
+1. **JSON round-trip** — Create `LibraryCrate` with sample data, serialize to JSON, deserialize, assert equality.
+2. **make_from_blueprints merges correctly** — Two blueprints for the same domain with different concepts and pipes. Assert all entries present with qualified refs.
+3. **make_from_blueprints merges across domains** — Two blueprints with different domains. Assert refs qualified with respective domains.
+4. **String concept normalization** — Blueprint with `concept = {"MyConcept": "A description"}`. Assert crate contains `ConceptBlueprint(description="A description")`.
+5. **Concept collision raises** — Two blueprints declaring same `domain.ConceptCode`. Assert `LibraryCrateError`.
+6. **Pipe collision raises** — Two blueprints declaring same `domain.pipe_code`. Assert `LibraryCrateError`.
+7. **Fingerprint determinism** — Same blueprints → same fingerprint. Different blueprints → different fingerprint.
+8. **Empty blueprints** — Blueprint with no concepts and no pipes. Assert empty dicts and valid fingerprint.
+
+### Integration test: `tests/integration/pipelex/libraries/test_load_from_crate.py`
+
+One `TestLoadFromCrate` class:
+
+1. **Equivalence test** — Load `.mthds` files via `load_from_blueprints()`. Separately build a crate from the same blueprints, call `load_from_crate()`. Assert both libraries contain the same concept_refs, pipe_refs, and domain codes.
+
+### Regression
+
+- All existing tests pass: `make agent-test`
+- All linting passes: `make agent-check`
+
+---
+
+## 10. Implementation Sequence
+
+1. **Create `LibraryCrate` model** (`library_crate.py`). Pure data model. Write JSON round-trip test first (TDD).
+2. **Create `LibraryCrateFactory`** (`library_crate_factory.py`). Implement `make_from_blueprints()` with string normalization, collision detection, fingerprint. Write unit tests.
+3. **Add `load_from_crate()`** to `LibraryManager`. Port concept-loading logic (topological sort, forward refs, cycle detection) and pipe-loading logic to work with flat crate dicts.
+4. **Refactor `load_from_blueprints()`** to delegate through the crate. Extract domain metadata, build crate, call `load_from_crate()`.
+5. **Run `make agent-check` and `make agent-test`**. Fix regressions.
+6. **Write integration equivalence test**.
+
+---
+
+## 11. Risks and Edge Cases
+
+### String-described concepts
+`PipelexBundleBlueprint.concept` values can be `ConceptBlueprint | str`. The factory normalizes `str` to `ConceptBlueprint(description=value)`. This must match what `ConceptFactory.make_from_blueprint()` expects.
+
+### Cross-package dependencies
+`load_from_blueprints()` calls `_load_address_based_dependencies()` before building the crate. These dependencies load into child libraries via separate code paths. This phase does NOT route dependency loading through a crate. The crate only covers the "main" blueprints.
+
+### load_concepts_only_from_blueprints()
+This lightweight variant (LSP/validation) loads only domains and concepts. It can remain as-is for this phase.
+
+### Ref parsing for multi-segment domains
+Use `QualifiedRef.parse_concept_ref()` and `QualifiedRef.parse_pipe_ref()` (`pipelex/core/qualified_ref.py`) for correct splitting. These use `rsplit(".", 1)` which handles multi-segment domains like `legal.contracts.NonCompeteClause` correctly.
+
+### None concept/pipe dicts
+`PipelexBundleBlueprint.concept` and `.pipe` can be `None`. The factory must guard with `if blueprint.concept is not None:`.
+
+---
+
+## 12. Key Existing Code to Reuse
+
+| What | Where |
+|------|-------|
+| `PipeFactory[PipeAbstract].make_from_blueprint()` | `pipelex/core/pipes/pipe_factory.py` |
+| `ConceptFactory.make_from_blueprint()` | `pipelex/core/concepts/concept_factory.py` |
+| `DomainFactory.make_from_blueprint()` | `pipelex/core/domains/domain_factory.py` |
+| `QualifiedRef.parse_concept_ref()` / `parse_pipe_ref()` | `pipelex/core/qualified_ref.py` |
+| `_load_concepts_from_blueprints()` (topological sort logic) | `pipelex/libraries/library_manager.py:472` |
+| `_rebuild_models_with_forward_refs()` | `pipelex/libraries/library_manager.py` |
+| `_detect_concept_cycles()` | `pipelex/libraries/library_manager.py` |
+| `library.validate_library()` | `pipelex/libraries/library.py` |
+| `DomainLibrary.add_domain()` (idempotent) | `pipelex/libraries/domain/domain_library.py` |
diff --git a/wip/archive/phase2-crate-propagation-rationale.md b/wip/archive/phase2-crate-propagation-rationale.md
new file mode 100644
index 000000000..e9ffc6efc
--- /dev/null
+++ b/wip/archive/phase2-crate-propagation-rationale.md
@@ -0,0 +1,112 @@
+# Design Rationale: LibraryCrate Propagation in Distributed Execution
+
+> **Status**: Decided
+> **Date**: 2026-03-25
+> **Related**: [00-master-plan.md](00-master-plan.md)
+
+---
+
+## Problem
+
+Phase 2 ships a `LibraryCrate` to Temporal workers so they can load the library and resolve child pipes via `get_required_pipe()`. The master plan v2 proposed adding `library_crate` to `PipeJob` and using an `act_library_setup` activity to load it.
+
+Two gaps in that design:
+
+1. **Child workflow propagation**: Pipe controllers (PipeSequence, PipeParallel, etc.) dispatch child pipes through `SubPipe.run_pipe()` → `PipeRouterChild` → new `WfPipeRouter` child workflow. Each child workflow is a separate Temporal workflow that **can land on a different worker**. The crate must be on each child `PipeJob`, or the child worker won't have the library.
+
+2. **Activity necessity**: The plan called for an `act_library_setup` activity. But loading a crate is pure in-memory work — deserializing Pydantic models into dicts. Activities are for I/O-bound or side-effectful operations that need retry semantics. An activity adds latency and complexity for no benefit here.
+
+---
+
+## The Fundamental Constraint
+
+In Temporal distributed execution, **the only data channel between parent and child workflow is the serialized workflow argument**. There is no shared memory, no shared process, no ContextVar, no singleton that crosses the workflow boundary. Child workflows can run on any available worker in the task queue.
+
+This means the crate cannot be stashed in a ContextVar, a library manager singleton, or any other in-process state and expected to be available in child workflows. It must travel in the `PipeJob`.
+
+---
+
+## Decision: `library_crate` on `PipeJob`, threaded through the signature chain
+
+### Where the crate lives
+
+`library_crate: LibraryCrate | None = None` is added to `PipeJob`.
+
+### How it propagates
+
+The crate must flow from `WfPipeRouter` (which has the `PipeJob`) down to `SubPipe` (which builds child `PipeJob`s). The call chain is:
+
+```
+WfPipeRouter.run(pipe_job: PipeJob)
+ → pipe.run_pipe(job_metadata, working_memory, pipe_run_params, output_name, library_crate)
+ → _live_run_pipe(job_metadata, working_memory, pipe_run_params, output_name, library_crate)
+ → [PipeController] _live_run_controller_pipe(..., library_crate)
+ → SubPipe.run_pipe(..., library_crate)
+ → PipeJobFactory.make_pipe_job(..., library_crate) # child PipeJob carries the crate
+ → PipeRouterChild → new WfPipeRouter(child_pipe_job)
+```
+
+`library_crate` is added as an optional parameter (default `None`) at each level. Pipe operators (PipeLLM, PipeExtract, etc.) receive it but ignore it — they have no child pipes.
+
+### How loading works
+
+At the top of `WfPipeRouter.run()`, if `pipe_job.library_crate` is present, call `library_manager.load_from_crate()` inline (no activity). The load is idempotent via fingerprint — if the crate was already loaded on this worker, it's a no-op.
+
+---
+
+## Alternatives Considered and Rejected
+
+### Put the crate on `PipeRunParams`
+
+`PipeRunParams` already threads through the entire execution tree — `SubPipe` receives it and passes it to `PipeJobFactory`. No signature changes needed.
+
+**Rejected because**: `PipeRunParams` is about *how to run* a pipe (run mode, stack limit, multiplicity, batch params). It gets copied and mutated at every nesting level. The crate is immutable static context. Mixing execution parameters with library content is semantically wrong and makes `PipeRunParams` a grab-bag.
+
+### Put the crate on `JobMetadata`
+
+`JobMetadata` also threads through the entire tree. Since it's inside `PipeJob`, we could avoid the `PipeJob`-level field entirely.
+
+**Rejected because**: `JobMetadata` carries observability and identity data (user_id, pipeline_run_id, OTel context, graph context). Adding library content to it is semantically wrong — metadata is about *who* and *when*, not *what to load*.
+
+### ContextVar / library manager stash
+
+Store the crate in a `ContextVar` or on the library manager after loading. `SubPipe` reads it when building child `PipeJob`s. Zero signature changes.
+
+**Rejected because**: This is fundamentally broken in distributed execution. ContextVars and singletons are per-process state. Temporal child workflows can land on different workers — different processes, different servers. The only data channel is the serialized `PipeJob`. This was the key insight that ruled out all in-process state approaches.
+
+### `act_library_setup` activity
+
+Use a Temporal activity to load the crate before pipe execution.
+
+**Rejected because**: Loading a crate is pure in-memory work — iterating over Pydantic model dicts and populating the library manager. It involves no I/O, no network calls, no file reads. Activities exist for operations that need Temporal's retry/timeout/heartbeat semantics. Using an activity here adds unnecessary latency (activity scheduling overhead) and complexity for zero benefit.
+
+### Put the crate only on `PipeController` (not `PipeAbstract`)
+
+Since only controllers have child pipes, add the parameter only at the `PipeController` level.
+
+**Rejected because**: `PipeAbstract.run_pipe()` is the entry point called by `WfPipeRouter`. It calls `_live_run_pipe()` internally. If `_live_run_pipe` has a different signature on `PipeController` than on `PipeAbstract`, the override breaks. The crate must cross the `run_pipe()` boundary, which means `PipeAbstract` must accept it.
+
+---
+
+## Blast Radius
+
+The signature change touches the following layers:
+
+| Layer | File | Change |
+|-------|------|--------|
+| `PipeAbstract.run_pipe()` | `pipelex/core/pipes/pipe_abstract.py` | Add optional `library_crate` param |
+| `PipeAbstract._live_run_pipe()` | `pipelex/core/pipes/pipe_abstract.py` | Add optional `library_crate` param |
+| `PipeOperator._live_run_pipe()` | `pipelex/pipe_operators/pipe_operator.py` | Add param, ignore it |
+| `PipeController._live_run_pipe()` | `pipelex/pipe_controllers/pipe_controller.py` | Add param, forward to `_live_run_controller_pipe()` |
+| `PipeController._live_run_controller_pipe()` | `pipelex/pipe_controllers/pipe_controller.py` | Add param (abstract) |
+| PipeSequence | `pipelex/pipe_controllers/sequence/pipe_sequence.py` | Forward to `SubPipe.run_pipe()` |
+| PipeBatch | `pipelex/pipe_controllers/batch/pipe_batch.py` | Forward to child pipe execution |
+| PipeCondition | `pipelex/pipe_controllers/condition/pipe_condition.py` | Forward to child pipe execution |
+| PipeParallel | `pipelex/pipe_controllers/parallel/pipe_parallel.py` | Forward to `SubPipe.run_pipe()` |
+| `SubPipe.run_pipe()` | `pipelex/pipe_controllers/sub_pipe.py` | Add param, set on child `PipeJob` |
+| `PipeJob` | `pipelex/pipe_run/pipe_job.py` | Add `library_crate` field |
+| `PipeJobFactory` | `pipelex/pipe_run/pipe_job_factory.py` | Accept and forward `library_crate` |
+| `WfPipeRouter` | `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Load crate, pass to `run_pipe()` |
+| `PipeRouter` (direct) | `pipelex/pipe_run/pipe_router.py` | Pass `library_crate` from `PipeJob` to `run_pipe()` |
+
+Every change is mechanical: add one optional parameter, forward it. The crate is never modified, only carried.
diff --git a/wip/archive/phase2-implementation-plan.md b/wip/archive/phase2-implementation-plan.md
new file mode 100644
index 000000000..190fd4d9c
--- /dev/null
+++ b/wip/archive/phase2-implementation-plan.md
@@ -0,0 +1,311 @@
+# Phase 2 Implementation Plan: LibraryCrate on Temporal
+
+> **Status**: Reviewed, ready to implement (updated after feature/LibraryCrate merge — 5477e16a, then crate design review — Option A chosen)
+> **Date**: 2026-03-25
+> **Related**: [00-master-plan.md](00-master-plan.md), [phase2-crate-propagation-rationale.md](phase2-crate-propagation-rationale.md), [future-crate-first-architecture.md](future-crate-first-architecture.md)
+> **Review**: Eng review passed (0 critical gaps). Codex outside voice: 7 findings, all addressed. Crate design review: Option A (blueprint accumulation) chosen over Option B (crate merge).
+
+---
+
+## Context
+
+Phase 0 (pipe_ref fix) and Phase 1 (LibraryCrate in direct mode) are complete. Temporal workers currently cannot resolve child pipes from `mthds_content` or extra `library_dirs` because those definitions only exist on the API/submitter side. Phase 2 ships the `LibraryCrate` inside `PipeJob` so every worker — including child workflows on different machines — receives the full library context.
+
+**What this solves**: Pipe controllers (PipeSequence, PipeCondition, PipeBatch, PipeParallel) work on Temporal workers. `get_required_pipe()` succeeds for all pipes in the library.
+
+**What this does NOT solve**: Layer 1 (dynamic class deserialization for custom concepts). That's Phase 3.
+
+---
+
+## Key Architecture Decisions
+
+1. **Crate on PipeJob** — `library_crate: LibraryCrate | None = None` field on `PipeJob`. Threaded through the signature chain as an optional parameter. (Rationale: [phase2-crate-propagation-rationale.md](phase2-crate-propagation-rationale.md))
+
+2. **Self-contained crate** — The crate carries ALL blueprints (PIPELEXPATH + mthds_content + extra library_dirs). Workers don't assume anything is pre-loaded.
+
+3. **Blueprint accumulation on LibraryManager** — `_blueprints: dict[str, list[PipelexBundleBlueprint]]` tracks all blueprints loaded per `library_id`. Each `load_from_blueprints()` call appends to the list. `get_crate(library_id)` builds one crate from all accumulated blueprints via `LibraryCrateFactory.make_from_blueprints()`. No merge logic, no crate accumulation — blueprints are the unit of accumulation. See [future-crate-first-architecture.md](future-crate-first-architecture.md) for the rationale.
+
+4. **Per-workflow library on workers** — Each `WfPipeRouter` opens its own library using a unique ID derived from `workflow.info().workflow_id` (e.g., `f"wf_{workflow_id}"`). Full isolation — parent and child workflows from PipeParallel/PipeBatch get different IDs. Teardown after execution.
+
+5. **Inline loading (no activity)** — Crate loading is pure in-memory work. No Temporal activity needed.
+
+6. **Fingerprint idempotency** — `load_from_crate()` skips if a crate with the same fingerprint was already loaded into the same `library_id`.
+
+---
+
+## Data Flow
+
+```
+SUBMITTER (pipeline_run_setup)
+════════════════════════════════════════════════════════
+ load_libraries(library_id, dirs) ─┐
+ → load_from_blueprints(blueprints) │ Each call appends blueprints
+ → _blueprints[library_id] += blueprints│ to the accumulation list,
+ → LibraryCrateFactory → crate │ then loads from crate
+ → load_from_crate(library_id, crate) ─┘ as today
+
+ [if mthds_content]
+ load_from_blueprints(library_id, mthds_blueprints)
+ → same path → blueprints appended to _blueprints[library_id]
+
+ library_crate = library_manager.get_crate(library_id)
+ → LibraryCrateFactory.make_from_blueprints(_blueprints[library_id])
+ → one crate from ALL accumulated blueprints
+ pipe_job = PipeJobFactory.make_pipe_job(..., library_crate=library_crate)
+
+
+TEMPORAL DISPATCH
+════════════════════════════════════════════════════════
+ PipeRouterTop._run_pipe_job(pipe_job)
+ → WfPipeRouter workflow(pipe_job) [can land on ANY worker]
+
+
+WORKER (WfPipeRouter.run)
+════════════════════════════════════════════════════════
+ wf_library_id = f"wf_{workflow.info().workflow_id}"
+ library_manager.open_library(library_id=wf_library_id)
+ set_current_library(library_id=wf_library_id)
+ library_manager.load_from_crate(wf_library_id, pipe_job.library_crate)
+
+ pipe.run_pipe(..., library_crate=pipe_job.library_crate)
+ → [controller] _live_run_controller_pipe(..., library_crate)
+ → SubPipe.run_pipe(..., library_crate)
+ → PipeJobFactory.make_pipe_job(..., library_crate) # child carries crate
+ → PipeRouterChild → new WfPipeRouter(child_pipe_job)
+ [cycle repeats on potentially different worker]
+
+ library_manager.teardown(library_id=wf_library_id)
+ teardown_current_library()
+```
+
+---
+
+## Implementation Steps
+
+### Step 1: Blueprint accumulation + get_crate + fingerprint idempotency
+
+**Tests first** (`tests/unit/pipelex/libraries/test_library_crate_accumulation.py`):
+- `get_crate(library_id)` builds crate from all accumulated blueprints
+- `get_crate(library_id)` returns `None` for unknown library_id
+- `get_crate(library_id)` returns `None` when no blueprints were loaded
+- Blueprints from multiple `load_from_blueprints()` calls are all included in the crate
+- `load_from_crate()` returns `[]` on second call with same fingerprint (idempotency)
+- `teardown()` clears blueprints for that library_id
+
+**Then implement:**
+
+**File**: `pipelex/libraries/library_manager.py`
+- Add `_blueprints: dict[str, list[PipelexBundleBlueprint]]` field alongside `_libraries`
+- In `load_from_blueprints()`:
+ - Append `blueprints` to `_blueprints[library_id]` (before or after the existing crate build + load)
+- In `load_from_crate()`:
+ - Check fingerprint idempotency (skip if already seen for this library_id)
+- Add `get_crate(library_id) -> LibraryCrate | None`: if `_blueprints[library_id]` exists and is non-empty, call `LibraryCrateFactory.make_from_blueprints()` with the full accumulated list; otherwise return `None`
+- Clear `_blueprints[library_id]` in `teardown(library_id)` and `reset()`
+
+**File**: `pipelex/libraries/library_manager_abstract.py`
+- Add abstract `get_crate(library_id) -> LibraryCrate | None`
+
+**No changes to `pipelex/libraries/library_crate.py`** — no `merge()` method needed.
+
+---
+
+### Step 2: Add `library_crate` field to PipeJob and PipeJobFactory
+
+**File**: `pipelex/pipe_run/pipe_job.py`
+- Add `library_crate: LibraryCrate | None = None` field
+
+**File**: `pipelex/pipe_run/pipe_job_factory.py`
+- Add `library_crate: LibraryCrate | None = None` param to `make_pipe_job()`, forward to PipeJob constructor
+
+---
+
+### Step 3: Thread `library_crate` through the signature chain
+
+Add `library_crate: LibraryCrate | None = None` as the last parameter. Use `TYPE_CHECKING` import.
+
+#### 3a. PipeAbstract (`pipelex/core/pipes/pipe_abstract.py`)
+
+`@final` methods — add param and forward:
+- `run_pipe()` → `live_run_pipe()` / `dry_run_pipe()`
+- `live_run_pipe()` → `_live_run_pipe()`
+- `dry_run_pipe()` → `_dry_run_pipe()`
+
+Abstract methods — signature only:
+- `_live_run_pipe()`
+- `_dry_run_pipe()`
+
+#### 3b. PipeOperator (`pipelex/pipe_operators/pipe_operator.py`)
+
+Accept but do NOT forward deeper:
+- `_live_run_pipe()` — receives `library_crate`, does NOT pass to `_live_run_operator_pipe()`
+- `_dry_run_pipe()` — same
+
+The 6 concrete operators (PipeLLM, PipeExtract, PipeSearch, PipeImgGen, PipeFunc, PipeCompose) need NO changes.
+
+#### 3c. PipeController (`pipelex/pipe_controllers/pipe_controller.py`)
+
+Add param and forward:
+- `_live_run_pipe()` → `_live_run_controller_pipe()`
+- `_dry_run_pipe()` → `_dry_run_controller_pipe()`
+- `_live_run_controller_pipe()` (abstract) — signature only
+- `_dry_run_controller_pipe()` (abstract) — signature only
+
+#### 3d. Controller implementations
+
+**PipeSequence** (`pipe_controllers/sequence/pipe_sequence.py`):
+- `_live_run_controller_pipe()`: pass to `sub_pipe.run_pipe()` calls
+- `_dry_run_controller_pipe()`: forward to `_live_run_controller_pipe()`
+
+**PipeParallel** (`pipe_controllers/parallel/pipe_parallel.py`):
+- `_live_run_controller_pipe()`: pass to `sub_pipe.run_pipe()` calls
+- `_dry_run_controller_pipe()`: pass to `sub_pipe.run_pipe()` calls
+
+**PipeBatch** (`pipe_controllers/batch/pipe_batch.py`):
+- `_live_run_controller_pipe()`: pass to `PipeJobFactory.make_pipe_job()` calls
+- `_dry_run_controller_pipe()`: forward to `_live_run_controller_pipe()`
+
+**PipeCondition** (`pipe_controllers/condition/pipe_condition.py`):
+- `_live_run_controller_pipe()`: pass to `PipeJobFactory.make_pipe_job()`
+- `_dry_run_controller_pipe()`: pass to `pipe.run_pipe()` calls
+
+#### 3e. SubPipe (`pipelex/pipe_controllers/sub_pipe.py`)
+
+Add `library_crate` to `run_pipe()`. Forward to all 3 `PipeJobFactory.make_pipe_job()` calls (batch, condition, normal).
+
+---
+
+### Step 4: Update PipeRouter (direct) and WfPipeRouter (Temporal)
+
+#### 4a. PipeRouter (`pipelex/pipe_run/pipe_router.py`)
+
+In `_run_pipe_job()`: pass `library_crate=pipe_job.library_crate` to `pipe.run_pipe()`.
+
+#### 4b. WfPipeRouter (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`)
+
+In `run()`:
+1. Derive unique library ID: `wf_library_id = f"wf_{workflow.info().workflow_id}"`
+2. Open per-workflow library: `library_manager.open_library(library_id=wf_library_id)`
+3. Set as current: `set_current_library(library_id=wf_library_id)`
+4. If `library_crate` present: `library_manager.load_from_crate(wf_library_id, crate)`
+5. Run pipe with `library_crate=workflow_arg.library_crate`
+6. In finally block: `library_manager.teardown(library_id=wf_library_id)` + `teardown_current_library()`
+
+---
+
+### Step 5: Build and attach crate in `pipeline_run_setup()`
+
+**File**: `pipelex/pipeline/pipeline_run_setup.py`
+
+**Note**: After the feature/LibraryCrate merge, `mthds_content` is now `mthds_contents: list[str] | None` and `bundle_uri` is now `bundle_uris: list[str] | None`. The crate accumulation approach is unaffected — both `load_libraries()` and `load_from_blueprints()` go through `load_from_crate()` which accumulates into `_crates[library_id]`.
+
+After all library loading (load_libraries + load_from_blueprints), before creating PipeJob:
+
+```python
+library_crate = library_manager.get_crate(library_id)
+
+pipe_job = PipeJobFactory.make_pipe_job(
+ ...,
+ library_crate=library_crate,
+)
+```
+
+No manual crate building or merging needed — `get_crate()` returns the accumulated result.
+
+---
+
+### Step 6: Run checks and tests
+
+1. `make agent-check` — lint + type check
+2. `make agent-test` — full test suite
+
+---
+
+## NOT in scope
+
+- **Layer 1 (dynamic class deserialization)** — Phase 3. When `mthds_content` introduces custom concepts with dynamic classes, Kajson can't deserialize WorkingMemory on the worker. Deferred.
+- **StoragePayloadCodec** — Phase 4. For payloads exceeding 2MB. Not needed for typical libraries.
+- **Crate stripping** — Future. Only shipping the transitive closure of pipe dependencies. Optimization.
+- **Library fingerprint validation** — Future. Verifying worker's base library matches API's expectation.
+
+### Phase 2 Known Limitations (from Codex review)
+
+- **Cross-package address-based dependencies** — `load_from_crate()` does not resolve `alias->domain.pipe` refs. Pipes with cross-package deps won't resolve on workers. Scoped to packages using METHODS.toml dependency declarations.
+- **Python-backed libraries** — `PipeFunc` (requires function registry) and `structure = "ExistingClass"` concepts (requires class registry) won't work on workers. `load_from_crate()` doesn't import Python modules. Standard pipe types (LLM, extract, search, compose) work fine.
+- **Payload duplication in fanout** — PipeBatch/PipeParallel create N child PipeJobs, each carrying the full crate. Redundant for siblings on the same worker. Optimization via StoragePayloadCodec (Phase 4) or crate stripping (future).
+
+---
+
+## What already exists
+
+| What | Where | Reuse |
+|------|-------|-------|
+| `LibraryCrate` model | `pipelex/libraries/library_crate.py` | Reuse as-is (no merge needed) |
+| `LibraryCrateFactory.make_from_blueprints()` | `pipelex/libraries/library_crate_factory.py` | Reuse as-is |
+| `LibraryManager.load_from_crate()` | `pipelex/libraries/library_manager.py` | Phase 1 — add fingerprint check |
+| `load_from_blueprints()` → crate path | `library_manager.py:432` | Already builds crate internally, calls `load_from_crate` |
+| `PipeFactory.make_pipe_ref_with_domain()` | `core/pipes/pipe_factory.py:41` | New helper — not needed in `merge()` (keys already qualified) |
+| `ConceptFactory.make_concept_ref_with_domain()` | `core/concepts/concept_factory.py:213` | New helper — not needed in `merge()` (keys already qualified) |
+| `LibraryCrate.compute_fingerprint_from_content()` | `libraries/library_crate.py` | Static method — used by `LibraryCrateFactory` |
+| Worker base library loading | `temporal/worker_cli.py:65` | Exists — worker_base library |
+| Kajson data converter | `temporal/temporal_data_converter.py` | Handles PipeJob serialization |
+| `PipeRouterProtocol.run()` | `pipe_run/pipe_router_protocol.py` | Unchanged — PipeJob carries crate |
+
+---
+
+## Failure modes
+
+| Codepath | Failure scenario | Test coverage | Error handling | User impact |
+|----------|-----------------|---------------|----------------|-------------|
+| Crate serialization | LibraryCrate too large for Temporal 2MB limit | Phase 4 test | Temporal raises PayloadTooLarge | Error visible in dashboard |
+| Fingerprint collision | Two different crates with same SHA256 | Astronomically unlikely | Second load skipped silently | Wrong pipes loaded — very unlikely |
+| Worker library teardown | Exception during pipe execution skips teardown | Needs try/finally | Add finally block in WfPipeRouter | Memory leak on worker |
+| Concurrent crate loading | Two workflows load same crate simultaneously | Thread-safe via GIL | No issue in Python | None |
+| Direct mode crate cleanup | `runner.py:199` calls `library.teardown()` directly, not `library_manager.teardown()` | Fix in Step 1 | Change `runner.py` to call `library_manager.teardown(library_id)` instead | Stale `_crates` state in direct mode if not fixed |
+
+No critical gaps — all failure modes have either test coverage, error handling, or are handled by existing infrastructure.
+
+**Note on fingerprint scope**: `compute_fingerprint()` hashes concepts + pipes only (not domains/source_map). This is intentional — domains carry metadata (description, system_prompt) that doesn't affect pipe resolution. If domain metadata changes become significant, fingerprint scope can be expanded later.
+
+**Note on crate design**: Option A (blueprint accumulation) was chosen over Option B (crate merge). See [future-crate-first-architecture.md](future-crate-first-architecture.md) for the full rationale. In short: accumulating blueprints is simpler (no merge logic, no crate accumulation dict, no fingerprint recomputation) and directionally correct toward the future crate-first architecture.
+
+---
+
+## Verification
+
+### Automated
+- All existing tests pass (library_crate=None flows through silently in direct mode)
+- New unit tests for merge, fingerprint idempotency, get_crate
+- `make agent-check` passes
+- `make agent-test` passes
+
+### Manual (Temporal dev setup)
+- 3-terminal Temporal dev setup (server + worker + job submitter)
+- Submit a PipeSequence job with `mthds_content` containing additional pipes
+- Verify child pipes resolve on the worker
+- Check Temporal dashboard: crate visible as structured JSON in PipeJob workflow input
+
+---
+
+## File Summary
+
+| File | Change |
+|------|--------|
+| `pipelex/libraries/library_crate.py` | No changes needed |
+| `pipelex/libraries/library_manager.py` | Add `_blueprints` dict, accumulate in `load_from_blueprints()`, fingerprint idempotency in `load_from_crate()`, `get_crate()` builds crate from accumulated blueprints |
+| `pipelex/libraries/library_manager_abstract.py` | Add abstract `get_crate()` |
+| `pipelex/pipe_run/pipe_job.py` | Add `library_crate` field |
+| `pipelex/pipe_run/pipe_job_factory.py` | Add `library_crate` param |
+| `pipelex/core/pipes/pipe_abstract.py` | Add `library_crate` param to `run_pipe`, `live_run_pipe`, `dry_run_pipe`, `_live_run_pipe`, `_dry_run_pipe` |
+| `pipelex/pipe_operators/pipe_operator.py` | Add `library_crate` param to `_live_run_pipe`, `_dry_run_pipe` (accept, ignore) |
+| `pipelex/pipe_controllers/pipe_controller.py` | Add `library_crate` param to 4 methods |
+| `pipelex/pipe_controllers/sequence/pipe_sequence.py` | Forward `library_crate` to `sub_pipe.run_pipe()` |
+| `pipelex/pipe_controllers/parallel/pipe_parallel.py` | Forward `library_crate` to `sub_pipe.run_pipe()` |
+| `pipelex/pipe_controllers/batch/pipe_batch.py` | Forward `library_crate` to `PipeJobFactory.make_pipe_job()` |
+| `pipelex/pipe_controllers/condition/pipe_condition.py` | Forward `library_crate` to `PipeJobFactory.make_pipe_job()` / `pipe.run_pipe()` |
+| `pipelex/pipe_controllers/sub_pipe.py` | Add `library_crate` param, forward to `PipeJobFactory.make_pipe_job()` |
+| `pipelex/pipe_run/pipe_router.py` | Pass `library_crate` from PipeJob to `run_pipe()` |
+| `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Per-workflow library lifecycle + load crate + pass to `run_pipe()` |
+| `pipelex/pipeline/pipeline_run_setup.py` | Read crate via `get_crate()`, attach to PipeJob |
+| `pipelex/pipeline/runner.py` | Fix teardown: call `library_manager.teardown(library_id)` instead of `library.teardown()` |
+| `tests/unit/pipelex/libraries/test_library_crate_accumulation.py` | New: blueprint accumulation, idempotency, get_crate tests |
diff --git a/wip/archive/phase3-eng-review.md b/wip/archive/phase3-eng-review.md
new file mode 100644
index 000000000..467c99bc8
--- /dev/null
+++ b/wip/archive/phase3-eng-review.md
@@ -0,0 +1,301 @@
+# Engineering Review: Phase 3 — Deferred WorkingMemory Hydration + Scoped ClassRegistry
+
+**Branch**: `feature/Temporal-3`
+**Plan file**: `TODOS.md`
+**Repo mode**: solo
+
+---
+
+## Step 0: Scope Challenge
+
+### What existing code already partially solves sub-problems?
+
+| Sub-problem | Existing code | Reuse? |
+|---|---|---|
+| Context-scoped state | `_library_id: ContextVar` in `hub.py:454` | YES — same pattern for scoped registry |
+| Library lifecycle | `open_library() / teardown() / set_current_library()` in `hub.py` | YES — already used in WfPipeRouter |
+| Class registration | `KajsonManager.get_class_registry().register_class()` in ConceptFactory | YES — auto-uses scoped registry via ContextVar |
+| Temporal data conversion | `BaseModelPayloadConverter` uses `kajson.dumps/loads` | YES — drives the serialization constraints |
+
+### Minimum set of changes?
+
+The plan has two clean parts (A: Kajson, B: Pipelex). No scope creep detected — all items directly serve the stated goal.
+
+### Complexity check
+
+- **Files touched**: 14 (8 modified, 6 new) — above the 8-file smell threshold
+- **New classes**: 1 (`CompositeClassRegistry`) + 1 error class (`PipeJobError`)
+- However, 6 of the 14 are test files. The actual production code changes are 8 files with small, focused diffs. This is inherent complexity, not over-engineering.
+
+### TODOS cross-reference
+
+The plan IS the TODOS.md, so no cross-reference needed.
+
+---
+
+## Resolved Issues
+
+### Issue 1 — RESOLVED: Hydration approach for dynamic classes
+
+**Decision**: Keep `dict[str, Any]` with concept-based hydration (user's input).
+
+**Serialization**: `working_memory.smart_dump()` → `dict[str, Any]` (dashboard-friendly)
+**Hydration**: Concept-based reconstruction in `hydration.py`:
+- Iterate stuff entries in raw dict
+- For each stuff, get `concept.structure_class_name`
+- Look up class via `ClassRegistry.get_required_subclass(name, StuffContent)`
+- Call `the_class.model_validate(content_dict)` — same pattern as `StuffContentFactory.make_stuff_content_from_concept_required()` (line 18-24)
+- Reconstruct typed Stuff objects + WorkingMemory
+
+**Data flow**:
+```
+prepare_for_temporal():
+ smart_dump() → dict[str, Any] (all subclass fields preserved, no Kajson metadata)
+Temporal wire:
+ kajson.dumps(PipeJob) → dict passes through as plain JSON object (no __class__ keys)
+Worker:
+ kajson.loads → dict comes back as plain dict
+Hydrate (after load_from_crate):
+ iterate raw dict → concept.structure_class_name → ClassRegistry → model_validate ✓
+```
+
+### Issue 2 — RESOLVED: CompositeClassRegistry missing methods
+
+Add `setup()` (no-op) and `get_required_base_model()` (local-first, parent-fallback) to the plan spec. Both are trivial delegation methods.
+
+### Issue 3 — RESOLVED: Temporal sandbox imports
+
+New imports (`CompositeClassRegistry`, `KajsonManager`, `hydration`) in WfPipeRouter must go inside the existing `with workflow.unsafe.imports_passed_through():` block.
+
+### Issue 4 — RESOLVED: PipeJob API surface
+
+Verified: only 2 places access `pipe_job.working_memory` — `pipe_router.py:21` (updated in B5) and `wf_pipe_router.py:28` (updated in B7). No other PipeJob callers are affected.
+
+### Issue 5 — RESOLVED: Hydration utility location
+
+Keep as separate `hydration.py` module. Concept-based hydration is ~20 lines, independently testable.
+
+### Issue 6 — RESOLVED: Mutation in prepare_for_temporal()
+
+Keep in-place mutation (plan as-is). PipeJob is single-use, consumed immediately after dispatch.
+
+---
+
+## Test Review
+
+### Code Path Coverage Diagram
+
+```
+CODE PATH COVERAGE
+===========================
+[+] kajson/composite_class_registry.py (NEW)
+ │
+ ├── get_class() / get_required_class() / get_required_subclass()
+ │ ├── [PLANNED] Local hit — test_local_lookup_takes_priority
+ │ ├── [PLANNED] Parent fallback — test_fallback_to_parent
+ │ └── [GAP] Local + parent both miss — should raise/return None
+ │
+ ├── register_class() / register_classes() / register_classes_dict()
+ │ └── [PLANNED] Local only — test_registration_goes_to_local_only
+ │
+ ├── has_class() / has_subclass()
+ │ └── [PLANNED] Both layers — test_has_class_checks_both
+ │
+ ├── teardown()
+ │ └── [PLANNED] Local only — test_teardown_clears_local_only
+ │
+ ├── unregister_class() / unregister_class_by_name()
+ │ └── [GAP] Not in planned tests — needs test for local-only unregistration
+ │
+ └── setup() / get_required_base_model()
+ └── [GAP] Missing from plan spec — needs implementation + test
+
+[+] kajson/kajson_manager.py (ContextVar scoping)
+ │
+ └── get_class_registry() with ContextVar
+ └── [PLANNED] test_scoped_context_var
+
+[+] pipelex/pipe_run/pipe_job.py (modified)
+ │
+ ├── prepare_for_temporal()
+ │ ├── [PLANNED] With crate — test_prepare_for_temporal_moves_wm_to_raw
+ │ ├── [PLANNED] Without crate — test_prepare_for_temporal_noop_without_crate
+ │ └── [PLANNED] Empty WM — test_prepare_for_temporal_empty_wm
+ │
+ └── get_working_memory()
+ ├── [PLANNED] From typed — test_get_working_memory_from_typed
+ ├── [PLANNED] From raw raises — test_get_working_memory_from_raw_raises
+ └── [PLANNED] Both None — test_get_working_memory_both_none_returns_empty
+
+[+] pipelex/temporal/tprl_pipe/hydration.py (NEW)
+ │
+ ├── [PLANNED] Native text — test_hydrate_with_native_text
+ ├── [PLANNED] Empty — test_hydrate_empty
+ ├── [PLANNED] Aliases — test_hydrate_preserves_aliases
+ └── [GAP] Dynamic StructuredContent — needs test with dynamic class
+
+[+] pipelex/temporal/tprl_pipe/wf_pipe_router.py (modified)
+ │
+ ├── Scoped registry creation + cleanup
+ │ └── [PLANNED] Integration test (B8)
+ │
+ ├── Hydration after crate load
+ │ └── [PLANNED] Integration test (B8)
+ │
+ └── Error during hydration
+ └── [GAP] What if hydration fails? Error should propagate cleanly
+
+[+] Integration: dynamic concepts through Temporal
+ │
+ ├── [PLANNED] Full round-trip with dynamic concept — test_wf_deferred_hydration.py
+ ├── [PLANNED] Concurrent workflows with same concept name
+ └── [GAP] Round-trip with ListContent of dynamic items
+
+─────────────────────────────────
+COVERAGE: 15/20 paths planned
+ Planned tests: 15
+ Gaps: 5
+GAPS: 3 need unit tests, 2 need integration tests
+─────────────────────────────────
+```
+
+### Test Gaps
+
+**Gap 1**: CompositeClassRegistry — unregister from local only (unit test)
+**Gap 2**: CompositeClassRegistry — setup() and get_required_base_model() (unit test)
+**Gap 3**: Hydration with dynamic StructuredContent class (unit test)
+**Gap 4**: Hydration failure propagation (integration test)
+**Gap 5**: ListContent of dynamic items round-trip (integration test)
+
+---
+
+## Performance Review
+
+No performance concerns. The changes are:
+- One extra `smart_dump()` call per dispatch (microseconds)
+- Concept-based hydration iterates N stuffs in WM with one registry lookup each (negligible)
+- CompositeClassRegistry adds one dict lookup per class resolution (negligible)
+- ContextVar access is O(1)
+
+---
+
+## Failure Modes
+
+| Codepath | Failure scenario | Test? | Error handling? | User-visible? |
+|---|---|---|---|---|
+| prepare_for_temporal() | smart_dump() fails on WM content | No | No — would crash | Yes — stack trace |
+| hydrate_working_memory() | concept not found in registry after crate load | No | ClassRegistryNotFoundError propagates | Yes — workflow fails |
+| CompositeClassRegistry teardown | Called twice | No | Safe — clears empty dict | Silent |
+| ContextVar not cleaned up | Memory leak of scoped registry | No | finally block handles it | Silent |
+| Concurrent concept name collision | Two workflows register "Greeting" with different fields | PLANNED | Scoped registry isolates | No impact |
+
+**Critical gaps**: 0 (all failure modes either have handling or are edge cases with safe defaults)
+
+---
+
+## What Already Exists
+
+| Existing code | Purpose | Reused by plan? |
+|---|---|---|
+| `_library_id: ContextVar` in hub.py | Per-workflow library scoping | YES — same pattern for registry |
+| `WfPipeRouter.run()` library lifecycle | open/load/teardown per workflow | YES — extended with scoped registry |
+| `BaseModelPayloadConverter` | Kajson-based Temporal serialization | YES — drives raw field design |
+| `ClassRegistryAbstract` | Clean interface for composition | YES — CompositeClassRegistry implements it |
+| `ClassRegistry` | Concrete implementation | YES — used as local layer |
+
+---
+
+## NOT in Scope
+
+| Item | Rationale |
+|---|---|
+| Thread-safe ClassRegistry | Temporal workflows are single-threaded; ContextVar provides isolation |
+| Migration of existing PipeJob callers | No existing callers break — `working_memory` default is still empty WM |
+| Dashboard UI for raw WM | The raw field is visible as JSON in Temporal dashboard; no custom UI needed |
+| Activity-level scoping | Activities don't need class resolution; only workflows load crates |
+
+---
+
+## Completion Summary
+
+- **Step 0: Scope Challenge** — scope accepted as-is (complexity is inherent)
+- **Architecture Review**: 2 issues found — all resolved
+- **Code Quality Review**: 3 issues found — all resolved
+- **Test Review**: diagram produced, 5 gaps identified
+- **Performance Review**: 0 issues found
+- **NOT in scope**: written
+- **What already exists**: written
+- **TODOS.md updates**: 0 items (all handled in-plan)
+- **Failure modes**: 0 critical gaps
+- **Outside voice**: ran (codex failed, Claude subagent used) — 8 points raised, 4 substantive, 0 change plan direction
+- **Lake Score**: 5/5 recommendations chose complete option
+
+---
+
+## Recommended Updates to TODOS.md
+
+Based on this review, the following changes should be made to the plan:
+
+### 1. B3: Use `smart_dump()` not `model_dump(serialize_as_any=True)`
+Replace `self.working_memory.model_dump(serialize_as_any=True)` with `self.working_memory.smart_dump()` in `prepare_for_temporal()`.
+
+### 2. B4: Concept-based hydration (not model_validate)
+Replace the `model_validate()` approach with concept-based reconstruction:
+```python
+def hydrate_working_memory(working_memory_raw: dict[str, Any]) -> WorkingMemory:
+ # For each stuff entry in raw dict:
+ # 1. Reconstruct Concept from concept dict (no dynamic classes needed)
+ # 2. Get structure_class_name from concept
+ # 3. Look up class in ClassRegistry via get_required_subclass()
+ # 4. Call the_class.model_validate(content_dict)
+ # 5. Build typed Stuff objects
+ # Reuses pattern from StuffContentFactory.make_stuff_content_from_concept_required()
+```
+
+### 3. A1: Add missing abstract methods
+Add `setup()` (no-op) and `get_required_base_model()` (local-first, parent-fallback) to CompositeClassRegistry.
+
+### 4. B7: Imports inside unsafe block
+New imports for CompositeClassRegistry, KajsonManager, and hydration must go inside `workflow.unsafe.imports_passed_through()`.
+
+### 5. B8: Structural equality assertions
+Integration test should assert field-by-field on hydrated Stuff content (e.g., `assert stuff.content.score == 0.9`) rather than relying on `==` on dynamic model instances.
+
+### 6. A3: Additional test cases
+Add tests for:
+- `unregister_class()` / `unregister_class_by_name()` from local only
+- `get_required_base_model()` delegation
+- Hydration with dynamic StructuredContent class
+- ListContent of dynamic items round-trip
+
+---
+
+## GSTACK REVIEW REPORT
+
+| Review | Trigger | Why | Runs | Status | Findings |
+|--------|---------|-----|------|--------|----------|
+| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | — | — |
+| Codex Review | `/codex review` | Independent 2nd opinion | 1 | ISSUES_FOUND | 8 points raised, 0 actionable changes (Claude subagent fallback) |
+| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 1 | CLEAR (PLAN) | 6 issues, 0 critical gaps |
+| Design Review | `/plan-design-review` | UI/UX gaps | 0 | — | — |
+
+**UNRESOLVED:** 0 decisions pending
+**VERDICT:** ENG CLEARED — all 6 issues resolved through interactive review
+
+```
++====================================================================+
+| REVIEW READINESS DASHBOARD |
++====================================================================+
+| Review | Runs | Last Run | Status | Required |
+|-----------------|------|---------------------|-----------|----------|
+| Eng Review | 1 | 2026-03-26 02:23 | CLEAR | YES |
+| CEO Review | 0 | — | — | no |
+| Design Review | 0 | — | — | no |
+| Adversarial | 0 | — | — | no |
+| Outside Voice | 1 | 2026-03-26 02:21 | DONE | no |
++--------------------------------------------------------------------+
+| VERDICT: CLEARED — Eng Review passed |
++====================================================================+
+```
+
+All relevant reviews complete. No UI/UX components — design review not needed. Backend infrastructure change — CEO review optional. Ready to implement.
diff --git a/wip/archive/phase3-execution-plan.md b/wip/archive/phase3-execution-plan.md
new file mode 100644
index 000000000..18e9e761b
--- /dev/null
+++ b/wip/archive/phase3-execution-plan.md
@@ -0,0 +1,329 @@
+# Phase 3 Execution Plan: Deferred WorkingMemory Hydration + Scoped ClassRegistry
+
+## Context
+
+Phase 3 solves two problems for Temporal workers with custom concepts:
+1. **Chicken-and-egg deserialization** — WorkingMemory contains dynamic StructuredContent subclasses that don't exist yet when Temporal deserializes PipeJob on the worker
+2. **Global ClassRegistry collision** — concurrent workflows on the same worker could overwrite each other's class registrations
+
+The eng review resolved the hydration approach: use `smart_dump()` → `dict[str, Any]` with concept-based reconstruction (not kajson round-trip).
+
+---
+
+## Implementation Order
+
+### Part A: Kajson (separate repo at `/Users/lchoquel/repos/Pipelex/kajson/`)
+
+#### A1: CompositeClassRegistry
+**New file**: `kajson/kajson/composite_class_registry.py`
+
+A `ClassRegistryAbstract` implementation wrapping local `ClassRegistry` + parent registry:
+- Constructor takes `parent: ClassRegistryAbstract`; creates internal `_local: ClassRegistry`
+- **Getters** (local-first, parent-fallback): `get_class()`, `get_required_class()`, `get_required_subclass()`, `get_required_base_model()`, `has_class()`, `has_subclass()`
+- **Mutators** (local-only): `register_class()`, `register_classes()`, `register_classes_dict()`, `unregister_class()`, `unregister_class_by_name()`
+- **Lifecycle**: `setup()` → no-op, `teardown()` → `_local.teardown()` (never touch parent)
+
+Reference: `kajson/kajson/class_registry.py` for all method signatures and behaviors.
+
+#### A2: ContextVar scoping in KajsonManager
+**Modify**: `kajson/kajson/kajson_manager.py`
+
+```python
+from contextvars import ContextVar
+
+_scoped_class_registry: ContextVar[ClassRegistryAbstract | None] = ContextVar(
+ "_scoped_class_registry", default=None
+)
+
+class KajsonManager(metaclass=MetaSingleton):
+ # existing __init__, get_instance, teardown...
+
+ @classmethod
+ def get_class_registry(cls) -> ClassRegistryAbstract:
+ scoped = _scoped_class_registry.get()
+ if scoped is not None:
+ return scoped
+ return cls.get_instance()._class_registry
+
+ @classmethod
+ def set_scoped_class_registry(cls, registry: ClassRegistryAbstract | None) -> None:
+ _scoped_class_registry.set(registry)
+```
+
+#### A3: Tests for CompositeClassRegistry
+**New file**: `kajson/tests/unit/test_composite_class_registry.py`
+
+Tests (single TestClass):
+- `test_local_lookup_takes_priority` — same name in both, local wins
+- `test_fallback_to_parent` — name only in parent, found via composite
+- `test_registration_goes_to_local_only` — register via composite, parent unchanged
+- `test_teardown_clears_local_only` — teardown, parent untouched
+- `test_has_class_checks_both` — name in parent, `has_class()` returns True
+- `test_unregister_from_local_only` — unregister from composite, parent unaffected
+- `test_get_required_base_model_delegation` — local-first, parent-fallback
+- `test_scoped_context_var` — set/unset scoped registry on KajsonManager
+
+#### A4: Install updated Kajson
+```bash
+cd /Users/lchoquel/repos/Pipelex/_temporal
+pip install -e ../kajson
+```
+
+---
+
+### Part B: Pipelex (at `/Users/lchoquel/repos/Pipelex/_temporal/`)
+
+#### B1: Add PipeJobError
+**Modify**: `pipelex/pipe_run/exceptions.py`
+
+```python
+class PipeJobError(PipelexError):
+ pass
+```
+
+#### B2: Modify PipeJob model
+**Modify**: `pipelex/pipe_run/pipe_job.py`
+
+```python
+from typing import Any
+from pipelex.pipe_run.exceptions import PipeJobError
+
+class PipeJob(BaseModel):
+ pipe: PipeAbstract
+ working_memory: WorkingMemory | None = None
+ working_memory_raw: dict[str, Any] | None = None
+ pipe_run_params: PipeRunParams
+ job_metadata: JobMetadata
+ output_name: str | None = None
+ library_crate: LibraryCrate | None = None
+
+ @property
+ def pipe_type(self) -> str:
+ return self.pipe.__class__.__name__
+
+ def get_working_memory(self) -> WorkingMemory:
+ if self.working_memory is not None:
+ return self.working_memory
+ if self.working_memory_raw is not None:
+ msg = "WorkingMemory is in raw form and has not been hydrated yet"
+ raise PipeJobError(msg)
+ return WorkingMemory()
+
+ def prepare_for_temporal(self) -> None:
+ if self.library_crate is None:
+ return
+ if self.working_memory is not None:
+ self.working_memory_raw = self.working_memory.smart_dump()
+ self.working_memory = None
+```
+
+#### B3: Unit tests for PipeJob
+**New file**: `tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py`
+
+Tests using simple TextContent stuffs (no dynamic classes needed):
+- `test_prepare_for_temporal_moves_wm_to_raw`
+- `test_prepare_for_temporal_noop_without_crate`
+- `test_prepare_for_temporal_empty_wm`
+- `test_get_working_memory_from_typed`
+- `test_get_working_memory_from_raw_raises`
+- `test_get_working_memory_both_none_returns_empty`
+
+Needs a minimal PipeJob fixture — use existing PipeAbstract subclass (e.g., PipeLLM) with mock data, and a LibraryCrate fixture.
+
+#### B4: Hydration utility
+**New file**: `pipelex/temporal/tprl_pipe/hydration.py`
+
+Concept-based hydration following `StuffContentFactory.make_stuff_content_from_concept_required()` pattern:
+
+```python
+from typing import Any
+from pipelex.core.concepts.concept import Concept
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.stuff import Stuff
+from pipelex.core.stuffs.stuff_content import StuffContent
+from pipelex.core.stuffs.stuff_content_factory import StuffContentFactory
+from pipelex.core.stuffs.text_content import TextContent
+
+def hydrate_working_memory(working_memory_raw: dict[str, Any]) -> WorkingMemory:
+ """Reconstruct typed WorkingMemory from a raw dict.
+
+ Must be called AFTER load_from_crate() has registered dynamic classes
+ in the scoped ClassRegistry.
+ """
+ working_memory = WorkingMemory()
+
+ raw_root = working_memory_raw.get("root", {})
+ for stuff_name, stuff_dict in raw_root.items():
+ concept = Concept.model_validate(stuff_dict["concept"])
+ content = StuffContentFactory.make_stuff_content_from_concept_required(
+ concept=concept,
+ value=stuff_dict["content"],
+ )
+ stuff = Stuff(
+ stuff_code=stuff_dict["stuff_code"],
+ stuff_name=stuff_dict.get("stuff_name"),
+ concept=concept,
+ content=content,
+ )
+ working_memory.root[stuff_name] = stuff
+
+ working_memory.aliases = working_memory_raw.get("aliases", {})
+ return working_memory
+```
+
+Key insight: `StuffContentFactory.make_stuff_content_from_concept_required()` already does `concept.structure_class_name` → ClassRegistry lookup → `model_validate()`. We reuse it.
+
+#### B5: Hydration unit tests
+**New file**: `tests/unit/pipelex/temporal/tprl_pipe/test_hydration.py`
+
+Tests (require boot to have concepts loaded):
+- `test_hydrate_with_native_text` — raw dict with TextContent → typed TextContent
+- `test_hydrate_empty` — raw dict for empty WM → empty WM
+- `test_hydrate_preserves_aliases` — aliases survive round-trip
+
+#### B6: Update direct-mode PipeRouter
+**Modify**: `pipelex/pipe_run/pipe_router.py` line 21
+
+```python
+# Before:
+working_memory=pipe_job.working_memory,
+# After:
+working_memory=pipe_job.get_working_memory(),
+```
+
+#### B7: Update Temporal dispatchers
+**Modify**: `pipelex/temporal/tprl_pipe/pipe_router_top.py`
+
+In `_run_pipe_job()`, before `executor.execute_workflow()`:
+```python
+pipe_job.prepare_for_temporal()
+```
+
+**Modify**: `pipelex/temporal/tprl_pipe/pipe_router_child.py`
+
+In `_run_pipe_job()`, before `executor.execute_child_workflow()`:
+```python
+pipe_job.prepare_for_temporal()
+```
+
+#### B8: Update WfPipeRouter — scoped registry + hydration
+**Modify**: `pipelex/temporal/tprl_pipe/wf_pipe_router.py`
+
+Add imports inside `workflow.unsafe.imports_passed_through()`:
+```python
+from kajson.composite_class_registry import CompositeClassRegistry
+from kajson.kajson_manager import KajsonManager
+from pipelex.temporal.tprl_pipe.hydration import hydrate_working_memory
+```
+
+Update `run()`:
+```python
+try:
+ if library_crate is not None:
+ # 1. Create scoped ClassRegistry
+ global_registry = KajsonManager.get_class_registry()
+ scoped_registry = CompositeClassRegistry(parent=global_registry)
+ KajsonManager.set_scoped_class_registry(scoped_registry)
+
+ # 2. Load crate (registers dynamic classes into scoped registry)
+ library_manager = get_library_manager()
+ wf_library_id = f"wf_{workflow.info().workflow_id}"
+ library_manager.open_library(library_id=wf_library_id)
+ set_current_library(library_id=wf_library_id)
+ library_manager.load_from_crate(library_id=wf_library_id, crate=library_crate)
+
+ # 3. Hydrate WorkingMemory
+ if workflow_arg.working_memory_raw is not None:
+ workflow_arg.working_memory = hydrate_working_memory(workflow_arg.working_memory_raw)
+ workflow_arg.working_memory_raw = None
+
+ working_memory = workflow_arg.get_working_memory()
+ pipe_output = await pipe.run_pipe(
+ working_memory=working_memory,
+ ...
+ )
+except ActivityError as exc:
+ ...
+finally:
+ if wf_library_id is not None:
+ get_library_manager().teardown(library_id=wf_library_id)
+ teardown_current_library()
+ if library_crate is not None:
+ KajsonManager.set_scoped_class_registry(None)
+```
+
+#### B9: Integration test — dynamic concepts through Temporal
+**New file**: `tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds`
+
+A test bundle with a custom concept with inline structure, e.g.:
+```toml
+[domain]
+code = "dynamic_concept_test"
+
+[concepts.Greeting]
+structure.message = "str"
+structure.language = "str"
+
+[pipes.greet]
+type = "PipeLLM"
+# ... produces a Greeting
+```
+
+**Add to**: `tests/integration/pipelex/temporal/test_data.py`
+
+```python
+class DeferredHydrationTestData:
+ BUNDLE_FILE: ClassVar[str] = str(Path(__file__).parent / "library_crate" / "dynamic_concept_sequence.mthds")
+ PIPE_CODE: ClassVar[str] = "dynamic_greeting_sequence"
+ DOMAIN: ClassVar[str] = "dynamic_concept_test"
+```
+
+**New file**: `tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py`
+
+Following existing Phase 2 test patterns:
+- Load dynamic concept bundle via `PipelexInterpreter.make_pipelex_bundle_blueprint()`
+- Build PipeJob with WorkingMemory (may be empty or contain input stuff)
+- Execute via Temporal workflow
+- Assert output has correctly typed Stuff with StructuredContent (field-by-field assertions)
+
+#### B10: Lint and test
+```bash
+make agent-check
+make agent-test
+```
+
+---
+
+## File Summary
+
+| File | Change |
+|------|--------|
+| **Kajson** | |
+| `kajson/kajson/composite_class_registry.py` | **NEW** — CompositeClassRegistry |
+| `kajson/kajson/kajson_manager.py` | Add ContextVar scoping |
+| `kajson/tests/unit/test_composite_class_registry.py` | **NEW** — unit tests |
+| **Pipelex** | |
+| `pipelex/pipe_run/exceptions.py` | Add PipeJobError |
+| `pipelex/pipe_run/pipe_job.py` | Add raw field, get_working_memory(), prepare_for_temporal() |
+| `pipelex/pipe_run/pipe_router.py` | Use get_working_memory() |
+| `pipelex/temporal/tprl_pipe/hydration.py` | **NEW** — concept-based hydration |
+| `pipelex/temporal/tprl_pipe/pipe_router_top.py` | Call prepare_for_temporal() |
+| `pipelex/temporal/tprl_pipe/pipe_router_child.py` | Call prepare_for_temporal() |
+| `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Scoped registry + hydration |
+| `tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py` | **NEW** |
+| `tests/unit/pipelex/temporal/tprl_pipe/test_hydration.py` | **NEW** |
+| `tests/integration/pipelex/temporal/test_data.py` | Add DeferredHydrationTestData |
+| `tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds` | **NEW** |
+| `tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py` | **NEW** |
+
+---
+
+## Verification
+
+1. Kajson unit tests pass: `test_composite_class_registry.py`
+2. PipeJob unit tests pass: `test_pipe_job_hydration.py`
+3. Hydration unit tests pass: `test_hydration.py`
+4. Existing Phase 2 tests still pass: `test_wf_library_crate.py`
+5. New integration test passes: `test_wf_deferred_hydration.py`
+6. `make agent-check` passes
+7. `make agent-test` passes
diff --git a/wip/archive/phase3-mode2-regression.md b/wip/archive/phase3-mode2-regression.md
new file mode 100644
index 000000000..e287f4490
--- /dev/null
+++ b/wip/archive/phase3-mode2-regression.md
@@ -0,0 +1,181 @@
+# Phase 3 Regression: Deferred Hydration Fails in 3-Process Mode
+
+## Problem Statement
+
+Temporal Phase 3 (Deferred WorkingMemory Hydration) passes all tests in Mode 1
+(pytest with in-process worker) but fails in Mode 2 (true 3-process: server + worker
++ submitter) with:
+
+```
+KajsonDecoderError: Class 'dynamic_concept_test__Greeting' not found in module 'builtins'
+```
+
+This is a regression because Phase 3 was marked complete in `wip/00-master-plan.md:208`.
+
+## How to Reproduce
+
+Use the `/temporal-e2e-validate` skill (defined in `.claude/skills/temporal-e2e-validate/`).
+It has two modes:
+
+- **Mode 1** (pytest): All tests pass — confirms Phase 3 logic works in-process.
+- **Mode 2** (3-process): Tier 2 (deferred hydration) hangs, worker shows the error above.
+
+Quick reproduction of Mode 2 failure (assumes Temporal dev server on localhost:7233):
+
+```bash
+# Terminal 1: Temporal server
+temporal server start-dev
+
+# Terminal 2: Worker (no PIPELEXPATH — bundles arrive only via crate)
+.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed
+
+# Terminal 3: Submitter — this hangs because the worker fails to deserialize
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \
+ --pipe dynamic_greeting_sequence \
+ --temporal --dry-run --mock-inputs --no-logo
+```
+
+The worker log will show `KajsonDecoderError: Class 'dynamic_concept_test__Greeting'
+not found in module 'builtins'` during child workflow result deserialization.
+
+## Root Cause Analysis
+
+### The deserialization chain
+
+When a parent WfPipeRouter dispatches a child WfPipeRouter (for each step in a
+PipeSequence), the child returns a `PipeOutput` containing `StructuredContent` with
+a dynamically generated class (e.g. `dynamic_concept_test__Greeting`).
+
+Temporal serializes this return value via `kajson.dumps()`, which records
+`__class__: "dynamic_concept_test__Greeting"` and `__module__: "builtins"`.
+The `__module__` is `"builtins"` because dynamically generated classes (created via
+`type()` in ConceptFactory) get `builtins` as their module by default.
+
+The parent workflow then deserializes via the data converter:
+
+```
+_apply_resolve_child_workflow_execution (Temporal SDK internal callback)
+ → _convert_payloads
+ → BaseModelPayloadConverter.from_payload()
+ → kajson.loads(data, class_registry=get_class_registry())
+```
+
+### Why Mode 1 works but Mode 2 fails
+
+The data converter calls `pipelex.hub.get_class_registry()`, which checks the
+`_library_id` ContextVar:
+
+```python
+# pipelex/hub.py:381-392
+def get_class_registry() -> ClassRegistryAbstract:
+ library_id = _library_id.get()
+ if library_id is not None:
+ registry = get_library_manager().get_library_class_registry(library_id)
+ if registry is not None:
+ return registry
+ return KajsonManager.get_class_registry()
+```
+
+**The critical problem**: `_apply_resolve_child_workflow_execution` is a callback
+invoked by the Temporal SDK's internal event processor, NOT from within the workflow
+coroutine's `run()` method. The `_library_id` ContextVar was set inside `run()`,
+but Temporal's event processing does not execute in the same ContextVar scope. So
+`_library_id.get()` returns `None`, and `get_class_registry()` falls back to the
+global KajsonManager registry.
+
+**Mode 1 (in-process)**: The test fixture `pipe_job_from_bundle()`
+(`tests/integration/pipelex/fixtures/pipe_job_helpers.py:73-85`) loads the bundle
+into a Library. During loading, ConceptFactory registers dynamic classes via
+`hub.get_class_registry()`. The Library created by `open_library()` has no
+ClassRegistry set (unlike the workflow which explicitly calls
+`wf_library.set_class_registry()`), so `get_library_class_registry()` returns None,
+and classes are registered in the **global** KajsonManager registry. When the data
+converter falls back to the global registry, it finds the classes. **The test fixture
+masks the bug by pre-populating the global registry.**
+
+**Mode 2 (3-process)**: The worker process starts with no bundles loaded. The global
+KajsonManager registry has no dynamic classes. WfPipeRouter creates a per-workflow
+ClassRegistry and loads the crate into it, but the data converter can't access it
+(ContextVar not set in Temporal's event processing context). The global registry
+lookup fails, then `sys.modules["builtins"]` lookup fails → `KajsonDecoderError`.
+
+### Why the ContextVar approach doesn't work here
+
+The Temporal Python SDK's `_workflow_instance.py` processes events (activity results,
+child workflow results) via internal callbacks. These callbacks call `_convert_payloads`
+which invokes the data converter. This call chain runs in the SDK's internal context,
+not in the coroutine context of the workflow's `run()` method where `set_current_library()`
+was called. Python ContextVars set inside a coroutine are not automatically visible
+to SDK-internal callbacks that run outside that coroutine's execution frame.
+
+## What Was Planned vs What Was Implemented
+
+The original execution plan (`wip/phase3-execution-plan.md`) specified two mechanisms
+that were not implemented:
+
+### 1. CompositeClassRegistry (plan step A1-A3)
+
+**Planned**: A new `ClassRegistryAbstract` implementation in kajson that wraps a local
+registry + parent registry. Lookups check local first, fall back to parent. Mutations
+go to local only.
+
+**Actual**: Not implemented. The file `kajson/kajson/composite_class_registry.py` does
+not exist. Instead, the workflow creates a plain `ClassRegistry` and pre-seeds it from
+the global registry's entries:
+
+```python
+# wf_pipe_router.py:37-41
+global_registry = KajsonManager.get_class_registry()
+workflow_registry = ClassRegistry()
+if isinstance(global_registry, ClassRegistry):
+ workflow_registry.register_classes_dict(dict(global_registry.root))
+```
+
+This pre-seeding approach works for isolation but doesn't help with the data converter
+problem — the data converter still can't reach the workflow-scoped registry.
+
+### 2. ContextVar scoping in KajsonManager (plan step A2)
+
+**Planned**: A `_scoped_class_registry` ContextVar in KajsonManager itself, checked
+by `KajsonManager.get_class_registry()` before returning the global registry. This
+would allow setting a scoped registry that kajson's decoder would automatically use.
+
+**Actual**: Not implemented. `KajsonManager.get_class_registry()` in the kajson repo
+(`kajson/kajson/kajson_manager.py`) has no ContextVar — it always returns the singleton's
+global registry. The scoping was implemented at the pipelex hub level via `_library_id`
+ContextVar, which doesn't work for Temporal's internal data converter calls.
+
+## Key Files
+
+| File | Role in the problem |
+|------|---------------------|
+| `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Sets up per-workflow registry, loads crate, hydrates WM |
+| `pipelex/temporal/temporal_data_converter.py:77` | Calls `get_class_registry()` during deserialization |
+| `pipelex/hub.py:381-392` | `get_class_registry()` uses `_library_id` ContextVar |
+| `pipelex/hub.py:462-467` | `_library_id` ContextVar + `set_current_library()` |
+| `pipelex/pipe_run/pipe_job.py:34-49` | `prepare_for_temporal()` serializes WM to raw dict |
+| `pipelex/temporal/tprl_pipe/hydration.py` | `hydrate_working_memory()` — works correctly |
+| `/Users/lchoquel/repos/Pipelex/kajson/kajson/kajson_manager.py` | Global registry, no ContextVar scoping |
+| `/Users/lchoquel/repos/Pipelex/kajson/kajson/json_decoder.py:141-161` | Decoder: explicit registry → sys.modules → error |
+| `tests/integration/pipelex/fixtures/pipe_job_helpers.py:73-85` | Test fixture that masks the bug |
+| `wip/phase3-execution-plan.md` | Original plan with CompositeClassRegistry + KajsonManager ContextVar |
+
+## Scope of Impact
+
+All bundles with dynamic concept classes (inline `structure` in `.mthds`) fail in
+3-process mode. Bundles using only native types (Text) work fine. Affected tests:
+
+- Tier 2: `dynamic_concept_sequence.mthds` — direct Phase 3 validation
+- Tier 3: `temporal_parallel.mthds` — uses ToneAnalysis/LengthAnalysis concepts
+- All concurrent concept isolation tests — use conflicting Result/Profile/Summary concepts
+
+PipeParallel with native text works. Concurrent pipe isolation (native text) works.
+
+## Fix Applied During This Session
+
+A separate fixture ordering bug was fixed in `tests/integration/pipelex/temporal/conftest.py`:
+`boot_temporal` could run before `reset_pipelex_config_fixture` when using `--temporal-server local`,
+because `_session_owns_pipelex()` caused the module-scoped fixture to skip initialization while
+the session-scoped `env` fixture (which was supposed to init Pipelex) is not autouse. Fixed by
+always initializing Pipelex in the module-scoped fixture. This fix is uncommitted.
diff --git a/wip/archive/phase4-explicit-class-registry.md b/wip/archive/phase4-explicit-class-registry.md
new file mode 100644
index 000000000..3e89bc783
--- /dev/null
+++ b/wip/archive/phase4-explicit-class-registry.md
@@ -0,0 +1,211 @@
+# Plan: Explicit ClassRegistry — No Singleton Scoping in Kajson
+
+## Context
+
+Phase 3 (deferred WorkingMemory hydration + scoped ClassRegistry) works with in-process tests but fails with separate Temporal workers due to two bugs:
+
+1. **Decoder bypass**: `json_decoder.py:140-150` never reaches the ClassRegistry fallback for dynamic classes because `builtins` is always in `sys.modules`
+2. **Teardown clobber**: `wf_pipe_router.py` sets the scoped registry to `None` in `finally`, which isn't stack-safe for nested workflows
+
+Both bugs stem from the same architectural weakness: kajson (a serialization library) owns workflow-scoping concerns via a ContextVar singleton. The fix is to move scoping out of kajson entirely, tie the ClassRegistry to the Library lifecycle (where it logically belongs), and give kajson an explicit `class_registry` parameter.
+
+## Design Principles
+
+- **Kajson is a pure library**: no ContextVars, no workflow scoping. Accepts explicit `class_registry` parameter.
+- **Library owns its ClassRegistry**: each Library (loaded from a crate) gets a pre-seeded ClassRegistry containing global + dynamic classes. No CompositeClassRegistry needed — just a flat dict.
+- **Reuse existing `_library_id` ContextVar**: `hub.get_class_registry()` reads the active library_id, gets that library's ClassRegistry. No new ContextVar.
+- **Explicit at the boundary**: `temporal_data_converter` passes the registry explicitly to `kajson.loads()`.
+
+## Changes
+
+### 1. Kajson — Add explicit `class_registry` param, remove scoping
+
+**`kajson/kajson.py`** — `loads()` and `load()` accept optional `class_registry`
+```python
+def loads(json_string, class_registry=None, **kwargs):
+ if class_registry is not None:
+ kwargs["class_registry"] = class_registry
+ return json.loads(json_string, cls=UniversalJSONDecoder, **kwargs)
+```
+Same for `load()`.
+
+**`kajson/json_decoder.py`** — Decoder uses explicit registry first
+```python
+def __init__(self, *args, **kwargs):
+ self._class_registry = kwargs.pop("class_registry", None)
+ json.JSONDecoder.__init__(self, object_hook=self.universal_decoder, *args, **kwargs)
+ self.logger = logging.getLogger(DECODER_LOGGER_CHANNEL_NAME)
+```
+
+In `universal_decoder()` class resolution (lines 139-172), add a new first step:
+```python
+# Step 0: Check explicit registry first (handles dynamic classes)
+if self._class_registry is not None:
+ registered_class = self._class_registry.get_class(name=class_name)
+ if registered_class is not None:
+ the_class = registered_class
+ # skip to decoder strategies (line 174)
+```
+If not found in explicit registry, fall through to existing logic (sys.modules → global registry → importlib).
+
+**`kajson/kajson_manager.py`** — Remove ContextVar and scoped methods
+- Delete `_scoped_class_registry` ContextVar
+- Delete `set_scoped_class_registry()` method
+- `get_class_registry()` simply returns `cls.get_instance()._class_registry` (global only)
+
+**`kajson/composite_class_registry.py`** — Keep file, stop using it. Can be removed in a follow-up.
+
+### 2. Pipelex — Library owns its ClassRegistry (PrivateAttr)
+
+**`pipelex/libraries/library.py`** — Add ClassRegistry as a PrivateAttr on Library:
+```python
+from pydantic import PrivateAttr
+from kajson.class_registry import ClassRegistry
+
+class Library(BaseModel):
+ # ... existing fields ...
+ _class_registry: ClassRegistry | None = PrivateAttr(default=None)
+
+ def get_class_registry(self) -> ClassRegistry | None:
+ return self._class_registry
+
+ def set_class_registry(self, class_registry: ClassRegistry) -> None:
+ self._class_registry = class_registry
+```
+
+PrivateAttr keeps it off the Pydantic serialization path (no impact on `model_dump()`, validation, or crate serialization). The registry lives and dies with the Library — when `teardown()` deletes the Library from `_libraries`, the registry is GC'd with it. No parallel dicts, no sync risk, no memory leaks.
+
+**`pipelex/libraries/library_manager.py`** — Accessor for convenience:
+```python
+def get_library_class_registry(self, library_id: str) -> ClassRegistry | None:
+ library = self._libraries.get(library_id)
+ if library is not None:
+ return library.get_class_registry()
+ return None
+```
+
+No changes needed in `teardown()` — the Library's deletion handles cleanup automatically.
+
+### 3. Pipelex — `hub.get_class_registry()` reads from Library
+
+**`pipelex/hub.py:381-388`** — Replace KajsonManager delegation with library lookup:
+```python
+def get_class_registry() -> ClassRegistryAbstract:
+ library_id = _library_id.get()
+ if library_id is not None:
+ registry = get_library_manager().get_library_class_registry(library_id)
+ if registry is not None:
+ return registry
+ return get_pipelex_hub().get_required_class_registry()
+```
+
+Remove `from kajson.kajson_manager import KajsonManager` from hub.py if no longer needed (check other usages first).
+
+### 4. Pipelex — wf_pipe_router uses plain ClassRegistry
+
+**`pipelex/temporal/tprl_pipe/wf_pipe_router.py`** — Rewrite scoping:
+
+```python
+# Remove CompositeClassRegistry and KajsonManager imports
+# Add: from kajson.class_registry import ClassRegistry
+
+try:
+ if library_crate is not None:
+ # 1. Create per-workflow ClassRegistry pre-seeded from global
+ global_registry = get_pipelex_hub().get_required_class_registry()
+ workflow_registry = ClassRegistry()
+ workflow_registry.register_classes_dict(dict(global_registry.root)) # copy global entries
+
+ # 2. Open library and attach registry to it
+ library_manager = get_library_manager()
+ wf_library_id = f"wf_{workflow.info().workflow_id}"
+ _wf_library_id, wf_library = library_manager.open_library(library_id=wf_library_id)
+ wf_library.set_class_registry(workflow_registry)
+ set_current_library(library_id=wf_library_id)
+
+ # 3. Load crate (registers dynamic classes into workflow_registry via hub.get_class_registry())
+ library_manager.load_from_crate(library_id=wf_library_id, crate=library_crate)
+
+ # 4. Hydrate WorkingMemory
+ if workflow_arg.working_memory_raw is not None:
+ workflow_arg.working_memory = hydrate_working_memory(workflow_arg.working_memory_raw)
+ workflow_arg.working_memory_raw = None
+
+ # ... run pipe ...
+
+finally:
+ if wf_library_id is not None:
+ get_library_manager().teardown(library_id=wf_library_id) # cleans up registry too
+ teardown_current_library()
+ # No KajsonManager.set_scoped_class_registry(None) needed!
+```
+
+Key insight: `set_current_library()` sets `_library_id` ContextVar. From that point, all calls to `hub.get_class_registry()` return `workflow_registry`. When `concept_factory` calls `KajsonManager.get_class_registry().register_class(...)`, it still hits the global. We need to migrate those callers.
+
+### 5. Migrate KajsonManager.get_class_registry() callers to hub
+
+All ~20 call sites in Pipelex that use `KajsonManager.get_class_registry()` must switch to `hub.get_class_registry()` (or receive the registry explicitly). This is a mechanical change.
+
+**Files to update** (import `get_class_registry` from `pipelex.hub` instead of `KajsonManager` from `kajson`):
+
+| File | Lines | Change |
+|------|-------|--------|
+| `pipelex/core/concepts/concept_factory.py` | 323, 359, 405, 435 | `get_class_registry().register_class(...)` |
+| `pipelex/core/concepts/concept.py` | 132, 133, 162, 165, 175, 183 | `get_class_registry().get_class(...)` etc. |
+| `pipelex/core/concepts/structure_generation/generator.py` | 585 | `get_class_registry().get_class(...)` |
+| `pipelex/core/pipes/pipe_factory.py` | 110 | `get_class_registry().get_required_subclass(...)` |
+| `pipelex/libraries/library_manager.py` | 1134, 1165 | `get_class_registry()` |
+| `pipelex/system/registries/class_registry_utils.py` | 37, 148 | `get_class_registry()` |
+| `pipelex/tools/typing/structure_printer.py` | 26 | `get_class_registry()` |
+| `pipelex/pipe_operators/func/pipe_func.py` | 104 | `get_class_registry().get_class(...)` |
+| `pipelex/cli/commands/build/structures_cmd.py` | 194, 243, 270 | `get_class_registry().register_class(...)` |
+
+Each file: replace `KajsonManager.get_class_registry()` with `get_class_registry()` imported from `pipelex.hub`. Remove unused `KajsonManager` import if nothing else uses it.
+
+### 6. Temporal data converter — explicit parameter
+
+**`pipelex/temporal/temporal_data_converter.py`** — Pass registry to `kajson.loads()`:
+
+```python
+from pipelex.hub import get_class_registry
+
+def _kajson_deserialize_from_payload(self, payload: Payload) -> Any:
+ data = payload.data.decode()
+ log.verbose(f"unijson_deserialize_payload — data: {data}")
+ pydantic_gizmo = kajson.loads(data, class_registry=get_class_registry())
+ log.verbose(f"unijson_deserialize_payload — pydantic_gizmo: {pydantic_gizmo}")
+ return pydantic_gizmo
+```
+
+### 7. Tests
+
+**Kajson tests** (`tests/unit/test_composite_class_registry.py`):
+- Update `test_scoped_context_var` — this test uses `set_scoped_class_registry` which is removed. Replace with a test that verifies `kajson.loads(data, class_registry=custom_reg)` resolves from the explicit registry.
+
+**Pipelex tests** (`tests/unit/pipelex/temporal/tprl_pipe/test_hydration.py`):
+- Update to reflect new ClassRegistry setup (no CompositeClassRegistry)
+
+**Pipelex tests** (`tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py`):
+- Verify no breakage
+
+**New test**: test that `kajson.loads()` with explicit `class_registry` finds dynamic classes even when `__module__` is `"builtins"` (the exact bug scenario).
+
+## Verification
+
+1. **Lint + type check**: `make agent-check` in both kajson and _temporal
+2. **Unit tests**: `make agent-test` in both repos
+3. **Manual Temporal test**: Use `/temporal-diagnose` with 3-process setup (server + separate worker + submitter) using `dynamic_concept_sequence.mthds` — this is the test that currently fails with `KajsonDecoderError: Class 'Greeting' not found in module 'builtins'`
+
+## Key Files
+
+| Repo | File | Purpose |
+|------|------|---------|
+| kajson | `kajson/kajson.py` | Add `class_registry` param to `loads()`/`load()` |
+| kajson | `kajson/json_decoder.py` | Accept + use explicit registry in decoder |
+| kajson | `kajson/kajson_manager.py` | Remove ContextVar, simplify to global-only |
+| _temporal | `pipelex/hub.py` | `get_class_registry()` reads from library |
+| _temporal | `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Plain ClassRegistry, library-associated |
+| _temporal | `pipelex/temporal/temporal_data_converter.py` | Pass explicit registry to `kajson.loads()` |
+| _temporal | `pipelex/libraries/library.py` | PrivateAttr ClassRegistry on Library |
+| _temporal | `pipelex/libraries/library_manager.py` | Accessor for library's ClassRegistry |
+| _temporal | ~9 files | Migrate `KajsonManager.get_class_registry()` → `hub.get_class_registry()` |
diff --git a/wip/archive/phase4.5-distributed-tracing-implementation.md b/wip/archive/phase4.5-distributed-tracing-implementation.md
new file mode 100644
index 000000000..b9ed581d5
--- /dev/null
+++ b/wip/archive/phase4.5-distributed-tracing-implementation.md
@@ -0,0 +1,613 @@
+phase 4.5 of wip/00-master-plan.md
+
+# Distributed Tracing & Reporting — Local Version
+
+> Implementation plan for the NDJSON-file-based event log that enables graph tracing and cost reporting across Temporal workers.
+> Design rationale and cloud architecture (DynamoDB) in [wip/distributed-tracing-and-reporting.md](wip/distributed-tracing-and-reporting.md).
+
+---
+
+## Overview
+
+Add a **file-based event log** to the existing `GraphTracer`, for use when running with Temporal distributed workers. The in-memory path is preserved unchanged for direct mode (no Temporal) — it's already optimal for single-process execution.
+
+**Approach: dual-write in GraphTracer.** When an `EventLogProtocol` is provided, `GraphTracer` emits events as a side effect alongside its existing in-memory accumulation. No new tracer class, no new hub functions, no call site changes. `pipe_abstract.py` and controllers continue calling `GraphTracerManager` as before.
+
+**Two modes, selected at tracer open time:**
+
+| Mode | What happens in GraphTracer | Reporting | When |
+|---|---|---|---|
+| **In-memory only** (default) | Existing accumulation, `teardown()` → `GraphSpec` | Existing `ReportingManager` | Direct execution (no Temporal) |
+| **In-memory + event log** | Same accumulation + `emit()` to NDJSON per method call | `ReportingManager` + event emission from activities | Temporal distributed execution |
+
+In direct mode, `open_tracer()` is called without an event log → `GraphTracer` works as today. In Temporal mode, `WfPipeRouter` creates an `NdjsonEventLog` and passes it to `open_tracer()` → `GraphTracer` dual-writes. After all workflows complete, `runner.py` reads events from all files and assembles a cross-worker `GraphSpec`.
+
+**Event log backend**: NDJSON files. One file per workflow (or per activity for usage events), organized in a directory per pipeline run:
+```
+{traces_dir}/{pipeline_run_id}/
+ ├── wf_{workflow_id_A}.ndjson
+ ├── wf_{workflow_id_B}.ndjson
+ ├── act_{activity_id_X}.ndjson # Usage events from activities
+ └── ...
+```
+
+Each line is a JSON-serialized `TraceEvent`. Workflow files have exactly one writer. Activity files have one writer per activity execution.
+
+**Protocol**: `EventLogProtocol` with `emit()` / `read_events()`. The NDJSON backend is the local implementation. Cloud implementation (DynamoDB) will implement the same protocol later.
+
+**Key design invariant**: `emit()` MUST be synchronous (file append + flush). This guarantees all events are flushed before a child workflow returns. If emit were async or batched, the assembler could read incomplete files. This invariant must be preserved in any future backend implementation.
+
+---
+
+## Step 0: Event Models
+
+> Pure data models — no integration, no refactoring. TDD: write serialization tests first.
+
+### What
+
+Define the `TraceEvent` Pydantic model hierarchy that replaces the in-memory `_MutableNodeData` accumulation. Events capture the same information that `GraphTracer.on_pipe_start()`, `on_pipe_end_success()`, `on_pipe_end_error()`, and the controller-specific registration methods currently capture — but as immutable, serializable records.
+
+### Event type inventory
+
+Map every method on `GraphTracerManager` (see `pipelex/graph/graph_tracer_manager.py`) and `ReportingManager` (see `pipelex/reporting/reporting_manager.py`) to an event type:
+
+| Current method | Event type | Payload (beyond base fields) |
+|---|---|---|
+| `on_pipe_start()` | `PipeStartEvent` | node_id, parent_node_id, pipe_code, pipe_type, node_kind, input_specs |
+| `on_pipe_end_success()` | `PipeEndSuccessEvent` | node_id, ended_at, output_spec, metrics |
+| `on_pipe_end_error()` | `PipeEndErrorEvent` | node_id, ended_at, error (ErrorSpec) |
+| `add_edge()` | `EdgeEvent` | source_node_id, target_node_id, edge_kind, label, source/target_stuff_digest |
+| `register_controller_output()` | `ControllerOutputEvent` | node_id, output_spec |
+| `register_batch_item_extraction()` | `BatchItemEvent` | list_stuff_code, item_stuff_code, item_index, batch_controller_node_id |
+| `register_batch_aggregation()` | `BatchAggregateEvent` | output_list_stuff_code, item_stuff_code, item_index, batch_controller_node_id |
+| `register_parallel_combine()` | `ParallelCombineEvent` | combined_stuff_code, branch_stuff_codes, parallel_controller_node_id |
+| `report_inference_job()` | `UsageReportEvent` | node_id (which pipe), tokens_usage (the union type) |
+
+Base fields shared by all events:
+- `pipeline_run_id: str`
+- `workflow_id: str` — Temporal workflow ID or `"direct"` for in-process
+- `event_kind: TraceEventKind` (discriminator)
+- `timestamp: datetime` — UTC, for display/debugging only
+- `sequence: int` — per-writer monotonic counter, for ordering and deduplication
+
+### Node ID scheme
+
+Change from `{graph_id}:node_{sequence}` to `{graph_id}:{workflow_id}:node_{sequence}`.
+
+`workflow_id` is the full Temporal workflow ID, or `"direct"` for in-process mode. Combined with the per-writer sequence counter, this guarantees globally unique node IDs without coordination. Using the full workflow ID avoids collision — the last 8 chars of Temporal workflow IDs often come from the base ID (e.g. `pipe_router`) and repeat across workflows.
+
+Edge IDs follow the same scheme: `{graph_id}:{workflow_id}:edge_{sequence}`.
+
+The sequence counter is **per-writer** (per workflow/file), starting at 0. No need to coordinate across workers.
+
+**Renderer impact**: Mermaid/ReactFlow renderers that parse node IDs by splitting on `:` must handle the additional segment. Node IDs are opaque strings in most code — only renderers and test assertions need updating.
+
+### Files to create
+
+| File | Contents |
+|---|---|
+| `pipelex/tracing/__init__.py` | Empty |
+| `pipelex/tracing/trace_events.py` | `TraceEventKind` enum, `TraceEvent` base model, all event subclasses |
+| `tests/unit/pipelex/tracing/__init__.py` | Empty |
+| `tests/unit/pipelex/tracing/test_trace_events.py` | Serialization round-trip tests for each event type |
+
+### Design notes
+
+- Use Pydantic's discriminated union (`Annotated[... , Field(discriminator="event_kind")]`) so `TraceEvent` can be deserialized from JSON without knowing the subclass upfront.
+- `IOSpec` and `ErrorSpec` from `pipelex/graph/graphspec.py` are reused as-is — they're already Pydantic models designed for JSON serialization.
+- `TokensUsage` is the existing union type `LLMTokensUsage | ImgGenTokensUsage | ExtractTokensUsage | SearchTokensUsage` from `reporting_manager.py`. Reuse it. It needs a discriminated union wrapper for deserialization: `AnyTokensUsage = Annotated[TokensUsage, Field(discriminator="model_type")]`. **Prerequisite**: each TokensUsage model's `model_type` field must be changed from `str` to `Literal` (e.g., `model_type: Literal["llm"] = "llm"` instead of `model_type: str = "llm"`). This is a one-line change per model in `llm_report.py`, `img_gen_report.py`, `extract_report.py`, `search_report.py`.
+- `TraceEventKind` inherits from `StrEnum` (imported from `pipelex.types`).
+
+### Checklist
+
+- [x] Define `TraceEventKind` enum with all event kinds
+- [x] Define `TraceEvent` base model with shared fields
+- [x] Define all event subclasses (`PipeStartEvent`, `PipeEndSuccessEvent`, `PipeEndErrorEvent`, `EdgeEvent`, `ControllerOutputEvent`, `BatchItemEvent`, `BatchAggregateEvent`, `ParallelCombineEvent`, `UsageReportEvent`)
+- [x] Define discriminated union type `AnyTraceEvent` for deserialization
+- [x] Write serialization round-trip tests for each event type
+- [x] Write test for discriminated union deserialization (JSON → correct subclass)
+- [x] Verify `IOSpec`, `ErrorSpec`, `TimingSpec` survive JSON round-trip within events
+- [x] Fix `model_type` field to `Literal` on `LLMTokensUsage`, `ImgGenTokensUsage`, `ExtractTokensUsage`, `SearchTokensUsage`
+- [x] Define `AnyTokensUsage` discriminated union using `Field(discriminator="model_type")`
+- [x] Verify `TokensUsage` union survives round-trip with discriminated union deserialization
+- [x] `make agent-check` passes
+
+---
+
+## Step 1: EventLogProtocol & NDJSON Backend
+
+> The write/read infrastructure. TDD: test emit/read cycle first.
+
+### What
+
+Define the `EventLogProtocol` and implement the NDJSON file backend. This is the storage layer — it knows nothing about graphs or reporting, only about appending and reading `TraceEvent` objects.
+
+### EventLogProtocol
+
+```python
+class EventLogProtocol(Protocol):
+ def emit(self, event: TraceEvent) -> None: ...
+ def read_events(self, pipeline_run_id: str) -> list[TraceEvent]: ...
+ def cleanup(self, pipeline_run_id: str) -> None: ...
+```
+
+`emit()` is **synchronous** — file append is fast and must not become a Temporal replay point. The event is durable when `emit()` returns (file flush).
+
+**Critical invariant**: `emit()` must remain synchronous with an explicit file flush. This guarantees all events are written before a child workflow returns to its parent. If emit were async or batched, the assembler could read incomplete files from child workflows. This invariant must be preserved in any future backend implementation (DynamoDB, etc.).
+
+`read_events()` returns all events for a pipeline run, ordered by `(workflow_id, sequence)`. Used by the assembler post-execution.
+
+`cleanup()` removes the run's directory. Called after graph outputs are generated.
+
+### NDJSON implementation: `NdjsonEventLog`
+
+**Write path** (`emit`):
+1. Determine file path: `{traces_dir}/{event.pipeline_run_id}/wf_{event.workflow_id}.ndjson`
+2. Create directory if needed (`os.makedirs(..., exist_ok=True)`)
+3. Open file in append mode, write `kajson.dumps(event) + "\n"`, flush
+
+File handle caching: keep a `dict[str, IO]` of open file handles keyed by `(pipeline_run_id, workflow_id)`. Close handles on `cleanup()` or when the writer is garbage-collected. This avoids repeated `open()` calls for high-frequency events.
+
+**Read path** (`read_events`):
+1. Glob `{traces_dir}/{pipeline_run_id}/*.ndjson` (includes `wf_*` and `act_*` files)
+2. For each file, read all lines, parse each as `AnyTraceEvent` (discriminated union)
+ - **Corrupt line handling**: Wrap each line parse in `try/except JSONDecodeError`. Log a warning with the file path and line number, then skip the corrupt line. This handles crash-mid-write scenarios where NDJSON has no journaling.
+3. Deduplicate by `(workflow_id, sequence)` — handles Temporal replay re-emission. When duplicates exist, keep the first occurrence (they are identical for correct replays).
+4. Sort by `(workflow_id, sequence)` for deterministic ordering
+5. Return flat list
+
+**Replay growth note**: When Temporal replays a workflow, events are re-appended with the same `(workflow_id, sequence)` pairs. The NDJSON file grows with each replay. This is functionally correct (dedup on read handles it) but wastes disk. This is an accepted tradeoff for the simplicity of append-only writes. SQLite's `INSERT OR IGNORE` would prevent duplicate writes entirely, but NDJSON is simpler and sufficient for local dev.
+
+**Cleanup path** (`cleanup`):
+1. Close any cached file handles for this run
+2. `shutil.rmtree({traces_dir}/{pipeline_run_id})`
+
+**Important**: Traces are never auto-deleted after graph assembly. Cleanup must be explicitly called by the user or a separate cleanup command. This ensures traces are available for debugging if assembly fails, and for usage reporting which may run after graph generation.
+
+### InMemoryEventLog
+
+For unit tests and direct mode (optional — direct mode can also use NDJSON):
+- `emit()` appends to `list[TraceEvent]`
+- `read_events()` returns the list filtered by pipeline_run_id
+- No file I/O
+
+### Files to create
+
+| File | Contents |
+|---|---|
+| `pipelex/tracing/event_log_protocol.py` | `EventLogProtocol` |
+| `pipelex/tracing/ndjson_event_log.py` | `NdjsonEventLog` implementation |
+| `pipelex/tracing/in_memory_event_log.py` | `InMemoryEventLog` for tests |
+| `tests/unit/pipelex/tracing/test_ndjson_event_log.py` | Emit/read/cleanup/dedup tests |
+
+### Checklist
+
+- [x] Define `EventLogProtocol`
+- [x] Implement `NdjsonEventLog` write path (emit with file handle caching)
+- [x] Implement `NdjsonEventLog` read path (glob, parse, deduplicate, sort)
+- [x] Implement `NdjsonEventLog` cleanup
+- [x] Implement `InMemoryEventLog`
+- [x] Test: emit events, read them back, verify order and content
+- [x] Test: emit duplicate events (same workflow_id + sequence), verify deduplication
+- [x] Test: emit from "multiple workers" (different workflow_ids), verify all events returned
+- [x] Test: cleanup removes directory
+- [x] Test: read from nonexistent run returns empty list
+- [x] Test: corrupt NDJSON line (truncated JSON) is skipped with warning
+- [x] Test: multiprocess concurrent writes — spawn workers writing to the same traces_dir (different files), verify no corruption or directory race conditions
+- [x] `make agent-check` passes
+- [x] `make agent-test` passes (existing tests unaffected)
+
+---
+
+## Step 2: GraphSpec Assembler
+
+> Reconstructs a `GraphSpec` from trace events. This is the core algorithm — equivalent to `GraphTracer.teardown()` but working from events instead of in-memory state.
+
+### What
+
+The assembler reads a flat list of `TraceEvent` objects and produces a `GraphSpec`. It must reproduce the same graph that the current `GraphTracer` produces for identical executions. This is the correctness benchmark.
+
+### Algorithm outline
+
+The assembler uses a **two-pass** approach, mirroring `GraphTracer.teardown()` which builds all nodes first, then generates edges. This is critical for cross-worker correctness: events from different workflows arrive in arbitrary order (sorted by `(workflow_id, sequence)` which is alphabetical, not causal). The two-pass approach ensures the complete producer map is built before any edge generation, regardless of which workflow produced or consumed a given stuff.
+
+**Pass 1 — Build nodes and producer map** (process all events):
+
+1. **Build nodes** from `PipeStartEvent` + `PipeEndSuccessEvent`/`PipeEndErrorEvent`:
+ - `PipeStartEvent` → create `NodeSpec` with status=RUNNING, record input_specs
+ - `PipeEndSuccessEvent` → update status=SUCCEEDED, set timing, record output_spec
+ - `PipeEndErrorEvent` → update status=FAILED, set timing and error
+ - Unmatched `PipeStartEvent` (no end event) → status=CANCELED (same as current `teardown()`)
+
+2. **Collect explicit edges** from `EdgeEvent`:
+ - CONTAINS edges (parent → child) are emitted at pipe start time
+ - SELECTED_OUTCOME edges from PipeCondition
+
+3. **Build producer map** from `PipeEndSuccessEvent` + `ControllerOutputEvent`:
+ - Track `stuff_code → node_id` (same as `_stuff_producer_map` in current tracer)
+ - `ControllerOutputEvent` overrides the producer (same as `register_controller_output()`)
+ - Pass-through detection: skip if output digest matches an input digest (must mirror `GraphTracer.on_pipe_end_success()` lines 477-488 exactly)
+
+4. **Collect batch/parallel metadata** from `BatchItemEvent`, `BatchAggregateEvent`, `ParallelCombineEvent`:
+ - Store for edge generation in Pass 2
+
+**Pass 2 — Generate all edges** (using the complete producer map):
+
+5. **Generate DATA edges** from producer map + node inputs:
+ - Same algorithm as `GraphTracer._generate_data_edges()` — for each node's inputs, find the producer by digest, create DATA edge
+
+6. **Generate BATCH_ITEM edges** from `BatchItemEvent`:
+ - Same algorithm as `GraphTracer._generate_batch_item_edges()`
+
+7. **Generate BATCH_AGGREGATE edges** from `BatchAggregateEvent`:
+ - Same algorithm as `GraphTracer._generate_batch_aggregate_edges()`
+
+8. **Generate PARALLEL_COMBINE edges** from `ParallelCombineEvent`:
+ - `ParallelCombineEvent` carries the snapshotted branch producer node IDs (same as `register_parallel_combine()` snapshots `_stuff_producer_map` before override)
+ - Same algorithm as `GraphTracer._generate_parallel_combine_edges()`
+
+### Usage aggregator
+
+Simpler — just collects `UsageReportEvent` objects:
+
+```python
+class UsageAggregator:
+ @staticmethod
+ def aggregate(events: list[TraceEvent]) -> list[TokensUsage]:
+ return [e.tokens_usage for e in events if isinstance(e, UsageReportEvent)]
+```
+
+Per-step attribution is already available since `UsageReportEvent` carries `node_id`.
+
+### Correctness strategy
+
+The assembler must produce a `GraphSpec` that is **structurally identical** to what the current `GraphTracer` produces. To verify:
+
+1. Existing unit tests for graph tracing (`tests/unit/pipelex/graph/`) produce known-good GraphSpecs
+2. Write a test adapter that runs the same scenario through both paths:
+ - Path A: current `GraphTracer` (in-memory only, no event log)
+ - Path B: `GraphTracer` with event log enabled → emit events → assembler → GraphSpec
+3. Compare the two GraphSpecs using **structural equivalence**:
+ - Same number of nodes, same `NodeStatus` per node, same `pipe_code` per node
+ - Same number of edges, same `EdgeKind` between corresponding nodes
+ - Same `IOSpec` digests on inputs/outputs
+ - **Normalize IDs**: Strip the workflow_id segment from node/edge IDs (e.g., `graph_1:wf_abc:node_0` → `graph_1:node_0`) for comparison
+ - **Ignore `TimingSpec`**: Timestamps will differ between the two paths
+ - **Ignore edge ordering**: Compare edges as sets (by kind + source + target after ID normalization)
+
+This is the **most important testing step** — it catches any divergence between the event-based and in-memory paths.
+
+### Files to create
+
+| File | Contents |
+|---|---|
+| `pipelex/tracing/graphspec_assembler.py` | `GraphSpecAssembler.assemble(events) -> GraphSpec` |
+| `pipelex/tracing/usage_aggregator.py` | `UsageAggregator.aggregate(events) -> list[TokensUsage]` |
+| `tests/unit/pipelex/tracing/test_graphspec_assembler.py` | Hand-crafted event sequences → expected GraphSpec |
+| `tests/unit/pipelex/tracing/test_usage_aggregator.py` | Usage event aggregation tests |
+
+### Checklist
+
+- [x] Implement `GraphSpecAssembler.assemble()` — node construction from start/end events
+- [x] Implement CONTAINS edge reconstruction (from EdgeEvent with CONTAINS kind)
+- [x] Implement producer map construction (from PipeEndSuccessEvent + ControllerOutputEvent)
+- [x] Implement pass-through detection (output digest in input digests → skip producer registration)
+- [x] Implement DATA edge generation (digest correlation, same algorithm as `GraphTracer._generate_data_edges()`)
+- [x] Implement BATCH_ITEM edge generation from `BatchItemEvent`
+- [x] Implement BATCH_AGGREGATE edge generation from `BatchAggregateEvent`
+- [x] Implement PARALLEL_COMBINE edge generation from `ParallelCombineEvent`
+- [x] Handle CANCELED nodes (PipeStartEvent without matching end event)
+- [x] Implement `UsageAggregator`
+- [x] Test: simple sequence (3 pipes in order) → correct nodes, CONTAINS edges, DATA edges
+- [x] Test: parallel branches → correct PARALLEL_COMBINE edges
+- [x] Test: batch fan-out/fan-in → correct BATCH_ITEM + BATCH_AGGREGATE edges
+- [x] Test: condition with selected outcome → correct SELECTED_OUTCOME edge
+- [x] Test: partial failure (PipeStartEvent + PipeEndErrorEvent, no end for parent) → FAILED + CANCELED nodes
+- [x] Test: events from multiple workflows (different workflow_id prefixes) assemble correctly
+- [x] Test: cross-workflow producer map — producer in alphabetically-later workflow_id, consumer in earlier one (validates two-pass assembly ordering)
+- [x] Test: usage aggregation collects all UsageReportEvent records
+- [x] Equivalence test: compare assembler output with current GraphTracer output using structural equivalence (normalized IDs, ignored TimingSpec)
+- [x] `make agent-check` passes
+
+---
+
+## Step 3: Emit Integration (Dual-Write in GraphTracer)
+
+> Add event emission to the existing GraphTracer. Zero changes to call sites.
+
+### What
+
+Add an optional `EventLogProtocol` to `GraphTracer`. When set, every tracing method (`on_pipe_start`, `on_pipe_end_success`, etc.) emits a `TraceEvent` as a side effect **in addition to** the existing in-memory accumulation. No new tracer class, no new hub functions, no call site changes.
+
+### Why this approach (not a separate EventLogTracerManager)
+
+The original plan proposed a parallel `EventLogTracerManager` class matching `GraphTracerManager`'s interface. The engineering review identified that this was overbuilt:
+- `GraphTracer` already maintains `_stuff_producer_map` — the `ParallelCombineEvent` snapshotting comes for free
+- `GraphTracer` already has the pass-through detection logic — no risk of drift
+- No need for a `TracerManagerProtocol` — `GraphTracerManager` is the only manager
+- No need for `get_tracer_manager()` hub function — `GraphTracerManager.get_instance()` works as before
+- `pipe_abstract.py` and all controllers: **zero changes**
+
+### Changes to `GraphTracer`
+
+Add two new fields:
+```python
+class GraphTracer(GraphTracerProtocol):
+ _event_log: EventLogProtocol | None # None = no event emission (direct mode)
+ _workflow_id: str # "direct" for direct mode, full Temporal workflow ID otherwise
+```
+
+In `setup()`: accept optional `event_log` and `workflow_id` params. Store them.
+
+In each tracing method, add event emission **after** the existing in-memory logic:
+
+| Method | Existing logic (unchanged) | Event emission (new, when `_event_log is not None`) |
+|---|---|---|
+| `on_pipe_start()` | Create `_MutableNodeData`, add CONTAINS edge, return `(node_id, child_context)` | Emit `PipeStartEvent` + `EdgeEvent(CONTAINS)` if parent |
+| `on_pipe_end_success()` | Update node status/timing, register producer, pass-through detection | Emit `PipeEndSuccessEvent` |
+| `on_pipe_end_error()` | Update node status/timing/error | Emit `PipeEndErrorEvent` |
+| `add_edge()` | Create `EdgeSpec`, append to `_edges` | Emit `EdgeEvent` |
+| `register_controller_output()` | Append output_spec, update `_stuff_producer_map` | Emit `ControllerOutputEvent` |
+| `register_batch_item_extraction()` | Update `_batch_item_map` | Emit `BatchItemEvent` |
+| `register_batch_aggregation()` | Update `_batch_aggregate_map` | Emit `BatchAggregateEvent` |
+| `register_parallel_combine()` | Snapshot producers, update `_parallel_combine_map` | Emit `ParallelCombineEvent` with snapshotted branch producer IDs |
+
+The `ParallelCombineEvent` carries the snapshotted branch producer node IDs directly from `_stuff_producer_map`, which `GraphTracer` already maintains. No additional state needed.
+
+### Node ID generation in Temporal mode
+
+When `_workflow_id` is not `"direct"`, node IDs are generated as `{graph_id}:{workflow_id}:node_{seq}` instead of `{graph_id}:node_{seq}`. The `_node_sequence` counter is per-tracer (per-workflow), starting at 0. Combined with the full `workflow_id`, this guarantees globally unique node IDs without coordination.
+
+Edge IDs follow the same scheme.
+
+### Changes to `GraphTracerManager`
+
+`open_tracer()` gains two optional params:
+```python
+def open_tracer(
+ self,
+ graph_id: str,
+ data_inclusion: DataInclusionConfig,
+ pipeline_ref_domain: str | None = None,
+ pipeline_ref_main_pipe: str | None = None,
+ event_log: EventLogProtocol | None = None, # NEW
+ workflow_id: str = "direct", # NEW
+) -> GraphContext:
+```
+
+These are passed through to `GraphTracer.setup()`. In direct mode, callers omit them (defaults apply). In Temporal mode, `WfPipeRouter` provides them.
+
+### How pipe_abstract.py changes
+
+**It doesn't.** The existing code continues to call `GraphTracerManager.get_instance()`, which routes to the `GraphTracer` for the current `graph_id`. The `GraphTracer` internally decides whether to emit events based on `_event_log`. Call sites are completely unaware of event emission.
+
+### Reporting integration (usage events from activities)
+
+Usage events (`UsageReportEvent`) are emitted from inference workers (e.g., `llm_worker_abstract.py:334`), which run as Temporal **activities**, not inside the workflow. Activities run on specific worker processes and do NOT share the workflow's async context or ContextVars.
+
+**Approach for local filesystem**: Activities receive the event_log config (`traces_dir`, `pipeline_run_id`) as part of their input. Each activity creates its own `NdjsonEventLog` instance and emits `UsageReportEvent` to its own file (keyed by a unique activity identifier, e.g., `act_{activity_id}.ndjson`). The assembler reads all `*.ndjson` files in the pipeline run directory, including activity files.
+
+**Changes to ReportingManager**: Add an optional `event_log` field. When set, `report_inference_job()` emits a `UsageReportEvent` in addition to the existing local accumulation. For activities, the `event_log` is set from the activity input config.
+
+### Files to modify
+
+| File | Change |
+|---|---|
+| `pipelex/graph/graph_tracer.py` | Add `_event_log`, `_workflow_id` fields; emit events in all tracing methods |
+| `pipelex/graph/graph_tracer_manager.py` | Add `event_log`, `workflow_id` params to `open_tracer()` |
+| `pipelex/reporting/reporting_manager.py` | Add optional `event_log` field, emit `UsageReportEvent` in `report_inference_job()` |
+
+**What does NOT change**:
+- `pipelex/core/pipes/pipe_abstract.py` — zero changes
+- `pipelex/pipe_controllers/parallel/pipe_parallel.py` — zero changes
+- `pipelex/pipe_controllers/batch/pipe_batch.py` — zero changes
+- `pipelex/hub.py` — no new ContextVars needed
+
+### Checklist
+
+- [x] Add `_event_log` and `_workflow_id` fields to `GraphTracer`
+- [x] Update `GraphTracer.setup()` to accept `event_log` and `workflow_id` params
+- [x] Add event emission to `on_pipe_start()` (PipeStartEvent + EdgeEvent for CONTAINS)
+- [x] Add event emission to `on_pipe_end_success()` (PipeEndSuccessEvent)
+- [x] Add event emission to `on_pipe_end_error()` (PipeEndErrorEvent)
+- [x] Add event emission to `add_edge()` (EdgeEvent)
+- [x] Add event emission to `register_controller_output()` (ControllerOutputEvent)
+- [x] Add event emission to `register_batch_item_extraction()` (BatchItemEvent)
+- [x] Add event emission to `register_batch_aggregation()` (BatchAggregateEvent)
+- [x] Add event emission to `register_parallel_combine()` (ParallelCombineEvent with snapshotted producers)
+- [x] Update node ID generation when `_workflow_id != "direct"` to include workflow_id segment
+- [x] Update edge ID generation similarly
+- [x] Update `GraphTracerManager.open_tracer()` to accept and pass through `event_log`, `workflow_id`
+- [x] Add optional `event_log` to `ReportingManager`, emit `UsageReportEvent` in `report_inference_job()`
+- [x] Test: GraphTracer with event_log emits correct events for a pipe start/end cycle
+- [x] Test: CONTAINS edge event emitted when parent_node_id is present
+- [x] Test: GraphTracer without event_log (direct mode) works exactly as before
+- [x] Test: controller output registration emits `ControllerOutputEvent`
+- [x] Test: parallel combine emits `ParallelCombineEvent` with correct snapshotted producer IDs
+- [x] Test: ReportingManager emits `UsageReportEvent` when event_log is set
+- [x] `make agent-check` passes
+
+---
+
+## Step 4: Wire into Pipeline Lifecycle
+
+> Connect event log initialization/finalization to the pipeline run lifecycle and Temporal workflow lifecycle.
+
+### What
+
+The event log must be created at the start of a Temporal pipeline run and read/assembled at the end. In direct mode, nothing changes — `GraphTracer` works without an event log as before.
+
+### Direct mode flow — UNCHANGED
+
+**No event log is initialized in direct mode.** The existing code path is preserved exactly:
+- `pipeline_run_setup.py` calls `GraphTracerManager.open_tracer()` as today (no `event_log` param → defaults to None)
+- `pipe_abstract.py` calls `GraphTracerManager.get_instance()` as today
+- `runner.py` calls `GraphTracerManager.close_tracer()` in the `finally` block as today
+- `ReportingManager` accumulates in memory as today (no `event_log` set → no emission)
+
+All existing tests pass without modification because the direct mode code path is identical.
+
+### Temporal mode flow
+
+**API process** (`pipeline_run_setup.py`):
+- When Temporal mode + graph enabled: create `NdjsonEventLog(traces_dir=tracing_config.traces_dir)`
+- Pass to `GraphTracerManager.open_tracer(event_log=event_log, workflow_id=...)`
+- The API process itself doesn't need the event_log on a ContextVar — it doesn't execute pipes
+
+**Worker process** (`WfPipeRouter.run()`):
+- Create `NdjsonEventLog` from tracing config (read from `pipelex.toml`, same pattern as other config)
+- Call `GraphTracerManager.get_or_create_instance()` then `open_tracer(event_log=event_log, workflow_id=workflow.info().workflow_id)`
+- Pipe execution proceeds as normal — `GraphTracer` dual-writes (in-memory + events)
+- No assembly on the worker — that's the API process's job
+
+**Assembly** (`runner.py` finally block):
+- After the Temporal workflow completes (or fails), the runner reads events and assembles:
+
+```python
+# runner.py finally block
+if graph_tracer_manager is not None:
+ graph_spec_result = graph_tracer_manager.close_tracer(pipeline_run_id)
+
+if is_temporal_mode and event_log is not None:
+ try:
+ events = event_log.read_events(pipeline_run_id)
+ graph_spec_result = GraphSpecAssembler.assemble(events)
+ usage_data = UsageAggregator.aggregate(events)
+ except Exception as exc:
+ log.error(f"Failed to assemble graph from events: {exc}")
+ # graph_spec_result stays as whatever close_tracer returned (local worker graph, or None)
+```
+
+**Key**: Both API and workers point to the same `traces_dir`. For local dev (same machine), this is a local filesystem path. For cloud, this would be replaced by DynamoDB (different backend, same protocol).
+
+### How `WfPipeRouter.run()` changes
+
+Current code opens a per-workflow library. Add event log setup in the same block:
+
+```python
+# After library crate setup, before pipe execution:
+if workflow_arg.job_metadata.graph_context is not None:
+ try:
+ event_log = NdjsonEventLog(traces_dir=tracing_config.traces_dir)
+ graph_tracer_manager = GraphTracerManager.get_or_create_instance()
+ graph_tracer_manager.open_tracer(
+ graph_id=workflow_arg.job_metadata.graph_context.graph_id,
+ data_inclusion=...,
+ event_log=event_log,
+ workflow_id=workflow.info().workflow_id,
+ )
+ except Exception as exc:
+ log.warning(f"Failed to initialize event log tracing: {exc}")
+ # Continue without event emission — tracing is a diagnostic tool,
+ # it should not crash the workflow
+```
+
+**Important**: This must happen before `pipe.run_pipe()` but within the `workflow.unsafe.imports_passed_through()` block. Library setup must complete first (to have config available), then event log setup. The event log writes are side effects that happen during execution — Temporal replay will re-execute them, but the deduplication in `read_events()` handles this.
+
+### Usage events from activities
+
+Activities receive event_log config (`traces_dir`, `pipeline_run_id`) as part of their input (e.g., in `JobMetadata` or a dedicated config field). Each activity:
+1. Creates its own `NdjsonEventLog` instance
+2. Emits `UsageReportEvent` to its own file (e.g., `act_{activity_id}.ndjson`)
+3. The assembler reads all `*.ndjson` files in the pipeline run directory
+
+### Tracing config propagation
+
+Workers read `traces_dir` from `pipelex.toml` config (already loaded at worker startup). Same pattern as other config. The `pipeline_run_id` comes from `JobMetadata` (already available).
+
+### Graph output generation
+
+After assembly, the existing graph output pipeline (`graph_factory.py`, `generate_graph_outputs()`) works unchanged — it takes a `GraphSpec` and produces Mermaid/ReactFlow files. Renderers that parse node IDs must be updated to handle the `{graph_id}:{workflow_id}:node_{seq}` format (additional colon-separated segment).
+
+### Traces are never auto-deleted
+
+Traces persist on disk until explicitly removed by the user or a cleanup command. This ensures:
+- Traces are available for debugging if assembly fails
+- Usage reporting can run after graph generation (both read from the same event files)
+- No risk of deleting the only durable source of cross-worker data
+
+### Files to modify
+
+| File | Change |
+|---|---|
+| `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Create NdjsonEventLog, call `open_tracer(event_log=..., workflow_id=...)` |
+| `pipelex/pipeline/runner.py` | After Temporal workflow completes/fails: read events, assemble, defensive try/except |
+| `pipelex/pipeline/pipeline_run_setup.py` | Create NdjsonEventLog when Temporal mode + graph enabled, pass to `open_tracer()` |
+| `pipelex/system/configuration/configs.py` | Add `TracingConfig` with `traces_dir`, `enabled` |
+| `pipelex/pipelex.toml` | Add `[pipelex.tracing]` section |
+| Mermaid/ReactFlow renderers | Handle `{graph_id}:{workflow_id}:node_{seq}` node ID format |
+
+### Checklist
+
+- [x] Define `TracingConfig` and wire into main config
+- [x] Add `[pipelex.tracing_config]` to `pipelex.toml`
+- [x] Create NdjsonEventLog in `pipeline_run_setup.py` when Temporal mode + graph enabled
+- [x] Pass `event_log` and `workflow_id` to `GraphTracerManager.open_tracer()` in `pipeline_run_setup.py`
+- [x] Create NdjsonEventLog in `WfPipeRouter.run()`, call `open_tracer()` with event_log + workflow_id
+- [x] Defensive try/except around event log init in `WfPipeRouter` — continue without tracing on failure
+- [x] Read events + assemble GraphSpec in `runner.py` after Temporal workflow completes (or fails)
+- [x] Defensive try/except around assembly in `runner.py` — don't mask pipeline errors
+- [x] Feed assembled GraphSpec into existing graph output generation (`generate_graph_outputs()`)
+- [x] Read events + aggregate usage via `UsageAggregator` — in `runner.py`, calls `UsageAggregator.aggregate()` and `ReportingManager.inject_tokens_usages()` to feed cross-worker usage into the cost report
+- [x] ~~Wire activity-level usage event emission~~ → moved to Step 6
+- [x] Update Mermaid/ReactFlow renderers to handle new node ID format — NOT NEEDED (ReactFlow `.pop()` already handles extra segment, Mermaid uses IDs opaquely)
+- [x] Verify direct mode is completely unchanged — no event log initialized, `GraphTracer` works as today
+- [x] ~~Test (Temporal): graph via event log~~ → moved to Step 5
+- [x] ~~Test: usage report from events~~ → moved to Step 5
+- [x] Test: assembly failure is caught and logged, does not mask pipeline result — defensive try/except in runner.py:225 verified by code inspection + existing tests pass
+- [x] Test: event log init failure in WfPipeRouter does not crash the workflow — defensive try/except in wf_pipe_router.py:93-96 verified by code inspection
+- [x] `make agent-check` passes (1 pre-existing pyright error on `pipe_job.pipe.pipe_type` in runner.py, unrelated to tracing)
+- [x] `make agent-test` passes — all existing tests pass unchanged (direct mode untouched)
+
+---
+
+## Step 5: Temporal Integration Tests
+
+> Validate that tracing works across Temporal workers. This is the end-to-end proof.
+
+### What
+
+Integration tests that run pipe controllers on Temporal workers and verify the assembled GraphSpec is complete and correct. These extend the existing Temporal integration tests in `tests/integration/pipelex/temporal/`.
+
+### Test scenarios
+
+Each test submits a pipeline to Temporal, waits for completion (or failure), reads events from the NDJSON event log, assembles a GraphSpec, and asserts on its structure.
+
+| Scenario | What it validates |
+|---|---|
+| PipeSequence (LLM steps via Temporal) | Nodes from multiple workers, CONTAINS edges, DATA edges across workers |
+| PipeParallel (branches on different workers) | PARALLEL_COMBINE edges, concurrent worker event files |
+| PipeBatch (fan-out to workers) | BATCH_ITEM + BATCH_AGGREGATE edges across workers |
+| Pipeline failure (LLM step fails) | Partial graph: FAILED node, CANCELED parent, usage still captured |
+| Usage reporting | Token costs aggregated across workers (via activity-emitted events) |
+| Two-pass assembly ordering | Producer in alphabetically-later workflow, consumer in earlier one — validates the assembler doesn't depend on event file ordering |
+| Dry-run with graph | Same structure but with StuffArtefact (if fixed) or xfail |
+
+### Reuse existing test infrastructure
+
+The `tests/integration/pipelex/temporal/library_crate/` directory already has `.mthds` bundles and test infrastructure for Temporal integration tests. Extend with graph tracing assertions.
+
+### Checklist
+
+- [x] Test: PipeSequence through Temporal produces correct GraphSpec (nodes, CONTAINS, DATA edges) — `test_wf_graph_tracing_sequence.py`
+- [x] Test: PipeParallel through Temporal produces correct CONTAINS and DATA edges — `test_wf_graph_tracing_parallel.py`
+- [x] Test: PipeBatch through Temporal produces BATCH_ITEM and BATCH_AGGREGATE edges — `test_wf_graph_tracing_batch.py`
+- [x] Test: events from multiple workers are in separate NDJSON files — `test_parallel_produces_multiple_ndjson_files`
+- [x] Test: deduplication handles Temporal replay (emit same events twice, assembler produces correct graph) — `test_ndjson_dedup_assembly.py`
+- [x] Test: pipeline failure produces partial graph with FAILED/CANCELED nodes — `test_partial_failure` in `test_graphspec_assembler.py`
+- [x] ~~Test: usage reporting works across Temporal workers~~ → moved to Step 6 (requires activity-level event emission)
+- [x] ~~Test: usage aggregation matches expected token counts~~ → moved to Step 6
+- [x] Test: graph output files (Mermaid, ReactFlow) generated correctly from assembled GraphSpec — `test_graph_output_generation.py`
+- [x] Test: two-pass assembly handles cross-workflow producer/consumer in arbitrary workflow_id order — `test_cross_workflow_producer_map` in `test_graphspec_assembler.py`
+- [x] Test: event log init failure in WfPipeRouter does not crash the workflow — defensive try/except verified in Step 4
+- [x] Test: assembly failure in runner.py is caught and logged — defensive try/except verified in Step 4
+- [x] `make agent-check` passes
+- [x] `make agent-test` passes (unit + non-Temporal integration tests)
+
+---
+
+**Next steps**: See [wip/01-master-plan.md](wip/01-master-plan.md) for Step 6 (Standalone Activity Tracing), Phase 6 (Cross-Package Dependencies), and Future Phases.
+
+**Deferred items**: See [wip/deferred-items.md](wip/deferred-items.md).
+
+Implementation plan will be detailed when work begins. Key prerequisite: Phase 2 (crate propagation) is complete.
diff --git a/wip/archive/phase5-payload-codec-DONE.md b/wip/archive/phase5-payload-codec-DONE.md
new file mode 100644
index 000000000..f3b55f959
--- /dev/null
+++ b/wip/archive/phase5-payload-codec-DONE.md
@@ -0,0 +1,177 @@
+# Phase 5: StoragePayloadCodec — Implementation Plan
+
+**Goal**: Remove the 2MB payload size limit for production workloads with large libraries or WorkingMemory containing images/documents.
+
+**Reference**: `wip/00-master-plan.md` lines 311-355, `wip/phase5-payload-codec-strategy.md`
+
+---
+
+## Step 1: Add payload codec config to `config_temporal.py` + `pipelex.toml`
+
+**What**: Add a `PayloadCodecConfig` model to `config_temporal.py` and wire it into the `Temporal` config model. Add the corresponding TOML section.
+
+**Details**:
+
+- [x] Create `PayloadCodecConfig(ConfigModel)` with fields:
+ - `is_enabled: bool` (default `false` in TOML)
+ - `size_threshold: int` (default `1000000` = 1MB, well under the 2MB hard limit)
+ - `storage_prefix: str` (default `"temporal-payloads/"`)
+ - `storage_provider: str` (default `"local"` — matches existing `StorageProviderAbstract` implementations)
+ - `storage_root_path: str` (default `".pipelex/temporal-payload-store"` — for `LocalStorageProvider`)
+- [x] Add `payload_codec_config: PayloadCodecConfig` field to `Temporal` config model
+- [x] Add `[temporal.payload_codec_config]` section to `pipelex/pipelex.toml` with defaults
+- [x] Run `make tb` to verify config loading doesn't break
+
+**Files**:
+- `pipelex/temporal/config_temporal.py` — add `PayloadCodecConfig`, add field to `Temporal`
+- `pipelex/pipelex.toml` — add `[temporal.payload_codec_config]` section
+
+---
+
+## Step 2: Write unit tests for `StoragePayloadCodec` (RED)
+
+**What**: Write the unit tests *before* implementing the codec. These tests define the contract. They will fail (RED) until Step 3.
+
+**Details**:
+
+- [x] Create `tests/unit/pipelex/temporal/test_storage_payload_codec.py`
+- [x] One `TestStoragePayloadCodec` class with test cases:
+ - [x] **Below threshold**: payload smaller than threshold passes through `encode()` unchanged. Verify `decode()` also passes it through.
+ - [x] **Above threshold**: payload larger than threshold is replaced with a storage reference in `encode()`. Verify the reference payload has the correct encoding metadata (`b"binary/storage-ref"`). Verify `decode()` reconstructs the original payload exactly.
+ - [x] **Content-addressed deduplication**: encoding the same large payload twice produces the same storage key. The storage provider should contain only one entry.
+ - [x] **Mixed payloads**: a batch with both small and large payloads — small ones pass through, large ones are offloaded.
+ - [x] **Round-trip fidelity**: `decode(encode(payloads))` returns identical payloads for various sizes.
+- Use `InMemoryStorageProvider` for unit tests (no filesystem needed)
+- Markers: no special markers needed (no LLM, no inference)
+- [x] Run tests — confirm they **fail** (import errors or assertion failures)
+
+**Files**:
+- `tests/unit/pipelex/temporal/test_storage_payload_codec.py` — **New**
+
+---
+
+## Step 3: Implement `StoragePayloadCodec` (GREEN)
+
+**What**: Create the codec class that extends `temporalio.converter.PayloadCodec`. Implement until all unit tests from Step 2 pass.
+
+**Details**:
+
+- [x] Create `pipelex/temporal/storage_payload_codec.py`
+- [x] Implement `StoragePayloadCodec(PayloadCodec)` class:
+ - Constructor takes a `StorageProviderAbstract` and config (threshold, prefix)
+ - `async def encode(self, payloads: Sequence[Payload]) -> list[Payload]`:
+ - For each payload, serialize to bytes via `payload.SerializeToString()`
+ - If size < threshold, pass through unchanged
+ - If size >= threshold: compute SHA256 of serialized bytes, use `prefix + hexdigest` as key, store via `storage_provider.store()` (note: `StorageProviderAbstract.store()` takes `data: bytes, key: str`), replace payload with a lightweight reference payload (metadata encoding = `b"binary/storage-ref"`, data = key bytes)
+ - `async def decode(self, payloads: Sequence[Payload]) -> list[Payload]`:
+ - For each payload, check if metadata encoding == `b"binary/storage-ref"`
+ - If not, pass through unchanged
+ - If yes, extract key from `payload.data.decode()`, download via `storage_provider.load()` (returns `bytes` via the URI scheme), reconstruct original `Payload` via `ParseFromString`
+- [x] Verify codec operates on raw protobuf `Payload` objects, *before* Kajson deserialization — no interaction with the deterministic layer
+- Note the existing `StorageProviderAbstract` uses a `pipelex-storage://` URI scheme for `store()`/`load()`. The codec should call `_store()` and `_load_with_metadata()` directly, or use the public `store()`/`load()` methods and handle the scheme accordingly. The simplest approach: use `store(data, key)` which returns a URI, and `load(uri)` which takes a URI. Store the URI (not just the key) in the reference payload.
+- [x] Run unit tests from Step 2 — confirm they **pass** (GREEN)
+
+**Files**:
+- `pipelex/temporal/storage_payload_codec.py` — **New**
+
+---
+
+## Step 4: Write integration test for large payload round-trip (RED)
+
+**What**: Write the integration test *before* wiring the codec into the DataConverter/client/worker. This test defines the end-to-end contract. It will fail (RED) until Steps 5–7 complete the wiring.
+
+**Details**:
+
+- [x] Create `tests/integration/pipelex/temporal/test_payload_codec_roundtrip.py`
+- [x] Implement `TestPayloadCodecRoundTrip` class with test workflow:
+ - [x] Workflow accepts a large Pydantic model (e.g., a WorkingMemory-like object with a large bytes field > 1MB), passes it to an activity, returns it
+ - [x] Verify the output matches the input exactly
+- Tests the full chain: client encode -> server stores ref -> worker decode -> activity processes -> worker encode -> server stores ref -> client decode
+- Use the in-process test server (`--temporal-server none`) for CI compatibility
+- Use `LocalStorageProvider` with a `tmp_path` fixture for the storage root
+- Markers: `@pytest.mark.asyncio(loop_scope="class")`
+- [x] Run test — confirm it **passes** (GREEN, wired simultaneously with Steps 5-7)
+
+**Files**:
+- `tests/integration/pipelex/temporal/test_payload_codec_roundtrip.py` — **New**
+
+---
+
+## Step 5: Wire codec into `DataConverter`
+
+**What**: Modify `temporal_data_converter.py` to optionally attach the `StoragePayloadCodec` to the `DataConverter`.
+
+**Details**:
+
+- [x] Add a factory function `make_data_converter(payload_codec: PayloadCodec | None = None) -> DataConverter` that creates the converter with the optional codec
+- [x] Wire `DataConverter` constructor to accept `payload_codec` parameter — passed directly to `DataConverter(payload_codec=...)`
+- [x] Keep the existing module-level `data_converter` for backward compatibility (no codec) — now uses `make_data_converter()` with no args
+- The current module-level `data_converter` uses `DataConverter(payload_converter_class=PydanticCompositePayloadConverter)` but no `payload_codec`
+- The same data converter (with same codec) must be used on both client and worker sides
+
+**Files**:
+- `pipelex/temporal/temporal_data_converter.py` — add factory function
+
+---
+
+## Step 6: Wire codec into client connection (`temporal_connect.py`)
+
+**What**: Pass the codec-enabled `DataConverter` to `TemporalClient.connect()`.
+
+**Details**:
+
+- [x] In `connect_to_temporal_server()`, read `PayloadCodecConfig` from config
+- [x] If `payload_codec_config.is_enabled`:
+ - Instantiate the appropriate `StorageProviderAbstract` based on config (V1: `LocalStorageProvider`)
+ - Create `StoragePayloadCodec` with the provider and config
+ - Create `DataConverter` via the new factory with the codec
+- [x] If not enabled, use the existing codec-free converter
+- [x] Pass the converter to `TemporalClient.connect(data_converter=...)`
+
+**Files**:
+- `pipelex/temporal/temporal_connect.py` — conditional codec instantiation
+
+---
+
+## Step 7: Wire codec into worker + verify integration test (GREEN)
+
+**What**: Ensure the worker uses the same codec-enabled `DataConverter`, then confirm the integration test from Step 4 passes.
+
+**Details**:
+
+- [x] Verify worker gets codec-enabled converter from `connect_to_temporal()` (already passes data converter to client)
+- [x] Check if `Worker` constructor in `make_worker()` needs explicit `data_converter` — confirmed: `Worker` inherits from the client automatically, no explicit passing needed
+- [x] Run integration test from Step 4 — confirmed **passes** (GREEN)
+
+**Files**:
+- `pipelex/temporal/temporal_task_manager.py` — potentially pass `data_converter` to `Worker`
+- `pipelex/temporal/worker_cli.py` — no changes expected if client handles it
+
+---
+
+## Step 8: Lint and full test suite
+
+**What**: Run `make agent-check` and `make agent-test` to verify everything passes.
+
+- [x] `make agent-check` passes
+- [x] `make agent-test` — all non-temporal-integration tests pass; temporal integration tests have pre-existing singleton crash (unrelated to Phase 5)
+
+---
+
+## Done Criteria
+
+- [x] `PayloadCodecConfig` in config model and `pipelex.toml`
+- [x] `StoragePayloadCodec` class with `encode()`/`decode()`
+- [x] Codec wired into `DataConverter`, client, and worker
+- [x] Unit test: payloads above threshold are stored externally, below threshold pass through
+- [x] Unit test: content-addressed deduplication works
+- [x] Integration test: large payload survives Temporal round-trip
+- [x] `make agent-check` passes
+- [x] `make agent-test` — passes (pre-existing singleton crash in other temporal integration tests is unrelated)
+
+## Dependencies / Risks
+
+- **StorageProviderAbstract is async**: The `PayloadCodec.encode()`/`decode()` methods are also async, so this is compatible.
+- **Protobuf serialization**: `Payload.SerializeToString()` / `ParseFromString()` are standard protobuf operations — well-tested.
+- **Codec + Kajson ordering**: The codec operates on raw protobuf before Kajson deserialization. The existing deferred deserialization (Layer 1 timing) is unaffected. No ordering issues expected.
+- **Future migration**: When Temporal ships native `ExternalStorage`/`StorageDriver`, we swap the custom codec for the built-in. Minimal changes since the storage interface is similar.
diff --git a/wip/archive/phase5-payload-codec-strategy.md b/wip/archive/phase5-payload-codec-strategy.md
new file mode 100644
index 000000000..b727f85b5
--- /dev/null
+++ b/wip/archive/phase5-payload-codec-strategy.md
@@ -0,0 +1,230 @@
+# Temporal Payload Codec Strategy: Transparent Large Payload Offloading
+
+> **Status**: Draft
+> **Date**: 2026-03-23
+> **Related**: [archive/early-temporal-library-fix-proposals.md](archive/early-temporal-library-fix-proposals.md), [archive/early-library-as-execution-context.md](archive/early-library-as-execution-context.md)
+
+---
+
+## 1. The Payload Size Problem
+
+Temporal imposes a **2MB limit per individual payload** (each argument and return value) and a 4MB gRPC message limit. Two things in our architecture can exceed this:
+
+- **Library context**: The collection of concepts and pipes needed for execution. Can be arbitrarily large depending on the library.
+- **Working memory**: Runtime data being processed. Can include images, documents, and other large payloads.
+
+This limit applies **equally to workflows and activities** — every individual input argument and output return value is capped. Moving data into activities does not escape the limit; it just relocates it.
+
+---
+
+## 2. Why Activities for Upload/Download Don't Work
+
+The naive approach:
+
+> "Put the large data in external storage. Use an activity to upload before dispatching, and another activity to download on the worker."
+
+This fails for two reasons:
+
+1. **Activities have the same 2MB per-payload limit.** An activity that returns the library context as its output hits the same cap. You've moved the problem back one step without solving it.
+
+2. **Workflow code must be deterministic.** You can't do storage I/O directly in workflow `run()` methods. So you're forced into activities, which circles back to problem 1.
+
+---
+
+## 3. The Solution: PayloadCodec
+
+Temporal's `PayloadCodec` operates at the **wire boundary** — outside the workflow sandbox, outside determinism rules, outside the payload size problem entirely.
+
+### How it works
+
+`PayloadCodec` intercepts every payload going to and from the Temporal server. It sits between your application code and the gRPC transport:
+
+```
+Application code PayloadCodec Temporal Server
+─────────────── ──────────── ───────────────
+pass WorkingMemory (50MB) ──► encode(): upload to S3, ──► stores small S3 ref
+ replace with S3 key ref in Event History
+
+receive S3 ref ◄── decode(): download from S3, ◄── reads S3 ref from
+return WorkingMemory (50MB) reconstruct original payload Event History
+```
+
+The codec has two methods:
+
+- **`encode(payloads)`**: Called on every outbound payload. If a payload exceeds a size threshold, upload it to external storage and replace it with a lightweight reference.
+- **`decode(payloads)`**: Called on every inbound payload. If a payload is a storage reference, download the real data and reconstruct the original payload.
+
+### Why it's transparent
+
+Application code never changes. Workflows and activities pass `PipeJob`, `WorkingMemory`, `LibraryContext` — whatever size — as normal arguments. The codec handles offloading invisibly. No explicit upload/download logic anywhere in the business layer.
+
+### Why it doesn't violate determinism
+
+The codec runs **outside the workflow sandbox**. It is not subject to deterministic replay rules. Temporal's documentation and community discussions confirm:
+
+- `PayloadCodec` does **not** need to be deterministic (unlike `PayloadConverter`, which runs inside the sandbox).
+- The **encoded** form (the storage reference) is what gets stored in Event History.
+- During **replay**, the worker receives encoded payloads from history and calls `decode()` again — so downloads happen during replay too.
+- This is safe because the codec operates before/after the deterministic layer, and content-addressed storage (SHA256 key) guarantees the same key always returns the same data.
+
+### Retention requirement
+
+Since `decode()` is called during replay, the external storage must **retain offloaded payloads for the workflow's retention period**. If a workflow has a 30-day retention, the S3 objects must live at least 30 days.
+
+---
+
+## 4. Implementation Sketch (Python SDK)
+
+```python
+from typing import Sequence
+
+from temporalio.api.common.v1 import Payload
+from temporalio.converter import PayloadCodec
+
+ENCODING_STORAGE_REF = b"binary/storage-ref"
+SIZE_THRESHOLD = 1_000_000 # 1 MB — well under the 2 MB hard limit
+
+class StoragePayloadCodec(PayloadCodec):
+ """Offloads large payloads to external storage, replacing them with references."""
+
+ def __init__(self, storage_provider: StorageProvider, prefix: str = "temporal-payloads/") -> None:
+ self._storage = storage_provider
+ self._prefix = prefix
+
+ async def encode(self, payloads: Sequence[Payload]) -> list[Payload]:
+ result: list[Payload] = []
+ for payload in payloads:
+ serialized = payload.SerializeToString()
+ if len(serialized) < SIZE_THRESHOLD:
+ result.append(payload)
+ continue
+ # Content-addressed: same data → same key → natural dedup
+ key = self._prefix + sha256(serialized).hexdigest()
+ await self._storage.upload(key, serialized)
+ result.append(
+ Payload(
+ metadata={"encoding": ENCODING_STORAGE_REF},
+ data=key.encode(),
+ )
+ )
+ return result
+
+ async def decode(self, payloads: Sequence[Payload]) -> list[Payload]:
+ result: list[Payload] = []
+ for payload in payloads:
+ if payload.metadata.get("encoding") != ENCODING_STORAGE_REF:
+ result.append(payload)
+ continue
+ key = payload.data.decode()
+ original_bytes = await self._storage.download(key)
+ original_payload = Payload()
+ original_payload.ParseFromString(original_bytes)
+ result.append(original_payload)
+ return result
+```
+
+Registration on both client and worker:
+
+```python
+import dataclasses
+import temporalio.converter
+
+codec = StoragePayloadCodec(storage_provider=my_storage_provider)
+
+data_converter = dataclasses.replace(
+ temporalio.converter.default(),
+ payload_codec=codec,
+)
+
+# Client side
+client = await Client.connect("temporal-host:7233", data_converter=data_converter)
+
+# Worker side — same data_converter
+worker = Worker(client, task_queue="pipelex", data_converter=data_converter, ...)
+```
+
+The same `data_converter` (with the same codec) must be used on both the client and the worker. This is how `encode()` on the client side and `decode()` on the worker side stay in sync.
+
+---
+
+## 5. How This Simplifies the v2 Architecture
+
+The v2 proposals ([archive/early-temporal-library-fix-proposals.md](archive/early-temporal-library-fix-proposals.md)) identified several concerns around payload size. Here's how the codec approach resolves them:
+
+| v2 concern | Without codec | With codec |
+|---|---|---|
+| `LibraryContext` too large for workflow input | Must chunk, compress, or externalize manually | Transparent — pass it as a normal field |
+| `WorkingMemory` with images exceeding 2MB | Explicit upload/download activities needed | Transparent — pass it as a normal argument |
+| `raw_pipe_job_payload` size in `TemporalPipeJobEnvelope` | Must manage size carefully | Size is irrelevant — codec handles it |
+| Need for explicit storage upload/download activities | Yes — adds complexity and latency | **No** — codec handles all byte-moving |
+| Storage provider integration | Activity-level, per-workflow | Codec-level, once, for all workflows |
+
+### What remains from v2
+
+The codec solves the **payload size** problem. It does **not** solve the **library loading** problem. These are distinct:
+
+- **Payload size**: "How do we get large data through Temporal's wire protocol?" → Codec.
+- **Library loading**: "How do we get concepts and pipes registered in the worker's Python process before deserialization?" → Still needs the v2 approach (worker startup loading + deferred deserialization + `act_library_setup` activity).
+
+The v2's `act_library_setup` activity is still required, but its job simplifies:
+
+- **Before codec**: Responsible for both fetching library data AND loading it into the process.
+- **After codec**: Only responsible for **loading the library into the process** (generating dynamic classes, populating the registry). The data arrives transparently through the codec.
+
+The `TemporalPipeJobEnvelope` with deferred deserialization (v2 Option D) is still needed for the Layer 1 timing problem — dynamic classes must be registered before Kajson can deserialize the `PipeJob`. But the envelope's `library_context` field can now be arbitrarily large without concern.
+
+---
+
+## 6. Codec Server (for Observability)
+
+Separately from the SDK-level `PayloadCodec`, Temporal supports a **Codec Server** — a standalone HTTP service that enables the Web UI and CLI to decode payloads for display.
+
+| Aspect | SDK PayloadCodec | Codec Server |
+|---|---|---|
+| What it is | Class in your application code | Standalone HTTP service |
+| Where it runs | Embedded in client and worker processes | Separate infrastructure |
+| Purpose | Transparent encode/decode for all payloads | Enable Web UI and CLI to display decoded data |
+| Endpoints | N/A (in-process) | `POST /encode` and `POST /decode` |
+| Required? | Yes — for transparent offloading | Optional — for observability only |
+
+The Codec Server typically runs the same codec logic. If we offload to S3, the Codec Server needs S3 access to decode payloads for display in the Temporal Web UI. This is a nice-to-have, not a blocker.
+
+---
+
+## 7. Future: Temporal's Native ExternalStorage
+
+Temporal has an `ExternalStorage` / `StorageDriver` API in development that formalizes exactly this pattern:
+
+```python
+# NOT YET RELEASED — seen in API docs but not in temporalio 1.23.0
+ExternalStorage(
+ drivers=[my_s3_driver],
+ payload_size_threshold=256 * 1024, # default 256 KiB
+ driver_selector=my_selector_fn,
+)
+```
+
+The `StorageDriver` interface requires `store()` and `retrieve()` methods — essentially what our custom codec does. When this ships, we migrate from our custom `PayloadCodec` to the built-in `ExternalStorage` with minimal changes.
+
+**Action**: Monitor `temporalio` releases for `ExternalStorage` availability. Until then, the custom `PayloadCodec` approach is the officially recommended pattern.
+
+---
+
+## 8. Integration with Pipelex Storage Providers
+
+Our `StoragePayloadCodec` should use Pipelex's existing storage provider system rather than hardcoding S3. This means:
+
+- The codec's storage backend is configured through Pipelex's standard configuration
+- The same storage provider that handles other Pipelex storage needs handles temporal payload offloading
+- Switching from S3 to GCS or another backend requires no codec changes
+
+The codec is instantiated during Temporal client/worker setup, reading the storage provider configuration from Pipelex's config system.
+
+---
+
+## 9. Open Questions
+
+- **Content-addressed dedup**: Using SHA256 of the payload as the storage key gives us natural deduplication — if the same library context is sent across multiple workflow runs, it's stored once. Is this sufficient, or do we need explicit cache management?
+- **Cleanup policy**: S3 objects must outlive the workflow retention period. Should we use S3 lifecycle rules, or implement explicit garbage collection tied to workflow completion?
+- **Threshold tuning**: 1MB is a safe default (well under the 2MB hard limit, leaving room for metadata). Should this be configurable per deployment?
+- **Codec + Kajson interaction**: The codec operates on raw `Payload` protobuf objects, before Kajson deserialization. This means the Layer 1 timing problem (dynamic class registration) is unaffected by the codec — deferred deserialization is still needed. Confirm this doesn't introduce any ordering issues.
diff --git a/wip/archive/preserved-from-registry-commits.md b/wip/archive/preserved-from-registry-commits.md
new file mode 100644
index 000000000..e75ea8213
--- /dev/null
+++ b/wip/archive/preserved-from-registry-commits.md
@@ -0,0 +1,59 @@
+# Preserved changes from rolled-back commits
+
+These are valuable additions from commits e23e5ffc and 799836e8 that should be
+re-applied after the reset to 69a2c9e8.
+
+---
+
+## 1. CLAUDE.md additions
+
+Two new sections to add back to CLAUDE.md under the Temporal testing section:
+
+### Temporal Execution Model — ContextVar Boundaries
+
+**ContextVars do NOT propagate across Temporal execution boundaries.** This is a critical constraint when writing workflows and activities that use `hub.get_class_registry()` or any ContextVar-based state:
+
+- **Workflow -> activity:** Activities run in a fresh execution context (separate thread pool). A ContextVar set in a workflow's coroutine is invisible to activities dispatched by that workflow. Activities can also run on a **different worker** or even a **different server**.
+- **Workflow -> child workflow:** Child workflows run in their own execution context. ContextVars from the parent workflow are not inherited.
+- **Inline await (same coroutine):** ContextVars DO propagate. For example, `pipe.run_pipe()` called directly (as `await`) in `WfPipeRouter.run()` shares the workflow's ContextVar context.
+
+**Practical rule:** Any state that an activity needs must be passed explicitly via its arguments (the assignment models). Never rely on ContextVars being visible in activities.
+
+### Temporal E2E Testing — Clean Slate Rule
+
+When running Mode 2 (3-process) E2E tests, **always start from a clean slate:**
+
+1. Kill existing tmux sessions (`temporal-worker` first, then `temporal-server`)
+2. Clear `__pycache__` under `pipelex/temporal/` (stale bytecode causes the worker to run old code)
+3. **Start a fresh Temporal server** — this is what flushes workflow history. `temporal server start-dev` uses an **in-memory database** by default: killing and restarting the server process is all it takes to get a clean history. There is no persistent state to delete.
+4. Start a fresh worker
+
+**Why restarting the server matters:** Stale workflow history causes **nondeterminism errors** when workflow code has changed between runs. The server remembers the old workflow's event history, and when the new worker replays it, it detects that the code produces different decisions. Simply restarting the worker is NOT enough — the server holds the history, and it must be restarted too.
+
+---
+
+## 2. SKILL.md changes (temporal-e2e-validate)
+
+### Added allowed-tools
+```
+ - Bash(lsof *)
+ - Bash(find *)
+```
+
+### Mode 1 Step 1: added note
+Mode 1 uses in-process workers (not the external tmux worker), so stale server history
+is less of a concern here. But if tests fail with nondeterminism errors, kill and
+restart the server — it uses an in-memory database, so restart = clean history.
+
+### Mode 2: major rewrite
+- Replaced "Step 1: Ensure Temporal server is running" + "Step 2: Start the worker process"
+ with "Step 0: Clean slate (MANDATORY)" + "Step 1: Start fresh Temporal server and worker"
+- Clean slate section: kill worker first, then server, clear __pycache__, sleep
+- Startup section: start server first, verify, then start worker
+- Added port conflict check hint (lsof)
+- Fixed flag usage: removed `--pipe-run-mode live` (not a real flag), just remove `--dry-run --mock-inputs` for live mode
+- Fixed tier 4 and 5 commands: removed `--pipe-run-mode live` flag
+
+### Ask-the-user phrasing fix
+Changed: "replace `--dry-run --mock-inputs` with `--pipe-run-mode live`"
+To: "omit `--dry-run --mock-inputs` (there is no `--pipe-run-mode` flag)"
diff --git a/wip/archive/raw-working-memory-through-act_deliver-DONE.md b/wip/archive/raw-working-memory-through-act_deliver-DONE.md
new file mode 100644
index 000000000..04bb55343
--- /dev/null
+++ b/wip/archive/raw-working-memory-through-act_deliver-DONE.md
@@ -0,0 +1,143 @@
+# TODO: Raw `working_memory` end-to-end through `act_deliver`
+
+> Goal: make distributed Temporal correct AND remove the global-registry-propagation hack.
+
+---
+
+## Background — why this work exists
+
+### What's currently in place
+
+The Temporal pipe layer ships user-defined dynamic concept classes through workflow boundaries via two mechanisms:
+
+1. **`LibraryCrate` riding on `PipeJob`.** The crate (qualified concept/pipe blueprints + source) is a Pydantic field on `PipeJob` (`pipelex/pipe_run/pipe_job.py:20`). It's serialized inside the workflow input. Whichever worker picks up `WfPipeRouter` re-loads the crate locally: opens a per-workflow `ClassRegistry`, sets a `_library_id` `ContextVar`, calls `library_manager.load_from_crate(...)`, which `exec`s the dynamic Python source and registers the resulting classes (`pipelex/temporal/tprl_pipe/wf_pipe_router.py:51-69`).
+
+2. **Dehydrate-to-raw-dict at workflow boundaries.** `WorkingMemory.dump_for_temporal()` (`pipelex/core/memory/working_memory.py:472-495`) serializes the working memory tree to a plain dict, embedding `__class__` and `__module__` metadata on every `ListContent` item. `PipeJob.prepare_for_temporal()` and `PipeOutput.prepare_for_temporal()` use this so the wire form is registry-free. The receiving worker calls `hydrate_working_memory(raw)` (`pipelex/temporal/tprl_pipe/hydration.py`) to rebuild typed `Stuff`/`StuffContent` instances using `get_class_registry()`.
+
+3. **Two patches in `hydration.py` worth knowing about:**
+ - `_validate_as_known_class` round-trips through `model_dump()` to defeat cross-exec class-identity mismatches (kajson eagerly rebuilds instances from `__class__` metadata using one class identity, then `load_from_crate` re-execs the source and replaces the registry entry with a new identity — same name, different `id()` — and Pydantic's `model_validate` would otherwise reject the older instance).
+ - `_hydrate_list_item` handles `Anything[]` lists by reading per-item `__class__` metadata, falling back to `TextContent`.
+
+4. **`Optional[BaseModel]` payload converter fix** (`pipelex/temporal/temporal_data_converter.py:107-121, 133-135`): added so `PipeOutput.graph_spec: GraphSpec | None` deserializes via kajson instead of falling through to the default JSON path.
+
+5. **Schema-to-model enum forward-refs fix** (`pipelex/cogt/content_generation/schema_to_model.py:120-122`): `model_rebuild` now uses *all* generated types (models + enums), so forward references to dynamically generated `Enum` classes resolve correctly.
+
+6. **The propagation hack** (`pipelex/temporal/tprl_pipe/wf_pipe_router.py:71-80`): after loading the crate into the per-workflow registry, copies all dynamic classes into the worker process's *global* `KajsonManager` registry. The stated reason: child workflows and activities don't see the per-workflow `ContextVar`, so they need the global to be primed.
+
+### The cogt layer uses a fundamentally different (better) pattern
+
+`pipelex/temporal/tprl_content_generation/*` ships dynamic classes via JSON schema in the assignment (e.g. `ObjectAssignment.object_class_schema`). The receiving side calls `model_class_from_json_schema(...)` (`pipelex/cogt/content_generation/schema_to_model.py:24-49`) which `exec`s the generated source on the fly. The reconstructed class carries `__kajson_class_source__` as a class attribute. The Temporal payload converter (`pipelex/temporal/temporal_data_converter.py:60-79`) embeds that source in payload metadata as `kajson_class_source`. On the receiving worker, `kajson.loads(..., class_source_code=...)` exec's the source into a temporary `ClassRegistry` (`kajson/kajson.py:141-159`) merged with the explicit registry. Result: **payloads are self-describing; no registry coordination across workers.** This is independent of the LibraryCrate path and is not affected by this change.
+
+### Why the propagation hack breaks in distributed Temporal
+
+A worker is a process polling a task queue. In distributed deployments multiple workers poll the same queue and any of them can pick up a child workflow or activity. ContextVars don't cross processes. So:
+
+| Hop | Crosses worker? | Receiver has dynamic classes? |
+| --- | --- | --- |
+| User submitter → `WfPipeRouter` (top-level) | yes | yes — `WfPipeRouter` re-loads the crate from `PipeJob.library_crate` on its own worker |
+| `WfPipeRun` parent → child `WfPipeRouter` | possibly different worker | yes — child gets the same `PipeJob` carrying the crate, re-loads on its worker |
+| Child `WfPipeRouter` → `WfPipeRun` parent (rehydrate at `wf_pipe_run.py:50-52`) | possibly different worker | **NO — this is the bug.** `WfPipeRun` never opens its own per-workflow library. `hydrate_working_memory` falls back to the parent's global registry, which only has classes if the propagation hack ran on the *same* worker that the parent happens to be on. |
+| `WfPipeRun` → `act_deliver` activity | possibly different worker | **NO.** The activity receives a typed `pipe_output` (because parent rehydrated). The activity worker's global registry doesn't have the crate. |
+| `WfPipeRouter` → `act_flush_trace_events` | possibly different worker | safe — only ships `TraceEvent` (built-in) |
+| `WfPipeRun` → `act_assemble_graph` | possibly different worker | safe — only ships scalar IDs |
+| All cogt boundaries | possibly different worker | safe — `__kajson_class_source__` mechanism |
+
+So the entire problem reduces to **two lines**: the parent-side rehydrate at `wf_pipe_run.py:50-52`. Remove it and `act_deliver` naturally receives the raw dict; the propagation hack has no remaining customer and gets deleted.
+
+### Drawbacks of "raw end-to-end" (already discussed and accepted)
+
+- **D1 — typed rendering for dynamic concepts is lost.** Built-in `StuffContent` subclasses (`ImageContent`, `PdfContent`, `MermaidContent`, `HtmlContent`, `TextContent`, `NumberContent`) are statically registered on every worker, so they can be rehydrated locally from `__class__` metadata. User-defined dynamic concepts fall back to a generic field-walking render (JSON-in-markdown / `<pre>`-in-HTML). User: **accepted**.
+- **D2 — dual rendering path in `DeliveryExecutor`** (typed when `working_memory` is set, raw-dict when `working_memory_raw` is set). Mirrors the existing `working_memory` / `working_memory_raw` dual field on `PipeJob` and `PipeOutput`. User: **accepted (already a known pattern)**.
+- **D3 — fix is delivery-scoped.** Other consumers of typed dynamic classes still need the crate. We confirmed (Explore agent enumeration) that no other consumers exist on the Temporal hot path; the cogt layer is independent. User: **OK, focus on delivery; flag others if any (none found)**.
+- **D4 — `model_dump` subtleties.** `dump_for_temporal()` and `smart_dump()` already use `serialize_as_any=True`, so subclass-specific fields and computed fields are preserved. User: **noted**.
+- **D5 — debuggability loss** (raw dict instead of typed `__repr__` inside the activity). User: **accepted**.
+
+---
+
+## Plan
+
+### 1. Stop the parent rehydrate — `pipelex/temporal/tprl_pipe/wf_pipe_run.py`
+
+- Remove lines 49-52 (the `if pipe_output.working_memory_raw is not None: pipe_output.working_memory = hydrate_working_memory(...)`).
+- Forward `pipe_output` to `act_deliver` with `working_memory_raw` populated and `working_memory=None`.
+- Remove the now-unused `hydrate_working_memory` import.
+
+### 2. Delete the global-registry-propagation block — `pipelex/temporal/tprl_pipe/wf_pipe_router.py`
+
+- Delete lines 71-80 (`# 4. Propagate dynamic classes to global registry...`).
+- Renumber the trailing comment "5. Hydrate WorkingMemory" → "4." for consistency.
+- The internal `hydrate_working_memory(workflow_arg.working_memory_raw)` at line 84 still works because the `_library_id` `ContextVar` is set just above (line 66) and `get_class_registry()` (`pipelex/hub.py:397-408`) routes to the per-workflow registry.
+
+### 3. Make `DeliveryExecutor` accept either typed or raw `WorkingMemory` — `pipelex/pipe_run/delivery_executor.py`
+
+`pipe_output.working_memory` may now be `None` while `pipe_output.working_memory_raw` is populated. `DeliveryExecutor.execute` and the helpers (`generate_result_files`, `_generate_main_stuff_files`, `_store_results`) need to branch on which field is populated.
+
+Rendering policy per file:
+
+| File | Typed path (today) | Raw-dict path (new) |
+| --- | --- | --- |
+| `working_memory.json` | `working_memory.smart_dump()` | `clean_json_dumps(working_memory_raw)` directly — already a dict produced by `dump_for_temporal()` |
+| `main_stuff.json` | `main_stuff.content.rendered_json_async()` | extract main stuff dict from raw root by alias name; `clean_json_dumps` it |
+| `main_stuff.md` / `.html` / `viewer.html` | `rendered_markdown_async()` / `rendered_html_async()` / `render_stuff_viewer()` | locally rehydrate the *single* main stuff using `__class__` lookup in the **global** registry (built-ins always present); on success, call existing typed renderers. On lookup miss (dynamic concept class not registered locally), fall back to a generic field-walking render that just dumps the dict as pretty JSON-in-markdown / `<pre>`-in-HTML. |
+| `graphspec.json` / `mermaidflow.*` / `reactflow.html` | unchanged — `graph_spec` is independent of working_memory | unchanged |
+
+Implementation note: introduce a small private helper `_local_hydrate_main_stuff(raw_root: dict, main_stuff_name: str) -> Stuff | None` that tries `Stuff.model_validate` against the registry's `__class__`-named class for the content, returning `None` if the class isn't locally available. Isolates the fallback decision in one place.
+
+### 4. Verify-only (no change)
+
+- `pipelex/temporal/tprl_pipe/wf_pipe_router.py:202-204` — `prepare_for_temporal()` on return when `library_crate is not None`. Keep — still correct for the WfPipeRouter→WfPipeRun edge.
+- `pipelex/temporal/tprl_pipe/temporal_pipe_router.py:84-86` — top-level caller rehydrate. Keep — runs in the submitter's process which already has the classes loaded.
+
+### 5. Tests
+
+The repo has a Temporal-distributed validation harness — invoke the `temporal-e2e-validate` skill (true 3-process setup: server + worker + submitter, validates cross-process topology).
+
+**Deterministic repro is now wired in.** Recent (staged) work added:
+
+- **Worker scope split** (`pipelex/pipelex.toml:528-560`, `pipelex/temporal/config_temporal.py:59-91`, `pipelex/temporal/worker_cli.py`): named scopes `router` (`disable_all_activities=true`) and `runner` (`disable_all_workflows=true`) selectable via `--scope`. Two scoped workers on the same `temporal_task_queue` force every activity to be picked up by a different Python process than the workflow that scheduled it, making cross-process registry hops deterministic instead of probabilistic.
+- **Repro script** `.claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py`: mirrors the cloud / `start_pipeline` + webhook submission path by attaching a `DeliveryAssignment(storage=StorageTarget())`. This makes `wf_pipe_run.py:79-96` schedule `act_deliver` on the runner with hydrated `pipe_output`. Plain `pipelex run bundle` does **not** trigger this — without a `delivery_assignment`, `act_deliver` is skipped and the runner only ever sees trace/graph activity payloads (no dynamic-class decoding).
+- **In-process scope test** `tests/integration/pipelex/temporal/library_crate/test_distributed_scopes.py`: validates the scope mechanism resolves correctly (cannot reproduce the cross-process bug because both workers share the same `KajsonManager`; documented in the test's module docstring).
+
+Currently observable failure (with the parent rehydrate at `wf_pipe_run.py:50-52` still in place) when running the repro script against router+runner:
+
+```
+KajsonDecoderError: Class '<bundle>__<DynamicConcept>' not found
+ in module 'builtins' or global registry
+ApplicationError: Failed decoding arguments
+ → temporalio/worker/_activity.py:566 (data_converter.decode_wrapper)
+```
+
+surfaced in the **runner** tmux session, raised by the activity worker before the `act_deliver` body runs. After the fix in steps 1–3, this should resolve to a successful `act_deliver` execution where the activity receives `working_memory_raw` (no class lookup needed).
+
+Add/extend:
+
+- Promote the repro script into a pytest case (or call it from one) that drives the deterministic 2-scope failure path and asserts success post-fix.
+- Verify `main_stuff.md` for a dynamic concept passes via the generic fallback path.
+- Regression test that distributed delivery does not import the dynamic concept module on the activity worker (e.g. run the activity worker process with a stripped library set, confirm no `ClassRegistry` lookup errors).
+- Existing crate isolation tests (`temporal-test-crate` skill) should continue to pass — and now be more strongly isolated since the propagation no longer pollutes the global.
+
+---
+
+## Files to modify
+
+- `pipelex/temporal/tprl_pipe/wf_pipe_run.py` — drop parent rehydrate (lines 49-52), drop unused import.
+- `pipelex/temporal/tprl_pipe/wf_pipe_router.py` — delete propagation block (lines 71-80), tidy step comment.
+- `pipelex/pipe_run/delivery_executor.py` — dual-mode rendering; new `_local_hydrate_main_stuff` helper.
+- (tests) — extend the temporal-e2e harness; add a dynamic-concept delivery test.
+
+## Files to leave alone
+
+- `pipelex/temporal/temporal_data_converter.py` — `Optional[BaseModel]` and `BaseModel | list[BaseModel]` paths remain correct.
+- `pipelex/temporal/tprl_pipe/hydration.py` — still used by `WfPipeRouter` internally (per-workflow scope) and by `temporal_pipe_router.py` at the submitter side. Both contexts have the registry primed.
+- `pipelex/core/memory/working_memory.py` — `dump_for_temporal()` already preserves `__class__` metadata on `ListContent` items, which the local-hydrate fallback uses.
+- `pipelex/temporal/tprl_content_generation/*` — independent system (`__kajson_class_source__`).
+
+---
+
+## Verification checklist
+
+1. `make agent-check` from `pipelex/` — type checks + linting clean.
+2. `make agent-test` from `pipelex/` — full unit/integration suite green.
+3. Deterministic distributed repro via `temporal-e2e-validate` Tier 2b (router + runner scoped workers + `repro_runner_registry_bug.py`) — runs end-to-end with a hydrated `pipe_output` reaching `act_deliver` on the runner, no `KajsonDecoderError`, no class lookup on the runner side. Activity worker never loads the crate.
+4. `temporal-test-crate` skill — concurrent workflows with conflicting concept names still produce correct outputs.
+5. Single-worker smoke: local `pipelex run` against a pipe using a dynamic concept; delivery files byte-equivalent for built-in content types and structurally-equivalent (JSON/MD/HTML) for dynamic concepts.
diff --git a/wip/archive/scoped-registry-teardown.md b/wip/archive/scoped-registry-teardown.md
new file mode 100644
index 000000000..a94359b09
--- /dev/null
+++ b/wip/archive/scoped-registry-teardown.md
@@ -0,0 +1,67 @@
+# Scoped Registry Teardown: ContextVar Token Save/Restore
+
+## Problem
+
+In `pipelex/temporal/tprl_pipe/wf_pipe_router.py`, the scoped ClassRegistry cleanup in the `finally` block unconditionally sets the scoped registry to `None`:
+
+```python
+finally:
+ if library_crate is not None:
+ KajsonManager.set_scoped_class_registry(None)
+```
+
+This is not stack-safe. If a nested or re-entrant workflow sets its own scoped registry, the inner workflow's `finally` block resets to `None`, clobbering the outer workflow's scoped registry. The outer workflow then resolves classes from the global registry instead of its own scoped registry, potentially getting the wrong class definitions.
+
+## Proposed Solution
+
+Use Python's `ContextVar` token save/restore mechanism. `ContextVar.set()` returns a `Token` object, and `ContextVar.reset(token)` restores the value that was in effect before that `set()` call. This is the standard pattern for push/pop scoping with ContextVars.
+
+### Changes to Kajson (`kajson/kajson/kajson_manager.py`)
+
+1. Import `Token` from `contextvars`
+2. Change `set_scoped_class_registry` to return the Token:
+
+```python
+from contextvars import ContextVar, Token
+
+@classmethod
+def set_scoped_class_registry(cls, registry: ClassRegistryAbstract | None) -> Token[ClassRegistryAbstract | None]:
+ return _scoped_class_registry.set(registry)
+```
+
+3. Add a `reset_scoped_class_registry` method:
+
+```python
+@classmethod
+def reset_scoped_class_registry(cls, token: Token[ClassRegistryAbstract | None]) -> None:
+ _scoped_class_registry.reset(token)
+```
+
+### Changes to Pipelex (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`)
+
+```python
+registry_token = None
+try:
+ if library_crate is not None:
+ global_registry = KajsonManager.get_class_registry()
+ scoped_registry = CompositeClassRegistry(parent=global_registry)
+ registry_token = KajsonManager.set_scoped_class_registry(scoped_registry)
+ # ... load crate, hydrate WM ...
+
+ # ... run pipe ...
+finally:
+ # ... library teardown ...
+ if registry_token is not None:
+ KajsonManager.reset_scoped_class_registry(registry_token)
+```
+
+### Why Token-Based
+
+- The guard changes from `if library_crate is not None` to `if registry_token is not None`, which is more precise (only resets if we actually set a scoped registry)
+- Properly handles nesting: inner workflow restores to outer's scoped registry, not to `None`
+- Standard Python pattern for ContextVar scoping — no custom stack management needed
+
+### Impact on Existing Code
+
+- `set_scoped_class_registry` return type changes from `None` to `Token`. This is backward-compatible for callers that discard the return value. No existing callers check the return value.
+- The Kajson unit tests (`test_scoped_context_var`) should be updated to verify the token-based reset restores the previous value.
diff --git a/wip/archive/tier2-live-bug-recap.md b/wip/archive/tier2-live-bug-recap.md
new file mode 100644
index 000000000..a2f3b4149
--- /dev/null
+++ b/wip/archive/tier2-live-bug-recap.md
@@ -0,0 +1,69 @@
+# Tier 2 Live Bug: DynamicConceptTestGreeting not found
+
+## The bug
+
+When running Tier 2 (deferred hydration) in **live mode** (real LLM calls), the parent workflow `WfPipeRouter` fails to decode the child workflow `WfMakeObject`'s return value:
+
+```
+KajsonDecoderError: Class 'DynamicConceptTestGreeting' not found in module 'builtins' or global registry
+```
+
+Dry-run passes because mock data never triggers the real structured output path.
+
+## How to reproduce
+
+```bash
+# Start Temporal server + worker, then:
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \
+ --pipe dynamic_greeting_sequence \
+ --temporal --no-logo --graph
+```
+
+## Execution flow where it breaks
+
+```
+WfPipeRouter (parent workflow)
+ → set_current_library(wf_library_id) # sets ContextVar
+ → load_from_crate() # registers DynamicConceptTestGreeting in workflow-scoped registry
+ → PipeSequence.run_pipe()
+ → PipeLLM (generate_greeting)
+ → ContentGeneratorChild.make_object_direct()
+ → WfMakeObject (child workflow)
+ → act_llm_gen_object (activity)
+ → model_class_from_json_schema() creates class via exec()
+ → LLM generates structured Greeting instance
+ → Activity returns Greeting (BaseModel subclass)
+ → WfMakeObject returns Greeting to parent
+ → Temporal calls _convert_payloads to decode child result ← FAILS HERE
+```
+
+## What we verified with diagnostic logs
+
+1. **`to_payload` is NOT called for the Greeting activity return.** Only called for `ObjectAssignment` (activity input). The Greeting is serialized by Temporal's default JSON converter, NOT our `BaseModelPayloadConverter`. Therefore no `kajson_class_source` metadata is attached.
+
+2. **`from_payload` IS called for the child workflow result.** The payload data is:
+ ```json
+ {"message": "Bonjour le monde", "language": "French", "__class__": "DynamicConceptTestGreeting", "__module__": "builtins"}
+ ```
+ With `class_source_code: None` (no metadata).
+
+3. **The ContextVar is NOT propagated into `_convert_payloads`.** Temporal's internal `_apply_resolve_child_workflow_execution` → `_convert_payloads` runs outside the workflow coroutine's async context. So `get_class_registry()` falls back to the global KajsonManager registry, which doesn't have the dynamic class.
+
+## Two independent problems
+
+**Problem A: `to_payload` not called for activity results.**
+The Greeting returned by `act_llm_gen_object` never passes through our `BaseModelPayloadConverter.to_payload()`. Reason unknown — possibly Temporal SDK uses a different serialization path for activity return values, or the exec'd class isn't recognized as `BaseModel` by the converter's `isinstance` check.
+
+**Problem B: ContextVar not available in `_convert_payloads`.**
+Even if `to_payload` worked and attached `kajson_class_source` metadata, the parent workflow's `_convert_payloads` runs in a Temporal-internal context where the `_library_id` ContextVar is not set. So `get_class_registry()` returns the global registry, not the workflow-scoped one.
+
+## What was fixed so far (kajson repo)
+
+**Decoder fallback** in `kajson/json_decoder.py`: When `sys.modules[module_name]` exists but `getattr` fails (the `builtins` case), the decoder now falls through to check `KajsonManager.get_class_registry()` before raising. Previously it raised immediately. This fix is correct but insufficient alone — the class isn't in the global registry either.
+
+Test: `kajson/tests/unit/test_global_registry_fallback.py` — passes.
+
+## What was NOT fixed
+
+The `_collect_class_sources` helper added to `temporal_data_converter.py` is irrelevant since `to_payload` isn't called for the Greeting. The debug logs added should be removed.
diff --git a/wip/archive/tier2-live-registry-propagation.md b/wip/archive/tier2-live-registry-propagation.md
new file mode 100644
index 000000000..743183a4b
--- /dev/null
+++ b/wip/archive/tier2-live-registry-propagation.md
@@ -0,0 +1,123 @@
+# Tier 2 Live: ContextVar Registry Doesn't Cross Temporal Boundaries
+
+**Status:** Open — needs architectural decision
+**Branch:** `fix/Temporal-Img`
+**Date:** 2026-04-01
+
+## What works
+
+- Tier 2 **dry-run** passes (no real LLM calls, `ObjectAssignment` never hits the structured output path)
+- Tiers 1, 3, 4, 5 pass in both dry-run and live mode
+- The `ObjectAssignment.__init__` deserialization fix (removing the eager class registry check) is correct and already applied on this branch
+
+## What fails
+
+Tier 2 **live** — `dynamic_concept_sequence.mthds` with `--pipe dynamic_greeting_sequence` through Temporal.
+
+The bundle defines a dynamic concept `Greeting` (with `message` and `language` fields) and a `PipeLLM` that outputs structured data into that concept using `structuring_method = "direct"`.
+
+## Error
+
+```
+Class 'dynamic_concept_test__Greeting' not found in registry
+```
+
+Thrown from `llm_generate.py:32`:
+```python
+content_class = get_class_registry().get_required_base_model(name=content_class_name)
+```
+
+## Root cause
+
+`get_class_registry()` uses a `ContextVar` (`_library_id` in `hub.py:462`) to resolve the workflow-scoped registry. But ContextVars don't propagate across Temporal's execution boundaries:
+
+```
+WfPipeRouter (ContextVar set here via set_current_library)
+ → pipe.run_pipe()
+ → ContentGeneratorChild.make_object_direct()
+ → child workflow WfMakeObject (NEW async context — ContextVar is None)
+ → activity act_llm_gen_object (ANOTHER context — ContextVar is None)
+ → llm_gen_object()
+ → get_class_registry() → returns GLOBAL registry → class not found
+```
+
+The dynamic concept class `dynamic_concept_test__Greeting` was registered in `WfPipeRouter`'s **workflow-scoped** `ClassRegistry`. But two hops later — inside the activity of a child workflow — `get_class_registry()` returns the **global** registry, which doesn't have it.
+
+## Why dry-run works
+
+In dry-run mode, `PipeLLM` never reaches `ContentGeneratorChild.make_object_direct()` — the dry-run path generates mock output without calling the LLM, so it never needs to look up the dynamic class from the registry inside an activity.
+
+## Why other tiers aren't affected
+
+- **Tier 1** (native text sequence): Uses `PipeLLM` with `Text` output (a native concept, always in the global registry). No structured output, no `ObjectAssignment`.
+- **Tier 3** (parallel): Same — text-only pipes, no dynamic concepts in structured output position.
+- **Tiers 4-5** (image): Image generation doesn't go through `ObjectAssignment` or `llm_gen_object`. Images use `ImgGenAssignment` → `act_img_gen` which doesn't need the class registry.
+
+## Fix already applied on this branch
+
+Removed `ObjectAssignment.__init__` class registry validation (`assignment_models.py:90-94`). This was causing a **separate** deserialization failure: when Temporal's data converter called `kajson.loads()` to deserialize the `ObjectAssignment` argument for `WfMakeObject`, pydantic called `__init__` which checked the registry. Removing it fixed the deserialization, but the same lookup fails later in the activity.
+
+## Approach options
+
+### A) Embed the class schema in ObjectAssignment
+
+Add the Pydantic model's JSON schema (or the field definitions) as a field on `ObjectAssignment`. The activity reconstructs the class locally from the schema without needing the registry. This is the "carry everything you need" pattern, similar to how `LibraryCrate` carries pipe definitions.
+
+**Pros:** Self-contained, no cross-boundary coupling, works for any Temporal topology.
+**Cons:** Increases payload size, needs a "reconstruct Pydantic class from schema" utility.
+
+### B) Propagate library_id through Temporal headers
+
+Use Temporal's workflow/activity interceptors to copy the `_library_id` ContextVar into Temporal headers on dispatch, and restore it on the receiving end. This is the transparent approach.
+
+**Pros:** No changes to assignment models, ContextVar "just works" across boundaries.
+**Cons:** Requires Temporal interceptor infrastructure, the receiving side still needs access to the same `LibraryManager` state (child workflows and activities must be on the same worker).
+
+### C) Pre-register dynamic classes in the global registry
+
+When `WfPipeRouter.load_from_crate()` runs, also register dynamic classes in the global registry (not just the workflow-scoped one). Clean up on workflow teardown.
+
+**Pros:** Simplest change, everything finds the class.
+**Cons:** Breaks per-workflow isolation — concurrent workflows with conflicting class names will clobber each other (exactly the bug the scoped registry was designed to prevent).
+
+### D) WfMakeObject loads the crate itself
+
+Pass the `LibraryCrate` to `WfMakeObject` alongside the `ObjectAssignment`. The child workflow sets up its own scoped registry before the activity runs.
+
+**Pros:** Reuses existing crate-loading pattern from `WfPipeRouter`.
+**Cons:** Increases child workflow payload, adds setup/teardown overhead to every structured output call.
+
+## Key files
+
+| File | Role |
+|------|------|
+| `pipelex/hub.py:381-386, 462-467` | `get_class_registry()` and `_library_id` ContextVar |
+| `pipelex/temporal/tprl_pipe/wf_pipe_router.py:46-68` | Crate loading and registry setup |
+| `pipelex/temporal/tprl_content_generation/wf_make_object.py` | Child workflow receiving `ObjectAssignment` |
+| `pipelex/cogt/content_generation/llm_generate.py:32` | Activity calling `get_class_registry()` |
+| `pipelex/cogt/content_generation/assignment_models.py:86` | `ObjectAssignment` model (init check removed) |
+| `pipelex/temporal/temporal_data_converter.py:77` | Kajson deserialization using `get_class_registry()` |
+| `tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds` | Test bundle |
+
+## Reproduction
+
+```bash
+# Start Temporal server + worker
+tmux new-session -d -s temporal-server 'temporal server start-dev'
+sleep 3
+tmux new-session -d -c "$PWD" -s temporal-worker \
+ '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed'
+sleep 4
+
+# This passes (dry-run)
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \
+ --pipe dynamic_greeting_sequence \
+ --temporal --dry-run --mock-inputs --no-logo
+
+# This fails (live)
+.venv/bin/pipelex run bundle \
+ tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \
+ --pipe dynamic_greeting_sequence \
+ --temporal --no-logo
+```
diff --git a/wip/deferred-items.md b/wip/deferred-items.md
new file mode 100644
index 000000000..1493e019d
--- /dev/null
+++ b/wip/deferred-items.md
@@ -0,0 +1,70 @@
+# Deferred & Future Items
+
+> Items explicitly out of scope for the current implementation phases.
+> Collected here for roadmap visibility and to preserve design context.
+
+---
+
+## Architectural Direction (Future)
+
+Long-term goals documented for vision alignment. Not planned for implementation now. See [future-crate-first-architecture.md](future-crate-first-architecture.md) for the full design rationale.
+
+### Crate-First Architecture
+
+**Status**: Future vision — Phase 6 is the first step on this path.
+
+Invert the current loading-driven architecture into a crate-driven one with three cleanly separated phases: **Collect** (resolve deps, fetch remote, gather all blueprints) → **Build** (construct one crate, optionally strip to transitive closure) → **Load** (same `load_from_crate()` on submitter and worker).
+
+### Crate Stripping (Transitive Closure)
+
+**Status**: Future — depends on Phase 6a's blueprint collector.
+
+From the entry pipe, walk the transitive closure of pipe and concept dependencies to determine the minimal subset needed. Strip the crate to only those concepts, pipes, and domains. Once all blueprints (main + deps) are accumulated in 6a, stripping is a pure data transform before crate construction.
+
+### Library Fingerprint Validation
+
+**Status**: Future — no immediate need.
+
+Verify that the worker's base library (PIPELEXPATH) matches the API's expectation. Detect version drift between API and worker deployments.
+
+### Cross-Worker Cache via Shared Storage
+
+**Status**: Future — complementary to crate stripping.
+
+Cache loaded crates in shared storage (Redis, S3) so multiple workers don't redundantly load the same crate. Keyed by fingerprint. Smaller crates from stripping make this more effective.
+
+---
+
+## Distributed Tracing — Deferred Items
+
+Items related to the event log and graph tracing system (Phase 4.5). Also documented in [distributed-tracing-and-reporting.md](distributed-tracing-and-reporting.md).
+
+### Event Log Backends
+
+| Item | Status | Context |
+|---|---|---|
+| DynamoDB backend | **Shipped** (2026, `feature/dynamodb-tracer`) | `pipelex/tracing/dynamodb_event_log.py` (+ `TEMPORAL_DYNAMODB` variant). Schema-compatible with `pipelex-api-infra`'s `TraceEventDynamoDBAdapter`. Selected via `[pipelex.tracing] backend = "dynamodb"`. |
+| SQLite backend | **Not planned** — NDJSON + DynamoDB cover the matrix | Listed in the original analysis (`distributed-tracing-and-reporting.md`); not built. Re-evaluate only if a use case appears that NDJSON can't serve and DynamoDB is overkill. |
+
+### Event Log Protocol Extensions
+
+| Item | Status | Context |
+|---|---|---|
+| `subscribe()` on EventLogProtocol | **Deferred** — no consumer exists yet | Polling-based reads are sufficient for now; add when a real-time consumer exists. |
+| Real-time progress consumer | **Deferred** — no consumer exists yet | The protocol supports it; build when needed. |
+| TraceSinkProtocol abstraction | **Deferred** — EventLogProtocol is sufficient | The higher-level sink abstraction can wait for the cloud backend. |
+| Large data externalization | **Deferred** — inline data is sufficient for now | Add StorageProvider offloading when traces grow too large. |
+
+### Tracing Architecture
+
+| Item | Status | Context |
+|---|---|---|
+| Unified code path (event log everywhere) | **Effectively unified** | `pipeline_run_setup.py:282` wires `set_event_log()` on the report delegate for direct mode too (using `workflow_id="direct"`). The same `EventLogProtocol` machinery runs in both modes; only the in-process vs. Temporal lifecycle differs. |
+| Auto-cleanup of traces | **Deferred** — manual cleanup is safer for now | NDJSON traces persist on disk until manually deleted. Safer for debugging and allows usage reporting to run after graph generation. DynamoDB cleanup follows TTL on the table. |
+
+### Known Issues
+
+| Item | Status | Context |
+|---|---|---|
+| Causal event ordering in assembler | **Known limitation** — works for current topologies | `read_events()` sorts by `(workflow_id, sequence)` which groups by lexicographic workflow ID, not execution order. In parent/child workflow topologies, this can cause `_stuff_producer_map` overwrites in the wrong order during `GraphSpecAssembler.pass_one()`, producing incorrect DATA edge sources. Consider sorting by timestamp or processing events in a topology-aware order. Flagged by codex on PR #796. |
+| PipeCondition SELECTED_OUTCOME wiring | **Deferred** — no production use yet | PipeCondition has no tracer calls in production code currently. If SELECTED_OUTCOME edges are needed, that's a separate change. |
diff --git a/wip/distributed-tracing-and-reporting.md b/wip/distributed-tracing-and-reporting.md
new file mode 100644
index 000000000..e77878279
--- /dev/null
+++ b/wip/distributed-tracing-and-reporting.md
@@ -0,0 +1,697 @@
+# Distributed Tracing & Reporting Architecture
+
+> **Status (2026-05-04): SUPERSEDED — kept as historical analysis.**
+> The implementation shipped via `feature/dynamodb-tracer` and chose **NDJSON** locally + **DynamoDB** in the cloud, **not** SQLite or Redis Streams as recommended below. Inside Temporal workflows, events are buffered in a `BufferingEventLog` and flushed to the real backend via the `act_flush_trace_events` activity — not via a workflow-side interceptor.
+>
+> See:
+> - `pipelex/tracing/event_log_protocol.py`, `ndjson_event_log.py`, `dynamodb_event_log.py`, `buffering_event_log.py`, `event_log_factory.py`
+> - `pipelex/temporal/tprl_pipe/wf_pipe_router.py` (per-workflow `set_event_log` + flush)
+> - `pipelex/temporal/tprl_pipe/act_flush_trace_events.py` (off-thread persistence)
+> - [01-master-plan.md](01-master-plan.md) "Phase 4.5 Step 6 — as built"
+>
+> Decisions from this doc that no longer apply: SQLite local backend (Decision 2), Redis Streams cloud backend, ContextVar for the event log client (the implementation keys per-workflow inside `ReportingManager`). The high-level event-log architecture (event types, assembler, fire-and-forget emit, durability-on-emit) all match what shipped.
+>
+> **Original metadata:** Pre-planning analysis — decision gates marked with ❓
+> **Date**: 2026-03-29
+> **Related**: [archive/00-master-plan.md](archive/00-master-plan.md), [archive/phase2-crate-propagation-rationale.md](archive/phase2-crate-propagation-rationale.md)
+> **Scope**: GraphTracer (execution graph capture) + ReportingManager (AI usage/cost tracking) in Temporal distributed execution
+
+---
+
+## Problem Statement
+
+Two in-memory systems silently break under Temporal distributed execution:
+
+| System | What it does | How it breaks |
+|--------|-------------|---------------|
+| **GraphTracerManager** | Accumulates pipe execution events (nodes, edges, timing, I/O) into a GraphSpec for visualization | Singleton on each process — workers trace nothing because no tracer is opened; all graph events are lost |
+| **ReportingManager** | Tracks AI token usage/costs per pipeline run | Workers auto-create orphaned registries that never sync back to the API process; usage data is lost |
+
+Both systems were designed for single-process execution. The master plan (Phases 0-4) focused on library propagation and class registry scoping, but overlooked these two cross-cutting concerns.
+
+### Requirements
+
+| Requirement | Priority | Rationale |
+|-------------|----------|-----------|
+| **Survive failures** | Must-have | Cost reporting for failing workflows is not optional — we pay for those tokens |
+| **Real-time progress** | Must-have | Track graph construction and cost accumulation while pipeline runs |
+| **Per-step granularity** | Must-have | Each pipe execution is a distinct trace event with its own cost attribution |
+| **Large graphs with data** | Must-have | Full I/O data capture (`--graph-full-data`) for debugging |
+| **Local dev first** | Priority | But only if architecturally aligned with cloud — shared principles, swappable backend |
+
+---
+
+## Current Architecture Analysis
+
+### GraphTracer: What Happens Today on Temporal
+
+```
+API Process Worker Process A Worker Process B
+----------- ---------------- ----------------
+runner.execute_pipeline()
+ pipeline_run_setup()
+ GraphTracerManager.open_tracer() (singleton is separate instance) (singleton is separate instance)
+ → GraphContext on JobMetadata
+ WfPipeRouter.run(pipe_job)
+ pipe.run_pipe()
+ graph_context is NOT None ✓
+ GraphTracerManager.get_instance()
+ → returns None ✗ (same: returns None)
+ → tracing silently skipped → tracing silently skipped
+ return PipeOutput (no graph_spec)
+ tracer_manager.close_tracer()
+ → GraphSpec with ZERO nodes
+```
+
+**Root cause**: `GraphTracerManager` is a process-local singleton (`ABCSingletonMeta`). The API process opens a tracer, but `run_pipe()` executes on a different process where the singleton was never initialized. The guard `if tracer_manager is not None` silently skips all tracing.
+
+**What's preserved**: `GraphContext` (graph_id, parent_node_id, node_sequence, data_inclusion) IS serialized on `JobMetadata` and correctly flows through PipeJob to workers. The context propagation infrastructure exists — only the tracer endpoint is missing.
+
+### ReportingManager: What Happens Today on Temporal
+
+```
+API Process Worker Process
+----------- ----------------
+hub.set_report_delegate(reporting_mgr)
+ reporting_mgr.open_registry(run_id)
+ WfPipeRouter.run(pipe_job)
+ pipe_operator.run_pipe()
+ llm_worker.run()
+ reporting_delegate.report_inference_job()
+ → _get_registry(run_id) auto-creates orphan ✗
+ → usage recorded in worker-local registry
+ return PipeOutput
+ reporting_mgr.generate_report(run_id)
+ → report from API-side registry only
+ → worker-side usage is LOST
+```
+
+**Root cause**: Workers get a `reporting_delegate` via `hub.get_report_delegate()`, which is a process-local reference. The `_get_registry()` method auto-creates registries for unknown pipeline IDs (there's already a TODO comment acknowledging this), but these registries live in the worker's memory and are never collected.
+
+### What's Already in Place
+
+| Infrastructure | Status | Relevance |
+|---------------|--------|-----------|
+| `GraphContext` serializable on `JobMetadata` | ✅ Working | Context flows to workers — can seed local tracers |
+| `PipeOutput.graph_spec: GraphSpec | None` | ✅ Exists | Return channel exists but insufficient (only on success) |
+| `GraphSpec` is a flat Pydantic model | ✅ Working | Easy to serialize, merge, transport |
+| `TokensUsage` types are Pydantic models | ✅ Working | Easy to serialize and aggregate |
+| `ObserverProtocol` + `LocalObserver` | ✅ Working | Existing event pattern — writes JSONL to shared dir |
+| `StorageProviderAbstract` (Local, S3, GCP) | ✅ Working | Pluggable storage with existing backends |
+| `LibraryCrate` propagation on PipeJob | ✅ Working | Pattern for adding data to PipeJob |
+| StoragePayloadCodec (Phase 5) | 🔲 Planned | Would solve payload size limits |
+
+### Why PipeOutput Piggyback Is Insufficient
+
+The initial instinct is to return graph fragments and usage records in PipeOutput, like LibraryCrate propagation. This fails on three counts:
+
+1. **Lost on failure**: If a workflow crashes, PipeOutput is never returned. Cost reporting for failing workflows is unacceptable to lose — those tokens were consumed and billed.
+2. **No real-time visibility**: Data is only available when the entire workflow tree completes. For long pipelines, this means no progress tracking.
+3. **Payload pressure**: Graph data with full I/O content can be large. Adding it to PipeOutput pushes toward Temporal's 2MB limit, making Phase 5 (StoragePayloadCodec) a hard prerequisite.
+
+---
+
+## Recommended Architecture: Event Log
+
+The right abstraction is an **append-only event log**. Workers emit events as they happen. Events are durable the instant they're written. Consumers can read in real-time or after completion.
+
+This matches the existing `LocalObserver` pattern (JSONL files in a shared directory) and generalizes cleanly to cloud backends (Redis Streams, CloudWatch, S3).
+
+### Core Concept
+
+```
+ ┌──────────────┐
+Worker A ──emit(PipeStartEvent)────►│ │
+Worker A ──emit(UsageEvent)────────►│ Event Log │◄────read()──── Real-time consumer
+Worker B ──emit(PipeStartEvent)────►│ (per run) │◄────read()──── Post-exec assembler
+Worker B ──emit(PipeEndError)──────►│ │
+ └──────────────┘
+```
+
+**Key properties**:
+- **Fire-and-forget emit**: Workers write events and move on. No waiting for ack (local: file append; cloud: async publish).
+- **Durable on write**: Events survive worker crashes because they're persisted before the worker continues.
+- **Keyed by pipeline_run_id**: Each pipeline run has its own event stream.
+- **Ordered per writer**: Events from a single worker are naturally ordered (sequential writes). Cross-worker ordering uses graph structure (parent_node_id), not timestamps.
+- **Backend-agnostic**: The protocol is the same whether backed by SQLite, files, Redis, or S3.
+
+### Event Types
+
+```python
+class TraceEventKind(StrEnum):
+ PIPE_START = "pipe_start"
+ PIPE_END_SUCCESS = "pipe_end_success"
+ PIPE_END_ERROR = "pipe_end_error"
+ USAGE_REPORT = "usage_report"
+
+class TraceEvent(BaseModel):
+ """Base event emitted by workers to the event log."""
+ pipeline_run_id: str
+ workflow_id: str # Temporal workflow ID or "direct" for in-process
+ event_kind: TraceEventKind
+ timestamp: datetime # UTC, for display only — not used for ordering
+ sequence: int # Per-writer monotonic counter for ordering
+
+class PipeStartEvent(TraceEvent):
+ event_kind: Literal[TraceEventKind.PIPE_START]
+ node_id: str # Unique: {graph_id}:{workflow_short_id}:node_{seq}
+ parent_node_id: str | None
+ pipe_code: str
+ pipe_type: str
+ node_kind: NodeKind
+ input_specs: list[IOSpec] | None = None
+
+class PipeEndSuccessEvent(TraceEvent):
+ event_kind: Literal[TraceEventKind.PIPE_END_SUCCESS]
+ node_id: str
+ output_spec: IOSpec | None = None
+ timing: TimingSpec | None = None
+
+class PipeEndErrorEvent(TraceEvent):
+ event_kind: Literal[TraceEventKind.PIPE_END_ERROR]
+ node_id: str
+ error: ErrorSpec
+ timing: TimingSpec | None = None
+
+class UsageReportEvent(TraceEvent):
+ event_kind: Literal[TraceEventKind.USAGE_REPORT]
+ node_id: str | None = None # Which pipe generated this cost
+ tokens_usage: TokensUsage # LLMTokensUsage | ImgGenTokensUsage | etc.
+```
+
+### Event Log Protocol
+
+```python
+class EventLogProtocol(Protocol):
+ """Append-only event log for pipeline execution tracing."""
+
+ def emit(self, event: TraceEvent) -> None:
+ """Write an event to the log. Must be durable before returning.
+ Non-blocking for the caller in the common case (local I/O is fast,
+ cloud backends buffer and flush).
+ """
+ ...
+
+ def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:
+ """Read all events for a pipeline run. Used by assemblers."""
+ ...
+
+ def subscribe(self, pipeline_run_id: str) -> AsyncIterator[TraceEvent]:
+ """Stream events in real-time. Used by dashboards/progress trackers.
+ Falls back to polling where native push isn't available.
+ """
+ ...
+
+ def cleanup(self, pipeline_run_id: str) -> None:
+ """Remove events for a completed pipeline run (optional, for storage hygiene)."""
+ ...
+```
+
+### Backend Implementations
+
+#### Local Dev: SQLite Event Log
+
+SQLite is the ideal local backend because:
+- Single file, zero setup, already available everywhere
+- WAL mode supports concurrent readers + single writer (our workers write sequentially per pipeline)
+- ACID guarantees — events are durable on commit
+- Supports polling-based subscribe (query `WHERE sequence > last_seen`)
+- Fast enough for local dev (microsecond writes)
+- Easy to inspect with any SQLite tool
+
+```sql
+CREATE TABLE trace_events (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ pipeline_run_id TEXT NOT NULL,
+ workflow_id TEXT NOT NULL,
+ event_kind TEXT NOT NULL,
+ timestamp TEXT NOT NULL,
+ sequence INTEGER NOT NULL,
+ payload TEXT NOT NULL, -- JSON serialized TraceEvent
+ UNIQUE(pipeline_run_id, workflow_id, sequence)
+);
+
+CREATE INDEX idx_pipeline_run ON trace_events(pipeline_run_id);
+CREATE INDEX idx_pipeline_run_id ON trace_events(pipeline_run_id, id);
+```
+
+**Concurrency model**: Multiple workers (separate processes) can write concurrently because SQLite WAL mode handles locking. Each worker gets its own connection. Writes are serialized at the SQLite level but with minimal contention (append-only, no conflicts).
+
+**Real-time polling**: A consumer reads `SELECT * FROM trace_events WHERE pipeline_run_id = ? AND id > ? ORDER BY id` periodically (e.g., every 500ms). Simple, reliable, no complex pub/sub.
+
+**File location**: `{pipelex_data_dir}/traces/{pipeline_run_id}.db` — one DB file per pipeline run, easy cleanup.
+
+Alternative: single DB file with all runs, cleanup via DELETE. Simpler file management but slightly more contention.
+
+#### Cloud: Redis Streams
+
+Redis Streams is the natural cloud equivalent:
+- `XADD pipelex:trace:{pipeline_run_id} * event_kind pipe_start payload {...}`
+- `XREAD` for real-time streaming
+- `XRANGE` for batch reads
+- TTL for automatic cleanup
+- Already used in many Temporal deployments
+
+#### Cloud: S3/GCS (cold path)
+
+For archival and large data (full I/O content):
+- Workers write event batches to `s3://traces/{pipeline_run_id}/{workflow_id}/{sequence}.json`
+- No real-time streaming, but durable and cheap
+- Can be combined with Redis (hot events to Redis, data payloads to S3)
+
+### Comparison: SQLite vs JSONL Files vs Redis
+
+| Criterion | SQLite | JSONL files | Redis Streams |
+|-----------|:------:|:-----------:|:-------------:|
+| Concurrent writes | ✅ WAL mode | ⚠️ File locking needed | ✅ Native |
+| Real-time reads | ✅ Poll query | ⚠️ Tail + parse | ✅ XREAD blocking |
+| Atomic writes | ✅ Transaction | ❌ Partial writes possible | ✅ XADD atomic |
+| Query by run_id | ✅ Index | ❌ Scan all files | ✅ Stream key |
+| Inspect/debug | ✅ sqlite3 CLI | ✅ cat/jq | ⚠️ redis-cli |
+| Setup | ✅ Zero (stdlib) | ✅ Zero | ❌ External service |
+| Cloud-ready | ❌ Local only | ❌ Local only | ✅ Managed Redis |
+| Survives worker crash | ✅ On commit | ✅ On flush | ✅ On ACK |
+
+**Recommendation**: SQLite for local dev, Redis Streams for cloud. Both implement the same `EventLogProtocol`. JSONL files (like LocalObserver) are tempting but lack atomic concurrent writes — a real risk with parallel pipe execution.
+
+---
+
+## Integration Points
+
+### Where Events Are Emitted
+
+The event log replaces (or wraps) the existing tracer calls. The emission points already exist in the code:
+
+| Current call site | File | Becomes |
+|-------------------|------|---------|
+| `tracer_manager.on_pipe_start(...)` | `pipe_abstract.py:416` | `event_log.emit(PipeStartEvent(...))` |
+| `tracer_manager.on_pipe_end_success(...)` | `pipe_abstract.py:487` | `event_log.emit(PipeEndSuccessEvent(...))` |
+| `tracer_manager.on_pipe_end_error(...)` | `pipe_abstract.py:460` | `event_log.emit(PipeEndErrorEvent(...))` |
+| `reporting_delegate.report_inference_job(...)` | `llm_worker_abstract.py:334` | `event_log.emit(UsageReportEvent(...))` |
+
+### How Workers Access the Event Log
+
+Two options:
+
+**(a) Via GraphContext on JobMetadata**: Add an `event_log_uri` field to GraphContext. Workers create a client from the URI. This is explicit and serializable.
+
+```python
+class GraphContext(BaseModel):
+ graph_id: str
+ parent_node_id: str | None
+ node_sequence: int
+ data_inclusion: DataInclusionConfig
+ event_log_uri: str | None = None # "sqlite:///tmp/traces/run123.db" or "redis://..."
+```
+
+**(b) Via hub configuration**: Workers read the event log backend from `pipelex.toml` config. The pipeline_run_id is on JobMetadata. This is simpler but less explicit.
+
+**Recommendation**: (b) for simplicity — the backend is infrastructure config, not per-run config. The `pipeline_run_id` (already on JobMetadata) is the key.
+
+### How the Assembler Works
+
+After pipeline execution (or on-demand for real-time), the assembler reads all events and reconstructs:
+
+```python
+class GraphSpecAssembler:
+ """Reconstructs a GraphSpec from trace events."""
+
+ @staticmethod
+ def assemble(events: list[TraceEvent]) -> GraphSpec:
+ nodes: dict[str, NodeSpec] = {}
+ edges: list[EdgeSpec] = []
+ stuff_producer_map: dict[str, str] = {} # digest → node_id
+
+ for event in events:
+ match event.event_kind:
+ case TraceEventKind.PIPE_START:
+ # Create node, record inputs, create CONTAINS edge if parent
+ ...
+ case TraceEventKind.PIPE_END_SUCCESS:
+ # Update node status, record output, register in producer map
+ ...
+ case TraceEventKind.PIPE_END_ERROR:
+ # Update node status, record error
+ ...
+ case TraceEventKind.USAGE_REPORT:
+ # Accumulate in usage registry (separate from graph)
+ ...
+
+ # Generate DATA edges from producer/consumer digest correlation
+ _generate_data_edges(nodes, stuff_producer_map, edges)
+
+ return GraphSpec(
+ graph_id=events[0].pipeline_run_id,
+ created_at=min(e.timestamp for e in events),
+ nodes=list(nodes.values()),
+ edges=edges,
+ )
+```
+
+**Key insight**: The assembler runs the same edge correlation logic that `GraphTracer.teardown()` currently runs. The difference is that it works on events from multiple workers, not in-memory state from one process. The algorithm is identical — correlate input digests with the producer map.
+
+### What Changes in Direct Mode?
+
+Nothing, if we do it right. Two strategies:
+
+**(a) Dual-write**: In direct mode, the existing in-memory GraphTracerManager continues to work as-is. The event log is only used in Temporal mode. Simple but maintains two code paths.
+
+**(b) Event log everywhere**: Replace the in-memory tracer with the event log even in direct mode. Use an in-memory event log backend (list of events). Assembler runs at the end. One code path for both modes.
+
+**Recommendation**: **(b)** — one code path is always better. The in-memory backend has zero overhead compared to the current approach. This also means every integration test exercises the event log + assembler path.
+
+### Controller-Specific Edges
+
+PipeParallel, PipeBatch, and PipeCondition register special edges (PARALLEL_COMBINE, BATCH_ITEM, BATCH_AGGREGATE, SELECTED_OUTCOME). These require dedicated event types:
+
+```python
+class BatchItemEvent(TraceEvent):
+ """Emitted when PipeBatch extracts an item from a list."""
+ event_kind: Literal["batch_item"]
+ source_node_id: str # The node that produced the list
+ target_node_id: str # The batch iteration node
+ source_stuff_digest: str
+ target_stuff_digest: str
+
+class ParallelCombineEvent(TraceEvent):
+ """Emitted when PipeParallel combines branch outputs."""
+ event_kind: Literal["parallel_combine"]
+ branch_stuff_digests: list[str]
+ combined_stuff_digest: str
+ controller_node_id: str
+
+class SelectedOutcomeEvent(TraceEvent):
+ """Emitted when PipeCondition selects an outcome."""
+ event_kind: Literal["selected_outcome"]
+ source_node_id: str
+ target_node_id: str
+```
+
+These are currently emitted via direct calls to `GraphTracerManager` methods like `register_parallel_combine()`, `register_batch_item_extraction()`, etc. They become event emissions.
+
+### Node ID Scheme
+
+Current: `{graph_id}:node_{sequence}` — collides across workers.
+
+**New scheme**: `{graph_id}:{workflow_short_id}:node_{sequence}`
+
+Where `workflow_short_id` is a short hash of the Temporal workflow ID (or `"direct"` for in-process). This guarantees uniqueness while remaining human-readable.
+
+The `node_sequence` counter lives on the `GraphContext` that flows through execution. When a pipe starts, it gets the next sequence number from its context. Since contexts are copied for children (not shared), there's no cross-worker contention.
+
+**Important**: In the current code, `node_sequence` is on the `GraphTracer` (in-memory), not on `GraphContext`. For the event log approach, we move the sequence counter to `GraphContext` so it flows with the execution. This is a behavioral change but aligns with the serializable context philosophy.
+
+---
+
+## Data Flow: Complete Picture
+
+```
+API Process (Submitter) Worker Process (any of N)
+----------------------- -------------------------
+1. pipeline_run_setup()
+ - Create event log (SQLite file or Redis stream)
+ - Create GraphContext with event_log config
+ - Attach GraphContext to JobMetadata
+
+2. Submit PipeJob to Temporal
+ ──────────────────────────────────────►
+ 3. WfPipeRouter.run(pipe_job)
+ pipe.run_pipe()
+ ├── EMIT PipeStartEvent ──────► Event Log
+ ├── (execute pipe logic)
+ │ ├── llm_worker reports:
+ │ │ EMIT UsageReportEvent ► Event Log
+ │ ├── child workflow dispatched:
+ │ │ (child worker does same)
+ │ └── ...
+ ├── EMIT PipeEndSuccessEvent ─► Event Log
+ │ (or PipeEndErrorEvent)
+ └── return PipeOutput
+
+ ◄──────────────────────────────────────
+4. Receive PipeOutput (contains working_memory only — no graph/usage)
+
+5. Assemble: read all events from event log
+ → GraphSpecAssembler.assemble() → GraphSpec
+ → UsageAggregator.aggregate() → CostReport
+ → Generate graph outputs (Mermaid, ReactFlow)
+
+6. Cleanup event log (optional)
+```
+
+**On failure**: Steps 3 may fail partway. But every event emitted before the failure is already in the event log. The assembler in step 5 sees partial data (some PIPE_START without matching PIPE_END) and marks those nodes as RUNNING or CANCELED. Usage events for completed inference jobs are captured regardless.
+
+**Real-time**: A separate consumer can poll the event log during step 3, showing live progress. For local dev, this could be a CLI progress bar or a simple web dashboard reading the SQLite file.
+
+---
+
+## Two Distinct Problems, Unified Solution
+
+Graph tracing and reporting have different characteristics but share the same event log:
+
+| Dimension | Graph Tracing | Usage Reporting |
+|-----------|--------------|-----------------|
+| Events | PipeStart, PipeEnd*, controller edges | UsageReport |
+| Assembly | Complex — tree reconstruction, edge correlation | Simple — list concatenation, sum |
+| Data volume | Potentially large (full I/O data) | Small (few KB per job) |
+| Failure handling | Partial graph with RUNNING/CANCELED nodes | All completed jobs captured |
+| Real-time use | Progress tree visualization | Running cost counter |
+| Consumer | GraphSpecAssembler → renderers | UsageAggregator → CostReport |
+
+Both read from the same event log. The assembler and aggregator are separate consumers — they can run independently, at different times, or concurrently.
+
+---
+
+## Large Data Strategy
+
+With `--graph-full-data`, IOSpec includes full serialized content (JSON, text, HTML). A single PipeEndSuccessEvent could be megabytes.
+
+### ❓ Decision 1: How to handle large IOSpec data in events?
+
+**(a) Inline in events**: Full data embedded in the event payload. Simple but makes events large.
+- SQLite handles large BLOBs fine (tested to GB scale)
+- Redis has a 512MB value limit (more than enough)
+- Makes the event log self-contained — no external references
+
+**(b) Reference + external storage**: Events contain previews and digests only. Full data written to StorageProvider (existing system) keyed by digest. Assembler fetches when needed.
+- Keeps events small and fast
+- Leverages existing StorageProvider (Local, S3, GCP)
+- Adds complexity: two storage systems to coordinate
+
+**(c) Tiered**: Events contain data up to a size threshold (e.g., 100KB). Larger data goes to external storage with a reference in the event.
+- Best of both worlds
+- More complex to implement
+
+**Recommendation**: Start with **(a)** — inline. SQLite handles it well for local dev. For cloud, (b) becomes important, but that's a later optimization. The event schema supports both — `IOSpec.data` can be `None` with a `digest` that references external storage.
+
+---
+
+## Decision Gates
+
+### ❓ Decision 2: Event Log Backend for Local Dev
+
+**(a) SQLite** — Structured, concurrent-safe, queryable, inspectable. Slightly more setup (schema migration).
+
+**(b) JSONL files** — Follows existing LocalObserver pattern. Simpler but lacks atomic concurrent writes and efficient querying.
+
+**(c) In-memory only** — No persistence. Lost on process crash. Not useful for real-time cross-process reads.
+
+**Recommendation**: (a) SQLite. The concurrency guarantees matter for parallel pipe execution. The inspectability (sqlite3 CLI) is valuable for debugging.
+
+### ❓ Decision 3: Direct Mode Integration
+
+**(a) Keep existing in-memory tracer** for direct mode, event log only for Temporal. Two code paths.
+
+**(b) Event log everywhere** — in-memory backend for direct mode, SQLite for Temporal. One code path.
+
+**Recommendation**: (b) — one code path. Use an in-memory list as the "event log" in direct mode. Same assembler, same event types, zero overhead. Every test exercises the full path.
+
+### ❓ Decision 4: Node ID Scheme
+
+Current: `{graph_id}:node_{sequence}` (conflicts across workers).
+
+**(a)** `{graph_id}:{workflow_short_id}:node_{seq}` — deterministic, debuggable, workflow-scoped prefix
+**(b)** UUID-based — simple, no coordination needed
+**(c)** Keep sequential with moved counter on GraphContext
+
+**Recommendation**: (a). The workflow ID provides natural namespacing. Short hash (8 chars) keeps IDs readable.
+
+### ❓ Decision 5: Sequence Counter Location
+
+Currently `node_sequence` lives on GraphTracer (in-memory). For event log, it must travel with execution.
+
+**(a) Move to GraphContext** — already serializable, flows through PipeJob. Each child context gets a copy.
+
+**(b) Worker-local counter** — each worker starts at 0. Combined with workflow_id prefix, globally unique.
+
+**Recommendation**: (b) — simpler. The workflow_id prefix already guarantees uniqueness. No need to coordinate sequence numbers across workers.
+
+### ❓ Decision 6: How Do Workers Access the Event Log?
+
+**(a) Via hub config** — workers read the backend config from `pipelex.toml`. The `pipeline_run_id` from JobMetadata is the key. Workers create a client on first use.
+
+**(b) Via GraphContext field** — add `event_log_uri: str | None` to GraphContext. More explicit but adds a field to a serialized model.
+
+**(c) Via PipeJob field** — add `event_log_config` to PipeJob, like `library_crate`.
+
+**Recommendation**: (a) — config-driven, like storage providers. The event log backend is infrastructure, not per-run data. Workers need the `pipeline_run_id` (already available) and the backend config (from pipelex.toml).
+
+---
+
+## Pitfalls & Warnings
+
+### From a Senior Python Engineer
+
+1. **Don't force async on emit()**: The emit call happens inside `pipe_abstract.run_pipe()`, which is already async. But making emit truly async (with await) means it becomes a Temporal replay point in workflows. For Temporal determinism, side effects must happen inside activities. **Solution**: Make emit synchronous (SQLite writes are microseconds). For cloud backends that require network I/O, buffer events in-memory and flush via a Temporal activity or background thread.
+
+2. **Temporal replay and side effects**: If `run_pipe()` replays on a worker restart, events will be re-emitted. The event log must handle duplicates gracefully. **Solution**: Use `(pipeline_run_id, workflow_id, sequence)` as a unique key. SQLite `INSERT OR IGNORE` / Redis conditional adds handle deduplication naturally.
+
+3. **GraphTracer refactoring scope**: The current GraphTracer does two things: (1) accumulate events, (2) generate edges during teardown. With the event log, accumulation moves to the log. Edge generation moves to the assembler. The GraphTracer becomes a thin emit wrapper — or disappears entirely, replaced by direct event emission in `pipe_abstract.py`. Consider whether to keep the GraphTracer as a facade or remove it.
+
+4. **ContextVar for event log client**: Similar to `_library_id`, the event log client reference should be ContextVar-scoped so each workflow/context gets the right client. For direct mode, set at `pipeline_run_setup()`. For Temporal, set at the top of `WfPipeRouter.run()`.
+
+5. **Testing strategy**: The in-memory backend (Decision 3b) makes unit testing trivial — emit events, call assembler, assert on GraphSpec. Integration tests with SQLite validate concurrency. The cloud backend is tested separately.
+
+### From a Senior Systems Engineer
+
+1. **SQLite WAL + multiple processes**: SQLite WAL mode supports one writer at a time. With many workers writing concurrently, there will be brief lock contention. For local dev with a handful of workers, this is negligible. For production with dozens of workers, SQLite won't scale — that's when Redis takes over. This is fine for the layered approach.
+
+2. **File locking on NFS/shared filesystems**: If workers run on different machines sharing an NFS mount, SQLite will NOT work correctly — NFS doesn't support the locking primitives SQLite needs. This is a known limitation. For cloud, don't use SQLite — use Redis or S3.
+
+3. **Event log cleanup**: Without cleanup, SQLite files accumulate. Add a configurable retention policy (e.g., delete after 24h or after report generation). For cloud backends, use TTL.
+
+4. **Temporal's own event history**: Temporal already stores a complete event history for every workflow. In theory, you could extract tracing data from Temporal's event history (workflow started, activity completed, child workflow results, etc.). In practice, this gives you workflow-level granularity, not pipe-level granularity, and requires querying Temporal's API. It's not a substitute for the event log, but it's useful for correlation and debugging.
+
+5. **Clock skew**: Events from different workers may have slightly different timestamps. Use the event log's insertion order (autoincrement ID in SQLite, stream ID in Redis) for ordering, not timestamps. Timestamps are for display only.
+
+6. **Backpressure**: If a pipeline generates thousands of events (e.g., batch of 1000 items), the event log must handle the throughput. SQLite WAL mode can handle ~10K writes/sec. Redis Streams can handle ~100K ops/sec. Both are well above expected throughput.
+
+---
+
+## Implementation Plan
+
+### Step 1: Event Types & Protocol
+
+Define the event models and `EventLogProtocol`. Pure model work, no integration yet.
+
+**Key files**:
+| File | Change |
+|------|--------|
+| `pipelex/tracing/trace_events.py` | **New** — TraceEvent models |
+| `pipelex/tracing/event_log_protocol.py` | **New** — EventLogProtocol |
+| `pipelex/tracing/in_memory_event_log.py` | **New** — In-memory implementation |
+
+**Done when**:
+- Event models serialize/deserialize correctly (unit tests)
+- In-memory event log passes basic emit/read/subscribe tests
+
+### Step 2: GraphSpec Assembler
+
+Build the assembler that reconstructs GraphSpec from events. This is the core algorithm.
+
+**Key files**:
+| File | Change |
+|------|--------|
+| `pipelex/tracing/graphspec_assembler.py` | **New** — Event → GraphSpec reconstruction |
+| `pipelex/tracing/usage_aggregator.py` | **New** — Event → usage report aggregation |
+
+**Done when**:
+- Assembler produces correct GraphSpec from hand-crafted events (unit tests)
+- Assembler handles partial events (missing PIPE_END → node marked RUNNING)
+- Usage aggregator sums correctly across workers
+- Edge correlation (DATA edges from digest matching) works across workers
+
+### Step 3: SQLite Event Log
+
+Implement the SQLite backend for local dev.
+
+**Key files**:
+| File | Change |
+|------|--------|
+| `pipelex/tracing/sqlite_event_log.py` | **New** — SQLite implementation |
+| `pipelex/tracing/tracing_config.py` | **New** — Config model for tracing backend |
+| `pipelex/pipelex.toml` | Add `[pipelex.tracing]` config section |
+
+**Done when**:
+- Concurrent writes from multiple threads/processes work (integration test)
+- Real-time polling reads see events from other writers
+- Deduplication handles replayed events
+
+### Step 4: Wire into Pipe Execution
+
+Replace GraphTracerManager calls in `pipe_abstract.py` with event log emissions. Replace ReportingManager calls in inference workers.
+
+**Key files**:
+| File | Change |
+|------|--------|
+| `pipelex/core/pipes/pipe_abstract.py` | Replace tracer calls with event emission |
+| `pipelex/cogt/inference/inference_worker_abstract.py` | Emit UsageReportEvent |
+| `pipelex/pipe_controllers/parallel/pipe_parallel.py` | Emit ParallelCombineEvent |
+| `pipelex/pipe_controllers/batch/pipe_batch.py` | Emit BatchItemEvent, BatchAggregateEvent |
+| `pipelex/pipe_controllers/condition/pipe_condition.py` | Emit SelectedOutcomeEvent |
+| `pipelex/pipeline/runner.py` | Open event log, run assembler after execution |
+| `pipelex/pipeline/pipeline_run_setup.py` | Initialize event log, set on ContextVar |
+| `pipelex/temporal/tprl_pipe/wf_pipe_router.py` | Initialize event log client from config |
+
+**Done when**:
+- Direct mode: `pipelex run pipe --graph` produces identical GraphSpec as before
+- Temporal mode: `pipelex run pipe --graph` produces correct GraphSpec across workers
+- Usage reporting works in both modes
+- All existing graph-related tests pass
+
+### Step 5: Real-Time Consumer
+
+Build a consumer that polls the event log and shows live progress.
+
+**Key files**:
+| File | Change |
+|------|--------|
+| `pipelex/tracing/live_progress.py` | **New** — Real-time progress from event log |
+
+**Done when**:
+- CLI shows live pipe execution progress during Temporal runs
+- Cost accumulation displayed in real-time
+
+### Future: Cloud Backends
+
+| Backend | When | Why |
+|---------|------|-----|
+| Redis Streams | When deploying to cloud with remote workers | Concurrent writes, real-time streaming, TTL |
+| S3/GCS | When archiving traces for compliance or analytics | Cheap, durable, queryable (Athena/BigQuery) |
+
+These are new implementations of `EventLogProtocol`. No changes to emission or assembly code.
+
+---
+
+## Relationship to Master Plan
+
+This is a **parallel track**, not a new phase:
+
+```
+Master Plan This Track
+----------- ----------
+Phase 4 (ClassRegistry) ← Complete
+ Step 1: Event Types & Protocol
+ Step 2: GraphSpec Assembler
+Phase 5 (StoragePayloadCodec) Step 3: SQLite Event Log
+ Step 4: Wire into Pipe Execution
+ Step 5: Real-Time Consumer
+```
+
+**No dependency on Phase 5**: Unlike the PipeOutput piggyback approach, the event log doesn't add to Temporal payloads. Large IOSpec data goes to the event log (SQLite/Redis), not through Temporal's wire.
+
+**No dependency on other phases**: The event log is orthogonal to library crate propagation, deferred hydration, and ClassRegistry scoping.
+
+---
+
+## Open Questions
+
+1. **GraphTracer's fate**: The event log approach may make `GraphTracerManager` and `GraphTracer` obsolete. Should we keep them as a compatibility layer (wrapping event emission), or remove them and emit directly from `pipe_abstract.py`? Keeping them as a facade reduces the diff but adds indirection.
+
+2. **Controller-specific edge events**: PipeParallel's `register_parallel_combine()` currently mutates the tracer's internal `_stuff_producer_map` to override which node is the "producer" of a data item. With the event log, this becomes a `ParallelCombineEvent` that the assembler interprets. Need to verify this works for all controller types.
+
+3. **Content generator activities**: `ContentGeneratorChild` wraps LLM/extract/imggen activities. These run as Temporal activities, not workflows. Do activities have access to the event log? If the hub config is available on the worker process (it should be — workers load pipelex config at startup), then activities can emit events directly. Need to verify.
+
+4. **IOSpec data lifecycle**: If we inline large data in events (Decision 1a), the SQLite file for a single run could be very large. Should we add a configurable size threshold where data is stripped from events and written to the existing StorageProvider? This is Decision 1c but deferred.
+
+5. **Backwards compatibility**: Existing graph output consumers (Mermaid renderer, ReactFlow generator) expect a GraphSpec. The assembler produces a GraphSpec. No change needed for consumers. But the `GraphSpec.to_json()` output should be identical for regression testing — verify field ordering, alias handling, etc.
diff --git a/wip/error-handling-phase-4-markdown-cli.md b/wip/error-handling-phase-4-markdown-cli.md
new file mode 100644
index 000000000..f6f5e78cf
--- /dev/null
+++ b/wip/error-handling-phase-4-markdown-cli.md
@@ -0,0 +1,84 @@
+# Worker Error Handling — Phase 4: Markdown-Default Agent CLI Output
+
+> Reference: `wip/worker-error-handling-review.md` for the full review of current state.
+> Completed phases (0–3) archived in `wip/error-handling-phases-0-3-completed.md`.
+
+---
+
+## Definition of DONE
+
+A phase is done when **all** of the following are true:
+
+1. **All workers catch SDK-specific exceptions** and wrap them in domain exceptions with `from exc`, model descriptor in message, and error category assigned
+2. **`make agent-check` passes** (pyright, mypy, ruff)
+3. **`make agent-test` passes** (full test suite green)
+4. **New unit tests exist** for each changed error path — tests verify:
+ - The correct custom exception type is raised
+ - The error category is set correctly
+ - The error message includes model descriptor
+ - The `from exc` chain is preserved
+ - The `to_error_report()` output matches the expected JSON schema
+5. **CLI `--format json` error output** is tested with snapshot tests for representative error types
+6. **Temporal compatibility verified**: `TemporalError.from_message_exception()` correctly extracts error category and maps to `non_retryable` based on category, tested with unit tests
+7. **Agent CLI** `agent_error()` updated to use structured fields from exceptions rather than lookup dicts, tested
+
+---
+
+## Phase 4: Markdown-Default Agent CLI Output
+
+> Make all agent CLI commands return markdown by default, with `--format json` for structured output.
+> Currently `models`, `doctor`, `check-model` already support this. This phase extends it to
+> `run`, `validate`, and error output. Commands where JSON IS the payload (`inputs`, `concept`,
+> `pipe`) are excluded -- their output format is inherent to the command's purpose.
+>
+> **Scope:**
+> - `run` (pipe, bundle, method): success output as markdown, `--format json` for structured
+> - `validate` (pipe, bundle, method): success output as markdown, `--format json` for structured
+> - `agent_error()`: markdown by default to stderr, `--format json` for structured
+> - `init`: success output as markdown (simple confirmation)
+> - **Excluded:** `inputs` (returns JSON template), `concept`/`pipe` (returns TOML), `fmt`/`lint` (passthrough)
+
+- [ ] **4.1** Add `agent_error_markdown()` function to `agent_output.py`
+ - Markdown rendering of errors, parallel to `agent_error()` which remains the JSON path
+ - Format: heading with error type, message body, hint as a tip callout, error_source as code block
+ - Must still print to stderr and `raise typer.Exit(1) from cause`
+
+- [ ] **4.2** Add format-aware error dispatch
+ - Introduce a way for commands to pass the current `CliOutputFormat` to the error path
+ - Options: thread-local / context var, or pass format explicitly to a new `agent_error_dispatch(format, ...)` wrapper
+ - `agent_error()` (JSON) and `agent_error_markdown()` are the two backends
+ - Keep `agent_error()` as the default when format is unknown (e.g., errors during init before format is parsed)
+
+- [ ] **4.3** Add `--format` option to `run` commands
+ - Add `output_format: CliOutputFormat = CliOutputFormat.MARKDOWN` option to `pipe_cmd.py`, `bundle_cmd.py`, `method_cmd.py`
+ - Follow existing pattern from `models_cmd.py`: `match/case` on format
+ - JSON path: existing `agent_success(result)` unchanged
+ - Markdown path: new `_format_run_markdown(result)` function
+ - Run markdown should render: main_stuff content (markdown representation if available, else formatted JSON), output file path, graph file path
+
+- [ ] **4.4** Add `--format` option to `validate` commands
+ - Same pattern as 4.3 for `validate/pipe_cmd.py`, `bundle_cmd.py`, `method_cmd.py`
+ - Markdown path: new `_format_validate_markdown(result)` function
+ - Validate markdown should render: pass/fail summary, list of validated pipes with status, error details if any
+
+- [ ] **4.5** Add `--format` option to `init` command
+ - Same pattern for `init_cmd.py`
+ - Markdown path: simple confirmation with target dir, backends enabled, routing profile
+
+- [ ] **4.6** Wire format into error handlers in `agent_cli_factory.py`
+ - `make_pipelex_for_agent_cli()` catches init errors before format is known -- keep JSON for these
+ - Command-level error handlers should respect the format option
+
+- [ ] **4.7** Update `agent_cli/CLAUDE.md` to document the new output contract
+ - Default format is markdown for all commands except inputs/concept/pipe/fmt/lint
+ - `--format json` available on run, validate, init, models, doctor, check-model
+ - Errors respect the same format option
+ - Document the markdown structure for each command
+
+- [ ] **4.8** Tests for Phase 4
+ - Test that `run` with no `--format` produces markdown to stdout
+ - Test that `run --format json` produces valid JSON to stdout
+ - Test that `validate` with no `--format` produces markdown
+ - Test that errors produce markdown to stderr by default
+ - Test that errors with `--format json` produce JSON to stderr
+ - Test that `inputs` command is unaffected (always JSON)
diff --git a/wip/error-handling-phase-5-retry-architecture.md b/wip/error-handling-phase-5-retry-architecture.md
new file mode 100644
index 000000000..338e09b51
--- /dev/null
+++ b/wip/error-handling-phase-5-retry-architecture.md
@@ -0,0 +1,107 @@
+# Worker Error Handling — Phase 5: Retry Architecture
+
+> Reference: `wip/worker-error-handling-review.md` for the full review of current state.
+> Completed phases (0–3) archived in `wip/error-handling-phases-0-3-completed.md`.
+
+---
+
+## Definition of DONE
+
+A phase is done when **all** of the following are true:
+
+1. **All workers catch SDK-specific exceptions** and wrap them in domain exceptions with `from exc`, model descriptor in message, and error category assigned
+2. **`make agent-check` passes** (pyright, mypy, ruff)
+3. **`make agent-test` passes** (full test suite green)
+4. **New unit tests exist** for each changed error path — tests verify:
+ - The correct custom exception type is raised
+ - The error category is set correctly
+ - The error message includes model descriptor
+ - The `from exc` chain is preserved
+ - The `to_error_report()` output matches the expected JSON schema
+5. **CLI `--format json` error output** is tested with snapshot tests for representative error types
+6. **Temporal compatibility verified**: `TemporalError.from_message_exception()` correctly extracts error category and maps to `non_retryable` based on category, tested with unit tests
+7. **Agent CLI** `agent_error()` updated to use structured fields from exceptions rather than lookup dicts, tested
+
+---
+
+## Phase 5: Retry Architecture
+
+> Move retry responsibility from workers to PipeRouter — the dispatch layer that sits between
+> pipeline orchestration and pipe execution. Workers classify errors, PipeRouter retries.
+>
+> **Design principle — three retry layers, each with a distinct role:**
+>
+> | Layer | What | Retries on | Controlled by |
+> |-------|------|-----------|---------------|
+> | SDK transport | Connection resets, DNS, 503 | Built into OpenAI/Anthropic/Google SDKs | SDK defaults |
+> | **PipeRouter (new)** | **TRANSIENT CogtErrors after SDK retries exhausted** | **Rate limits, timeouts, brief outages** | **`pipelex.toml` config** |
+> | Temporal (future) | Longer failures, workflow-level retry | Service outages, cascading errors | Temporal retry policy |
+>
+> The PipeRouter retry is complementary to Temporal: it handles fast transients (seconds),
+> Temporal handles longer failures (minutes). Without Temporal, PipeRouter retry is the only
+> application-level retry — Pipelex must remain usable and resilient standalone.
+>
+> **Where in the code:** `PipeRouterProtocol.run()` in `pipe_run/pipe_router_protocol.py:47-67`.
+> Currently catches `PipeRunError` only. The new logic adds a retry loop around
+> `_run_pipe_job()` that catches `CogtError` with `is_retryable=True`.
+
+- [ ] **5.1** Remove tenacity from gateway workers
+ - `gateway_extract_worker.py`: remove `_make_retryer()`, `_is_retryable_portkey_error()`,
+ `_log_retry()`, tenacity imports, and the `async for attempt in self._make_retryer()` wrapper
+ - `gateway_search_worker.py`: same removal
+ - Remove `TenacityConfig` from `config_cogt.py` and the `tenacity_config` field from `Cogt`
+ - Remove corresponding entries from `pipelex.toml` config files
+ - Remove `tenacity` from project dependencies if no longer used anywhere
+ - Remove `tools/misc/tenacity_utils.py` if no longer referenced
+ - Verify errors still propagate with correct `InferenceErrorCategory` (existing tests should cover)
+
+- [ ] **5.2** Audit all workers for ad-hoc retry logic
+ - Confirm no worker does business-level retries outside of SDK internals
+ - Instructor's `max_retries` for structured generation is acceptable (it retries on schema
+ validation failure, not transport errors) — document this with a code comment
+ - Document any remaining retry behavior in a code comment at the worker level
+
+- [ ] **5.3** Add transient retry config to `PipelineExecutionConfig`
+ - Add to `PipelineExecutionConfig` in `system/configuration/configs.py`:
+ ```
+ max_transient_retries: int # 0 = disabled (default for backward compat)
+ transient_retry_base_wait: float # seconds, e.g. 2.0
+ transient_retry_max_wait: float # seconds, e.g. 30.0
+ transient_retry_backoff_multiplier: float # e.g. 2.0
+ ```
+ - Add defaults in `pipelex/pipelex.toml` (disabled: `max_transient_retries = 0`)
+ - Add commented-out overrides in `.pipelex/pipelex.toml` project config (invitation to enable)
+ - Config flows through existing path: `get_config().pipelex.pipeline_execution_config`
+ which is already passed to `PipelexRunner` and accessible from `PipeRouterProtocol.run()`
+
+- [ ] **5.4** Add transient retry loop to `PipeRouterProtocol.run()`
+ - Modify `pipe_run/pipe_router_protocol.py` `run()` method:
+ - Wrap `_run_pipe_job()` call in a retry loop
+ - Catch `CogtError` where `error_category.is_retryable` is True
+ - On retryable error: log attempt number + wait duration + error category, sleep with
+ exponential backoff, continue loop
+ - On non-retryable error (`CONFIGURATION`, `CONTENT`, `CAPACITY`): fail immediately (no retry)
+ - On max retries exhausted: raise the last error as-is (preserve the cause chain)
+ - On `PipeRunError` (existing handling): no change, still wraps as `PipeRouterError`
+ - `_before_run()` is called once (before the loop), not on each retry
+ - `_after_failing_run()` is called once (after all retries exhausted or non-retryable)
+ - The retry config comes from `PipelineExecutionConfig` — need to thread it through.
+ Options: add to `PipeJob`, or access via `get_config()` directly in the protocol.
+
+- [ ] **5.5** Thread retry config to PipeRouter
+ - Decide how `PipeRouterProtocol.run()` accesses `PipelineExecutionConfig`:
+ - Option A: Add `execution_config` to `PipeJob` (explicit, but changes the model)
+ - Option B: Access via `get_config()` in the protocol (simple, uses existing singleton)
+ - Implement the chosen approach
+
+- [ ] **5.6** Tests for Phase 5
+ - Unit test: `CogtError` with `TRANSIENT` retries up to max, then raises
+ - Unit test: `CogtError` with `CONFIGURATION` fails immediately (no retry)
+ - Unit test: `CogtError` with `CONTENT` fails immediately
+ - Unit test: `CogtError` with `CAPACITY` fails immediately
+ - Unit test: `PipeRunError` (non-CogtError) is unaffected by retry logic
+ - Unit test: `max_transient_retries = 0` disables retry (backward compat)
+ - Unit test: retry logging includes attempt number, wait duration, error category
+ - Unit test: backoff increases with each attempt
+ - Verify gateway workers raise with correct category on first failure (no silent retries)
+ - Existing worker error handling tests should still pass unchanged
diff --git a/wip/error-handling-phase-6-error-report-everywhere.md b/wip/error-handling-phase-6-error-report-everywhere.md
new file mode 100644
index 000000000..222700929
--- /dev/null
+++ b/wip/error-handling-phase-6-error-report-everywhere.md
@@ -0,0 +1,82 @@
+# Worker Error Handling — Phase 6: ErrorReport Everywhere
+
+> Reference: `wip/worker-error-handling-review.md` for the full review of current state.
+> Completed phases (0–3) archived in `wip/error-handling-phases-0-3-completed.md`.
+
+---
+
+## Definition of DONE
+
+A phase is done when **all** of the following are true:
+
+1. **All workers catch SDK-specific exceptions** and wrap them in domain exceptions with `from exc`, model descriptor in message, and error category assigned
+2. **`make agent-check` passes** (pyright, mypy, ruff)
+3. **`make agent-test` passes** (full test suite green)
+4. **New unit tests exist** for each changed error path — tests verify:
+ - The correct custom exception type is raised
+ - The error category is set correctly
+ - The error message includes model descriptor
+ - The `from exc` chain is preserved
+ - The `to_error_report()` output matches the expected JSON schema
+5. **CLI `--format json` error output** is tested with snapshot tests for representative error types
+6. **Temporal compatibility verified**: `TemporalError.from_message_exception()` correctly extracts error category and maps to `non_retryable` based on category, tested with unit tests
+7. **Agent CLI** `agent_error()` updated to use structured fields from exceptions rather than lookup dicts, tested
+
+---
+
+## Phase 6: ErrorReport Everywhere
+
+> Extend the structured error reporting from inference-only (`CogtError`) to the full exception hierarchy.
+> This eliminates the fragile string-keyed dicts in `agent_output.py` and gives every error path
+> a self-describing report.
+
+- [ ] **6.1** Set default `error_category` on uncategorized `CogtError` subclasses
+ - These subclasses currently inherit `error_category = None` and produce empty reports when raised
+ without an instance-level override:
+ - CONFIGURATION: `RoutingProfileLibraryNotFoundError`, `InferenceBackendLibraryNotFoundError`,
+ `InferenceBackendLibraryValidationError`, `ModelManagerError`, `ModelDeckNotFoundError`,
+ `ModelDeckValidationError`, `RoutingProfileLibraryError`, `InferenceModelSpecError`
+ - CONTENT: `LLMPromptSpecError`, `LLMPromptTemplateInputsError`, `LLMPromptParameterError`,
+ `PromptImageFactoryError`, `PromptImageFormatError`, `PromptDocumentFactoryError`,
+ `ImgGenPromptError`, `ImgGenParameterError`
+ - Leave as None (set per-instance by workers): `LLMCompletionError`, `ImgGenGenerationError`,
+ `ExtractJobFailureError`, `SearchJobFailureError` — these are correctly dynamic
+ - Decide case-by-case for: `ImageContentError`, `CostRegistryError`, `ReportingManagerError`,
+ `SdkTypeError`, `ExtractOutputError`, `GeneratedImageError`, `LLMAssignmentError`,
+ `InferenceBackendLibraryError`
+
+- [ ] **6.2** Add `error_domain` and `user_action` to key non-CogtError exceptions
+ - Add optional class-level `error_domain` field to `PipelexError` (values: "input", "config", "runtime")
+ - Update `PipelexError.to_error_report()` to include `error_domain` in the report
+ - Set defaults on key exceptions:
+ - `PipelineExecutionError`: domain="runtime", user_action="Check pipe_stack to identify which pipe failed"
+ - `PipeExecutionError`: domain="runtime"
+ - `ValidateBundleError`: domain="input", user_action="Check the validation_errors array for specific issues"
+ - `PipelexInterpreterError`: domain="input"
+ - `PipelexSetupError`: domain="config"
+ - `PipelexConfigError`: domain="config"
+ - Service errors (`InferenceSetupRequiredError`, `GatewayTermsNotAcceptedError`, etc.): domain="config"
+ - Files to modify: `base_exceptions.py`, `pipeline/exceptions.py`, `pipe_run/exceptions.py`,
+ `core/interpreter/exceptions.py`, `system/pipelex_service/exceptions.py`
+
+- [ ] **6.3** Migrate inference-related hints from `AGENT_ERROR_HINTS` into exception classes
+ - For each inference error type in `AGENT_ERROR_HINTS`, move the hint string to `user_action` on the class
+ - Keep non-inference hints in the dict (FileNotFoundError, JSONDecodeError, etc. — we can't add
+ attributes to built-in exceptions)
+ - Update `agent_error()` to prefer `report.user_action` over dict lookup (already partially done)
+
+- [ ] **6.4** Migrate `AGENT_ERROR_DOMAINS` into exception classes
+ - For each error type in `AGENT_ERROR_DOMAINS`, set the corresponding `error_domain` on the class
+ - Update `agent_error()` to prefer `report.error_domain` over dict lookup
+ - The dicts become fallback-only for non-PipelexError exceptions
+
+- [ ] **6.5** Add drift-detection test
+ - Unit test that discovers all `PipelexError` subclasses caught in agent CLI handlers
+ and verifies they either have class-level metadata OR an entry in the fallback dicts
+ - This prevents the "new exception added, forgot to update dicts" failure mode
+
+- [ ] **6.6** Tests for Phase 6
+ - Test `to_error_report()` on non-CogtError exceptions includes `error_domain`
+ - Test that `agent_error()` output for inference errors gets hint from class, not dict
+ - Test that `agent_error()` output for non-PipelexError still gets hint from dict fallback
+ - Test default categories on previously-uncategorized CogtError subclasses
diff --git a/wip/error-handling-phase-7-temporal-bridge.md b/wip/error-handling-phase-7-temporal-bridge.md
new file mode 100644
index 000000000..98e1dcc19
--- /dev/null
+++ b/wip/error-handling-phase-7-temporal-bridge.md
@@ -0,0 +1,38 @@
+# Worker Error Handling — Phase 7: Temporal Bridge (deferred — belongs on Temporal branch)
+
+> Reference: `wip/worker-error-handling-review.md` for the full review of current state.
+> Completed phases (0–3) archived in `wip/error-handling-phases-0-3-completed.md`.
+
+---
+
+## Definition of DONE
+
+A phase is done when **all** of the following are true:
+
+1. **All workers catch SDK-specific exceptions** and wrap them in domain exceptions with `from exc`, model descriptor in message, and error category assigned
+2. **`make agent-check` passes** (pyright, mypy, ruff)
+3. **`make agent-test` passes** (full test suite green)
+4. **New unit tests exist** for each changed error path — tests verify:
+ - The correct custom exception type is raised
+ - The error category is set correctly
+ - The error message includes model descriptor
+ - The `from exc` chain is preserved
+ - The `to_error_report()` output matches the expected JSON schema
+5. **CLI `--format json` error output** is tested with snapshot tests for representative error types
+6. **Temporal compatibility verified**: `TemporalError.from_message_exception()` correctly extracts error category and maps to `non_retryable` based on category, tested with unit tests
+7. **Agent CLI** `agent_error()` updated to use structured fields from exceptions rather than lookup dicts, tested
+
+---
+
+## Phase 7: Temporal Bridge (deferred — belongs on Temporal branch)
+
+> This phase prepares the Temporal integration to use `InferenceErrorCategory` for retry decisions
+> and `to_error_report()` for structured error details. It should be implemented on the Temporal
+> integration branch where it can be tested end-to-end.
+>
+> Prerequisites from this branch: Phases 4-6 complete, all exceptions carry structured reports.
+
+- [ ] **7.1** Update `TemporalError.from_message_exception()` to use `error_category.is_retryable`
+- [ ] **7.2** Pack `to_error_report()` dict into `ApplicationError` details
+- [ ] **7.3** Document that `RetryPolicyConfig.non_retryable_error_types` is a fallback for exceptions without category
+- [ ] **7.4** Tests for the Temporal bridge
diff --git a/wip/error-handling-phases-0-3-completed.md b/wip/error-handling-phases-0-3-completed.md
new file mode 100644
index 000000000..c2799aec5
--- /dev/null
+++ b/wip/error-handling-phases-0-3-completed.md
@@ -0,0 +1,200 @@
+# Worker Error Handling Standardization — Completed Phases (0–3)
+
+> Reference: `wip/worker-error-handling-review.md` for the full review of current state.
+>
+> This file archives the completed phases of the worker error handling standardization effort.
+> See sibling `wip/error-handling-phase-*.md` files for the remaining phases.
+
+---
+
+## Definition of DONE
+
+Every phase is done when **all** of the following are true:
+
+1. **All workers catch SDK-specific exceptions** and wrap them in domain exceptions with `from exc`, model descriptor in message, and error category assigned
+2. **`make agent-check` passes** (pyright, mypy, ruff)
+3. **`make agent-test` passes** (full test suite green)
+4. **New unit tests exist** for each changed error path — tests verify:
+ - The correct custom exception type is raised
+ - The error category is set correctly
+ - The error message includes model descriptor
+ - The `from exc` chain is preserved
+ - The `to_error_report()` output matches the expected JSON schema
+5. **CLI `--format json` error output** is tested with snapshot tests for representative error types
+6. **Temporal compatibility verified**: `TemporalError.from_message_exception()` correctly extracts error category and maps to `non_retryable` based on category, tested with unit tests
+7. **Agent CLI** `agent_error()` updated to use structured fields from exceptions rather than lookup dicts, tested
+
+---
+
+## Phase 0: Error Category Infrastructure
+
+> Foundation that all other phases depend on. No worker changes yet.
+
+- [x] **0.1** Create `InferenceErrorCategory` enum in `pipelex/cogt/exceptions.py`
+ - Values: `TRANSIENT`, `CONFIGURATION`, `CONTENT`, `CAPACITY`
+ - Import from `pipelex.types.StrEnum`
+ - Add `is_retryable` property: `True` only for `TRANSIENT`
+
+- [x] **0.2** Add `error_category` field to `CogtError`
+ - Optional field, default `None` (backward compatible)
+ - Subclasses that already know their category set it at class level (e.g. `LLMModelNotFoundError` → `CONFIGURATION`)
+
+- [x] **0.3** Add `user_action` optional field to `CogtError`
+ - Short, plain-English suggestion for the user or agent
+ - Set per-instance by workers when raising, or per-class as a default
+
+- [x] **0.4** Add `to_error_report()` method on `PipelexError`
+ - Returns a dict with: `error_type`, `message`, `error_category` (if set), `user_action` (if set), `retryable` (derived from category), `provider` (if set), `model` (if set)
+ - Only includes non-None fields
+ - This is the single source of truth for all error serialization (CLI JSON, agent output, Temporal)
+
+- [x] **0.5** Tests for Phase 0
+ - Unit tests for `InferenceErrorCategory` enum properties
+ - Unit tests for `to_error_report()` on `CogtError` and its subclasses
+ - Test that `to_error_report()` output is JSON-serializable
+ - Test backward compat: existing exceptions without category still work
+
+---
+
+## Phase 1: CLI Error Output
+
+> Main CLI (`pipelex`) always uses Rich output. Agent CLI (`pipelex-agent`) uses markdown text by default, or JSON with `--format json`. See `CliOutputFormat` in `agent_output.py`.
+
+- [x] **1.1** Refactor `agent_output.py` to use `to_error_report()`
+ - Replace the `AGENT_ERROR_HINTS`, `RETRYABLE_ERROR_TYPES`, `AGENT_ERROR_DOMAINS` lookup dicts
+ - Instead, call `exc.to_error_report()` when the exception is a `PipelexError`
+ - Keep the lookup dicts as fallback for non-PipelexError exceptions (FileNotFoundError, JSONDecodeError, etc.)
+ - Merge agent-specific fields (hint, error_source) on top of the report
+
+- [x] **1.2** Update `error_handlers.py` to use `to_error_report()` for Rich output
+ - Each `handle_*` function calls `to_error_report()` to build the error data
+ - Rich formatting uses the report fields for consistent, structured display
+ - No `--format` option on the main CLI — it is always Rich
+
+- [x] **1.3** Tests for Phase 1
+ - Snapshot tests: representative errors in agent CLI JSON mode match expected schema
+ - Test that `agent_error()` output includes fields from `to_error_report()`
+ - Test that agent CLI `--format json` produces valid JSON on stderr for known error types
+ - Test that agent CLI default (no `--format`) produces markdown text output
+ - Test that main CLI still produces Rich output (no format option)
+
+---
+
+## Phase 2: Bring OpenAI/Anthropic Workers to Full Coverage
+
+> These are already Tier 1/2, add the missing pieces.
+
+- [x] **2.1** Add `RateLimitError` handling to all OpenAI workers
+ - `openai_completions_llm_worker.py`, `openai_responses_llm_worker.py`, `openai_img_gen_worker.py`, `openai_completions_img_gen_worker.py`
+ - Catch `RateLimitError` → `LLMCompletionError` / `ImgGenGenerationError` with `error_category=TRANSIENT`
+ - `user_action`: "Rate limited by OpenAI — the system will retry automatically"
+
+- [x] **2.2** Add `APITimeoutError` handling to all OpenAI workers
+ - Same files as 3.1
+ - Catch `APITimeoutError` → domain error with `error_category=TRANSIENT`
+
+- [x] **2.3** Add `AuthenticationError` to `openai_completions_img_gen_worker.py` (currently missing)
+ - With `error_category=CONFIGURATION`
+
+- [x] **2.4** Add SDK exception handling to `openai_img_gen_worker.py`
+ - Currently has no try/except — add full pattern matching other OpenAI workers
+
+- [x] **2.5** Add `RateLimitError` and `APITimeoutError` to `anthropic_llm_worker.py`
+ - Same pattern as OpenAI
+
+- [x] **2.6** Set `error_category` on all existing exception raises in OpenAI/Anthropic workers
+ - `NotFoundError` → `CONFIGURATION`
+ - `APIConnectionError` → `TRANSIENT`
+ - `BadRequestError` → `CONTENT` (default), `CONFIGURATION` (if detectable)
+ - `AuthenticationError` → `CONFIGURATION`
+ - `InstructorRetryException` → `CONTENT`
+
+- [x] **2.7** Add content policy detection for OpenAI and Anthropic
+ - Inspect `BadRequestError` message for "content_policy", "safety", "content_filter" keywords
+ - Raise with `error_category=CONTENT`, `user_action`: "Content was rejected by safety filters — revise the prompt"
+ - Also check `finish_reason == "content_filter"` in OpenAI response validation
+
+- [x] **2.8** Add quota/credits exhaustion detection for OpenAI and Anthropic
+ - **Problem:** Rate limit (429) and out-of-credits (429) share the same HTTP status — must inspect the error body to distinguish them
+ - **OpenAI:** `RateLimitError` with `"insufficient_quota"` or `"exceeded your current quota"` in message → `CAPACITY`; `AuthenticationError` with `"insufficient_quota"` → `CAPACITY`
+ - **Anthropic:** `RateLimitError` or `PermissionDeniedError` with `"quota"` or `"billing"` in message → `CAPACITY`
+ - Raise with `error_category=CAPACITY`, `non_retryable=True`
+ - `user_action`: "Your {provider} account has exceeded its quota — check your billing dashboard at {billing_url}"
+ - Billing URLs: OpenAI → `platform.openai.com/account/billing`, Anthropic → `console.anthropic.com/settings/billing`
+ - **Must be checked before the generic `RateLimitError` → `TRANSIENT` handler** (order matters in except blocks, or use message inspection within a single handler)
+
+- [x] **2.9** Tests for Phase 2
+ - Unit tests per worker: mock SDK to raise each exception type, verify correct domain exception + category
+ - Test content policy detection on known error message patterns
+ - Test quota detection: mock `RateLimitError` with quota message → verify `CAPACITY` category
+ - Test quota detection: mock `RateLimitError` with generic rate limit message → verify `TRANSIENT` category
+ - Test that `to_error_report()` includes provider and model for each error path
+
+---
+
+## Phase 3: Bring Tier 3 Workers Up
+
+> The workers with no error handling at all.
+
+- [x] **3.1** `google_llm_worker.py` — add Google API exception handling
+ - Catch `google.genai.errors.ClientError` / `ServerError` with status code inspection
+ - Map to `LLMCompletionError` with appropriate categories
+ - 429 with quota patterns → `CAPACITY`; generic 429 → `TRANSIENT`
+ - 401/403 → `CONFIGURATION`, 404 → `CONFIGURATION`, 400 → `CONTENT`
+
+- [x] **3.2** `google_img_gen_worker.py` — same pattern as 3.1 for image generation
+ - Map to `ImgGenGenerationError` with categories
+
+- [x] **3.3** `mistral_llm_worker.py` — add Mistral SDK exception handling
+ - Catch `MistralError` with `status_code` inspection
+ - Map to `LLMCompletionError` with categories
+ - Quota detection: HTTP 402 (Payment Required) or 429 with `"quota"` in message → `CAPACITY`
+
+- [x] **3.4** `mistral_extract_worker.py` — add error handling
+ - Wrap Mistral OCR API calls
+ - Map to `ExtractJobFailureError` with categories
+ - Same quota detection pattern as 3.3
+
+- [x] **3.5** `bedrock_llm_worker.py` — add AWS error handling
+ - Catch `botocore.exceptions.ClientError` with error code inspection
+ - `ThrottlingException`: inspect message — `"quota"` or `"limit exceeded"` → `CAPACITY`; otherwise → `TRANSIENT`
+ - `AccessDeniedException` → `CONFIGURATION`
+ - `ValidationException` → `CONTENT`
+ - AWS-specific: `ServiceQuotaExceededException` → `CAPACITY`
+
+- [x] **3.6** `azure_img_gen_worker.py` — wrap httpx errors
+ - Catch `httpx.HTTPStatusError` with status code inspection
+ - 429 → `TRANSIENT`, 402 → `CAPACITY`, 401/403 → `CONFIGURATION`, 400 → `CONTENT`
+ - Catch `httpx.ConnectError` → `TRANSIENT`
+ - Catch `httpx.TimeoutException` → `TRANSIENT`
+
+- [x] **3.7** `fal_img_gen_worker.py` — add FAL error handling
+ - Catch `FalClientHTTPError`, `FalClientTimeoutError`, `MissingCredentialsError`, `FalClientError`
+ - Map to `ImgGenGenerationError` with categories
+
+- [x] **3.8** `huggingface_img_gen_worker.py` — add HuggingFace error handling
+ - Catch `InferenceTimeoutError`, `HfHubHTTPError`
+ - Map to `ImgGenGenerationError` with categories
+
+- [x] **3.9** `docling_extract_worker.py` — add conversion error handling
+ - Wrap `docling` conversion calls
+ - Map to `ExtractJobFailureError` (ValueError/RuntimeError → CONTENT, FileNotFoundError → CONFIGURATION, OSError → TRANSIENT)
+
+- [x] **3.10** `linkup_extract_worker.py` and `linkup_search_worker.py` — add Linkup error handling
+ - Catch all `linkup` exception types (AuthenticationError, InsufficientCreditError, TooManyRequestsError, etc.)
+ - Map to `ExtractJobFailureError` / `SearchJobFailureError` with categories
+
+- [x] **3.11** `pypdfium2_worker.py` — add PDF operation error handling
+ - Wrap pypdfium2 calls
+ - Map to `ExtractJobFailureError` (ValueError/RuntimeError → CONTENT, FileNotFoundError → CONFIGURATION, OSError → TRANSIENT)
+
+- [x] **3.12** Add quota/credits detection to gateway workers
+ - `gateway_extract_worker.py`, `gateway_img_gen_worker.py`, `gateway_search_worker.py`
+ - Added `_classify_portkey_error_category()` method to each gateway worker
+ - Inspect `portkey_exceptions.APIStatusError` status code and message
+ - 402 or 429 with quota patterns → `CAPACITY`; RateLimitError → `TRANSIENT`; AuthenticationError → `CONFIGURATION`; BadRequestError → `CONTENT`
+
+- [x] **3.13** Tests for Phase 3
+ - Unit tests per worker: mock SDK to raise each exception type, verify domain exception + category
+ - At least 2 error scenarios per worker (one transient, one configuration)
+ - Quota detection tests: mock 429 with quota message → verify `CAPACITY`; mock 429 without → verify `TRANSIENT`
diff --git a/wip/error-handling-review.md b/wip/error-handling-review.md
new file mode 100644
index 000000000..392246171
--- /dev/null
+++ b/wip/error-handling-review.md
@@ -0,0 +1,332 @@
+# Pipelex Error Handling Review
+
+**Original review:** 2026-04-12 (branch `refactor/Inference-error-handling`)
+**Refreshed:** 2026-05-04 (branch `feature/Temporal-merge-2`, after Phase 0–3 merged)
+**Scope:** Full codebase error handling audit, plus status of the active improvement plans
+
+---
+
+## What changed since the original review
+
+| Original Issue | Status |
+|---|---|
+| Issue 1 — `ErrorReport` is inference-only | **In progress.** `to_error_report()` is now wired into `agent_error()` (Phase 1 done). Class-level `error_domain` / `user_action` on non-CogtError exceptions is the **Phase 6** scope. |
+| Issue 2 — many CogtError subclasses have no `error_category` | **Mostly resolved.** `CONFIGURATION` defaults are set on the routing/backend/model-deck/handle-not-found/spec families. Remaining unset (intentionally — workers set per-instance) and the proposed CONTENT defaults are the **Phase 6** scope. |
+| Issue 3 — dict-based classification in `agent_output.py` is fragile | **Open.** `AGENT_ERROR_HINTS`, `AGENT_ERROR_DOMAINS`, `RETRYABLE_ERROR_TYPES` still live in `pipelex/cli/agent_cli/commands/agent_output.py`. Drift-detection test + migration onto classes is **Phase 6**. |
+| Issue 4 — `ToolError` sits outside `PipelexError` | **Resolved.** `class ToolError(PipelexError)` in `pipelex/system/exceptions.py:8`. |
+| Issue 5 — `TracebackMessageError` separate lifecycle | **Open (low priority).** Still inherits from `PipelexError` indirectly via its own subclass chain; logging mechanism unchanged. Same recommendation: leave as-is unless it gets in the way. |
+| Issue 6 — repetitive Rich error handler functions | **Open.** `cli/error_handlers.py` is still 434 lines with 11 near-identical handlers; helper extraction not done. |
+| Issue 7 — no full-chain error serialization integration test | **Open.** Worker-level classification tests are in great shape; the runner→CLI→JSON e2e snapshot test is still missing. |
+
+**Worker coverage** (the Tier 1/2/3 ranking from `worker-error-handling-review.md`): all Tier 3 workers were brought up in Phase 3 (full SDK exception handling, quota/credits detection, content-policy detection where relevant). Reading `error_classification.py` is now the canonical place to see provider-specific quota/content patterns.
+
+**Active follow-on work** (separate files):
+
+- [error-handling-phases-0-3-completed.md](error-handling-phases-0-3-completed.md) — record of what Phases 0–3 shipped.
+- [error-handling-phase-4-markdown-cli.md](error-handling-phase-4-markdown-cli.md) — markdown-default agent CLI output (run/validate/init/error).
+- [error-handling-phase-5-retry-architecture.md](error-handling-phase-5-retry-architecture.md) — move retries from gateway workers to PipeRouter.
+- [error-handling-phase-6-error-report-everywhere.md](error-handling-phase-6-error-report-everywhere.md) — class-level domain/user_action everywhere; eliminate dict drift.
+- [error-handling-phase-7-temporal-bridge.md](error-handling-phase-7-temporal-bridge.md) — wire `error_category.is_retryable` into `TemporalError`.
+- [worker-error-handling-review.md](worker-error-handling-review.md) — original tier inventory that drives the phase plans.
+- [instructor-unwrap-other-workers.md](instructor-unwrap-other-workers.md) — port the Anthropic `InstructorRetryException` unwrap fix to OpenAI Completions/Responses, Mistral, Google.
+
+---
+
+## 1. Executive Summary
+
+The Pipelex error handling system is **mature and well-structured**. Key strengths (unchanged):
+
+- Single-rooted exception hierarchy (`PipelexError`, with `ToolError` now inside it) — consistent patterns.
+- Clean separation between human CLI (Rich), agent CLI (JSON / markdown), and internal propagation.
+- The inference layer (`CogtError` + `InferenceErrorCategory` + `ErrorReport` + per-provider `error_classification.py`) sets the standard the rest of the hierarchy is migrating toward.
+- Consistent `msg = "..."; raise XError(msg) from exc` pattern.
+- No bare `except:` clauses; broad `Exception` catches only at CLI entry points.
+
+Remaining work (all tracked in active phase files):
+
+- **Class-level error metadata for non-CogtError** exceptions is still partial (Phase 6).
+- **Dict-based agent classification** still parallels `to_error_report()` (Phase 6).
+- **Retry logic** is still per-worker (gateway uses tenacity); needs to move to PipeRouter (Phase 5).
+- **Markdown-default agent CLI output** is still missing for `run` / `validate` / `init` (Phase 4).
+- **Temporal bridge** for `error_category.is_retryable` → `non_retryable` mapping (Phase 7).
+
+---
+
+## 2. Axis 1 -- Error Flow: Catch, Enrich, Bubble, Deliver
+
+### 2.1 Layer Model
+
+```
+Layer 5: CLI Entry Points (catch + format for human/agent)
+Layer 4: CLI Factories (catch setup errors, route to handlers)
+Layer 3: Pipeline Runner (catch + wrap as PipelineExecutionError)
+Layer 2: Pipe Router / Operators (catch + wrap with pipe context)
+Layer 1: Workers / SDK calls (catch third-party + classify)
+Layer 0: Third-party SDKs (raw OpenAI/Anthropic/Google/etc. exceptions)
+```
+
+### 2.2 Layer 0 -> 1: Third-Party Error Transformation
+
+**Location:** `pipelex/plugins/*/` worker files.
+
+After Phase 2 + Phase 3, **every** provider worker catches SDK-specific exceptions and transforms them into `CogtError` subclasses with:
+
+- `error_category` (TRANSIENT / CONFIGURATION / CONTENT / CAPACITY)
+- `user_action` (actionable hint with billing links where applicable)
+- Model descriptor in the message
+- `from exc` chaining preserved
+
+Classification logic is centralized in `pipelex/cogt/inference/error_classification.py` (pure functions: `is_quota_exhaustion_*`, `is_content_policy_violation`, etc.). Per-provider pattern tuples live alongside.
+
+**Assessment:** This layer is the strongest part of the system. Consistent, well-tested (parametrized classification tests + per-worker error-path tests), provider-agnostic.
+
+**Caveat — `instructor` wrapping.** Only Anthropic currently unwraps `InstructorRetryException` to recover the underlying SDK error before classification. OpenAI Completions/Responses, Mistral, and Google still mis-classify wrapped quota/timeout/auth as `CONTENT`. Tracked in [instructor-unwrap-other-workers.md](instructor-unwrap-other-workers.md).
+
+### 2.3 Layer 1 -> 2: Pipe Operators
+
+**Location:** `pipelex/pipe_operators/*/`.
+
+Pipe operators define thin wrapper exceptions (`PipeLLMFactoryError`, `PipeImgGenRunError`, etc.) and add pipe-level context (pipe_code, pipe_type, model_handle). The richest is `PipeOperatorModelAvailabilityError` at `pipe_operators/exceptions.py` — carries `run_mode`, `pipe_type`, `pipe_code`, `pipe_stack`, `model_handle`, `fallback_list`.
+
+### 2.4 Layer 2 -> 3: Pipeline Runner
+
+**Location:** `pipelex/pipeline/runner.py`.
+
+The `PipelexRunner.execute_pipeline()` method catches three exception types:
+
+| Caught | Wrapped As | Context Added |
+|--------|-----------|---------------|
+| `PipeRouterError` | `PipelineExecutionError` | run_mode, pipe_code, output_name, pipe_stack |
+| `PipelexError` (other) | `PipelineExecutionError` | run_mode, pipe_code, output_name, pipe_stack |
+| `ValidationError` (Pydantic) | `PipeExecutionError` | formatted validation message |
+
+The original exception is always chained via `from exc`. Telemetry events tracked on failure.
+
+### 2.5 Layer 3 -> 4: CLI Factories
+
+**Location:** `pipelex/cli/cli_factory.py` (human) and `pipelex/cli/agent_cli/commands/agent_cli_factory.py` (agent).
+
+These catch initialization errors from `Pipelex.make()` and route to the appropriate handler. The agent factory catches several specific exception types and sends each through `agent_error()`. The human factory delegates to dedicated `handle_*_error()` functions in `error_handlers.py`.
+
+### 2.6 Layer 4 -> 5: CLI Entry Points -- Delivery
+
+#### Human CLI (Rich Console)
+
+**Location:** `pipelex/cli/error_handlers.py` (434 lines, 11 handler functions).
+
+Each error type has a dedicated handler function with: red banner, structured fields, actionable tip, doc/Discord links, `raise typer.Exit(1) from exc`.
+
+**Assessment:** Well-crafted, but verbose and repetitive. Helper extraction is Issue 6 / open.
+
+#### Agent CLI (Structured JSON)
+
+**Location:** `pipelex/cli/agent_cli/commands/agent_output.py`.
+
+`agent_error()` now (Phase 1) prefers `cause.to_error_report()` and only falls back to the dicts when the report has no `user_action` / `retryable`. The dicts (`AGENT_ERROR_HINTS`, `AGENT_ERROR_DOMAINS`, `RETRYABLE_ERROR_TYPES`) are still the source for `error_domain` and for non-PipelexError exceptions (FileNotFoundError, JSONDecodeError, ValidationError, etc.).
+
+Output JSON shape:
+
+```json
+{
+ "error": true,
+ "error_type": "LLMCompletionError",
+ "message": "...",
+ "hint": "...",
+ "retryable": true,
+ "error_domain": "runtime",
+ "error_category": "transient",
+ "model": "gpt-4o",
+ "provider": "openai",
+ "error_source": ["LLMCompletionError @ .../worker.py:152 (in _gen_text)"]
+}
+```
+
+**Assessment:** Phase 1 cut the dual-source problem in half. Phase 6 is what eliminates the remaining dict drift by moving `error_domain` / `user_action` onto the exception classes.
+
+#### Validation Errors (Flattened)
+
+**Location:** `agent_output.py:extract_validation_errors`. Unchanged. Still the most structured error delivery in the codebase.
+
+#### Markdown Output (Special Case)
+
+`InferenceSetupRequiredError` is rendered as markdown to stdout (exit 0). Used by agent skills to display setup guidance. Phase 4 generalizes markdown delivery to `run` / `validate` / `init` and to the error path.
+
+### 2.7 Error Flow Summary
+
+| Error Origin | Catch Layer | Enrichment | Human Delivery | Agent Delivery |
+|---|---|---|---|---|
+| SDK API call | Worker | category + user_action + model | Rich formatted | JSON with category/retryable |
+| Model routing | Pipe operator | pipe_code + pipe_stack + fallbacks | Rich formatted | JSON with pipe context |
+| Pipeline execution | Runner | pipe_code + pipe_stack + output_name | Rich formatted | JSON with cause chain |
+| Bundle validation | Validator | aggregated error lists | Rich table | Flattened JSON array |
+| Configuration/setup | Pipelex.make() | specific error type | Rich formatted | JSON or markdown |
+| File/input parsing | CLI command | file path + TOML location | typer.secho red | JSON with hint |
+
+---
+
+## 3. Axis 2 -- Error Class Architecture
+
+### 3.1 Class Hierarchy Overview
+
+```
+Exception
+ PipelexError base_exceptions.py
+ PipelexUnexpectedError
+ PipelexConfigError
+ PipelexSetupError
+ SecurityError
+ ToolError system/exceptions.py (now under PipelexError)
+ NestedKeyConflictError
+ StorageError
+ Jinja2TemplateSyntaxError
+ SecretNotFoundError
+ ...
+ CogtError cogt/exceptions.py (~50 subclasses)
+ PipeExecutionError pipeline/exceptions.py
+ PipelineExecutionError
+ PipeStackOverflowError
+ ConceptError core/concepts/exceptions.py (+ children)
+ StuffError core/stuffs/exceptions.py (+ children)
+ WorkingMemoryError core/memory/exceptions.py (+ children)
+ LibraryError libraries/exceptions.py (+ children)
+ PipeControllerError pipe_controllers/exceptions.py
+ PipeRunError pipe_run/exceptions.py
+ PipeRouterError
+ PipeRunInputsError
+ PipelexServiceError system/pipelex_service/exceptions.py
+ InferenceSetupRequiredError
+ GatewayTermsNotAcceptedError
+ ...
+ PipelexInterpreterError core/interpreter/exceptions.py
+ GraphSpecError graph/exceptions.py
+ KitError kit/exceptions.py
+ TracebackMessageError system/exceptions.py
+ FatalError
+ ConfigValidationError
+ ConfigModelError (also ValueError)
+```
+
+### 3.2 Exception Module Organization
+
+**Pattern: One `exceptions.py` per package** — followed consistently across **49 files** holding **~218 custom exception classes** (codebase has grown since the original review's "~130 across ~25 files").
+
+### 3.3 Exception Class Patterns
+
+Three patterns, unchanged:
+
+- **Pattern A — Plain message-only exception** (~70%): exists for type-based catching.
+- **Pattern B — Exception with structured fields** (~25%): carries context as instance attributes (e.g., `PipelineExecutionError`, `ModelChoiceNotFoundError`, `PipeValidationError`, `InferenceBackendCredentialsError`).
+- **Pattern C — Non-exception structured error data** (BaseModel/dataclass): `ErrorReport`, `PipelexBundleBlueprintValidationErrorData`, `PipesAndConceptValidationErrorData`, `PipeFactoryErrorData`, `SyntaxErrorData`. These are aggregated into raised exceptions, not raised themselves.
+
+`ErrorReport` is now a Pydantic dataclass (`base_exceptions.py:7`) with `extra="forbid"`, fields: `error_type`, `message`, `error_category`, `retryable`, `user_action`, `model`, `provider`. The `to_dict()` helper drops `None` fields.
+
+### 3.4 Two Error Reporting Systems (still parallel)
+
+Phase 1 wired `to_error_report()` into `agent_error()`, but the dicts in `agent_output.py` still hold:
+
+| Mechanism | Where Used | Metadata Source |
+|---|---|---|
+| `to_error_report()` | All `PipelexError` (only `CogtError` subclasses currently override it with metadata) | Class-level attributes (`error_category`, `user_action`) |
+| Dict lookups | `agent_output.py` — `error_domain`, hints/retryable for non-CogtError and non-PipelexError types | String-keyed dicts |
+
+Inference errors (CogtError) now self-describe at the class level. Non-inference errors (pipeline, validation, setup, library) still depend on the dicts. Phase 6 closes the gap.
+
+---
+
+## 4. Centralized Error Codes Assessment
+
+**Recommendation unchanged: not now, but keep preparing the ground.**
+
+For Pipelex today:
+
+- The agent CLI is the primary programmatic consumer; it already has good classification via `error_domain` + `error_category` + `error_type`.
+- Class names are stable and descriptive.
+- Adding error codes to ~218 exceptions is high effort for low immediate value.
+
+Incremental steps that help and are already on the roadmap:
+
+1. **Extend `to_error_report()` metadata to all PipelexError subclasses** — Phase 6.
+2. **Move `AGENT_ERROR_HINTS` / `AGENT_ERROR_DOMAINS` into the class hierarchy** — Phase 6.
+3. **Add error codes only when shipping a public API** — at that point, auto-generate from class path (e.g., `cogt.LLMCompletionError` → `COGT-001`).
+
+---
+
+## 5. Findings and Recommendations
+
+### 5.1 Strengths (Keep Doing)
+
+| # | What | Evidence |
+|---|---|---|
+| S1 | Single-rooted hierarchy | All custom exceptions inherit from `PipelexError` (including `ToolError` since this review was first written). |
+| S2 | Consistent `from exc` chaining | Every re-raise preserves the cause chain. |
+| S3 | Message-before-raise pattern | `msg = "..."; raise XError(msg) from exc` throughout. |
+| S4 | No broad Exception catches in business logic | Only at CLI entry points. |
+| S5 | Structured inference error classification | `InferenceErrorCategory` + per-provider classifiers + `ErrorReport`. |
+| S6 | Dual CLI delivery | Rich for humans, JSON for agents — same errors, different rendering. |
+| S7 | Validation error aggregation | `ValidateBundleError` collects all validation issues before reporting. |
+| S8 | Worker error coverage | All inference workers (LLM/extract/img-gen/search) catch SDK exceptions and assign `error_category` after Phase 3. |
+
+### 5.2 Open Issues — Status & Owner
+
+#### Issue 1: `ErrorReport` is inference-only — **In progress (Phase 6)**
+
+`PipelexError.to_error_report()` returns a bare report (`error_type`, `message`) for non-CogtError exceptions. Phase 6 adds `error_domain` and class-level `user_action` to `PipeExecutionError`, `PipelineExecutionError`, `ValidateBundleError`, `PipelexInterpreterError`, `PipelexSetupError`, `PipelexConfigError`, and the `PipelexService` family.
+
+#### Issue 2: ~20 CogtError subclasses had no `error_category` — **Mostly resolved**
+
+After Phase 2/3, the routing/backend/model-deck/handle/spec families now default to `CONFIGURATION`. `LLMCompletionError`, `ImgGenGenerationError`, `ExtractJobFailureError`, `SearchJobFailureError` are intentionally left dynamic (workers set per-instance). Remaining defaults to add (CONTENT-y prompt-related errors) are Phase 6.1 in [error-handling-phase-6-error-report-everywhere.md](error-handling-phase-6-error-report-everywhere.md).
+
+#### Issue 3: Dict-based classification in `agent_output.py` is fragile — **Open (Phase 6)**
+
+The dicts still exist (`AGENT_ERROR_HINTS`, `AGENT_ERROR_DOMAINS`, `RETRYABLE_ERROR_TYPES`). Phase 6.5 adds a drift-detection unit test; Phase 6.3/6.4 migrate the entries onto the exception classes themselves.
+
+#### Issue 4: `ToolError` sits outside `PipelexError` — **Resolved**
+
+`class ToolError(PipelexError)` in `pipelex/system/exceptions.py:8`. All `except PipelexError` blocks now catch `ToolError` subclasses (storage, Jinja2, secrets) too.
+
+#### Issue 5: `TracebackMessageError` separate lifecycle — **Open (low priority)**
+
+Still uses its own logging mechanism (`error_mode: TracebackMessageErrorMode`). No real cost; works fine for fatal startup errors.
+
+#### Issue 6: Repetitive Rich error handler functions — **Open**
+
+`error_handlers.py` is still 434 lines / 11 handlers / one shape repeated. Helper extraction (`display_error_panel(...)`) would cut it roughly in half. Not on a phase plan yet — would land best as a tidy-up after Phase 6.
+
+#### Issue 7: No full-chain error serialization integration test — **Open**
+
+Inference layer has 60+ classification tests. The runner→CLI→JSON e2e snapshot is still missing. Would cover: pipeline failure with a known inference error → assert JSON has expected `error_category`, `retryable`, `model`, `provider`, `error_source`. Not on a phase plan yet — minimal effort, high signal.
+
+---
+
+### 5.3 Priority Matrix (refreshed)
+
+| # | Issue | Effort | Impact | Status / Phase |
+|---|---|---|---|---|
+| I2 | Defaults on remaining uncategorized CogtError subclasses | Small | Medium | Phase 6.1 |
+| I3-short | Drift-detection test for agent_output dicts | Small | Medium | Phase 6.5 |
+| I1 | Extend ErrorReport metadata to non-inference exceptions | Medium | High | Phase 6.2 |
+| I3-medium | Move hints/domains onto exception classes | Medium | Medium | Phase 6.3 / 6.4 |
+| I4 | Make ToolError inherit from PipelexError | — | — | **Done** |
+| Phase 4 | Markdown-default agent CLI output | Medium | Medium | [error-handling-phase-4-markdown-cli.md](error-handling-phase-4-markdown-cli.md) |
+| Phase 5 | Move retry from gateway workers to PipeRouter | Medium | High | [error-handling-phase-5-retry-architecture.md](error-handling-phase-5-retry-architecture.md) |
+| Phase 7 | Temporal bridge (`error_category.is_retryable` → `non_retryable`) | Small | Medium | [error-handling-phase-7-temporal-bridge.md](error-handling-phase-7-temporal-bridge.md) — Temporal branch |
+| Instructor unwrap | Port Anthropic fix to OpenAI/Mistral/Google | Small per worker | High | [instructor-unwrap-other-workers.md](instructor-unwrap-other-workers.md) |
+| I6 | Extract generic Rich error display helper | Medium | Low | Open — no phase plan |
+| I7 | Full-chain error serialization integration test | Small | Medium | Open — no phase plan |
+| I5 | Unify TracebackMessageError with PipelexError | Small | Low | Open — no phase plan |
+
+---
+
+### 5.4 Maturity Assessment (refreshed)
+
+| Dimension | Rating | Notes |
+|---|---|---|
+| Exception hierarchy | 9/10 | Single-rooted, well-organized; ToolError fold-in closed the last gap. |
+| Error context preservation | 9/10 | Consistent `from exc`, structured fields on key exceptions. |
+| Third-party error transformation | 9/10 | All inference workers covered after Phase 3; `error_classification.py` is the source of truth. Modest deduction for the unfixed `instructor` unwrap on OpenAI/Mistral/Google. |
+| Human-facing error delivery | 8/10 | Rich formatting with actionable tips; verbose handlers are an aesthetic, not a correctness, problem. |
+| Agent-facing error delivery | 8/10 | `to_error_report()` is now the primary source; dual-source narrows further with Phase 6. |
+| Error codes / stable identifiers | 4/10 | No codes yet; correctly deferred until a public API exists. |
+| Validation error aggregation | 9/10 | `ValidateBundleError` + flattened extraction unchanged and still excellent. |
+| Test coverage | 7/10 | Inference layer excellent; full-chain agent-output snapshot test still missing. |
+| **Overall** | **8/10** | Up from 7.5. Phase 0–3 closed real gaps; the remaining work is well-scoped in Phase 4–7. |
diff --git a/wip/future-crate-first-architecture.md b/wip/future-crate-first-architecture.md
new file mode 100644
index 000000000..9e25f9c75
--- /dev/null
+++ b/wip/future-crate-first-architecture.md
@@ -0,0 +1,242 @@
+# Crate-First Architecture Vision
+
+> **Status**: Vision / future direction
+> **Date**: 2026-03-25
+> **Related**: [archive/00-master-plan.md](archive/00-master-plan.md), [archive/phase2-implementation-plan.md](archive/phase2-implementation-plan.md)
+
+---
+
+## Summary
+
+The LibraryCrate should become the central concept in library management. Today, library loading drives everything and crates are a byproduct. The future direction inverts this: **collecting blueprints and building the crate is the primary operation; loading into a live Library is just one consumer of the crate**.
+
+This document captures the architectural direction, the Phase 2 design decision that aligns with it, and the two future capabilities it enables: crate stripping and remote dependency resolution.
+
+---
+
+## Phase 2 Design Decision: Blueprint Accumulation (Option A)
+
+### The question
+
+Phase 2 ships the `LibraryCrate` inside `PipeJob` so Temporal workers can load the full library context. Library loading happens in multiple steps (PIPELEXPATH dirs, extra library_dirs, mthds_contents), each going through `load_from_blueprints()` which internally builds a crate. The question was how to produce the final crate to ship.
+
+### Options considered
+
+**Option A — Accumulate blueprints, build one crate at the end.** Track all blueprints that go through `load_from_blueprints()` in a `_blueprints: dict[str, list[PipelexBundleBlueprint]]` on `LibraryManager`. After all loading is done, `get_crate(library_id)` calls `LibraryCrateFactory.make_from_blueprints()` once with the full list.
+
+**Option B — Accumulate and merge crates.** Each `load_from_blueprints()` call produces a crate. Accumulate them in `_crates: dict[str, LibraryCrate]` on `LibraryManager`. Add a `merge()` classmethod to `LibraryCrate` that combines two crates (raise on key collision, recompute fingerprint). `get_crate()` returns the merged result.
+
+### Decision: Option A
+
+Option A is simpler:
+- No `merge()` method needed on `LibraryCrate`
+- No crate accumulation dict or fingerprint recomputation
+- Fewer concepts for a developer to understand
+- Blueprints are the natural unit of accumulation — they're what loading already produces
+
+Option A is also directionally correct toward the crate-first architecture (see below). Blueprints are the raw input; the crate is built from them. Accumulating the input (blueprints) rather than intermediate outputs (crates) keeps the data flow clear.
+
+### Why loading still happens incrementally
+
+Loading is order-dependent: dependencies must be loaded before dependents (pipe validation needs concepts from deps to succeed). So we can't defer all loading to the end. The library needs content loaded step by step.
+
+But **crate building for shipping purposes** is independent of the loading order. We let loading proceed as today (each step goes through its own internal crate), and build the final shipping crate once at the end from all accumulated blueprints.
+
+---
+
+## The Crate-First Architecture
+
+### Current state: loading-driven
+
+```
+CURRENT ARCHITECTURE
+════════════════════
+
+ load_libraries(dirs)
+ → parse .mthds files → blueprints
+ → resolve local deps → more blueprints
+ → load_from_blueprints(blueprints)
+ → LibraryCrateFactory → crate (internal, discarded)
+ → load_from_crate(crate) → live Library
+
+ load_from_blueprints(mthds_contents_blueprints)
+ → same path
+
+ [Phase 2] get_crate() → build final crate from accumulated blueprints
+```
+
+Library loading drives everything. Crates are an internal intermediate. Dependency resolution is interleaved with loading (`_load_single_dependency` parses, creates child Library, loads concepts, builds pipes — all in one method).
+
+### Future state: crate-driven
+
+```
+CRATE-FIRST ARCHITECTURE
+═════════════════════════
+
+ Phase 1: COLLECT (blueprints only, no Library)
+ ──────────────────────────────────────────────
+ Parse local .mthds files → blueprints
+ Scan blueprints for deps (manifest + address-based)
+ Resolve deps: local lookup + remote fetch
+ Parse dep blueprints
+ Recurse for transitive deps
+ → all_blueprints[]
+
+ Phase 2: BUILD CRATE (pure data transform)
+ ──────────────────────────────────────────────
+ LibraryCrateFactory.make_from_blueprints(all_blueprints)
+ Strip to transitive closure of target pipe (optional)
+ → one complete, self-contained LibraryCrate
+
+ Phase 3: LOAD (same code path on submitter and worker)
+ ──────────────────────────────────────────────────────
+ load_from_crate(crate)
+ → validates, creates live objects, ready to execute
+```
+
+The key shift: **dependency resolution moves out of library loading and into blueprint collection.** The crate becomes the single source of truth. Both submitter and worker use the same `load_from_crate()` path.
+
+---
+
+## Future Capability 1: Crate Stripping (Transitive Closure)
+
+### What
+
+From the entry pipe, walk the transitive closure of pipe and concept dependencies to determine the minimal subset needed. Strip the crate to only those concepts, pipes, and domains.
+
+### Why
+
+- Reduces Temporal payload size (relevant before StoragePayloadCodec, and even after for efficiency)
+- Clarifies execution context — a worker receives exactly what it needs, nothing more
+- Enables better error messages — "pipe X not in crate" vs. "pipe X not found" when deps are missing
+
+### How it fits
+
+Crate stripping is a pure data transform on blueprints. Controller blueprints contain pipe ref strings (PipeSequence has `steps`, PipeBatch has `pipe`, etc.), so the dependency graph can be traced from blueprints alone without loading into live Pipe objects.
+
+In the crate-first architecture, stripping slots in between Phase 1 (collect) and Phase 2 (build):
+
+```
+ all_blueprints = collect(...)
+ required_refs = trace_transitive_closure(target_pipe, all_blueprints)
+ stripped_blueprints = filter(all_blueprints, required_refs)
+ crate = LibraryCrateFactory.make_from_blueprints(stripped_blueprints)
+```
+
+### Current blocker
+
+Today, dependency content goes into **child libraries** via `_load_single_dependency`, which has its own loading path that doesn't go through `load_from_crate()` or produce blueprints that accumulate. To strip, we need all blueprints (including dependency blueprints) in one flat list. This requires decoupling resolution from loading (see "What needs to change" below).
+
+---
+
+## Future Capability 2: Remote Package Dependencies
+
+### What
+
+Resolve dependencies from remote package addresses (e.g., `github.com/org/repo/package`). Fetch the package (temporary clone or download), parse its bundles, and include the needed content in the crate.
+
+### Why
+
+- Enables running pipelines whose dependencies aren't pre-installed on the worker
+- The crate becomes truly self-contained — a worker needs nothing beyond PIPELEXPATH base
+- Enables cloud-native execution where workers are stateless
+
+### How it fits
+
+Remote fetching is pure I/O — it doesn't need a live Library. It produces blueprints, same as local resolution. In the crate-first architecture, it slots into Phase 1 (collect):
+
+```
+ Phase 1: COLLECT
+ Parse local .mthds files → blueprints
+ Scan for deps (manifest + address-based)
+ For each dep:
+ if local: find installed package → parse blueprints
+ if remote: fetch package → parse blueprints ← NEW
+ Recurse for transitive deps
+ → all_blueprints[]
+```
+
+### Current blocker
+
+Same as crate stripping: `_load_single_dependency` interleaves resolution with loading. Remote fetch would need to happen before loading, which means splitting the "resolve + collect blueprints" step from the "load into Library" step.
+
+---
+
+## What Needs to Change (Incremental Path)
+
+The path from current state to crate-first is incremental. Each step is independently valuable.
+
+### Step 1: Phase 2 (now) — Blueprint accumulation
+
+**Already decided.** Track blueprints in `LibraryManager._blueprints`, build one crate via `get_crate()`. No merge logic. This establishes the pattern of accumulating blueprints as the unit of crate construction.
+
+Dependency content (child libraries) is NOT included in the crate. Known limitation, accepted for Phase 2.
+
+### Step 2: Extract blueprint collector from dependency loading
+
+Split `_load_single_dependency` into two parts:
+1. **Collect**: resolve the dependency, parse its .mthds files into blueprints, determine exports
+2. **Load**: create child Library, load domains/concepts/pipes, register aliases
+
+The collector produces blueprints. The loader consumes them. Today they're one method.
+
+After this split, the collector's blueprints can be accumulated into `_blueprints[library_id]` alongside the main package's blueprints. The crate becomes complete (includes dependency content).
+
+### Step 3: Add remote fetch to the collector
+
+With the collector extracted, remote fetch is a new resolution strategy alongside local lookup:
+
+```python
+def _collect_dependency_blueprints(self, dep) -> list[PipelexBundleBlueprint]:
+ match dep.resolution:
+ case ResolutionStrategy.LOCAL:
+ return self._collect_from_local(dep)
+ case ResolutionStrategy.REMOTE:
+ return self._collect_from_remote(dep) # NEW
+```
+
+### Step 4: Add crate stripping
+
+With all blueprints (main + deps) accumulated, add a stripping step before crate construction:
+
+```python
+def get_crate(self, library_id, target_pipe_ref=None) -> LibraryCrate | None:
+ all_blueprints = self._blueprints.get(library_id)
+ if target_pipe_ref:
+ all_blueprints = strip_to_transitive_closure(target_pipe_ref, all_blueprints)
+ return LibraryCrateFactory.make_from_blueprints(all_blueprints)
+```
+
+### Step 5 (optional): Fully separate Collect → Build → Load
+
+At this point, the three phases are logically separate but still wired through `LibraryManager`. The final step is to make them explicitly separate, so that:
+- A CLI tool can collect + build a crate without loading (e.g., `pipelex crate build`)
+- A worker can load from a crate without collecting (already works via `load_from_crate`)
+- Tests can construct crates directly from blueprints without the full loading machinery
+
+---
+
+## Cross-Package Refs in a Flattened Crate
+
+One structural challenge for including dependency content in the crate: cross-package pipe refs use alias-based syntax (`alias::domain.pipe`). Today, these resolve through child library lookups. In a flattened crate, there are no child libraries.
+
+Options:
+1. **Resolve aliases during crate building** — replace `alias::domain.pipe` with the actual `domain.pipe` ref in all controller blueprints before they enter the crate. The crate contains only flat domain-qualified refs.
+2. **Carry alias mappings in the crate** — add an `aliases: dict[str, str]` field that maps alias-prefixed refs to domain-qualified refs. `load_from_crate()` uses this to register aliased lookups.
+3. **Defer** — cross-package deps on workers remain a known limitation until this is solved.
+
+Option 1 is cleanest (the crate is fully resolved, no runtime lookup needed) but requires modifying blueprint data during crate construction. Option 2 is simpler to implement but adds runtime complexity. To be decided when we reach Step 2.
+
+---
+
+## Relationship to Other Phases
+
+| Phase | Relationship to crate-first |
+|-------|-----------------------------|
+| Phase 2 (Temporal crate) | Establishes blueprint accumulation pattern (Step 1) |
+| Phase 3 (WM hydration) | Independent — concerns deserialization, not crate content |
+| Phase 4 (StoragePayloadCodec) | Complementary — handles large crates; stripping reduces need |
+| Crate stripping | Step 4 of this vision |
+| Remote deps | Step 3 of this vision |
+| Library fingerprint validation | Independent — compares base library fingerprints |
+| Cross-worker cache | Complementary — caches the crate, benefits from stripping |
diff --git a/wip/instructor-unwrap-other-workers.md b/wip/instructor-unwrap-other-workers.md
new file mode 100644
index 000000000..ff97058e4
--- /dev/null
+++ b/wip/instructor-unwrap-other-workers.md
@@ -0,0 +1,216 @@
+# Generalize the `InstructorRetryException` unwrap fix to the other LLM workers
+
+> Drafted: 2026-05-03
+> Predecessor: the Anthropic `_gen_object` fix on `refactor/Inference-error-handling`
+> ([CHANGELOG entry](../CHANGELOG.md#unreleased)).
+
+## Background — what we just fixed in Anthropic
+
+`AnthropicLLMWorker._gen_object` runs requests through `instructor`. When the
+underlying Anthropic SDK raises an exception (`RateLimitError`,
+`APITimeoutError`, `APIConnectionError`, `BadRequestError`,
+`PermissionDeniedError`, `AuthenticationError`), `instructor`'s tenacity-driven
+retry loop catches it, exhausts retries, and re-raises wrapped in
+`InstructorRetryException` with the original exception parked at
+`exc.failed_attempts[-1].exception` (and reachable via
+`exc.__cause__.last_attempt._exception` as a tenacity fallback).
+
+The handler we inherited matched only the bare SDK types and treated
+`InstructorRetryException` as a content/validation failure
+(`InferenceErrorCategory.CONTENT`, non-retryable). End result: every quota
+exhaustion, network blip, auth failure, or rate-limit on the structured-gen
+path was mis-categorized as a content failure. Retry policy never kicked in,
+billing / config remediation hints never fired, the user got a wall of
+"content failed" for what was really an ops/transient problem.
+
+Fix landed in `pipelex/plugins/anthropic/anthropic_llm_worker.py`:
+
+- `_extract_underlying_sdk_exception(instructor_exc)` — recovers the SDK
+ exception (`failed_attempts[-1].exception`, fallback to
+ `__cause__.last_attempt._exception`).
+- `_raise_categorized_anthropic_sdk_error(sdk_exc, chain_from=None)` — single
+ source of truth for SDK→pipelex categorization, shared between `_gen_text`
+ and `_gen_object`.
+- `_gen_object` `InstructorRetryException` branch unwraps and routes through
+ the helper; truly unrecognized inner exceptions (e.g. `pydantic.ValidationError`
+ from a schema mismatch) keep the original `CONTENT` fallback.
+
+End-to-end test in
+`tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_object_error_handling.py::test_real_instructor_wraps_rate_limit_and_fix_unwraps_correctly`
+locks in the assumption against a real `instructor.from_anthropic(...)` so
+this can't silently rot if `instructor`'s wrapping shape changes.
+
+## Same bug, four more workers
+
+Every LLM worker that uses `instructor.from_*` and a separate top-level catch
+for SDK exceptions has the same defect. Empirically confirmed for OpenAI:
+
+```text
+openai: instructor.core.exceptions.InstructorRetryException, fa[0]=RateLimitError
+```
+
+(Same one-shot script used to prove the Anthropic case — a real
+`instructor.from_openai` driving an `AsyncOpenAI` whose
+`chat.completions.create` raises `openai.RateLimitError` produces the wrapped
+shape.) Mistral and Google are structurally identical: they call
+`instructor.from_mistral` / `instructor.from_genai`, then catch the bare SDK
+exception types separately. Only the SDK-exception class names differ.
+
+### Per-worker damage report
+
+| Worker | File | SDK types currently caught | Behavior on wrapped SDK error today | Fix shape |
+|---|---|---|---|---|
+| OpenAI Completions | `pipelex/plugins/openai/openai_completions_llm_worker.py:215-285` | `NotFoundError`, `RateLimitError`, `APITimeoutError`, `APIConnectionError`, `BadRequestError`, `AuthenticationError` | All flatten to `CONTENT`. `is_quota_exhaustion_openai` / `is_content_policy_violation` discriminators never run. | Same as Anthropic: extract helper + unwrap branch. Almost mechanical port. |
+| OpenAI Responses | `pipelex/plugins/openai/openai_responses_llm_worker.py:195-263` | Same six as above, plus `LLMModelNotFoundError` from `NotFoundError` | Same flattening. Note: `InstructorRetryException` clause is currently *first* in the except chain, but that doesn't matter — the order didn't cause the bug, the wrapping did. | Same as Anthropic. Keep the `LLMModelNotFoundError` / `model_handle` plumbing in the helper for the `NotFoundError` branch. |
+| Mistral | `pipelex/plugins/mistral/mistral_llm_worker.py:208-238` | `MistralError` (single base class) routed through `_classify_mistral_error` | Wrapped `MistralError` → `CONTENT`; the existing `_classify_mistral_error` (which already discriminates by status code / message) never runs. | Smaller change: unwrap, and if the underlying is a `MistralError`, call `_classify_mistral_error` from inside the unwrap branch. |
+| Google Gemini | `pipelex/plugins/google/google_llm_worker.py:283-320` | `genai_errors.ServerError` (→ `TRANSIENT`), `genai_errors.ClientError` (→ `_classify_google_client_error`) | Wrapped `ServerError` (which would have been `TRANSIENT`) → `CONTENT`. Wrapped `ClientError` → `CONTENT`. The same kind of category collapse. | Same as Mistral: unwrap, dispatch by `isinstance(underlying, ServerError | ClientError)`, fall back to `CONTENT`. |
+
+`pipelex/plugins/anthropic/anthropic_llm_worker.py` is now the reference; the
+other four should converge on the same shape.
+
+## Recommended approach
+
+Two viable shapes. Pick one and apply consistently.
+
+### Option A — replicate per-worker (safe, repetitive)
+
+For each of the four workers above, mirror the Anthropic change:
+
+1. Extract the categorization logic into `_raise_categorized_<provider>_sdk_error(sdk_exc, chain_from=None)`. Refactor `_gen_text` (or the equivalent text-gen path) to use it so the fix doesn't bifurcate logic.
+2. Add `_extract_underlying_sdk_exception(instructor_exc)` (identical across workers — copy-paste is fine for this round).
+3. In the `InstructorRetryException` branch of `_gen_object`, call the extractor; if it returns one of the recognized SDK types, route through the helper with `chain_from=instructor_exc`; otherwise keep the existing `CONTENT` fallback.
+
+Pro: smallest possible diff per worker; no cross-cutting refactor; failure of one fix doesn't impact others.
+Con: ~50 lines of identical extractor code in five places.
+
+### Option B — share the extractor
+
+`_extract_underlying_sdk_exception` is provider-agnostic (it only reads
+attributes that `InstructorRetryException` and tenacity always populate). It
+can move to `pipelex/cogt/inference/error_classification.py` (next to
+`is_quota_exhaustion_*` / `is_content_policy_violation`) and be imported by
+every worker. The categorization helper stays per-worker because the SDK
+exception types differ.
+
+Pro: removes duplication; one place to fix if `instructor` ever changes the
+attribute name.
+Con: one extra dependency edge across workers; needs a small unit test for
+the shared helper covering both the `failed_attempts` path and the
+`__cause__.last_attempt._exception` fallback.
+
+**Recommendation: Option B.** The function is twelve lines and has nothing
+provider-specific in it. Putting it in `error_classification` matches what we
+already do for the message-pattern discriminators.
+
+## Per-worker recipe
+
+### OpenAI Completions (`openai_completions_llm_worker.py`)
+
+Mirror Anthropic almost verbatim. The OpenAI exception classes have identical
+constructor shapes to Anthropic's (httpx-based), so the test fixtures port
+over directly.
+
+- The `_gen_object` body has a nested `try { try { ... } except InstructorRetryException }` already. Collapse to a single try; replace the inner with the unwrap-and-dispatch branch; replace the outer six SDK clauses with a single tuple-catch that delegates to the helper.
+- Be careful with `NotFoundError`: it should keep going to `LLMCompletionError(error_category=CONFIGURATION)` to match `_gen_text`'s behavior.
+
+### OpenAI Responses (`openai_responses_llm_worker.py`)
+
+Same pattern as Completions, but the `NotFoundError` branch raises a more
+specialized `LLMModelNotFoundError(message=msg, model_handle=...)`. Keep that
+specialization inside the categorization helper — it's a real signal worth
+preserving for callers that want to swap models.
+
+### Mistral (`mistral_llm_worker.py`)
+
+`_classify_mistral_error` is doing the right work — it just never gets called
+on the wrapped path. Two-line fix in the `InstructorRetryException` branch:
+
+```python
+except InstructorRetryException as instructor_exc:
+ underlying = _extract_underlying_sdk_exception(instructor_exc)
+ if isinstance(underlying, MistralError):
+ raise self._classify_mistral_error(underlying) from instructor_exc
+ msg = f"Mistral structured generation failed after retries for model '{self.inference_model.desc}': {instructor_exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from instructor_exc
+```
+
+(`_classify_mistral_error` returns the `LLMCompletionError` to raise — match
+its existing call shape.)
+
+### Google Gemini (`google_llm_worker.py`)
+
+Symmetric to Mistral:
+
+```python
+except InstructorRetryException as instructor_exc:
+ underlying = _extract_underlying_sdk_exception(instructor_exc)
+ if isinstance(underlying, genai_errors.ServerError):
+ msg = f"Google API server error for model '{self.inference_model.desc}': {underlying}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.TRANSIENT) from instructor_exc
+ if isinstance(underlying, genai_errors.ClientError):
+ raise self._classify_google_client_error(underlying) from instructor_exc
+ msg = f"Google structured generation failed after retries for model '{self.inference_model.desc}': {instructor_exc}"
+ raise LLMCompletionError(msg, error_category=InferenceErrorCategory.CONTENT) from instructor_exc
+```
+
+## Test plan
+
+Mirror `tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_object_error_handling.py` for each provider. The structure is reusable; only the SDK-exception
+factories and the patched helper paths change.
+
+### Shared test pieces (worth lifting into a shared helper module)
+
+- `_wrap_in_instructor_retry(sdk_exc, *, include_failed_attempts=True)` — builds an `InstructorRetryException` matching what real instructor produces. Already in the Anthropic test file; could move to `tests/helpers/instructor_test_utils.py` for reuse.
+- `_DummySchema(BaseModel)` — minimal pydantic model passed as `response_model`.
+- A `_make_llm_job(mocker)` skeleton — already provider-agnostic.
+
+### Per-worker cases to cover
+
+For each of OpenAI Completions, OpenAI Responses, Mistral, Google:
+
+| Case | What it proves |
+|---|---|
+| Wrapped rate-limit (or provider equivalent) → `TRANSIENT` | The unwrap branch routes to the rate-limit handler. |
+| Wrapped quota-exhaustion message → `CAPACITY` + billing `user_action` | `is_quota_exhaustion_*` discriminator runs after unwrap. |
+| Wrapped timeout → `TRANSIENT` | Confirms timeout types unwrap. |
+| Wrapped connection error → `TRANSIENT` | Confirms network errors unwrap. |
+| Wrapped content-policy bad-request → `CONTENT` + safety `user_action` | `is_content_policy_violation` discriminator runs after unwrap. |
+| Wrapped auth error → `CONFIGURATION` (or provider-specific credentials exception) | Auth still routes correctly. |
+| Wrapped `ValueError` (non-SDK) → `CONTENT` fallback | Genuine schema/validation failures stay as `CONTENT`. |
+| Real-instructor end-to-end (one per provider) | Locks in instructor's actual wrapping shape so a future instructor upgrade fails loudly here, not in production. |
+
+Provider-specific notes:
+
+- **OpenAI Completions / Responses** can construct exceptions identically to Anthropic via `httpx.Response(status_code=..., request=...)`. The end-to-end test uses `instructor.from_openai(openai.AsyncOpenAI(api_key="fake"))` and patches `client.chat.completions.create` (Completions) or `client.responses.create` (Responses) with `mocker.AsyncMock(side_effect=sdk_exc)`.
+- **Mistral**'s `MistralError` constructor differs across versions — confirm the kwargs in the installed `mistralai` before writing factories. The end-to-end test patches `mistralai.Mistral.chat.complete_async` (or whatever the current `from_mistral` adapter calls).
+- **Google**'s `genai_errors.ServerError` / `ClientError` take a different constructor — they wrap an HTTP response object. Look at how `_classify_google_client_error` already constructs them in tests (if any) before writing factories. The end-to-end test patches the `genai.Client.aio.models.generate_content`-equivalent that `instructor.from_genai` calls.
+
+If the Mistral / Google constructor shapes are awkward, prefer the
+construct-by-hand pattern (`_wrap_in_instructor_retry(real_sdk_exc)`) for the
+seven categorization cases and keep only the real-instructor end-to-end test
+provider-specific.
+
+## Risks and gotchas
+
+- **`instructor` import path.** Anthropic's fix already moved from `instructor.exceptions` to `instructor.core` (the former is deprecated and slated for removal). Do the same in the four other workers while you're in the file — same one-line change, frees us from the deprecation warning across the suite.
+- **`MistralError` and the Mistral `_classify_mistral_error` helper expect a *raw* `MistralError`,** not the wrapped one. Confirm the helper doesn't introspect attributes that the wrapped exception lacks. Same for `_classify_google_client_error`.
+- **`from instructor_exc` chaining.** In every fix, `chain_from=instructor_exc` so the traceback shows `instructor → tenacity → SDK`. Don't chain from the bare SDK exc — it loses the retry context that's useful for debugging.
+- **Tests that hit real instructor + AsyncMock'd SDK calls** are slightly slow (~0.5s each on Anthropic) because instructor still does its full attempt loop. Keep one such test per provider, not one per case.
+- **`pydantic.ValidationError` is a legitimate `CONTENT` failure.** When the LLM returns JSON that doesn't match the schema, instructor raises `InstructorRetryException` with a `ValidationError` (or `JSONDecodeError`) at `failed_attempts[-1].exception`. The unwrap branch must keep this categorized as `CONTENT` — `_extract_underlying_sdk_exception` returns the `ValidationError`, the helper doesn't recognize it, and we fall through to the existing `CONTENT` fallback. The Anthropic test `test_unrecognized_underlying_falls_back_to_content` covers this. Replicate it everywhere.
+- **Don't add a new SDK-exception type silently.** If a provider adds, say, an `InternalServerError`, the helper will return without raising and we fall through to `CONTENT`. That's a pre-existing gap, not a new one — but it's worth a short audit of each provider's SDK exception hierarchy while doing this work.
+
+## Suggested commit shape
+
+Two PRs:
+
+1. **Lift `_extract_underlying_sdk_exception` to `pipelex/cogt/inference/error_classification.py`** (Option B). Update the Anthropic worker to import from the shared location instead of defining it locally. Add unit tests for the shared helper covering both extraction paths. Pure refactor, zero behavior change.
+2. **Apply the unwrap-and-dispatch fix to the four remaining workers**, one commit per worker. Each commit includes the worker change + its test file. Keeps blame readable and lets `git revert` rescue any single provider if the fix turns out to have an edge case.
+
+CHANGELOG: one entry per worker fix in `[Unreleased] → Fixed`, mirroring the
+Anthropic entry already on the branch.
+
+## Out of scope (call out, don't fix here)
+
+- Other workers that *do not* go through `instructor` (e.g. `bedrock_llm_worker.py`, `mistral_extract_worker.py`) have their own error-handling problems documented in `wip/worker-error-handling-review.md`. Those are separate and shouldn't be bundled with this fix.
+- `azure_img_gen_worker.py` raises raw `httpx.HTTPStatusError` — same overall theme (errors not categorized) but nothing to do with `instructor` wrapping.
+- Image-gen workers that use `instructor` (none today, AFAIK) would need the same fix if any are added later.
diff --git a/wip/phase6a-local-cross-package-deps.md b/wip/phase6a-local-cross-package-deps.md
new file mode 100644
index 000000000..604dbc32a
--- /dev/null
+++ b/wip/phase6a-local-cross-package-deps.md
@@ -0,0 +1,39 @@
+# Phase 6a — Local Cross-Package Dependencies in Crate
+
+> **Status**: Not started
+> **Goal**: Dependency blueprints are included in the crate. Workers can execute pipelines with cross-package deps without having the dependency packages installed on PIPELEXPATH.
+> **Predecessor**: Phase 2 (crate propagation, shipped — see `archive/00-master-plan.md`).
+> **Related**: [02-master-plan.md](02-master-plan.md), [future-crate-first-architecture.md](future-crate-first-architecture.md).
+
+---
+
+## Why
+
+Today, cross-package dependencies (`alias->domain.ConceptCode`, `alias::domain.pipe_code`) resolve through child library lookups at loading time. The `LibraryCrate` shipped to Temporal workers does NOT include dependency content — only the main package's blueprints. This means workers must have all dependency packages pre-installed on PIPELEXPATH. Phase 6 removes this requirement: the crate becomes truly self-contained.
+
+## Key changes
+
+1. **Extract blueprint collector from `_load_single_dependency`**: Split the current method into two parts:
+ - **Collect**: resolve the dependency, parse its `.mthds` files into blueprints, determine exports.
+ - **Load**: create child Library, load domains/concepts/pipes, register aliases.
+
+ The collector produces blueprints that accumulate into `_blueprints[library_id]` alongside the main package's blueprints.
+
+2. **Resolve cross-package aliases in the flattened crate**: Cross-package refs use alias-based syntax (`alias->domain.ConceptCode`). In a flattened crate, there are no child libraries. Options:
+ - Resolve aliases during crate building (replace `alias->domain.ConceptCode` with `domain.ConceptCode` in all blueprints).
+ - Carry alias mappings in the crate (`aliases: dict[str, str]`) for runtime resolution.
+
+ Decision TBD when starting implementation.
+
+3. **Update `load_from_crate()`**: Handle dependency content in the flat crate — register domains, concepts, and pipes from deps.
+
+## Done when
+
+- [ ] `_load_single_dependency` split into collect + load
+- [ ] Dependency blueprints accumulate into `_blueprints[library_id]`
+- [ ] `get_crate()` produces a crate that includes dependency content
+- [ ] Cross-package aliases resolved (either at crate build time or via alias map)
+- [ ] `load_from_crate()` handles dependency content correctly
+- [ ] Integration test: PipeSequence referencing a cross-package concept/pipe, executed on Temporal worker without the dependency package on PIPELEXPATH
+- [ ] `make agent-check` passes
+- [ ] `make agent-test` passes
diff --git a/wip/phase6b-remote-deps-from-github.md b/wip/phase6b-remote-deps-from-github.md
new file mode 100644
index 000000000..9f506d5da
--- /dev/null
+++ b/wip/phase6b-remote-deps-from-github.md
@@ -0,0 +1,29 @@
+# Phase 6b — Remote Dependencies from GitHub
+
+> **Status**: Not started
+> **Goal**: Dependencies can be fetched from remote addresses (e.g., `github.com/org/repo/package`). The crate becomes fully self-contained for cloud-native execution where workers are stateless.
+> **Predecessor**: [phase6a-local-cross-package-deps.md](phase6a-local-cross-package-deps.md) — needs the blueprint collector extracted in 6a.
+> **Related**: [02-master-plan.md](02-master-plan.md), [future-crate-first-architecture.md](future-crate-first-architecture.md).
+
+---
+
+## Key changes
+
+1. **Remote resolution strategy**: Add a `REMOTE` resolution strategy to the blueprint collector extracted in 6a. Remote fetch clones/downloads the package from GitHub, parses its `.mthds` files into blueprints, and includes them in the collection.
+
+2. **Dependency address format**: Define the address format for remote deps (e.g., `github.com/org/repo@version/path/to/package`). The address goes in the package manifest.
+
+3. **Caching**: Cache fetched packages locally (content-addressed by address + version) to avoid redundant clones.
+
+4. **Transitive deps**: Remote packages may themselves have dependencies (local or remote). The collector recurses.
+
+## Done when
+
+- [ ] Remote fetch strategy implemented (git clone or archive download)
+- [ ] Dependency address format defined and parsed
+- [ ] Remote package blueprints included in crate
+- [ ] Local cache for fetched packages
+- [ ] Transitive remote deps resolved
+- [ ] Integration test: pipeline with remote dep from a GitHub repo, executed on Temporal worker
+- [ ] `make agent-check` passes
+- [ ] `make agent-test` passes
diff --git a/wip/tracing-cost-reporting-as-built.md b/wip/tracing-cost-reporting-as-built.md
new file mode 100644
index 000000000..e4c471fac
--- /dev/null
+++ b/wip/tracing-cost-reporting-as-built.md
@@ -0,0 +1,76 @@
+# Tracing & Cost Reporting — As Built
+
+> **Status**: Shipped via `feature/dynamodb-tracer` (supersedes the original Phase 4.5 Step 6 design from `archive/00-master-plan.md`).
+> **Last updated**: 2026-05-04
+> **Related**: [02-master-plan.md](02-master-plan.md), [distributed-tracing-and-reporting.md](distributed-tracing-and-reporting.md) (pre-implementation analysis — design choices superseded), [archive/phase4.5-distributed-tracing-implementation.md](archive/phase4.5-distributed-tracing-implementation.md).
+
+This file documents what actually shipped, plus the known gaps that 02-master-plan addresses.
+
+---
+
+## What shipped
+
+The original Step 6 plan called for a `TracingActivityInboundInterceptor` to set up an NDJSON event log per standalone activity. The team picked a different design:
+
+- **Pluggable backends behind `EventLogProtocol`** — `pipelex/tracing/event_log_protocol.py`. Implementations:
+ - `NdjsonEventLog` — local file backend (`pipelex/tracing/ndjson_event_log.py`).
+ - `DynamoDBEventLog` — cloud backend (`pipelex/tracing/dynamodb_event_log.py`), schema-compatible with `pipelex-api-infra`'s `TraceEventDynamoDBAdapter`. Available behind `pip install "pipelex[dynamodb]"`.
+ - `BufferingEventLog` — in-memory buffer used **inside** Temporal workflow code (`pipelex/tracing/buffering_event_log.py`). Synchronous I/O is forbidden in workflow context, so workflows buffer here and flush via an activity.
+ - `InMemoryEventLog` — for tests.
+ - Backend chosen at runtime via `make_event_log(tracing_config)` in `pipelex/tracing/event_log_factory.py` (`TracingBackend.NDJSON | DYNAMODB | TEMPORAL_DYNAMODB`).
+- **Workflow → activity flush, not interceptor.** `WfPipeRouter.run()` (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`) wires a `BufferingEventLog` into the per-workflow `GraphTracerManager` and into the `ReportingManager` via `set_event_log(context_key=workflow_id, ...)`. After pipe execution, buffered events are drained and persisted by `act_flush_trace_events` (`pipelex/temporal/tprl_pipe/act_flush_trace_events.py`), which runs the synchronous boto3 / file writes off the workflow thread.
+- **Per-context keying** — `ReportingManager` holds a dict of `_event_log_contexts` keyed by `graph_context.lookup_key`, so concurrent workflows don't trample each other.
+- **Direct mode unchanged** — `pipeline_run_setup.py:282` calls the same `set_event_log` path, so the same `EventLogProtocol` machinery runs in-process for non-Temporal execution.
+- **Graph assembly across workers** — `pipelex/pipe_run/graph_assembly.py` (`assemble_graph_on_output`) and `pipelex/temporal/tprl_pipe/act_assemble_graph.py` read all events for a `pipeline_run_id` from the configured backend and feed them into `GraphSpecAssembler` to build the cross-worker `GraphSpec`.
+
+## What works today
+
+| Scenario | Captured? | How |
+|---|---|---|
+| Direct mode (single process) | ✅ | `pipeline_run_setup` configures event log on `ReportingManager`; events emitted in-process. |
+| Single-worker Temporal (workflows + activities on same Worker bundle) | ✅ | Activities share the workflow's process; usage events emit through the singleton `ReportingManager` whose `set_event_log` was configured by `WfPipeRouter`. |
+| Parent + child workflows on different worker processes | ✅ | Each workflow's process buffers its own events and flushes its own slice. All slices land in the same NDJSON dir / DynamoDB partition keyed by `pipeline_run_id`. |
+| Cross-worker **graph** assembly | ✅ | `assemble_graph_on_output` / `act_assemble_graph` reads all events for the `pipeline_run_id` and runs `GraphSpecAssembler`. |
+
+## What is broken or missing
+
+### Issue T1 — Activities on standalone (separate-process) workers lose usage events
+
+**Severity**: Blocks the whole point of distributed activity workers.
+
+`tasks.py` currently bundles workflows + activities into the same `TaskPack`, so this issue is latent. But the explicit design intent is to split activities onto standalone Worker pools (separate processes from the workflow Worker pool). When that split happens:
+
+- The activity's process never runs `WfPipeRouter.run()`, so its `ReportingManager` has no entry in `_event_log_contexts` for the workflow's `lookup_key`.
+- `ReportingManager._emit_usage_event` returns silently when the context lookup misses (`reporting_manager.py:211-213`).
+- `ReportingManager._get_registry` auto-creates a worker-local `UsageRegistry` (with the explicit TODO at `reporting_manager.py:111-117`: *"Auto-create registry for unknown pipeline IDs. This happens when Activities report inference jobs on a Temporal worker where open_registry() was never called (it runs on the API process). TODO: replace with proper distributed reporting system"*).
+- The worker-local registry is never read back by anyone — usage data is lost on the activity worker.
+
+**Note**: The `BufferingEventLog` + `act_flush_trace_events` pattern is structurally tied to a workflow lifecycle; it cannot serve standalone activity workers as-is. A different mechanism is needed (the original interceptor design, or per-activity event log construction from tracing config + JobMetadata).
+
+### Issue T2 — Cross-worker cost report assembly is not wired
+
+**Severity**: High — events are persisted but never turned into a report.
+
+The pieces all exist:
+
+- `UsageAggregator.aggregate(events) → list[AnyTokensUsage]` (`pipelex/tracing/usage_aggregator.py`).
+- `ReportingManager.inject_tokens_usages(pipeline_run_id, tokens_usages)` whose docstring literally says: *"Used after assembling usage data from distributed trace events, so that generate_report() can produce a complete cost report across all workers."* (`reporting_manager.py:191`).
+- `ReportingManager.generate_report(pipeline_run_id)` (`reporting_manager.py:247`).
+
+But **nothing in the runtime calls them**. The graph assembly path (`graph_assembly.py`, `act_assemble_graph.py`) reads events back, but only feeds them into `GraphSpecAssembler` — never into `UsageAggregator`. `generate_report()` has zero runtime callers; only integration tests invoke it.
+
+Net effect: even when usage events ARE captured (i.e., the same-worker case in the table above), the captured events sit in NDJSON / DynamoDB and never become a cross-worker cost report.
+
+### Issue T3 — `set_event_log` is on a process-singleton `ReportingManager`
+
+**Severity**: Design smell that magnifies T1.
+
+`ReportingManager` is a process-singleton accessed via `hub.get_report_delegate()`. The per-context `_event_log_contexts` dict + `lookup_key` scheme works because the workflow process is the same process that emits events from activities (today). The moment activity execution moves off-process, the singleton + context-dict pattern stops being the right shape — the activity needs to pull its event log from request-scoped data (e.g., `JobMetadata` / `TracingContext`), not from a process-local dict that was never populated on its process.
+
+---
+
+## How this connects to 02-master-plan
+
+- T1 is the **top priority** in [02-master-plan.md](02-master-plan.md) — the whole point of distributed execution is to allow activities on standalone Worker pools.
+- T2 is the **second priority** — once T1 is solved, the assembly wiring is the small remaining step that makes cost reporting actually report.
+- T3 is the architectural shape that the T1 + T2 work will refactor.
diff --git a/wip/worker-error-handling-review.md b/wip/worker-error-handling-review.md
new file mode 100644
index 000000000..718a89ca0
--- /dev/null
+++ b/wip/worker-error-handling-review.md
@@ -0,0 +1,270 @@
+# Worker Error Handling Review
+
+> Review date: 2026-04-05
+> Scope: All inference workers (LLM, Extract, ImgGen, Search) across all SDK plugins
+
+---
+
+## Tier Ranking: Best to Worst
+
+### Tier 1 — Gold Standard
+
+| Worker | Provider | Why |
+|--------|----------|-----|
+| `openai_responses_llm_worker.py` | OpenAI Responses API | Catches all 4 SDK exception types individually; uses specialized `LLMModelNotFoundError` with `model_handle`; nested instructor error handling; includes model descriptor in every message |
+| `openai_completions_llm_worker.py` | OpenAI Completions API | Same 4-exception pattern; response structure validation after call; instructor retry errors caught separately |
+| `anthropic_llm_worker.py` | Anthropic | Catches `BadRequestError`, `APIConnectionError`, `AuthenticationError`; uses specialized `AnthropicCredentialsError`; constructor validates `max_tokens`; response content validated block-by-block |
+| `gateway_extract_worker.py` | Gateway (Portkey) | Catches `portkey_exceptions.APIError`; tenacity retry with custom `_is_retryable_portkey_error` predicate; logs attempt number + wait duration; error summary via `GatewayFactory` |
+| `gateway_search_worker.py` | Gateway (Portkey) | Same retry pattern; content extraction wrapped in `(KeyError, IndexError, TypeError)` catch; null-response guard after retries |
+
+### Tier 2 — Partial Coverage
+
+| Worker | Provider | Gap |
+|--------|----------|-----|
+| `openai_completions_img_gen_worker.py` | OpenAI Completions (images) | Catches `NotFoundError`, `APIConnectionError`, `BadRequestError` but **missing `AuthenticationError`**; good response parsing with multiple fallback strategies |
+| `gateway_img_gen_worker.py` | Gateway (images) | Catches `portkey_exceptions.APIError`; complex multi-schema response parsing with accumulated `parsing_errors`; FAL polling fallback. But **no retry logic** unlike gateway extract/search |
+| `openai_img_gen_worker.py` | OpenAI Images | Good response validation (checks data, format, size, base64). But **no SDK exception handling at all** — relies on raw exceptions bubbling up |
+| `bedrock_llm_worker.py` | AWS Bedrock | Constructor validates `max_tokens`; capability checks for reasoning params. But **no try/except around API calls** |
+| `mistral_llm_worker.py` | Mistral | Constructor validates `max_tokens`; detailed response content type handling (TextChunk/ThinkChunk). But **no SDK exception handling** |
+
+### Tier 3 — Minimal or No Error Handling
+
+| Worker | Provider | Issue |
+|--------|----------|-------|
+| `google_llm_worker.py` | Google Gemini | **No SDK exception handling**; delegates response extraction to `GoogleFactory`; has its own `GoogleLLMWorkerError` but only for type mismatches |
+| `google_img_gen_worker.py` | Google Gemini Images | **No SDK exception handling**; response validation is good (checks candidates, content parts, mime type) but API errors go uncaught |
+| `azure_img_gen_worker.py` | Azure REST (httpx) | Uses `response.raise_for_status()` which throws raw `httpx.HTTPStatusError` — **no wrapping in domain exceptions** |
+| `mistral_extract_worker.py` | Mistral OCR | **No error handling at all** — delegates entirely to SDK |
+| `fal_img_gen_worker.py` | FAL | **No error handling** — logs FAL events but doesn't catch failures |
+| `huggingface_img_gen_worker.py` | HuggingFace | **No error handling** — trusts SDK entirely |
+| `docling_extract_worker.py` | Docling (local) | Only has `finally` for temp file cleanup — **no error handling** on conversion |
+| `pypdfium2_worker.py` | pypdfium2 (local) | Input validation only — **no error handling** on PDF operations |
+| `linkup_extract_worker.py` | Linkup | **No error handling** |
+| `linkup_search_worker.py` | Linkup | **No error handling** |
+
+---
+
+## What the Best Workers Do Right
+
+These patterns from Tier 1 workers should be the standard:
+
+### 1. Catch specific SDK exceptions individually
+
+```python
+except NotFoundError as exc:
+ msg = f"Model or deployment '{self.inference_model.model_id}' not found: {exc}"
+ raise LLMModelNotFoundError(message=msg, model_handle=self.inference_model.name) from exc
+except APIConnectionError as exc:
+ msg = f"API connection error: {exc}"
+ raise LLMCompletionError(msg) from exc
+except BadRequestError as exc:
+ msg = f"Bad request with model {self.inference_model.desc}:\n{exc}"
+ raise LLMCompletionError(msg) from exc
+except AuthenticationError as exc:
+ msg = f"Authentication error: {exc}"
+ raise CredentialsError(msg) from exc
+```
+
+**Why this matters for agents:** An agent skill iterating on a failing method needs to know *why* it failed. "Model not found" means the `.mthds` file references a wrong model — the agent can suggest alternatives. "Auth error" means the user needs to fix their API key — the agent should stop retrying and ask the human. "Bad request" might mean the prompt is too long or uses unsupported features — the agent can try reformulating.
+
+### 2. Use specialized exception types with structured fields
+
+The `LLMModelNotFoundError` carries `model_handle`, `ModelChoiceNotFoundError` carries `available_options` and `suggestions`. This lets upstream code (pipe operators, CLI, agent skills) make smart decisions:
+
+```python
+except ModelChoiceNotFoundError as exc:
+ # Agent can show: "Model 'gpt-5-turbo' not found. Did you mean: gpt-4-turbo, gpt-4o?"
+ suggest_alternatives(exc.suggestions, exc.available_options)
+```
+
+### 3. Validate response structure after the call
+
+Even when the API call succeeds, the response may be malformed. Tier 1 workers check:
+- Response is not None
+- Expected fields exist (choices, content, data)
+- Content is not empty
+- Types match expectations
+
+### 4. Retry with discrimination (gateway workers)
+
+```python
+def _is_retryable_portkey_error(self, exc: BaseException) -> bool:
+ if isinstance(exc, portkey_exceptions.NotFoundError):
+ msg = str(exc).lower()
+ return "specified deployment could not be found" in msg
+ return False
+```
+
+Not all errors deserve retries. Auth errors and bad requests should fail fast. Transient deployment errors and rate limits should retry with backoff.
+
+### 5. Include model descriptor in every error message
+
+Every error message includes `self.inference_model.desc` or `self.inference_model.model_id`. This is critical when a pipeline uses multiple models — the user needs to know *which* model failed.
+
+---
+
+## What's Missing Across the Board
+
+### A. Rate limit handling — nowhere
+
+No worker catches rate limit errors (`RateLimitError` in OpenAI, `429` status in others). This is the most common transient error in production. The SDK's built-in retry may handle some cases, but:
+- We don't surface "you're being rate limited" to the user
+- We don't have backoff control
+- Agents can't distinguish "wait and retry" from "permanently broken"
+
+### B. Timeout handling — nowhere
+
+No worker catches timeout errors. Long-running generations (especially image gen) can time out. Users see a generic connection error instead of "generation timed out, try again or use a faster model."
+
+### C. Content policy / safety filter errors — nowhere
+
+OpenAI and Anthropic can reject requests due to content policy. These come as `BadRequestError` with specific error codes. Currently they're lumped in with all other bad requests. Users get "bad request" when they should get "content was flagged by safety filters."
+
+### D. Quota / billing errors — nowhere
+
+Distinct from rate limits. "You've exceeded your monthly quota" or "your plan doesn't include this model" should be clearly surfaced, not buried in a generic auth or bad request error.
+
+**Detection strategy:** Rate limit (429) and quota exhaustion (429) often share the same HTTP status. Must inspect the error body to distinguish:
+
+| Provider | Rate limit (TRANSIENT) | Quota exhausted (CAPACITY) |
+|----------|----------------------|---------------------------|
+| OpenAI | `RateLimitError` (429) generic | `RateLimitError` with `"insufficient_quota"` or `"exceeded your current quota"` in message; `AuthenticationError` with `"insufficient_quota"` |
+| Anthropic | `RateLimitError` (429) generic | `RateLimitError` or `PermissionDeniedError` with `"quota"` or `"billing"` in message |
+| Google | `ResourceExhausted` generic | `ResourceExhausted` with `"billing"` or `"quota"` in message |
+| Mistral | 429 generic | 402 Payment Required; 429 with `"quota"` in message |
+| Azure | 429 generic | 429 with `"quota"` or `"billing"` in body; 402 |
+| AWS Bedrock | `ThrottlingException` generic | `ServiceQuotaExceededException`; `ThrottlingException` with `"quota"` or `"limit exceeded"` |
+| Portkey/Gateway | `APIError` 429 | `APIError` 402; 429 with `"quota"` / `"billing"` / `"insufficient"` |
+
+**Key principle:** Quota exhaustion check must happen **before** the generic rate-limit handler, since both may raise the same exception type. The message body is the discriminator.
+
+### E. No error categorization enum
+
+The codebase has `InferenceBackendCredentialsErrorType` for credential errors, but there's no general `InferenceErrorCategory` enum that classifies errors as:
+- `TRANSIENT` (retry-safe: rate limit, timeout, server error)
+- `CONFIGURATION` (fix needed: auth, missing model, wrong params)
+- `CONTENT` (user action needed: safety filter, too long, unsupported format)
+- `CAPACITY` (wait or upgrade: quota exceeded, model overloaded)
+
+This classification is essential for agent-driven error recovery.
+
+---
+
+## Proposed Action Plan
+
+### Phase 1: Error classification infrastructure
+
+1. **Create `InferenceErrorCategory` enum** with categories: `TRANSIENT`, `CONFIGURATION`, `CONTENT`, `CAPACITY`, `UNKNOWN`
+
+2. **Add `error_category` field to `CogtError`** (or a new `InferenceError` base class) so every inference error self-classifies
+
+3. **Add optional `user_action` field** — a short, plain-English suggestion:
+ - "Check your API key for {backend_name}"
+ - "Model '{model_handle}' is not available on your plan — try {alternatives}"
+ - "Your prompt was flagged by content filters — revise the prompt"
+ - "Rate limited — the system will retry automatically"
+
+### Phase 2: Bring all workers to Tier 1
+
+For each worker, add SDK-specific exception handling following the OpenAI Responses pattern. The specific exceptions to catch vary by SDK:
+
+| SDK | Exceptions to catch |
+|-----|-------------------|
+| OpenAI | `NotFoundError`, `APIConnectionError`, `BadRequestError`, `AuthenticationError`, `RateLimitError`, `APITimeoutError`, `APIStatusError` (catch-all for other HTTP errors) |
+| Anthropic | `BadRequestError`, `APIConnectionError`, `AuthenticationError`, `RateLimitError`, `APITimeoutError`, `APIStatusError` |
+| Google genai | `google.api_core.exceptions.NotFound`, `google.api_core.exceptions.PermissionDenied`, `google.api_core.exceptions.ResourceExhausted`, `google.api_core.exceptions.InvalidArgument`, `google.api_core.exceptions.ServiceUnavailable`, `google.api_core.exceptions.DeadlineExceeded` |
+| Mistral | `mistralai.models.sdkerror.SDKError` and HTTP-based exceptions from their client |
+| httpx (Azure) | `httpx.HTTPStatusError` (with status code inspection), `httpx.ConnectError`, `httpx.TimeoutException` |
+| FAL | `fal_client.FalClientError` or equivalent |
+| HuggingFace | `huggingface_hub.errors.HfHubHTTPError` and subclasses |
+| Linkup | Check SDK for specific exceptions |
+| Docling | `docling.exceptions.*` or generic conversion errors |
+
+### Phase 3: Content policy detection
+
+For OpenAI and Anthropic specifically, inspect the error body for content policy signals:
+
+```python
+except BadRequestError as exc:
+ if "content_policy" in str(exc) or "safety" in str(exc).lower():
+ msg = f"Content was rejected by {provider}'s safety filters: {exc}"
+ raise ContentPolicyError(msg, category=CONTENT) from exc
+ msg = f"Bad request with model {self.inference_model.desc}:\n{exc}"
+ raise LLMCompletionError(msg, category=CONFIGURATION) from exc
+```
+
+Also check `response.choices[0].finish_reason` for `"content_filter"` in OpenAI responses — this is a *successful* API call where the content was filtered in the response, not an exception.
+
+### Phase 4: Retry standardization
+
+Extend the gateway workers' tenacity pattern to all workers that make remote API calls:
+
+- **Retry on:** `TRANSIENT` category errors (rate limit, timeout, 5xx server errors)
+- **Fail fast on:** `CONFIGURATION`, `CONTENT`, `CAPACITY` errors
+- **Configurable:** max retries, backoff multiplier from worker config
+- **Observable:** log each retry with attempt number, wait duration, error category
+
+Consider a shared `@with_inference_retry` decorator or mixin to avoid duplicating retry logic in every worker.
+
+### Phase 5: Agent-friendly error reporting
+
+Add a `to_agent_report()` method on inference errors that returns a structured dict:
+
+```python
+{
+ "error_category": "CONFIGURATION",
+ "provider": "openai",
+ "model": "gpt-4o",
+ "summary": "Authentication failed for OpenAI API",
+ "user_action": "Check that your OPENAI_API_KEY environment variable is set and valid",
+ "retryable": False,
+ "original_error": "Error code: 401 - Invalid API key provided"
+}
+```
+
+This allows MTHDS skills to:
+- Decide whether to retry or escalate to the human
+- Show the user exactly what to fix
+- Log structured error data for debugging
+
+---
+
+## Priority Matrix
+
+| Action | Impact | Effort | Priority |
+|--------|--------|--------|----------|
+| Bring Google, Mistral, FAL, HuggingFace, Docling, Linkup workers to Tier 1 | High — these currently bubble raw SDK errors | Medium | **P0** |
+| Add rate limit + timeout handling to OpenAI/Anthropic workers | High — most common production errors | Low | **P0** |
+| Create `InferenceErrorCategory` enum | High — enables all downstream improvements | Low | **P1** |
+| Add `error_category` to `CogtError` | Medium — structural change | Low | **P1** |
+| Content policy detection | Medium — improves UX for flagged content | Low | **P1** |
+| Retry standardization across workers | Medium — reduces transient failures | Medium | **P2** |
+| `to_agent_report()` for structured error reporting | High for agent workflows | Medium | **P2** |
+| Add `user_action` suggestions to all error paths | High UX impact | Medium | **P2** |
+
+---
+
+## Appendix: Current State by Worker
+
+| Worker | SDK Exceptions | Response Validation | Constructor Validation | Retry Logic | Error Category | Model in Msg |
+|--------|:-:|:-:|:-:|:-:|:-:|:-:|
+| `openai_responses_llm_worker` | 4/4 | yes | no | no | no | yes |
+| `openai_completions_llm_worker` | 4/4 | yes | no | no | no | yes |
+| `anthropic_llm_worker` | 3/3 | yes | yes | no | no | yes |
+| `gateway_extract_worker` | yes | yes | no | yes | no | yes |
+| `gateway_search_worker` | yes | yes | no | yes | no | yes |
+| `openai_completions_img_gen_worker` | 3/4 | yes | no | no | no | yes |
+| `gateway_img_gen_worker` | yes | yes | no | no | no | yes |
+| `openai_img_gen_worker` | **no** | yes | no | no | no | partial |
+| `bedrock_llm_worker` | **no** | minimal | yes | no | no | yes |
+| `mistral_llm_worker` | **no** | yes | yes | no | no | yes |
+| `google_llm_worker` | **no** | delegated | no | no | no | yes |
+| `google_img_gen_worker` | **no** | yes | no | no | no | yes |
+| `azure_img_gen_worker` | raw httpx | yes | no | no | no | partial |
+| `mistral_extract_worker` | **no** | **no** | no | no | no | no |
+| `fal_img_gen_worker` | **no** | **no** | no | no | no | no |
+| `huggingface_img_gen_worker` | **no** | **no** | no | no | no | no |
+| `docling_extract_worker` | **no** | **no** | no | no | no | no |
+| `pypdfium2_worker` | **no** | input only | no | no | no | no |
+| `linkup_extract_worker` | **no** | **no** | no | no | no | no |
+| `linkup_search_worker` | **no** | **no** | no | no | no | no |