Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Refactor/plugins 5 (#1006)
Louis Choquel3a8ffcf96626
diff --git a/.claude/skills/temporal-e2e-validate/SKILL.md b/.claude/skills/temporal-e2e-validate/SKILL.mddeleted file mode 100644index cbb561d78..000000000--- a/.claude/skills/temporal-e2e-validate/SKILL.md+++ /dev/null@@ -1,386 +0,0 @@-----name: temporal-e2e-validate-description: >- Full end-to-end validation of Temporal distributed execution (Phases 2-5 +- v1 routing + v2 queue options / worker-runtime profiles). Two modes:- (1) pytest against a real Temporal server for detailed assertions on all- test cases, and (2) true 3-process setup (server + separate worker process- + submitter) that validates the actual deployment topology including- cross-process serialization, LibraryCrate propagation, deferred hydration,- concurrent isolation, image payload storage, cross-worker graph- tracing with GraphSpec assembly, and cross-worker cost-report assembly- (runner-side usage aggregated into a single submitter cost report, validated- cheaply via the hidden `--dry-run --mock-usage` trigger). Step 8 validates v1 per-activity routing.- Step 9 validates v2 per-queue submitter options (timeouts, retry,- rate-limit), per-handle option overrides, named worker-runtime profiles- selected via `--profile`, and the strict `--task-queue` CLI typo check- with "did you mean?" suggestion. Mode 1 also folds in the error-handling- suite (activity error boundary, workflow error-report full chain, local- parity arm); Mode 2 Tiers 13-16 validate that a worker-side failure — from an- LLM, extract, image-gen, or batched child-workflow activity — carries- a structured `ErrorReport` across the activity -> workflow -> submitter- boundary. Tier 17 validates that a `--temporal --dry-run` run dispatches the- real inference activities and mocks INSIDE them (run mode orthogonal to- backend — no API keys, no storage IO, zero-token suppressed usage).- Use when the user says "validate temporal", "e2e temporal",- "temporal regression", "temporal validation", "3-process test", "full temporal- test", "validate phases", "image temporal", "queue options", "runtime profile",- "validate error handling", "error report", "error propagation",- "cost report", "distributed cost", "mock usage",- or wants comprehensive verification that distributed execution works- end-to-end. Also use proactively after changes to LibraryCrate, deferred- hydration, ClassRegistry scoping, graph tracing, image generation activities,- content storage, queue options resolution, worker-runtime profiles,- usage / cost reporting, the `--costs` / `--mock-usage` flags,- error classification / `ErrorReport` / the Temporal error boundary, or- Temporal workflow code.-allowed-tools:- - Bash(tmux *)- - Bash(curl *)- - Bash(.venv/bin/pytest *)- - Bash(.venv/bin/pipelex *)- - Bash(ls *)- - Bash(which *)- - Bash(open *)- - Bash(cat *)- - Bash(.venv/bin/python *)- - Bash(grep *)- - Bash(sort *)- - Bash(head *)- - Bash(tail *)- - Bash(temporal *)- - Bash(jq *)- - Bash(timeout *)- - Bash(pkill *)- - Bash(sleep *)- - Bash(echo *)- - Bash(seq *)------# Temporal E2E Validation Suite--This skill validates that Pipelex pipelines execute correctly when distributed across-Temporal workers — separate processes that receive serialized work, run pipes, and return-results. It covers the full chain: shipping pipe definitions to the worker (Phase 2),-deserializing dynamic concepts the worker has never seen (Phase 3), isolating concurrent-workflows so they don't corrupt each other (Phase 4), assembling an execution graph-from trace events emitted across workers (Phase 4.5), and verifying that image-heavy-pipelines (image generation, image-to-LLM flow) don't blow up Temporal's payload-limits by ensuring images are stored at the activity level.--## Important: surface results immediately--After each command completes, **immediately tell the user** the outcome in plain text —-PASS/FAIL, what it means, output paths. Do NOT silently move on to the next command.-The user sees collapsed tool outputs by default and relies on your text messages.--Use multi-line formatting with clean indentation — never cram everything onto one line:--```-Tier 1 PASS- The worker received the pipe definitions and executed a 2-step sequence correctly.- Output: results/native_text_sequence_output_01/- Graph: results/native_text_sequence_output_01/reactflow.html-```--For failures, include the error message and what it means:--```-Tier 2 FAIL- KajsonDecoderError: Class 'Greeting' not found- The worker tried to deserialize WorkingMemory before registering dynamic concepts.- Check worker output: tmux capture-pane -t temporal-worker-router -p -S -200- (or tmux capture-pane -t temporal-worker-runner / temporal-worker depending on setup)-```-------## Timeouts policy — REQUIRED for every command--A hung Temporal test will lock the session and consume the agent's wall time until the-user interrupts. Workers, workflow polls, and gRPC retries will happily wait forever.-**Every** `.venv/bin/pytest` and `.venv/bin/pipelex run bundle` invocation in this skill-runs under a hard shell `timeout` and, for pytest, under `--timeout=N` per test as well.-`pytest --timeout=N` alone is not sufficient — it caps a single test, not fixture-setup/teardown, so a broken session-scoped fixture can still hang forever without the-shell cap.--**Defaults to use (apply unless a step explicitly overrides):**--| Command kind | Shell cap | Per-test cap |-|---|---|---|-| Single dry-run pipeline (`pipelex run bundle ... --dry-run --mock-inputs`) | `timeout 120` | — |-| Two concurrent dry-run pipelines (isolation pairs) | `timeout 180` | — |-| Live pipeline (`pipelex run bundle ... --temporal` without `--dry-run`) | `timeout 600` | — |-| Repro/diagnostic script (`.venv/bin/python <path-to-script>.py`) | `timeout 120` | — |-| Pytest folder (dry, `library_crate/` or single module) | `timeout 300` | `--timeout=90` |-| Pytest single test class/case (dry) | `timeout 180` | `--timeout=60` |-| Pytest folder (live, real LLM/img-gen) | `timeout 900` | `--timeout=300` |--**Form to use everywhere:**--```bash-timeout 300 .venv/bin/pytest -x -v tests/integration/pipelex/temporal/library_crate/ \- -m temporal --temporal-server local --timeout=90 2>&1 | tail -80-```--```bash-timeout 120 .venv/bin/pipelex run bundle <bundle> --pipe <pipe> \- --temporal --dry-run --mock-inputs --no-logo --graph 2>&1 | tail -15-echo "EXIT=$?"-```--**If a command times out (exit 124 from `timeout`) or behaves as if hung, the response is-ALWAYS the same:**--1. Capture worker output: `tmux capture-pane -t temporal-worker-router -p -S -200`- and `tmux capture-pane -t temporal-worker-runner -p -S -200` (or- `temporal-worker` for the single-worker setup). Quote the actual error.-2. Kill zombies before retrying: `pkill -f "pipelex.temporal.worker_cli"`,- `pkill -f "pytest.*temporal"`. Stale workers from a previous timed-out run- will swallow new dispatches and make every subsequent step look hung.-3. Diagnose root cause from the captured output. Do **not** raise the timeout- and rerun blind — that wastes another minute and produces the same hang. The- "Interpreting failures" table at the bottom of this skill maps common error- strings to causes.-------## Prerequisites--```bash-which tmux && echo "ok" || echo "MISSING: brew install tmux"-which temporal && echo "ok" || echo "MISSING: brew install temporal"-.venv/bin/python -c "import temporalio; print(f'temporalio {temporalio.__version__}')"-```-------## Mode 1: Automated Test Suite (pytest)--Runs integration tests against a real Temporal server (localhost:7233), but the worker-runs in-process — no separate worker needed. This is the fast path for catching-regressions.--### Step 1: Ensure the Temporal dev server is running--```bash-curl -s http://localhost:8233 > /dev/null 2>&1 && echo "running" || echo "not running"-```--If not running:--```bash-tmux new-session -d -s temporal-server 'temporal server start-dev'-sleep 3-curl -s http://localhost:8233 > /dev/null 2>&1 && echo "running" || echo "FAILED"-```--Do NOT start if already running — it will fail with a bind error.--### Step 2: Run the tests--The conftest auto-registers the Pipelex custom search attributes listed in-`pipelex.temporal.config_temporal.BUILTIN_SEARCH_ATTRIBUTES` on the first-session that hits a fresh dev server. No manual setup needed.--**Dry mode (fast, no LLM costs):**--```bash-timeout 300 .venv/bin/pytest -v tests/integration/pipelex/temporal/library_crate/ \- -m temporal --temporal-server local --timeout=90 2>&1 | tail -80-```--**Live mode (real LLM calls):**--```bash-timeout 900 .venv/bin/pytest -v tests/integration/pipelex/temporal/library_crate/ \- -m temporal --temporal-server local --pipe-run-mode live --timeout=300 2>&1 | tail -80-```--Note on `-x`: omitted here so we see the full failure surface, not just the first hang.-If you want the classic stop-on-first-failure behavior, add `-x` back — but expect to-diagnose root cause from the first failure before retrying.--### Step 2b: Run the error-handling tests--These verify that a Pipelex failure crosses the Temporal boundary carrying a-structured `ErrorReport` (`error_category` / `error_domain` / `retryable` /-`model` / `provider` / `user_action`), with local/Temporal parity. They live-*outside* `library_crate/`, so Step 2 does not pick them up. The LLM call is-mocked in every one of these tests — no API keys, no inference cost.--```bash-# Error-handling tests — Temporal boundary (real server, in-process worker, mocked LLM)-timeout 300 .venv/bin/pytest -v \- tests/integration/pipelex/temporal/test_activity_error_boundary.py \- tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py \- tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py \- -m temporal --temporal-server local --timeout=90 2>&1 | tail -60-```--```bash-# Error-handling tests — local parity arm (no Temporal server, mocked LLM)-timeout 180 .venv/bin/pytest -v \- tests/integration/pipelex/error_handling/ --timeout=60 2>&1 | tail -40-```--The local arm is **not** marked `temporal`, so it must run as its own-invocation — a `-m temporal` filter would deselect it.--### Step 3: Report results--Tell the user what each suite validated and whether it passed. Here is what the suites test:--- **TestWfLibraryCrate** — Can a worker receive a "crate" (a portable bundle of pipe- definitions) and execute a PipeSequence from it? Also tests that submitting *without* a- crate correctly fails with `PipeNotFoundError`.-- **TestWfDeferredHydration** — When a pipe creates a brand-new concept type at runtime- (e.g. `Greeting` with `message` and `language` fields), can the worker deserialize- WorkingMemory that contains instances of that concept, even though the worker has never- seen the class before?-- **TestWfConcurrentConceptIsolation** — Two workflows run simultaneously on the same- worker, both defining a concept called `Result` — but with different fields (`score, label`- vs `value, confidence, is_valid`). Does each workflow get the right version, or do they- clobber each other?-- **TestWfConcurrentPipeIsolation** — Same idea, but for pipes: two workflows both define- a pipe called `shared_step` with different prompts. Does each workflow execute its own- version?-- **TestWfMultiConceptIsolation** — Worst case: two workflows define *two* overlapping- concepts each (`Profile` + `Summary`) with incompatible structures, running across 6- concurrent workflows.-- **TestWfPipeParallel** — A PipeParallel controller dispatches branches as concurrent- child workflows. Do the branches execute independently and merge correctly?-- **TestWfPipeCondition / PipeBatch / PipeCompose / CombinedPipeline** — Controller- coverage for condition routing, fan-out/fan-in, composition, and nested controller- combinations.--For Step 2b (error-handling), here is what the suites validate:--- **TestActivityErrorBoundary** — A real activity failure (LLM and non-LLM) is caught- by the `convert_pipelex_errors` boundary and surfaces a classified `ErrorReport` on- the workflow side, with the correct `non_retryable` flag derived from the error- category.-- **TestWfPipeRunFailurePath** — When the pipe router fails inside the workflow,- delivery still fires with `DeliveryStatus.FAILED` (and `pipe_output` unset), then the- failure re-raises.-- **TestWorkflowErrorReportFullChain** — The `ErrorReport` survives the full- activity → workflow → submitter Temporal round-trip with `error_category` /- `retryable` / `model` / `provider` / `user_action` intact — the recovered report- carries the real failure message, not the generic `"Failed to execute workflow"`.-- **TestErrorReportLocalFullChain** — The local (non-Temporal) parity arm: the same- failing pipe run through `PipeRouter` directly. Both arms assert the same- `ErrorReportParityTestData` constants, so local/Temporal `ErrorReport` parity holds- by construction.--**Known xfails:** Some controller execution tests fail in dry-run mode because-`StuffArtefact` objects can't serialize through Temporal's data converter. The crate-structure tests pass. These xfails will resolve once StuffArtefact serialization is fixed.--The test `test_missing_crate_fails_pipe_resolution` is a **negative test** — it intentionally-submits without a crate. The Temporal warning `Failed activation on workflow` with-`RuntimeError: No current library set` is expected.-------## Mode 2: True 3-Process Validation--The real deployment test: three separate OS processes — Temporal server, worker, and-submitter — with no shared memory. The worker has its own Python runtime, its own-`sys.modules`, its own `ClassRegistry`. This is the only way to catch bugs an in-process-worker (Mode 1) cannot — e.g. the worker failing to deserialize a PipeJob because concept-classes aren't registered, the Kajson decoder bypassing class lookup, or the Temporal-data converter silently dropping fields.--Mode 2 is a battery of tiers and scenarios. It is long, and most invocations need only-part of it — so the procedure lives in **reference files under `references/`**, each-loaded only for the work at hand. Read them as needed:--| Reference file | Covers | Read it when |-|---|---|---|-| `references/mode-2-setup.md` | Steps 1–2: Temporal server + worker processes; run-mode (dry vs live) | **Always — start here.** Every other Mode 2 file assumes the server + worker(s) are already up. |-| `references/mode-2-tiers.md` | Steps 3–7: Tiers 1–16 (sequence, hydration, parallel, image gen/flow, isolation, cross-worker usage + cost report, object gen, extract, CV batch, error propagation), codec, final report | Validating core distributed execution, cost reporting, or running the full suite. |-| `references/routing-battery.md` | Step 8: v1 `activity_queues` per-activity / per-handle routing (Tiers 10a–10c) | Validating that `activity_queues` isolates workers. Opt-in. |-| `references/queue-options-battery.md` | Step 9: v2 per-queue options, per-handle overrides, rate limits, worker-runtime profiles, CLI typo check (Scenarios A–F) | Validating queue options / worker-runtime profiles. Opt-in. |--**Choosing what to run** — match the request, then read only those files:--- "validate temporal" / "full temporal test" / a broad regression check → `mode-2-setup.md`, then `mode-2-tiers.md`; offer the two batteries as opt-in extras.-- "validate temporal error handling" / "error report" / "error propagation" → Mode 1 Step 2b above (the precise pytest assertions) **plus** `mode-2-setup.md` + `mode-2-tiers.md` Step 5f (Tiers 13–16, cross-process).-- "dry run temporal" / "dry-run dispatch" / "leaf mock" → `mode-2-setup.md` + `mode-2-tiers.md` Step 5g (Tier 17: DRY honors the backend — dispatch + mock inside the activity, no keys needed), plus the Mode-1 pytest `tests/integration/pipelex/temporal/tracing/test_dry_run_dispatches_and_mocks.py`.-- "cost report" / "distributed cost" / "mock usage" → `mode-2-setup.md` (split workers), then `mode-2-tiers.md` Step 5b' (Tier 8b). This routing is **scope-aware**: once a request has landed here (via one of those three triggers), read the scope manifest at the top of Step 5b' and run the arms it selects:- - **default** (bare "cost report" / "distributed cost" / "mock usage") → the free, deterministic mock arms only (`--dry-run --mock-usage`); no live spend.- - **explicit spend opt-in** — the canonical token `full` (aliases `thorough`, `every`, `with-spend`) qualifying the cost request (e.g. "cost reporting full", "cost report — every arm") → **every** cost arm, real spend authorized: mock primary + cross-child + CSV cross-check + the `--no-costs` negative gate + live LLM + live img-gen + live extract. Run them all and report PASS/FAIL per arm. Do **not** treat bare "live" or "all" as a spend opt-in — they're too easily incidental ("live" also collides with the default mock arm, which already runs in LIVE mode); if that's the only signal, confirm with the user before spending real money.-- "queue options" / "runtime profile" / "routing" → `mode-2-setup.md`, then `routing-battery.md` and/or `queue-options-battery.md`.-- Anything image-related → `mode-2-setup.md` + `mode-2-tiers.md` Tiers 4–5, run **live** — dry-run cannot surface payload-size bugs.--The **Timeouts policy** and the **"surface results immediately"** rule above apply to-every command in every reference file. The **Interpreting failures** table and **Bundle-reference** below cover all of Mode 2 — keep them in mind while running any reference file.-------## Cleanup--Propose these to the user — do NOT run them automatically:--- Kill tmux sessions: `tmux kill-session -t temporal-worker-router` / `tmux kill-session -t temporal-worker-runner` (or `tmux kill-session -t temporal-worker` if you used the single full worker) / `tmux kill-session -t temporal-server`. If any of Tiers 13–16 ran, also kill `temporal-worker-err` and confirm `.env` is clean (`grep tier-err .env` returns nothing). A blanket `pkill -f "pipelex.temporal.worker_cli"` clears any stray worker process.-- Clean results directory: `rm -rf results/`-- Clean trace files: `rm -rf .pipelex/traces/`-- Remove temporary override if still present: `.venv/bin/python -c "from pathlib import Path; Path('.pipelex/pipelex_temporary_override.toml').unlink(missing_ok=True)"`--Leave the server running if the user plans to iterate.-------## Interpreting failures--| Error | What it means |-|-------|--------------|-| `PipeNotFoundError` on worker | The LibraryCrate didn't arrive or wasn't loaded — the worker can't find the pipes it's supposed to run |-| `KajsonDecoderError: Class 'X' not found` | The worker tried to deserialize data containing a dynamic concept before registering that concept's class — hydration order bug |-| `RuntimeError: Failed decoding arguments` | Temporal's data converter can't deserialize the PipeJob on the worker — usually a serialization format issue |-| `WorkflowFailureError` wrapping `TemporalError` | The pipe itself failed during execution — read the inner error for the real cause |-| Submitter shows `"Failed to execute workflow ..."` with no inner detail (Tier 13–16) | The `ErrorReport` did not cross the boundary — submitter-side report recovery (`recover_error_report` / `WorkflowExecutionError`) is broken, or the activity bridge didn't pack the report into `ApplicationError.details` |-| Tier 13–16 run exits 0 (success) instead of failing | A healthy worker is still polling `temporal_task_queue` and stole the activity (kill all workers, keep only `temporal-worker-err`); or the worker booted before `.env` was tampered — it captures env at boot, so it must be started *after* the `cat >> .env` step and *before* `.env` is restored |-| `AssertionError: StructuredContent missing field` | Per-workflow ClassRegistry isolation failed — the worker used the wrong concept class (from another workflow's definitions) |-| Submitter hangs indefinitely | The worker crashed during deserialization — check `tmux capture-pane -t temporal-worker-router -p -S -200` (and the runner session, or `temporal-worker` for the single-worker setup) |-| Both concurrent jobs succeed but wrong data | ContextVar leak between workflows — per-workflow scoping is broken, one workflow's class definitions bled into the other |-| No `reactflow.html` generated | GraphSpec assembly failed — either tracing is disabled in `pipelex.toml` or NDJSON events weren't emitted by the worker |-| `PayloadSizeWarning` in worker logs | Image data (base64) is being passed inline through Temporal payloads instead of being stored at the activity level — the fix is to call storage in the image generation activity before returning results |-| `NotImplementedError` during image generation | The image generation or content storage path isn't wired up for the Temporal execution path — activity-level storage is missing |-| Payload too large / `DataConverterError` on image pipes | Raw image bytes exceed Temporal's payload size limit (~2MB default) — images must be stored and referenced by URI, not passed inline |-| `ImageContent` missing or has no `uri` field | The image generation activity returned raw image data instead of a stored `ImageContent` with a storage URI |-| Codec enabled but no storage files | Payloads are below the threshold (1MB default) — expected in dry-run mode where mock data is small. Run live mode for realistic payload sizes |-| `FileNotFoundError` in codec decode | The codec is trying to load a payload from storage that doesn't exist — storage root path may be misconfigured or files were cleaned up mid-run |-------## Bundle reference--**Crate/isolation bundles** — in `tests/integration/pipelex/temporal/library_crate/`:--| Bundle | What it tests | Main pipe |-|--------|--------------|-----------|-| `native_text_sequence.mthds` | Basic crate propagation (native Text, 2 steps) | `native_text_sequence` |-| `dynamic_concept_sequence.mthds` | Deferred hydration (Greeting concept created at runtime) | `dynamic_greeting_sequence` |-| `conflict_concept_alpha.mthds` | Concept isolation — Result(score, label) | `alpha_pipeline` |-| `conflict_concept_beta.mthds` | Concept isolation — Result(value, confidence, is_valid) | `beta_pipeline` |-| `conflict_pipe_alpha.mthds` | Pipe isolation — shared_step with alpha prompt | `pipe_alpha_pipeline` |-| `conflict_pipe_beta.mthds` | Pipe isolation — shared_step with beta prompt | `pipe_beta_pipeline` |-| `multi_concept_alpha.mthds` | Multi-concept isolation — Profile(name, age) + Summary(headline, body) | `multi_alpha_pipeline` |-| `multi_concept_beta.mthds` | Multi-concept isolation — Profile(title, department, level) + Summary(content) | `multi_beta_pipeline` |-| `temporal_parallel.mthds` | PipeParallel — concurrent child workflows (ToneAnalysis + LengthAnalysis) | `temporal_parallel_sequence` |-| `temporal_batch.mthds` | PipeBatch — fan-out to per-item child workflows | `temporal_batch_sequence` |-| `temporal_condition.mthds` | PipeCondition — conditional routing via child workflow | `temporal_condition_sequence` |-| `temporal_compose.mthds` | PipeCompose — operator composition + deferred hydration | `temporal_compose_sequence` |-| `temporal_combined.mthds` | Nested PipeParallel + PipeCondition | `temporal_combined_pipeline` |-| `large_payload_sequence.mthds` | Codec stress test — 3-step verbose sequence accumulating large WorkingMemory | `large_payload_sequence` |-| `cv_batch_screening.mthds` | Deeply-nested controller stack (PipeSequence -> PipeSequence + PipeBatch -> PipeSequence) over PipeExtract + PipeLLM. Inputs JSON: `cv_batch_screening_inputs.json` | `batch_analyze_cvs_for_job_offer` |--**Image payload bundles** — in `tests/integration/pipelex/pipes/pipelines/`:--| Bundle | What it tests | Main pipe |-|--------|--------------|-----------|-| `crazy_image_generation.mthds` | Image generation pipeline — LLM imagines scene + PipeImgGen renders it, custom ImagePrompt concept | `generate_crazy_image` |-| `test_image_out_in.mthds` | Image flow — PipeImgGen generates image, then PipeLLM (vision) describes it | `image_out_in` |diff --git a/.claude/skills/temporal-e2e-validate/references/mode-2-setup.md b/.claude/skills/temporal-e2e-validate/references/mode-2-setup.mddeleted file mode 100644index 5a90ad063..000000000--- a/.claude/skills/temporal-e2e-validate/references/mode-2-setup.md+++ /dev/null@@ -1,154 +0,0 @@-# Temporal E2E — Mode 2 setup: server + worker processes--> Reference file for the **temporal-e2e-validate** skill — Mode 2, Steps 1–2. **Read this first** before any other Mode 2 reference.-> The **Timeouts policy** and the **surface-results-immediately** rule in `SKILL.md` apply to every command here.->-> Mode 2 reference files (read as the work requires):->-> - `mode-2-setup.md` — Steps 1–2: Temporal server + worker processes (this file)-> - `mode-2-tiers.md` — Steps 3–7: Tiers 1–14, graph, isolation, codec, final report-> - `routing-battery.md` — Step 8: v1 `activity_queues` routing battery-> - `queue-options-battery.md` — Step 9: v2 queue options + worker-runtime profiles--## Why Mode 2: True 3-Process Validation--This is the real deployment test. Three separate OS processes — Temporal server, worker,-and submitter — with no shared memory. The worker has its own Python runtime, its own-`sys.modules`, its own ClassRegistry. This is the only way to catch bugs like:--- The worker can't deserialize the PipeJob because concept classes aren't registered yet-- The Kajson decoder bypasses class lookup when `__module__="builtins"`-- The Temporal data converter silently drops fields during encoding/decoding--Each command runs a pipeline through Temporal with `--graph`, which also validates that-the worker emits NDJSON trace events and the submitter assembles them into a GraphSpec-with an interactive ReactFlow HTML visualization.--### Run mode: dry-run vs live--By default, Mode 2 runs in **dry-run** mode (`--dry-run --mock-inputs`) — fast, no LLM-costs, validates serialization and crate propagation. But dry-run produces tiny mock-data, so it **cannot catch payload size issues** (e.g. large image payloads blowing up-Temporal's data converter).--Ask the user which mode they want. If they say "live", simply omit `--dry-run --mock-inputs`-from all commands below. The `pipelex run bundle` CLI has no `--pipe-run-mode` flag — live-is the default when neither `--dry-run` nor `--mock-inputs` is specified (note: pytest in-Mode 1 does accept `--pipe-run-mode live`, but the CLI does not). Live mode makes real LLM and image-generation calls — it costs money and is slower, but it's the only way to validate that-real-sized payloads (especially images) flow correctly through Temporal.--**This matters most for Tiers 4 and 5** (image generation and image flow). In dry-run,-mock images are trivially small, so payload size bugs don't surface. In live mode,-generated images are hundreds of KB to several MB — exactly the size that breaks Temporal-if activity-level storage isn't implemented.--### Step 1: Ensure Temporal server is running--Same as Mode 1 Step 1 in `SKILL.md` — start `temporal server start-dev` in a `temporal-server` tmux session if `curl -s http://localhost:8233` shows it is not already up.--**Search-attribute registration on a fresh dev server.** Worker boot performs a-hard-fail audit (`check_required_search_attributes`) and refuses to start if any-of the Pipelex custom attributes (defined in `BUILTIN_SEARCH_ATTRIBUTES`) is-missing from the namespace. The error message includes the exact-`pipelex setup-temporal-namespace` command to run. For a freshly started-`temporal server start-dev`, register them once (idempotent):--```bash-.venv/bin/pipelex setup-temporal-namespace-```--This wraps the same helper the test conftest uses, so the set stays in sync-with `BUILTIN_SEARCH_ATTRIBUTES` automatically. (If you need to invoke the raw-Temporal CLI for some reason — e.g. an environment where the Pipelex CLI is-unavailable — the command shape is-`temporal operator search-attribute create --namespace <ns> --name <Name> --type Keyword`,-one per attribute name from `BUILTIN_SEARCH_ATTRIBUTES`.)--Mode 1 (pytest) handles this automatically via the test conftest. Mode 2 needs-it done before worker startup.--### Step 2: Start the worker processes--The workers should NOT have test bundles in their PIPELEXPATH — the whole point is-that bundles arrive via the LibraryCrate in the PipeJob.--**Two scoped workers (cross-process regression setup — recommended).**-Splits responsibilities so workflows and activities run in different processes:-- `router` worker registers all workflows (`WfPipeRouter`, `WfPipeRun`, …),- `disable_all_activities = true`.-- `runner` worker registers all activities (`act_deliver`, `act_llm_*`,- `act_assemble_tracing`, `act_flush_trace_events`, …), `disable_all_workflows = true`.--This forces every activity to be picked up by a *different* Python process than the-workflow that scheduled it. The runner process never executes `WfPipeRouter`, so it-never loads the LibraryCrate and its global `ClassRegistry` stays cold for any-dynamic concept defined in the bundle.--> ⚠️ **Important — what this setup does NOT reproduce on its own.**-> Plain `pipelex run bundle` (Tiers 1–3 below) does NOT trigger the runner-side-> registry decode bug, because:-> - `WfPipeRouter` dehydrates `pipe_output` via `prepare_for_temporal()` before-> returning, so workflow-level transit carries raw dicts (no class lookup).-> - `WfPipeRun` rehydrates back on the *router* (same process that loaded the crate).-> - Activities the runner actually executes in dry-run (`act_assemble_tracing`,-> `act_flush_trace_events`) operate on raw event records — no dynamic class needed.-> - `act_deliver` is **only scheduled when `delivery_assignment is not None`**-> (`wf_pipe_run.py:79`), and `pipelex run bundle` does not pass one.->-> To deterministically force a hydrated `pipe_output` across the process boundary,-> run **Tier 2b** below (mirrors the cloud / `start_pipeline` + webhook path).--```bash-tmux has-session -t temporal-worker-router 2>/dev/null && tmux kill-session -t temporal-worker-router-tmux has-session -t temporal-worker-runner 2>/dev/null && tmux kill-session -t temporal-worker-runner-tmux new-session -d -c "$PWD" -s temporal-worker-router \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope router'-tmux new-session -d -c "$PWD" -s temporal-worker-runner \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner'-# Bounded wait: fail fast with captured logs if a worker can't start within 30s-# (boot is normally <5s — anything past that means a stuck config validator,-# missing search attributes, or the worker crashed early).-for session in temporal-worker-router temporal-worker-runner; do- for attempt in $(seq 1 30); do- if tmux capture-pane -t "$session" -p 2>/dev/null | grep -q "Temporal Worker started"; then break; fi- if [ "$attempt" -eq 30 ]; then- echo "TIMEOUT: $session did not report 'Temporal Worker started' within 30s — last 50 lines:"- tmux capture-pane -t "$session" -p -S -50- exit 1- fi- sleep 1- done-done-tmux capture-pane -t temporal-worker-router -p -S -30 | grep -E "scope|started for|search-attribute" | head -5-tmux capture-pane -t temporal-worker-runner -p -S -30 | grep -E "scope|started for|search-attribute" | head -5-```--Look for `Temporal Worker started for 'temporal_task_queue'` in each session, plus-`profile='default' scope='router'` and `'runner'` respectively. If a worker fails with-`SearchAttributeRegistrationError`, register the missing attributes per Step 1 and-restart the worker — do **not** bump the timeout and retry blind.--**Alternative — single full worker** (simpler, but masks distributed-execution bugs;-use only when you don't need the regression coverage):--```bash-tmux has-session -t temporal-worker 2>/dev/null && tmux kill-session -t temporal-worker-tmux new-session -d -c "$PWD" -s temporal-worker \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed'-# Bounded wait: fail fast with captured logs if the worker can't start within 30s-for attempt in $(seq 1 30); do- if tmux capture-pane -t temporal-worker -p 2>/dev/null | grep -q "Temporal Worker started"; then break; fi- if [ "$attempt" -eq 30 ]; then- echo "TIMEOUT: temporal-worker did not report 'Temporal Worker started' within 30s — last 50 lines:"- tmux capture-pane -t temporal-worker -p -S -50- exit 1- fi- sleep 1-done-tmux capture-pane -t temporal-worker -p -S -30-```--When using two scoped workers, replace any later capture commands like-`tmux capture-pane -t temporal-worker ...` with the appropriate session name-(`temporal-worker-router` or `temporal-worker-runner`).diff --git a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md b/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.mddeleted file mode 100644index 0b119a1b6..000000000--- a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md+++ /dev/null@@ -1,1392 +0,0 @@-# Temporal E2E — Mode 2 tiers: Steps 3–7--> Reference file for the **temporal-e2e-validate** skill — Mode 2, Steps 3–7.-> **Run `mode-2-setup.md` first** (Steps 1–2: Temporal server + worker processes) — every tier here assumes the server and worker(s) are already up.-> The **Timeouts policy** and the **surface-results-immediately** rule in `SKILL.md` apply to every command here.-> Sibling reference files: `routing-battery.md` (Step 8 — v1 routing), `queue-options-battery.md` (Step 9 — v2 queue options / worker-runtime profiles).--## Contents--- **Step 3** — Sequential tests: Tier 1 (sequence), Tier 2 (hydration), Tier 2b (cross-process registry), Tier 2c (validate sweep stays in-process), Tier 2d (dry-run+validate as one in-memory activity), Tier 3 (parallel), Tier 4 (image generation), Tier 5 (image flow)-- **Step 4** — Verify graph output-- **Step 5** — Concurrent isolation tests (concept / pipe / multi-concept)-- **Step 5b** — Tier 8: cross-worker usage emission; Tier 8b: cross-worker cost report assembly (`--dry-run --mock-usage`, free)-- **Step 5c** — Tier 9: object generation cross-process-- **Step 5d** — Tier 11: `make_extract_pages` two-activity cross-process-- **Step 5e** — Tier 12: deeply-nested controller stack (CV batch screening)-- **Step 5f** — Tiers 13–16: error propagation across the activity → workflow → submitter boundary (LLM, extract, image-gen, and batched child-workflow failures)-- **Step 6** — StoragePayloadCodec tests: Tier 6 (codec transparency), Tier 7 (large payload)-- **Step 7** — Final report (master results table)-------### Step 3: Sequential tests — run one at a time, report after each--Do NOT clean previous results automatically — the user may want to compare runs.--**Tier 1 — Can the worker execute a simple pipe sequence?**--This sends a 2-step PipeSequence (step_one → step_two) to a worker that has never seen-these pipes. The worker must unpack the LibraryCrate, register the pipes, and execute-them in order. This is the most basic "does Temporal work at all" test.--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --dry-run --mock-inputs --no-logo --graph-```--After this completes, tell the user: PASS/FAIL, output dir, graph file path.--**Tier 2 — Can the worker handle concepts it has never seen? (in-process hydration)**--This sends a pipeline that defines a custom concept (`Greeting` with `message` and-`language` fields) at runtime. `WfPipeRouter` must register this dynamic class-into the per-workflow registry before hydrating the WorkingMemory. If the hydration-order is wrong **on the router**, this fails with-`KajsonDecoderError: Class 'Greeting' not found`.--> Note: in dry-run with `pipelex run bundle` (no `delivery_assignment`), this tier-> only exercises hydration **on the router process**, not cross-process. It will-> pass even if the runner's registry is cold — see Tier 2b for the deterministic-> cross-process repro.--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \- --pipe dynamic_greeting_sequence \- --temporal --dry-run --mock-inputs --no-logo --graph-```--After this completes, tell the user: PASS/FAIL, output dir, graph file path.--**Tier 2b — Cross-process registry regression (deterministic, dry-run).**--Same bundle as Tier 2, but submitted with a `DeliveryAssignment` attached. This-mirrors the cloud / `start_pipeline` + webhook path: `wf_pipe_run.py:79` then-schedules `act_deliver` on the runner with `pipe_output` carrying a hydrated-`WorkingMemory`. Temporal's data converter on the **runner process** must decode-the dynamic concept class — which is the exact path the `wf_pipe_router.py:71-80`-in-process propagation hack does NOT cover.--If the registry propagation across processes is broken, this fails every time with:--```-KajsonDecoderError: Class '<bundle>__<DynamicConcept>' not found- in module 'builtins' or global registry-ApplicationError: Failed decoding arguments- → temporalio/worker/_activity.py:566 (data_converter.decode_wrapper)-```--The error surfaces on the **runner** tmux session, raised by the activity worker-*before* the activity body runs.--```bash-timeout 120 .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py-```--To reproduce against a different bundle:--```bash-timeout 120 .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py \- --bundle <path/to/bundle.mthds> --pipe <pipe_code>-```--After this completes, tell the user:-- PASS/FAIL.-- If FAIL: capture the runner log with- `tmux capture-pane -t temporal-worker-runner -p -S -300 | tail -120`- and quote the `KajsonDecoderError` and `ApplicationError: Failed decoding arguments`- lines verbatim, plus the activity name (`act_deliver`).--**Tier 2c — Validate sweep stays in-process under a Temporal-enabled boot (submitter-side leak guard).**--This is **not** a `run` — it's a `validate`. It guards the production bug where the `/validate`-dry-run sweep leaked nested controller sub-pipes to Temporal: a standalone `PipeBatch` swept directly-fans out over a mock list and dispatches each branch through `get_pipe_router()`, which under a-Temporal-enabled hub is the `TemporalPipeRouter`. Those concurrent same-id top-level dispatches raised-`WorkflowAlreadyStartedError` → the API returned **HTTP 422** with `Failed to execute workflow-WfPipeRouter`. The fix scopes an in-process router for the whole sweep; the contract here is that-validation **never dispatches to Temporal**, even when booted Temporal-enabled.--The `--temporal` flag (parity with `pipelex run`) flips the boot's hub default to the Temporal router-with no `pipelex_temporary_override.toml` juggling. With the fix, the sweep still stays in-process —-which is exactly what this asserts. `temporal_batch.mthds` declares a standalone `type = "PipeBatch"`-pipe (`batch_temporal_describe_topics`), so its sweep fans out — the exact shape that 422'd.--```bash-# Server + worker up (Mode 2). The boot connects Temporal-enabled; with the fix the sweep stays-# in-process, so the worker must receive NO workflow for this validate run.-# Capture pipelex's OWN exit code (load-bearing for GREEN/RED). Don't pipe the command into-# `tail` and read `$?` — that's tail's exit; and `${PIPESTATUS[0]}` is bash-only (blank in zsh).-# Redirect to a file, read `$?` with no pipe in between, then tail the log. Works in bash AND zsh.-timeout 120 .venv/bin/pipelex validate bundle \- tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \- --temporal > /tmp/tier2c-validate.log 2>&1; echo "EXIT=$?"-tail -20 /tmp/tier2c-validate.log-```--GREEN: `EXIT=0` and `Successfully validated bundle ...`. **Strong check** (the point of the scenario):-the worker stayed idle — it received no top-level dispatch. Capture both worker sessions and confirm-no new `WfPipeRouter` / `WfPipeRun` execution appeared for this run (split workers are the Step 2-default; fall back to `temporal-worker` for the single-worker setup):--```bash-tmux capture-pane -t temporal-worker-router -p -S -200 | grep -i WfPipeRouter # expect: nothing new-tmux capture-pane -t temporal-worker-runner -p -S -200 | grep -i WfPipeRun # expect: nothing new-```--RED (prove the scenario bites) — the fix is committed, so neutralize it in the working tree. Surgical:-in `pipelex/pipeline/bundle_validator.py`, inside `validate_pipes`, drop the `with-scoped_pipe_router(self._pipe_router):` wrapper so the sweep loop runs unscoped (de-indent the loop-one level). Re-run the GREEN command — expect `EXIT=1`, `Dry run failed with 1 unexpected pipe-failure(s): 'temporal_batch_test.batch_temporal_describe_topics': ... Failed to execute workflow-WfPipeRouter`, and the worker session now shows `WfPipeRouter` activity (the leak). **Restore the fix-immediately:** `git checkout -- pipelex/pipeline/bundle_validator.py`.--After this completes, tell the user:-- PASS (GREEN exits 0 **and** worker idle — no dispatch) / FAIL.-- Caveats worth stating: only a *standalone* `PipeBatch`/`PipeParallel` swept directly turns the leak- fatal (concurrent same-id collision); a batch reached only as a sequence sub-pipe round-trips- Temporal but passes (a false pass), so "validate exited 0" alone is **not** sufficient — the- worker-idle check is the real assertion. Worker up vs down are **both** RED (collision- `WorkflowAlreadyStartedError` vs no-worker `RPCError`, same `except` branch); keep the worker up to- match production.-- The cheaper CI-automated companion to this scenario is the Mode-1 pytest- `tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py` (real- `TemporalPipeRouter` as hub default, spies `WorkflowExecutor.execute_workflow`, asserts never- called). This Mode-2 scenario is the deployment-faithful demonstration across the real API↔worker- process boundary.--**Tier 2d — Dry-run + validation runs as ONE in-process, in-memory activity.**--Sibling to Tier 2c: where 2c proves the *direct* `/validate` sweep doesn't leak to Temporal, 2d-proves the *Temporal-dispatched* path runs the whole sweep **+** graph dry-run inside a single-activity (`act_dry_validate`, dispatched via the one-step wrapper workflow `wf_dry_validate`), in-memory, returning `{status map, GraphSpec}` in one round-trip. The submitter script below is the-same dispatch shape the Temporal-enabled API `/validate` uses.--```bash-# Server + split workers up (Mode 2). Dispatch the wrapper-workflow→activity over the parallel-# controller bundle (interesting graph: sequence → parallel fan-out → 2 branches → summary).-timeout 120 .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/submit_dry_validate.py \- --bundle tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds \- --pipe temporal_parallel_test.temporal_parallel_sequence > /tmp/tier2d-validate.log 2>&1; echo "EXIT=$?"-tail -20 /tmp/tier2d-validate.log-```--GREEN: `EXIT=0` · the STATUS MAP lists every pipe in the bundle as `SUCCESS` · `GRAPH:` line shows a-non-empty GraphSpec (nodes for the whole controller topology) and the JSON landed at-`/tmp/tier2d-graph-spec.json`.--**Strong check (the point):** during the run the worker ran the wrapper workflow + exactly one-`act_dry_validate` and **nothing else** — NO child `WfPipeRouter`/`WfPipeRun`, NO `act_llm_gen_*`,-NO `act_assemble_tracing`/`act_flush_trace_events`. Capture both worker sessions and grep — expect-only `wf_dry_validate` / `act_dry_validate` for this run (mirrors Tier 2c's worker-idle check, but-here the one activity is expected; what must be absent is everything *nested*):--```bash-tmux capture-pane -t temporal-worker-router -p -S -200 | grep -iE "WfPipeRouter|WfPipeRun|act_llm_gen|act_assemble_tracing|act_flush_trace_events" # expect: nothing new-tmux capture-pane -t temporal-worker-runner -p -S -200 | grep -iE "WfPipeRouter|WfPipeRun|act_llm_gen|act_assemble_tracing|act_flush_trace_events" # expect: nothing new-```--**In-memory tracing:** no new NDJSON partition appears under `.pipelex/traces/` for the activity's-internal graph dry-run, and no DynamoDB write — the `GraphSpec` rode back on the activity result,-assembled from the in-memory log. **No usage/cost:** no cost table, no `usage_report` events.--**Best-effort graph sub-case:** dispatch a bundle with no `main_pipe` and no `--pipe` (so the graph-arm has nothing to target):--```bash-timeout 120 .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/submit_dry_validate.py \- --bundle tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \- --no-pipe > /tmp/tier2d-nograph.log 2>&1; echo "EXIT=$?"-tail -5 /tmp/tier2d-nograph.log-```--GREEN: `EXIT=0`, full STATUS MAP, and `GRAPH: None (best-effort — validation still succeeded)`.--**Concurrency:** launch two submitter invocations in parallel (e.g. `temporal_parallel.mthds` and-`temporal_batch.mthds --no-pipe` with `--graph-out` pointing at distinct files) and confirm both-exit 0 with distinct `graph_id`s / status maps — no shared or merged trace events.--RED (prove it bites) — the fix is committed, so neutralize it in the working tree. Either arm:--- **Drop the content-generator scope:** in `pipelex/pipe_run/dry_run_in_process.py`- (`dry_run_pipe_in_process`), remove `scoped_content_generator(content_generator)` from the- `with` line. The DRY mock lives at the cogt leaf, so the leaf resolves the hub- `ContentGeneratorInWorkflow` and dies with `_NotInWorkflowEventLoopError` / the strong check shows- dispatch. (The CI-cheap Mode-1 companion is- `test_dry_run_graph_in_process.py::test_leaf_level_mock_stays_in_process`.)-- **Drop the shared event log:** in the same function, remove `scoped_event_log(event_log)` from- the `with` line — the two-instance regression: emit and assemble no longer share the instance, the- assembly finds zero events, and `dry_run_pipe_in_process` raises `In-process dry-run of pipe '...'- did not produce a graph spec`. In the activity that PipelexError degrades per the parity- best-effort catch, so the submitter still exits 0 — the RED signal is `GRAPH: None` where the- GREEN run produced a real graph (nodes/edges + the JSON file). The loud, CI-cheap guard for the- same regression is the Mode-1 companion `test_dry_run_graph_in_process.py` (it calls- `dry_run_pipe_in_process` directly and asserts a non-empty graph).--**Restore immediately:** `git checkout -- pipelex/pipe_run/dry_run_in_process.py`.--**API arm (after Phase 4 — the real `/validate` route in `../pipelex-api`).** With the server +-split workers up, exercise the actual route code performing the real dispatch (the sibling repo's-venv must have this pipelex installed, e.g. `cd ../pipelex-api && uv pip install -e ../<this-worktree>`):--```bash-# From ../pipelex-api: boot PYTEST-mode with temporal_enabled=True, point the dispatch at the-# e2e queue, and POST a real bundle through TestClient (full route + handler chain, real worker).-cd ../pipelex-api && timeout 120 .venv/bin/python - <<'PY'-import os-os.environ.setdefault("COMPLETION_CALLBACK_SECRET", "tier2d-placeholder")-from fastapi import FastAPI-from fastapi.testclient import TestClient-from pipelex.config import get_config-from pipelex.pipelex import Pipelex-from pipelex.system.runtime import IntegrationMode--BUNDLE = open("../pipelex/tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds").read() # or any main_pipe bundle-Pipelex.make(IntegrationMode.PYTEST, temporal_enabled=True)-get_config().temporal.worker_config.default_task_queue = "temporal_task_queue"-from api.exception_handlers import register_exception_handlers-from api.routes import router as api_router-app = FastAPI(); app.include_router(api_router, prefix="/api/v1"); register_exception_handlers(app)-resp = TestClient(app).post("/api/v1/validate", json={"mthds_contents": [BUNDLE]})-print(resp.status_code, resp.json().get("graph_spec") is not None)-Pipelex.teardown_if_needed()-PY-```--GREEN: `200 True` (one round-trip returned the envelope with a worker-assembled `graph_spec`), the-worker shows only `wf_dry_validate` + one `act_dry_validate` (same strong check as above), and a-strict-mode signature bundle POSTed the same way returns **422** with-`error_type=ValidateBundleError`, `error_domain=input` — identical to the direct path. The failed-workflow must appear exactly ONCE in `temporal workflow list` (the dispatch pins-`RetryPolicy(maximum_attempts=1)` on the wrapper workflow — a deterministic validation failure-must not re-run). The bundle's graph is best-effort: a bundle with no `main_pipe` 422s on the-API-side precondition; the worker-side `graph_spec=None` arm is covered by the route's unit tests-(`tests/unit/test_validate_temporal_dispatch.py`).--After this completes, tell the user:--- PASS (GREEN exits 0 · status map all SUCCESS · non-empty GraphSpec · worker shows ONLY the- wrapper workflow + one `act_dry_validate` · no NDJSON/DDB write · best-effort sub-case returns- `GRAPH: None` with exit 0) / FAIL.-- The cheaper CI-automated companion is the Mode-1 pytest- `tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py` (real worker against- the in-process server; zero-nested-dispatch asserted on the Temporal history; in-memory tracing- with `make_event_log` forbidden; best-effort graph; structured `ErrorReport` on validation- failure; concurrent isolation).--**Tier 3 — Do parallel branches execute as separate child workflows?**--This sends a PipeParallel controller that fans out into two branches (tone analysis and-length analysis), each running as its own child workflow on the worker. The branches-execute concurrently and their results merge back. The `--graph` flag here is especially-interesting: it shows cross-worker execution with child workflow branches in the-ReactFlow visualization.--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds \- --pipe temporal_parallel_sequence \- --temporal --dry-run --mock-inputs --no-logo --graph-```--After this completes, tell the user: PASS/FAIL, output dir, graph file path.--**Tier 4 — Can the worker handle image generation pipelines?**--**Important — split workers REQUIRED for this tier.** Run Tier 4 only when the-two scoped workers from Step 2 (`mode-2-setup.md`) are alive (`temporal-worker-router` and-`temporal-worker-runner`). On a single full worker, image-gen routing accidents-are silently masked: every activity ends up on one process so a stray-`task_queue=` kwarg pointing at the wrong queue would still execute correctly.-Split workers are the only way to surface that class of bug as a hang or-wrong-pool execution.--**Important:** This tier only catches payload size bugs in **live mode**. In dry-run,-mock images are tiny and will pass even without activity-level storage. If the user-wants to validate image payload handling, they must run live.--This is a critical payload-size test. Image generation produces large binary data-(base64-encoded images) that must be stored at the activity level and passed as-storage URIs through Temporal — not inline in the workflow payload. If storage is-missing, this will either fail with a Temporal payload size error or trigger-`PayloadSizeWarning` in the worker logs.--The `generate_crazy_image` bundle runs a 2-step PipeSequence: an LLM imagines a-scene description, then PipeImgGen renders it as an image. This exercises the full-image generation path through Temporal, including the custom `ImagePrompt` concept-(which refines `Text`).--Dry-run:--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/pipes/pipelines/crazy_image_generation.mthds \- --pipe generate_crazy_image \- --temporal --dry-run --mock-inputs --no-logo --graph-```--Live (real image generation — required to catch payload size bugs):--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/pipes/pipelines/crazy_image_generation.mthds \- --pipe generate_crazy_image \- --temporal --no-logo --graph-```--After this completes, tell the user: PASS/FAIL, output dir, graph file path.-Also check worker logs for `PayloadSizeWarning`:--```bash-tmux capture-pane -t temporal-worker-router -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings (router)"-tmux capture-pane -t temporal-worker-runner -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings (runner)"-```--> For per-activity routing validation that proves image-gen lands on a-> dedicated worker pool (via `activity_queues` override), see **Step 8 —-> Routing validation battery** in `routing-battery.md`.--**Tier 5 — Can generated images flow between pipes in a sequence?**--**Important — split workers REQUIRED for this tier.** Same as Tier 4 — the-mis-routing of image-gen across worker pools is only observable when router and-runner run as separate processes. On a single full worker, the bug surfaces as-"works on my machine" but breaks in production.--**Important:** Like Tier 4, this only catches real payload bugs in **live mode**.--This is the "image out → image in" test. A PipeSequence first generates an image-via PipeImgGen, then passes that image as input to a PipeLLM (vision model) that-describes it. This exercises:-- Image generation storage at the activity level-- Image content serialization through Temporal's data converter-- Image content deserialization on the worker for the next pipe step-- Vision model receiving an image reference (URI) rather than inline base64--If activity-level storage is not implemented, this will fail because the raw image-bytes exceed Temporal's payload limit, or because the worker can't deserialize the-`ImageContent` object from the previous step.--Dry-run:--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/pipes/pipelines/test_image_out_in.mthds \- --pipe image_out_in \- --temporal --dry-run --mock-inputs --no-logo --graph-```--Live (real image generation + vision — required to catch payload size bugs):--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/pipes/pipelines/test_image_out_in.mthds \- --pipe image_out_in \- --temporal --no-logo --graph-```--After this completes, tell the user: PASS/FAIL, output dir, graph file path.-Also check for payload warnings on both workers:--```bash-tmux capture-pane -t temporal-worker-router -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings (router)"-tmux capture-pane -t temporal-worker-runner -p -S -50 | grep -i "payload\|warning\|size" || echo "No payload warnings (runner)"-```--> For per-activity routing validation across the image-gen + LLM-text-> activities exercised here, see **Step 8 — Routing validation battery**-> in `routing-battery.md`.--If any tier hangs for more than 30 seconds, check worker output. With the-recommended split workers (Step 2, `mode-2-setup.md`), capture both sessions; for the-alternative single full worker setup, capture `temporal-worker` instead:--```bash-# Split workers (default setup from Step 2, mode-2-setup.md)-tmux capture-pane -t temporal-worker-router -p -S -100-tmux capture-pane -t temporal-worker-runner -p -S -100--# Single full worker (alternative setup)-tmux capture-pane -t temporal-worker -p -S -100-```--### Step 4: Verify graph output--Check that the `--graph` flag produced ReactFlow HTML files:--```bash-ls results/*/reactflow.html 2>/dev/null-```--Tell the user where the files are and propose opening the most interesting one (the-PipeParallel graph shows cross-worker execution):--```bash-open results/temporal_parallel_sequence_output_01/reactflow.html-```--If no `reactflow.html` exists, GraphSpec assembly failed — check that tracing is enabled-in `pipelex.toml` (`[pipelex.tracing_config]` with `is_enabled = true`).--### Step 5: Concurrent isolation tests — can conflicting workflows coexist?--These are the most important safety tests. They submit two workflows simultaneously to-the same worker, where both workflows define identically-named concepts or pipes but-with different structures. If per-workflow isolation is broken, one workflow will use-the other's class definitions and either crash or silently produce wrong data.--Each pair runs with `--graph`. The graph output from backgrounded jobs goes to separate-`results/<pipe_code>_output_NN/` dirs.--**Test A — Conflicting concept names:**-Two workflows both define a concept called `Result`, but alpha's has `score` + `label`-while beta's has `value` + `confidence` + `is_valid`. If isolation fails, one workflow-will try to populate the wrong fields.--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/conflict_concept_alpha.mthds \- --pipe alpha_pipeline \- --temporal --dry-run --mock-inputs --no-logo --graph &-PID_ALPHA=$!--timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/conflict_concept_beta.mthds \- --pipe beta_pipeline \- --temporal --dry-run --mock-inputs --no-logo --graph &-PID_BETA=$!--wait $PID_ALPHA && echo "Alpha: PASS" || echo "Alpha: FAIL"-wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"-```--After this, tell the user both results.--**Test B — Conflicting pipe names:**-Two workflows both define a pipe called `shared_step` but with different prompts.-Does each workflow execute its own version?--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/conflict_pipe_alpha.mthds \- --pipe pipe_alpha_pipeline \- --temporal --dry-run --mock-inputs --no-logo --graph &-PID_ALPHA=$!--timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/conflict_pipe_beta.mthds \- --pipe pipe_beta_pipeline \- --temporal --dry-run --mock-inputs --no-logo --graph &-PID_BETA=$!--wait $PID_ALPHA && echo "Alpha: PASS" || echo "Alpha: FAIL"-wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"-```--After this, tell the user both results.--**Test C — Worst case: multiple conflicting concepts at high concurrency:**-Two workflows define both `Profile` AND `Summary` with incompatible field structures.-This is the hardest isolation test — if ContextVar scoping leaks between workflows,-this is where it shows up.--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/multi_concept_alpha.mthds \- --pipe multi_alpha_pipeline \- --temporal --dry-run --mock-inputs --no-logo --graph &-PID_ALPHA=$!--timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/multi_concept_beta.mthds \- --pipe multi_beta_pipeline \- --temporal --dry-run --mock-inputs --no-logo --graph &-PID_BETA=$!--wait $PID_ALPHA && echo "Alpha: PASS" || echo "Alpha: FAIL"-wait $PID_BETA && echo "Beta: PASS" || echo "Beta: FAIL"-```--After this, tell the user both results.--### Step 5b: Tier 8 — Cross-worker usage emission (writer_id fallback)--This validates that `UsageReportEvent`s emitted from inference activities running on-the runner process land in the same NDJSON partition as the router-side trace events,-stamped with a runner-process `writer_id` (`act_{pid}_{uuid8}`). It's the e2e-counterpart to the Phase 4 integration test-`tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py`.--**Important — the default dry run does NOT exercise the *reportable*-usage path.**-`--temporal --dry-run` DOES dispatch `act_llm_gen_text`-to the runner (the leaf mocks inside the activity — that's Tier 17's subject),-but its default synthetic usage is **zero-token and suppressed by design**, so no-rendered cost report and no reportable usage events come back from it.--**Cheap deterministic CLI way: `--dry-run --mock-usage` (Tier 8b below).** To observe-runner-side `act_*` writer files from the CLI *without LLM spend*, add the hidden-`--mock-usage` flag to a dry run: the dry run dispatches `act_llm_gen_text` to the-runner exactly as a real run does, and the flag flips the dry leaf's reporting to-reportable non-zero synthetic usage. This is the deterministic, free path — prefer it-over live mode for writer-id observation, and see **Tier 8b** for the full-cross-worker cost-report assertion. The two other paths below remain valid: the-Phase 4 integration test (which substitutes the inference activity with a wrapper-that synthesizes a real `LLMJob` server-side), and live mode (real spend).--```bash-timeout 180 .venv/bin/pytest -x -v \- tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py \- -m temporal --temporal-server local --timeout=60 2>&1 | tail -60-```--Expected: both test cases pass —-`test_runner_usage_event_lands_in_same_ndjson_dir` (lands in same dir, with an-`__w_act_*` file) and `test_no_double_emit_in_split_worker_pool` (no-double-counting between fast path and fallback).--**To observe `act_*` writer files via the CLI**, run live mode (real LLM call —-costs money) so `act_llm_gen_text` is actually dispatched to the runner:--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --no-logo --graph-```--After completion, list the trace files for the run:--```bash-RUN_ID=$(ls -t .pipelex/traces/ | head -1)-ls -la .pipelex/traces/$RUN_ID/-grep -l '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson-grep -hoE '"writer_id":"[^"]+"' .pipelex/traces/$RUN_ID/*.ndjson | sort -u-```--Expect:--- At least one `wf_*__w_act_{pid}_{uuid}.ndjson` file alongside the- router-side `wf_*.ndjson` files.-- The runner-side file contains `usage_report` events with- `writer_id` starting `act_`.-- Router-side files contain `pipe_start` / `pipe_end_success` events with- `writer_id="primary"`.--If only `writer_id="primary"` appears, the inference activity ran in the-router process. Check `worker_config.activity_queues` (and the workflow's-own `task_queue`) so that `act_llm_gen_text` resolves to a queue the runner-listens on, and confirm both workers are running with the latest code-(restart them if they predate the Phase 2 runner-side fallback commit).--### Step 5b': Tier 8b — Cross-worker cost report assembly (`--dry-run --mock-usage`, free + deterministic)--#### Scope manifest — which arms to run--Tier 8b has several arms. By default, run only the free, deterministic mock arms (the **default** scope — arms A–B). If the request carries an **explicit spend opt-in** — the canonical token `full` (aliases `thorough`, `every`, `with-spend`), shown as the **full** scope in the table below — run **every** arm, including the live ones that cost real money, and report PASS/FAIL for each. Do not stop after the cheap arms when a spend opt-in is present. Do **not** treat bare "live" or "all" as the opt-in (too easily incidental; "live" also collides with the default mock arm, which already runs in LIVE mode) — if that's the only signal, confirm before spending.--Run each arm in listed order. After each, surface the asserted token totals (where the arm produces them) and PASS/FAIL, then continue. **Arm D is the exception** — it runs no assertion script and emits no `RESULT: PASS` or token totals; its pass is the *absence* of a cost table and usage events (see its sub-section), so do not treat the missing totals as a failure. For every arm that *does* run the assertion script, if it cannot reach `RESULT: PASS`, stop and report it rather than silently continuing.--| # | Arm | Scope | Spend | Expected assertion |-|---|---|---|---|---|-| A | **Mock primary** — `--dry-run --mock-usage` `native_text_sequence` | default + full | free | 2 events / 200 input / 100 output, `--expected-model-type llm --require-fallback` |-| B | **Cross-child fan-out** — `--dry-run --mock-usage` `temporal_parallel` | default + full | free | 3 events / 300 input / 150 output, `--expected-model-type llm --require-fallback` (the CLI assert checks the summed total, **not** workflow-span; the cross-child guarantee itself is enforced by the pytest counterpart `test_split_worker_cross_child_usage.py`) |-| C | **CSV un-truncated cross-check** — flip `reporting_config.is_generate_cost_report_file_enabled=true`, rerun arm A, confirm CSV totals == NDJSON totals, restore to `false` | full | free | `csv tokens` line matches NDJSON totals |-| D | **`--no-costs` negative gate** | full | free | no cost table, no `usage_report` events, `reactflow.html` still assembles |-| E | **Live LLM arm** — drop `--dry-run --mock-usage` on `native_text_sequence` | full | **real** | non-zero total, real `model_names` (not `mock_usage`), `--require-fallback --require-nonzero` |-| F | **Live img-gen arm** — run a Tier 4/5 image bundle live with `--graph --costs` | full | **real** | `--expected-model-type img_gen --require-fallback --require-nonzero` |-| G | **Live extract arm** — run the extract bundle `pdf_extract_page_views.mthds` (`--pipe pdf_extract_with_page_views`) live against the plain split workers with `--graph --costs` | full | **real** | `--expected-model-type extract --require-fallback --require-nonzero` |--Arms F and G validate non-LLM usage, which the mock arms cannot reach (non-LLM dry leaves emit no synthetic usage), so they are **live-only** — they are the sole way to prove img-gen / extract token usage crosses the runner fallback and aggregates into the cost report. Each arm's full command + assertion is detailed in the sub-sections below; arms A/C map to "Primary check", B to "Cross-child aggregation", D to "Negative check", E to "Live arm", F/G to "Non-LLM cross-worker cost". Arms F and G run their bundles on the **same plain router+runner split workers** as the other arms — they do **not** need the routing battery's multi-queue (`q_extract` / `q_image_gen`) setup. On plain split workers the img-gen / extract activity runs on the runner and emits via the `act_*` fallback, which is exactly what `--require-fallback` checks.--Tier 8 (Step 5b above — the writer-id landing pytest) is the precursor to all of these and should pass first under any scope.-------Tier 8 proves a runner-side `UsageReportEvent` *lands* in the NDJSON partition.-Tier 8b proves the next link end-to-end: those cross-worker usage events-*assemble* onto `PipeOutput.tokens_usages` (via `act_assemble_tracing` with-`assemble_usage=True`) and the submitter renders a **single** end-of-run cost-report covering usage from all workers — with **no LLM spend**.--**Why the `--mock-usage` sub-flag.** A dry run-dispatches `act_llm_gen_text` to the runner (the leaf mocks inside the-activity), but its default synthetic usage is zero-token → the cost report is-*suppressed* by design. The hidden `--mock-usage` flag (requires `--dry-run`)-keeps that exact dispatch and flips the dry leaf's reporting to reportable-non-zero synthetic usage (model `mock_usage`). So the runner emits `act_*`-usage events AND the report is non-suppressed — the one thing the default dry-run can't validate cheaply.--**Requires split router+runner workers** (Step 2 in `mode-2-setup.md`). The point-is that the usage events originate on the *runner* process and must aggregate-into the report assembled for the *router*/submitter. `--mock-usage` is a hidden-option on the main `run` subcommands (pipe/method/bundle) and requires-`--dry-run`.--**Primary check — both artifacts assemble from one event read (D3):**--```bash-timeout 180 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --dry-run --mock-usage --mock-inputs --no-logo --graph --costs 2>&1 | tail -30-echo "EXIT=$?"-```--Expect, at the end of the submitter output:--- A Rich **cost table** with model `mock_usage`, non-zero input/output token- counts, and a run total (the synthetic per-call counts are- `MOCK_USAGE_NB_TOKENS_BY_CATEGORY = {INPUT: 100, OUTPUT: 50}`, so a- 2-LLM-step sequence totals 200 input / 100 output).-- A `reactflow.html` for the run (graph assembles from the *same* event read).-- Exit 0.--Then **assert the numbers** (don't eyeball — the terminal table truncates wide-columns to `0 … …`). The `assert_cross_worker_cost.py` helper sums input/output-tokens straight from the NDJSON usage events, counts them, checks a runner `act_*`-writer engaged, and (if a CSV report exists) cross-checks the un-truncated CSV-totals. For mock-usage the per-call counts are fixed-(`MOCK_USAGE_NB_TOKENS_BY_CATEGORY = {INPUT: 100, OUTPUT: 50}`), so a-2-LLM-step sequence must total exactly 2 events / 200 input / 100 output:--```bash-RUN_ID=$(ls -t .pipelex/traces/ | head -1)-ls -la .pipelex/traces/$RUN_ID/-.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \- --run-dir .pipelex/traces/$RUN_ID \- --expected-events 2 --expected-input 200 --expected-output 100 \- --expected-model-type llm --require-fallback-```--Expect `RESULT: PASS`, with:--- At least one `wf_*__w_act_{pid}_{uuid}.ndjson` file (runner-side) carrying- `usage_report` events with `writer_id` starting `act_` (the `--require-fallback`- gate), alongside router-side `wf_*.ndjson` with `writer_id="primary"`.-- The `usage events` count equal to the number of LLM steps (one per mocked- `act_llm_gen_text`), no double-count between the fast path and the fallback.-- `total tokens : input=200 output=100`.--**Un-truncated CSV cross-check (optional but recommended).** The Rich console-table truncates; the CSV does not. Enable-`reporting_config.is_generate_cost_report_file_enabled = true` in-`.pipelex/pipelex.toml` before the run, then the run also writes-`reports/cost_report*.csv`. The script auto-detects it and asserts the CSV token-totals equal the NDJSON totals (`csv tokens` line in its output). Restore the-flag to `false` afterwards.--**Cross-child aggregation (A+B+C → one report).** Run a fan-out bundle so usage-from multiple child workflows aggregates into a single submitter report:--```bash-timeout 180 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds \- --pipe temporal_parallel_sequence \- --temporal --dry-run --mock-usage --mock-inputs --no-logo --graph --costs 2>&1 | tail -30-echo "EXIT=$?"-```--Expect a single end-of-run cost table whose token totals sum the parent and both-child-workflow branches — not one table per branch. Assert it numerically: the-parallel bundle has three LLM steps (`branch_tone`, `branch_length`,-`summarize_results`), so 3 events / 300 input / 150 output, and usage must span-**more than one workflow** (cross-child):--```bash-RUN_ID=$(ls -t .pipelex/traces/ | head -1)-.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \- --run-dir .pipelex/traces/$RUN_ID \- --expected-events 3 --expected-input 300 --expected-output 150 \- --expected-model-type llm --require-fallback-```--The pytest counterpart for this cross-child aggregation (no spend) is-`tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py`.--**Negative check — `--no-costs` gates costs only, graph unaffected:**--```bash-timeout 180 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --dry-run --mock-usage --mock-inputs --no-logo --graph --no-costs 2>&1 | tail -30-echo "EXIT=$?"-RUN_ID=$(ls -t .pipelex/traces/ | head -1)-grep -l '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson || echo "No usage_report events (expected with --no-costs)"-ls results/*/reactflow.html 2>/dev/null && echo "graph still rendered (expected)"-```--Expect: **no** cost table rendered, **no** `usage_report` events in the NDJSON-(the usage event-log isn't wired when `--no-costs`), but the `reactflow.html`-still assembles — proving `--costs` gates only the cost channel, independent of-`--graph`.--**Live arm (real spend, opt-in)** — mirrors Tier 8's live block for the-real-payload case. Drop `--dry-run --mock-usage` so the real `act_llm_gen_text` runs on-the runner and bills real tokens:--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --no-logo --graph --costs 2>&1 | tail -30-```--Real token counts are not predictable, so assert non-zero rather than exact —-the run still must capture real provider tokens, cross the runner boundary, and-aggregate to a non-zero total:--```bash-RUN_ID=$(ls -t .pipelex/traces/ | head -1)-.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \- --run-dir .pipelex/traces/$RUN_ID \- --expected-model-type llm --require-fallback --require-nonzero-```--Expect `RESULT: PASS` with non-zero `total tokens` and a real `model_names` entry-(not `mock_usage`). The pytest counterpart for this real-inference cross-worker-path (gated, opt-in spend) is-`tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py`-(marked `inference`/`llm`; the no-spend counterpart is-`tests/integration/pipelex/temporal/tracing/test_mock_usage_temporal.py`).--**Non-LLM cross-worker cost (img-gen / extract — live only).** `--mock-usage`-cannot cover image generation or extraction: non-LLM dry leaves emit no synthetic-usage, so non-LLM token usage only crosses the runner boundary-on a real run. The img-gen / extract live tiers (Tiers 4 / 5 / 10c) already-exercise those activities cross-process but never check the cost numbers. After-running one of those live tiers with `--graph --costs` against split workers,-assert its run dir surfaced non-zero non-LLM usage with the right model handle:--```bash-RUN_ID=$(ls -t .pipelex/traces/ | head -1)-# --expected-model-type img_gen for an image-gen bundle, extract for an extract bundle-.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \- --run-dir .pipelex/traces/$RUN_ID \- --expected-model-type img_gen --require-fallback --require-nonzero-```--Expect `RESULT: PASS` proving image-gen/extract token usage (not just LLM) is-captured, emitted via the runner fallback, and aggregated into the submitter's-cost report. The no-spend unit counterparts (img-gen/extract usage through the-fallback + aggregator) are-`tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py` and-`tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py`.--After each run, tell the user: the script's `RESULT: PASS/FAIL`, whether the cost-table rendered, the distinct `writer_id` set (must include an `act_*`), the-usage-event count and the summed input/output tokens (NDJSON, and CSV if-enabled), and the graph file path.--### Step 5c: Tier 9 — Object generation through Temporal cross-process--Validates that `act_llm_gen_object` and `act_llm_gen_object_list` survive a-cross-process round-trip. Every existing `library_crate/*.mthds` outputs `Text`,-so the structured-output activities never go through the real Temporal data-converter cross-process — Tier 9 closes that gap. The `make_object` /-`make_object_list` round-trip uses-`model_dump(mode="json", serialize_as_any=True)` and `model_validate(...)` on-the activity boundary; if either side regresses, nested fields silently drop-or fail to validate.--The `structured_output_sequence` bundle defines an `Invoice` concept with a-nested `Customer` concept and a `LineItem` list — at least one nested field-is required so the test exercises non-trivial JSON serialization.--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/structured_output_sequence.mthds \- --pipe generate_invoice_single \- --temporal --dry-run --mock-inputs --no-logo --graph-```--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/structured_output_sequence.mthds \- --pipe generate_invoice_list \- --temporal --dry-run --mock-inputs --no-logo --graph-```--After each completes, tell the user: PASS/FAIL, output dir, graph file path.-The pytest counterpart in `tests/integration/pipelex/temporal/tracing/test_split_worker_object_gen.py`-runs the same scenario through the in-process server with split workers and-asserts the round-trip preserves nested field values exactly.--### Step 5d: Tier 11 — `make_extract_pages` two-activity cross-process--Validates the most fragile post-collapse path: the only `ContentGeneratorProtocol`-method that dispatches more than one activity. `make_extract_pages` runs-`act_extract_gen_extract_pages` then conditionally `act_render_page_views`,-attaching each rendered page to the corresponding `PageContent.page_view`. This-is also the only method whose `activity_id` uniqueness mitigation-(`f"{base_id}-pages"` and `f"{base_id}-render-page-views"`) actually matters in-production.--The test fixture workflow `WfTestContentGeneratorPdfPageViews` (registered in-`TEMPORAL_TEST_WORKFLOWS`) exercises this flow. The pytest counterpart-substitutes both activities with canonical fixtures so it runs without Azure-Document Intelligence credentials or pypdfium2 — the goal is to pin the-activity_id contract, not to re-validate the OCR backend (already covered by-`content_generation/test_tprl_content_generator_pdf_page_views.py`):--```bash-timeout 180 .venv/bin/pytest -x -v \- tests/integration/pipelex/temporal/tracing/test_split_worker_extract_pages.py \- -m temporal --temporal-server local --timeout=60 2>&1 | tail -60-```--After this completes, tell the user: PASS/FAIL. The test asserts via-`WorkflowHandle.fetch_history()` that exactly two `ActivityTaskScheduled`-events appear with `activity_id` ending in `-pages` and `-render-page-views`.-For manual Temporal Web UI inspection, point `temporal-worker-router` /-`temporal-worker-runner` at the same dev server and inspect the workflow's-event timeline — the two scheduled activities should be visible with their-distinct activity_ids.--### Step 5e: Tier 12 — Deeply-nested controller stack (CV batch screening)--Validates the full pipelex-demos example 21 pipeline through Temporal: a-nested PipeSequence -> PipeSequence (job-offer prep) + PipeBatch (per-CV-fan-out) -> PipeSequence (extract + analyze + match) over PipeExtract +-PipeLLM operators. This is the canonical "real-world" e2e bundle for the-skill — controller composition that the other tiers only exercise piecewise.--The pytest counterpart lives at-`tests/integration/pipelex/temporal/library_crate/test_wf_cv_batch_screening.py`-(in-process server, dry mode). The direct-mode (no Temporal) e2e counterpart-lives at `tests/e2e/pipelex/cv_batch_screening/test_cv_batch_screening.py`.--Dry-run (CLI mocks inputs — no PDFs needed):--```bash-timeout 180 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/cv_batch_screening.mthds \- --pipe batch_analyze_cvs_for_job_offer \- --temporal --dry-run --mock-inputs --no-logo --graph-```--Live (real Azure Doc Intel extract + LLM analysis — uses `John-Doe-CV.pdf`-and `Job-Offer.pdf` from `tests/data/documents/`):--```bash-timeout 900 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/cv_batch_screening.mthds \- --pipe batch_analyze_cvs_for_job_offer \- --inputs tests/integration/pipelex/temporal/library_crate/cv_batch_screening_inputs.json \- --temporal --no-logo --graph-```--After completion, tell the user: PASS/FAIL, output dir, graph file path. The-generated ReactFlow graph is unusually rich for this pipeline (nested-PipeSequence containers + PipeBatch fan-out edges + per-step extract/LLM-nodes) and is the recommended one to open at the end of the run:--```bash-open results/batch_analyze_cvs_for_job_offer_output_01/reactflow.html-```--If the live run errors with `Extract choice '...' was not found in the model-deck`, the deck is not loading Azure Doc Intel — fix the deck before falling-back to a different handle; user preference is Azure Doc Intel via the-Pipelex Gateway, period (do not substitute `mistral-ocr` / `deepseek-ocr`).--### Step 5f: Tiers 13–16 — Error propagation across the activity → workflow → submitter boundary--These four tiers validate the error-handling deliverable in a *true* 3-process-setup: a worker-side failure must travel back to the submitter carrying its-`ErrorReport` — the submitter must surface the *real classified inner error*,-not a bare "workflow failed" wrapper. The in-process pytest suite (Mode 1 Step 2b-in `SKILL.md`) asserts the structured `ErrorReport` fields directly; these tiers-prove the same survives real OS-process serialization. Tiers 13–15 fail a-*single* top-level activity, one per activity family (LLM, extract, image-gen);-Tier 16 fails inside a fanned-out child workflow.--**Submitter note.** The human CLI (`pipelex run bundle`) is the only CLI that-dispatches to Temporal — the agent CLI has no `--temporal`. It prints a flat-message, so the PASS check is at the message level. Precise field-level-assertions (`error_category` etc.) belong to Mode 1 Step 2b in `SKILL.md`.--**Inducing the failure — why `.env` must be edited.** The worker is a separate-process; you make its inference calls fail by giving it invalid credentials.-A shell-env prefix on the `tmux` command does NOT work: Pipelex loads `.env`-with `override=True` (`pipelex/system/environment.py`), so `.env` always wins-over the shell env. The reliable method is to temporarily append invalid keys-to `.env`, start the worker (it captures the tampered env at boot), then restore-`.env` immediately — the submitter and everything after run with a clean `.env`,-while the already-booted worker keeps the bad keys.--**Each tier owns the task queue exclusively.** The bad-credential worker must be-the *only* worker polling `temporal_task_queue` — else a healthy shared worker-picks up the activity and succeeds. Kill all workers first; restart the shared-ones (Step 2 in `mode-2-setup.md`) only if you continue to later tiers (Step 6 restarts its own).--**Start the bad-credential worker** — shared by Tiers 13–16, start it once:--```bash-# Back up .env, append invalid inference credentials, start the worker, restore .env-cp .env /tmp/dotenv-tier-err-bak-cat >> .env << 'EOF'--# --- temporal-e2e-validate Tier 13-16: invalid inference credentials (temporary) ----OPENAI_API_KEY=invalid-key-tier-err-PIPELEX_GATEWAY_API_KEY=invalid-key-tier-err-PIPELEX_INFERENCE_API_KEY=invalid-key-tier-err-ANTHROPIC_API_KEY=invalid-key-tier-err-GOOGLE_API_KEY=invalid-key-tier-err-AZURE_API_KEY=invalid-key-tier-err-EOF-pkill -f "pipelex.temporal.worker_cli" 2>/dev/null; sleep 1-tmux kill-session -t temporal-worker-err 2>/dev/null-tmux new-session -d -c "$PWD" -s temporal-worker-err \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed'-for attempt in $(seq 1 30); do- if tmux capture-pane -t temporal-worker-err -p 2>/dev/null | grep -q "Temporal Worker started"; then break; fi- if [ "$attempt" -eq 30 ]; then- echo "TIMEOUT: temporal-worker-err did not start within 30s — last 50 lines:"- tmux capture-pane -t temporal-worker-err -p -S -50- fi- sleep 1-done-cp /tmp/dotenv-tier-err-bak .env && echo ".env restored — the worker holds the invalid keys"-grep -q "tier-err" .env && echo "WARNING: .env still tampered — restore manually" || echo ".env clean"-```--**Tier 13 — LLM activity error propagation**--Submits `native_text_sequence` live. The first `act_llm_gen_text` call fails-provider-side with a 401 auth error — classified `CONFIGURATION` (non-retryable,-so no retry storm), converted to `TemporalError`, carried back as an-`ErrorReport`. The auth rejection happens before any tokens are billed — free.--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --no-logo > /tmp/tier13.log 2>&1-echo "EXIT=$?"-tail -25 /tmp/tier13.log-```--Report to the user:--- **PASS** = non-zero exit (within ~30s — `CONFIGURATION` is non-retryable), and- the final `Failed to execute pipeline '...': <cause>` line carries the *real- inner error* — an authentication / `401` / `Invalid API Key` message. That- proves the worker's `ErrorReport` crossed the boundary: without recovery the- cause would be a bare workflow-failed wrapper with no inner detail. Confirm on- the worker:- `tmux capture-pane -t temporal-worker-err -p -S -200 | grep -iE "act_llm_gen_text|TemporalError|authentication|401"`.-- **FAIL** = hangs > 60s; exits 0 (a stray healthy worker stole the activity —- check no other worker polls the queue); or the cause is only a generic- workflow-failed wrapper with no inner provider error (see "Interpreting- failures").--Note: the string `Failed to execute workflow WfPipeRun:` appears in a normal-`ERROR` log line — that is expected. The signal is the *inner cause* after it.--**Tier 14 — non-LLM (extract) activity error propagation**--The counterpart through a non-LLM activity, against the *same* bad-credential-worker. `act_extract_gen_extract_pages` fails on the worker; the `ErrorReport`-must cross the same way. Mirrors the LLM + non-LLM split that-`test_activity_error_boundary.py` covers in-process.--Create the inputs JSON if it does not exist:--```bash-cat > /tmp/pdf_extract_inputs.json << EOF-{- "source_pdf": {- "concept": "native.Document",- "content": { "url": "$PWD/tests/data/documents/Job-Offer.pdf" }- }-}-EOF-```--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/pdf_extract_page_views.mthds \- --pipe pdf_extract_with_page_views \- --inputs /tmp/pdf_extract_inputs.json \- --temporal --no-logo > /tmp/tier14.log 2>&1-echo "EXIT=$?"-tail -25 /tmp/tier14.log-```--Report to the user:--- **PASS** = non-zero exit, and the `Failed to execute pipeline '...': <cause>`- line carries the real extract auth error (`Extract service error ... 401 ...- Invalid API Key`). Confirm on the worker:- `tmux capture-pane -t temporal-worker-err -p -S -200 | grep -iE "act_extract_gen_extract_pages|TemporalError|authentication|401"`.-- **FAIL** = same modes as Tier 13.--**Tier 15 — image-generation activity error propagation**--The counterpart through the image-generation activity (`act_img_gen_images`),-against the *same* bad-credential worker. The error boundary-(`@convert_pipelex_errors`) is wired on every in-scope activity — Tier 13 proves-it for LLM, Tier 14 for extract, this tier closes it for image generation.--`generate_image` is a single `PipeImgGen` step with a static prompt, so-`act_img_gen_images` is the first (and only) inference activity dispatched — a-failure here is unambiguously the image-gen path, not a preceding LLM step. The-image model (`gpt-image-1-mini`) rejects the request on the tampered-OpenAI / Gateway key before any image is billed — free.--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/pipes/pipelines/test_image_out_in.mthds \- --pipe generate_image \- --temporal --no-logo > /tmp/tier15.log 2>&1-echo "EXIT=$?"-tail -25 /tmp/tier15.log-```--Report to the user:--- **PASS** = non-zero exit, and the `Failed to execute pipeline '...': <cause>`- line carries the real image-gen auth error (`401` / `Invalid API Key` from the- image model). Confirm on the worker:- `tmux capture-pane -t temporal-worker-err -p -S -200 | grep -iE "act_img_gen_images|TemporalError|authentication|401"`.-- **FAIL** = same modes as Tier 13.--**Tier 16 — error propagation out of a fanned-out child workflow (PipeBatch)**--Tiers 13–15 fail a *single* top-level activity. This tier proves the report also-crosses when the failure happens *inside a fanned-out child workflow*: a-`PipeBatch` dispatches one child workflow per item, and each child's-`act_llm_gen_text` fails on the bad-credential worker. The first failing child's-`ErrorReport` must propagate up through the batch controller → `WfPipeRouter` →-submitter — not collapse into a generic "a batch item failed" wrapper.--Running the `PipeBatch` pipe (`batch_temporal_describe_topics`) directly — not-the wrapping `temporal_batch_sequence` — is deliberate: the sequence would fail-at its first LLM step (topic generation) before ever reaching the fan-out.-Feeding the topic list via `--inputs` skips topic generation so the batch-fan-out is the first inference dispatched.--Create the inputs JSON:--```bash-cat > /tmp/batch_topics_inputs.json << 'EOF'-{- "topics": {- "concept": "temporal_batch_test.Topic",- "content": ["science", "history", "music"]- }-}-EOF-```--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \- --pipe batch_temporal_describe_topics \- --inputs /tmp/batch_topics_inputs.json \- --temporal --no-logo > /tmp/tier16.log 2>&1-echo "EXIT=$?"-tail -25 /tmp/tier16.log-```--Report to the user:--- **PASS** = non-zero exit, and the `Failed to execute pipeline '...': <cause>`- line carries the real per-branch auth error (`401` / `Invalid API Key`) —- proving the child workflow's `ErrorReport` survived the fan-in, not a generic- batch wrapper. Confirm on the worker:- `tmux capture-pane -t temporal-worker-err -p -S -200 | grep -iE "act_llm_gen_text|temporal_describe_topic|TemporalError|401"`.-- **FAIL** = hangs > 60s (a retry storm — the per-branch failure should be- non-retryable `CONFIGURATION`); exits 0 (a stray healthy worker stole a- branch activity); or the cause is a generic batch/workflow-failed wrapper with- no inner provider error (the child `ErrorReport` did not cross the fan-in —- the exact gap this tier exists to catch).--**Teardown** (after Tiers 13–16):--```bash-tmux kill-session -t temporal-worker-err 2>/dev/null-# Safety net: ensure .env was restored, then drop the backup-if [ -f /tmp/dotenv-tier-err-bak ]; then- grep -q "tier-err" .env && cp /tmp/dotenv-tier-err-bak .env && echo ".env restored from backup"- rm /tmp/dotenv-tier-err-bak-fi-grep -q "tier-err" .env && echo "WARNING: .env still tampered" || echo ".env clean"-```--If you continue to Step 6 or later tiers, restart the shared worker(s) per Step 2 in `mode-2-setup.md`.-------### Step 5g: Tier 17 — DRY honors the Temporal backend (leaf mock inside the activity)--Contract under test: `run_mode` is orthogonal to the backend — a `--temporal --dry-run` run goes-through the REAL distribution machinery (`WfPipeRouter` → child workflows → `act_*_gen_*`-activities on the worker) and the cogt **leaf inside each activity** mocks instead of calling a-provider (`run_mode=DRY` rides `CogtRunParams` on every assignment across the wire). A regression-to the old behavior — dry-run short-circuiting at the operator/router and dispatching nothing —-must turn this tier red. Purpose: test dispatch, scheduling, serialization, routing, and-cross-worker propagation at zero AI cost.--The Mode-1 (pytest) companion is-`tests/integration/pipelex/temporal/tracing/test_dry_run_dispatches_and_mocks.py` — it asserts the-dispatch via workflow history (`ActivityTaskScheduled` across the parent + child workflows), the-`DRY RUN:` leaf output, and the zero-token suppressed usage over the in-process server.--**Setup.** Split workers up (router + runner per Step 2 in `mode-2-setup.md`), Temporal-enabled.-The dry leaves never touch a provider or storage, so this tier needs NO API keys — the "dry works-with no inference configured" invariant must hold even though activities dispatch.--**GREEN arm — LLM bundle:**--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --dry-run --mock-inputs --no-logo --graph > /tmp/tier17.log 2>&1-echo "EXIT=$?"-tail -15 /tmp/tier17.log-# The runner DID execute the LLM activity (pre-Part-B: nothing inference-related ran here):-tmux capture-pane -t temporal-worker-runner -p -S -300 | grep -c "act_llm_gen_text"-# The leaf mocked INSIDE the activity — no provider call, no storage write on the runner:-tmux capture-pane -t temporal-worker-runner -p -S -300 | grep -iE "openai|anthropic|provider|storing|uploaded" | head -5-```--Report to the user:--- **PASS** = exit 0 · the runner pane shows `act_llm_gen_text` executions (count ≥ 1) · no- provider/storage lines on the runner · the output `main_stuff` text starts with `DRY RUN:` ·- **no cost table rendered** (zero-token dry usage is suppressed by design — contrast Tier 8b) ·- `reactflow.html` generated (cross-worker graph tracing assembled, LibraryCrate propagated).-- **FAIL** = non-zero exit; or the runner pane shows NO `act_llm_gen_text` (the dry run didn't- dispatch — the pre-Part-B shortcut is back); or any provider/storage activity on the runner.--**GREEN arm — non-LLM leaves** (every leaf mocks under DRY — img/extract DRY branches sit at the-`*_and_store` layer, so also assert no storage writes):--```bash-# Extract (two activities: act_extract_gen_extract_pages + act_render_page_views):-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/pdf_extract_page_views.mthds \- --pipe pdf_extract_with_page_views \- --temporal --dry-run --mock-inputs --no-logo > /tmp/tier17-extract.log 2>&1-echo "EXIT=$?"-tmux capture-pane -t temporal-worker-runner -p -S -200 | grep -cE "act_extract_gen_extract_pages|act_render_page_views"--# Image generation:-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/pipes/pipelines/test_image_out_in.mthds \- --pipe generate_image \- --temporal --dry-run --mock-inputs --no-logo > /tmp/tier17-img.log 2>&1-echo "EXIT=$?"-tmux capture-pane -t temporal-worker-runner -p -S -200 | grep -c "act_img_gen_images"--# Web search:-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_search.mthds \- --pipe native_search \- --temporal --dry-run --mock-inputs --no-logo > /tmp/tier17-search.log 2>&1-echo "EXIT=$?"-tmux capture-pane -t temporal-worker-runner -p -S -200 | grep -c "act_search_gen_sourced_answer"-```--- **PASS** per arm = exit 0 + the named activity count ≥ 1 on the runner + no storage writes- (no bucket/object upload lines in the runner pane) + zero provider calls.--**No-keys GREEN arm — prove dry needs no inference configured.** The dispatched-and-mocked-behavior must hold on a worker with NO usable credentials. Tamper `.env` with invalid keys and-restart the runner using the same backup/restore method as Tiers 13–16 (Step 5f), then re-run the-GREEN LLM command:--- **PASS** = exit 0 with the keyless runner — the dry leaves never resolve a provider, so invalid- credentials are never exercised. Keep this runner for the RED arm below.--**RED arm — prove the leaf branch bites.** There is no router-level mock to revert to,-so the RED mutation is at the leaf: temporarily disable the DRY branch in `llm_gen_text`-(`pipelex/cogt/content_generation/llm_generate.py` — comment out the-`if llm_assignment.cogt_run_params.run_mode.is_dry:` short-circuit), restart the **runner** worker-(it must boot WITHOUT inference credentials — same `.env` tamper method as Tiers 13–16 if your-`.env` has real keys), and re-run the GREEN LLM command:--- **RED-PASS** = the dispatched `act_llm_gen_text` attempts a real call and **fails loud** on- missing/invalid keys — the dispatched + mocked-inside + no-spend + no-keys quartet flips.-- Restore the leaf branch (`git checkout -- pipelex/cogt/content_generation/llm_generate.py`),- restore `.env` if tampered, restart the workers, and re-run GREEN to confirm recovery.-------### Step 6: StoragePayloadCodec tests — does the codec work end-to-end?--These tests validate Phase 5: large payloads are transparently offloaded to external-storage by the `StoragePayloadCodec`, keeping Temporal's event history small. The worker-and submitter both use the codec — it's wired at the data converter level.--**Important:** The codec config is managed via `pipelex_temporary_override.toml` — a-config layer that loads after `pipelex_override.toml` and takes highest priority. This-file is ephemeral and gitignored. **NEVER modify `pipelex_override.toml`** — that is-the user's personal config.--**Step 6a: Create the temporary override to enable the codec**--```bash-cat > .pipelex/pipelex_temporary_override.toml << 'EOF'-# Temporary override for E2E codec testing — delete when done-[temporal.payload_codec_config]-is_enabled = true-EOF-echo "Temporary override created"-```--This takes precedence over whatever codec setting exists in `pipelex_override.toml`.--**Step 6b: Restart the worker** (so it picks up the new config)--```bash-tmux has-session -t temporal-worker 2>/dev/null && tmux kill-session -t temporal-worker-tmux new-session -d -c "$PWD" -s temporal-worker \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed'-sleep 4-tmux capture-pane -t temporal-worker -p -S -30 | grep -i "payload codec\|Temporal Worker started"-```--Look for `Payload codec enabled` and `Temporal Worker started`.--**Tier 6 — Codec transparency: existing pipelines work unchanged**--Re-runs Tier 1 and Tier 2 with the codec on. If the codec is truly transparent, the-same pipelines produce the same results. The only difference: payloads above 1MB are-stored in `.pipelex/temporal-payload-store/` instead of inline in Temporal's event history.--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --dry-run --mock-inputs --no-logo --graph-```--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \- --pipe dynamic_greeting_sequence \- --temporal --dry-run --mock-inputs --no-logo --graph-```--After both complete, check that storage was used:--```bash-ls -la .pipelex/temporal-payload-store/ 2>/dev/null && echo "Storage files found" || echo "No storage files (payloads may be below threshold in dry-run)"-```--Tell the user: PASS/FAIL for each, and whether storage files were created. In dry-run-mode, payloads may be below the 1MB threshold, so no storage files is expected. In live-mode, larger payloads should trigger storage. Either way, the key assertion is that the-pipelines complete successfully.--**Tier 7 — Large payload stress test**--A 3-step pipeline that accumulates verbose text output across steps. Each step's-WorkingMemory carries the results of all previous steps, so the payload grows. In live-mode this produces realistic multi-KB to multi-MB payloads. In dry-run mode, mock-outputs are small but the codec path is still exercised.--```bash-timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds \- --pipe large_payload_sequence \- --temporal --dry-run --mock-inputs --no-logo --graph-```--Live (real LLM calls — produces larger payloads, better stress test):--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds \- --pipe large_payload_sequence \- --temporal --no-logo --graph-```--After this completes, check storage and worker logs:--```bash-ls -la .pipelex/temporal-payload-store/ 2>/dev/null | head -20-tmux capture-pane -t temporal-worker -p -S -50 | grep -i "payload\|warning\|size\|codec" || echo "No payload warnings"-```--Tell the user: PASS/FAIL, output dir, graph file path, storage file count.--**Step 6c: Remove the temporary override**--```bash-.venv/bin/python -c "from pathlib import Path; Path('.pipelex/pipelex_temporary_override.toml').unlink(missing_ok=True)"-echo "Temporary override removed"-```--Optionally restart the worker to restore the base codec config, or leave it if done.--### Step 7: Final report--List all graph files and present a summary table with results. Tier 10a–10c are defined in `routing-battery.md` and Scenarios A–F in `queue-options-battery.md` — include those rows only if you ran those batteries:--```bash-ls results/*/reactflow.html-```--| Test | What it proved | Status | Graph | Payload warnings |-|------|---------------|--------|-------|-----------------|-| Tier 1: Sequence | Worker can unpack crate and run a pipe sequence | PASS/FAIL | path | — |-| Tier 2: Hydration | Worker handles dynamic concepts it has never seen | PASS/FAIL | path | — |-| Tier 2b: Cross-process registry | `act_deliver` decodes hydrated `pipe_output` on runner (forced via `delivery_assignment`) | PASS/FAIL | — | — |-| Tier 2c: Validate sweep stays in-process | `validate bundle --temporal` over a standalone `PipeBatch` exits 0 **and** the worker received no `WfPipeRouter` dispatch (the sweep never leaks to Temporal) | PASS/FAIL | — | — |-| Tier 2d: Dry-run+validate as one in-memory activity | the Temporal-dispatched /validate runs the whole sweep + graph dry-run inside ONE in-process activity (zero nested dispatch), traces the graph in memory (no NDJSON/DDB), returns {status, GraphSpec}; best-effort graph → None on failure | PASS/FAIL | path | — |-| Tier 3: Parallel | Branches execute as concurrent child workflows | PASS/FAIL | path | — |-| Tier 4: ImgGen | Image generation pipeline works through Temporal | PASS/FAIL | path | yes/no |-| Tier 5: Image flow | Generated image flows as input to next pipe step | PASS/FAIL | path | yes/no |-| Tier 6: Codec transparency | Existing pipelines work unchanged with codec enabled | PASS/FAIL | path | — |-| Tier 7: Large payload | Multi-step pipeline with codec stress test | PASS/FAIL | path | — |-| Tier 8: Cross-worker usage | Runner-side `UsageReportEvent` lands in same NDJSON dir with `act_*` writer_id (live mode or integration test) | PASS/FAIL | — | — |-| Tier 8b: Cross-worker cost report | `--dry-run --mock-usage` (free): runner-side usage assembles onto `PipeOutput.tokens_usages` and the submitter renders a single non-suppressed cost report (model `mock_usage`); `--no-costs` renders none while `--graph` still assembles. **Numeric assertion** via `scripts/assert_cross_worker_cost.py` (sums NDJSON usage tokens, checks count + `act_*` fallback + optional CSV cross-check) — mock: 2 events/200 input/100 output; parallel: 3/300/150; live arms `--require-nonzero`; img-gen/extract `--expected-model-type` (live only) | PASS/FAIL | path | — |-| Tier 9: Object gen cross-process | `act_llm_gen_object` / `act_llm_gen_object_list` survive the JSON round-trip with nested fields intact | PASS/FAIL | path | — |-| Tier 10a: Multi-activity routing | `activity_queues.default` routes both `act_llm_gen_text` and `act_img_gen_images` to their dedicated worker pools; default runner sees 0 hits for either | PASS/FAIL/SKIPPED | — | — |-| Tier 10b: Per-handle routing | `activity_queues.by_handle` overrides the activity default per model handle — two distinct handles in one workflow land on two distinct workers | PASS/FAIL/SKIPPED | path | — |-| Tier 10c: Two activities, one route | `act_extract_gen_extract_pages` + `act_render_page_views` (no routing key) both land on a shared dedicated queue via Azure Doc Intel through Pipelex Gateway | PASS/FAIL | — | — |-| Tier 11: Extract two-activity | `act_extract_gen_extract_pages` + `act_render_page_views` dispatched cross-process with distinct activity_ids | PASS/FAIL | — | — |-| Tier 12: CV batch screening | Deeply-nested controller stack (PipeSequence → PipeSequence + PipeBatch → PipeSequence) over PipeExtract + PipeLLM (pipelex-demos example 21) | PASS/FAIL | path | — |-| Tier 13: LLM error propagation | A worker-side LLM auth failure crosses activity→workflow→submitter; submitter shows the real error, not the generic `"Failed to execute workflow"` wrapper | PASS/FAIL | — | — |-| Tier 14: Extract error propagation | Same for a non-LLM (`act_extract_gen_extract_pages`) activity failure | PASS/FAIL | — | — |-| Tier 15: Image-gen error propagation | Same for the image-generation (`act_img_gen_images`) activity failure | PASS/FAIL | — | — |-| Tier 16: Batch child-workflow error propagation | A per-branch failure inside a `PipeBatch` fan-out child workflow crosses the fan-in carrying its `ErrorReport`, not a generic batch wrapper | PASS/FAIL | — | — |-| Tier 17: DRY honors the backend | a `--temporal --dry-run` run dispatches `act_llm_gen_*` (+ extract/img-gen/search) to the worker and mocks INSIDE the activity — no real IO, no API keys needed, zero-token suppressed usage, cross-worker graph still assembles | PASS/FAIL | path | — |-| Scenario A: v2 multi-class routing | Specialized scopes + per-class runners cover LLM/img-gen/extract without cross-class leakage | PASS/FAIL/SKIPPED | — | — |-| Scenario B: per-queue timeout | `queue_options[X].start_to_close_timeout` overrides the worker_config baseline and flows into `ActivityTaskScheduled.start_to_close_timeout` | PASS/FAIL/SKIPPED | — | — |-| Scenario C: per-handle override | `handle_options[<handle>].start_to_close_timeout` wins over per-queue value for that one handle | PASS/FAIL/SKIPPED | — | — |-| Scenario D: queue rate limit | `queue_options[X].max_task_queue_activities_per_second` throttles dispatch — burst of N at rate R produces non-zero schedule_to_start latency on the tail | PASS/FAIL/SKIPPED | — | — |-| Scenario E: missing-worker timeout | A queue referenced by routing but polled by no worker produces a bounded `schedule_to_start` timeout, not a hang | PASS/FAIL/SKIPPED | — | — |-| Scenario F: CLI typo | `--task-queue typo_q` fails fast with `WorkerTaskQueueUnknownError` and a "Did you mean?" suggestion | PASS/FAIL | — | — |-| Concept isolation (alpha) | Conflicting `Result` concepts don't cross-contaminate | PASS/FAIL | path | — |-| Concept isolation (beta) | (same test, other side) | PASS/FAIL | path | — |-| Pipe isolation (alpha) | Conflicting `shared_step` pipes use correct version | PASS/FAIL | path | — |-| Pipe isolation (beta) | (same test, other side) | PASS/FAIL | path | — |-| Multi-concept (alpha) | Two overlapping concepts with incompatible structures | PASS/FAIL | path | — |-| Multi-concept (beta) | (same test, other side) | PASS/FAIL | path | — |--After reporting, propose opening the PipeParallel graph (most interesting cross-worker view):--```bash-open results/temporal_parallel_sequence_output_01/reactflow.html-```--If any concurrent tests fail, capture worker output for diagnosis (split workers-from Step 2 in `mode-2-setup.md` are the default — fall back to `temporal-worker`-only if you used the single full worker setup):--```bash-tmux capture-pane -t temporal-worker-router -p -S -200-tmux capture-pane -t temporal-worker-runner -p -S -200-```diff --git a/.claude/skills/temporal-e2e-validate/references/queue-options-battery.md b/.claude/skills/temporal-e2e-validate/references/queue-options-battery.mddeleted file mode 100644index 4fe12da75..000000000--- a/.claude/skills/temporal-e2e-validate/references/queue-options-battery.md+++ /dev/null@@ -1,404 +0,0 @@-# Temporal E2E — Mode 2 Step 9: queue options + worker-runtime profiles battery (v2)--> Reference file for the **temporal-e2e-validate** skill — Mode 2, Step 9.-> **Run `mode-2-setup.md` first** (Steps 1–2: Temporal server + worker processes).-> The **Timeouts policy** and the **surface-results-immediately** rule in `SKILL.md` apply to every command here.-> Sibling reference files: `mode-2-tiers.md` (Steps 3–7 — Tiers 1–14), `routing-battery.md` (Step 8 — v1 routing, the layer this battery builds on).--### Step 9: Queue options + worker-runtime profiles battery (v2)--This step validates the v2 surfaces shipped on top of v1 routing:--- **Per-queue submitter options** (`[temporal.queue_options.<q>]`) — timeouts and retry policy attach to the queue, not the activity.-- **Per-handle option overrides** (`activity_queues.*.handle_options.<handle>`) — a single handle on a queue can override the queue baseline.-- **Cluster-wide queue rate limit** (`queue_options[q].max_task_queue_activities_per_second`).-- **Named worker-runtime profiles** (`[temporal.worker_runtime_profiles.profiles.<name>]`) — concurrency slots, pollers, and rate-limit knobs become per-worker config selected via `--profile`.-- **Startup validation** — warn on unknown routing queues; fail with "did you mean?" on unknown `--task-queue`.--Scenarios A, B, D, E below are **live** for the same reason as Step 8 (`routing-battery.md`): dry-run short-circuits inference inside the workflow process, so `act_*` activities never get scheduled and the routing/timeout/rate-limit assertions are meaningless. Scenario C is a pytest/unit check and Scenario F is a CLI-startup check — neither submits a live workflow.--**Step 9.0 — Preflight + setup**--Build a temporary override exercising the new schema. Notice the named `worker_runtime_profiles` and `queue_options` blocks layered on top of v1 `activity_queues`:--```bash-cat > .pipelex/pipelex_temporary_override.toml << 'EOF'-# v1 routing — drives which queue each activity lands on.-[temporal.worker_config.activity_queues.act_llm_gen_text]-default = "q_llm"-by_handle = { "claude-4.6-sonnet" = "q_llm_anthropic" }--[temporal.worker_config.activity_queues.act_img_gen_images]-default = "q_imggen"--[temporal.worker_config.activity_queues.act_extract_gen_extract_pages]-default = "q_extract"--[temporal.worker_config.activity_queues.act_render_page_views]-default = "q_extract"--# Scenario B — per-queue timeout. q_slow has a generous start_to_close; the baseline is tight.-[temporal.worker_config]-default_activity_start_to_close_timeout = "0:00:30" # baseline tight on purpose--[temporal.queue_options.q_llm]-start_to_close_timeout = "0:05:00"--[temporal.queue_options.q_llm_anthropic]-start_to_close_timeout = "0:05:00"--# Every queue named under activity_queues must have a matching queue_options-# entry. Empty stanza = "use worker_config defaults for this queue".-[temporal.queue_options.q_imggen]-[temporal.queue_options.q_extract]--# NOTE: q_capped (Scenario D's rate-limited queue) is intentionally NOT declared-# here. The config validator rejects a queue_options entry that no activity_queues-# route references (orphan-queue / "the overlay will never apply" error), and the-# router refuses to boot. Scenario D routes act_llm_gen_text -> q_capped in its own-# override block, so q_capped lives there, not in this base setup.--# Scenario C — per-handle override on top of per-queue.-[temporal.worker_config.activity_queues.act_llm_gen_text.handle_options."claude-opus-4-7-1m"]-start_to_close_timeout = "0:25:00"--# Scenario A also exercises differently-shaped worker profiles per pool.-[temporal.worker_runtime_profiles]-default_profile = "default"--[temporal.worker_runtime_profiles.profiles.llm-throughput]-tuning_mode = "explicit"-max_cached_workflows = 10000-max_concurrent_workflow_tasks = 1000-max_concurrent_activities = 50-max_concurrent_local_activities = 1000-max_concurrent_workflow_task_polls = 100-max_concurrent_activity_task_polls = 20-max_activities_per_second = 60-sticky_queue_schedule_to_start_timeout = "0:30:00"-max_heartbeat_throttle_interval = "1:00:00"-default_heartbeat_throttle_interval = "1:00:00"-graceful_shutdown_timeout = "0:30:00"--[temporal.worker_runtime_profiles.profiles.img-gen-tight]-tuning_mode = "explicit"-max_cached_workflows = 10000-max_concurrent_workflow_tasks = 1000-max_concurrent_activities = 2 # small parallelism, large payloads-max_concurrent_local_activities = 1000-max_concurrent_workflow_task_polls = 100-max_concurrent_activity_task_polls = 100-max_activities_per_second = 2-sticky_queue_schedule_to_start_timeout = "0:30:00"-max_heartbeat_throttle_interval = "1:00:00"-default_heartbeat_throttle_interval = "1:00:00"-graceful_shutdown_timeout = "0:10:00"-EOF-```--Restart the router (only the router reads `activity_queues` / `queue_options` at dispatch time):--```bash-tmux kill-session -t temporal-worker-router 2>/dev/null-tmux new-session -d -c "$PWD" -s temporal-worker-router \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope router'-sleep 4-tmux capture-pane -t temporal-worker-router -p -S -30 | grep "Temporal Worker started"-```--**Also keep a general `runner` on the default task queue.** Un-routed activities — tracing (`act_flush_trace_events`, `act_assemble_tracing`) and `act_deliver` — are not in `activity_queues`, so they fall through to `default_task_queue` (`temporal_task_queue`). The specialized runners below only poll their own named queues, so without a general runner those activities have no poller and any `--graph` run (Scenarios A–B use `--graph`) **hangs** on the first trace flush (the activity sits `PENDING_SCHEDULED` forever). Spawn it:--```bash-tmux kill-session -t temporal-worker-runner 2>/dev/null-tmux new-session -d -c "$PWD" -s temporal-worker-runner \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner'-sleep 3-tmux capture-pane -t temporal-worker-runner -p -S -20 | grep "Temporal Worker started"-```--Spawn one specialized runner per queue, each with a profile shaped for its workload. Note the use of the new `runner-llm` / `runner-img-gen` / `runner-extract` scopes shipped in Phase 5:--```bash-tmux has-session -t temporal-worker-q-llm 2>/dev/null && tmux kill-session -t temporal-worker-q-llm-tmux new-session -d -c "$PWD" -s temporal-worker-q-llm \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner-llm --profile llm-throughput --task-queue q_llm'--tmux has-session -t temporal-worker-q-llm-anthropic 2>/dev/null && tmux kill-session -t temporal-worker-q-llm-anthropic-tmux new-session -d -c "$PWD" -s temporal-worker-q-llm-anthropic \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner-llm --profile llm-throughput --task-queue q_llm_anthropic'--tmux has-session -t temporal-worker-q-imggen 2>/dev/null && tmux kill-session -t temporal-worker-q-imggen-tmux new-session -d -c "$PWD" -s temporal-worker-q-imggen \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner-img-gen --profile img-gen-tight --task-queue q_imggen'--tmux has-session -t temporal-worker-q-extract 2>/dev/null && tmux kill-session -t temporal-worker-q-extract-tmux new-session -d -c "$PWD" -s temporal-worker-q-extract \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner-extract --task-queue q_extract'--sleep 5-for q in q-llm q-llm-anthropic q-imggen q-extract; do- echo "=== temporal-worker-$q ==="- tmux capture-pane -t "temporal-worker-$q" -p -S -20 | grep -E "started for|profile=|scope=" | head -3-done-```--Each line should report `profile='<name>' scope='<name>' task_queue='<q>'`. Profile selection should be visible in the worker startup log (introduced by the Phase 3 logging).--**Scenario A — Multi-class routing with specialized scopes (live)**--The v2 counterpart of Tier 10a (`routing-battery.md`), but using the specialized `runner-llm` / `runner-img-gen` / `runner-extract` scopes instead of bare `runner`. Each worker only registers its own activity class, so a stray routing decision lands on the wrong queue and the activity never executes (instead of being silently picked up by a generalist runner).--Submit a pipeline that fires at least one activity of each class:--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/pipes/pipelines/crazy_image_generation.mthds \- --pipe generate_crazy_image \- --temporal --no-logo --graph-```--After completion:--```bash-LLM_HITS=$(tmux capture-pane -t temporal-worker-q-llm -p -S -500 | grep -c "act_llm_gen_text")-IMG_HITS=$(tmux capture-pane -t temporal-worker-q-imggen -p -S -500 | grep -c "act_img_gen_images")-LLM_IMG_CROSS=$(tmux capture-pane -t temporal-worker-q-llm -p -S -500 | grep -c "act_img_gen_images")-IMG_LLM_CROSS=$(tmux capture-pane -t temporal-worker-q-imggen -p -S -500 | grep -c "act_llm_gen_text")-echo "q_llm llm=$LLM_HITS cross-class img=$LLM_IMG_CROSS (want llm≥1, img=0)"-echo "q_imggen img=$IMG_HITS cross-class llm=$IMG_LLM_CROSS (want img≥1, llm=0)"-if [ "$LLM_HITS" -ge 1 ] && [ "$IMG_HITS" -ge 1 ] && [ "$LLM_IMG_CROSS" -eq 0 ] && [ "$IMG_LLM_CROSS" -eq 0 ]; then- echo "Scenario A PASS"-else- echo "Scenario A FAIL — see hit table"-fi-```--**Scenario B — Per-queue timeout applied (live)**--The override sets `default_activity_start_to_close_timeout = "0:00:30"` (baseline tight) and `queue_options.q_llm.start_to_close_timeout = "0:05:00"` (generous). An LLM activity routed to `q_llm` should use 5min, not 30s. Read it back from workflow history:--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --no-logo --graph-```--After completion, find the latest workflow run in the dev UI (http://localhost:8233) and inspect any `ActivityTaskScheduled` event for `act_llm_gen_text`. Its `start_to_close_timeout` field should show `300s` (= 5min), not `30s`. To check programmatically, use the Temporal CLI:--```bash-RUN_ID=$(temporal workflow list --limit 1 --output json | jq -r '.[0].execution.run_id')-WF_ID=$(temporal workflow list --limit 1 --output json | jq -r '.[0].execution.workflow_id')-temporal workflow show --workflow-id "$WF_ID" --run-id "$RUN_ID" --output json \- | jq '.events[] | select(.activityTaskScheduledEventAttributes.activityType.name == "act_llm_gen_text") | .activityTaskScheduledEventAttributes.startToCloseTimeout'-```--Expected output: `"300s"`. Anything else (especially `"30s"`) means the resolver didn't pick up the per-queue overlay — Scenario B FAIL.--**Scenario C — Per-handle override wins over per-queue (pytest/unit)**--Two `act_llm_gen_text` calls (queue baseline `0:05:00`): one with a regular handle (stays at the `0:05:00` baseline), one with `claude-opus-4-7-1m` (the handle_options entry overrides to `0:25:00`). The queue is incidental — the per-handle override applies on top of whatever queue the (activity, handle) pair resolves to (here `q_llm`, since only `claude-4.6-sonnet` is routed to `q_llm_anthropic`). The reusable `per_handle_routing.mthds` bundle from Step 8 (`routing-battery.md`) can serve as a starting point if you want a live variant, but Scenario C is validated via pytest, not a live submission — inspect the `start_to_close_timeout` per scheduled activity: one should be `300s`, one should be `1500s`.--The full pytest integration sibling for this scenario already exists at:-`tests/integration/pipelex/temporal/tracing/test_split_worker_extract_pages.py::TestSplitWorkerExtractPages::test_queue_options_start_to_close_timeout_flows_to_dispatch`.-Run that to validate the resolver layer without spinning up the live CLI:--```bash-timeout 120 .venv/bin/pytest -xvs tests/integration/pipelex/temporal/tracing/test_split_worker_extract_pages.py::TestSplitWorkerExtractPages::test_queue_options_start_to_close_timeout_flows_to_dispatch \- -m temporal --temporal-server local --timeout=60-```--PASS = per-queue timeout flows through the resolver into the actual dispatch. The handle-options layer is covered by the unit test below — note the class in the node ID (`TestResolveDispatch::`), which the bare function name omits:--```bash-timeout 90 .venv/bin/pytest -x -q \- "tests/unit/pipelex/temporal/test_resolve_dispatch.py::TestResolveDispatch::test_handle_options_override_queue" \- --timeout=60-```--**Scenario D — Queue-level rate limit observed (live)**--`max_task_queue_activities_per_second = 2` on `q_capped` is server-enforced across all pollers. Fan out ≥10 activities to `q_capped` and the server releases them at ~2/s, so the tail activities show a growing `schedule_to_start` latency.--Scenario D needs its **own** override — it routes `act_llm_gen_text` to `q_capped`, which conflicts with the base setup's `act_llm_gen_text -> q_llm`. Rewrite the override, restart the router, and run a `q_capped` worker plus the general runner (for tracing/deliver):--```bash-cat > .pipelex/pipelex_temporary_override.toml << 'EOF'-[temporal.worker_config.activity_queues.act_llm_gen_text]-default = "q_capped"--[temporal.queue_options.q_capped]-max_task_queue_activities_per_second = 2-EOF--# kill the base-setup specialized runners (their routing no longer applies)-for s in temporal-worker-q-llm temporal-worker-q-llm-anthropic temporal-worker-q-imggen temporal-worker-q-extract; do- tmux kill-session -t "$s" 2>/dev/null-done-# router (reads new routing) + general runner (tracing/deliver) + q_capped worker (runs the LLM activity)-tmux kill-session -t temporal-worker-router 2>/dev/null-tmux new-session -d -c "$PWD" -s temporal-worker-router \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope router'-tmux kill-session -t temporal-worker-runner 2>/dev/null-tmux new-session -d -c "$PWD" -s temporal-worker-runner \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner'-tmux kill-session -t temporal-worker-q-capped 2>/dev/null-tmux new-session -d -c "$PWD" -s temporal-worker-q-capped \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner-llm --task-queue q_capped'-sleep 5-for s in temporal-worker-router temporal-worker-runner temporal-worker-q-capped; do- tmux capture-pane -t "$s" -p -S -15 | grep -q "Temporal Worker started" && echo "$s: started" || echo "$s: NOT started"-done-```--Reuse the existing batch bundle with a 10-item input — no need to author a bundle. `batch_temporal_describe_topics` dispatches one `act_llm_gen_text` per topic (each in its own child workflow):--```bash-cat > /tmp/batch10_inputs.json << 'EOF'-{ "topics": { "concept": "temporal_batch_test.Topic",- "content": ["science","history","music","sports","cooking","travel","cinema","biology","economics","architecture"] } }-EOF--timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \- --pipe batch_temporal_describe_topics \- --inputs /tmp/batch10_inputs.json \- --temporal --no-logo-echo "EXIT=$?"-```--Measure the throttle. The `act_llm_gen_text` activities live in **child** workflows, so aggregate scheduled→started times across all children of the run (a single `workflow show` on the root will NOT contain them). `jq`'s `fromdateiso8601` can't parse the fractional-second timestamps, so do the math in Python:--```bash-ROOT=$(temporal workflow list --limit 40 --output json | jq -r '.[] | select(.type.name=="wf_pipe_run") | .execution.workflowId' | head -1)-tmpf=$(mktemp)-for wf in $(temporal workflow list --limit 80 --output json | jq -r --arg r "$ROOT" '.[] | select(.execution.workflowId|startswith($r)) | .execution.workflowId' | sort -u); do- temporal workflow show --workflow-id "$wf" --output json | jq -c '- [.events[]? | select(.activityTaskScheduledEventAttributes.activityType.name=="act_llm_gen_text" and .activityTaskScheduledEventAttributes.taskQueue.name=="q_capped") | {sid:.eventId, sched:.eventTime}] as $sch- | [.events[]? | select(.activityTaskStartedEventAttributes!=null) | {sid:(.activityTaskStartedEventAttributes.scheduledEventId), started:.eventTime}] as $st- | $sch | map(. as $s | {sched:$s.sched, started:(($st[] | select(.sid==$s.sid) | .started) // null)}) | .[]' >> "$tmpf"-done-.venv/bin/python - "$tmpf" << 'PY'-import sys, json-from datetime import datetime, timezone-def t(s):- # Temporal eventTime is RFC 3339 UTC: it may carry 0 or 1-9 fractional digits.- # strptime's %f only accepts 1-6, so normalize: drop the Z, pad/truncate the- # fraction to microseconds, and pick the format with/without a fractional part.- s = s.rstrip("Z")- if "." in s:- head, frac = s.split(".", 1)- s = f"{head}.{(frac + '000000')[:6]}"- fmt = "%Y-%m-%dT%H:%M:%S.%f"- else:- fmt = "%Y-%m-%dT%H:%M:%S"- return datetime.strptime(s, fmt).replace(tzinfo=timezone.utc).timestamp()-recs=[json.loads(l) for l in open(sys.argv[1]) if l.strip()]-recs=[r for r in recs if r.get("started")]-starts=sorted(t(r["started"]) for r in recs)-lat=sorted(t(r["started"])-t(r["sched"]) for r in recs)-print(f"activities={len(recs)} started_span={starts[-1]-starts[0]:.2f}s max_schedule_to_start={lat[-1]:.2f}s")-print(f"schedule->start latencies={[round(x,1) for x in lat]}")-PY-rm -f "$tmpf"-```--PASS = `started_span` is clearly non-zero (≳4–5s for 10 activities at 2 RPS) with a growing tail latency (e.g. `[0,0,…,1,1,3]`); the initial token-bucket burst (~first 4 immediate) is expected. Don't pin exact timing — the server's rate-limit precision is per-second.--**Scenario E — Missing-worker negative (live, bounded)**--Route a real, triggerable activity to a queue nothing polls; the bounded `schedule_to_start_timeout` must make it fail fast instead of hanging forever.--> Do **not** use `act_jinja2_gen_text` here. `PipeJinja2` was renamed to `PipeCompose`, which renders templates **inline** via `render_template` (deterministic, in-process) — no pipe dispatches `act_jinja2_gen_text` anymore, so it can't be triggered from a bundle (the activity is vestigial: still registered in `tasks.py` and the `runner-jinja2` scope, but unreachable via any pipe). Use `act_llm_gen_text` and the stock `native_text_sequence` bundle instead — the objective (bounded timeout vs hang) is activity-agnostic.--```bash-cat > .pipelex/pipelex_temporary_override.toml << 'EOF'-[temporal.worker_config.activity_queues.act_llm_gen_text]-default = "q_orphan"--# q_orphan has a route (above) AND a queue_options entry (below) so it passes the-# orphan-queue validator; the schedule_to_start_timeout bounds the wait so a-# dispatch to a queue nothing polls fails fast instead of hanging. Keep it small-# (10s) — see the retry note below for why.-[temporal.queue_options.q_orphan]-schedule_to_start_timeout = "0:00:10"-EOF--# router reads the new routing; keep a general runner up but start NO q_orphan worker-tmux kill-session -t temporal-worker-router 2>/dev/null-tmux new-session -d -c "$PWD" -s temporal-worker-router \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope router'-tmux has-session -t temporal-worker-runner 2>/dev/null || tmux new-session -d -c "$PWD" -s temporal-worker-runner \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner'-sleep 4-pgrep -fl "q_orphan" >/dev/null && echo "WARNING: a worker polls q_orphan" || echo "confirmed: nothing polls q_orphan"--timeout 120 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds \- --pipe native_text_sequence \- --temporal --no-logo-echo "EXIT=$?"-```--Expected: non-zero exit with `activity ScheduleToStart timeout` in the message. Confirm in workflow history that the scheduled `act_llm_gen_text` event on `q_orphan` carries `scheduleToStartTimeout=10s` and fails with `TIMEOUT_TYPE_SCHEDULE_TO_START`:--```bash-WF=$(temporal workflow list --limit 20 --output json | jq -r '.[] | select(.type.name=="wf_pipe_router") | .execution.workflowId' | grep step_one | head -1)-temporal workflow show --workflow-id "$WF" --output json \- | jq -r '.events[]? | select(.activityTaskScheduledEventAttributes.activityType.name=="act_llm_gen_text") | "taskQueue=\(.activityTaskScheduledEventAttributes.taskQueue.name) s2s=\(.activityTaskScheduledEventAttributes.scheduleToStartTimeout)"'-temporal workflow show --workflow-id "$WF" --output json \- | jq -r '.events[]? | .activityTaskTimedOutEventAttributes.failure.timeoutFailureInfo.timeoutType // empty'-```--PASS = bounded timeout, clear error. **Note on total time:** each pipe step runs as a child workflow that **retries on failure** (≈3 attempts), so the run takes ≈`maxAttempts × schedule_to_start` (≈30s at a 10s bound), not a single bound — that's why the bound is kept small. It is still bounded; the point is "fails fast per attempt, no infinite hang." FAIL = the run exceeds `maxAttempts × bound` substantially (true hang) or the error is an unclear wrapper with no `ScheduleToStart`.--**Scenario F — CLI startup typo (no live submission)**--`pipelex worker --task-queue typo_q` (where `typo_q` isn't in `queue_options`, `activity_queues`, or `default_task_queue`) must exit non-zero with the Phase 4 "did you mean?" suggestion. No tmux session needed — run inline:--```bash-.venv/bin/python -m pipelex.temporal.worker_cli --task-queue temporal_task_queu 2>&1 | tail -20-```--PASS = process exits with non-zero status and the error contains:--```-WorkerTaskQueueUnknownError: --task-queue 'temporal_task_queu' is not referenced by any routing or options entry.-Known queues: [...]. Did you mean 'temporal_task_queue'?-```--FAIL = process starts up, hangs, or errors with a different message.--Sanity-check that a **known** queue is accepted by the validator. Target `temporal_task_queue` (the `default_task_queue`) — it is always known regardless of which scenario's override is currently loaded. Do **not** use `q_llm` here: each scenario's override is written with `cat >` (full replace), so by this point the active override is Scenario E's, which declares only `q_orphan` — `q_llm` would be (correctly) rejected as unknown.--```bash-OUT=$(timeout 60 .venv/bin/python -m pipelex.temporal.worker_cli \- --task-queue temporal_task_queue --is-unit-testing 2>&1); RC=$?-if echo "$OUT" | grep -q "WorkerTaskQueueUnknownError"; then- echo "Scenario F sanity FAIL — known queue 'temporal_task_queue' wrongly rejected by the task-queue validator"-else- echo "Scenario F sanity PASS — 'temporal_task_queue' passed the task-queue validator (no WorkerTaskQueueUnknownError; rc=$RC)"-fi-```--PASS = no `WorkerTaskQueueUnknownError` — the run proceeds *past* the task-queue validator, which runs early (`worker_cli.py`, before library load and `Worker(...)` construction) and only guards the `--task-queue` name; it does **not** boot a worker by itself.--> Note: under `--is-unit-testing` the run then dies on an unrelated `RuntimeError: Failed validating workflow <test-only workflow>` (the exact one varies, e.g. `wf_test_content_generator_child` or `wf_test_structured_output_cross_process`) — temporalio sandbox validation of a test-only workflow, queue-name-independent and orthogonal to this check. (If that test-workflow issue is ever fixed, the known queue boots a real polling worker instead and the `timeout 60` fires at `rc=124`.) Either outcome is "past the validator" and PASSES; only `WorkerTaskQueueUnknownError` is a Scenario F failure. The sandbox-validation issue is tracked as a product follow-up in `wip/temporal-e2e-validate-skill.md`.--**Step 9.t — Teardown**--```bash-.venv/bin/python -c "from pathlib import Path; Path('.pipelex/pipelex_temporary_override.toml').unlink(missing_ok=True)"-rm -f /tmp/batch10_inputs.json-for q in q-llm q-llm-anthropic q-imggen q-extract q-capped; do- tmux kill-session -t "temporal-worker-$q" 2>/dev/null-done-tmux kill-session -t temporal-worker-runner 2>/dev/null-tmux kill-session -t temporal-worker-router 2>/dev/null-tmux new-session -d -c "$PWD" -s temporal-worker-router \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope router'-tmux new-session -d -c "$PWD" -s temporal-worker-runner \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner'-sleep 4-tmux capture-pane -t temporal-worker-router -p -S -10 | grep "Temporal Worker started"-```diff --git a/.claude/skills/temporal-e2e-validate/references/routing-battery.md b/.claude/skills/temporal-e2e-validate/references/routing-battery.mddeleted file mode 100644index 94ac3f842..000000000--- a/.claude/skills/temporal-e2e-validate/references/routing-battery.md+++ /dev/null@@ -1,241 +0,0 @@-# Temporal E2E — Mode 2 Step 8: routing validation battery (v1)--> Reference file for the **temporal-e2e-validate** skill — Mode 2, Step 8.-> **Run `mode-2-setup.md` first** (Steps 1–2: Temporal server + worker processes).-> The **Timeouts policy** and the **surface-results-immediately** rule in `SKILL.md` apply to every command here.-> Sibling reference files: `mode-2-tiers.md` (Steps 3–7 — Tiers 1–14), `queue-options-battery.md` (Step 9 — v2 queue options / worker-runtime profiles).--### Step 8: Routing validation battery — does `activity_queues` actually isolate workers?--This step validates the v1 per-activity, per-handle routing (PR #879) end-to-end-against a real Temporal server. It is **opt-in** — Tiers 1–11 in `mode-2-tiers.md` all run with-the default empty `activity_queues`, where every activity lands on-`worker_config.default_task_queue` and either of the split workers picks it up. Step 8-proves the routing feature works as advertised when operators actually configure-it: each activity (and, in Tier 10b, each model handle) lands on its dedicated-worker pool, never on the fallback runner.--Step 8 is **live-only** for its spend-bearing assertions. Since Part B, dry-run-also dispatches `act_*` activities (the cogt leaf mocks inside them), so routing-CAN be observed dry — but this battery predates that and its arms assert on real-provider effects (model handles, spend), so run it live as written.--**Step 8.0 — Preflight + setup**--Verify base split workers from Step 2 (`mode-2-setup.md`) are still alive (router +-runner). If not, go back and start them.--Write the routing override:--```bash-cat > .pipelex/pipelex_temporary_override.toml << 'EOF'-[temporal.worker_config.activity_queues.act_llm_gen_text]-default = "q_inference"-by_handle = { "claude-4.6-sonnet" = "q_handle_a", "gemini-flash-latest" = "q_handle_b" }--[temporal.worker_config.activity_queues.act_img_gen_images]-default = "q_image_gen"--[temporal.worker_config.activity_queues.act_extract_gen_extract_pages]-default = "q_extract"--[temporal.worker_config.activity_queues.act_render_page_views]-default = "q_extract"--# Every queue named under activity_queues must have a matching-# [temporal.queue_options.<q>] entry. Empty stanza = "use worker_config-# defaults for this queue" — required by the orphan-queue validator.-[temporal.queue_options.q_inference]-[temporal.queue_options.q_handle_a]-[temporal.queue_options.q_handle_b]-[temporal.queue_options.q_image_gen]-[temporal.queue_options.q_extract]-EOF-```--The override needs to be visible to the **router** process (where `resolve_queue`-runs inside the workflow). The dedicated activity workers don't read this config-— they just listen on the queue named in their `--task-queue` flag. Restart the-router so it reloads config:--```bash-tmux kill-session -t temporal-worker-router-tmux new-session -d -c "$PWD" -s temporal-worker-router \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope router'-sleep 4-tmux capture-pane -t temporal-worker-router -p -S -30 | grep "Temporal Worker started"-```--Spawn the dedicated activity workers, one per named queue:--```bash-for q in q_inference q_handle_a q_handle_b q_image_gen q_extract; do- session="temporal-worker-${q//_/-}"- tmux has-session -t "$session" 2>/dev/null && tmux kill-session -t "$session"- tmux new-session -d -c "$PWD" -s "$session" \- ".venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope runner --task-queue $q"-done-sleep 5-for q in q_inference q_handle_a q_handle_b q_image_gen q_extract; do- session="temporal-worker-${q//_/-}"- echo "=== $session ==="- tmux capture-pane -t "$session" -p -S -20 | grep -B 1 -A 1 "started for"-done-```--Each session should report `Temporal Worker started for '<queue>'`. If any-worker failed to start, stop and diagnose before running the sub-tiers.--**Tier 10a — Multi-activity isolation (live)**--Runs an image-generation pipeline that dispatches `act_llm_gen_text` (handle-resolves to `gpt-4o-mini` via `@default-small` — not in `by_handle`, falls through-to activity default `q_inference`) AND `act_img_gen_images` (default → `q_image_gen`).-Both activities must land on their dedicated workers; the inference runner from-Step 2 (`mode-2-setup.md`) must see 0 hits for either.--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/pipes/pipelines/crazy_image_generation.mthds \- --pipe generate_crazy_image \- --temporal --no-logo --graph-```--After completion:--```bash-INF=$(tmux capture-pane -t temporal-worker-q-inference -p -S -500 | grep -c "act_llm_gen_text")-IMG=$(tmux capture-pane -t temporal-worker-q-image-gen -p -S -500 | grep -c "act_img_gen_images")-INF_IMG=$(tmux capture-pane -t temporal-worker-q-inference -p -S -500 | grep -c "act_img_gen_images")-IMG_LLM=$(tmux capture-pane -t temporal-worker-q-image-gen -p -S -500 | grep -c "act_llm_gen_text")-RUN_LLM=$(tmux capture-pane -t temporal-worker-runner -p -S -500 | grep -c "act_llm_gen_text")-RUN_IMG=$(tmux capture-pane -t temporal-worker-runner -p -S -500 | grep -c "act_img_gen_images")-echo "q_inference llm=$INF img=$INF_IMG (want llm≥1, img=0)"-echo "q_image_gen llm=$IMG_LLM img=$IMG (want llm=0, img≥1)"-echo "runner llm=$RUN_LLM img=$RUN_IMG (want llm=0, img=0)"-if [ "$INF" -ge 1 ] && [ "$IMG" -ge 1 ] && [ "$INF_IMG" -eq 0 ] && [ "$IMG_LLM" -eq 0 ] && [ "$RUN_LLM" -eq 0 ] && [ "$RUN_IMG" -eq 0 ]; then- echo "Tier 10a PASS: multi-activity isolation verified"-else- echo "Tier 10a FAIL — see hit table above"-fi-```--**Tier 10b — Per-handle routing (live)**--Runs `per_handle_routing.mthds`, a 2-step PipeSequence that dispatches `act_llm_gen_text`-twice — step 1 with `model = "claude-4.6-sonnet"`, step 2 with `model = "gemini-flash-latest"`.-The override maps each handle to its own queue via `by_handle`. After execution,-each per-handle worker should show exactly 1 hit for `act_llm_gen_text`, and-`q_inference` (the activity default) should see 0 — proving the per-handle layer-wins over the activity default.--```bash-timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/per_handle_routing.mthds \- --pipe per_handle_routing_sequence \- --temporal --no-logo --graph-```--After completion:--```bash-HA=$(tmux capture-pane -t temporal-worker-q-handle-a -p -S -500 | grep -c "act_llm_gen_text")-HB=$(tmux capture-pane -t temporal-worker-q-handle-b -p -S -500 | grep -c "act_llm_gen_text")-INF=$(tmux capture-pane -t temporal-worker-q-inference -p -S -500 | grep -c "act_llm_gen_text")-RUN=$(tmux capture-pane -t temporal-worker-runner -p -S -500 | grep -c "act_llm_gen_text")-echo "q_handle_a (claude) hits=$HA (want ≥1)"-echo "q_handle_b (gemini) hits=$HB (want ≥1)"-echo "q_inference hits=$INF (want delta=0 from Tier 10a baseline — by_handle wins)"-echo "runner hits=$RUN (want delta=0 from Tier 10a baseline)"-if [ "$HA" -ge 1 ] && [ "$HB" -ge 1 ]; then- echo "Tier 10b PASS: per-handle routing verified (both handles landed on their dedicated workers)"-else- echo "Tier 10b FAIL — see hit table above"-fi-```--Note: `q_inference` and `runner` hit counts include Tier 10a's `act_llm_gen_text`-dispatch (1 from Tier 10a's `@default-small` → `gpt-4o-mini` call landing on-`q_inference`). The Tier 10b assertion is that neither counter incremented after-this run — i.e. both Tier 10b dispatches landed on their per-handle workers.-If you ran Step 8 from a fresh session restart, `q_inference` should be exactly-1 (from Tier 10a) and `runner` should be 0.--**Tier 10c — Two activities, one route (live)**--**Credentials note.** This repo's `.env` provides `PIPELEX_GATEWAY_API_KEY`-and `PIPELEX_INFERENCE_API_KEY` — the Pipelex Gateway proxies extract-backends (including Azure Document Intelligence) without needing direct-`AZURE_DOCUMENT_INTELLIGENCE_*` env vars. Use the-`azure-document-intelligence` handle directly; do **NOT** substitute-`mistral-ocr` or `deepseek-ocr` even when those handles seem available.-User preference: extract = Azure Doc Intel via the gateway, period. (The-`mistral-ocr` handle defined in `mistral.toml` is not auto-registered in-the deck on this setup — `is_model_handle_defined` returns False — so-trying it produces `Extract choice '...mistral-ocr...' was not found in the-model deck`. Skip that path.)--The existing bundle at-`tests/integration/pipelex/temporal/library_crate/pdf_extract_page_views.mthds`-already references `@default-extract-document` (→ `azure-document-intelligence`)-and sets `page_views = true`, exercising both activities. There is no-matching inputs JSON for the CLI run — write one at `/tmp/pdf_extract_inputs.json`-pointing at any `tests/data/documents/*.pdf` (e.g. `Job-Offer.pdf`):--```bash-cat > /tmp/pdf_extract_inputs.json << EOF-{- "source_pdf": {- "concept": "native.Document",- "content": {- "url": "$PWD/tests/data/documents/Job-Offer.pdf"- }- }-}-EOF--timeout 600 .venv/bin/pipelex run bundle \- tests/integration/pipelex/temporal/library_crate/pdf_extract_page_views.mthds \- --pipe pdf_extract_with_page_views \- --inputs /tmp/pdf_extract_inputs.json \- --temporal --no-logo --graph-```--After completion:--```bash-EXTR=$(tmux capture-pane -t temporal-worker-q-extract -p -S -500 | grep -c "act_extract_gen_extract_pages")-RNDR=$(tmux capture-pane -t temporal-worker-q-extract -p -S -500 | grep -c "act_render_page_views")-RUN_EXTR=$(tmux capture-pane -t temporal-worker-runner -p -S -500 | grep -c "act_extract_gen_extract_pages")-RUN_RNDR=$(tmux capture-pane -t temporal-worker-runner -p -S -500 | grep -c "act_render_page_views")-echo "q_extract: extract=$EXTR (want ≥1) render=$RNDR (want ≥1)"-echo "runner: extract=$RUN_EXTR render=$RUN_RNDR (want both 0)"-if [ "$EXTR" -ge 1 ] && [ "$RNDR" -ge 1 ] && [ "$RUN_EXTR" -eq 0 ] && [ "$RUN_RNDR" -eq 0 ]; then- echo "Tier 10c PASS: both extract activities routed to q_extract (activity-default fallback for routing_key=None works)"-else- echo "Tier 10c FAIL — see hit table above"-fi-```--If the pipeline ever errors with `Extract choice '...' was not found in the-model deck`, the deck is not loading Azure Doc Intel — fix the deck before-falling back to a different handle; do not substitute another OCR backend.--**Step 8.d — Teardown**--Restore the default empty `activity_queues` and kill the dedicated workers:--```bash-.venv/bin/python -c "from pathlib import Path; Path('.pipelex/pipelex_temporary_override.toml').unlink(missing_ok=True)"-for q in q_inference q_handle_a q_handle_b q_image_gen q_extract; do- tmux kill-session -t "temporal-worker-${q//_/-}" 2>/dev/null-done-# Restart the router so it reverts to empty activity_queues-tmux kill-session -t temporal-worker-router-tmux new-session -d -c "$PWD" -s temporal-worker-router \- '.venv/bin/python -m pipelex.temporal.worker_cli --is-not-sandboxed --scope router'-sleep 4-tmux capture-pane -t temporal-worker-router -p -S -10 | grep "Temporal Worker started"-```--Optionally re-run Tier 1 (default routing, `mode-2-tiers.md`) to confirm the baseline is restored.diff --git a/.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py b/.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.pydeleted file mode 100755index a862ca7b6..000000000--- a/.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py+++ /dev/null@@ -1,143 +0,0 @@-#!/usr/bin/env python3-"""Numeric cross-worker cost assertion for temporal-e2e-validate Tier 8b.--Turns the old "eyeball that a cost table rendered" check into a real assertion:-reads the NDJSON usage events a split-worker run produced, sums input/output-tokens straight from the events, and (optionally) cross-checks them against the-un-truncated CSV cost report and against expected per-call counts. The terminal-Rich table truncates wide columns to ``0 … …``, so this is the only reliable way-to confirm the real numbers cross the runner boundary and aggregate.--Pure stdlib (no pipelex import) so it runs fast and never perturbs the run.--Usage:- .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \- --run-dir .pipelex/traces/<RUN_ID> \- [--expected-events N] [--expected-input N] [--expected-output N] \- [--expected-model-type llm|img_gen|extract|search] \- [--reports-dir reports] [--require-fallback]--Exit code 0 = all assertions passed, 1 = at least one failed.-"""--from __future__ import annotations--import argparse-import csv-import json-import sys-from pathlib import Path-from typing import Any--_INPUT_KEYS = ("input",)-_OUTPUT_KEYS = ("output",)---def _load_usage_events(run_dir: Path) -> list[dict[str, Any]]:- events: list[dict[str, Any]] = []- for ndjson_path in sorted(run_dir.glob("*.ndjson")):- for line in ndjson_path.read_text(encoding="utf-8").splitlines():- line = line.strip()- if not line:- continue- event = json.loads(line)- if event.get("event_kind") == "usage_report":- events.append(event)- return events---def _sum_tokens(events: list[dict[str, Any]]) -> tuple[int, int]:- total_input = 0- total_output = 0- for event in events:- by_category = event.get("tokens_usage", {}).get("nb_tokens_by_category", {})- total_input += sum(int(by_category.get(key, 0)) for key in _INPUT_KEYS)- total_output += sum(int(by_category.get(key, 0)) for key in _OUTPUT_KEYS)- return total_input, total_output---def _csv_token_totals(reports_dir: Path) -> tuple[int, int] | None:- """Sum input-joined + output token columns across the most recent cost_report CSV, if any."""- csv_paths = sorted(reports_dir.glob("cost_report*.csv"))- if not csv_paths:- return None- latest = csv_paths[-1]- total_input = 0- total_output = 0- with latest.open(encoding="utf-8") as csv_file:- for row in csv.DictReader(csv_file):- total_input += int(float(row.get("nb_tokens_input_joined") or 0))- total_output += int(float(row.get("nb_tokens_output") or 0))- return total_input, total_output---def main() -> int:- parser = argparse.ArgumentParser(description=__doc__)- parser.add_argument("--run-dir", required=True, type=Path, help="The .pipelex/traces/<RUN_ID> directory")- parser.add_argument("--expected-events", type=int, default=None, help="Expected usage-event count (== number of inference calls)")- parser.add_argument("--expected-input", type=int, default=None, help="Expected total input tokens")- parser.add_argument("--expected-output", type=int, default=None, help="Expected total output tokens")- parser.add_argument("--expected-model-type", default=None, help="Assert every usage record has this model_type")- parser.add_argument("--reports-dir", type=Path, default=Path("reports"), help="Dir holding the un-truncated cost_report*.csv")- parser.add_argument("--require-fallback", action="store_true", help="Require at least one act_* writer_id (runner fallback engaged)")- parser.add_argument(- "--require-nonzero", action="store_true", help="Require total input AND output tokens > 0 (real-spend arms, where exact counts are unknown)"- )- args = parser.parse_args()-- run_dir: Path = args.run_dir- if not run_dir.is_dir():- print(f"FAIL: run dir not found: {run_dir}")- return 1-- events = _load_usage_events(run_dir)- failures: list[str] = []-- if not events:- print(f"FAIL: no usage_report events under {run_dir}")- return 1-- writer_ids = sorted({event.get("writer_id", "") for event in events})- model_types = sorted({event.get("tokens_usage", {}).get("model_type", "?") for event in events})- model_names = sorted({event.get("tokens_usage", {}).get("inference_model_name", "?") for event in events})- total_input, total_output = _sum_tokens(events)-- print(f"usage events : {len(events)}")- print(f"writer_ids : {writer_ids}")- print(f"model_types : {model_types}")- print(f"model_names : {model_names}")- print(f"total tokens : input={total_input} output={total_output} (joined+output={total_input + total_output})")-- csv_totals = _csv_token_totals(args.reports_dir)- if csv_totals is not None:- csv_input, csv_output = csv_totals- print(f"csv tokens : input={csv_input} output={csv_output}")- if (csv_input, csv_output) != (total_input, total_output):- failures.append(f"CSV totals {csv_totals} != NDJSON totals {(total_input, total_output)}")- else:- print(f"csv tokens : (no cost_report*.csv under {args.reports_dir}; enable is_generate_cost_report_file_enabled to cross-check)")-- if args.require_fallback and not any(writer_id.startswith("act_") for writer_id in writer_ids):- failures.append(f"no act_* writer_id (runner fallback did not engage); writer_ids={writer_ids}")- if args.require_nonzero and not (total_input > 0 and total_output > 0):- failures.append(f"expected non-zero input AND output tokens, got input={total_input} output={total_output}")- if args.expected_events is not None and len(events) != args.expected_events:- failures.append(f"event count {len(events)} != expected {args.expected_events}")- if args.expected_input is not None and total_input != args.expected_input:- failures.append(f"input tokens {total_input} != expected {args.expected_input}")- if args.expected_output is not None and total_output != args.expected_output:- failures.append(f"output tokens {total_output} != expected {args.expected_output}")- if args.expected_model_type is not None and model_types != [args.expected_model_type]:- failures.append(f"model_types {model_types} != [{args.expected_model_type!r}]")-- if failures:- print("RESULT: FAIL")- for failure in failures:- print(f" - {failure}")- return 1- print("RESULT: PASS")- return 0---if __name__ == "__main__":- sys.exit(main())diff --git a/.claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py b/.claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.pydeleted file mode 100644index 142b6d0da..000000000--- a/.claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py+++ /dev/null@@ -1,69 +0,0 @@-# ruff: noqa: INP001-r"""Submitter that mirrors `pipelex run bundle` but attaches a DeliveryAssignment so-`act_deliver` fires on the runner worker. This forces Temporal's data converter on-the runner process to deserialize a hydrated PipeOutput — the path that surfaces-the cross-process dynamic-class registry bug when the bundle defines a dynamic-concept (e.g. Greeting) the runner has never registered.--Equivalent to the cloud / webhook submission path (`start_pipeline` with a-WebhookTarget): the runner must decode the dynamic concept class even though the-crate was only loaded on the router process.--Usage (with both scoped workers up — router + runner on temporal_task_queue):-- .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/repro_runner_registry_bug.py \\- --bundle tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds \\- --pipe dynamic_greeting_sequence--Defaults reproduce the canonical Greeting case.-"""--import argparse-import asyncio-from pathlib import Path--from pipelex.cli.cli_factory import make_pipelex_for_cli-from pipelex.cli.error_handlers import ErrorContext-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, StorageTarget-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipelex import Pipelex-from pipelex.pipeline.runner import PipelexMTHDSProtocol--DEFAULT_BUNDLE = "tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds"-DEFAULT_PIPE = "dynamic_greeting_sequence"---async def submit(bundle: str, pipe_code: str) -> None:- bundle_content = Path(bundle).read_text(encoding="utf-8")- runner = PipelexMTHDSProtocol(- bundle_uris=[bundle],- pipe_run_mode=PipeRunMode.DRY,- execution_config=None,- library_dirs=None,- )- delivery = DeliveryAssignment(storage=StorageTarget())- print(f"Submitting pipe='{pipe_code}' with delivery_assignment={delivery!r}")- response = await runner.execute(- pipe_code=pipe_code,- mthds_contents=[bundle_content],- inputs=None,- delivery_assignment=delivery,- )- print(f"OK: pipeline_run_id={response.pipeline_run_id} state={response.state}")---def main() -> None:- parser = argparse.ArgumentParser(description=__doc__)- parser.add_argument("--bundle", default=DEFAULT_BUNDLE)- parser.add_argument("--pipe", default=DEFAULT_PIPE, dest="pipe_code")- args = parser.parse_args()-- make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN, library_dirs=None, temporal_enabled=True)- try:- asyncio.run(submit(bundle=args.bundle, pipe_code=args.pipe_code))- finally:- Pipelex.teardown_if_needed()---if __name__ == "__main__":- main()diff --git a/.claude/skills/temporal-e2e-validate/scripts/submit_dry_validate.py b/.claude/skills/temporal-e2e-validate/scripts/submit_dry_validate.pydeleted file mode 100644index 82ae00e41..000000000--- a/.claude/skills/temporal-e2e-validate/scripts/submit_dry_validate.py+++ /dev/null@@ -1,93 +0,0 @@-# ruff: noqa: INP001-r"""Tier-2d submitter: dispatch the dry-run+validation job as ONE Temporal activity.--Dispatches the one-step wrapper workflow ``WfDryValidate`` (which runs the single-``act_dry_validate`` activity fully in-process on a worker) and awaits the-``{status map, GraphSpec}`` result in one round-trip — the Temporal-enabled ``/validate``-dispatch shape. Prints the per-pipe status map and writes the GraphSpec JSON when produced.--Usage (with the Temporal server + split workers up — see mode-2-setup.md):-- .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/submit_dry_validate.py \\- --bundle tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds \\- --pipe temporal_parallel_test.temporal_parallel_sequence--Exit codes: 0 = validation succeeded (graph may still be None — best-effort); 1 = validation-failed (the structured ErrorReport is printed).-"""--import argparse-import asyncio-import sys-from pathlib import Path--from pipelex.cli.cli_factory import make_pipelex_for_cli-from pipelex.cli.error_handlers import ErrorContext-from pipelex.pipelex import Pipelex-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.tprl_pipe.act_dry_validate import DryValidateArg-from pipelex.temporal.tprl_pipe.dry_validate_dispatch import dispatch_dry_validate--DEFAULT_BUNDLE = "tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds"-DEFAULT_PIPE = "temporal_parallel_test.temporal_parallel_sequence"-DEFAULT_GRAPH_OUT = "/tmp/tier2d-graph-spec.json" # noqa: S108---async def submit(bundle: str, pipe_code: str | None, allow_signatures: bool, graph_out: str) -> int:- bundle_content = Path(bundle).expanduser().read_text(encoding="utf-8")- arg = DryValidateArg(- mthds_contents=[bundle_content],- allow_signatures=allow_signatures,- pipe_code=pipe_code,- )- print(f"Dispatching wf_dry_validate for bundle='{bundle}' pipe_code={pipe_code!r} allow_signatures={allow_signatures}")- try:- result = await dispatch_dry_validate(arg)- except WorkflowExecutionError as exc:- print("VALIDATION FAILED — structured ErrorReport recovered from the workflow failure:")- print(f" error_type: {exc.error_report.error_type if exc.error_report else 'n/a'}")- print(f" error_domain: {exc.error_report.error_domain if exc.error_report else 'n/a'}")- print(f" message: {exc.message}")- return 1-- print(f"STATUS MAP ({len(result.dry_run_outputs)} pipes):")- for pipe_ref, output in sorted(result.dry_run_outputs.items()):- suffix = f" — {output.error_message}" if output.error_message else ""- print(f" {output.status:<8} {pipe_ref}{suffix}")-- if result.graph_spec is None:- print("GRAPH: None (best-effort — validation still succeeded)")- else:- print(f"GRAPH: graph_id={result.graph_spec.graph_id} nodes={len(result.graph_spec.nodes)} edges={len(result.graph_spec.edges)}")- graph_path = Path(graph_out).expanduser()- try:- graph_path.parent.mkdir(parents=True, exist_ok=True)- graph_path.write_text(result.graph_spec.model_dump_json(indent=2), encoding="utf-8")- except OSError as exc:- # The graph file is a best-effort artifact: an IO failure must not be reported as a validation failure (exit 1)- print(f"GRAPH JSON write to {graph_path} failed: {exc}")- else:- print(f"GRAPH JSON written to {graph_path}")- return 0---def main() -> None:- parser = argparse.ArgumentParser(description=__doc__)- parser.add_argument("--bundle", default=DEFAULT_BUNDLE)- parser.add_argument("--pipe", default=DEFAULT_PIPE, dest="pipe_code")- parser.add_argument("--no-pipe", action="store_true", help="Do not pass a pipe_code (use the bundle's main_pipe, if any)")- parser.add_argument("--allow-signatures", action="store_true")- parser.add_argument("--graph-out", default=DEFAULT_GRAPH_OUT)- args = parser.parse_args()-- pipe_code: str | None = None if args.no_pipe else args.pipe_code- make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN, library_dirs=None, temporal_enabled=True)- try:- exit_code = asyncio.run(submit(bundle=args.bundle, pipe_code=pipe_code, allow_signatures=args.allow_signatures, graph_out=args.graph_out))- finally:- Pipelex.teardown_if_needed()- sys.exit(exit_code)---if __name__ == "__main__":- main()diff --git a/.claude/skills/temporal-test-crate/SKILL.md b/.claude/skills/temporal-test-crate/SKILL.mddeleted file mode 100644index 55b66333e..000000000--- a/.claude/skills/temporal-test-crate/SKILL.md+++ /dev/null@@ -1,138 +0,0 @@-----name: temporal-test-crate-description: "Run and diagnose the LibraryCrate integration tests for Temporal. Tests that PipeSequence controllers execute on Temporal workers via LibraryCrate propagation, including concurrent isolation tests for conflicting concepts and pipes. Use when the user says 'test temporal crate', 'run crate tests', 'isolation tests', or wants to verify LibraryCrate on Temporal works."------# Temporal LibraryCrate Integration Tests--Run the integration tests that verify LibraryCrate propagation and per-workflow isolation through Temporal workflows.--## How to narrate the test run--Before running the tests, print a brief schematic explanation for the user so they understand what they're about to see. Use this narration:-------### What we're testing--These tests verify that **LibraryCrate** — the serializable snapshot of pipes, concepts, and dynamic classes — propagates correctly from the client to Temporal workers, and that concurrent workflows remain isolated from each other.--**The components and how they interact:**--```-Client (test fixture) Temporal Worker-───────────────────── ───────────────--1. Load .mthds bundle-2. Build LibraryCrate (pipes,- concepts, fingerprint)-3. Build PipeJob with crate attached-4. Submit workflow ──────────────────► 5. WfPipeRouter.run() receives PipeJob- 6. load_from_crate() → registers pipes- into a scoped Library (ContextVar)- 7. Dynamic concept classes generated- into a scoped ClassRegistry (ContextVar)- 8. WorkingMemory hydrated (deferred)- 9. PipeSequence executes child pipes- via get_required_pipe() from the- scoped library- 10. PipeOutput returned to client-```--**The 5 test suites (19 tests total):**--| Suite | What it proves | Key risk if broken |-|-------|---------------|--------------------|-| **TestWfLibraryCrate** (5 tests) | Crate structure is valid, PipeSequence runs end-to-end via Temporal, both `library_dirs` and `mthds_content` loading paths work. Includes a **negative test** (`test_missing_crate_fails_pipe_resolution`) that verifies a PipeJob *without* a crate correctly fails. | Pipes can't resolve on worker |-| **TestWfDeferredHydration** (2 tests) | Dynamic concept classes (e.g. `Greeting`) are generated on the worker from crate data, WorkingMemory is hydrated after class registration, StructuredContent fields are accessible. | `KajsonDecoderError: Class not found` |-| **TestWfConcurrentConceptIsolation** (4 tests) | Two workflows define conflicting `Result` concepts (different fields) and run simultaneously on one worker. Per-workflow ClassRegistry scoping keeps them isolated. Repeated 5x to catch races. | Silent data corruption — wrong fields |-| **TestWfConcurrentPipeIsolation** (4 tests) | Two workflows define conflicting `shared_step` pipes (different prompts) and run simultaneously. Per-workflow Library scoping ensures each resolves its own pipe. Repeated 5x. | Wrong pipe executed silently |-| **TestWfMultiConceptIsolation** (4 tests) | Worst case: two workflows each define `Profile` AND `Summary` with incompatible structures. 6 concurrent workflows stress-test ContextVar scoping. | Multiple classes cross-contaminate |-------After running the tests, report results with a summary table showing pass/fail per suite.--## Run the tests--### Step 1: Dry run (default)--Always start with dry mode — fast, no API calls:--```bash-.venv/bin/pytest -x -v -s tests/integration/pipelex/temporal/library_crate/ -m temporal-```--To run only the isolation tests:--```bash-.venv/bin/pytest -x -v -s tests/integration/pipelex/temporal/library_crate/ -m temporal -k "Concurrent or Multi"-```--### Step 2: Live run (optional)--After a successful dry run, ask the user if they want to run in **live mode**. Live mode makes real LLM API calls and exercises realistic concurrency timing — it's the closest thing to production behavior:--```bash-.venv/bin/pytest -x -v -s tests/integration/pipelex/temporal/library_crate/ -m temporal --pipe-run-mode live-```--When reporting live-mode results, show:-- Pass/fail per test with the test name-- For passing tests, confirm that structured outputs had the expected fields (e.g. "Alpha Result had score + label", "Beta Result had value + confidence + is_valid")-- For concurrent tests, confirm that both workflows returned correct results simultaneously-- Total wall-clock time (live mode is slower due to real API calls)--## Expected warnings in test output--`test_missing_crate_fails_pipe_resolution` is a **negative test** — it intentionally submits a PipeJob *without* a LibraryCrate and asserts that `WorkflowFailureError` is raised. During this test, Temporal logs a `WARNING Failed activation on workflow` with a full traceback ending in `RuntimeError: No current library set`. **This is expected behavior, not a failure.** The test passes (PASSED) because the expected error was correctly raised.--## Prerequisites--```bash-# Verify venv and temporalio-.venv/bin/python -c "import temporalio; print(f'temporalio {temporalio.__version__}')"-# Verify test bundles exist-ls tests/integration/pipelex/temporal/library_crate/*.mthds-```--## Interpreting failures--| Error | Cause | Action |-|-------|-------|--------|-| `PipeNotFoundError` on worker | LibraryCrate not loaded or not propagated | Check `WfPipeRouter.run()` loads crate; check `library_crate` is set on PipeJob |-| `KajsonDecoderError: Class 'X' not found` | Dynamic concept classes not registered on worker | Check deferred hydration path in `WfPipeRouter.run()` |-| `RuntimeError: Failed decoding arguments` | Temporal can't deserialize PipeJob | Check Kajson data converter; may be a serialization issue with LibraryCrate |-| `WorkflowFailureError` wrapping `TemporalError` | Pipe execution failed on worker | Read the inner error — it's the real cause |-| Fixture setup error in `pipe_job_from_*` | Library loading failed before Temporal dispatch | Check bundle file exists and is valid MTHDS |-| `AssertionError: StructuredContent missing field` | ClassRegistry scoping failed — wrong class used | Per-workflow ContextVar is leaking; check `set_current_library` / `clear_current_library` |-| `AssertionError` in repeated/high-concurrency tests | Intermittent ContextVar race condition | Scoping mechanism has a race under load |--## Interactive debugging--For tmux-based 3-terminal debugging (server + worker + submitter), use the `/temporal-e2e-validate` skill (Mode 2) instead.--## Test files--| File | Purpose |-|------|---------|-| `tests/integration/pipelex/temporal/library_crate/test_wf_library_crate.py` | Single-workflow crate propagation (native Text concepts) |-| `tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py` | Deferred hydration with dynamic concept (Greeting) |-| `tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_concept_isolation.py` | Two workflows with conflicting 'Result' concepts |-| `tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_pipe_isolation.py` | Two workflows with conflicting 'shared_step' pipes |-| `tests/integration/pipelex/temporal/library_crate/test_wf_multi_concept_isolation.py` | Worst-case multi-class conflicts (Profile + Summary) |-| `tests/integration/pipelex/temporal/library_crate/conftest.py` | PipeJob fixtures with LibraryCrate for all scenarios |-| `tests/integration/pipelex/temporal/test_data.py` | Test constants for all scenarios |--## Bundle files--| Bundle | Domain | Concepts | Pipes |-|--------|--------|----------|-------|-| `native_text_sequence.mthds` | `native_text_test` | (native Text only) | native_text_sequence, step_one, step_two |-| `dynamic_concept_sequence.mthds` | `dynamic_concept_test` | Greeting(message, language) | dynamic_greeting_sequence, generate_greeting, summarize_greeting |-| `conflict_concept_alpha.mthds` | `conflict_alpha` | Result(score, label) | alpha_pipeline, alpha_generate, alpha_summarize |-| `conflict_concept_beta.mthds` | `conflict_beta` | Result(value, confidence, is_valid) | beta_pipeline, beta_generate, beta_summarize |-| `conflict_pipe_alpha.mthds` | `pipe_conflict_alpha` | (native Text only) | pipe_alpha_pipeline, shared_step, alpha_finalize |-| `conflict_pipe_beta.mthds` | `pipe_conflict_beta` | (native Text only) | pipe_beta_pipeline, shared_step, beta_finalize |-| `multi_concept_alpha.mthds` | `multi_alpha` | Profile(name, age), Summary(headline, body) | multi_alpha_pipeline, generate_profile, generate_summary, finalize |-| `multi_concept_beta.mthds` | `multi_beta` | Profile(title, department, level), Summary(content) | multi_beta_pipeline, generate_profile, generate_summary, finalize |diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.tomlindex de39d5c70..75af5d268 100644--- a/.pipelex/pipelex.toml+++ b/.pipelex/pipelex.toml@@ -223,130 +223,10 @@ is_dump_response_enabled = falseis_dump_error_enabled = false####################################################################################################-# Temporal Config+# Plugins####################################################################################################-[temporal]-is_enabled = false--# Custom Temporal search attributes Pipelex populates on every workflow start-# so the dashboard can group/filter by PipeCode, PipelineRunId, SessionId,-# UserId, and DomainCode. Set `enabled = false` to skip both the worker-boot-# registration check and the per-workflow attribute attachment — useful when-# the operator owns the namespace policy and can't grant search-attribute-# admin to the worker.-[temporal.search_attributes]-enabled = false-attributes = ["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"]--[temporal.payload_codec_config]-is_enabled = false-size_threshold = 1_048_576 # in bytes (~1 MB, well under Temporal's 2 MB hard limit)-storage_prefix = "temporal-payloads/"--[temporal.payload_codec_config.storage_provider_config]-method = "local"--[temporal.payload_codec_config.storage_provider_config.local]-uri_format = "{hash}"-local_storage_path = ".pipelex/temporal-payload-store"--[temporal.worker_config]-default_task_queue = "temporal_task_queue"-workflow_execution_timeout = "1:00:00" # workflow-level budget (not the activity timeout)-run_timeout = "1:00:00"-task_timeout = "0:00:10"-start_delay = "0:00:00"-rpc_timeout = "1:00:00"-# Baseline activity timeout. Per-queue and per-handle overlays in-# [temporal.queue_options] / activity_queues.*.handle_options can override.-default_activity_start_to_close_timeout = "0:10:00"-# Per-activity, per-handle routing. Empty by default — every activity lands on-# `default_task_queue` above (same as legacy behavior). Populate this to route-# activities to dedicated worker pools (e.g. OpenAI vs Anthropic, a heavy-payload-# image pool, isolated Mistral OCR), with optional per-handle queue overrides and-# rare per-handle option overrides. See pipelex/kit/configs/pipelex.toml for the-# full schema and examples.-[temporal.worker_config.activity_queues]--[temporal.worker_config.retry_policy_config]-initial_interval = "0:00:03"-backoff_coefficient = 2.0-maximum_interval = "unlimited"-maximum_attempts = 3-non_retryable_error_types = [- "DryRunMockBuildError",- "DryRunObjectFidelityError",- "ExtractHandleNotFoundError",- "FileNotFoundError",- "ImgGenHandleNotFoundError",- "LLMHandleNotFoundError",- "ModelNotFoundError",- "ValidationError",-]-# Per-queue / per-handle overlays add to the list above via the layer-specific-# `non_retryable_error_types_extra` field (see the commented example below).-# That field belongs only on overlays — the baseline schema deliberately does-# not accept it.--# Per-queue submitter options. Layer on top of worker_config defaults at-# dispatch time. Add `[temporal.queue_options.<queue>]` entries to match each-# backend pool's latency/retry profile (timeouts, retry policy, cluster-wide-# rate caps). Ships empty to match the package default — see the rationale in-# pipelex/pipelex.toml. Each deployment configures the overlay for whatever-# name it picked for `default_task_queue`.-[temporal.queue_options]--# Named worker-runtime profiles. A worker process selects one via `--profile`.-# The `default` profile below mirrors the pre-v2 hardcoded knobs, so default-# behavior is unchanged. Add more profiles to shape pools differently — e.g.-# small concurrency on a GPU pool sitting next to a high-throughput LLM pool.-# See pipelex/kit/configs/pipelex.toml for examples.-[temporal.worker_runtime_profiles]-default_profile = "default"--[temporal.worker_runtime_profiles.profiles.default]-tuning_mode = "explicit"-max_cached_workflows = 10000-max_concurrent_workflow_tasks = 1000-max_concurrent_activities = 1000-max_concurrent_local_activities = 1000-max_concurrent_workflow_task_polls = 100-max_concurrent_activity_task_polls = 100-max_activities_per_second = 1000-sticky_queue_schedule_to_start_timeout = "0:30:00"-max_heartbeat_throttle_interval = "1:00:00"-default_heartbeat_throttle_interval = "1:00:00"-graceful_shutdown_timeout = "0:30:00"--[temporal.temporal_config]-selected_server = "local"--[temporal.temporal_config.temporal_server_configs.local]-description = "default localhost"-target_host = "localhost:7233"-namespace = "default"-api_key_method = "none"-api_key_id = ""--[temporal.temporal_config.temporal_server_configs.testing]-description = "Temporal server for testing"-target_host = "${TESTING_TEMPORAL_TARGET_HOST}"-namespace = "${TESTING_TEMPORAL_NAMESPACE}"-api_key_method = "secret_provider"-api_key_id = "${TESTING_TEMPORAL_MAIN_API_KEY}"--[temporal.temporal_config.temporal_log_config]-is_workflow_info_on_message = false-is_workflow_info_on_extra = true-is_full_workflow_info_on_extra = false-is_activity_info_on_message = false-is_activity_info_on_extra = true-is_full_activity_info_on_extra = false-is_formatter_enabled = true-is_prefix_enabled = true-managed_loggers = [- "temporalio.activity",- "temporalio.workflow",- "temporalio.worker._activity",-]+[plugins]+# Names of discovered plugins to skip at startup (a denylist). Discovery is the+# source of truth for which plugins are present; this only turns one off.+disabled = []diff --git a/.vscode/settings.json b/.vscode/settings.jsonindex 87cb4fb30..f766c3264 100644--- a/.vscode/settings.json+++ b/.vscode/settings.json@@ -10,7 +10,7 @@},"ruff.configuration": "--config=pyproject.toml","mypy.enabled": true,- "mypy.runUsingActiveInterpreter": true,+ "mypy.runUsingActiveInterpreter": true,"search.exclude": {".mypy_cache/*": true,"**/dev_context": truediff --git a/CHANGELOG.md b/CHANGELOG.mdindex 3f8c40593..e8e63cc7c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,28 @@# Changelog+## [Unreleased]++### Added++- **Orchestrator-dispatched `/validate`:** A new per-call bundle-validator seam (`BundleValidatorProtocol` / `BundleValidatorRegistry`) makes `/validate` dispatch by `orchestration_mode` the way `/start` runs a pipe — `direct` validates in-process, distributed modes dispatch the job to a worker — with a byte-identical verdict across backends. The core `direct` plugin registers an in-process `DirectBundleValidator`.+- **Explicit graph targets for validation:** Graph generation during validation and dry runs now accepts an explicit `pipe_code` target, so developers can generate graphs for a specific pipe in a bundle even when no `main_pipe` is declared. This is wired through the CLI (`validate bundle --pipe <target> --graph`/`--view`), `dry_run_pipeline`, `generate_graph_for_bundle`, `generate_view_for_bundle`, and `PipelexMTHDSProtocol.validate(extra={"graph_pipe_code": "..."})`.+- **GraphSpec source path enrichment:** `GraphSpec` artifacts now include the declaration `source` file path for pipes and concepts in `pipe_registry` and `concept_registry`, powered by `LibraryCrate.source_map`.+- **CI/environment heartbeats:** Added `WAIT_WITH_HEARTBEAT` and `RUN_WITH_HEARTBEAT` Makefile macros that emit a periodic heartbeat (default every 20s) to prevent CI runners and sandboxes from timing out during long tasks; applied to the `agent-test`, `pyright`, `mypy`, and `pylint` targets.+- **Developer guidelines:** Expanded `python_standards.md` with rules on filesystem paths (`pathlib.Path` over `str`/`os.path`), data-holder shapes (`NamedTuple` vs. `pydantic.dataclasses` vs. `BaseModel`), keyword-only arguments (enforced via `make cko`/`make fko`), and custom exception placement (`exceptions.py` or `<topic>_exceptions.py`).+- **Architecture docs:** Added design documents `wip/graph/graphspec-source-map-enrichment.md` and `wip/graph/validate-explicit-graph-target.md` covering the new graph validation features.+- **Environment config:** Added an autogenerated `.codex/environments/environment.toml` to standardize the `pipelex` environment setup.++### Changed++- **Orchestration & Delivery split (pre-1.0 breaking, plugin SPI):** The flat execution-mode axis is split into an open `orchestration_mode` token (which orchestrator runs the job — `direct` in core, `temporal`/`mistralai-workflows` from plugins) and a closed `DeliveryMode` enum (the wait-semantics axis — `BLOCKING` / `FIRE_AND_FORGET`). The orchestrator plugin contract changes accordingly: `OrchestratorProtocol.run` now takes a required `delivery: DeliveryMode` keyword, an orchestrator must declare a `supports_fire_and_forget: bool` capability, and the registries are re-keyed by the open `orchestration_mode` token (the former `PipelexExecutionMode` enum is removed). A plugin built against the prior contract is incompatible and must be rebuilt. `PipelexPipeRunInput` now carries `orchestration_mode` + `delivery`.+- **Honest fire-and-forget rejection:** `/start` now rejects a fire-and-forget request when the resolved orchestrator cannot do genuine async (e.g. the in-process `direct` mode), returning a 4xx instead of silently running to completion and falsely acking.+- **`dry_run_pipeline` return shape:** `dry_run_pipeline` now returns the fully resolved, domain-qualified pipe reference alongside the `GraphSpec`.++### Fixed++- **Graph generation on bundles without a `main_pipe`:** Requesting a graph or view via the CLI no longer fails with a missing `main_pipe` error when a valid pipe is targeted with `--pipe`.+- **Developer guidelines typo:** Fixed a minor spacing typo in `pytest_standards.md`.+## [v0.35.1] - 2026-06-22### Changed@@ -184,7 +207,7 @@ This cycle reworks **cost reporting** so it survives distributed execution and s### Changed-- **Mistral Workflows integration extracted into a dedicated package.** The optional `pipelex[mistralai-workflows]` extra and the `pipelex.plugins.mistralai_workflows.*` modules have been removed from `pipelex`. The Mistral Workflows integration now ships separately as its own package — coming soon; install and import details will follow once it publishes. The framework-agnostic runtime-bridge core (boundary types, `run_pipe_via_bridge`, `PipelexExecutionMode`, `ensure_pipelex_booted`) has been promoted from `pipelex.plugins.mistralai_workflows.*` to `pipelex.runtime_bridge.*` so any host runtime — not just Mistral Workflows — can embed Pipelex. No behavior changes; activities, boundary types, and execution modes are identical.+- **Mistral Workflows integration extracted into a dedicated package.** The optional `pipelex[mistralai-workflows]` extra and the `pipelex.plugins.mistralai_workflows.*` modules have been removed from `pipelex`. The Mistral Workflows integration now ships separately as its own package, `pipelex-mistralai-workflows`. The framework-agnostic runtime-bridge core (boundary types, `run_pipe_via_bridge`, `PipelexExecutionMode`, `ensure_pipelex_booted`) has been promoted from `pipelex.plugins.mistralai_workflows.*` to `pipelex.runtime_bridge.*` so any host runtime — not just Mistral Workflows — can embed Pipelex. No behavior changes; activities, boundary types, and execution modes are identical.- **`--cost-report` removed, folded into `--costs`.** Breaking: `--cost-report/--no-cost-report` is gone from `run pipe|method|bundle`. Use `--costs` (default on) instead.@@ -315,7 +338,7 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli- **New `PipeStructure` operator** that turns text into a structured concept via a single LLM call. Takes one `Text`-compatible input (or a domain concept that `refines = "Text"`), produces any structured output with the usual multiplicity options (`Foo`, `Foo[]`, `Foo[N]`). Useful whenever the source text comes from a PDF extraction, a search result, an upstream pipe, or any non-LLM origin. Documented at [building-methods/pipes/pipe-operators/PipeStructure.md](building-methods/pipes/pipe-operators/PipeStructure.md).-- **Custom search attributes on the Temporal namespace, with a strict prerequisite at worker boot.** Pipelex sets five custom Keyword search attributes on every workflow start — `PipeCode`, `PipelineRunId`, `SessionId`, `UserId`, `DomainCode` — so the Temporal dashboard can filter, group, and audit workflows by their actual semantic identity rather than by opaque workflow ids. A real Temporal cluster rejects every `StartWorkflowExecution` RPC that references an unregistered attribute, so the check at worker boot is a hard fail (raises `SearchAttributeRegistrationError`) on a reachable namespace — the previous "warn and continue" framing was dishonest because workflows would then fail at every dispatch with a much less actionable error. The exception message embeds both the `pipelex setup-temporal-namespace` invocation and the equivalent raw `temporal operator search-attribute create` command. `RPCError` (control-plane unreachable) stays a soft fail; the in-process test server is registered up front by the integration conftest. Documented at [distributed-execution/cluster-setup.md](distributed-execution/cluster-setup.md).+- **Custom search attributes on the Temporal namespace, with a strict prerequisite at worker boot.** Pipelex sets five custom Keyword search attributes on every workflow start — `PipeCode`, `PipelineRunId`, `SessionId`, `UserId`, `DomainCode` — so the Temporal dashboard can filter, group, and audit workflows by their actual semantic identity rather than by opaque workflow ids. A real Temporal cluster rejects every `StartWorkflowExecution` RPC that references an unregistered attribute, so the check at worker boot is a hard fail (raises `SearchAttributeRegistrationError`) on a reachable namespace — the previous "warn and continue" framing was dishonest because workflows would then fail at every dispatch with a much less actionable error. The exception message embeds both the `pipelex setup-temporal-namespace` invocation and the equivalent raw `temporal operator search-attribute create` command. `RPCError` (control-plane unreachable) stays a soft fail; the in-process test server is registered up front by the integration conftest.- **`[temporal.search_attributes]` config block** — master `enabled` toggle and an `attributes` subset selector for the five built-ins. `enabled = false` skips both the worker-boot registration check and the per-workflow attribute attachment (use this when the namespace policy is owned by a third party who will not register attributes). The validator rejects unknown attribute names so typos like `"PipelineRunID"` surface at config load instead of silently producing no attribute. The Pydantic model lives in `pipelex/temporal/config_temporal.py` so the validator can reference `BUILTIN_SEARCH_ATTRIBUTES` without pulling `temporalio` into the config-load path.diff --git a/CLAUDE.md b/CLAUDE.mdindex 646101c02..f574e13d4 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -93,30 +93,6 @@```Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.-### Temporal Integration Test Options-- The Temporal integration tests support different server modes via the `--temporal-server` pytest CLI option:-- - `--temporal-server`: Which Temporal server to use- - `none` (default): in-process test server — no external dependencies, used in CI- - `time-skipping`: in-process server with deterministic time control- - A profile name from `temporal_server_configs` in `pipelex.toml` (e.g. `local`, `testing`): connects to a real Temporal server using the profile's host, namespace, and API key settings-- ```bash- # CI default: in-process server- .venv/bin/pytest tests/integration/pipelex/temporal/-- # Dev with local Temporal server- .venv/bin/pytest tests/integration/pipelex/temporal/ \- --temporal-server local-- # Dev with cloud/testing server- .venv/bin/pytest tests/integration/pipelex/temporal/ \- --temporal-server testing- ```------### Prerequisites for running command lines: use virtual environment**CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST use the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.diff --git a/Makefile b/Makefileindex 8dd95d386..081b908c7 100644--- a/Makefile+++ b/Makefile@@ -148,8 +148,6 @@ make test-extract - Run unit tests only for extract (with prints)make te - Shorthand -> test-extractmake test-img-gen - Run unit tests only for img_gen (with prints)make test-g - Shorthand -> test-img-gen-make test-temporal - Run temporal tests (SRV=local|testing MODE=live REG=isolated)-make ttm - Shorthand -> test-temporalmake check-unused-imports - Check for unused imports without fixingmake fix-unused-imports - Fix unused imports with ruff@@ -172,17 +170,6 @@ make serve-graph - Start HTTP server to view ReactFlow graphs (PORTmake stop-graph-server - Stop the graph viewer HTTP servermake view-graph - Start server and open ReactFlow graph in browser-make temporal-server - Start a local Temporal dev server (requires 'temporal' CLI)-make ts - Shorthand -> temporal-server-make temporal-stop - Kill the local Temporal dev server (port 7233)-make tstop - Shorthand -> temporal-stop-make temporal-worker - Start a Temporal worker (separate process)-make tw - Shorthand -> temporal-worker-make temporal-run - Run a pipe through Temporal (real LLM calls)-make trun - Shorthand -> temporal-run-make temporal-run-dry - Run a pipe through Temporal (dry run, no LLM)-make trund - Shorthand -> temporal-run-dry-make check - Shorthand -> format lint mypymake c - Shorthand -> checkmake cc - Shorthand -> cleanderived check@@ -209,7 +196,7 @@ export HELPrules rules-claude-standalone up-kit-configs ukc check-config-sync ccs check-keyword-only cko fix-keyword-only fko check-rules check-urls cu insert-skeleton \cleanderived cleanenv cleanall \test test-xdist t test-quiet tq test-with-prints tp test-inference ti \- test-llm tl test-img-gen tg test-extract te test-temporal ttm codex-tests gha-tests \+ test-llm tl test-img-gen tg test-extract te codex-tests gha-tests \run-all-tests run-manual-trigger-gha-tests run-gha_disabled-tests \validate v check c cc agent-check agent-test agent-test-debug atd \test-durations td test-durations-serial tds test-time tt test-time-serial tts \@@ -221,8 +208,6 @@ export HELPupdate-gateway-models update-gateway-models-quiet ugm check-gateway-models cgm up \test-count check-test-badge \serve-graph serve-graph-bg stop-graph-server view-graph sg vg \- temporal-server ts temporal-stop tstop temporal-worker tw temporal-worker-router twr temporal-worker-runner twn \- temporal-run trun temporal-run-dry trund \docs-deploy-rootall help:@@ -691,42 +676,6 @@ test-img-gen: envtg: test-img-gen@echo "> done: tg = test-img-gen"-SRV ?=-MODE ?=-REG ?=-TEMPORAL_PYTEST_MARKERS := $(if $(filter live,$(MODE)),"temporal","temporal and (dry_runnable or not inference)")-TEMPORAL_TESTS_DIR := tests/integration/pipelex/temporal/--test-temporal: env- $(call PRINT_TITLE,"Unit testing Temporal")- @if [ -n "$(TEST)" ]; then \- if [ "$(TEST)" = "LF" ] || [ "$(TEST)" = "lf" ]; then \- $(VENV_PYTEST) --exitfirst -m $(TEMPORAL_PYTEST_MARKERS) -s --lf \- $(if $(SRV),--temporal-server $(SRV),) \- $(if $(REG),--class-registry $(REG),) \- $(if $(filter live,$(MODE)),--pipe-run-mode live,) \- $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))) \- $(TEMPORAL_TESTS_DIR); \- else \- $(VENV_PYTEST) --exitfirst -m $(TEMPORAL_PYTEST_MARKERS) -s -k "$(TEST)" \- $(if $(SRV),--temporal-server $(SRV),) \- $(if $(REG),--class-registry $(REG),) \- $(if $(filter live,$(MODE)),--pipe-run-mode live,) \- $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))) \- $(TEMPORAL_TESTS_DIR); \- fi; \- else \- $(VENV_PYTEST) --exitfirst -m $(TEMPORAL_PYTEST_MARKERS) -s \- $(if $(SRV),--temporal-server $(SRV),) \- $(if $(REG),--class-registry $(REG),) \- $(if $(filter live,$(MODE)),--pipe-run-mode live,) \- $(if $(filter 1,$(VERBOSE)),-v,$(if $(filter 2,$(VERBOSE)),-vv,$(if $(filter 3,$(VERBOSE)),-vvv,))) \- $(TEMPORAL_TESTS_DIR); \- fi--ttm: test-temporal- @echo "> done: ttm = test-temporal"-test-pipelex-api: env$(call PRINT_TITLE,"Unit testing")@if [ -n "$(TEST)" ]; then \@@ -1111,67 +1060,6 @@ sg: serve-graphvg: view-graph@echo "> done: vg = view-graph"-temporal-server:- $(call PRINT_TITLE,"Starting local Temporal dev server")- @if ! command -v temporal >/dev/null 2>&1; then \- echo "Error: 'temporal' CLI not found. Install it with: brew install temporal"; \- exit 1; \- fi- @echo "• Temporal Web UI will be available at http://localhost:8233"- @echo "• Temporal gRPC service at localhost:7233"- @echo "• Press Ctrl+C to stop"- temporal server start-dev--ts: temporal-server--temporal-stop:- $(call PRINT_TITLE,"Stopping local Temporal dev server")- @PID=$$(lsof -tiTCP:7233 -sTCP:LISTEN 2>/dev/null); \- if [ -z "$$PID" ]; then \- echo "• No process found on port 7233"; \- else \- kill $$PID && echo "• Killed Temporal server (PID $$PID)"; \- fi--tstop: temporal-stop--TEMPORAL_BUNDLE ?= tests/integration/pipelex/pipes/controller/pipe_sequence/pipe_sequence_1.mthds-TEMPORAL_PIPE ?= simple_text_sequence-TEMPORAL_LIB ?=--TEMPORAL_SCOPE ?=--temporal-worker: env- $(call PRINT_TITLE,"Starting Temporal worker$(if $(TEMPORAL_SCOPE), (scope: $(TEMPORAL_SCOPE)),)")- $(if $(TEMPORAL_LIB),PIPELEXPATH=$(TEMPORAL_LIB),) $(VENV_PYTHON) -m pipelex.temporal.worker_cli --is-not-sandboxed \- $(if $(TEMPORAL_SCOPE),--scope $(TEMPORAL_SCOPE),)--tw: temporal-worker--temporal-worker-router: env- $(MAKE) temporal-worker TEMPORAL_SCOPE=router--twr: temporal-worker-router--temporal-worker-runner: env- $(MAKE) temporal-worker TEMPORAL_SCOPE=runner--twn: temporal-worker-runner--temporal-run: env- $(call PRINT_TITLE,"Running pipe through Temporal")- $(VENV_PIPELEX) run bundle $(TEMPORAL_BUNDLE) --temporal --mock-inputs --no-logo --graph \- $(if $(TEMPORAL_PIPE),--pipe $(TEMPORAL_PIPE),)--trun: temporal-run--temporal-run-dry: env- $(call PRINT_TITLE,"Running pipe through Temporal - dry run")- $(VENV_PIPELEX) run bundle $(TEMPORAL_BUNDLE) --temporal --dry-run --mock-inputs --no-logo --graph \- $(if $(TEMPORAL_PIPE),--pipe $(TEMPORAL_PIPE),)--trund: temporal-run-dry-############################################################################################# GRAPH UI ASSET SYNC (from mthds-ui)##########################################################################################diff --git a/README.md b/README.mdindex 8e3075998..496d06190 100644--- a/README.md+++ b/README.md@@ -533,9 +533,7 @@ Install all extras:uv pip install "pipelex[anthropic,google,google-genai,mistralai,bedrock,fal,linkup,docling]"```-> **Mistral Workflows orchestration** *(coming soon)* is not a pipelex extra. It runs pipes through the runtime bridge inside Mistral Workflows activities and requires **Python 3.12+**, so it will ship as a separate, dedicated package that bundles pipelex. pipelex itself supports **Python 3.10+**; only this optional orchestration path needs 3.12.->-> This package is still in active development and not published yet — install and import instructions will land here once it ships.+> **Mistral Workflows orchestration** is not a pipelex extra. It runs pipes through the runtime bridge inside Mistral Workflows activities, and ships as a separate private package (`pipelex-mistralai-workflows`) that bundles pipelex — part of Pipelex's [workflow-orchestration offer](https://pipelex.com/products#mistral-workflows). It requires **Python 3.12+**; pipelex itself supports **Python 3.10+**, so only this optional orchestration path needs 3.12.---diff --git a/TODOS.md b/TODOS.mdindex 16e2d8466..4341002dd 100644--- a/TODOS.md+++ b/TODOS.md@@ -1,91 +1,339 @@-# Active plan — GraphSpec source paths + explicit validate graph target+# Orchestrator-agnostic runner & deployment flavors — implementation TODOS-> **Status: ready to start (2026-06-23).** This is a two-part Pipelex runtime graph/validation plan. Work from the Pipelex repo root (`/Users/lchoquel/repos/Pipelex/pipelex`). Start by reading the two design notes: `wip/graph/graphspec-source-map-enrichment.md` and `wip/graph/validate-explicit-graph-target.md`. Implement them in that order: source-map enrichment first, explicit graph target second.+**Effort:** make `pipelex-api` an orchestrator-agnostic base, then build N thin deployment flavors on top of it (Temporal today, Mistral next). The *consumer-side* sequel to the plugin-system externalization (that tracker is archived at [`wip/archive/plugin-system-implementation-tracker.md`](wip/archive/plugin-system-implementation-tracker.md)).-## Cold-Start Context+**Status:** **Phases A + B + C DONE (committed & pushed) + two post-C follow-on extensions DONE (pushed).** Phase A/B core seams (`pipelex`: `4b5449f88` Phase A, `75196e2e3` seam refinement, `7378afdd8` Phase B docs; `pipelex-temporal` `6d9adff`). **Phase C** = `pipelex-api` made orchestrator-agnostic, commit **`a39841e`** on `refactor/orchestrator-agnostic-base` (cut from `dev`): F1 execution via the orchestrator-registry hub seam (byte-equivalent — see the F1 deviation note in the as-built), F2 validate always DIRECT, F3 exception-handler mappers consumed at app construction, `[api]` config = Option C `api.toml` via `load_plugin_config`, `temporal` extra dropped. Clean-context `/code-review`s + all gates green (308 tests, pyright 0, mypy 0, pylint 10, openapi-check). **MAJOR GATE 2 / Checkpoint C (THE gate) cleared.** **Follow-on extension plans then landed on top of Phase C** (their own plan docs + `feature/*` branches — see "Follow-on extensions" below): orchestrator-dispatched `/validate` (reverses/extends F2), per-request `execution_mode` on `/execute` (extends F1, **PR #27**), and `/start` deriving the fire-and-forget variant of `execution_mode` (`f5e3ef7` + review fixes `7e4bc13`). **NEXT = Phase D** (`pipelex-api-hosted` Temporal flavor — MAJOR GATE 3), unchanged by the follow-ons. The in-`pipelex` plugin-system foundation (seams inverted, Temporal externalized, `plugins.boot_orchestrator` gate) was done earlier on `refactor/Plugins-3`.-Today `GraphSpec` is an execution trace for one root pipe. It has `nodes[]`, `edges[]`, `pipe_registry`, and `concept_registry`. It does not currently include the `.mthds` bundle path where each pipe or concept was declared. The path information exists elsewhere: `LibraryCrate.source_map` maps `domain.pipe_code` and `domain.ConceptCode` to source paths, `LibraryManager.get_pipe_source()` exposes pipe origins for diagnostics, and `PipeJob.library_crate` is already attached by `prepare_pipe_job()` and crosses the Temporal boundary.+**Current branches:** the follow-on work + this tracker live on **`feature/Orchestrator-dispatched-validate`** — `pipelex` (`_plugins`) and `pipelex-temporal` (tip `459e04d`). `pipelex-api` is a stack: `a39841e` (Phase C, `refactor/orchestrator-agnostic-base`) → `72c0efc` (validate, `feature/Orchestrator-dispatched-validate`) → `feature/Execute-per-request-mode` (execute **PR #27** `191c900`, then `/start` f&f derivation `f5e3ef7` + review fixes `7e4bc13`, **tip `7e4bc13`**). The PR #27 stack through `191c900` is pushed/CI-green; `f5e3ef7` + `7e4bc13` and this tracker update are **not pushed**. Phase D onward each lands in its own repo/branch.-Validation graph generation also has a separate root-selection constraint. The validation report graph arm currently uses `select_primary_blueprint(result.blueprints).main_pipe_ref`: first blueprint declaring `main_pipe`, else `None`. `best_effort_graph_spec(pipe_ref=None, ...)` returns `None`, so protocol/API validation succeeds without a graph when no `main_pipe` exists. CLI graph/view helpers are stricter: `dry_run_pipeline()` raises `PipelexInterpreterError("Bundle does not declare a main_pipe, cannot generate graph")` when graph output is explicitly requested and no `main_pipe` exists.+### Follow-on extensions (post-Phase-C — DONE; not part of the original A–F phases)-The product direction is: keep `GraphSpec` as an execution-trace artifact, but make it more useful. First, enrich existing registries with declaration source paths. Second, let callers request a graph for a specific pipe, so bundles without `main_pipe` can still validate and graph a selected pipe.+Extension plans written + executed **after** Phase C; each has its own plan doc and `feature/*` branch line. They do **not** change the D/E/F roadmap below.-## Work Order+- **Orchestrator-dispatched `/validate`** — plan [`wip/plugins/orchestrator-dispatched-validate.md`](wip/plugins/orchestrator-dispatched-validate.md). **Reverses/extends F2:** `/validate` is now `execution_mode`-aware via a NEW per-call `BundleValidatorRegistry` seam (mirrors `OrchestratorRegistry`) — `direct` validates in-process (F2 kept for the agnostic base), `temporal_*` dispatches the whole job to a worker. Verdict-as-value (`ApiRunner.validate` override removed → `validate_verdict`; route discriminates on `isinstance(verdict, PipelexValidationReport)`). Landed across all 3 repos on `feature/Orchestrator-dispatched-validate`: core PR **#998** (CI-green + Greptile-happy), `pipelex-api` tip **`72c0efc`**, `pipelex-temporal` tip **`459e04d`**. **DONE.**+- **Per-request `execution_mode` on `/execute`** — plan [`wip/plugins/execute-per-request-execution-mode.md`](wip/plugins/execute-per-request-execution-mode.md) (+ `execute-per-request-mode-deferred.md`). **Extends F1:** `/execute` now dispatches by resolved `execution_mode` through the `OrchestratorRegistry` (symmetric with `/start`), rejects fire-and-forget with a 400, and rehydrates the orchestrator's JSON-safe output back into the full `PipeOutput`. **`pipelex-api`-only** (reuses the existing seam — no core/temporal change). **PR #27** (`feature/Execute-per-request-mode` → `feature/Orchestrator-dispatched-validate`, tip **`191c900`**): CI green, Greptile 5/5, gstack `/review` merge-ready; **not merged (user's call).** Deferred items in `wip/plugins/execute-per-request-mode-deferred.md` (DIRECT round-trip cost; pre-existing `/start` resource leak; `/start` schema gap + dangling-`#/$defs/` OpenAPI ref pattern; failing-pipe error double-wrap via the shared seam).+ - **CI gotcha (any `pipelex-api` PR on these branches):** the editable `pipelex = ../_plugins` pin breaks CI (no sibling worktree on the runner). Fix = pin core to a **git+https SHA** in `pyproject.toml` `[tool.uv.sources]` (core repo is PUBLIC → no creds) + relock `uv.lock`; flip back to the PyPI `==` pin at release. Done on PR #27 as `c60be1f` (pinned to core `51ff09417`).+- **`/start` derives the fire-and-forget variant of `execution_mode`** — commit **`f5e3ef7`** (+ review fixes **`7e4bc13`**) on `feature/Execute-per-request-mode`, stacked on PR #27. **Endpoint, not deployment:** fire-and-forget vs blocking is a property of the *endpoint*. A deployment configures ONE *synchronous* `execution_mode`; `/execute` + `/validate` dispatch it as-is, while `/start` derives its f&f sibling (`temporal_blocking → temporal_fire_and_forget`; `direct`/`mistral_native` have none → dispatched unchanged, blocking). Kills the `/execute` 400 / forced-direct footgun — one coherent mode across all three endpoints. Route-local `_async_start_mode` (kept off the core enum per smallest-surface). **`pipelex-api`-only** (no core/temporal change). **xhigh-workflow `/code-review` + fixes (`7e4bc13`):** (1) `ApiConfig` now rejects a *configured* f&f `execution_mode` (field validator on `is_fire_and_forget`) — the docs declared it unconfigurable but nothing enforced it (a baked f&f would 400 every `/execute` + misroute `/validate`); fail-fast at load, per-request f&f override on `/start` still allowed (policy-gated). (2) Corrected the `mistral_native` `workflow_id` claim in the `_async_start_mode` docstring + call-site comment + `configuration.md` — `mistral_native` runs per-call and answers with a *non-null* `workflow_id` (the run id); only `direct` returns `None`. (3) Added a pure-mapping test of `_async_start_mode` over every mode + a `mistral_native` end-to-end `/start` case (both previously unverified despite the test docstring claiming the coverage); `test_api_config`'s locked-Temporal helper switched to the synchronous `temporal_blocking` + a configured-f&f-rejection test added. Gates green (ruff/pyright 0/mypy 0/pylint 10, full unit suite pass, OpenAPI in sync). Review verdict: the F1-gate ordering (403 runs *before* the f&f derivation) and single-caller transform are correct; deferred low-priority items (route-local-vs-enum altitude; a custom blocking-only orchestrator plugin now needs a f&f arm for `/start`) noted as by-design. **DONE; not pushed.**+- **DONE (Phases 1–4) — re-cut `execution_mode` into `orchestration_mode` (open string token) + `DeliveryMode` (closed enum)** — plan [`wip/plugins/orchestration-mode-and-delivery-split.md`](wip/plugins/orchestration-mode-and-delivery-split.md). **Phase 1 (core, `_plugins` `bd4e52827`) + Phase 2 (`pipelex-temporal` `01d7dc3`) + Phase 3 (`pipelex-api` `dd84a9a`, THE gate) landed on `feature/orchestration-mode-delivery-split` in each repo — gates green + clean-context `/code-review` each (Phase 3: no must-fix). Phase 4 = the external `pipelex-mistralai-workflows` plugin migrated to the new core surface (deleted-enum import + registration under the deleted member + own `execution_mode` wire field + old `run` signature → `orchestration_mode`/`delivery`/new protocol), commit `c2013fb` on `feature/Mistral-native`, green against the merged core (agent-check + full integration suite). §8 `/start`-honesty risk resolved (honest 400 safe). DEVIATION (Phase 4): the Mistral plugin pins core editable at `../_workflows` (branch `feature/mistralai-2x-bump`), NOT the orchestration-split worktree — so Phase 4 first required merging `feature/orchestration-mode-delivery-split` into `feature/mistralai-2x-bump` (`_workflows` merge commit `a1448ad8f`, conflict-free; core gates green) to give the plugin a core carrying BOTH the split and the mistralai-2x bump. Spec/conformance untouched (no `docs/specs/` heading or `conformance/` test asserts mode values → `make check-spec-links` not triggered). See the plan's §10 Phase 4 as-built.** **Supersedes the flat-`PipelexExecutionMode` framing of the two bullets above:** it removes the conflation those plans worked around — `_async_start_mode`, the `/execute` fire-and-forget 400, the `ApiConfig` f&f field-validator, and the `/validate` double-registration. Two orthogonal axes: an **open** orchestration token (core owns only `"direct"`; plugins own `"temporal"` / `"mistralai-workflows"` — realigns with "the base names no orchestrator") + a **closed** `DeliveryMode {BLOCKING, FIRE_AND_FORGET}` that endpoints set, never receive. `/start` becomes **honest** (4xx when the resolved mode can't do async) instead of degrading. Design **locked 2026-06-22**; Direction A (core → `pipelex-temporal` → `pipelex-api`, one repo/branch/commit per phase). **Top risk to settle before the API phase:** does the MTHDS-Protocol/conformance suite require `/start` to return `202` on a non-async (`direct`) backend? See the plan's §8.-1. **Source-map enrichment first.** This is additive and low-risk. It uses existing `LibraryCrate.source_map` and existing free-form registry payloads. It should not change root selection or validation behavior.-2. **Explicit graph target second.** This changes validation/graph routing across direct, Temporal/API, and CLI surfaces. It benefits from the source-enriched registries once graphs can target non-`main_pipe` pipes.+----## Phase 1 — Enrich GraphSpec Registries With Source Paths+This is the **execution tracker**. The *why* and *how* live in [`wip/plugins/orchestrator-agnostic-runner-and-flavors.md`](wip/plugins/orchestrator-agnostic-runner-and-flavors.md) (the reviewed, decision-locked plan). **Read that doc before starting any phase** — this file tracks progress against it; it does not replace it. **Resuming in a fresh session?** Start with [`wip/plugins/resume-and-verify-state.md`](wip/plugins/resume-and-verify-state.md) — paste-ready orientation + commands to verify the state against the repos before doing anything. Adjacent background: [`wip/plugins/temporal-config-out-of-core.md`](wip/plugins/temporal-config-out-of-core.md) (the boot-gate refactor that immediately precedes this work) and the plugin-system design/SPI docs in [`wip/plugins/`](wip/plugins/).-Goal: when `pipe_and_concept_registry` is enabled and a `LibraryCrate.source_map` entry exists, `graph_spec.pipe_registry[pipe_ref]["source"]` and `graph_spec.concept_registry[concept_ref]["source"]` carry the source bundle path. If source is unavailable, omit the field rather than emitting `source: null`.+> **How to use this file.** Tick boxes as you land work. **Do not skip a `🛑 CHECKPOINT`** — each is a hard stop with three mandatory actions (verify · capture cold-start context · fan-out `/code-review`). Three checkpoints are flagged **MAJOR GATE** — they are the points the source plan calls out as the ones that must hold before the effort can continue.+>+> **Fan-out convention for `/code-review`.** Spawn each review sub-agent with **no inherited context** — hand it only a pointer to the changes under review (the phase's commit SHA, `git diff <base>..HEAD`, or the unstaged/working-tree files), never the plan, the rationale, or your own conclusions. A clean-context reviewer reviews the code as written, not your story about it — that independence is the point of fanning it out.-Implementation shape:+----- In `pipelex/core/pipes/pipe_abstract.py`, add a source-aware pipe registry helper. It should start from `self.model_dump(mode="json")`, look up `library_crate.source_map.get(self.pipe_ref)`, and add `source` only when present.-- Update `_make_single_concept_data_for_registry()` and `_make_concept_data_for_registry()` to accept `library_crate: LibraryCrate | None`, look up `concept.concept_ref`, and add `source` only when present. Keep the existing `json_schema` behavior.-- In `_run_pipe_traced()`, replace `pipe_data = self.model_dump(mode="json")` and `concept_data = self._make_concept_data_for_registry()` with the source-aware helpers, passing the existing `library_crate` parameter.-- Do not add `source` to `NodeSpec`. A node is an invocation; the source path is declaration metadata, so it belongs in `pipe_registry` / `concept_registry`.-- Do not change `GraphSpec` model fields. The registries are already `dict[str, dict[str, Any]]`, and `PipeStartEvent.pipe_data` / `concept_data` already carry generic dictionaries.+## Cold-start primer (read this first if you're new to the session)-Tests:+**Goal of the whole effort:** today `pipelex-api` is wired directly to Temporal in production code — it is *not* an orchestrator-agnostic runner. Make the public `pipelex/pipelex-api` base name **no** orchestrator (ship only the in-process/direct path), then express each deployment as **base image + exactly one orchestrator plugin + that plugin's activation**. That is what lets a Temporal flavor (`pipelex-api-hosted`) and a future Mistral flavor (`pipelex-api-mistral`) both exist on one base.-- Add focused coverage proving registry entries include `source` when a fake or real `LibraryCrate.source_map` provides entries.-- Add a direct/in-process graph test using a real `.mthds` file loaded from disk: assert the pipe registry source is the bundle path and declared concept registry source is present.-- Add or extend Temporal/event-assembled coverage so `PipeStartEvent.pipe_data` / `concept_data` with `source` survive through `GraphSpecAssembler`.-- Add a sourceless path check: raw in-memory `mthds_contents` without sources still produces a graph and omits `source`.+**The model shift — internalize before touching any repo.** The plugin system split a previously-monolithic idea ("Temporal is on") into orthogonal concepts:-> **Checkpoint A — source paths in registries. DONE (2026-06-23).** Direct graph registry payloads now enrich pipes and concepts from `LibraryCrate.source_map`; event-assembled graph paths preserve `source` because `PipeStartEvent.pipe_data` / `concept_data` and `PipeEndSuccessEvent.output_concept_data` are copied unchanged by `GraphSpecAssembler`; sourceless in-memory bundle graphs still omit `source`. Edge-case decision: source enrichment uses only the executing job's active `LibraryCrate.source_map`; dependency or native entries with no source-map entry keep the previous behavior and omit `source` rather than emitting `source = null`. Verification: `CI=true .venv/bin/pytest tests/unit/pipelex/tracing/test_graphspec_assembler.py::TestGraphSpecAssembler::test_registry_source_payloads_survive_assembly tests/integration/pipelex/pipeline/test_dry_run_pipeline_graph.py::TestDryRunPipelineGraphTransport::test_graph_produced_with_tracing_disabled tests/integration/pipelex/pipeline/test_validate_graph_registry_sources.py -q` passed; `CI=true make agent-check` passed. `CI=true make agent-test` was attempted in the sandbox and reached unrelated environment failures (Temporal dev server `Operation not permitted`, restricted DNS/network HTTP tests, local HTTP server permission errors); unsandboxed rerun was rejected by approval policy because the broad suite contacts external services and starts local/networked servers.+- **Presence** = the plugin distribution is installed, so its `pipelex.plugins` entry point is discovered at boot and `register()` runs, contributing its `execution_mode` arms to the orchestrator registry. Presence makes a mode *dispatchable*; it does **not** make it *used*.+- **Activation** = the runtime is told to use it. It decomposes into **two independent knobs that are not derivable from each other:**-## Phase 2 — Add Explicit Graph Target Support+| Knob | Owns | Set by | Temporal | Mistral | Direct |+|---|---|---|---|---|---|+| `execution_mode` | which mode a *top-level* run dispatches as, through the bridge | runner config default + optional per-request override | `temporal_blocking` / `temporal_fire_and_forget` | `mistral_native` | `direct` |+| `plugins.boot_orchestrator` | whether to claim the process-global hub slots (content generator / pipe router / pipe run / task manager) at boot | deployment config TOML | `"temporal"` | *unset* | *unset* |-Goal: callers can request graph generation for a specific pipe. The default remains unchanged: use the selected `main_pipe` when no explicit target is provided. If neither explicit target nor `main_pipe` exists, protocol/API validation should keep returning `graph_spec=None`, while CLI `--graph` / `--view` should keep failing clearly because the user explicitly requested a graph.+**Mistral is the proof the two knobs are independent.** A Temporal deployment needs *both* (the boot gate stands up the Temporal client/task-manager the orchestrator enqueues against; `execution_mode` tells the bridge to enqueue). A Mistral deployment needs **only** `execution_mode = mistral_native` — its plugin claims **no** hub slots (self-contained per-call orchestrator), so there is no boot gate to set. A plain in-process deployment sets neither.-Implementation shape:+**Boot-global vs per-call orchestrators** — *why* the two knobs exist. A **boot-global** orchestrator (Temporal) replaces process-wide execution machinery → must claim hub slots at boot, gated by `boot_orchestrator`. A **per-call** orchestrator (Mistral) only adds a dispatch arm → presence is enough; selection is per-run via `execution_mode`. Classify any new orchestrator first; the classification decides whether `boot_orchestrator` is even meaningful for it.-- Add `graph_pipe_code: str | None = None` to `pipelex/pipeline/validate_in_process.py::validate_bundles_in_process()`. Compute `graph_target_ref = graph_pipe_code or select_primary_blueprint(result.blueprints).main_pipe_ref`, then pass it to `best_effort_graph_spec()`.-- Decide how `PipelexMTHDSProtocol.validate()` exposes this. If using protocol `extra`, validate and accept only a known key such as `graph_pipe_code`; otherwise keep protocol unchanged and expose the option only on concrete API/CLI surfaces.-- Temporal path already has `DryValidateArg.pipe_code`; ensure the hosted/API dispatch layer maps the explicit graph target into it. Rename only if the surrounding API request field also uses `graph_pipe_code`; avoid churn otherwise.-- Update `pipelex/pipeline/dry_run_pipeline.py` to accept `pipe_code: str | None = None`. If provided, execute that pipe; otherwise preserve the current `main_pipe` selection and missing-main error.-- Update `pipelex/graph/graph_rendering.py` helpers (`_dry_run_bundle()`, `generate_graph_for_bundle()`, `generate_view_for_bundle()`) to accept/pass the pipe override.-- Update `pipelex/cli/agent_cli/commands/validate/bundle_cmd.py` so `validate bundle --pipe X --graph` and `--view` graph `X` instead of ignoring `--pipe` and falling back to `main_pipe`.-- Decide and document bare vs qualified target behavior. Recommended: accept both; bare targets must be unique in the loaded library; existing ambiguity errors can surface on explicit CLI graph requests and degrade on best-effort report paths.+**Install channel follows license, not convenience.**-Tests:+- `pipelex-temporal` is **private** → pinned `git+ssh` ref (precedent: `pipelex-shared @ git+ssh://…/infra-python-tools.git@<ref>`). Only **private** images may pin it.+- `pipelex-mistralai-workflows` is **public** → plain PyPI (`pipelex-mistralai-workflows==X`). Currently pre-publish; gated on its own `mistralai-2.x` work — treat as PyPI once published.-- Protocol/direct validation: no `main_pipe` plus explicit graph target produces non-null `graph_spec`; explicit graph target overrides declared `main_pipe`; unknown explicit target degrades to `graph_spec=None` on best-effort validation reports.-- Agent CLI: `validate bundle no_main.mthds --pipe some_pipe --view` succeeds and includes `graphspec`; `validate bundle no_main.mthds --view` still errors; `validate bundle bundle_with_main.mthds --pipe other_pipe --view` graphs `other_pipe`.-- Temporal/API path: explicit graph target maps to `DryValidateArg.pipe_code`; direct and Temporal modes agree on the target behavior.+**Verified anchors (2026-06-21, this worktree)** — the plan's foundations are present and current:-> **Checkpoint B — explicit graph target parity. DONE (2026-06-23).** Direct/protocol validation now accepts `extra = {"graph_pipe_code": "..."}` as the only local runtime validate extension and passes it to `validate_bundles_in_process(graph_pipe_code=...)`; unknown, non-string, and blank validate extension args are rejected. The shared in-process graph arm targets `graph_pipe_code` first, else the selected `main_pipe`, else keeps `graph_spec = None`. CLI graph/view generation now passes `validate bundle --pipe X` into `generate_graph_for_bundle(..., pipe_code=X)` / `generate_view_for_bundle(..., pipe_code=X)`, and `dry_run_pipeline(..., pipe_code=X)` can graph bundles without `main_pipe`. Bare explicit dry-run targets now return the resolved domain-qualified graph target, so callers do not report the original ambiguous ref. Temporal keeps the existing wire field `DryValidateArg.pipe_code` as the explicit graph target to avoid churn; no field rename was made. Bare and qualified targets are both accepted via the existing pipe-library resolver; ambiguous or missing explicit targets degrade on best-effort validation report paths and surface as graph/view errors on explicit CLI graph requests. Tests were added for direct/protocol no-main, override, blank, and missing-target behavior; dry-run no-main explicit target; CLI pipe override passthrough; graph-rendering passthrough; and Temporal no-main/override behavior. Per user instruction, pytest was not run for this phase. Verification run: `CI=true make agent-check` passed.+- `plugins.boot_orchestrator` gate exists: `configs.py:258`, applied in `pipelex.py:206-207` (only overridden when the param is non-`None`, so a TOML value is **preserved** — D1 safe).+- Plugin contract: `PLUGIN_API_VERSION = 1` (`pipelex/plugins/contract.py:10`); `PluginRegistrar` already hosts `add_inference_backend` / `add_model_lister` / `add_orchestrator` / `claim_*` / `add_teardown` (no `add_cli_command` — removed by Option A). F3 adds one method here.+- `ErrorReport` is a core type (`pipelex/base_exceptions.py:309`) → F3's mapper signature uses it (no FastAPI in core).+- Config loader env-layering pattern to mirror for D2: `ConfigManager._override_files_for_dir` / `load_config` (`pipelex/system/configuration/config_loader.py`), keyed on `runtime_manager.environment`.+- The §2 gap is real and current: `pipelex-api/api/` still names temporal in `exception_handlers.py`, `routes/pipelex/validate.py`, `routes/pipelex/pipeline.py`. All sibling repos exist (`pipelex-temporal`, `pipelex-api`, `pipelex-api-hosted`, `pipelex-worker`, `pipelex-mistralai-workflows`).-## Non-Goals+**Locked decisions (D1, D2, F1, F2, F3)** — do not re-litigate; if you believe one is wrong, raise it explicitly before deviating. Full text in §3 of the source doc.-- Do not infer a root from "the only pipe in the bundle" unless a product decision explicitly asks for that. It creates a hidden second default alongside `main_pipe`.-- Do not create a static all-pipes `GraphSpec` under the same field. That would overload an execution-trace artifact with non-execution semantics.-- Do not require `main_pipe` to validate a bundle. Current validation correctly supports concept-only files, membership-only files, and top-down partial bundles.-- Do not move declaration source onto runtime `PipeAbstract` / `Concept` models unless registry enrichment proves insufficient. The source-map already exists on `LibraryCrate`.+- **D1** — `boot_orchestrator` is set from the flavor's deployment TOML (`[plugins] boot_orchestrator = "temporal"`). No `pipelex-api` change for the gate; verified preserved by `setup()`.+- **D2** — plugin config files become env-aware, mirroring the main config. Add a **reusable core helper** so every plugin self-loads `name.toml` → `name_{env}.toml` → `name_override.toml` with deep-merge. `pipelex-temporal` adopts it. One image bakes all envs; `PIPELEX_ENV` selects at runtime.+- **F1** — `execution_mode` = runner config default + optional per-request override, gated by a per-deployment policy (a locked-down Temporal runner can refuse a caller forcing `direct`). Runner builds `PipelexPipeRunInput(execution_mode=…)` and dispatches via `run_pipe_via_bridge`; the bridge output already carries `workflow_id`, so async-start survives.+- **F2** — validate is **always DIRECT in-process** (`validate_bundles_in_process`). Drops the Temporal validate branch. **Behavior change:** the API runner now loads the method library to validate (prior design pushed that onto the worker) — document the resource/isolation implication on the hosted runner. **→ REVERSED / EXTENDED** by orchestrator-dispatched `/validate` (new per-call `BundleValidatorRegistry` seam): `/validate` is now `execution_mode`-aware — `direct` in-process (F2 kept for the agnostic base) | `temporal_*` dispatched to a worker (restored). Same canonical verdict either way. See [`wip/plugins/orchestrator-dispatched-validate.md`](wip/plugins/orchestrator-dispatched-validate.md).+- **F3** — orchestrator-specific HTTP error handling via a **new framework-agnostic plugin seam**: `PluginRegistrar.add_http_error_mapper(*, exc_type, to_error_report: Callable[[Exception], ErrorReport])`. The orchestrator plugin registers its transport-fault mapping; `pipelex-api` iterates the mappers at app construction and wraps each into a FastAPI handler using its existing RFC 7807 + `DisclosureMode` rendering. **FastAPI/Starlette stays only in `pipelex-api`; core and the plugin import neither.** Grows the contract by one optional capability → bump `PLUGIN_API_VERSION`.-## Verification Guidance+**Delivery model (cross-repo).** Unlike the archived plugin-system tracker (one repo, one-commit-per-checkpoint, single PR), this effort spans repos. **Each phase lands in its own repo on its own branch as its own commit(s) + PR — there is no single cross-repo PR.** Ordering is load-bearing: the `pipelex` (A) and `pipelex-temporal` (B) enabling changes must land before `pipelex-api` (C) can consume the F3 seam.-Follow Pipelex repo standards: use `make agent-test`, not `make test`. For narrow iterations, run the smallest relevant pytest path through the repo's supported agent-test entrypoint if available, then run broader graph/validation coverage before handing off. Likely target areas: `tests/unit/pipelex/graph/`, `tests/integration/pipelex/pipeline/`, `tests/integration/pipelex/temporal/`, and agent CLI validate tests. If implementation touches hosted API request mapping, also update and verify the matching `../pipelex-api` surface in a separate repo change.+**Local cross-pin strategy (for dev before anything is published/released):** pin editable across the chain so the seam can be exercised end-to-end locally — `pipelex-temporal` → editable `pipelex` (`[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }`), and `pipelex-api` → editable `pipelex` (+ a synthetic in-test plugin to prove the F3 seam without importing `temporalio`). The published `==` / git+ssh pins are the release-time flip (Phases D–F).-## Files Likely To Touch+**Commands** (from `_plugins/CLAUDE.md`, for the `pipelex` Phase-A work): `make agent-check` (lint/types — always after changes) · `make agent-test` (full suite, silent on success — at end of phase) · `make tb` (boot/config sanity — fast, after config or seam edits). Always use the venv: `.venv/bin/...`. Consumer repos use their own `make agent-check` / `make agent-test` (same family conventions).-- `pipelex/core/pipes/pipe_abstract.py`-- `pipelex/pipeline/validate_in_process.py`-- `pipelex/pipeline/runner.py`-- `pipelex/pipeline/dry_run_pipeline.py`-- `pipelex/graph/graph_rendering.py`-- `pipelex/cli/agent_cli/commands/validate/bundle_cmd.py`-- `pipelex/temporal/tprl_pipe/act_dry_validate.py` only if field naming changes-- tests under `tests/unit/pipelex/graph/`, `tests/integration/pipelex/pipeline/`, `tests/integration/pipelex/temporal/`, and agent CLI validate coverage-- hosted API validate request/dispatch mapping in `../pipelex-api` if this is exposed over the public API+----## Definition Of Done+## Image architecture (target — handles public + private plugins uniformly)-- `GraphSpec.pipe_registry` and `GraphSpec.concept_registry` include `source` when a source-map entry exists and omit it when not available.-- Source enrichment works through direct in-memory graphs and event-assembled Temporal graphs.-- Validation can graph an explicit pipe target without `main_pipe`.-- Default validation behavior stays unchanged when no explicit target is provided.-- CLI `validate bundle --pipe X --graph/--view` graphs `X`.-- Tests pin source enrichment, no-main explicit target, target-overrides-main, missing target degradation/error behavior, and direct/Temporal parity.+A **two-layer** build per flavor isolates plugin-install complexity (private SSH, version-locking) into one place and keeps the env-config layer trivial:++```+pipelex/pipelex-api:<ver> (public base, orchestrator-agnostic, Docker Hub)+ ├─ flavored base = base + ONE plugin (built once per flavor → private ECR)+ │ pipelex-api-temporal:<ver> = base + pipelex-temporal (git+ssh)+ │ pipelex-api-mistral:<ver> = base + pipelex-mistralai-workflows (PyPI)+ └─ env-config child = flavored base + COPY env TOMLs (per env; config-only)+ pipelex-api-{dev,staging,prod} (Temporal)+ pipelex-api-mistral-{dev,staging,prod}+```++**One parameterized install recipe.** Flavored-base Dockerfile takes `PLUGIN_SPEC` and always exposes a BuildKit SSH mount. PyPI plugin → plain install (SSH mount unused). Private plugin → `--mount=type=ssh` with the deploy key in the CI builder (`docker build --ssh default`); never bake a key into a layer. Install mechanics that bite (encode in the Dockerfile): base Python is a uv project venv at `/app/.venv` (install into it; don't let the resolver swap the pinned `pipelex`); the base purges `git`/`build-essential` in a late layer (re-add `git` + `openssh-client` before installing the private plugin); the plugin ref and base `pipelex` version are a **coupled pair that bump together** — track it in the flavor's version file.++---++## Phase A — Core enabling changes (`pipelex`, the `_plugins` worktree)++**Goal:** add the two seams `pipelex-api` will consume, in the public core, with no orchestrator named and no FastAPI imported. Lands on `refactor/Plugins-3`.++- [x] **D2 — env-aware plugin config loader.** Add a reusable helper on `ConfigManager` (e.g. `load_plugin_config(*, name, schema)`) resolving `name.toml` (the calling plugin's packaged default) → `name_{environment}.toml` → `name_override.toml` with deep-merge, mirroring `load_config`'s env layering. Anchor: `pipelex/system/configuration/config_loader.py` (`_override_files_for_dir` / `load_config`, keyed on `runtime_manager.environment`).+- [x] **F3 — HTTP-error-mapper seam.** Add `add_http_error_mapper(*, exc_type: type[Exception], to_error_report: Callable[[Exception], ErrorReport])` to `PluginRegistrar` (`pipelex/plugins/registrar.py`, alongside the existing menu methods) plus a read accessor for the collected mappers. Signature uses `ErrorReport` (`pipelex/base_exceptions.py`) — **no FastAPI import.**+- [x] **Bump `PLUGIN_API_VERSION`** (`pipelex/plugins/contract.py:10`, `1 → 2`) and document the new optional capability in the orchestrator SPI doc (`docs/under-the-hood/orchestrator-plugins.md`). Note the contract bump means every plugin's `targets_api` must be re-confirmed (built-ins + `pipelex-temporal` in Phase B).+- [x] **Tests:** env-layered plugin-config round-trip (packaged default → `_{env}` → `_override` deep-merge); registrar collects/returns mappers; import-light boot still green (the seam adds no SDK/HTTP import — extend the existing import-light subprocess guard if needed).++### 🛑 CHECKPOINT A — hard stop (core seams exist; consumed at B/C)++- [x] **Verify:** `make agent-check` clean · `make tb` green · `make agent-test` green. Core still names **no** orchestrator and imports **no** FastAPI (`grep -rn "fastapi\|starlette" pipelex/` → only telemetry labels + `WorkerMode.FASTAPI` enum, no imports; temporal refs in core unchanged). The seam's *end-to-end* exercise lands at Checkpoint B (when the plugin contributes through it) — don't expect integration proof here.+- [x] **Capture cold-start context:** append `### Phase A — as-built` below (final helper name/signature, the registrar method + accessor signature, the `PLUGIN_API_VERSION` value, which plugins' `targets_api` still need re-confirming, test locations).+- [x] **Fan-out `/code-review`:** sub-agent runs `/code-review` on the Phase A `pipelex` diff. Triage findings into the as-built (apply cheap ones; defer design-tradeoffs to a `wip/plugins/` follow-up per the deferral convention). _(done — verdict clean; nit #2 applied; tradeoff #1 deferred to `wip/plugins/phase-c-http-error-mapper-consumer-path.md`)_+- [x] **Commit** on `refactor/Plugins-4`. This is a natural session boundary before switching to the `pipelex-temporal` repo.++---++## Phase B — `pipelex-temporal` adopts the new seams (additive, private repo)++**Goal:** the private Temporal plugin uses the D2 helper and contributes through the F3 seam. Additive — no behavior change to its existing orchestrator/slot-claim contributions.++- [x] **D2 adoption.** Switch `load_temporal_config()` to the new helper; ship `temporal_{env}.toml` resolution (the packaged default `temporal.toml` stays). One image bakes all envs; `PIPELEX_ENV` selects.+- [x] **F3 mapper.** In `TemporalPlugin.register()`, register `temporalio.TemporalError → ErrorReport` classified transient / `RUNTIME` (port the classification currently in `pipelex-api`'s `handle_temporal_error`). Keep `register` import-light — via a lazy `exc_type_provider` thunk + a lazy SDK import, `register` imports no `temporalio` (the Phase A seam was refined to take the provider; see Phase A as-built "Why lazy").+- [x] **Re-confirm `targets_api`** matches the new `PLUGIN_API_VERSION` from Phase A. (`targets_api = PLUGIN_API_VERSION`, auto-tracks to 2 via the editable pin — verified by the suite running against the Phase A core.)+- [x] Pin editable `pipelex` (`../_plugins`) for local testing against the Phase A core. (Already present in `pyproject.toml` `[tool.uv.sources]`.)++### 🛑 CHECKPOINT B — hard stop · **MAJOR GATE 1** (source-doc Checkpoint 1)++- [x] **Verify:** `pipelex-temporal` `make agent-check` + `make agent-test` green against the Phase A `pipelex`. The seam now has a real consumer: a `pipelex-temporal` install contributes its mapper through `add_http_error_mapper` and self-loads `temporal_{env}.toml`. Import-light `register` still proven (no `temporalio` at registration) — subprocess blocks `temporalio` and runs `register` to completion.+- [x] **Capture cold-start context:** `### Phase B — as-built` appended below.+- [x] **Fan-out `/code-review`:** two forked `/code-review`s (core seam refinement + pipelex-temporal diff) — both **clean, no bugs**. Triage recorded in the Phase B as-built (one NIT applied: slot-claim count in the import-light test; rest deferred).+- [x] **Commit** in `pipelex-temporal` (`refactor/own-temporal-config`, `6d9adff`). Core seam refinement + Phase B docs on `refactor/Plugins-4` (`75196e2e3`, `7378afdd8`).++---++## Phase C — `pipelex-api` decoupling (the big one, public Docker Hub)++**Goal:** make the base orchestrator-agnostic. **No `pipelex.temporal.*`, no `temporalio`, anywhere in `api/`.** Each of F1/F2/F3 removes one of the §2 coupling sites.++- [x] **Execution (F1).** Rewrite `ApiRunner.start()` (`api/routes/pipelex/pipeline.py`). **DEVIATION (flagged in as-built):** dispatch via `get_orchestrator_registry().get_optional(mode=execution_mode).run(pipe_job, delivery_assignment)` (the identical final dispatch `run_pipe_via_bridge` performs) fed the **rich** `pipeline_run_setup` job — NOT `PipelexPipeRunInput`+`run_pipe_via_bridge`, which is lossy (drops `request_id`/`output_multiplicity`/`dynamic_output_concept_ref`/registration/telemetry) and would need a forbidden 2nd core change to fix. `workflow_id` off `run_output`. `execution_mode` from `api.toml` (Option C) + per-request override gated by `allow_request_execution_mode_override`. `make_temporal_pipe_run` import/usage gone. (`execute` was already agnostic via the `get_pipe_run()` hub slot — untouched.)+- [x] **Validate (F2).** Collapsed `ApiRunner.validate()` to the single `validate_bundles_in_process` path. Deleted the `temporal.is_enabled` branch + `dispatch_dry_validate`/`DryValidateArg`/`PipelexInterpreter`/`build_validation_report` imports + the route's `WorkflowExecutionError` catch. Behavior change documented in `pipelex-api/docs/pipe-validate.md`.+- [x] **Exception handlers (F3).** Removed the `temporalio` import + hardcoded `handle_temporal_error`. `register_exception_handlers(…, http_error_mappers=…)` registers one handler per `exc_type` via `_make_orchestrator_error_handler` → existing `_problem_response`. `api/main.py` resolves the mappers at import via `build_registrar(...).get_http_error_mappers()` (deferred-doc option 1 — no 2nd core change). `handle_pipelex_error` kept.+- [x] **Deps.** Dropped the `temporal` extra from `pyproject.toml`; removed the `[temporal]` block from the base `.pipelex/pipelex.toml`. Base depends on **no** orchestrator plugin.+- [x] **Tests.** `test_exception_handlers.py` rewritten to the synthetic-plugin shape (real `PluginRegistrar` + `add_http_error_mapper` + `get_http_error_mappers`, no `temporalio`); `test_validate_*` temporal arms removed; conformance webhook test mocks `get_orchestrator_registry`. New `test_api_config.py` covers the default + override policy (forbidden → 403, allowed honored, same-as-default ok) at the resolver AND end-to-end on `POST /start`.++### 🛑 CHECKPOINT C — hard stop · **MAJOR GATE 2 — THE gate** (source-doc Checkpoint 2)++Flavors are meaningless until the base is clean. This is the decisive checkpoint.++- [x] **Verify:** `pipelex-api` suite green (307 pass) with **no** `pipelex.temporal`/`temporalio` import in `api/` (`grep` → only incidental prose; `import api.main` → `temporalio` not in `sys.modules`). Direct-mode validate + execute work with no orchestrator plugin installed. Override policy refuses a forbidden per-request mode (403). All gates green (lint/pyright 0/mypy 0/pylint 10/openapi-check).+- [x] **Capture cold-start context:** `### Phase C — as-built` appended below (config Option C + root-key gotcha, the F1 dispatch deviation + why, override policy, synthetic-plugin test pattern, validate doc location).+- [x] **Fan-out `/code-review`:** two clean-context reviews on the working-tree diff (correctness/byte-equivalence + tests/config) — both **correct, no silent bugs, byte-equivalence verified**. Triage recorded in the Phase C as-built (5 applied incl. a re-added coverage test; 2 pre-existing deferred).+- [x] **Commit:** `pipelex-api` `a39841e` (`refactor/orchestrator-agnostic-base`) + this tracker update on `refactor/Plugins-4`. Nothing pushed. Strong session boundary before building flavors.++---++## Phase D — `pipelex-api-hosted` (Temporal flavor, private ECR)++**Goal:** the first real flavor = agnostic base + `pipelex-temporal` + Temporal activation.++- [ ] **Flavored base** (`pipelex-api-temporal`): `FROM pipelex/pipelex-api:<ver>` + install `pipelex-temporal @ git+ssh://…@<ref>` per the image architecture (re-add `git` + `openssh-client`, SSH mount, version-locked to `<ver>`). Wire the deploy key into CI (`--ssh default`).+- [ ] **Config migration** in the baked `.pipelex/pipelex_{env}.toml`:+ - [ ] Replace `[temporal] is_enabled = true` → `[plugins] boot_orchestrator = "temporal"`.+ - [ ] Add `[api] execution_mode = "temporal_fire_and_forget"` (or `temporal_blocking` per route semantics) + the override policy.+ - [ ] **Move** the connection tree (`temporal_config`, `worker_config`, `queue_options`, `search_attributes`) **out** of `pipelex_{env}.toml` into baked `.pipelex/temporal_{env}.toml` (the D2 plugin-config file). The main `pipelex_{env}.toml` carries no `[temporal.*]` keys — core would reject them.+- [ ] **Env-config child Dockerfile** stays a `COPY` — now copying `pipelex_{env}.toml` **and** `temporal_{env}.toml`.+- [ ] Record the (base `pipelex-api` version ⇄ `pipelex-temporal` ref) pair in the flavor's version file.++### 🛑 CHECKPOINT D — hard stop · **MAJOR GATE 3** (source-doc Checkpoint 3)++First flavor boots end-to-end — validates the whole chain before replicating for Mistral.++- [ ] **Verify:** dev image boots with Temporal active; a run enqueues to Temporal Cloud; `PIPELEX_ENV` selects the right `temporal_{env}.toml`. Validate runs DIRECT in-process (F2) and the runner is sized for the library load.+- [ ] **Capture cold-start context:** append `### Phase D — as-built` (the two-layer Dockerfile shape, the SSH/deploy-key CI wiring, the exact config keys moved to `temporal_{env}.toml`, the version-pair record location, any e2e boot evidence).+- [ ] **Fan-out `/code-review`:** sub-agent runs `/code-review` on the `pipelex-api-hosted` diff (Dockerfile + config migration). Triage as above.+- [ ] **Commit** in `pipelex-api-hosted` (its own branch). This is where outward-facing/deploy-breaking work begins — confirm go/no-go with the user before any actual deploy.++---++## Phase E — `pipelex-api-mistral` (Mistral flavor, new repo) — GATED++**Goal:** prove the base supports a *second*, differently-shaped (per-call) orchestrator. **Gated on `pipelex-mistralai-workflows` being published to PyPI and its in-flight `mistralai-2.x` work settling — coordinate, don't disrupt.**++- [ ] **Scaffold** mirroring `pipelex-api-hosted` conventions.+- [ ] **Flavored base** (`pipelex-api-mistral`): `FROM pipelex/pipelex-api:<ver>` + `uv pip install pipelex-mistralai-workflows==X` (PyPI — no SSH, no `git` re-add).+- [ ] **Config** in baked `.pipelex/pipelex_{env}.toml`: `[api] execution_mode = "mistral_native"`. **No** `boot_orchestrator` (per-call orchestrator), **no** plugin config file (the Mistral plugin self-loads none today).++### 🛑 CHECKPOINT E — hard stop++- [ ] **Verify:** image boots with `mistral_native` dispatchable; a run routes through the Mistral orchestrator; no Temporal anything present (proves the base is genuinely agnostic, not Temporal-shaped).+- [ ] **Capture cold-start context:** append `### Phase E — as-built` (confirming the per-call vs boot-global distinction held in practice: no `boot_orchestrator`, no plugin config file needed).+- [ ] **Fan-out `/code-review`:** sub-agent runs `/code-review` on the new-repo scaffold + flavor config. Triage as above.+- [ ] **Commit** in `pipelex-api-mistral` (new repo).++---++## Phase F — `pipelex-worker` pin flip (private; lands with the release)++**Goal:** flip the worker to the externalized Temporal plugin. Per the archived plugin-system cut-list — land in the **same commit** as the `pipelex` release pin bump.++- [ ] `pyproject.toml`: `pipelex[dynamodb,s3,temporal]==X` → `pipelex[dynamodb,s3]==X` + `pipelex-temporal @ git+ssh://…@<ref>`.+- [ ] `Dockerfile` CMD `["pipelex","worker",…]` → `["pipelex-temporal","worker",…]`; `Makefile` `pipelex worker` → `pipelex-temporal worker`.+- [ ] Set `boot_orchestrator = "temporal"` for the worker process.+- [ ] Run downstream consumers' suites against the new pins **before** publishing.++### 🛑 CHECKPOINT F — hard stop (final)++- [ ] **Verify:** worker runs on the externalized plugin; Temporal is fully external (no `temporal` extra anywhere in the public base); consumer suites green against the new pins.+- [ ] **Capture cold-start context:** append `### Phase F — as-built` (the published `pipelex-api` version, the `pipelex-temporal` ref, the pins flipped per repo, consumer-suite green evidence).+- [ ] **Fan-out `/code-review`:** sub-agent reviews the pin-flip diffs.+- [ ] **Ship:** coordinate the cross-repo release. Effort complete.++---++## Per-repo handoff summary++| Phase | Repo | Change | Channel |+|---|---|---|---|+| A | `pipelex` (`_plugins`) | D2 env-aware plugin-config helper; F3 `add_http_error_mapper` seam + `PLUGIN_API_VERSION` bump; SPI doc | public, normal release |+| B | `pipelex-temporal` | use D2 helper (`temporal_{env}.toml`); register F3 `TemporalError → ErrorReport` mapper | private git+ssh |+| C | `pipelex-api` | drop `temporal` extra; execution via bridge (F1); validate DIRECT (F2); consume F3 mappers; remove all `pipelex.temporal`/`temporalio` imports | public Docker Hub |+| D | `pipelex-api-hosted` | flavored-base install of `pipelex-temporal` (git+ssh); `boot_orchestrator=temporal` + `[api] execution_mode`; connection tree → `temporal_{env}.toml` | private ECR |+| E | `pipelex-api-mistral` (new) | scaffold; flavored-base install of `pipelex-mistralai-workflows` (PyPI); `[api] execution_mode=mistral_native`, no boot gate | private ECR |+| F | `pipelex-worker` | pin `pipelex-temporal` (git+ssh); CLI → `pipelex-temporal worker`; `boot_orchestrator=temporal` | private |++---++## Open items / risks to verify during execution++- **F3 — confirm transport faults are wrapped, or rely on the mapper.** Verify whether the Temporal orchestrator already wraps bare `temporalio` transport errors into `PipelexError`s (`TemporalServerError` etc.) before they reach HTTP. If it does, the F3 mapper is belt-and-suspenders; if not, it is the only thing keeping such errors off the catch-all 500. Either way the mapper makes the base correct without naming Temporal.+- **F1 — define the override policy precisely.** Decide the exact shape (`allow_request_execution_mode_override` bool vs a per-deployment allowlist of modes) and the default — recommend override **off** on hosted flavors. A caller must never force `direct` on a runner whose whole point is distributed execution.+- **F2 — quantify the library-load cost on the hosted runner.** Validate now loads the method library API-side. Confirm the hosted runner image/resources are sized for it; document in `pipelex-api` / `pipelex-api-hosted` docs.+- **Version-lock bookkeeping.** Record the (base `pipelex-api` version ⇄ plugin ref) pair in each flavor repo's version file so a bump can't drift them apart.+- **Mistral publish gating.** `pipelex-mistralai-workflows` must be published to PyPI and its `mistralai-2.x` work settled before Phase E.+- **Docs deploy.** The Temporal error pages left in core `docs/errors/` (type_uri dereference targets) still need a home decision when `pipelex-temporal` docs exist — tracked in the archived Phase-5 cut-over notes, not here. **Concrete instance from Phase B:** the F3 mapper emits `type_uri = .../errors/temporal-transport-error/`, but `TemporalTransportError` is a synthetic `error_type` string (not a `PipelexError` subclass), so `generate-error-pages` never emits a page — the RFC 7807 `type` link 404s. This is **pre-existing parity** (pipelex-api's old `handle_temporal_error` emitted the identical dangling URI), now plugin-owned. Resolve with the same "where do Temporal docs live" decision: author a `<!-- pipelex:authored -->` page once a home exists.++---++## As-built log (append per phase at each checkpoint — keep this current for cold starts)++> Each checkpoint appends an `### Phase N — as-built` subsection here with: final names/signatures, divergences from plan, test evidence, and anything a cold resume needs.++### Phase A — as-built++**Branch:** `refactor/Plugins-4` (worktree `_plugins`), cut from `refactor/Plugins-3`. One commit for the whole phase.++**D2 — env-aware plugin config loader.** `pipelex/system/configuration/config_loader.py`:++- `ConfigLoader.load_plugin_config(*, name: str, package_dir: Path, schema: type[_PluginConfigT], extra_overrides: dict[str, Any] | None = None) -> _PluginConfigT` — loads, deep-merges and **validates** (returns the `schema` instance, not a raw dict — the `schema` param from the plan is honored as a validated return).+- **Signature divergence from the plan sketch `load_plugin_config(*, name, schema)`:** added `package_dir: Path`. The packaged default `{name}.toml` lives inside the *plugin's own* distribution; core cannot locate it, so the plugin passes its package dir (`Path(__file__).parent`). This is necessary, not optional.+- Layering (later wins per leaf, via `deep_update`): `{package_dir}/{name}.toml` → global `~/.pipelex/{name}_{env}.toml` → global `~/.pipelex/{name}_override.toml` → project `.pipelex/{name}_{env}.toml` → project `.pipelex/{name}_override.toml` (project only if distinct from global) → `extra_overrides`. `env` = `runtime_manager.environment`.+- Helper `ConfigLoader._plugin_override_files_for_dir(config_dir, *, name)` mirrors `_override_files_for_dir` but is **intentionally narrower**: only `{name}_{env}` + `{name}_override` — no `local` / `run_mode` / `temporary` tiers (a plugin config is env-selected + deployment-baked, not developer-scratch-layered). Missing files at any tier are skipped, so the packaged default alone is valid. Does **not** call `ensure_global_config_exists()` (no kit-template copy for plugin configs).+- New module-level `_PluginConfigT = TypeVar(..., bound=BaseModel)`; added `from pydantic import BaseModel`.++**F3 — HTTP-error-mapper seam.** `pipelex/plugins/registrar.py` (reflects the **lazy-provider refinement** landed during Phase B — see the next paragraph for why):++- Type aliases `HttpErrorMapperFn = Callable[[Exception], ErrorReport]` and `HttpErrorTypeProviderFn = Callable[[], type[Exception]]` (`ErrorReport` from `pipelex.base_exceptions` — a core type, no cycle, no FastAPI/SDK pull).+- Menu method `PluginRegistrar.add_http_error_mapper(*, exc_type_provider: HttpErrorTypeProviderFn, to_error_report: HttpErrorMapperFn) -> None` — the exc type is supplied as a **thunk**, not the bare class. `register` only appends a `_HttpErrorMapperContribution(exc_type_provider, to_error_report, source_plugin)` NamedTuple to `self.http_error_mappers: list[...]` + a generic `"http error mapper"` contribution line; it neither resolves the type nor imports the SDK.+- Read accessor `PluginRegistrar.get_http_error_mappers() -> dict[type[Exception], HttpErrorMapperFn]` **runs every provider** to build a freshly-keyed `{exc_type: mapper}` dict (so any SDK import is paid here, at the host's read time, not at registration), fail-loud `DuplicateHttpErrorMapperError` naming both plugins on a duplicate *resolved* type. Consumer can't mutate registrar state.+- New error class `DuplicateHttpErrorMapperError(PluginError)` in `pipelex/plugins/exceptions.py` (mirrors `DuplicateOrchestratorError`; fields `exc_type`/`first_plugin`/`second_plugin`).++**Why lazy (the Phase-B-forced refinement).** Phase A originally shipped `add_http_error_mapper(*, exc_type: type[Exception], …)` keyed on the concrete class. Phase B exposed the conflict: naming `temporalio.exceptions.TemporalError` imports the whole `temporalio` SDK (measured ~146 ms — pulls the `temporalio.api`/`temporalio.bridge` protobuf chain), so a bare `exc_type=` forces that import inside `TemporalPlugin.register` — violating Checkpoint B's hard acceptance ("import-light register, no temporalio at registration") and the plugin's own thunk discipline. The provider thunk defers the SDK import to `get_http_error_mappers` (a host runtime's app construction), where the plugin — and therefore its SDK — is by definition installed. Duplicate-by-type detection moves with it (to resolution time). `PLUGIN_API_VERSION` stays **2** (same capability, refined signature; nothing external consumed v2 yet). Landed as its own commit on `refactor/Plugins-4`.++**No hub wiring (deliberate).** Phase A is exactly the planned surface: registrar method + accessor. The mappers are **not** pushed onto the hub. **How `pipelex-api` reaches them in Phase C:** call `build_registrar(config=get_config())` at app construction (documented pure/repeatable fn; `cli/commands/plugins_cmd.py` already does this standalone) and iterate `registrar.get_http_error_mappers()`. No further core exposure needed → Phase C stays pipelex-api-only. If Phase C prefers reading the boot registrar instead of rebuilding, that needs a *new* public getter (the registrar is `Pipelex._plugin_registrar`, private today) — small follow-up, decide in Phase C.++**Version bump.** `pipelex/plugins/contract.py`: `PLUGIN_API_VERSION: int = 1 → 2` (+ comment noting v2 = the optional `add_http_error_mapper` capability). `PipelexPlugin` docstring corrected (dropped the stale "CLI commands" mention removed by Option A; added "model listers" and "HTTP-error mappers"). **`targets_api` re-confirmation:** all built-ins set `targets_api = PLUGIN_API_VERSION` (auto-tracks the bump — verified). **Still to re-confirm in Phase B:** the external `pipelex-temporal` plugin (and `pipelex-mistralai-workflows`, already an entry-point plugin) — both import `PLUGIN_API_VERSION` from core, so an editable-pinned core gives them `2` automatically, but their suites must run against the Phase A core to confirm.++**SPI doc.** `docs/under-the-hood/orchestrator-plugins.md`: new section "HTTP error mappers: rendering an orchestrator's transport faults" (plugin owns classification / core owns transport / host owns presentation), the registrar-menu table row now lists `add_http_error_mapper` + `get_http_error_mappers`, and the `PLUGIN_API_VERSION = 2` note. Error reference page `docs/errors/duplicate-http-error-mapper-error.md` generated via `pipelex-dev generate-error-pages` (+ index line in `inference-and-providers.md`).++**Tests (all green).**++- `tests/unit/pipelex/system/test_plugin_config_loader.py` — `TestPluginConfigLoader`: packaged-default-only, env-file deep-merge + sibling-env-ignored, override-beats-env, project-beats-global, extra_overrides-win-last, validation-after-merge. Hermetic via `PropertyMock` on `global_config_dir`/`project_config_dir`/`runtime_manager.environment`.+- `tests/unit/pipelex/plugins/test_http_error_mapper_seam.py` — `TestHttpErrorMapperSeam`: add+get (resolved type), **provider resolved lazily not at registration** (the import-light proof — an exploding provider survives `register`), contribution recorded without resolving, duplicate fails loud naming both at resolution time, accessor returns a fresh dict, empty-by-default.+- `tests/unit/pipelex/plugins/test_import_light_boot.py` — extended the BLOCKED set with `fastapi`/`starlette` so the subprocess guard now also proves registration imports no web framework (locks the F3 invariant at the discovery layer).++**Gates:** `make agent-check` clean (ruff/plxt/pyright 0/mypy 0/keyword-only ✓) · `make tb` green · `make agent-test` green. Core names no orchestrator; no `import fastapi`/`starlette` anywhere in core (the only `fastapi` strings are telemetry-integration labels + the `WorkerMode.FASTAPI` enum value, not imports).++**Code-review triage** (forked `/code-review`, high effort — verdict: clean, correct, faithful to D1/D2/F1/F2/F3; no bugs):++- **Applied:** test `begin_plugin(targets_api=…)` now uses `PLUGIN_API_VERSION` instead of a literal `2` (nit #2 — latent stale-on-bump).+- **Deferred (design tradeoff):** the Phase-C consumer path for the mappers (registrar has the accessor but it isn't on the hub and the boot registrar is private) → captured in [`wip/plugins/phase-c-http-error-mapper-consumer-path.md`](wip/plugins/phase-c-http-error-mapper-consumer-path.md). Recommended default: Phase C calls `build_registrar(config=get_config())` and iterates `get_http_error_mappers()` (keeps Phase C pipelex-api-only).+- **Info / no action:** `DuplicateHttpErrorMapperError` message uses `__qualname__` (only ever fires on a true same-class dup, so unambiguous); `load_plugin_config` has no `is_unit_testing` hermetic branch (correct — no run_mode tier; Phase B tests must mock the config dirs); D2 intentionally has no `.pipelex/{name}.toml` *base* tier (packaged default is the only base — worth a one-line plugin-authoring-doc callout when Phase B lands).++### Phase B — as-built++**Repos/branches:** core seam refinement on `refactor/Plugins-4` (`pipelex`, worktree `_plugins`) commit `75196e2e3`; Phase B proper on `refactor/own-temporal-config` (`pipelex-temporal`). pipelex-temporal pins core editable (`[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }`), so it builds against the Phase A core live.++**Seam refinement (core, `75196e2e3`) — prerequisite, see Phase A as-built "Why lazy".** `add_http_error_mapper(*, exc_type_provider: Callable[[], type[Exception]], to_error_report)`; `register` only stores a `_HttpErrorMapperContribution` NamedTuple; `get_http_error_mappers()` runs the providers at the host's read time and dedups on the resolved type. This is what lets `TemporalPlugin.register` name `temporalio.TemporalError` without importing `temporalio` (which costs ~146 ms — the whole api/bridge protobuf chain).++**D2 adoption.** `pipelex_temporal/temporal_config_loader.py`: `load_temporal_config()` now delegates to `config_manager.load_plugin_config(name="temporal", package_dir=Path(__file__).resolve().parent, schema=Temporal)`. Dropped the hand-rolled `load_toml_from_path` + single-`resolve_config_file` override; the helper layers packaged `temporal.toml` → global `~/.pipelex/temporal_{env}.toml` → `temporal_override.toml` → project `.pipelex/{...}` with deep-merge, `env = runtime_manager.environment`. **Behavior change:** the old loader read a plain `.pipelex/temporal.toml`; the new layering has **no plain-`.pipelex/temporal.toml` base tier** (packaged default is the only base — D2 design). No such override file existed in the repo, so nothing silently dropped. The env files themselves are baked per-deployment in Phase D; Phase B only switches the loader to support the layering. **Phase-D guardrail (deferred, from the Phase B `/code-review`):** a stray suffix-less `.pipelex/temporal.toml` left over after a botched migration would now vanish silently — a one-line `log.warning` when such a file is found would catch it. Deferred as an enhancement (not added now per smallest-correct-surface; revisit when Phase D migrates the connection tree).++**F3 mapper.** `pipelex_temporal/temporal_plugin.py`: two new module-level fns + one `register` line (unconditional, alongside the orchestrators — an API runner that only dispatches Temporal per-request still needs the mapper):++- `_temporal_transport_error_type() -> type[Exception]` — lazily `from temporalio.exceptions import TemporalError; return TemporalError`. Keyed on the **base** `TemporalError`: `RPCError` and `WorkflowFailureError` both subclass it, so the host's exception-MRO walk catches every bare transport fault. Invoked only by `get_http_error_mappers()`, never at register.+- `_temporal_transport_error_to_report(exc) -> ErrorReport` — ports pipelex-api's old `handle_temporal_error`: `error_type="TemporalTransportError"`, `message=str(exc)`, `title=pascal_case_to_sentence(...)` → "Temporal transport error", `type_uri=f"{URLs.error_docs_base}/{pascal_case_to_kebab(...)}/"` → `.../errors/temporal-transport-error/`, `error_category="transient"`, `error_domain=ErrorDomain.RUNTIME`, `retryable=True`. Imports no `temporalio` (only `str(exc)`); the identity pair uses the same core helpers `PipelexError.title`/`type_uri` use, so the docs URI matches the canonical scheme.+- Module-top imports added: `ErrorDomain`, `ErrorReport` (`pipelex.base_exceptions`), `pascal_case_to_kebab`/`pascal_case_to_sentence` (`pipelex.tools.misc.string_utils`), `URLs` (`pipelex.urls`) — all import-light core, no SDK.++**F3 is mostly belt-and-suspenders (open-item resolved).** `pipelex_temporal/tprl/workflow_caller.py` already wraps `RPCError` / `WorkflowFailureError` / `WorkflowAlreadyStartedError` into `WorkflowExecutionError` (a `PipelexError`) inside `execute_workflow`/`start_workflow`, so the common transport faults reach the API as `PipelexError` (handled generically by `handle_pipelex_error`). The F3 mapper is the safety net for any temporalio error that escapes that wrapping (e.g. a connection error raised outside execute/start) — and, more importantly, it is what makes the public `pipelex-api` base render such a fault correctly **without naming Temporal**. Implemented as planned regardless.++**`targets_api`.** Unchanged: `TemporalPlugin.targets_api = PLUGIN_API_VERSION` auto-tracks to 2 via the editable core pin. The whole suite running against the Phase A core is the confirmation.++**Tests (pipelex-temporal, all green).**++- `tests/unit/pipelex_temporal/test_temporal_config_env_layering.py` — `TestTemporalConfigEnvLayering`: packaged-default-only valid; `temporal_{env}.toml` deep-merges onto the real packaged default (overridden leaf wins, untouched leaves + sibling env ignored). Hermetic via `PropertyMock` on `ConfigLoader.global_config_dir`/`project_config_dir` + `runtime_manager.environment` (mirrors core's `test_plugin_config_loader`).+- `tests/unit/pipelex_temporal/test_temporal_plugin_http_error_mapper.py` — `TestTemporalPluginHttpErrorMapper`: register → `get_http_error_mappers()` keyed on `TemporalError` + full classification through the public seam; **import-light proof** = a fresh subprocess with a `temporalio` meta-path blocker imports the plugin and runs `register` (both `boot_orchestrator=None` and `"temporal"`) to completion, asserting `temporalio` never lands in `sys.modules`.+- Existing `test_temporal_config_loader.py` (packaged-default + hub cache) still green against the new loader.++**Gates:** pipelex-temporal `make agent-check` clean (ruff ✓ / pyright 0 / mypy 0) · `make agent-test` GREEN against the Phase A core (full suite incl. `temporal`-marked workflow integration tests, exit 0). Core (`refactor/Plugins-4`) `make agent-check` + plugins tests + `make tb` also green.++**Code-review triage** (two forked `/code-review`s, high effort — both verdicts: **clean, no bugs**):++- _Core seam refinement (`75196e2e3`):_ correct + internally consistent (no dangling `_http_error_mapper_sources`; `_add` still drives the 3 sibling adders; dedup attribution at resolution time correct; repeated `get_http_error_mappers()` side-effect-safe; NamedTuple standards-correct). **Deferred (recorded in `wip/plugins/phase-c-http-error-mapper-consumer-path.md`):** dedup raise now lands at the host's app-construction read time (theoretical for one-plugin-per-image); `plugins list` shows a generic `"http error mapper"` line (lost the per-type label — optional `display_label` param if it ever matters).+- _pipelex-temporal Phase B:_ F3 classification verified **byte-exact** vs pipelex-api `handle_temporal_error` (title/type_uri identical — same `URLs.error_docs_base` + helpers); `TemporalError`-base keying complete via MRO; import-light genuinely proven. **Applied:** import-light subprocess now also asserts `slot_claims` count per boot arm (0 off-gate, 4 on `"temporal"`). **Deferred:** dangling `temporal-transport-error` docs URI (pre-existing parity → folded into the "Temporal error pages need a home" open item); D2 base-tier-drop `log.warning` guardrail (Phase D).++### Phase C — as-built++**Repo/branch:** `pipelex-api` on `refactor/orchestrator-agnostic-base` (cut from `dev`), commit `a39841e`. The `pipelex` worktree only carries this tracker update (`refactor/Plugins-4`). Nothing pushed. pipelex-api pins core **editable** (`[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }`) so it builds against the Phase A core live; the published `==<ver>` pin (no `temporal` extra) is the release-time flip. Nothing pushed.++**Decision — `[api]` config home = Option C (the user chose it).** `execution_mode` + override policy live in a **pipelex-api-owned `api/api.toml`**, loaded via the Phase A D2 helper `config_manager.load_plugin_config(name="api", package_dir=Path(api/), schema=ApiConfig)` — NOT a core `[api]` section (core config is `extra="forbid"`, would reject it → would need a 2nd core change, which the plan forbids). **Mechanics gotcha:** `load_plugin_config` validates the **whole** merged TOML document against the schema (exactly like `temporal.toml`), so `api.toml` keys are **root-level — no `[api]` table wrapper** (the user's preview showed `[api]`; the real file has root keys; functionally identical). `ApiConfig(BaseModel, extra="forbid")` has NO field defaults (defaults live in `api.toml`, mirroring core's discipline): `execution_mode: PipelexExecutionMode`, `allow_request_execution_mode_override: bool`. Base ships `execution_mode = "direct"`, override off. `get_api_config()` is `@cache`'d (immutable per process), **warmed in `lifespan` after `Pipelex.make`** for fail-fast (matches `ERROR_DISCLOSURE`). A flavor bakes `.pipelex/api_{env}.toml` (Phase D).++**F1 — dispatch DEVIATION from the literal plan (flagged; the only option satisfying all three locked constraints).** The plan's F1 wording said "build `PipelexPipeRunInput` and dispatch via `run_pipe_via_bridge`." **That path is lossy:** `PipelexPipeRunInput` carries no `request_id` / `output_multiplicity` / `dynamic_output_concept_ref`, and `build_pipe_job_from_input` skips pipeline-manager registration (the 409-conflict path) + telemetry — so routing `start` through it would silently regress those features. The only non-lossy bridge variant is extending the **core** `PipelexPipeRunInput` = a 2nd core change the plan forbids ("Phase C = pipelex-api only"). **Chosen instead (Design B):** keep `pipeline_run_setup(...)` to build the rich `PipeJob` (preserves everything → byte-equivalent, which Checkpoint C explicitly demands), then dispatch via the agnostic hub seam `get_orchestrator_registry().get_optional(mode=execution_mode).run(pipe_job=…, delivery_assignment=…)` — the **identical final dispatch** `run_pipe_via_bridge` performs internally (bridge.py:92-95), just fed a live job instead of a serialized lossy input. Names no orchestrator, imports no SDK. `None` orchestrator → `MissingOrchestratorError(mode)` (a `PipelexError` → `handle_pipelex_error`, carries the install hint). `workflow_id` read off `run_output.workflow_id` (Temporal FF sets it, DIRECT returns `None`); returned `pipeline_run_id` stays the `pipeline_run_setup`-resolved id (byte-equivalent to old). Verified equivalent against `TemporalFireAndForgetOrchestrator.run` (pipelex-temporal) which calls the same `temporal_pipe_run.start(pipe_job)` the old code did. `execution_mode` resolved **first** in `start()` (before `pipeline_run_setup`) so a forbidden override 403s before any library load.++**F1 — override policy.** Per-request mode rides `PipelineApiExtras.execution_mode` (API-server extra; route `_validate_extras` parses it, `start` route passes `requested_execution_mode=`). `resolve_execution_mode(requested, *, config)`: `None`/equal-to-default → default; differs + override on → requested; differs + override off → `raise_forbidden(403, ErrorType.EXECUTION_MODE_OVERRIDE_FORBIDDEN)`. Documented in OpenAPI only via the `PipelineApiExtras` field description (NOT added to `PipelexApiStartRequest`, so the committed OpenAPI artifact's `/start` schema is unchanged — only the validate-route docstring regen'd).++**F2 — validate always DIRECT.** `ApiRunner.validate` collapsed to the single `validate_bundles_in_process(...)` path; dropped the `get_config().temporal.is_enabled` branch + `dispatch_dry_validate`/`DryValidateArg`/`PipelexInterpreter`/`build_validation_report` imports; dropped the route's `WorkflowExecutionError` catch in `validate.py` (unreachable). **Behavior change documented** in `pipelex-api/docs/pipe-validate.md` ("Where validation runs" + a deployment resource note: the runner now loads the method library API-side — size a flavor's runner for it).++> **F2 SUBSEQUENTLY REVERSED / EXTENDED** by the orchestrator-dispatched-`/validate` follow-on (its own plan: [`wip/plugins/orchestrator-dispatched-validate.md`](wip/plugins/orchestrator-dispatched-validate.md)). `/validate` is now `execution_mode`-aware through a new per-call `BundleValidatorRegistry` seam (V0 core, V1 `pipelex-temporal`, V2 `pipelex-api`): `ApiRunner.validate` (the override) was **removed** in favor of `ApiRunner.validate_verdict` (verdict-as-value); the route discriminates on `isinstance(verdict, PipelexValidationReport)` instead of catching `ValidateBundleError`. `direct` keeps the in-process path; `temporal_*` dispatches to a worker and assembles the same canonical report API-side. The library-load note is now flavor-conditional. Branches: all three repos on `feature/Orchestrator-dispatched-validate`.++**F3 — consume the mappers.** `api/exception_handlers.py`: dropped `from temporalio.exceptions import TemporalError` + hardcoded `handle_temporal_error`; `register_exception_handlers(app, *, disclosure_mode, http_error_mappers=None)` now registers one FastAPI handler per `(exc_type, mapper)` via `_make_orchestrator_error_handler` (runs the mapper → existing `_problem_response` RFC 7807 + DisclosureMode render). **Consumer path = deferred-doc option 1 (no 2nd core change):** `api/main.py` resolves the mappers at module import via `build_registrar(config=PipelexConfig.model_validate(config_manager.load_config())).get_http_error_mappers()` (the standalone pattern `plugins_cmd.py` uses — pure, pre-boot-safe; smoke-confirmed empty in the base). The module keeps its import-side-effect-free contract (mappers are a param, not a `build_registrar` call inside it), so tests register handlers on throwaway apps. **Base = zero mappers** ⇒ a bare orchestrator transport fault with no plugin installed falls to the sanitized catch-all 500 (the dedicated transient classification is now plugin-owned; intended for the agnostic base — proven by `test_unmapped_transport_error_falls_back_to_sanitized_500`).++**F3 test pattern (synthetic plugin, no temporalio).** `test_exception_handlers.py`: `_SyntheticTransportError(Exception)` (bare, non-PipelexError) + `_SyntheticOrchestratorError(PipelexError)` + `_synthetic_transport_to_report` mapper. `_synthetic_http_error_mappers()` builds a real `PluginRegistrar`, `begin_plugin(...)`, `add_http_error_mapper(exc_type_provider=lambda: _SyntheticTransportError, …)`, returns `get_http_error_mappers()` — exercises the **real producer→consumer seam** (same as `api/main.py`) without installing a plugin or importing an SDK. Conformance webhook test now mocks `get_orchestrator_registry` (fake orchestrator `run` does the real `DeliveryExecutor` delivery + returns a FF `PipelexPipeRunOutput`).++**Deps + base config.** `pyproject.toml`: `pipelex[mistralai,anthropic,google,google-genai,bedrock,fal]` (dropped `temporal`). Removed the whole `[temporal]` block from the base `.pipelex/pipelex.toml` (core would reject it). **Local-env cleanups (NOT committed, machine-only):** a stale `[temporal]` section in the gitignored `.pipelex/pipelex_override.toml` (had `worker_scopes`/search-attributes) broke loading against the temporal-free core — removed it; the global `~/.pipelex/pipelex.toml` only had the valid `tracing_config.temporal_dynamodb` backend (left alone). `tests/unit/conftest.py`: dropped the removed `temporal_enabled=False` kwarg from `Pipelex.make` (agnostic base defaults to in-process DIRECT, which is what the hermetic suite wants).++**Gates (all green):** `make lint` (ruff, incl. 2 TC002 moves — `PipelexValidationReport`+`PipelexExecutionMode` to TYPE_CHECKING in pipeline.py, both annotation-only under `from __future__ import annotations`; pydantic `models.py` keeps its runtime import) · `make pyright` 0 · `make mypy` 0 · `make pylint` 10.00/10 · **307 tests pass** (`make t`, xdist) · `make openapi-check` ✓ (only the validate docstring regen'd). `grep -rn temporal api/` → only incidental prose (`pipelex-temporal` plugin name, `temporalio.TemporalError` in a docstring, the execution-mode enum value strings). `import api.main` → `temporalio` NOT in `sys.modules` (agnostic base proven).++**Docs.** `docs/configuration.md` (new "Execution mode" section + de-Temporal'd the intro), `docs/pipe-validate.md` ("Where validation runs" + resource note), `docs/pipe-run.md` / `docs/index.md` / `docs/error-responses.md` (de-Temporal'd stale mentions), `CHANGELOG.md` `[Unreleased]`.++**Code-review triage** (two clean-context `/code-review`s on the working-tree diff — correctness/byte-equivalence + tests/config; both verdicts: **correct, byte-equivalence holds, no silent bugs, no 500 regression, clean coupling**):++- **Applied (5):** (1) _Real coverage loss_ — re-added the validate **no-verdict→5xx** invariant test (deleted with the Temporal arm; route still upholds it but nothing tested it) as `test_non_verdict_failure_is_not_a_200_verdict` (mocks `validate` to raise a non-`ValidateBundleError` `PipelexConfigError` → asserts ≥500 + problem+json + `is_valid` absent). (2) _Direct-mode `/start` blocks_ — the agnostic base runs `direct` in-process to completion before the 202, so the "asynchronous/non-blocking" promise is a **distributed-flavor** property; clarified in the `start` route docstring + `docs/pipe-run.md` (callback still fires either way). (3) Conformance webhook test now asserts `get_optional.assert_called_once_with(mode=DIRECT)` + `run.assert_awaited_once()` (closes the "wrong mode silently dispatched" gap). (4) conftest `get_api_config.cache_clear()` on setup+teardown (insurance vs `@cache` leak across tests). (5) Stale-doc fixes: `handle_unexpected_error` docstring + `ErrorType.INTERNAL_SERVER_ERROR` comment (dropped the `TemporalError` mention) + `test_validate_errors` module docstring.+- **Deferred (pre-existing / design, recorded here):** _`_validate_extras` mislabels error_type_ — a malformed `execution_mode`/`pipeline_run_id` in the start extras is a correct 422 but labeled `InvalidCallbackUrls` (one catch covers all three extras; the `message` still names the real field). Pre-existing pattern my new field slightly widened; proper fix = per-field error_type discrimination (more surface) → follow-up, untested today. _Double `build_registrar`_ — runs once at `api/main.py` import (mapper resolution) and once in `Pipelex.make` boot; harmless (pure fn), the deferred-doc option-1 tradeoff.+- **Verdict on the F1 deviation:** the correctness reviewer independently **verified byte-equivalence** of the orchestrator-registry dispatch vs the old `make_temporal_pipe_run().start()` (same `pipe_job`, same `delivery_assignment`, `pipeline_run_id` still from `pipeline_run_setup`, `workflow_id` off `run_output` — the F&F orchestrator calls the identical `temporal_pipe_run.start`). No silent change.++**Gates re-run after triage:** lint ✓ · pyright 0 · mypy 0 · pylint 10.00/10 · **308 tests pass** · openapi-check ✓ (the `start` + `validate` route docstrings regen'd the artifact).diff --git a/docs/404.html b/docs/404.htmlindex 323d51cd6..dfa8b81cf 100644--- a/docs/404.html+++ b/docs/404.html@@ -103,7 +103,13 @@'home/8-analytics/': 'analytics/','home/9-tools/': 'tools/','home/10-advanced-customizations/': 'advanced/',- 'under-the-hood/temporal-deployment': 'distributed-execution/cluster-setup',+ 'under-the-hood/temporal-deployment': 'distributed-execution',+ 'under-the-hood/temporal-integration': 'under-the-hood/runtime-bridge-and-transport',+ 'distributed-execution/temporal': 'distributed-execution',+ 'distributed-execution/cluster-setup': 'distributed-execution',+ 'distributed-execution/workers': 'distributed-execution',+ 'distributed-execution/task-routing': 'distributed-execution',+ 'distributed-execution/observability': 'distributed-execution',};function rewritePath(p) {diff --git a/docs/agents/debugging-hanging-pytest-runs.md b/docs/agents/debugging-hanging-pytest-runs.mdindex 3cf8c7ae4..2fcfe6b38 100644--- a/docs/agents/debugging-hanging-pytest-runs.md+++ b/docs/agents/debugging-hanging-pytest-runs.md@@ -117,9 +117,10 @@ echo "exit=$?" # 0=pass, 1=fail, 124=outer timeout# 3. If any failures, grep by error class namegrep -B 2 -A 10 "YourErrorClass" /tmp/pytest_unit.log-# 4. If integration temporal tests are in scope, run them serially with shorter timeout-timeout 240 .venv/bin/pytest --timeout=60 --timeout-method=thread -q \- tests/integration/pipelex/temporal/ > /tmp/pytest_temporal.log 2>&1+# 4. The Temporal integration tests live in the sibling `pipelex-temporal` repo —+# if they're in scope, run them there, serially, with a shorter timeout+( cd ../pipelex-temporal && timeout 240 .venv/bin/pytest --timeout=60 --timeout-method=thread -q \+ tests/integration/pipelex_temporal/ > /tmp/pytest_temporal.log 2>&1 )echo "exit=$?"```@@ -158,7 +159,7 @@ In all four cases, write up what you know in 3–4 sentences and surface to the## Codebase-specific notes (Pipelex / Temporal)- `make agent-test` uses `-n auto -q` — both buffer output and obscure progress. For debug runs, invoke pytest directly with `-v` and direct file redirect.-- `boot_temporal` fixture teardown (in `tests/integration/pipelex/temporal/conftest.py`) is the known fixture-teardown hang point. If a hang traceback mentions `manager.teardown()` or `temporal_hub.reset()`, it's this.+- `boot_temporal` fixture teardown (in the sibling `pipelex-temporal` repo's `tests/integration/pipelex_temporal/conftest.py`) is the known fixture-teardown hang point. If a hang traceback mentions `manager.teardown()` or `temporal_hub.reset()`, it's this.- xdist worker crashes on integration temporal tests often show as `[gw N] node down: Not properly terminated` followed by `replacing crashed worker`. Treat as infrastructure flake until you've proven otherwise serially.- `is_in_temporal_workflow()` is the right gate for code that might be invoked from inside a Temporal sandbox — `get_config()` is unsafe to read from inside, and many singletons return sandboxed-empty copies. This was the actual bug I hit; it would have been caught instantly by the playbook above if I'd run the integration tests serially on first iteration instead of trusting the xdist run.- **A payload-conversion error inside a workflow task looks like a clean hang, not a failure.** When the data converter raises while decoding an activity result (or workflow argument), Temporal fails the *workflow task* and retries it indefinitely — the submitter's `execute_workflow` never returns, pytest-timeout's thread-method kill fires, and xdist reports `node down` with the real traceback destroyed along with the process. Historical trigger: kajson ≤ 0.6.0 decoded tz-aware datetimes via `ZoneInfo(...)`, which needs the IANA tz database — absent on slim containers and uv-managed standalone Pythons. Fixed upstream in kajson 0.7.0 (self-sufficient timezone wire format + `tzdata` declared as kajson's own dependency), but any other deterministic decode error (registry miss, schema drift) still presents as this same hang. To diagnose this class of hang on a remote runner, set `PIPELEX_HANG_DUMP_DIR=<dir>`: the temporal integration conftest arms a per-test watchdog that dumps all thread stacks to a file before the kill AND mirrors temporalio's failure logs (which carry the workflow-task traceback) into the same directory; the tests workflow prints the directory on job failure. To reproduce a missing-tz-database environment locally, run with `PYTHONTZPATH=/nonexistent` after uninstalling `tzdata`.diff --git a/docs/building-methods/pipes/run-modes-and-backends.md b/docs/building-methods/pipes/run-modes-and-backends.mdindex b7799a6c3..5e16bf8f1 100644--- a/docs/building-methods/pipes/run-modes-and-backends.md+++ b/docs/building-methods/pipes/run-modes-and-backends.md@@ -22,7 +22,10 @@ Pipelex separates **what the inference leaves do** (the run mode) from **where t| Backend | Trigger | What runs where ||---|---|---|| **Direct** (default) | — | Everything in your process |-| **Temporal** | `--temporal`, or config `[temporal] is_enabled = true` | Controllers as child workflows, inference leaves as activities on workers |+| **Temporal** | `--orchestrator temporal`, or config `plugins.boot_orchestrator = "temporal"` | Controllers as child workflows, inference leaves as activities on workers |++!!! info "Durable backends are a commercial capability"+ The Temporal backend is part of Pipelex's [durable & distributed execution offer](https://pipelex.com/products#durable-execution) — the [Temporal backend](https://pipelex.com/products#temporal) is delivered through the Pipelex platform. The `--orchestrator <name>` flag (and its `plugins.boot_orchestrator` config equivalent) boots the process under the named orchestrator plugin; omit it for in-process execution.The run mode rides across the wire with every inference job, so the backend never changes *what* a leaf does — only *where* it does it. In particular, when you run a **pipeline** in dry mode on Temporal, the real inference activities are still dispatched and mock **inside** them on the worker. That's by design: it lets you test the entire distribution machinery — dispatch, scheduling, serialization, task-queue routing, cross-worker graph tracing — with zero AI spend and no credentials.@@ -45,10 +48,10 @@ pipelex run bundle my_bundle.mthdspipelex run bundle my_bundle.mthds --dry-run --mock-inputs# Live + Temporal-pipelex run bundle my_bundle.mthds --temporal+pipelex run bundle my_bundle.mthds --orchestrator temporal# Dry run + Temporal — dispatches real activities, mocks inside them-pipelex run bundle my_bundle.mthds --temporal --dry-run --mock-inputs+pipelex run bundle my_bundle.mthds --orchestrator temporal --dry-run --mock-inputs```From Python, the same knobs live on the runner — the backend comes from config:@@ -68,10 +71,10 @@ response = await runner.execute_pipeline()`pipelex validate` (see the [validate commands](../../tools/cli/validate.md)) combines static validation with a dry run of every pipe. It relates to the matrix in two specific ways:-- **The CLI sweep always runs in-process**, even when Temporal is enabled. The `--temporal` flag on `pipelex validate` exists precisely to verify that: it flips the boot config so you can confirm a validation sweep does not dispatch anything under a Temporal-enabled setup.-- **The runner API's `/validate` route ships the sweep to a worker — as one activity**: with Temporal enabled, it offloads the *whole* sweep as a single activity (`act_dry_validate`, wrapped in the `WfDryValidate` workflow). Inside that activity, the pipe router and the content generator are scoped to their in-process variants, so nothing nested is dispatched: every dry run in the sweep behaves exactly like the dry × direct cell of the matrix, just hosted on a worker instead of your machine. The per-pipe statuses plus the execution graph come back in one round-trip. See [Temporal integration](../../under-the-hood/temporal-integration.md) for the mechanics.+- **The CLI sweep always runs in-process**, even when Temporal is enabled. The `--orchestrator temporal` flag on `pipelex validate` exists precisely to verify that: it boots validation under the orchestrator plugin so you can confirm a validation sweep does not dispatch anything under a Temporal-enabled setup.+- **The runner API's `/validate` route ships the sweep to a worker — as one activity**: with Temporal enabled, it offloads the *whole* sweep as a single activity (`act_dry_validate`, wrapped in the `WfDryValidate` workflow). Inside that activity, the pipe router and the content generator are scoped to their in-process variants, so nothing nested is dispatched: every dry run in the sweep behaves exactly like the dry × direct cell of the matrix, just hosted on a worker instead of your machine. The per-pipe statuses plus the execution graph come back in one round-trip. See [Runtime Bridge & Transport](../../under-the-hood/runtime-bridge-and-transport.md) for how a job crosses the worker boundary.-This is not a contradiction of "the run mode never changes the backend" — it's the same leaf-mock foundation with a different *backend choice* for the nested work. A `--temporal --dry-run` pipeline run picks the Temporal backend all the way down, fanning out for real to test the distribution machinery. A Temporal validation sweep picks the Temporal backend for exactly one hop (get the sweep onto a worker) and the in-process backend for everything inside, because its goal is a cheap interactive answer, not distribution testing. Two deliberate opposites built from the same parts.+This is not a contradiction of "the run mode never changes the backend" — it's the same leaf-mock foundation with a different *backend choice* for the nested work. A `--orchestrator temporal --dry-run` pipeline run picks the Temporal backend all the way down, fanning out for real to test the distribution machinery. A Temporal validation sweep picks the Temporal backend for exactly one hop (get the sweep onto a worker) and the in-process backend for everything inside, because its goal is a cheap interactive answer, not distribution testing. Two deliberate opposites built from the same parts.## Forced Dry Modediff --git a/docs/distributed-execution/cluster-setup.md b/docs/distributed-execution/cluster-setup.mddeleted file mode 100644index 38d52b22d..000000000--- a/docs/distributed-execution/cluster-setup.md+++ /dev/null@@ -1,156 +0,0 @@-----title: "Cluster Setup"-description: "One-time Temporal namespace prerequisites for Pipelex: custom search attributes, the strict worker-boot check, and the `pipelex setup-temporal-namespace` registration command."------# Cluster Setup--This page covers the one-time cluster-side configuration Pipelex requires before workers can dispatch workflows on a real Temporal cluster. If you're new to distributed execution with Pipelex, start with the [Overview](index.md).-------## Custom search attributes are a strict prerequisite--Pipelex attaches a fixed set of custom `Keyword` search attributes on every workflow start. They are the operator-facing surface in the Temporal dashboard — column filters, list-view queries, audit trails. They are also a **strict prerequisite**: a real Temporal cluster rejects every `StartWorkflowExecution` RPC that references an unregistered custom search attribute with `Namespace ... has no mapping defined for search attribute ...`. Workflows do not silently run with a degraded dashboard — they do not run at all.--The five built-in attributes Pipelex can populate:--| Attribute | Type | Value source |-|-----------------|-----------|---------------------------------------------|-| `PipeCode` | `Keyword` | `pipe_job.pipe.code` |-| `PipelineRunId` | `Keyword` | `pipe_job.job_metadata.pipeline_run_id` |-| `SessionId` | `Keyword` | `TemporalManager.get_instance().session_id` |-| `UserId` | `Keyword` | `pipe_job.job_metadata.user_id` |-| `DomainCode` | `Keyword` | `pipe_job.pipe.domain_code` |--Pipelex only knows how to populate these five built-ins. Arbitrary custom names are out of scope — they would require code that knows the value source.-------## Configurable surface--The `[temporal.search_attributes]` block in `pipelex.toml` controls both what the workflow-start side attaches AND what the worker-boot check requires. The two stay in lockstep by construction.--```toml-[temporal.search_attributes]-# Master toggle. When false, workflow starts don't attach any custom search-# attributes, the worker-boot check is skipped, and the dashboard view falls-# back to WorkflowType / WorkflowId / StartTime only.-enabled = true--# Subset of the five built-in attributes to populate. Names not in this list-# are skipped at workflow-start time AND not required at worker boot. Names-# that are not built-ins are rejected by the config validator with a-# helpful error message.-attributes = ["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"]-```--Two operational modes:--- **`enabled = true` (default).** Every workflow start attaches the listed subset. The worker-boot check refuses to start the worker unless those names are registered on the namespace. Use this when the deployment owns its Temporal namespace and can register attributes once.-- **`enabled = false`.** No custom search attributes are attached at workflow start; the worker-boot check is skipped. Use this when the namespace policy is managed by a third party who will not register attributes for you, or in restricted dev/sandbox setups where the dashboard view does not matter.-------## Hard-fail at worker boot--Every Pipelex worker process performs a one-shot `ListSearchAttributes` call on boot against its connected namespace. The contract:--- **All configured attributes present** → silent, worker proceeds.-- **Some configured attributes missing on a reachable namespace** → worker boot aborts with `SearchAttributeRegistrationError`. The exception message names the missing attributes and includes **both** the Pipelex CLI invocation and the equivalent raw `temporal` CLI command, so operators on either side of the fence can fix the gap.-- **Operator service unreachable (`RPCError`)** → `log.warning(...)` notes the namespace was not reachable. Worker proceeds; the cluster will reject workflow starts at dispatch time if the attributes are not registered, but blocking worker boot during a control-plane outage would be more harmful than helpful.-- **`enabled = false`** → the check is skipped entirely. No `ListSearchAttributes` call is made.-- **Any other exception** → propagates and crashes the worker (real bug).--Failing fast at worker boot — rather than at every workflow dispatch — is the design point of this phase. The previous warn-and-continue behavior produced unactionable `RPCError`s in the activity logs, one per dispatch, instead of one loud, copy-paste-fixable error at boot.--The check runs **once per worker process at boot**. There is no caching across processes — every worker performs its own check.-------## Registering the attributes--Two equivalent registration paths. Pick whichever fits your operator topology.--### Path 1 — `pipelex setup-temporal-namespace`--Run this on the same host (and same `pipelex.toml`) the worker will use. It reads `[temporal.search_attributes].attributes` and `[temporal.temporal_config]`, connects via the same `connect_to_temporal_selected_server` helper as the worker, and registers the missing attributes via `OperatorService.AddSearchAttributes`. Idempotent — re-running after success is a no-op.--```bash-# Default: register against the selected server profile-pipelex setup-temporal-namespace--# Print the equivalent `temporal operator search-attribute create` invocation-# without executing — useful when the namespace admin needs to register on-# the operator's behalf.-pipelex setup-temporal-namespace --dry-run--# Target a non-default server profile from temporal_server_configs.-pipelex setup-temporal-namespace --server testing-```--Use this when the API key your worker uses has the `OperatorService.AddSearchAttributes` permission. On self-hosted Temporal that is usually the case. On Temporal Cloud, worker API keys typically do **not** have this permission — see Path 2.--### Path 2 — Temporal CLI / Cloud admin--Run by a namespace admin (someone whose credentials carry `Namespace Admin` or equivalent permission). The worker API key is not used.--```bash-temporal operator search-attribute create \- --namespace <namespace> \- --name PipeCode --type Keyword \- --name PipelineRunId --type Keyword \- --name SessionId --type Keyword \- --name UserId --type Keyword \- --name DomainCode --type Keyword-```--For Temporal Cloud, the admin can also use:--```bash-tcld namespace search-attributes add \- --namespace <namespace> \- --search-attribute PipeCode=Keyword \- --search-attribute PipelineRunId=Keyword \- --search-attribute SessionId=Keyword \- --search-attribute UserId=Keyword \- --search-attribute DomainCode=Keyword-```--or the Cloud UI: **Namespace → Custom Search Attributes**.--### Path 3 — `pipelex setup-temporal-namespace` falls back to the runbook--When `pipelex setup-temporal-namespace` runs against a Temporal Cloud namespace and the worker API key lacks `OperatorService.AddSearchAttributes` permission, the command catches the `PERMISSION_DENIED` RPC error and prints the exact Path 2 commands. Forward the output to your namespace admin — they can copy-paste it.-------## Permission model for Temporal Cloud self-managed workers--If you run Pipelex workers against a Temporal Cloud namespace your own customer (or your customer's IT team) administers, the practical model is:--- The customer issues you an API key scoped to "workflow start / activity execution" — enough to dispatch workflows but **not** to register search attributes.-- The first time you stand up a worker against a fresh namespace, worker boot will fail with `SearchAttributeRegistrationError` listing the five attribute names.-- You run `pipelex setup-temporal-namespace --dry-run` and send the output to the customer's namespace admin.-- The admin runs `tcld namespace search-attributes add ...` (or uses the Cloud UI) — a one-time setup.-- Subsequent worker boots are silent.--Set `[temporal.search_attributes] enabled = false` if the customer refuses to register the attributes and you accept losing the Pipelex-specific dashboard filters.-------## Bootstrap scripts for production--For production environments — Pipelex Cloud, Temporal Cloud, self-hosted Temporal — the registration step belongs in the namespace bootstrap script alongside other one-time namespace setup (replication config, retention policies, audit hooks). `pipelex setup-temporal-namespace` is a fine choice for that script; it shares the worker's `pipelex.toml` so the names registered there can never drift from the names the worker will dispatch with.-------## In-process test server--The Temporal in-process test server (`WorkflowEnvironment.start_local()`, used by CI and unit tests) starts with no custom attributes registered. The integration test conftest registers them up front via `ensure_required_search_attributes_registered(..., configured_attributes=BUILTIN_SEARCH_ATTRIBUTES)` so all temporal tests can start workflows without per-test bootstrap. There is no need to set `enabled = false` for tests — the conftest covers it.-------## What's next--Once the namespace is set up, head to **[Worker Deployment](workers.md)** to run the worker process that will execute your workflows.diff --git a/docs/distributed-execution/index.md b/docs/distributed-execution/index.mdindex 52c6b1a1c..3c2a21cca 100644--- a/docs/distributed-execution/index.md+++ b/docs/distributed-execution/index.md@@ -1,28 +1,34 @@---title: "Distributed Execution"-description: "Run Pipelex methods as durable workflows on Temporal with your own Pipelex Temporal workers. A second, managed backend (Mistral Workflows) is coming soon."+description: "Run Pipelex methods as durable, horizontally-scaled workflows — crash survival, automatic per-step retries, and operational visibility, delivered through the Pipelex platform."---# Distributed Execution-Pipelex methods run in-process by default — you call `pipelex run pipe ...` or invoke a pipe from Python and everything happens in one process. When you need durability, retries that survive failure, and horizontal scale, you run the same methods as **durable workflows** on Temporal.+Pipelex methods run in-process by default — you call a pipe and everything happens in one process. When you move to production and need work that survives crashes, retries every step under failure, and scales across machines, the **same methods** run as durable workflows. No rewrite: the methods you build locally are the methods that run distributed.-## Pipelex on Temporal+## What it gives you-**[Pipelex on Temporal](temporal/index.md)** runs Pipelex's own Temporal workers against a Temporal cluster you operate (self-hosted or Temporal Cloud) — you own the control plane. Python 3.10+, installed with `pipelex[temporal]`. Generally available.+- **Crash survival** — a long-running pipeline resumes exactly where it left off after a restart or worker crash, instead of starting over.+- **Per-step retries** — every LLM call, image generation, and document extraction retries on its own, with its own timeout and retry policy, so one flaky provider call doesn't sink the whole run.+- **Horizontal scale** — fan work out across many workers, and route different workloads — a provider, a model, OCR — to their own pools so they scale and fail independently.+- **Operational visibility** — every run is durable and observable: see what's running, what each step is doing, and replay history when you need it.-The same `.mthds` methods run distributed without rewriting. Pipelex's runtime bridge classifies controller pipes as child workflows and leaf operators (LLM calls, image generation, document extraction) as activities — so durability, retries, and observability attach at the right granularity.+## How it works-## Where to go next+You don't change your methods. Under the hood, Pipelex maps each method onto a durable execution graph — controller pipes become workflows, and each leaf operation (the actual AI calls) becomes an independently-retried unit of work. Pipelex handles the AI; the orchestration layer handles durability, retries, scheduling, and visibility. Which durable backend runs your methods is a deployment choice, not a code change.-- **[Overview](temporal/index.md)** — what it is, when you'd want it, the big picture, quick start.-- **[Cluster Setup](cluster-setup.md)** — search attributes and `pipelex setup-temporal-namespace`.-- **[Worker Deployment](workers.md)** — `pipelex worker`, scopes, runtime profiles, multi-worker topologies.-- **[Task-Queue Routing](task-routing.md)** — per-activity routing, queue options, per-handle overrides.-- **[Workflow Observability](observability.md)** — workflow ids, summary fields, search-attribute filtering.+## Backends-For the failure-handling model that motivates durable execution, see [Retries & Resilience](../reliability/retries-and-resilience.md).+Distributed execution is delivered through the Pipelex platform, on top of proven orchestration engines:-## Coming soon: a managed backend+- **[Temporal-backed execution](https://pipelex.com/products#temporal)** — run your methods as durable workflows on a Temporal control plane, with per-step retries and full replay history.+- **[Mistral Workflows](https://pipelex.com/products#mistral-workflows)** — run your pipes inside Mistral's managed Workflows orchestration, so there's no control plane for you to operate.-A second distributed-execution backend — running Pipelex pipes inside **Mistral Workflows**, Mistral's managed orchestration control plane (itself built on Temporal) — is in active development. It runs the same `.mthds` methods through the same runtime bridge, with no cluster for you to operate. Documentation and install instructions will land here once it ships.+Both run the identical `.mthds` methods through the same Pipelex runtime, so the durability model and the [error contract](../reliability/failure-classification.md) are the same whichever one executes your work.++## Get started++Distributed and durable execution is part of the Pipelex platform rather than something you wire up yourself. To run your methods durably at scale — or to talk through which backend fits your deployment — see **[Pipelex products](https://pipelex.com/products#durable-execution)**.++For the failure-handling model that durable execution builds on, see [Retries & Resilience](../reliability/retries-and-resilience.md).diff --git a/docs/distributed-execution/mistral-workflows/choosing-a-backend.md b/docs/distributed-execution/mistral-workflows/choosing-a-backend.mddeleted file mode 100644index cb0ad59f7..000000000--- a/docs/distributed-execution/mistral-workflows/choosing-a-backend.md+++ /dev/null@@ -1,41 +0,0 @@-----title: "Choosing a Backend"-description: "Pipelex on Temporal vs Pipelex on Mistral Workflows — both are durable execution on Temporal; pick based on who runs the control plane."------# Choosing a Backend--Both distributed-execution backends are durable execution on Temporal underneath. They give you the same core guarantees: crash survival, activity-level retries, and deterministic replay (on recovery the workflow re-runs from its event history, reusing already-stored activity results rather than re-executing completed activities). The real choice is **which control plane you want to operate**: your own Temporal deployment, or Mistral's managed Workflows service.--## Side by side--| | Pipelex on Temporal | Pipelex on Mistral Workflows |-|---|---|---|-| **Control plane** | You run it (self-hosted Temporal or Temporal Cloud) | Mistral runs it (managed) |-| **Install** | `pipelex[temporal]` | `pipelex-mistralai-workflows` |-| **Python** | 3.10+ | 3.12+ |-| **How you invoke** | Enable `[temporal] is_enabled` in `.pipelex/pipelex.toml` and call `pipe_run(...)` — it dispatches through the Temporal hub | Invoke a Pipelex pipe through the runtime bridge inside your Mistral Workflows activity |-| **Best when** | You already operate Temporal, or want full control over the cluster | You're already building on Mistral's platform |-| **Status** | Generally available | Preview |--## When to pick Temporal--- You already run a Temporal cluster or use Temporal Cloud.-- You want to own worker topology, task-queue routing, and retry policy in detail.-- You need the production operational surface available today: cluster setup, multi-worker scaling, per-activity routing, dashboard observability.--## When to pick Mistral Workflows--- You're already building workflows on Mistral's platform and want pipes to run alongside the rest of your orchestration.-- You'd rather not operate a Temporal cluster yourself.-- You're comfortable with a preview integration while the package stabilizes.--## You can run the same methods on either--Your `.mthds` methods don't change between backends. The choice is purely about where they run and who operates the control plane — so you can start in-process, move to one backend, and switch later without rewriting methods.--## Related--- **[Pipelex on Temporal](../temporal/index.md)** — the self-operated backend.-- **[Pipelex on Mistral Workflows](index.md)** — the managed backend (preview).-- **[Retries & Resilience](../../reliability/retries-and-resilience.md)** — the failure-handling model both backends build on.diff --git a/docs/distributed-execution/mistral-workflows/execution-modes.md b/docs/distributed-execution/mistral-workflows/execution-modes.mddeleted file mode 100644index 6b4479898..000000000--- a/docs/distributed-execution/mistral-workflows/execution-modes.md+++ /dev/null@@ -1,48 +0,0 @@-----title: "Execution Modes"-description: "How a Pipelex pipe runs inside a Mistral Workflows activity — DIRECT, TEMPORAL_BLOCKING, TEMPORAL_FIRE_AND_FORGET, and MISTRAL_NATIVE — and what each requires."------# Execution Modes--When you run a pipe through the bridge, the `execution_mode` field on the input selects *how* that pipe runs. The same activity code can run a pipe in-process, hand it off to your own Temporal workers, or decompose it into native Mistral Workflows primitives.--## The runtime bridge--Pipes are invoked through `run_pipe_via_bridge`, which takes a `PipelexPipeRunInput` and returns a `PipelexPipeRunOutput`. Both are plain JSON-safe models — the boundary never carries Pipelex internal types. The bridge boots Pipelex on first call, validates the input, optionally opens a per-call scoped library from a crate, then dispatches on `execution_mode`.--## The modes--| Mode | What it does | Requires |-|------|--------------|----------|-| `direct` | Runs the pipe in-process, inside the calling activity, blocking until it completes. Fastest feedback, simplest ops. The default. | Pipelex core |-| `temporal_blocking` | Dispatches the pipe as a Pipelex Temporal workflow and awaits completion. The pipe runs durably on your own Temporal worker fleet. | `pipelex[temporal]` |-| `temporal_fire_and_forget` | Dispatches the pipe as a Pipelex Temporal workflow and returns immediately with the `workflow_id`. Completion is signalled out-of-band via a delivery assignment (webhook / storage). | `pipelex[temporal]`, and `delivery_assignment_dump` |-| `mistral_native` | Decomposes the pipe into native Mistral Workflows primitives — controllers as child workflows, leaf operators as activities — surfacing per-step retry, signals, and cancellation through the host runtime. | `pipelex-mistralai-workflows` |--The `Mode` values above are the literal strings you set on the `execution_mode` field — lowercase, matching the `PipelexExecutionMode` enum values. (Mode names written in uppercase elsewhere in this guide refer to the same modes by name.)--For local and most in-activity use, `direct` is the right default: the pipe simply runs where the activity runs, and Mistral Workflows provides the durability and retry around the activity as a whole.--## Input fields that matter--`PipelexPipeRunInput` carries everything the bridge needs:--- `pipe_code` — the pipe to run.-- `inputs` — the pipe's inputs, as a JSON dict.-- `output_name` — optional override for the output variable name.-- `pipeline_run_id`, `user_id` — identity carried into tracing and observability.-- `execution_mode` — one of the modes above (defaults to `direct`).-- `library_crate_dump` — an optional serialized library snapshot. Use it when the pipe isn't already loaded at boot: the bridge opens a per-call scoped library from the crate, runs the pipe, and tears it down afterward. This is the dump-based library transport that lets a worker run methods it didn't load at startup.-- `delivery_assignment_dump` — required for `TEMPORAL_FIRE_AND_FORGET`; describes where to deliver the result (webhook / storage).--## Missing-dependency behavior--The bridge fails loudly and helpfully when a mode's dependency isn't installed:--- Requesting a `TEMPORAL_*` mode without the extra raises `MissingPipelexTemporalExtraError` — *"Install with: pip install 'pipelex[temporal]'"*.-- Requesting `MISTRAL_NATIVE` without the plugin raises `MissingMistralWorkflowsPluginError` — *"Install with: pip install pipelex-mistralai-workflows"*.-- Requesting `TEMPORAL_FIRE_AND_FORGET` without a `delivery_assignment_dump` is rejected up front, so a completion is never silently dropped.--!!! warning "MISTRAL_NATIVE is preview"- The dispatch hook for `MISTRAL_NATIVE` lives in Pipelex core, but the implementation that decomposes a pipe into native Workflows primitives ships in the preview `pipelex-mistralai-workflows` package. Treat end-to-end `MISTRAL_NATIVE` as preview and verify behavior against the installed package. For most integrations, running pipes in `DIRECT` mode inside your own activity already gives you durable, retryable execution from Mistral Workflows.diff --git a/docs/distributed-execution/mistral-workflows/index.md b/docs/distributed-execution/mistral-workflows/index.mddeleted file mode 100644index 259f98c79..000000000--- a/docs/distributed-execution/mistral-workflows/index.md+++ /dev/null@@ -1,38 +0,0 @@-----title: "Pipelex on Mistral Workflows"-description: "Preview: run Pipelex pipes inside Mistral Workflows, Mistral's managed orchestration control plane built on Temporal, via the pipelex-mistralai-workflows package."------# Pipelex on Mistral Workflows--!!! warning "Preview"- The `pipelex-mistralai-workflows` package is in preview and under active development. Install it with `pip install pipelex-mistralai-workflows` (Python 3.12+). APIs may change, and some integration symbols shown in this guide are illustrative — they are flagged where they appear. Pipelex itself supports Python 3.10+; only this orchestration path needs 3.12.--> Mistral Workflows is an orchestration control plane designed to accelerate the development and reliable execution of complex, AI-driven workflows. Built on Temporal for fault-tolerant workflow execution, Workflows combines a user-friendly API with a rich Python framework optimized for Mistral's AI services.--This guide shows how to run your Pipelex `.mthds` methods *inside* Mistral Workflows — so a single Mistral Workflows worker can orchestrate Pipelex pipes alongside the rest of your AI workflow, with durability, retries, and observability handled by the platform.--## How it fits together--You don't rewrite your methods. The same pipes that run in-process locally run unchanged here:--- You write a Mistral Workflows worker (workflows and activities) the usual way.-- Inside an activity, you invoke a Pipelex pipe through Pipelex's runtime bridge.-- Mistral Workflows — on its Temporal foundation — gives durability, automatic retries, streaming progress, and observability; Pipelex runs the AI method.--!!! info "Both backends are Temporal underneath"- Pipelex on Mistral Workflows and [Pipelex on Temporal](../temporal/index.md) are the same durability primitive packaged two ways. The difference is *who runs the control plane*: here it's Mistral's managed Workflows service; there it's a Temporal deployment you operate yourself. See [Choosing a Backend](choosing-a-backend.md).--## What you get--- **Durable execution** — pipe runs survive worker restarts and transient failures, resuming from history.-- **Automatic retries** — what retries depends on the execution mode: in `direct` mode the whole pipe runs inside one host activity that retries as a unit; in `mistral_native` mode each leaf operator (LLM call, extraction, image generation) runs as its own host activity and retries independently. See [Execution Modes](execution-modes.md).-- **Streaming progress** — surface live pipe progress through the Workflows Task event API. See [Streaming Progress](streaming.md).-- **The same methods, unchanged** — your `.mthds` bundles don't change between local, Temporal, and Mistral Workflows execution.--## Next steps--- **[Installation & Preview Status](installation.md)** — install the preview package and its prerequisites.-- **[Your First Pipelex Workflow](your-first-pipelex-workflow.md)** — register a Pipelex activity and run a pipe end to end.-- **[Execution Modes](execution-modes.md)** — how a pipe runs inside an activity, and what each mode requires.-- **[Choosing a Backend](choosing-a-backend.md)** — Mistral Workflows vs your own Temporal cluster.diff --git a/docs/distributed-execution/mistral-workflows/installation.md b/docs/distributed-execution/mistral-workflows/installation.mddeleted file mode 100644index daa61aa6b..000000000--- a/docs/distributed-execution/mistral-workflows/installation.md+++ /dev/null@@ -1,50 +0,0 @@-----title: "Installation & Preview Status"-description: "Install the preview pipelex-mistralai-workflows package (Python 3.12+) and understand what 'preview' means for the Mistral Workflows backend."------# Installation & Preview Status--!!! warning "Preview"- `pipelex-mistralai-workflows` is not yet a stable release. Its public surface may change between versions, and there is no separate published documentation site for it yet — this guide is the reference for now.--## Prerequisites--Before installing, make sure you have:--- **Python 3.12 or later.** Mistral Workflows requires 3.12+. Pipelex core supports 3.10+, but this orchestration path needs 3.12 because of the Workflows SDK.-- **A Mistral Workflows setup** — a workspace and API key from the [Mistral Console](https://console.mistral.ai/), and the `mistralai-workflows` SDK. Follow Mistral's own Workflows installation and first-workflow guides for the worker, console, and `.env` setup; this guide does not duplicate them.-- **A Pipelex method to run** — a `.mthds` bundle available on your `PIPELEXPATH`, or passed at call time as a library crate (see [Execution Modes](execution-modes.md)).--## Install--```bash-pip install pipelex-mistralai-workflows-```--This pulls in `pipelex` automatically. If your project uses `uv`, the equivalent is `uv add pipelex-mistralai-workflows`.--The package depends on the Mistral Workflows SDK (`mistralai-workflows`). If you are scaffolding a new Workflows project, install the SDK as Mistral documents it (`uv add mistralai-workflows`) and add `pipelex-mistralai-workflows` alongside it.--## Verify--Confirm both pieces import in your environment:--```bash-python -c "import mistralai.workflows; import pipelex_mistralai_workflows; print('ready')"-```--<!-- ILLUSTRATIVE: the importable top-level module name `pipelex_mistralai_workflows` matches the package's Python distribution name; confirm the exact public import path against the released package. -->--!!! note "Preview verify"- The exact public import surface of `pipelex_mistralai_workflows` is still settling. If the import above fails after a successful install, check the package's release notes for the current module layout.--## What "preview" means--- The API may change between releases without a deprecation window.-- Some examples in this guide are marked illustrative — confirm them against the installed package before relying on them in production.-- Watch the changelog and release notes for breaking changes.--## Next--Continue to **[Your First Pipelex Workflow](your-first-pipelex-workflow.md)**.diff --git a/docs/distributed-execution/mistral-workflows/streaming.md b/docs/distributed-execution/mistral-workflows/streaming.mddeleted file mode 100644index 3cfee5cfa..000000000--- a/docs/distributed-execution/mistral-workflows/streaming.md+++ /dev/null@@ -1,41 +0,0 @@-----title: "Streaming Progress"-description: "Surface live Pipelex pipe progress through Mistral Workflows' Task event API — one event per activity, or per-step updates in DIRECT mode."------# Streaming Progress--!!! warning "Preview"- Streaming integration ships in the preview `pipelex-mistralai-workflows` package. The Mistral Workflows streaming primitives shown here are stable; the Pipelex-specific entry points are flagged inline.--Mistral Workflows publishes real-time events from activities through its **Task** API, which powers live UIs and progress indicators. A Pipelex activity wraps a pipe run in a Task and emits progress as the pipe executes.--## How streaming works--Inside an activity you open a `Task` and push state updates as work proceeds:--```python-from mistralai.workflows.core.task import Task--async with Task(type="pipelex_progress", state={"completed_steps": []}) as task:- # ... run the pipe, updating task.state as steps complete ...- await task.update_state({"completed_steps": task.state["completed_steps"] + ["answer_question"]})-```--Mistral Workflows wraps each update with workflow/activity context and publishes it through its event streaming system, which consumers subscribe to via the Workflows client's event-streaming APIs.--## Two granularities--The Pipelex integration surfaces progress at one of two levels:--- **Per-activity** — a single Task per pipe run: started, in-progress, completed (or failed). This is the cheapest option and works regardless of execution mode.-- **Per-step (`DIRECT` mode)** — driven by Pipelex's own execution-trace events, the activity emits a Task update for each pipe step as it starts and finishes, exposing started-steps, completed-steps, and the current pipe code. Because it rides on in-process trace events, per-step granularity applies to `DIRECT` execution; Temporal modes keep the single started/finished pair.--<!-- ILLUSTRATIVE: the package's streaming entry-point names and the exact per-step task `type`/state schema are still settling in preview; the per-activity vs per-step distinction is the design-locked behavior. Confirm symbols against the installed package. -->--## Consuming events--Subscribe to the published events with the Mistral Workflows client's event-streaming APIs — see Mistral's "Consuming Streaming Events" guide. This guide doesn't duplicate the consumer side; the events a Pipelex activity emits are ordinary Workflows Task events.--!!! note "Payload limit"- NATS messages are capped at **1 MB**. Stream small progress state — step names, counts, a current pipe code — not large pipe outputs. For large results, store them externally (or use the bridge's delivery assignment) and stream a reference.diff --git a/docs/distributed-execution/mistral-workflows/your-first-pipelex-workflow.md b/docs/distributed-execution/mistral-workflows/your-first-pipelex-workflow.mddeleted file mode 100644index 96226116e..000000000--- a/docs/distributed-execution/mistral-workflows/your-first-pipelex-workflow.md+++ /dev/null@@ -1,95 +0,0 @@-----title: "Your First Pipelex Workflow on Mistral Workflows"-description: "Register a Pipelex activity on a Mistral Workflows worker, call it from a workflow, and run it — the preview end-to-end path."------# Your First Pipelex Workflow--!!! warning "Preview"- This walkthrough targets the preview `pipelex-mistralai-workflows` package. The Pipelex bridge calls shown here (`run_pipe_via_bridge`, `PipelexPipeRunInput`) are part of Pipelex core and stable; anything specific to the plugin package is flagged inline.--We'll register an activity that runs a Pipelex pipe, call it from a Mistral Workflow, and execute the whole thing on a worker.--## Prerequisites--- The preview package installed — see [Installation & Preview Status](installation.md).-- A working Mistral Workflows setup (workspace, API key, `.env`) per Mistral's own first-workflow guide.-- A Pipelex method available on your `PIPELEXPATH` (or passed as a library crate — see [Execution Modes](execution-modes.md)). The example below assumes a pipe named `answer_question`.--## Step 1 — Define a worker that runs a Pipelex pipe--Pipelex exposes a small, JSON-in / JSON-out entry point — `run_pipe_via_bridge(PipelexPipeRunInput(...))` — designed to be called from inside any host-runtime activity. The cleanest first integration is to write your own activity that calls it. Every symbol here is part of Pipelex core.--Create `my_pipelex_worker.py`:--```python-import mistralai.workflows as workflows--from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge---async def run_pipelex_pipe(pipe_code: str, inputs: dict) -> dict:- """Run a Pipelex pipe inside a Mistral Workflows activity.-- The bridge boots Pipelex on first call (idempotent), runs the pipe, and- returns a JSON-safe result. Only JSON crosses the activity boundary — no- Pipelex internal objects.- """- output = await run_pipe_via_bridge(- PipelexPipeRunInput(pipe_code=pipe_code, inputs=inputs)- )- return output.model_dump()--[email protected](name="pipelex_example_workflow")-class PipelexExampleWorkflow:- @workflows.workflow.entrypoint- async def run(self, pipe_code: str, inputs: dict) -> dict:- return await run_pipelex_pipe(pipe_code, inputs)---async def main() -> None:- await workflows.run_worker([PipelexExampleWorkflow])---if __name__ == "__main__":- import asyncio-- asyncio.run(main())-```--!!! note "Illustrative: ready-made activity"- The plugin package may also ship a ready-to-register Pipelex activity so you can add it to a worker without writing the wrapper yourself. Its exact name and registration signature are still settling in preview — confirm them against the installed package. Writing your own `@workflows.activity()` around `run_pipe_via_bridge`, as above, works today and is the safest starting point.--## Step 2 — Run the worker--```bash-uv run python my_pipelex_worker.py-```--The worker connects to the Mistral API and registers `pipelex_example_workflow`. You should see it in the [Mistral Console](https://console.mistral.ai/build/workflows).--## Step 3 — Trigger execution--From the Console, select `pipelex_example_workflow`, click **Start Workflow**, and pass input such as:--```json-{ "pipe_code": "answer_question", "inputs": { "question": "What is MTHDS?" } }-```--You can also start it programmatically with the Mistral SDK via `client.workflows.*`.--<!-- ILLUSTRATIVE: the exact client.workflows.* call shape is generic Mistral Workflows usage, not Pipelex-specific. Confirm against the Mistral SDK docs. -->--## What just happened--- The activity called `run_pipe_via_bridge`, which **booted Pipelex on first call** (idempotent — subsequent activities reuse the running instance).-- With no `execution_mode` set, the pipe ran in **`direct`** mode: in-process inside the activity, blocking until it completed.-- The bridge returned a `PipelexPipeRunOutput` — `output_dict`, `main_stuff_name`, `pipeline_run_id`, `is_completed`, and more — which the activity serialized with `model_dump()`.-- The boundary is **JSON-only**: no Pipelex internal types (`PipeJob`, `WorkingMemory`, `PipeOutput`) cross the activity edge, which is what keeps the pipe replayable and the activity serializable.--## Next--- **[Execution Modes](execution-modes.md)** — run the pipe in-process, delegate to your own Temporal workers, or fire-and-forget.-- **[Streaming Progress](streaming.md)** — surface live progress as the pipe runs.diff --git a/docs/distributed-execution/observability.md b/docs/distributed-execution/observability.mddeleted file mode 100644index ca3f13df6..000000000--- a/docs/distributed-execution/observability.md+++ /dev/null@@ -1,144 +0,0 @@-----title: "Workflow Observability"-description: "What Pipelex shows in the Temporal Web UI — workflow ids, activity ids, static summaries, per-activity summaries, and custom search-attribute filters."------# Workflow Observability--Every Pipelex workflow carries human-readable identity into the Temporal Web UI: a workflow id derived from your pipeline run id, a Markdown summary of the pipe and its inputs, per-activity summary lines that explain what each LLM or extract call is doing, and a set of custom search attributes for filtering. This page is the reference for everything you'll see in the dashboard.--For the cluster-side registration that enables search attributes, see [Cluster Setup](cluster-setup.md).-------## Why this matters--Operators triage incidents from the Temporal dashboard. They sort by start time, filter by user or session, drill into a specific workflow, and read the activity history to see where things went wrong. Opaque ids and bytes-style summaries make that work fast or slow. Pipelex chooses verbosity at the dashboard boundary so the operator never has to leave it to identify a workflow.-------## Workflow IDs--### Top-level workflows--Pipelex derives the workflow id from `JobMetadata.pipeline_run_id` with a short prefix that identifies the run environment:--```-{env_prefix}{pipeline_run_id}-```--Example: `ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c`.--Environment prefixes:--| Prefix | Run mode | When |-|--------|---------------------|-----------------------------------|-| `ut-` | `UNIT_TEST` | Unit test suites |-| `ci-` | `CI_TEST` | GitHub Actions / CI |-| `cc-` | `CODEX_CLOUD` | Codex Cloud agent runs |-| `cct-` | `CODEX_CLOUD_TEST` | Codex Cloud test runs |-| `` (none) | `NORMAL` | Production / local development |--Because the workflow id is derived from `pipeline_run_id`, a caller that controls that id (via `PipelineFactory.make_pipeline(pipeline_run_id=...)`) can replay onto the same Temporal **Workflow Execution Chain**. Each replay produces a fresh `run_id` under the SDK-default `ALLOW_DUPLICATE` reuse policy. This is documented behavior — the previous releases produced fresh workflow ids per execution by accident, via truncated session-id and random-shortuuid components, not by design.--### Child workflows--Pipelex builds child workflow ids off the parent's workflow id, joined with an underscore:--```-{parent_workflow_id}_pipe-router # the fixed WfPipeRouter child of WfPipeRun-{parent_workflow_id}_{pipe_code}-{8-hex} # a sub-pipe spawned by a router-```--The 8-hex suffix comes from `workflow.uuid4()` — Temporal's replay-safe UUID generator — so child workflow ids stay deterministic across worker restarts. The separator is an underscore, never a slash, so a workflow id can be reused verbatim as an S3 key or file name without creating spurious path segments.--Operational tooling that parses the nested-id format must handle `_` as the separator.-------## Activity IDs--Pipelex does not customize activity ids. The Temporal Python SDK auto-assigns deterministic sequential integers (`"1"`, `"2"`, …) per workflow run. This guarantees per-`(workflow_id, run_id)` uniqueness by construction and is replay-safe — the SDK assigns ids by history position.--Per-call semantic meaning lives in the per-activity `summary=` field instead. Examples:--```-LLM text · pipe=translate_doc · model=openai-gpt-4o-LLM object · pipe=invoice_extractor · class=Invoice-LLM object list · pipe=extract_line_items · class=LineItem-Img gen N× · pipe=cover · model=fal-flux-dev · n=3-Render page views · pipe=extract_doc-Extract pages · pipe=extract_doc · handle=azure-document-intelligence-extract-Templated text · pipe=summary-```--Anything that previously filtered or grouped Event History by the old semantic activity-id strings (`"craft-text"`, `"craft-object-direct"`, `"jinja2-text"`, `"extract-pages"`, etc.) must now read the per-activity `summary` field or the Activity Type.-------## Static summary and details--Every top-level workflow start sets two fields the Temporal dashboard renders as Markdown:--### `static_summary`--A 200-byte one-liner shown in the workflow list view. Shape:--```-{pipe_code} — {description}-```--Example: `translate_doc — Translate a document from English to French`. The description is the optional `description` field on the pipe; when empty, only the pipe code is shown.--### `static_details`--A 20 KB Markdown table shown in the workflow detail view. Always includes:--| Field | Value |-|--------------|------------------------------------|-| Pipe | `translate_doc` |-| Domain | `documents` |-| Pipeline run | `3f9c8b2a-...` |-| User | `user-42` |-| Session | `session-abc` |--Two optional rows are appended when the data is available:--- **Library crate** — the first 12 characters of the bundled `LibraryCrate.fingerprint`, when the workflow's `PipeJob` carries a crate.-- **Input** — comma-separated input stuff names, when the pipe declares inputs.--The formatting policy for both fields lives in `pipelex/temporal/tprl/observability.py`. Length caps follow Temporal's documented limits.-------## Filtering and grouping with search attributes--Every workflow start attaches up to five custom `Keyword` search attributes:--| Attribute | Value |-|-----------------|--------------------------------------------------------------------|-| `PipeCode` | The pipe being executed |-| `PipelineRunId` | The pipeline run id (= the workflow id, minus env prefix) |-| `SessionId` | The submitter session id (stamped at the top-level dispatch) |-| `UserId` | The user id from `JobMetadata` |-| `DomainCode` | The pipe's domain |--The set is configurable — only the attributes listed in `[temporal.search_attributes].attributes` are attached, and setting `enabled = false` skips them entirely. See [Cluster Setup](cluster-setup.md) for the registration steps the cluster needs before these can be used.--In the Temporal Web UI, use the **List Workflows** search bar with queries like:--```-PipeCode = "translate_doc"-UserId = "user-42" AND DomainCode = "documents"-PipelineRunId = "3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c"-```--Child workflows inherit `PipelineRunId`, `UserId`, and `SessionId` from their parent's `PipeJob`; `PipeCode` and `DomainCode` reflect the child's own pipe. Reading every field off the `PipeJob` (the workflow input) keeps the attribute build deterministic across replays.-------## What's next--- **[Cluster Setup](cluster-setup.md)** — register the five search attributes before workers can dispatch workflows that use them.-- **[Worker Deployment](workers.md)** — run the workers that produce these events.-- **[Task-Queue Routing](task-routing.md)** — control which worker pool each activity lands on; the dashboard shows the resolved queue per call.diff --git a/docs/distributed-execution/task-routing.md b/docs/distributed-execution/task-routing.mddeleted file mode 100644index fd312c154..000000000--- a/docs/distributed-execution/task-routing.md+++ /dev/null@@ -1,176 +0,0 @@-----title: "Task-Queue Routing"-description: "Route Pipelex activities to specific Temporal task queues with per-activity, per-handle precision. Layer per-queue and per-handle timeout, retry, and rate overrides on top."------# Task-Queue Routing--When one task queue isn't enough, Pipelex routes each activity to a specific queue based on the activity name and (optionally) the runtime handle — the model id, image-gen handle, or extract handle the activity is about to call. On top of routing, each queue and each handle can carry its own timeouts, retry policies, and rate caps.--This page is the reference for the three composable layers: per-activity routing, per-queue overlays, per-handle overlays. If you're running a single worker on a single queue, you can skip this page entirely.-------## Why you'd route--Three motivations cover almost every real-world case:--- **Provider isolation** — OpenAI and Anthropic have independent rate limits, retry semantics, and outage windows. Routing each provider's LLM calls to its own pool means one provider's incident doesn't drain the other's throughput.-- **Activity-class isolation** — A long-running OCR call shouldn't share a worker pool with thousands of short LLM calls. Routing extract activities to a dedicated `extract_q` keeps slow work from starving the rest.-- **Handle-specific tuning** — One specific model (say, a long-context Claude variant with 60-second timeouts) needs a different retry and timeout policy than the rest of the Anthropic pool. Per-handle overlays let you tune one model without forking the whole queue.-------## Per-activity, per-handle routing--Routing lives under `[temporal.worker_config.activity_queues.<activity_name>]`. Each entry declares a `default` queue and an optional `by_handle` map keyed by the activity's routing key:--```toml-[temporal.worker_config.activity_queues.act_llm_gen_text]-default = "openai_q"-by_handle.anthropic-default = "anthropic_q"-by_handle.gemini-pro = "gemini_q"--[temporal.worker_config.activity_queues.act_llm_gen_object]-default = "openai_q"-by_handle.anthropic-default = "anthropic_q"--[temporal.worker_config.activity_queues.act_img_gen_images]-default = "imggen_q"--[temporal.worker_config.activity_queues.act_extract_gen_extract_pages]-default = "extract_q"-```--The **routing key** is the runtime handle the activity is about to call:--| Activity | Routing key | Source |-|-----------------------------------|------------------------|---------------------------------|-| `act_llm_gen_text` | LLM model handle | The `llm_handle` on the call |-| `act_llm_gen_object` | LLM model handle | The `llm_handle` on the call |-| `act_llm_gen_object_list` | LLM model handle | The `llm_handle` on the call |-| `act_img_gen_images` | `img_gen_handle` | The image generation backend |-| `act_extract_gen_extract_pages` | `extract_handle` | The OCR / extract backend |-| `act_search_gen_sourced_answer` | `search_handle` | The web search backend |-| `act_search_gen_structured` | `search_handle` | The web search backend |-| `act_jinja2_gen_text` | (none — pass `None`) | Template rendering has no handle |-| `act_render_page_views` | (none — pass `None`) | Page-view rendering has no handle |--`WorkerConfig.resolve_queue(activity_name, routing_key)` walks three layers, returning the first match:--1. `worker_config.activity_queues[activity_name].by_handle[routing_key]`-2. `worker_config.activity_queues[activity_name].default`-3. `default_task_queue` (worker-wide fallback)--When `activity_queues` is completely empty (the shipping default), `resolve_queue` returns `None` and dispatch omits the `task_queue` kwarg — every activity rides the workflow's own queue, preserving single-queue behavior for installs that haven't opted into routing.--!!! note "Every routed queue needs a `queue_options` entry"-- Any queue named under `[temporal.worker_config.activity_queues.*]` must have a matching `[temporal.queue_options.<q>]` entry (an empty stanza is fine and means "use worker-config baselines for this queue"). Routing to an undeclared queue is treated as a typo and fails at config-load with `TemporalConfigError`. `default_task_queue` is the one implicit exception — it never needs its own `queue_options` entry.-------## Per-queue option overlays--Each queue can carry its own timeouts, retry policy, and cluster-wide rate cap under `[temporal.queue_options.<queue>]`:--```toml-[temporal.queue_options.openai_q]-start_to_close_timeout = "0:05:00"-heartbeat_timeout = "0:01:00"-max_task_queue_activities_per_second = 500--[temporal.queue_options.openai_q.retry_policy_config]-initial_interval = "0:00:01"-maximum_attempts = 4-non_retryable_error_types_extra = ["OpenAIBadRequestError"]--[temporal.queue_options.anthropic_q]-start_to_close_timeout = "0:10:00"-max_task_queue_activities_per_second = 50--[temporal.queue_options.extract_q]-start_to_close_timeout = "0:30:00"-heartbeat_timeout = "0:05:00"--# Empty stanzas for queues routed in activity_queues that don't need any-# tuning — required by the orphan-queue validator at config-load.-[temporal.queue_options.gemini_q]-[temporal.queue_options.imggen_q]-```--All fields are optional — anything you leave out falls through to the worker-config baseline:--| Field | Type | When unset |-|---------------------------------------------|-------------|---------------------------------------------|-| `start_to_close_timeout` | duration | Use `default_activity_start_to_close_timeout` |-| `schedule_to_close_timeout` | duration | Omitted; SDK applies its default |-| `schedule_to_start_timeout` | duration | Omitted; SDK applies its default |-| `heartbeat_timeout` | duration | Use `default_activity_heartbeat_timeout` |-| `retry_policy_config` | overlay | Use worker-config `retry_policy_config` |-| `max_task_queue_activities_per_second` | float | No cluster-wide cap on this queue |--`max_task_queue_activities_per_second` is the **cluster-wide** rate cap on the queue — every worker on the queue sends this value to the Temporal server; the latest writer wins. Before this config knob existed, Pipelex hard-coded `1000` on every worker. The shipped default keeps the cap at `1000` for the default queue:--```toml-[temporal.queue_options.temporal_task_queue]-max_task_queue_activities_per_second = 1000-```--Deployments using non-default queue names should add their own `[temporal.queue_options.<queue>]` entries with the cap appropriate for that backend pool.-------## Per-handle option overlays--For the rare case where one model variant needs different timeouts or retry from its queue baseline, declare a `handle_options.<handle>` entry on the activity route:--```toml-[temporal.worker_config.activity_queues.act_llm_gen_text]-default = "openai_q"-by_handle.anthropic-default = "anthropic_q"-by_handle.anthropic-claude-3-5-sonnet-long-context = "anthropic_q"--[temporal.worker_config.activity_queues.act_llm_gen_text.handle_options.anthropic-claude-3-5-sonnet-long-context]-start_to_close_timeout = "0:30:00"--[temporal.worker_config.activity_queues.act_llm_gen_text.handle_options.anthropic-claude-3-5-sonnet-long-context.retry_policy_config]-maximum_attempts = 2-non_retryable_error_types_extra = ["AnthropicLongContextOverflow"]-```--Handle overlays deliberately cover only timeout and retry. Heartbeat cadence is a property of the backend, not the model, so it lives on the queue overlay. Add a field to `HandleOptions` only when a real case requires it.--### Resolution order--At dispatch time, `WorkerConfig.resolve_dispatch(activity_name, routing_key)` composes the three layers in last-wins order for each scalar:--1. **Baseline** — worker-config defaults (`default_activity_start_to_close_timeout`, `default_activity_heartbeat_timeout`, `retry_policy_config`).-2. **Queue overlay** — `queue_options[resolved_queue]` if it exists.-3. **Handle overlay** — `worker_config.activity_queues[activity_name].handle_options[routing_key]` if it exists.--Each layer can replace a scalar (timeout, retry interval, max attempts) or leave it untouched. The result is splatted directly into `workflow.execute_activity(...)`.--`non_retryable_error_types` is the one exception to last-wins: it **composes additively** across layers. The baseline declares a main list; each overlay can declare `non_retryable_error_types_extra` to add more entries. Removing entries via overlay is deliberately not supported — per-queue and per-handle layers can only add to the no-retry list. This is a safety lean: it's easier to say "never retry this" than to undo it by accident at a narrower scope.--The `RetryPolicyConfig` schema is split into a baseline class (owns `non_retryable_error_types`) and an overlay class (owns `non_retryable_error_types_extra`). `extra = "forbid"` rejects the wrong field at each layer so the additive rule is enforced at config load.-------## Tracing the resolution--When timeouts or retries behave unexpectedly, turn on dispatch resolution tracing:--```toml-[temporal.temporal_config.temporal_log_config]-is_dispatch_resolution_traced = true-```--Every `workflow.execute_activity` call then emits one INFO log line with the resolved queue, the resolved `start_to_close_timeout`, the resolved retry attempts, and — most useful — **which layer each scalar came from** (`baseline`, `queue_options`, or `handle_options`). This is verbose, so leave it off by default; turn it on when you're hunting a misconfigured overlay.-------## What's next--- **[Worker Deployment](workers.md)** — pair routing with `--scope` so each worker pool only registers the activities it should receive.-- **[Workflow Observability](observability.md)** — see what the dashboard shows for each routed activity.diff --git a/docs/distributed-execution/temporal/index.md b/docs/distributed-execution/temporal/index.mddeleted file mode 100644index 049a3ee4a..000000000--- a/docs/distributed-execution/temporal/index.md+++ /dev/null@@ -1,100 +0,0 @@-----title: "Pipelex on Temporal"-description: "Run Pipelex pipelines as durable Temporal workflows on your own Temporal cluster or Temporal Cloud. Cluster setup, worker deployment, task-queue routing, and dashboard observability."------# Pipelex on Temporal--Run your Pipelex pipelines as durable Temporal workflows across one or more worker processes. This section walks through setting up the cluster, running workers, routing work to the right pools, and reading the Temporal Web UI.--This is the Temporal backend for distributed execution: you run **Pipelex's own Temporal workers** against a Temporal cluster you operate (self-hosted or Temporal Cloud) — you own the control plane. A second, managed alternative (Pipelex on Mistral Workflows) is coming soon. See the [Distributed Execution overview](../index.md) for the big picture.--## What it is--Pipelex pipelines normally run in-process — you call `pipelex run pipe ...` or invoke a pipe from Python and everything happens in one Python process. With the optional `pipelex[temporal]` integration, the same pipelines run as **Temporal workflows**: each pipe execution becomes a workflow on a Temporal cluster, child pipes become child workflows, and every LLM call, image generation, or document extraction becomes a Temporal activity executed on a worker. Temporal handles durability, retries, scheduling, and visibility; Pipelex handles the AI work.--The whole layer is opt-in. Flip `[temporal] is_enabled = true` in `.pipelex/pipelex.toml`, install `pipelex[temporal]`, and the same `.mthds` methods run distributed without changing a line.--## When you'd want it--You don't need Temporal for local development — pipes run in-process by default. Reach for distributed execution when you need:--- **Durability** — long-running pipelines survive process restarts and worker crashes. Temporal replays history to resume exactly where execution left off.-- **Retries with budget control** — every activity (LLM call, extract, image gen) retries independently with per-activity timeouts and retry policies.-- **Horizontal scale** — fan out work across multiple worker machines. Pipelex lets you route OpenAI calls to one pool, Anthropic to another, OCR to a third.-- **Operational visibility** — every workflow shows up in the Temporal Web UI with the pipe code, pipeline run id, user, session, and a per-activity summary line.--## The big picture--```-Your application- │- │ pipe_run(...) ──→ starts a Temporal workflow- ▼-┌────────────────────┐-│ Temporal cluster │ durability, history, retries, search-└─────────┬──────────┘- │ tasks distributed by queue- ▼- ┌─────────────┐ ┌─────────────┐ ┌─────────────┐- │ worker │ │ worker │ │ worker │ poll task queues,- │ (router) │ │ (LLM pool) │ │ (img-gen) │ execute workflows- └─────────────┘ └─────────────┘ └─────────────┘ and activities-```--A submitter (your app or `pipelex run`) starts a workflow; one or more worker processes poll task queues for work; results flow back through the workflow's history. You can run a single worker that handles everything, or split workers by activity class for independent scaling and isolation.--## What you'll set up--Three things, in this order:--- **[Cluster Setup](../cluster-setup.md)** — one-time namespace prerequisites: register the custom search attributes Pipelex needs for the dashboard view. Run `pipelex setup-temporal-namespace` and you're done.-- **[Worker Deployment](../workers.md)** — how to run `pipelex worker`, what `--scope` and `--profile` do, and how to compose multiple workers into a deployment.-- **[Task-Queue Routing](../task-routing.md)** — when you outgrow a single queue: per-activity routing, per-queue timeouts and retry policies, per-handle overrides. Skip on day one; revisit when you need to isolate workloads.--## What you'll see in the Temporal dashboard--Every workflow Pipelex starts carries human-readable identity: a workflow id derived from your pipeline run id, a Markdown summary of the pipe and its inputs, and per-activity summary lines that explain what each LLM or extract call is doing. Filter by `PipeCode`, `PipelineRunId`, `UserId`, `SessionId`, or `DomainCode` from the dashboard's search attribute selector. See **[Workflow Observability](../observability.md)** for the full reference.--## Quick start--Minimal configuration in `.pipelex/pipelex.toml`:--```toml-[temporal]-is_enabled = true--[temporal.worker_config]-default_task_queue = "temporal_task_queue"--[temporal.temporal_config]-selected_server = "local"--[temporal.temporal_config.temporal_server_configs.local]-target_host = "localhost:7233"-namespace = "default"-api_key_method = "none"-```--Register the custom search attributes once (assumes Temporal is reachable):--```bash-pipelex setup-temporal-namespace-```--Boot a worker against the default task queue:--```bash-pipelex worker-```--Submit work from your application — the same `pipe_run(...)` you'd use locally now executes as a Temporal workflow because `[temporal] is_enabled = true`. The Pipelex Python API automatically dispatches through the Temporal hub.--## Where to go next--- **[Cluster Setup](../cluster-setup.md)** — search attributes, `pipelex setup-temporal-namespace`, Temporal Cloud permission model.-- **[Worker Deployment](../workers.md)** — `pipelex worker`, scopes, runtime profiles, multi-worker topologies.-- **[Task-Queue Routing](../task-routing.md)** — `activity_queues`, queue options, per-handle overrides, dispatch tracing.-- **[Workflow Observability](../observability.md)** — workflow ids, activity ids, summary fields, search-attribute filtering.--For the runtime mechanics that make distributed execution work under the hood — LibraryCrate propagation, deferred hydration, per-workflow class registry isolation — see [Temporal Integration](../../under-the-hood/temporal-integration.md) and [Distributed Content Generation](../../under-the-hood/distributed-content-generation.md).diff --git a/docs/distributed-execution/workers.md b/docs/distributed-execution/workers.mddeleted file mode 100644index f14f0a4c9..000000000--- a/docs/distributed-execution/workers.md+++ /dev/null@@ -1,190 +0,0 @@-----title: "Worker Deployment"-description: "Run and tune Pipelex Temporal worker processes — the `pipelex worker` CLI, default task queue, worker scopes, and named runtime profiles."------# Worker Deployment--A worker process polls one Temporal task queue for workflows and activities, executes them, and reports results. This page covers how to run a worker, how to scope it to a subset of work, and how to tune its concurrency.--For the cluster-side prerequisites that must be in place before a worker can boot, see [Cluster Setup](cluster-setup.md).-------## Running a worker--The simplest invocation reads everything from `pipelex.toml`:--```bash-pipelex worker-```--This boots a worker that polls `[temporal.worker_config].default_task_queue`, uses the `[temporal.worker_scopes].default_scope` worker scope, and applies the `[temporal.worker_runtime_profiles].default_profile` runtime profile. For a single-machine deployment that's all you need.--Available flags:--| Flag | Purpose |-|------------------|----------------------------------------------------------------------------------|-| `--task-queue` | Override the polled queue. Must be a known queue (see strict validation below). |-| `--scope` | Pick a worker scope from `[temporal.worker_scopes.scopes]`. |-| `--profile` | Pick a runtime profile from `[temporal.worker_runtime_profiles.profiles]`. |-| `--no-sandbox` | Run workflows without Temporal's sandbox restrictions. For tests / debugging. |-| `--unit-testing` | Run in unit-testing mode (sets `RunMode.UNIT_TEST`). |--Example:--```bash-pipelex worker --task-queue anthropic_q --scope runner-llm --profile anthropic-tier4-```-------## The default task queue--```toml-[temporal.worker_config]-default_task_queue = "temporal_task_queue"-```--`default_task_queue` is the queue a worker polls when no `--task-queue` flag is passed. It's also the fallback used when [Task-Queue Routing](task-routing.md) is configured but a given activity has no specific routing entry.--The key was previously named `task_queue`; the migration map in `pipelex.toml` rewrites legacy configs on load, so existing deployments keep working without manual edits.-------## Strict `--task-queue` validation--`pipelex worker --task-queue <q>` fast-fails with `WorkerTaskQueueUnknownError` when `<q>` is not declared anywhere in the Temporal config — not in `default_task_queue`, not in any `[temporal.activity_queues.<activity>]` entry, and not in any `[temporal.queue_options.<queue>]` overlay. The error message includes a Levenshtein "Did you mean?" suggestion when a close match exists.--This catches operator typos in seconds rather than after a multi-second library boot. Programmatic callers (tests, library code) get the same check inside `TemporalTaskManager.run_worker`.-------## Worker scopes--A worker scope picks which workflows and activities a worker process registers with Temporal. Scopes are declared under `[temporal.worker_scopes.scopes.<name>]` and selected via `--scope <name>`.--Built-in scopes:--| Scope | Workflows | Activities | Use case |-|-------------------|-----------|-----------------------------------------------------|---------------------------------------------------------------------------|-| `full` | All | All | Single-worker deployment. Default. |-| `router` | All | None | Workflow-only worker. Pair with `runner` to force cross-process dispatch. |-| `runner` | None | All | Activity-only worker. Counterpart to `router`. |-| `runner-llm` | None | `act_llm_gen_text`, `act_llm_gen_object`, `act_llm_gen_object_list` | One pool dedicated to LLM calls. |-| `runner-img-gen` | None | `act_img_gen_images`, `act_render_page_views` | One pool dedicated to image generation. |-| `runner-extract` | None | `act_extract_gen_extract_pages`, `act_render_page_views` | One pool dedicated to OCR / document extraction. |-| `runner-jinja2` | None | `act_jinja2_gen_text` | One pool dedicated to template rendering (rare; usually fine in-band). |-| `runner-search` | None | `act_search_gen_sourced_answer`, `act_search_gen_structured` | One pool dedicated to web search calls. |--`act_render_page_views` is intentionally registered under **both** `runner-img-gen` and `runner-extract` so neither pool breaks when the activity is routed to either. The `runner-llm` scope splits all three LLM activity types (text, object, object-list) onto the same pool — they share the same provider connections so splitting further would only add operational overhead.--The `full` scope is the right default for almost every deployment. Reach for `router`/`runner` splits when you need to enforce cross-worker execution boundaries (durability under partial failure, deterministic activity placement) or when one activity class has very different resource needs from the rest.-------## Worker runtime profiles--A runtime profile bundles the worker's concurrency knobs: how many workflow tasks it polls in parallel, how many activities it runs concurrently, how aggressively it heartbeats, and how long it waits during graceful shutdown.--The default profile is shipped in `pipelex.toml`:--```toml-[temporal.worker_runtime_profiles]-default_profile = "default"--[temporal.worker_runtime_profiles.profiles.default]-tuning_mode = "explicit"-max_cached_workflows = 10000-max_concurrent_workflow_tasks = 1000-max_concurrent_activities = 1000-max_concurrent_local_activities = 1000-max_concurrent_workflow_task_polls = 100-max_concurrent_activity_task_polls = 100-max_activities_per_second = 1000-sticky_queue_schedule_to_start_timeout = "0:30:00"-max_heartbeat_throttle_interval = "1:00:00"-default_heartbeat_throttle_interval = "1:00:00"-graceful_shutdown_timeout = "0:30:00"-```--Add a custom profile when one worker pool needs different tuning from another — for example, an LLM pool tied to a tier-4 provider quota:--```toml-[temporal.worker_runtime_profiles.profiles.anthropic-tier4]-tuning_mode = "explicit"-max_cached_workflows = 0-max_concurrent_workflow_tasks = 1-max_concurrent_activities = 50-max_concurrent_local_activities = 50-max_concurrent_workflow_task_polls = 1-max_concurrent_activity_task_polls = 50-max_activities_per_second = 8-sticky_queue_schedule_to_start_timeout = "0:30:00"-max_heartbeat_throttle_interval = "1:00:00"-default_heartbeat_throttle_interval = "1:00:00"-graceful_shutdown_timeout = "0:30:00"-```--Select it on the worker process:--```bash-pipelex worker --profile anthropic-tier4-```--`tuning_mode` must be `"explicit"` in this release. A `"resource_based"` value is reserved but not implemented yet — the config validator rejects it with a clear message.-------## Putting it together--A common production topology splits workers along three lines:--- One **router** worker pool that owns workflow orchestration and dispatches activities back out to specialized runner pools.-- One **LLM runner** pool per provider quota — separating OpenAI and Anthropic lets each scale independently.-- One **image-gen runner** pool that owns image generation activities (and their occasional page-view rendering side effects).--The corresponding `pipelex.toml` adds two profiles plus the routing config (see [Task-Queue Routing](task-routing.md) for the routing details):--```toml-[temporal.worker_config]-default_task_queue = "temporal_task_queue"--[temporal.activity_queues.act_llm_gen_text]-default = "openai_q"-by_handle.anthropic-default = "anthropic_q"--[temporal.activity_queues.act_img_gen_images]-default = "imggen_q"--[temporal.worker_runtime_profiles.profiles.openai-pool]-# ... explicit tuning for the OpenAI pool ...--[temporal.worker_runtime_profiles.profiles.anthropic-tier4]-# ... explicit tuning for the Anthropic pool ...-```--Boot four workers — typically one process each, on its own machine or container:--```bash-# Router: orchestrates workflows, no activity execution-pipelex worker --scope router --task-queue temporal_task_queue--# OpenAI LLM pool-pipelex worker --scope runner-llm --task-queue openai_q --profile openai-pool--# Anthropic LLM pool-pipelex worker --scope runner-llm --task-queue anthropic_q --profile anthropic-tier4--# Image generation pool-pipelex worker --scope runner-img-gen --task-queue imggen_q-```--The router worker hosts every workflow; each runner worker hosts only the activities its scope declares; routing decides which runner pool receives each activity at dispatch time.-------## What's next--- **[Task-Queue Routing](task-routing.md)** — route specific activities to specific worker pools with per-activity / per-handle precision, then layer per-queue timeouts and retry policies on top.-- **[Workflow Observability](observability.md)** — what each worker writes to the Temporal Web UI: workflow ids, activity summaries, search-attribute filters.diff --git a/docs/errors/broken-plugin-error.md b/docs/errors/broken-plugin-error.mdnew file mode 100644index 000000000..3315334be--- /dev/null+++ b/docs/errors/broken-plugin-error.md@@ -0,0 +1,21 @@+---+title: "Broken plugin"+description: "Reference for the `BrokenPluginError` Pipelex error class."+---++<!-- pipelex:generated -->++# Broken plugin++A discovered plugin failed while loading or registering itself.++| Field | Value |+|---|---|+| `error_type` | `BrokenPluginError` |+| `title` | Broken plugin |+| `type_uri` | `https://docs.pipelex.com/latest/errors/broken-plugin-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/concept-library-error.md b/docs/errors/concept-library-error.mdindex 96e9fe75b..cee0ad2b2 100644--- a/docs/errors/concept-library-error.md+++ b/docs/errors/concept-library-error.md@@ -7,6 +7,8 @@ description: "Reference for the `ConceptLibraryError` Pipelex error class."# Concept library+A concept-library failure (unresolved / duplicate / invalid concept reference).+| Field | Value ||---|---|| `error_type` | `ConceptLibraryError` |@@ -14,6 +16,6 @@ description: "Reference for the `ConceptLibraryError` Pipelex error class."| `type_uri` | `https://docs.pipelex.com/latest/errors/concept-library-error/` || `error_domain` | _(inherited from parent)_ || Defined in | `pipelex.libraries.concept.exceptions` |-| Parent class | [`LibraryError`](library-error.md) |+| Parent class | [`LibraryLoadingError`](library-loading-error.md) |[Back to Error Reference](index.md)diff --git a/docs/errors/content-generation-error.md b/docs/errors/content-generation-error.mddeleted file mode 100644index 4951fec69..000000000--- a/docs/errors/content-generation-error.md+++ /dev/null@@ -1,19 +0,0 @@-----title: "Content generation"-description: "Reference for the `ContentGenerationError` Pipelex error class."------<!-- pipelex:generated -->--# Content generation--| Field | Value |-|---|---|-| `error_type` | `ContentGenerationError` |-| `title` | Content generation |-| `type_uri` | `https://docs.pipelex.com/latest/errors/content-generation-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalFlowError`](temporal-flow-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/core-unconditional-plugin-disabled-error.md b/docs/errors/core-unconditional-plugin-disabled-error.mdnew file mode 100644index 000000000..1ad2db2f1--- /dev/null+++ b/docs/errors/core-unconditional-plugin-disabled-error.md@@ -0,0 +1,21 @@+---+title: "Core unconditional plugin disabled"+description: "Reference for the `CoreUnconditionalPluginDisabledError` Pipelex error class."+---++<!-- pipelex:generated -->++# Core unconditional plugin disabled++A plugin that core requires unconditionally was named in plugins.disabled.++| Field | Value |+|---|---|+| `error_type` | `CoreUnconditionalPluginDisabledError` |+| `title` | Core unconditional plugin disabled |+| `type_uri` | `https://docs.pipelex.com/latest/errors/core-unconditional-plugin-disabled-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/duplicate-bundle-validator-error.md b/docs/errors/duplicate-bundle-validator-error.mdnew file mode 100644index 000000000..1c983d7d9--- /dev/null+++ b/docs/errors/duplicate-bundle-validator-error.md@@ -0,0 +1,21 @@+---+title: "Duplicate bundle validator"+description: "Reference for the `DuplicateBundleValidatorError` Pipelex error class."+---++<!-- pipelex:generated -->++# Duplicate bundle validator++Two plugins registered a bundle validator for the same orchestration mode.++| Field | Value |+|---|---|+| `error_type` | `DuplicateBundleValidatorError` |+| `title` | Duplicate bundle validator |+| `type_uri` | `https://docs.pipelex.com/latest/errors/duplicate-bundle-validator-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/duplicate-http-error-mapper-error.md b/docs/errors/duplicate-http-error-mapper-error.mdnew file mode 100644index 000000000..af5c7d472--- /dev/null+++ b/docs/errors/duplicate-http-error-mapper-error.md@@ -0,0 +1,21 @@+---+title: "Duplicate http error mapper"+description: "Reference for the `DuplicateHttpErrorMapperError` Pipelex error class."+---++<!-- pipelex:generated -->++# Duplicate http error mapper++Two plugins registered an HTTP-error mapper for the same exception type.++| Field | Value |+|---|---|+| `error_type` | `DuplicateHttpErrorMapperError` |+| `title` | Duplicate http error mapper |+| `type_uri` | `https://docs.pipelex.com/latest/errors/duplicate-http-error-mapper-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/duplicate-inference-backend-error.md b/docs/errors/duplicate-inference-backend-error.mdnew file mode 100644index 000000000..f38b88ccd--- /dev/null+++ b/docs/errors/duplicate-inference-backend-error.md@@ -0,0 +1,21 @@+---+title: "Duplicate inference backend"+description: "Reference for the `DuplicateInferenceBackendError` Pipelex error class."+---++<!-- pipelex:generated -->++# Duplicate inference backend++Two plugins registered an inference backend for the same (family, sdk).++| Field | Value |+|---|---|+| `error_type` | `DuplicateInferenceBackendError` |+| `title` | Duplicate inference backend |+| `type_uri` | `https://docs.pipelex.com/latest/errors/duplicate-inference-backend-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/duplicate-model-lister-error.md b/docs/errors/duplicate-model-lister-error.mdnew file mode 100644index 000000000..2ffdbc7e0--- /dev/null+++ b/docs/errors/duplicate-model-lister-error.md@@ -0,0 +1,21 @@+---+title: "Duplicate model lister"+description: "Reference for the `DuplicateModelListerError` Pipelex error class."+---++<!-- pipelex:generated -->++# Duplicate model lister++Two plugins registered a model lister for the same sdk.++| Field | Value |+|---|---|+| `error_type` | `DuplicateModelListerError` |+| `title` | Duplicate model lister |+| `type_uri` | `https://docs.pipelex.com/latest/errors/duplicate-model-lister-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/duplicate-orchestrator-error.md b/docs/errors/duplicate-orchestrator-error.mdnew file mode 100644index 000000000..8d86f62eb--- /dev/null+++ b/docs/errors/duplicate-orchestrator-error.md@@ -0,0 +1,21 @@+---+title: "Duplicate orchestrator"+description: "Reference for the `DuplicateOrchestratorError` Pipelex error class."+---++<!-- pipelex:generated -->++# Duplicate orchestrator++Two plugins registered an orchestrator for the same orchestration mode.++| Field | Value |+|---|---|+| `error_type` | `DuplicateOrchestratorError` |+| `title` | Duplicate orchestrator |+| `type_uri` | `https://docs.pipelex.com/latest/errors/duplicate-orchestrator-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/execution-and-runtime.md b/docs/errors/execution-and-runtime.mdindex 689fdb967..047664dd9 100644--- a/docs/errors/execution-and-runtime.md+++ b/docs/errors/execution-and-runtime.md@@ -36,24 +36,10 @@ own page. Classes are grouped by subsystem.- [`PipelineManagerNotFoundError`](pipeline-manager-not-found-error.md) — Pipeline manager not found- [`ValidateBundleError`](validate-bundle-error.md) — Validate bundle-## Temporal execution--- [`ContentGenerationError`](content-generation-error.md) — Content generation-- [`SearchAttributeRegistrationError`](search-attribute-registration-error.md) — Search attribute registration-- [`TemporalConfigError`](temporal-config-error.md) — Temporal config-- [`TemporalFlowError`](temporal-flow-error.md) — Temporal flow-- [`TemporalServerError`](temporal-server-error.md) — Temporal server-- [`UnrecoverableWorkflowFailureError`](unrecoverable-workflow-failure-error.md) — Unrecoverable workflow failure-- [`WorkerProfileConfigError`](worker-profile-config-error.md) — Worker profile config-- [`WorkerScopeConfigError`](worker-scope-config-error.md) — Worker scope config-- [`WorkerTaskQueueUnknownError`](worker-task-queue-unknown-error.md) — Worker task queue unknown-- [`WorkflowExecutionError`](workflow-execution-error.md) — Workflow execution-- [`WorkflowInputError`](workflow-input-error.md) — Workflow input-## Runtime bridge-- [`MissingMistralWorkflowsPluginError`](missing-mistral-workflows-plugin-error.md) — Missing mistral workflows plugin-- [`MissingPipelexTemporalExtraError`](missing-pipelex-temporal-extra-error.md) — Missing pipelex temporal extra+- [`MissingBundleValidatorError`](missing-bundle-validator-error.md) — Missing bundle validator+- [`MissingOrchestratorError`](missing-orchestrator-error.md) — Missing orchestrator- [`PipelexBridgeDispatchError`](pipelex-bridge-dispatch-error.md) — Pipelex bridge dispatch- [`PipelexRuntimeBridgeError`](pipelex-runtime-bridge-error.md) — Pipelex runtime bridgediff --git a/docs/errors/hub-slot-already-claimed-error.md b/docs/errors/hub-slot-already-claimed-error.mdnew file mode 100644index 000000000..02b23e731--- /dev/null+++ b/docs/errors/hub-slot-already-claimed-error.md@@ -0,0 +1,21 @@+---+title: "Hub slot already claimed"+description: "Reference for the `HubSlotAlreadyClaimedError` Pipelex error class."+---++<!-- pipelex:generated -->++# Hub slot already claimed++Two plugins claimed the same hub slot.++| Field | Value |+|---|---|+| `error_type` | `HubSlotAlreadyClaimedError` |+| `title` | Hub slot already claimed |+| `type_uri` | `https://docs.pipelex.com/latest/errors/hub-slot-already-claimed-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/index.md b/docs/errors/index.mdindex c1bdf8a3e..6ec44d69e 100644--- a/docs/errors/index.md+++ b/docs/errors/index.md@@ -16,7 +16,7 @@ The classes are grouped into a few macro areas — pick the one that matches whethe error came from:- [Authoring & language](authoring-and-language.md) — Core language, Pipe operators, Pipe controllers, Pipe signatures, Builder, Libraries.-- [Execution & runtime](execution-and-runtime.md) — Pipe execution, Pipeline execution, Temporal execution, Runtime bridge, Graph, Tracing.+- [Execution & runtime](execution-and-runtime.md) — Pipe execution, Pipeline execution, Runtime bridge, Graph, Tracing.- [Inference & providers](inference-and-providers.md) — Inference (Cogt), Provider plugins.- [Platform & tooling](platform-and-tooling.md) — Base & root errors, Tools, Kit, System & configuration, CLI.diff --git a/docs/errors/inference-and-providers.md b/docs/errors/inference-and-providers.mdindex 876905eea..f54242e1f 100644--- a/docs/errors/inference-and-providers.md+++ b/docs/errors/inference-and-providers.md@@ -53,6 +53,7 @@ own page. Classes are grouped by subsystem.- [`ModelDeckPresetValidatonError`](model-deck-preset-validaton-error.md) — Model deck preset validaton- [`ModelDeckValidationError`](model-deck-validation-error.md) — Model deck validation- [`ModelDeckValidatonError`](model-deck-validaton-error.md) — Model deck validaton+- [`ModelListingUnsupportedError`](model-listing-unsupported-error.md) — Model listing unsupported- [`ModelManagerError`](model-manager-error.md) — Model manager- [`ModelNotFoundError`](model-not-found-error.md) — Model not found- [`ModelReferenceParseError`](model-reference-parse-error.md) — Model reference parse@@ -82,6 +83,13 @@ own page. Classes are grouped by subsystem.- [`AzureCredentialsError`](azure-credentials-error.md) — Azure credentials- [`BedrockFactoryError`](bedrock-factory-error.md) — Bedrock factory- [`BedrockWorkerConfigurationError`](bedrock-worker-configuration-error.md) — Bedrock worker configuration+- [`BrokenPluginError`](broken-plugin-error.md) — Broken plugin+- [`CoreUnconditionalPluginDisabledError`](core-unconditional-plugin-disabled-error.md) — Core unconditional plugin disabled+- [`DuplicateBundleValidatorError`](duplicate-bundle-validator-error.md) — Duplicate bundle validator+- [`DuplicateHttpErrorMapperError`](duplicate-http-error-mapper-error.md) — Duplicate http error mapper+- [`DuplicateInferenceBackendError`](duplicate-inference-backend-error.md) — Duplicate inference backend+- [`DuplicateModelListerError`](duplicate-model-lister-error.md) — Duplicate model lister+- [`DuplicateOrchestratorError`](duplicate-orchestrator-error.md) — Duplicate orchestrator- [`GatewayCredentialsError`](gateway-credentials-error.md) — Gateway credentials- [`GatewayDeckError`](gateway-deck-error.md) — Gateway deck- [`GatewayError`](gateway-error.md) — Gateway@@ -90,11 +98,15 @@ own page. Classes are grouped by subsystem.- [`GatewaySearchResponseError`](gateway-search-response-error.md) — Gateway search response- [`GoogleImgGenWorkerError`](google-img-gen-worker-error.md) — Google img gen worker- [`GoogleLLMWorkerError`](google-llm-worker-error.md) — Google LLM worker+- [`HubSlotAlreadyClaimedError`](hub-slot-already-claimed-error.md) — Hub slot already claimed+- [`InferenceBackendNotFoundError`](inference-backend-not-found-error.md) — Inference backend not found- [`MistralExtractResponseError`](mistral-extract-response-error.md) — Mistral extract response- [`MistralModelListingError`](mistral-model-listing-error.md) — Mistral model listing- [`MistralPluginError`](mistral-plugin-error.md) — Mistral plugin- [`MistralWorkerConfigurationError`](mistral-worker-configuration-error.md) — Mistral worker configuration- [`OpenAIClientFactoryError`](open-ai-client-factory-error.md) — OpenAI client factory error+- [`PluginApiVersionMismatchError`](plugin-api-version-mismatch-error.md) — Plugin api version mismatch+- [`PluginError`](plugin-error.md) — Plugin error- [`PortkeyCredentialsError`](portkey-credentials-error.md) — Portkey credentials- [`PortkeyError`](portkey-error.md) — Portkey- [`PortkeyFactoryError`](portkey-factory-error.md) — Portkey factorydiff --git a/docs/errors/inference-backend-not-found-error.md b/docs/errors/inference-backend-not-found-error.mdnew file mode 100644index 000000000..502029f37--- /dev/null+++ b/docs/errors/inference-backend-not-found-error.md@@ -0,0 +1,21 @@+---+title: "Inference backend not found"+description: "Reference for the `InferenceBackendNotFoundError` Pipelex error class."+---++<!-- pipelex:generated -->++# Inference backend not found++No inference backend is registered for a requested (family, sdk).++| Field | Value |+|---|---|+| `error_type` | `InferenceBackendNotFoundError` |+| `title` | Inference backend not found |+| `type_uri` | `https://docs.pipelex.com/latest/errors/inference-backend-not-found-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/missing-bundle-validator-error.md b/docs/errors/missing-bundle-validator-error.mdnew file mode 100644index 000000000..948936185--- /dev/null+++ b/docs/errors/missing-bundle-validator-error.md@@ -0,0 +1,21 @@+---+title: "Missing bundle validator"+description: "Reference for the `MissingBundleValidatorError` Pipelex error class."+---++<!-- pipelex:generated -->++# Missing bundle validator++Raised when no bundle validator is registered for a requested orchestration mode.++| Field | Value |+|---|---|+| `error_type` | `MissingBundleValidatorError` |+| `title` | Missing bundle validator |+| `type_uri` | `https://docs.pipelex.com/latest/errors/missing-bundle-validator-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.runtime_bridge.exceptions` |+| Parent class | [`PipelexRuntimeBridgeError`](pipelex-runtime-bridge-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/missing-mistral-workflows-plugin-error.md b/docs/errors/missing-mistral-workflows-plugin-error.mddeleted file mode 100644index 62eef13ca..000000000--- a/docs/errors/missing-mistral-workflows-plugin-error.md+++ /dev/null@@ -1,21 +0,0 @@-----title: "Missing mistral workflows plugin"-description: "Reference for the `MissingMistralWorkflowsPluginError` Pipelex error class."------<!-- pipelex:generated -->--# Missing mistral workflows plugin--Raised when MISTRAL_NATIVE mode is requested without ``pipelex-mistralai-workflows`` installed.--| Field | Value |-|---|---|-| `error_type` | `MissingMistralWorkflowsPluginError` |-| `title` | Missing mistral workflows plugin |-| `type_uri` | `https://docs.pipelex.com/latest/errors/missing-mistral-workflows-plugin-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.runtime_bridge.exceptions` |-| Parent class | [`PipelexRuntimeBridgeError`](pipelex-runtime-bridge-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/missing-orchestrator-error.md b/docs/errors/missing-orchestrator-error.mdnew file mode 100644index 000000000..3441929b0--- /dev/null+++ b/docs/errors/missing-orchestrator-error.md@@ -0,0 +1,21 @@+---+title: "Missing orchestrator"+description: "Reference for the `MissingOrchestratorError` Pipelex error class."+---++<!-- pipelex:generated -->++# Missing orchestrator++Raised when no orchestrator is registered for a requested orchestration mode.++| Field | Value |+|---|---|+| `error_type` | `MissingOrchestratorError` |+| `title` | Missing orchestrator |+| `type_uri` | `https://docs.pipelex.com/latest/errors/missing-orchestrator-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.runtime_bridge.exceptions` |+| Parent class | [`PipelexRuntimeBridgeError`](pipelex-runtime-bridge-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/missing-pipelex-temporal-extra-error.md b/docs/errors/missing-pipelex-temporal-extra-error.mddeleted file mode 100644index 2f77cbd95..000000000--- a/docs/errors/missing-pipelex-temporal-extra-error.md+++ /dev/null@@ -1,21 +0,0 @@-----title: "Missing pipelex temporal extra"-description: "Reference for the `MissingPipelexTemporalExtraError` Pipelex error class."------<!-- pipelex:generated -->--# Missing pipelex temporal extra--Raised when a TEMPORAL_* execution mode is requested without the pipelex[temporal] extra.--| Field | Value |-|---|---|-| `error_type` | `MissingPipelexTemporalExtraError` |-| `title` | Missing pipelex temporal extra |-| `type_uri` | `https://docs.pipelex.com/latest/errors/missing-pipelex-temporal-extra-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.runtime_bridge.exceptions` |-| Parent class | [`PipelexRuntimeBridgeError`](pipelex-runtime-bridge-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/model-listing-unsupported-error.md b/docs/errors/model-listing-unsupported-error.mdnew file mode 100644index 000000000..d55c18cf4--- /dev/null+++ b/docs/errors/model-listing-unsupported-error.md@@ -0,0 +1,21 @@+---+title: "Model listing unsupported"+description: "Reference for the `ModelListingUnsupportedError` Pipelex error class."+---++<!-- pipelex:generated -->++# Model listing unsupported++A registered lister cannot enumerate models for an SDK variant at runtime.++| Field | Value |+|---|---|+| `error_type` | `ModelListingUnsupportedError` |+| `title` | Model listing unsupported |+| `type_uri` | `https://docs.pipelex.com/latest/errors/model-listing-unsupported-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.cogt.exceptions` |+| Parent class | [`CogtError`](cogt-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/plugin-api-version-mismatch-error.md b/docs/errors/plugin-api-version-mismatch-error.mdnew file mode 100644index 000000000..c6d741b6c--- /dev/null+++ b/docs/errors/plugin-api-version-mismatch-error.md@@ -0,0 +1,21 @@+---+title: "Plugin api version mismatch"+description: "Reference for the `PluginApiVersionMismatchError` Pipelex error class."+---++<!-- pipelex:generated -->++# Plugin api version mismatch++A discovered plugin targets a plugin-API version this Pipelex does not support.++| Field | Value |+|---|---|+| `error_type` | `PluginApiVersionMismatchError` |+| `title` | Plugin api version mismatch |+| `type_uri` | `https://docs.pipelex.com/latest/errors/plugin-api-version-mismatch-error/` |+| `error_domain` | _(inherited from parent)_ |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PluginError`](plugin-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/plugin-error.md b/docs/errors/plugin-error.mdnew file mode 100644index 000000000..73e94185d--- /dev/null+++ b/docs/errors/plugin-error.md@@ -0,0 +1,21 @@+---+title: "Plugin error"+description: "Reference for the `PluginError` Pipelex error class."+---++<!-- pipelex:generated -->++# Plugin error++Base for plugin-discovery and plugin-registration failures.++| Field | Value |+|---|---|+| `error_type` | `PluginError` |+| `title` | Plugin error |+| `type_uri` | `https://docs.pipelex.com/latest/errors/plugin-error/` |+| `error_domain` | `config` |+| Defined in | `pipelex.plugins.exceptions` |+| Parent class | [`PipelexError`](pipelex-error.md) |++[Back to Error Reference](index.md)diff --git a/docs/errors/search-attribute-registration-error.md b/docs/errors/search-attribute-registration-error.mddeleted file mode 100644index bec56b448..000000000--- a/docs/errors/search-attribute-registration-error.md+++ /dev/null@@ -1,21 +0,0 @@-----title: "Search attribute registration"-description: "Reference for the `SearchAttributeRegistrationError` Pipelex error class."------<!-- pipelex:generated -->--# Search attribute registration--Raised at worker boot when the namespace is reachable but missing a configured custom search attribute. The error message includes both the ``pipelex setup-temporal-namespace`` invocation and the equivalent raw ``temporal operator search-attribute create`` command so operators on either side of the fence can fix the gap.--| Field | Value |-|---|---|-| `error_type` | `SearchAttributeRegistrationError` |-| `title` | Search attribute registration |-| `type_uri` | `https://docs.pipelex.com/latest/errors/search-attribute-registration-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalConfigError`](temporal-config-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/temporal-config-error.md b/docs/errors/temporal-config-error.mddeleted file mode 100644index 6ca9fc905..000000000--- a/docs/errors/temporal-config-error.md+++ /dev/null@@ -1,19 +0,0 @@-----title: "Temporal config"-description: "Reference for the `TemporalConfigError` Pipelex error class."------<!-- pipelex:generated -->--# Temporal config--| Field | Value |-|---|---|-| `error_type` | `TemporalConfigError` |-| `title` | Temporal config |-| `type_uri` | `https://docs.pipelex.com/latest/errors/temporal-config-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalFlowError`](temporal-flow-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/temporal-flow-error.md b/docs/errors/temporal-flow-error.mddeleted file mode 100644index ad227b9ed..000000000--- a/docs/errors/temporal-flow-error.md+++ /dev/null@@ -1,19 +0,0 @@-----title: "Temporal flow"-description: "Reference for the `TemporalFlowError` Pipelex error class."------<!-- pipelex:generated -->--# Temporal flow--| Field | Value |-|---|---|-| `error_type` | `TemporalFlowError` |-| `title` | Temporal flow |-| `type_uri` | `https://docs.pipelex.com/latest/errors/temporal-flow-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`PipelexError`](pipelex-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/temporal-server-error.md b/docs/errors/temporal-server-error.mddeleted file mode 100644index 3e77e4d38..000000000--- a/docs/errors/temporal-server-error.md+++ /dev/null@@ -1,19 +0,0 @@-----title: "Temporal server"-description: "Reference for the `TemporalServerError` Pipelex error class."------<!-- pipelex:generated -->--# Temporal server--| Field | Value |-|---|---|-| `error_type` | `TemporalServerError` |-| `title` | Temporal server |-| `type_uri` | `https://docs.pipelex.com/latest/errors/temporal-server-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalFlowError`](temporal-flow-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/unrecoverable-workflow-failure-error.md b/docs/errors/unrecoverable-workflow-failure-error.mddeleted file mode 100644index 7a12d8a3d..000000000--- a/docs/errors/unrecoverable-workflow-failure-error.md+++ /dev/null@@ -1,21 +0,0 @@-----title: "Unrecoverable workflow failure"-description: "Reference for the `UnrecoverableWorkflowFailureError` Pipelex error class."------<!-- pipelex:generated -->--# Unrecoverable workflow failure--Synthesized when a Temporal workflow failure carries no recoverable ``ErrorReport``.--| Field | Value |-|---|---|-| `error_type` | `UnrecoverableWorkflowFailureError` |-| `title` | Unrecoverable workflow failure |-| `type_uri` | `https://docs.pipelex.com/latest/errors/unrecoverable-workflow-failure-error/` |-| `error_domain` | `runtime` |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalFlowError`](temporal-flow-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/worker-profile-config-error.md b/docs/errors/worker-profile-config-error.mddeleted file mode 100644index 35a5d79ec..000000000--- a/docs/errors/worker-profile-config-error.md+++ /dev/null@@ -1,19 +0,0 @@-----title: "Worker profile config"-description: "Reference for the `WorkerProfileConfigError` Pipelex error class."------<!-- pipelex:generated -->--# Worker profile config--| Field | Value |-|---|---|-| `error_type` | `WorkerProfileConfigError` |-| `title` | Worker profile config |-| `type_uri` | `https://docs.pipelex.com/latest/errors/worker-profile-config-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalConfigError`](temporal-config-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/worker-scope-config-error.md b/docs/errors/worker-scope-config-error.mddeleted file mode 100644index 09b0db4c7..000000000--- a/docs/errors/worker-scope-config-error.md+++ /dev/null@@ -1,19 +0,0 @@-----title: "Worker scope config"-description: "Reference for the `WorkerScopeConfigError` Pipelex error class."------<!-- pipelex:generated -->--# Worker scope config--| Field | Value |-|---|---|-| `error_type` | `WorkerScopeConfigError` |-| `title` | Worker scope config |-| `type_uri` | `https://docs.pipelex.com/latest/errors/worker-scope-config-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalConfigError`](temporal-config-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/worker-task-queue-unknown-error.md b/docs/errors/worker-task-queue-unknown-error.mddeleted file mode 100644index 56d4a5dae..000000000--- a/docs/errors/worker-task-queue-unknown-error.md+++ /dev/null@@ -1,19 +0,0 @@-----title: "Worker task queue unknown"-description: "Reference for the `WorkerTaskQueueUnknownError` Pipelex error class."------<!-- pipelex:generated -->--# Worker task queue unknown--| Field | Value |-|---|---|-| `error_type` | `WorkerTaskQueueUnknownError` |-| `title` | Worker task queue unknown |-| `type_uri` | `https://docs.pipelex.com/latest/errors/worker-task-queue-unknown-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalConfigError`](temporal-config-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/workflow-execution-error.md b/docs/errors/workflow-execution-error.mddeleted file mode 100644index 170396a1b..000000000--- a/docs/errors/workflow-execution-error.md+++ /dev/null@@ -1,21 +0,0 @@-----title: "Workflow execution"-description: "Reference for the `WorkflowExecutionError` Pipelex error class."------<!-- pipelex:generated -->--# Workflow execution--A Temporal workflow failure observed on the submitter side.--| Field | Value |-|---|---|-| `error_type` | `WorkflowExecutionError` |-| `title` | Workflow execution |-| `type_uri` | `https://docs.pipelex.com/latest/errors/workflow-execution-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalFlowError`](temporal-flow-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/errors/workflow-input-error.md b/docs/errors/workflow-input-error.mddeleted file mode 100644index 3cbdc8c2c..000000000--- a/docs/errors/workflow-input-error.md+++ /dev/null@@ -1,19 +0,0 @@-----title: "Workflow input"-description: "Reference for the `WorkflowInputError` Pipelex error class."------<!-- pipelex:generated -->--# Workflow input--| Field | Value |-|---|---|-| `error_type` | `WorkflowInputError` |-| `title` | Workflow input |-| `type_uri` | `https://docs.pipelex.com/latest/errors/workflow-input-error/` |-| `error_domain` | _(inherited from parent)_ |-| Defined in | `pipelex.temporal.exceptions` |-| Parent class | [`TemporalFlowError`](temporal-flow-error.md) |--[Back to Error Reference](index.md)diff --git a/docs/features/distributed-execution.md b/docs/features/distributed-execution.mdindex a7c5f10ac..4ff7b4cd5 100644--- a/docs/features/distributed-execution.md+++ b/docs/features/distributed-execution.md@@ -1,45 +1,27 @@---title: "Distributed Execution"-description: "Run Pipelex methods as durable workflows on Temporal — your own cluster today, with Mistral's managed Workflows backend coming soon."+description: "Run Pipelex methods as durable, horizontally-scaled workflows — crash survival, per-step retries, and operational visibility, delivered through the Pipelex platform."---# Distributed Execution-Run your `.mthds` methods as durable workflows.+Run your `.mthds` methods as durable workflows — the same methods you build locally, executed with crash survival, per-step retries, and horizontal scale.## Overview-Pipelex pipelines normally run in-process. When you need durability, retries that survive failure, and horizontal scale, the same pipelines run as durable workflows — each pipe becomes a workflow, child pipes become child workflows, and every LLM call, image generation, or document extraction becomes an activity. The orchestration layer handles durability, retries, scheduling, and visibility; Pipelex handles the AI work, and the same methods run distributed without changing a line of method code.+Pipelex pipelines normally run in-process. When you need durability, retries that survive failure, and scale, the same pipelines run as durable workflows: each pipe becomes a workflow, child pipes become child workflows, and every LLM call, image generation, or document extraction becomes an independently-retried unit of work. Pipelex handles the AI work; the orchestration layer handles durability, retries, scheduling, and visibility — and your method code doesn't change.-## Backends--Distributed execution runs on Temporal. Today there is one backend, with a managed option coming soon:--- **[Pipelex on Temporal](../distributed-execution/temporal/index.md)** — you run Pipelex's own Temporal workers against a Temporal cluster you operate (self-hosted or Temporal Cloud). Python 3.10+, `pipelex[temporal]`. Generally available.-- **Pipelex on Mistral Workflows** *(coming soon)* — run pipes inside Mistral Workflows, Mistral's managed orchestration control plane (itself built on Temporal), with no cluster to operate. In active development; docs land once it ships.--## Pipelex on Temporal+## What you get-Flip `[temporal] is_enabled = true` in `.pipelex/pipelex.toml`, install `pipelex[temporal]`, and the same methods run as Temporal workflows on workers you operate.+- **Crash survival** — long pipelines resume exactly where they left off after a restart.+- **Per-step retries** — each LLM call, extraction, or image generation retries on its own, with its own timeout and retry policy.+- **Horizontal scale** — fan work across workers and route workloads to pools that scale and fail independently.+- **Operational visibility** — every run is durable, observable, and replayable.-### Supported deployment patterns--- **Single worker** — one process polls one task queue, runs everything. Right for most deployments.-- **Router + runners** — a dedicated workflow worker dispatches activities to one or more runner pools (LLM, image-gen, extract). Each runner pool scales independently and isolates failures.-- **Per-provider isolation** — separate worker pools for OpenAI, Anthropic, image generation, and OCR, each on its own task queue with its own retry policy and rate cap.--### Configuration--All knobs live under `[temporal.*]` in `.pipelex/pipelex.toml`:+## Backends-- `[temporal.search_attributes]` — custom search attributes attached to every workflow start.-- `[temporal.worker_config]` — default task queue, workflow and activity timeouts, baseline retry policy.-- `[temporal.activity_queues.<activity>]` — per-activity, per-handle task-queue routing.-- `[temporal.queue_options.<queue>]` — per-queue timeout, retry, and rate-cap overlays.-- `[temporal.worker_runtime_profiles.profiles.<name>]` — named runtime profiles for `pipelex worker --profile`.-- `[temporal.worker_scopes.scopes.<name>]` — named scopes for `pipelex worker --scope`.-- `[temporal.temporal_config]` — server profiles (`local`, `testing`, …) and log toggles.+Distributed execution is delivered through the Pipelex platform, on proven orchestration engines — [Temporal-backed durable execution](https://pipelex.com/products#temporal), and a [Mistral Workflows](https://pipelex.com/products#mistral-workflows) integration that runs your pipes inside Mistral's managed Workflows. Both run the identical methods through the same Pipelex runtime, so the durability model and the error contract are the same whichever one executes your work.## Get started-See the **[Distributed Execution](../distributed-execution/index.md)** guide for the full walkthrough — overview, cluster setup, worker deployment, task-queue routing, and workflow observability. A managed Mistral Workflows backend is coming soon.+See the **[Distributed Execution](../distributed-execution/index.md)** capability guide, or explore the platform at **[Pipelex products](https://pipelex.com/products#durable-execution)**.diff --git a/docs/features/index.md b/docs/features/index.mdindex 8ccdfa375..327587b5d 100644--- a/docs/features/index.md+++ b/docs/features/index.md@@ -97,7 +97,7 @@ This section covers all Pipelex capabilities, organized by area.- **[Distributed Execution](distributed-execution.md)**- Run methods as durable workflows on Temporal with your own workers, with a managed Mistral Workflows backend coming soon. Per-activity routing, named runtime profiles, and dashboard observability on the Temporal backend.+ Run the same methods as durable workflows on a Temporal or Mistral Workflows backend, [delivered through the Pipelex platform](https://pipelex.com/products#durable-execution). Per-activity routing, named runtime profiles, and dashboard observability.- **[Telemetry & Observability](telemetry.md)**diff --git a/docs/features/validation-dry-run.md b/docs/features/validation-dry-run.mdindex 828daf97a..821675a1b 100644--- a/docs/features/validation-dry-run.md+++ b/docs/features/validation-dry-run.md@@ -39,4 +39,4 @@ List specific pipe codes in `pipelex.dry_run_config.allowed_to_fail_pipes` so drFor configuration details, see [Dry Run Configuration](../configuration/config-pipeline-validation/dry-run-config.md).-Dry run is one of two run modes (live, dry run), each of which works in-process or distributed on Temporal — see [Run Modes & Backends](../building-methods/pipes/run-modes-and-backends.md) for the full matrix.+Dry run is one of two run modes (live, dry run), each of which works in-process or distributed on a durable backend — see [Run Modes & Backends](../building-methods/pipes/run-modes-and-backends.md) for the full matrix.diff --git a/docs/reliability/durable-execution.md b/docs/reliability/durable-execution.mdindex 330cd9361..a43dd6085 100644--- a/docs/reliability/durable-execution.md+++ b/docs/reliability/durable-execution.md@@ -14,14 +14,14 @@ Stay on direct execution while you are developing, prototyping, or running shortMove to a [durable backend](../distributed-execution/index.md) when you need:- **Crash survival** — a long pipeline resumes exactly where it left off after a worker restart.-- **Retry under failure** — each LLM call, extraction, or image generation retries independently, with per-activity timeouts and a retry policy keyed off the [error category](failure-classification.md). (This per-leaf retry is the Temporal model, where each leaf maps to one host activity. On Mistral Workflows in `direct` mode the whole pipe runs inside a single host activity, so it retries as a unit; per-leaf retry there needs `mistral_native` mode.)+- **Retry under failure** — each LLM call, extraction, or image generation retries independently, with per-step timeouts and a retry policy keyed off the [error category](failure-classification.md). (With per-leaf retry, each leaf retries on its own; depending on the backend, a pipe may instead retry as a single unit.)- **Large durable batches** — running a pipe over thousands of items, durably and rate-limited.- **Horizontal scale** — fan work out across multiple worker machines.-The same `.mthds` methods run on every path without changing a line — which durable backend runs them is a deployment choice, not a code change. The [Pipelex on Temporal](../distributed-execution/temporal/index.md) backend is switched on with `[temporal] is_enabled = true`, which dispatches the work through your Temporal cluster. A second, managed backend (Mistral Workflows) is coming soon: that flag will not apply to it — pipes run inside Workflows activities via the runtime bridge.+The same `.mthds` methods run on every path without changing a line — which durable backend runs them is a deployment choice, not a code change. [Distributed execution](../distributed-execution/index.md) is delivered through the Pipelex platform, on a Temporal-backed engine or a Mistral Workflows integration.!!! note "Durable backend"- Durable execution (Tier 2) runs on Temporal: your own Temporal cluster ([Pipelex on Temporal](../distributed-execution/temporal/index.md)). A managed option — Mistral's Workflows control plane, also built on Temporal — is coming soon; the resilience model described here will apply to it too, the difference being who operates the control plane.+ Durable execution (Tier 2) runs on a durable backend — [Temporal-backed execution or a Mistral Workflows integration](../distributed-execution/index.md) — delivered through the Pipelex platform. The resilience model described here applies the same way whichever backend runs your work.!!! tip "The error you see is the same on both paths"A pipe that fails on a Temporal worker reaches your CLI or HTTP adapter with the *same* classification — category, retryable flag, model, provider, suggested action — as the identical failure run locally. Switching to a durable backend changes the resilience, not the error contract.diff --git a/docs/reliability/retries-and-resilience.md b/docs/reliability/retries-and-resilience.mdindex e4a74b7d7..edb548507 100644--- a/docs/reliability/retries-and-resilience.md+++ b/docs/reliability/retries-and-resilience.md@@ -57,7 +57,8 @@ This section goes deeper on each piece:| `transport_max_retries` | `[cogt]` | `2` | Tier 1 — transport retry attempts per request || `schema_reask_max_attempts` | `[cogt.llm_config]` | `3` | `instructor` schema re-ask attempts (structured output) || `max_concurrency` | `[pipelex.pipeline_execution_config]` | `8` | `PipeBatch` bounded fan-out (set `"unbounded"` to disable) |-| `is_enabled` | `[temporal]` | `false` | Opt into durable execution (Tier 2) |++Durable execution (Tier 2) is **not** a core configuration toggle. You opt into it by booting the process under an orchestrator plugin — `plugins.boot_orchestrator` (config), `--orchestrator <name>` (CLI), or `Pipelex.make(boot_orchestrator=...)` (Python) — delivered through Pipelex's [durable execution offer](https://pipelex.com/products#durable-execution). See [Durable Execution](durable-execution.md) and [Distributed Execution](../distributed-execution/index.md).---diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.mdindex 649963160..644e954d7 100644--- a/docs/under-the-hood/distributed-content-generation.md+++ b/docs/under-the-hood/distributed-content-generation.md@@ -1,35 +1,33 @@----title: "Distributed Content Generation"-description: "How Pipelex serializes dynamic Pydantic models and large binary content across Temporal workflow/activity boundaries — JSON schema propagation, class source injection, and storage-based payload offloading."+title: "Content Generation Across Boundaries"+description: "How Pipelex serializes dynamic Pydantic models and large binary content across a worker boundary — JSON schema propagation, class source injection, and storage-based payload offloading — independent of the host runtime."----# Distributed Content Generation+# Content Generation Across Boundaries-This page is for contributors working on Pipelex internals. If you're deploying or operating Pipelex on Temporal, see the user-facing [Distributed Execution with Temporal](../distributed-execution/temporal/index.md) guide instead.+This page is for contributors working on Pipelex internals. For the capability overview, see the user-facing [Distributed Execution](../distributed-execution/index.md) page instead.-This page covers the serialization mechanisms that enable content generation (LLM structured output, image generation, PDF extraction, web search) to run across Temporal worker processes. For the broader Temporal architecture — LibraryCrate propagation, deferred hydration, per-workflow scoping — see [Temporal Integration](./temporal-integration.md).+When content generation (LLM structured output, image generation, PDF extraction, web search) runs on a separate worker process, two serialization problems appear that single-process execution never faces. This page covers them backend-neutrally — the mechanisms live in open core and are the same regardless of which host runtime carries the payload. The concrete activity dispatch is a [commercial platform capability](https://pipelex.com/products#durable-execution); each backend (Temporal, Mistral Workflows) realizes it in its own plugin.!!! note "Every inference leaf goes through the same seam"- All inference operators dispatch their leaf call through the swappable `ContentGenerator` abstraction: direct inline (`ContentGenerator`) or as a Temporal activity (`ContentGeneratorInWorkflow`); the backend choice is independent of the run mode — under `run_mode=DRY` the chosen backend still dispatches and the cogt leaf mocks inside it. Web search (`PipeSearch`) joined this seam last — its `act_search_gen_*` activities make search-on-Temporal replay-safe (the result is recorded in workflow history) and let search failures cross the workflow boundary as classified errors. An operator that runs its leaf inline instead would re-execute it on every replay and, on failure, hang the submitter with an unclassified workflow-task error.+ All inference operators dispatch their leaf call through the swappable `ContentGenerator` abstraction (`pipelex/cogt/content_generation/content_generator.py`): direct inline, or — on a host runtime — wrapped as an activity by the runtime's in-workflow content generator. The backend choice is independent of the run mode: under `run_mode=DRY` the chosen backend still dispatches and the leaf mocks inside it. Wrapping the leaf as a host-runtime activity is what makes it replay-safe (the result is recorded in the run's history) and lets a leaf failure cross the worker boundary as a classified error, instead of re-executing on every replay and hanging the submitter with an unclassified fault.----## The Two Serialization Challenges--Content generation introduces two cross-cutting problems that don't exist in single-process mode:+## The two serialization challenges| Challenge | Trigger | Root cause | Solution ||-----------|---------|------------|----------|-| **Dynamic class propagation** | LLM structured output (`PipeLLM` with object concept) | Submitter creates a Pydantic model at runtime from `.mthds` concept definitions; the worker process has no such class | Embed JSON schema in assignment, reconstruct class on worker, carry source code through Temporal payload metadata |-| **Large payload management** | Image generation (`PipeImgGen`), PDF extraction (`PipeExtract`) | Binary data (base64 images, rendered pages) exceeds Temporal's ~2MB payload limit | Activities store binary data to external storage, return lightweight URI references |+| **Dynamic class propagation** | LLM structured output (`PipeLLM` with object concept) | The submitter creates a Pydantic model at runtime from `.mthds` concept definitions; the worker process has no such class | Embed the JSON schema in the assignment, reconstruct the class on the worker, carry the source code through the transport's payload metadata |+| **Large payload management** | Image generation (`PipeImgGen`), PDF extraction (`PipeExtract`) | Binary data (base64 images, rendered pages) exceeds a host runtime's payload limit (Temporal's is ~2 MB) | Activities store binary data to external storage, return lightweight URI references |----## Dynamic Class Propagation+## Dynamic class propagation### Schema embedding-When a pipe produces structured output, the caller captures the Pydantic model's JSON schema into the assignment — a serializable Temporal argument that carries everything the worker needs:+When a pipe produces structured output, the caller captures the Pydantic model's JSON schema into the assignment — a serializable argument that carries everything the worker needs:```pythonclass ObjectAssignment(BaseModel):@@ -49,11 +47,11 @@ class ObjectAssignment(BaseModel):)```-The schema is a plain `dict` — fully serializable, visible in the Temporal dashboard, and self-contained. No class reference crosses the process boundary.+The schema is a plain `dict` — fully serializable, inspectable on the wire, and self-contained. No class reference crosses the process boundary.### Schema-to-model reconstruction-On the worker, `model_class_from_json_schema()` rebuilds a live Pydantic class from the embedded schema:+On the worker, `model_class_from_json_schema()` (`pipelex/cogt/content_generation/schema_to_model_factory.py`) rebuilds a live Pydantic class from the embedded schema:```pythonsource_code = _generate_source_from_schema(schema) # datamodel-code-generator@@ -66,16 +64,16 @@ The reconstruction pipeline:1. **Generate** — `datamodel-code-generator` converts the JSON schema into Python source code defining a `BaseModel` subclass.2. **Execute** — `exec()` compiles the source in an isolated namespace and extracts the named class.3. **Cache** — A thread-safe SHA-256 hash cache (with double-check locking) avoids redundant generation for the same schema.-4. **Tag** — The generated source code is attached as `__kajson_class_source__` on the class, enabling Kajson to carry it through Temporal payloads.+4. **Tag** — The generated source code is attached as `__kajson_class_source__` on the class, enabling Kajson to carry it through the transport.!!! warning "Class name normalization"`datamodel-code-generator` normalizes schema titles to PascalCase (e.g., `dynamic_concept_test__Greeting` becomes `DynamicConceptTestGreeting`). The code tries both the original and normalized names. If neither matches, it raises `ValueError` listing the available classes.-### Source code injection into Temporal payloads+### Carrying the class source across the boundary-When the activity returns a `BaseModel` whose class has `__kajson_class_source__`, the custom `BaseModelPayloadConverter` injects it into the Temporal payload metadata:+The reconstructed class's source rides in the transport's **payload metadata**, not in the payload data itself — keeping the JSON payload clean and the reconstruction transparent. A host runtime's data converter injects it on serialize and re-attaches it on deserialize:-**Serialization** (`to_payload`):+**Serialize:**```pythonclass_source = getattr(type(value), "__kajson_class_source__", None)@@ -83,7 +81,7 @@ if class_source is not None:metadata["kajson_class_source"] = class_source.encode()```-**Deserialization** (`from_payload`):+**Deserialize:**```pythonsource_bytes = payload.metadata.get("kajson_class_source")@@ -93,105 +91,71 @@ pydantic_gizmo = kajson.loads(class_registry=get_class_registry(),class_source_code=class_source_code,)-# Re-attach source so it survives further Temporal hops-if class_source_code is not None:- if isinstance(pydantic_gizmo, BaseModel):- type(pydantic_gizmo).__kajson_class_source__ = class_source_code- elif isinstance(pydantic_gizmo, list):- for item in pydantic_gizmo:- if isinstance(item, BaseModel):- type(item).__kajson_class_source__ = class_source_code+# Re-attach source so it survives further hops+if class_source_code is not None and isinstance(pydantic_gizmo, BaseModel):+ type(pydantic_gizmo).__kajson_class_source__ = class_source_code```-The source code rides in Temporal metadata (not the payload data itself), keeping the JSON payload clean and the class reconstruction transparent. After `kajson.loads()` reconstructs the object, the converter re-attaches `__kajson_class_source__` to the deserialized class — this ensures the source survives if the object crosses another Temporal boundary (e.g. workflow returning a result to a parent activity that forwards it elsewhere).+Re-attaching after `kajson.loads()` ensures the source survives if the object crosses another boundary (e.g. a worker returning a result to a parent that forwards it). The `__kajson_class_source__` attachment is open core; the converter that injects it into payload metadata is the host-runtime plugin's (Pipelex's Temporal backend realizes it as `BaseModelPayloadConverter`).### The type bridge-The class reconstructed on the worker is a structural match to the original, but it is a different Python class in memory. Standard `isinstance()` checks would fail. The caller bridges this gap with a `model_validate` round-trip:+The class reconstructed on the worker is a structural match to the original, but a different Python class in memory — so `isinstance()` checks would fail. The caller bridges the gap with a `model_validate` round-trip (`pipelex/cogt/content_generation/content_generator.py`):```pythonraw_obj = await llm_gen_object(object_assignment=object_assignment)return object_class.model_validate(raw_obj.model_dump(serialize_as_any=True))```-`model_dump(serialize_as_any=True)` produces a plain dict from the reconstructed object, and `model_validate()` rebuilds it as a proper instance of the original class. This restores type safety for downstream code.--### End-to-end flow--```mermaid-sequenceDiagram- participant S as Submitter- participant W as Worker Activity- participant LLM as LLM Provider-- S->>S: object_class.model_json_schema()- S->>W: ObjectAssignment (class_name + schema)-- Note over W: model_class_from_json_schema()- W->>W: datamodel-code-generator → source- W->>W: exec() → class + __kajson_class_source__-- W->>LLM: gen_object(schema=reconstructed_class)- LLM-->>W: structured JSON-- W-->>S: BaseModel result<br/>(metadata: kajson_class_source)-- Note over S: kajson.loads(class_source_code=...)- S->>S: model_validate() → original type-```+`model_dump(serialize_as_any=True)` produces a plain dict from the reconstructed object, and `model_validate()` rebuilds it as a proper instance of the original class, restoring type safety for downstream code.### Structured web search reuses the same mechanism-Structured web search (`PipeSearch` with a non-text output concept) faces the identical dynamic-class problem and solves it the same way. `SearchObjectAssignment` mirrors `ObjectAssignment` — it ships `output_class_name` + `output_class_schema` alongside the `SearchAssignment`, the activity reconstructs a throwaway class via `SchemaToModelFactory` to drive the provider call, and the activity returns the **raw result dict**. The submitter re-validates that dict against the original output class (`output_structure_class.model_validate(result_dict)`) — a pure, deterministic step that keeps the dynamic class on the submitter side and never ships it across the boundary. The sourced-answer path (`make_search_sourced_answer`) has no dynamic class at all: it returns a `SearchResultContent`, a native serializable model.+Structured web search (`PipeSearch` with a non-text output concept) faces the identical dynamic-class problem and solves it the same way. `SearchObjectAssignment` mirrors `ObjectAssignment` — it ships `output_class_name` + `output_class_schema` alongside the `SearchAssignment`, the activity reconstructs a throwaway class to drive the provider call, and returns the **raw result dict**. The submitter re-validates that dict against the original output class (`output_structure_class.model_validate(result_dict)`) — a pure, deterministic step that keeps the dynamic class on the submitter side and never ships it across the boundary. The sourced-answer path (`make_search_sourced_answer`) has no dynamic class at all: it returns a `SearchResultContent`, a native serializable model.----## Large Payload Management+## Large payload management### The payload size constraint-Temporal payloads have a practical limit around 2MB. A single generated image can exceed this in base64 encoding. PDF extraction produces multiple page images. Sending raw binary through Temporal would either fail with a `DataConverterError` or severely degrade workflow history performance.+A host runtime's payload has a practical size limit (Temporal's is around 2 MB). A single generated image can exceed it in base64; PDF extraction produces multiple page images. Sending raw binary through the runtime would fail conversion or degrade the run history.### The store-then-reference pattern-Activities handle the full binary lifecycle internally — generate, store, return a lightweight reference:+Activities handle the full binary lifecycle internally — generate, store, return a lightweight reference. The storing is open-core: `GeneratedContentFactory` (`pipelex/cogt/content_generation/generated_content_factory.py`) uploads binary data via the configured `StorageProviderAbstract` (S3, local filesystem, …) and returns `ImageContent` or `PageContent` — Pydantic models carrying URIs, MIME types, and metadata, but never raw bytes. A host runtime's activity simply wraps that helper:```python-async def act_img_gen_images(img_gen_assignment: ImgGenAssignment) -> list[ImageContent]:- """Generate images and store them, returning lightweight ImageContent references.-- Large binary data (base64/bytes) is stored within the activity and never crosses- the Temporal workflow boundary — only URLs are returned.- """- storage_provider = get_storage_provider()- generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)+async def generate_and_store_images(img_gen_assignment):+ """Large binary (base64/bytes) is stored inside the activity and never+ crosses the runtime boundary — only URLs come back."""+ factory = GeneratedContentFactory(storage_provider=get_storage_provider())return await img_gen_image_list_and_store(img_gen_assignment=img_gen_assignment,- generated_content_factory=generated_content_factory,+ generated_content_factory=factory,)```-`GeneratedContentFactory` uploads binary data via the configured `StorageProviderAbstract` (S3, local filesystem, etc.) and returns `ImageContent` or `PageContent` — Pydantic models carrying URIs, MIME types, and metadata, but never raw bytes.-!!! info "What crosses the boundary"`ImageContent` carries `url` (storage URI), `public_url`, `mime_type`, `size`, `caption` — but never raw bytes. The `url` can be an S3 URI, HTTP URL, or local file path depending on storage configuration.-### Content type scenarios+What gets stored vs. what crosses the boundary, by content type:++| Content type | What gets stored | What crosses the boundary |+|-------------|-----------------|----------------------------|+| Generated image | Base64/bytes → storage via `GeneratedContentFactory` | `list[ImageContent]` with URIs |+| Extracted pages | Extracted page images → storage | `list[PageContent]` with URI references |+| Rendered page views | PDF page renders → storage | `list[ImageContent]` with URIs |+| LLM text | Nothing (text is small) | Plain `str` |+| LLM object | Nothing (JSON is small) | `BaseModel` + `__kajson_class_source__` in metadata |+| Search sourced answer | Nothing (answer + source refs are small) | `SearchResultContent` (answer + `DocumentContent` sources) |+| Search structured | Nothing (JSON is small) | Raw `dict`, re-validated against the output class on the submitter |-| Content type | Activity | What gets stored | What crosses the Temporal boundary |-|-------------|----------|-----------------|-----------------------------------|-| Generated image | `act_img_gen_images` | Base64/bytes → S3 via `GeneratedContentFactory` | `list[ImageContent]` with URIs |-| Extracted pages | `act_extract_gen_extract_pages` | Extracted page images → S3 | `list[PageContent]` with URI references |-| Rendered page views | `act_render_page_views` | PDF page renders → S3 | `list[ImageContent]` with URIs |-| LLM text | `act_llm_gen_text` | Nothing stored (text is small) | Plain `str` |-| LLM object | `act_llm_gen_object` | Nothing stored (JSON is small) | `BaseModel` + `__kajson_class_source__` in metadata |-| Search sourced answer | `act_search_gen_sourced_answer` | Nothing stored (answer + source refs are small) | `SearchResultContent` (answer + `DocumentContent` sources) |-| Search structured | `act_search_gen_structured` | Nothing stored (JSON is small) | Raw `dict`, re-validated against the output class on the submitter |+Each host-runtime plugin dispatches these through its own activities and queue routing; for the Temporal realization (the `act_*` activity set, per-activity task queues), see the `pipelex-temporal` plugin's docs.----## Kajson Class Resolution+## Kajson class resolutionWhen `kajson.loads()` receives both a `class_registry` and `class_source_code`, the resolution follows a priority chain:@@ -209,7 +173,7 @@ if class_source_code is not None:Resolution order:-1. **Explicit `class_registry`** — the per-workflow scoped registry (see [Temporal Integration](./temporal-integration.md#per-workflow-scoping))+1. **Explicit `class_registry`** — the per-call scoped registry (see [Runtime Bridge & Transport](./runtime-bridge-and-transport.md#per-call-scoping))2. **Source-derived classes** — from `class_source_code`, filling gaps in the explicit registry3. **`sys.modules`** — already-imported classes4. **Dynamic import** — last resort, importing the module path from `__module__` metadata@@ -229,19 +193,16 @@ Resolution order:| LLM generation functions | `pipelex/cogt/content_generation/llm_generate.py` || Search generation functions | `pipelex/cogt/content_generation/search_generate.py` || Generated content factory (storage) | `pipelex/cogt/content_generation/generated_content_factory.py` |-| Temporal data converter | `pipelex/temporal/temporal_data_converter.py` |-| Image generation activity | `pipelex/temporal/tprl_content_generation/act_img_gen_generate.py` |-| Extract activity | `pipelex/temporal/tprl_content_generation/act_extract_generate.py` |-| Search activity | `pipelex/temporal/tprl_content_generation/act_search_generate.py` |-| Page view rendering activity | `pipelex/temporal/tprl_content_generation/act_render_page_views.py` || Kajson serialization | `kajson` (external PyPI package) || ImageContent model | `pipelex/core/stuffs/image_content.py` || PageContent model | `pipelex/core/stuffs/page_content.py` |+The host-runtime data converter and the `act_*` activities that carry these payloads are backend-specific and live in their plugin repos, not in open core.+---## Next Steps-- [Temporal Integration](./temporal-integration.md) — LibraryCrate propagation, deferred hydration, per-workflow scoping+- [Runtime Bridge & Transport](./runtime-bridge-and-transport.md) — boundary DTOs, LibraryCrate propagation, deferred hydration, per-call scoping- [Pipe Routing & Execution](./pipe-routing-and-execution.md) — how PipeJobs travel through the system- [Architecture Overview](./architecture-overview.md) — the two-layer design and how components fit togetherdiff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.mdindex d113b1811..a84ce4a3f 100644--- a/docs/under-the-hood/error-model.md+++ b/docs/under-the-hood/error-model.md@@ -333,7 +333,7 @@ A downstream FastAPI exception handler calls `ErrorReport.http_status` and is a**Inputs.** `to_error_report()` takes a live `PipelexError`. `recover_error_report()` takes any `BaseException` and walks its `__cause__` chain. `ErrorReport.from_dict()` takes a `to_dict()` payload — strictly, raising `ValidationError` on drift.-**Outputs.** `to_error_report()` returns an `ErrorReport`; `to_dict()` returns a `None`-free `dict`. Side effects: telemetry events emitted on pipeline failure at Layer 3; the agent CLI writes to stderr and raises `typer.Exit(...)` — code 1 by default, or the validate surface's 0/1/2 policy (see [Validate exit-code policy](#validate-exit-code-policy-0--1--2)).+**Outputs.** `to_error_report()` returns an `ErrorReport`; `to_dict()` returns a `None`-free `dict`. Side effects: telemetry events emitted on pipeline failure at Layer 3; the agent CLI writes to stderr and raises `typer.Exit(...)` — code 1 by default, or the validate surface's 0/1/2 policy (see [Validate exit-code policy](#validate-exit-code-policy-0-1-2)).---@@ -436,9 +436,9 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT| `pipelex/cogt/inference/provider_name.py` | `ProviderName` enum keying the extract-fn registry || `pipelex/plugins/*/` | Per-provider inference workers — Layer 0 → 1 classification || `pipelex/pipeline/exceptions.py` | `PipelineExecutionError`, `PipeExecutionError` |-| `pipelex/temporal/tprl/temporal_error.py` | `TemporalError`, `from_message_exception`, `recover_error_report` |-| `pipelex/temporal/tprl/activity_error_boundary.py` | `convert_pipelex_errors` decorator |-| `pipelex/temporal/tprl/workflow_caller.py` | `WorkflowExecutor`, `WorkflowExecutionError` recovery |+| `pipelex_temporal/tprl/temporal_error.py` *(in the closed `pipelex-temporal` plugin)* | `TemporalError`, `from_message_exception`, `recover_error_report` |+| `pipelex_temporal/tprl/activity_error_boundary.py` *(plugin)* | `convert_pipelex_errors` decorator |+| `pipelex_temporal/tprl/workflow_caller.py` *(plugin)* | `WorkflowExecutor`, `WorkflowExecutionError` recovery || `pipelex/cli/error_handlers.py` | Human CLI Rich panels — `display_error_panel()` || `pipelex/cli/agent_cli/commands/agent_output.py` | Agent CLI JSON / markdown delivery |@@ -462,6 +462,6 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT## Next Steps- [Pipe Routing & Execution](./pipe-routing-and-execution.md) — the layer model errors rise through-- [Temporal Integration](./temporal-integration.md) — the activity → workflow boundary the error bridge spans+- [Runtime Bridge & Transport](./runtime-bridge-and-transport.md) — the process boundary the error bridge spans (the per-backend error converters live in the host-runtime plugins)- [Cogt Configuration](../configuration/config-technical/cogt-config.md) — `transport_max_retries` and the Tier 1 retry policy- [Agent CLI](../tools/cli/agent-cli.md) — the JSON / markdown error contractdiff --git a/docs/under-the-hood/index.md b/docs/under-the-hood/index.mdindex ea5a59792..72499e1b3 100644--- a/docs/under-the-hood/index.md+++ b/docs/under-the-hood/index.md@@ -19,15 +19,16 @@ Welcome to the technical deep-dives of Pipelex. This section is for contributors- **Test Profile Configuration** - How to configure which models are used in tests- **Dry Run Mock Generation** - How mock objects satisfy field validation constraints- **Init CLI Flows** - How `pipelex init` sets up the configuration directory-- **Distributed Content Generation** - How dynamic classes and large payloads cross Temporal boundaries+- **Runtime Bridge & Transport** - The backend-neutral seam that carries a pipe run across process boundaries — boundary DTOs, LibraryCrate propagation, deferred hydration, and per-call isolation+- **Content Generation Across Boundaries** - How dynamic classes and large payloads cross a worker boundary- **Technical Design Decisions** - Why we chose X over Y- **Module Deep-Dives** - Detailed explanations of specific subsystems!!! info "Not Required for Using Pipelex"You don't need to read this section to use Pipelex effectively. The [Home](../index.md) section covers everything you need to build methods.-!!! tip "Looking for Temporal deployment and operations?"- For running Pipelex on Temporal — cluster setup, workers, task-queue routing, and dashboard observability — see the user-facing [Distributed Execution with Temporal](../distributed-execution/temporal/index.md) guide. The pages below cover the runtime mechanics that make distributed execution work under the hood.+!!! tip "Looking for distributed execution?"+ For running Pipelex methods as durable, distributed workflows, see the user-facing [Distributed Execution](../distributed-execution/index.md) overview. The pages below cover the runtime mechanics that make distributed execution work under the hood.---@@ -44,6 +45,6 @@ Welcome to the technical deep-dives of Pipelex. This section is for contributors- [:material-console: Init CLI Flows](./init-cli-flows.md){ .md-button }- [:material-pipe: Pipe Routing & Execution](./pipe-routing-and-execution.md){ .md-button }- [:material-alert-circle-outline: Error Model](./error-model.md){ .md-button }-- [:material-cloud-sync: Temporal Integration](./temporal-integration.md){ .md-button }-- [:material-swap-horizontal: Distributed Content Generation](./distributed-content-generation.md){ .md-button }+- [:material-transit-connection-variant: Runtime Bridge & Transport](./runtime-bridge-and-transport.md){ .md-button }+- [:material-swap-horizontal: Content Generation Across Boundaries](./distributed-content-generation.md){ .md-button }diff --git a/docs/under-the-hood/inference-backend-plugins.md b/docs/under-the-hood/inference-backend-plugins.mdnew file mode 100644index 000000000..77074607c--- /dev/null+++ b/docs/under-the-hood/inference-backend-plugins.md@@ -0,0 +1,246 @@+---+title: "Inference Backend Plugins"+description: "How Pipelex discovers inference backends through the plugin seam, the Inference SPI a backend plugin compiles against, and how to author one."+---++# Inference Backend Plugins++Every model call in Pipelex — an LLM completion, an image generation, a document extraction, a web search — is served by an **inference worker**. Which worker handles a given model is decided entirely by data: a model's `sdk` field selects a backend, and a **backend plugin** is what teaches Pipelex how to build the worker for that `sdk`.++Core names no backend by import or by string. The built-in drivers (OpenAI, Gateway, Anthropic, Mistral, Bedrock, Google, FAL, HuggingFace, Docling, …) are plugins too — they ride the exact same seam an out-of-tree plugin would. This page documents that seam, the **Inference SPI** a plugin compiles against, and how to write one.++---++## The seam in one view++```+boot (Pipelex.setup)+ └─ build_registrar(config) # pure, import-light+ ├─ for each plugin in BUILTIN_PLUGINS+ └─ for each installed "pipelex.plugins" entry point+ └─ plugin.register(registrar) # side-effect-free+ └─ registrar.add_inference_backend(family=…, sdk=…, make_worker=…)+ └─ InferenceBackendRegistry(registrar.inference_backends) # stored on the hub++run time (e.g. LLMWorkerFactory.make_llm_worker)+ └─ model_handle = ModelHandle.make_for_inference_model(...)+ └─ make_worker = get_inference_backend_registry().lookup(family=LLM, sdk=model_handle.sdk)+ └─ worker = make_worker(inference_model=…, backend=…, sdk_clients=…, reporting_delegate=…)+```++The worker factories (`LLMWorkerFactory`, `ImgGenWorkerFactory`, `ExtractWorkerFactory`, `SearchWorkerFactory`) hold **no** `match` over SDK strings. They build a `ModelHandle`, resolve the `InferenceBackend` config, look up the backend's `make_worker` by `(family, sdk)`, and call it. A lookup miss raises a friendly `NotImplementedError` ("… Is its plugin installed?").++---++## The contract: `PipelexPlugin`++A plugin is any object satisfying the `@runtime_checkable` `PipelexPlugin` protocol:++```python+class PipelexPlugin(Protocol):+ name: str # unique, lowercase identifier (e.g. "openai")+ targets_api: int # must equal PLUGIN_API_VERSION+ def register(self, registrar: PluginRegistrar) -> None: ...+```++`register` is the **only** method core calls, and it is **side-effect-free**: it may call the registrar's menu methods and nothing else — no hub access, no I/O, no client/SDK construction. This is what makes `build_registrar` safe to run more than once (it runs at boot, and again at CLI-build to harvest plugin-contributed commands).++`targets_api` is checked against `PLUGIN_API_VERSION`. A mismatch fails loud with `PluginApiVersionMismatchError` — a single coarse integer gate, not semver-range matching.++---++## Registering an inference backend++A backend plugin's `register` calls one menu method per `(family, sdk)` it serves:++```python+registrar.add_inference_backend(+ family=InferenceFamily.LLM, # LLM | IMG_GEN | EXTRACT | SEARCH+ sdk="acme", # the model's `sdk` string+ make_worker=_make_acme_worker, # a MakeWorkerFn (a plain callable)+)+```++A registry key is `(family, sdk)`. The same `sdk` string may appear in two families (e.g. `google` serves both `LLM` and `IMG_GEN`); they are distinct keys. A duplicate `(family, sdk)` fails loud with `DuplicateInferenceBackendError` naming **both** contributing plugins.++One plugin may register across several families from a single `register` — the built-in `gateway` plugin serves all four, `mistral` serves `LLM` + `EXTRACT`, `linkup` serves `EXTRACT` + `SEARCH`. This is the cross-family-vendor coordination point: one plugin, many backends.++---++## `MakeWorkerFn` — the locked call shape++A backend is a typed **callable**, not a one-method object. Its signature is the contract:++```python+def _make_acme_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ ...+```++The factory always passes all four keyword arguments. A stateless backend simply ignores the ones it doesn't need (`# noqa: ARG001` on the unused parameter — see the built-in `pypdfium2` and `azure_rest` plugins).++Two invariants shape what goes *inside* the closure:++- **Import-light.** The plugin module must import no backend SDK at module load. Do the SDK import **inside** the closure (`# noqa: PLC0415`), so merely discovering the plugin never pulls a heavy/optional dependency. Booting Pipelex with the built-ins registered imports none of `anthropic`, `mistralai`, `google.genai`, `boto3`, `fal_client`, `huggingface_hub`, `docling`, `linkup`, … — enforced by a subprocess import-blocker guard.+- **Fail at use, not at boot.** Guard an optional dependency with `require_sdk(...)` *inside* the closure. A missing extra then raises `MissingDependencyError` (naming the package and the `pipelex[<extra>]` install hint) only when the backend is actually used.++Client memoization goes through `sdk_clients.get_or_create(handle=…, build=lambda: …)` — the registry caches one SDK client per `ModelHandle`, so repeated worker construction reuses the connection.++---++## Listing models — an optional capability++Alongside its worker factory, a backend plugin **may** register a model lister — the callable behind `pipelex show models <backend>`. It is optional: a backend that cannot enumerate its models simply never calls `add_model_lister`, and the listing command reports that SDK as unsupported-for-listing. The contract grows by one *optional* method, so progressive disclosure is preserved.++```python+registrar.add_model_lister(sdk="acme", lister=_list_acme_models)+```++A `ListModelsFn` mirrors `MakeWorkerFn` — import-light to reference, lazy inside. It is always `async` (the listing loop awaits it) and is keyed by `sdk` alone (listing is per-SDK, not per-`(family, sdk)`):++```python+async def _list_acme_models(+ *,+ sdk: str,+ backend_name: str,+ backend: InferenceBackend,+ flat: bool,+ any_listed: bool,+) -> None:+ from my_pkg.acme_list import list_acme_models # noqa: PLC0415++ await list_acme_models(sdk=sdk, backend_name=backend_name, backend=backend, flat=flat, any_listed=any_listed)+```++The same import-light / fail-at-use rules apply: a missing optional extra must raise `MissingDependencyError` only when the backend is actually listed — use the `require_sdk(...)` helper, or an equivalent `find_spec` guard inside the function the lister delegates to (the built-in listers do the latter, reusing the guard already in their `list_*_models` functions). If a registered lister's client variant cannot enumerate models at runtime (e.g. a Bedrock-backed Anthropic client), raise `ModelListingUnsupportedError(sdk=…)`: the loop treats that as the same soft "unsupported for listing" outcome as a missing lister — never a hard failure. A duplicate `sdk` lister fails loud with `DuplicateModelListerError` naming both plugins.++---++## Authoring a backend plugin (minimal example)++A complete LLM backend plugin for a hypothetical `acme` SDK:++```python+# acme_plugin.py+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol++_ACME_MISSING_MSG = "The acme SDK is required in order to use Acme models."+++def _make_acme_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="acme", extra="acme", msg=_ACME_MISSING_MSG)++ from acme import AcmeClient # noqa: PLC0415++ from my_pkg.acme_llm_worker import AcmeLLMWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: AcmeClient(api_key=backend.api_key),+ )+ return AcmeLLMWorker(+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class AcmePlugin:+ name = "acme"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="acme", make_worker=_make_acme_worker)+```++### Shipping it as an out-of-tree plugin++Declare an entry point in the `pipelex.plugins` group; discovery finds it automatically once the distribution is installed (no central enable-list — *presence* is the source of truth):++```toml+# pyproject.toml of your plugin package+[project.entry-points."pipelex.plugins"]+acme = "my_pkg.acme_plugin:AcmePlugin"+```++The entry point may resolve to a plugin instance or a zero-argument factory. A broken entry point is isolated and reported as `BrokenPluginError`, never a silent skip.++### Disabling a discovered plugin++`build_registrar` skips (and logs) any plugin whose `name` appears in the core-owned `plugins.disabled` denylist:++```toml+# .pipelex/pipelex.toml+[plugins]+disabled = ["acme"]+```++For external entry points the denylist is matched against the **entry-point name** *before* the plugin is loaded, so a broken or dependency-missing installed plugin can still be disabled to recover startup (it would otherwise raise `BrokenPluginError` at load). A working external plugin that sets a `name` different from its entry-point name is denylistable by **either** name.++Disabling a core-unconditional plugin (`openai`) is a configuration error, not a no-op — it raises `CoreUnconditionalPluginDisabledError`. There is intentionally **no** allowlist: a plugin's presence (built-in or installed entry point) is what enables it.++Use `pipelex plugins list` to see every discovered plugin, what each contributed, and its denylist state.++---++## The Inference SPI++What an out-of-tree backend plugin imports *is* the contract. The published surface:++| Symbol | Module | Role |+|--------|--------|------|+| `PipelexPlugin`, `PLUGIN_API_VERSION` | `pipelex.plugins.contract` | the plugin protocol + version gate |+| `PluginRegistrar` | `pipelex.plugins.registrar` | the accumulator `register` writes into |+| `InferenceFamily`, `MakeWorkerFn`, `require_sdk` | `pipelex.plugins.inference_backend_registry` | family enum, callable type, dependency guard |+| `ListModelsFn` | `pipelex.plugins.model_lister_registry` | the optional model-listing callable type |+| `ModelListingUnsupportedError` | `pipelex.cogt.exceptions` | soft signal a lister raises when its client variant cannot list |+| `ModelHandle` | `pipelex.plugins.model_handle` | the backend selector derived from a model spec |+| `SdkClientRegistry` | `pipelex.plugins.sdk_client_registry` | per-handle client memoization |+| `InferenceModelSpec` | `pipelex.cogt.model_backends.model_spec` | the resolved model record |+| `InferenceBackend` | `pipelex.cogt.model_backends.backend` | the backend config record (api key, extras) |+| `InferenceWorkerAbstract` and the `{LLM,ImgGen,Extract,Search}WorkerAbstract` subclasses | `pipelex.cogt.…` | the worker contracts a plugin returns |+| `MissingDependencyError` | `pipelex.exceptions` | raised by `require_sdk` |++The SPI is a documented, versioned **module/symbol list** gated by `PLUGIN_API_VERSION` — not an `__init__.py` re-export shim (the repo bans re-exports; import by full path). Anything a plugin needs to import outside this surface is a design gap to resolve, not an accident to live with.++---++## Fail-loud guarantees++| Condition | Error |+|-----------|-------|+| `targets_api` ≠ `PLUGIN_API_VERSION` | `PluginApiVersionMismatchError` |+| duplicate `(family, sdk)` | `DuplicateInferenceBackendError` (names both plugins) |+| duplicate `sdk` model lister | `DuplicateModelListerError` (names both plugins) |+| `name` in `plugins.disabled` but core-unconditional | `CoreUnconditionalPluginDisabledError` |+| entry point raises while loading/registering | `BrokenPluginError` |+| lookup for an unregistered `(family, sdk)` | `InferenceBackendNotFoundError` ("… Is its plugin installed and enabled?") |+| optional SDK missing at use | `MissingDependencyError` (package + `pipelex[<extra>]` hint) |++---++## Related++- [Pipe Routing & Execution](pipe-routing-and-execution.md) — where worker construction sits in the run path+- [Error Model](error-model.md) — how these errors render and dereference+- [Architecture Overview](architecture-overview.md)diff --git a/docs/under-the-hood/orchestrator-plugins.md b/docs/under-the-hood/orchestrator-plugins.mdnew file mode 100644index 000000000..9338fc903--- /dev/null+++ b/docs/under-the-hood/orchestrator-plugins.md@@ -0,0 +1,174 @@+---+title: "Orchestrator Plugins"+description: "How Pipelex dispatches a pipe run by orchestration mode through the orchestrator seam, the Orchestrator SPI a host-runtime plugin compiles against, and how the external pipelex-temporal plugin rides it."+---++# Orchestrator Plugins++A pipe a host runtime invokes through the runtime bridge runs along **two orthogonal axes**:++- **`orchestration_mode`** — *which* orchestrator runs the pipe. An **open string token**, not a closed enum: core owns only `"direct"` (in-process); every other token is contributed by the plugin that owns its orchestrator — [`"temporal"`](https://pipelex.com/products#temporal) (durable, on a Temporal worker fleet) by `pipelex-temporal`, [`"mistralai-workflows"`](https://pipelex.com/products#mistral-workflows) (decomposed into Mistral Workflows primitives) by `pipelex-mistralai-workflows`. Both are commercial host-runtime backends, part of Pipelex's [workflow-orchestration offer](https://pipelex.com/products#durable-execution).+- **`delivery`** — *whether the caller waits*. A **closed** core `DeliveryMode` enum (`BLOCKING` / `FIRE_AND_FORGET`), set by the endpoint and passed as a parameter to `run`, never received from a caller. An orchestrator honors it per its nature; `supports_fire_and_forget` advertises whether it can do genuine async.++An **orchestrator** is what knows how to run a pipe under one token. Core names no orchestrator by import or by string. The bridge resolves the orchestrator for the requested token from a registry (keyed by the token `str`) and calls its `run` — `"direct"` is contributed by a core plugin, `"temporal"` by the Temporal plugin, `"mistralai-workflows"` by the external `pipelex-mistralai-workflows` plugin. A lookup miss raises a generic `MissingOrchestratorError` that names no orchestrator. This page documents that seam, the **Orchestrator SPI** a host-runtime plugin compiles against, and how the Temporal plugin is wired.++---++## The seam in one view++```+run_pipe_via_bridge(input_payload) # pipelex_transport.bridge — closed host-runtime library+ → build the PipeJob (boundary decode + library scope + trace_context)+ → orchestrator = get_orchestrator_registry().get_optional(mode=orchestration_mode)+ → if orchestrator is None: raise MissingOrchestratorError(mode) # generic, names no orchestrator+ → return await orchestrator.run(pipe_job=..., delivery_assignment=..., delivery=...)+```++The registry is built once at boot from whatever the discovered plugins contributed (`build_registrar` → `OrchestratorRegistry` on the hub). There is no `match orchestration_mode:` anywhere in the bridge — the token set is open, so validation is the registry lookup itself; adding a mode's behavior means registering an orchestrator for its token, nothing in core changes.++!!! note "Open seam, closed host-side entry"+ The host-side entry — `run_pipe_via_bridge` / `build_pipe_job_from_input` and the dispatch primitives that decode the boundary payload, scope the per-call library, and deliver results — lives in the **closed `pipelex-transport` library**. Embedding Pipelex into a host runtime (Temporal, Mistral, Airflow, your own) is a Pipelex [commercial capability](https://pipelex.com/products#durable-execution), not an open extension point. What stays **open** is the seam a third-party orchestrator extends: the orchestrator registry, `OrchestratorProtocol`, and the `pipelex.runtime_bridge` modules listed in [The Orchestrator SPI](#the-orchestrator-spi) below.++---++## The orchestrator contract++An orchestrator satisfies `OrchestratorProtocol` (`pipelex/plugins/orchestrator_registry.py`):++```python+class OrchestratorProtocol(Protocol):+ supports_fire_and_forget: bool++ async def run(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None, delivery: DeliveryMode) -> PipelexPipeRunOutput: ...+```++`run` honors the endpoint-chosen `delivery` per the orchestrator's nature (in-process always blocks; a distributed orchestrator awaits completion for `BLOCKING` and returns a workflow id for `FIRE_AND_FORGET`). `supports_fire_and_forget` is the capability a runner reads *before* dispatch — `/start` rejects honestly (4xx) when the resolved mode cannot do genuine async, instead of silently running blocking and acking.++A plugin contributes one per token it serves by calling the registrar menu in its `register`, passing the token as a raw string (no enum, no cast):++```python+registrar.add_orchestrator(mode="temporal", orchestrator=TemporalOrchestrator())+```++Constructing the orchestrator instance must be **import-light** — it must not import the host-runtime SDK (`temporalio`, …) at module scope or in its `__init__`. The heavy import happens lazily inside `run` (and a friendly `MissingOrchestratorError` is raised there if the mode's extra is absent), so discovering and registering the plugin never pulls the SDK. This is what keeps boot import-light even on a process that will never use the mode.++### A missing orchestrator is a generic, plugin-decoupled error++A token with no registered orchestrator (its plugin is not installed) raises `MissingOrchestratorError(mode=...)` (`pipelex/runtime_bridge/exceptions.py`). The message names the token but **no orchestrator** — *"No orchestrator is registered for orchestration mode '{mode}'; is its plugin installed?"* — so core stays fully decoupled from its plugins (it never spells out `pipelex-temporal` / `pipelex-mistralai-workflows`). The one special case is the core `"direct"` token: its orchestrator is always present, so a miss there reports a boot/discovery fault. The message survives STRICT error disclosure.++---++## HTTP error mappers: rendering an orchestrator's transport faults++An orchestrator's *runtime* (a Temporal client, a Mistral workflow runner) raises SDK-specific transport faults — a server unreachable, a workflow timeout — that a host runtime serving HTTP must turn into a proper error response, not a catch-all 500. But core names no web framework and the SDK lives only in the plugin, so the host (`pipelex-api`) cannot itself know how to classify `temporalio.TemporalError`.++The plugin bridges that gap by contributing a **framework-agnostic mapper** — a function from one exception type to a structured `ErrorReport` (`pipelex/base_exceptions.py`):++```python+registrar.add_http_error_mapper(+ exc_type_provider=lambda: TemporalError, # SDK imported only when a host resolves the mappers, never at register+ to_error_report=lambda exc: ErrorReport(...), # classified transient / RUNTIME+)+```++The exc type is supplied as a **provider thunk**, not the bare class, on purpose: naming `temporalio.TemporalError` requires importing `temporalio` (the whole SDK), so a bare `exc_type=` would force that import at `register` — breaking the import-light invariant for a plugin that hard-depends on a heavy SDK. The provider defers the import to read time (a host runtime's app construction), where the plugin — and therefore its SDK — is by definition installed.++The contract is deliberately split so no layer overreaches:++- **The plugin** owns *classification* — which exception, transient or not, which error domain. It stays import-light: `register` only records the provider + closure; the SDK import happens when the provider runs at read time (and the `to_error_report` closure when the mapper is first invoked), never at registration.+- **Core** owns *transport* — `registrar.get_http_error_mappers()` runs every provider, builds the `{exc_type: mapper}` dict, and is fail-loud on a duplicate *resolved* exception type (naming both plugins). `ErrorReport` is a core type, so the seam carries **no** web-framework import.+- **The host runtime** owns *presentation* — at app construction it iterates the resolved mappers and wraps each into one framework error handler (FastAPI, …) that runs the mapper, then renders the `ErrorReport` through its own RFC 7807 + `DisclosureMode` path. FastAPI / Starlette stays only in the host; core and the plugin import neither.++This is what lets the public `pipelex-api` base be orchestrator-agnostic and still render a Temporal (or Mistral) transport fault correctly: install the flavor's plugin and its mapper rides in; install none and there is simply nothing to wrap. The capability is optional, so it grew the plugin contract by one method → `PLUGIN_API_VERSION` is now **2**.++---++## Boot-orchestrator plugins: claiming the runtime++Some orchestrators don't just serve a per-call mode — they reconfigure the whole process to run *as* that runtime (a Temporal worker). Such a plugin **claims process-global hub slots**, but only when the core-owned boot gate names it. `plugins.boot_orchestrator == self.name` means "boot this process as a Temporal-default runtime", not "the Temporal plugin is on". The gate is a backend-agnostic name-match — core names no orchestrator, and `register` reads no config file (the rich orchestrator config self-loads inside the thunks):++```python+if registrar.config.plugins.boot_orchestrator == self.name:+ registrar.claim_content_generator(_make_temporal_content_generator) # a thunk, not an instance+ registrar.claim_task_manager(_setup_temporal_task_manager)+ registrar.claim_pipe_router(_make_temporal_pipe_router)+ registrar.claim_pipe_run(_make_temporal_pipe_run)+ registrar.add_teardown(_teardown_temporal)+```++Each `claim_*` takes a **thunk** (a zero-arg factory), never a constructed instance. The thunk runs only at the boot apply-point, so `register` itself imports no `temporalio` — even on a worker. This is the deferred-thunk rule that keeps the import-light invariant intact at boot.++### Injection precedence++At each ordered hub slot, `Pipelex.setup` resolves in this precedence:++1. an explicit `setup()` parameter (test/host injection) — always wins;+2. a plugin slot-claim thunk;+3. the core default.++A slot claim must never silently override an explicit injection. Teardown runs the plugin-registered teardown callbacks **LIFO**, before core teardown, so a worker's in-flight runtime resources release first.++---++## Operational commands ship as console scripts++The plugin seam does **not** contribute commands to the host `pipelex` CLI. An operational command — a worker daemon, a one-time namespace bootstrap — is a daemon/utility, not a way a pipe *runs*, so a plugin that needs one ships its own `[project.scripts]` console script, which pip materializes into a standalone executable. Nothing is harvested onto `pipelex` at import time, so a broken or colliding plugin can never brick `pipelex --help` / `doctor` / `init`.++The `pipelex-temporal` plugin follows this rule: its `worker` and `setup-namespace` commands ship as the `pipelex-temporal` console script (`pipelex-temporal worker`, `pipelex-temporal setup-namespace`), declared in the plugin distribution's `pyproject.toml`:++```toml+[project.scripts]+pipelex-temporal = "pipelex_temporal.temporal_cli:app"+```++Because the script is declared by the `pipelex-temporal` distribution itself, it travels with that dist — nothing in core to harvest or move.++---++## The Orchestrator SPI++What an out-of-tree orchestrator imports *is* a contract. The SPI is a documented, versioned set of modules and symbols (gated by `PLUGIN_API_VERSION` in `pipelex/plugins/contract.py`) — not an `__init__.py` re-export shim. It is sized to what a real orchestrator (`pipelex-mistralai-workflows`) actually imports, not guessed. Anything an orchestrator needs that is *outside* this surface is a design bug — promote it into the SPI or remove the need.++| Area | Modules / symbols |+|---|---|+| Boundary serialization + boot | `pipelex.runtime_bridge.serialization` (`serialize_pipe_output`, `serialize_completed_output`, `PIPE_DISPATCH_ERRORS`), `pipelex.runtime_bridge.payloads` (`PipelexPipeRunInput`, `PipelexPipeRunOutput`), `pipelex.runtime_bridge.bootstrap` (`ensure_pipelex_booted`) |+| Mode + delivery + errors | `pipelex.runtime_bridge.orchestration_mode` (`OrchestrationMode`, `DIRECT_ORCHESTRATION_MODE`), `pipelex.runtime_bridge.delivery_mode` (`DeliveryMode`), `pipelex.runtime_bridge.exceptions` (`MissingOrchestratorError`, `PipelexBridgeDispatchError`) |+| Working-memory hydration | `pipelex.runtime_bridge.primitives.hydration` (re-hydrate `working_memory_raw` → typed `WorkingMemory`; stayed open because it is host-agnostic — used by core delivery and the open `pipelex-api` runner, and re-used across the boundary by `pipelex-transport`) |+| Plugin contract | `pipelex.plugins.contract` (`PipelexPlugin`, `PLUGIN_API_VERSION`), `pipelex.plugins.registrar` (`PluginRegistrar` menu: `add_orchestrator`, `add_http_error_mapper`, `claim_*`, `add_teardown`; read accessor: `get_http_error_mappers`), `pipelex.plugins.orchestrator_registry` (`OrchestratorProtocol`) |+| Execution protocols | `PipeRouterProtocol`, `PipeRunProtocol`, `ContentGeneratorProtocol`, the task-manager protocol |+| Payload / core types | `PipeJob`, `PipeOutput`, `DeliveryAssignment`, `WorkingMemory` (+ factory, `dump_for_transport`), `JobMetadata`, `LibraryCrate` |+| Library + hub scoping | `set_current_library` / `get_current_library`, `scoped_pipe_router`, `get_class_registry` (per-call library hydration via `library_crate_dump`) |+| Tracing / graph hooks | `trace_events`, `graph_tracer_manager`, `tracing_assembly` (per-step trace/usage events across the boundary) |++!!! warning "Not in the open SPI — the closed `pipelex-transport` layer"+ The host-side bridge entry (`run_pipe_via_bridge`, `build_pipe_job_from_input`), the cross-process dispatch primitives (`delivery`, `pipe_classification`, `submitter_hydration`, `trace_flush`, `scoped_library`, `pipe_run_arg`), and the transport prepare helpers (`prepare_job_for_transport`, `prepare_output_for_transport`) were extracted out of open core into the **closed `pipelex-transport` library**. Embedding Pipelex into a host runtime is a Pipelex commercial capability — Pipelex's own host-runtime plugins (`pipelex-temporal`, `pipelex-mistralai-workflows`) compile against `pipelex-transport`; a third-party out-of-tree orchestrator compiles against the **open** rows above only.++---++## Worked example: the `pipelex-temporal` plugin++`pipelex_temporal/temporal_plugin.py` (in the external `pipelex-temporal` distribution) is the reference orchestrator plugin. Its `register`:++- **always** (regardless of the boot gate): contributes a single `TemporalOrchestrator` registered once under the `"temporal"` token (import-light; `temporalio` is pulled lazily inside `run`). It advertises `supports_fire_and_forget = True`, and its `run` branches on the endpoint-chosen `delivery`: `BLOCKING` awaits completion and reports `make_workflow_id(...)`; `FIRE_AND_FORGET` calls `.start(...)` and returns an `is_completed=False` output carrying the workflow id;+- **only when `plugins.boot_orchestrator == "temporal"`**: claims the content-generator / task-manager / pipe-router / pipe-run hub slots with thunks and registers the teardown callback — booting this process as a Temporal-default runtime.++The orchestrator itself (`pipelex_temporal/temporal_orchestrators.py`) carries both delivery bodies behind one exhaustive `match delivery`, keeping the `WorkflowExecutionError` catch and the `make_workflow_id` recompute in the blocking arm. It serializes its `PipeOutput` through `pipelex.runtime_bridge.serialization`, shared with the core DIRECT orchestrator so the boundary shape cannot drift.++The Temporal plugin is **external** — it ships as the `pipelex-temporal` distribution and is discovered through a `pipelex.plugins` entry point in that dist's `pyproject.toml`, not through `BUILTIN_PLUGINS`. Core's `BUILTIN_PLUGINS` (`pipelex/plugins/builtins.py`) holds only the always-shipped inference and `direct` plugins and explicitly excludes Temporal; installing `pipelex-temporal` is all it takes to make the `"temporal"` orchestrator available — zero config, no core import of `temporalio`. Its operational `worker` / `setup-namespace` commands ship as the standalone `pipelex-temporal` console script, so they travel with that dist.++---++## Authoring an out-of-tree orchestrator plugin++A third-party host-runtime plugin is a distribution that:++1. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_orchestrator(mode=..., orchestrator=...)` for the mode(s) it serves — import-light;+2. compiles its orchestrator against the Orchestrator SPI above (and nothing outside it);+3. advertises itself under the `pipelex.plugins` entry-point group:++```toml+[project.entry-points."pipelex.plugins"]+my_runtime = "my_package.my_plugin:MyRuntimePlugin"+```++Installing the distribution makes the mode available; uninstalling removes it. No core change, no central registration list. A discovered plugin can be quarantined without uninstalling via the `plugins.disabled` denylist (see [Inference Backend Plugins](inference-backend-plugins.md) for the shared discovery/denylist machinery).diff --git a/docs/under-the-hood/pipe-routing-and-execution.md b/docs/under-the-hood/pipe-routing-and-execution.mdindex 7697c7cc2..11e19a828 100644--- a/docs/under-the-hood/pipe-routing-and-execution.md+++ b/docs/under-the-hood/pipe-routing-and-execution.md@@ -5,7 +5,7 @@ description: "How PipeJobs are created, routed, and executed — covering both d# Pipe Routing & Execution-Pipelex runs pipes either **direct** (single-process) or **distributed** on a host runtime. Both share the same pipe definitions, library loading, and controller logic. The key difference is *where* the pipe runs and how the PipeJob travels to get there. The distributed host runtime today is your own Temporal workers ([Pipelex on Temporal](../distributed-execution/temporal/index.md)); a managed Mistral Workflows backend (also built on Temporal) is coming soon. Both reach pipe execution through the same framework-agnostic runtime bridge.+Pipelex runs pipes either **direct** (single-process) or **distributed** on a host runtime. Both share the same pipe definitions, library loading, and controller logic. The key difference is *where* the pipe runs and how the PipeJob travels to get there. The distributed host runtime is a durable backend — Temporal-backed execution or a Mistral Workflows integration — delivered through the Pipelex platform (see [Distributed Execution](../distributed-execution/index.md)). Both reach pipe execution through the same framework-agnostic runtime bridge.---@@ -19,7 +19,7 @@ Pipelex runs pipes either **direct** (single-process) or **distributed** on a ho---!!! note "The runtime bridge"- Both distributed paths go through the framework-agnostic runtime bridge (`pipelex/runtime_bridge/`). Its `primitives/` layer is shared by `pipelex.temporal` and the in-development Mistral Workflows host package: it classifies controller pipes as child workflows and leaf operators as activities (`pipe_classification.py`), hydrates working memory, flushes trace events, and delivers results in a host-neutral way. Graph and token-usage assembly is not a bridge primitive — it lives in `pipelex/pipe_run/tracing_assembly.py` and runs the same way for local and distributed runs.+ Both distributed paths reach pipe execution through the framework-agnostic runtime bridge. The open `pipelex/runtime_bridge/` keeps the boot, serialization, mode/delivery, and worker-side hydration (`primitives/hydration.py`) pieces; the cross-process **dispatch** primitives — classifying controller pipes as child workflows and leaf operators as activities (`pipe_classification`), flushing trace events (`trace_flush`), and delivering results (`delivery`) — were extracted into the closed `pipelex-transport` library, which both commercial host-runtime plugins import. Graph and token-usage assembly is not a bridge primitive — it lives in `pipelex/pipe_run/tracing_assembly.py` and runs the same way for local and distributed runs.---@@ -28,7 +28,7 @@ Pipelex runs pipes either **direct** (single-process) or **distributed** on a hoEvery pipe execution — regardless of mode — follows the same pattern:1. **Setup**: `pipeline_run_setup()` loads the library, resolves the pipe, initializes working memory, and creates a `PipeJob`-2. **Route**: A router (`PipeRouter` or `PipeRouterTop`) receives the PipeJob and dispatches it+2. **Route**: A router (`PipeRouter` for direct mode, or the booted orchestrator's distributed router) receives the PipeJob and dispatches it3. **Execute**: The pipe's `run_pipe()` method runs, potentially resolving and executing child pipesThe `PipeJob` is the universal unit of execution. It carries everything needed to run a single pipe.@@ -49,7 +49,7 @@ The `PipeJob` is the universal unit of execution. It carries everything needed t| `output_name` | `str \| None` | Override for the output variable name || `library_crate` | `LibraryCrate \| None` | Serializable library snapshot for distributed execution |-`PipeJob` is created by `pipeline_run_setup()`, which handles library loading, pipe resolution, working memory initialization, and telemetry setup. For Temporal dispatch, `prepare_for_temporal()` moves `working_memory` to `working_memory_raw` (deferred hydration) and ensures the crate is attached.+`PipeJob` is created by `pipeline_run_setup()`, which handles library loading, pipe resolution, working memory initialization, and telemetry setup. For distributed dispatch, `prepare_job_for_transport(pipe_job)` (in the closed `pipelex-transport` library) moves `working_memory` to `working_memory_raw` (deferred hydration) and ensures the crate is attached.---@@ -83,7 +83,7 @@ These dynamically-generated classes become the `content` type of `Stuff` objects## Direct Execution-In direct execution, everything runs in a single Python process. This is the default mode when Temporal is not enabled.+In direct execution, everything runs in a single Python process. This is the default mode when no orchestrator plugin is booted.### Flow@@ -109,13 +109,15 @@ PipeRouter.run(pipe_job)### Router Selection-The router is selected during `Pipelex.setup()`:+The router is selected during `Pipelex.setup()` by resolving the hub's `PIPE_ROUTER` slot. A boot-orchestrator plugin (e.g. the closed `pipelex-temporal` plugin) claims that slot in its `register()` when `plugins.boot_orchestrator` names it; otherwise the default in-process router is used:```python-if get_config().temporal.is_enabled:- effective_pipe_router = make_tprl_pipe_router_top() # Distributed-else:- effective_pipe_router = PipeRouter(observer=...) # Direct+# A boot-orchestrator plugin swaps in its distributed router when+# plugins.boot_orchestrator == its own name; otherwise this default runs.+effective_pipe_router = self._resolve_hub_slot(+ slot=HubSlot.PIPE_ROUTER,+ default=lambda: PipeRouter(observer=multi_observer), # Direct, in-process+)```### How PipeRouter Works@@ -155,22 +157,21 @@ All controllers follow the same pattern:### Auto-Switching Router-The hub (`get_pipe_router()`) automatically returns the correct router based on context:+The hub (`get_pipe_router()`) returns the router for whichever orchestrator the process is booted under:-- **Direct execution**: Returns `PipeRouter` — child pipes run in-process-- **Distributed execution, outside workflow** (submitter side): Returns `PipeRouterTop` — submits a top-level Temporal workflow-- **Distributed execution, inside workflow** (worker side): Returns `PipeRouterChild` — creates a Temporal child workflow via `execute_child_workflow()`+- **Direct execution**: the in-process `PipeRouter` — child pipes run in the same process.+- **Distributed execution**: the booted host-runtime orchestrator's router, which has claimed the hub's `PIPE_ROUTER` slot. That router auto-detects whether it is dispatching from the **submitter** (start a top-level workflow) or from **inside a running workflow** (start a child workflow). Pipelex's Temporal backend realizes this as `TemporalPipeRouter`, which picks `execute_workflow` vs `execute_child_workflow` accordingly.-This means each child pipe in a controller gets its own Temporal workflow boundary in distributed mode — enabling independent retries, separate worker assignment, and per-pipe visibility in the Temporal UI.+This means each child pipe in a controller gets its own workflow boundary in distributed mode — enabling independent retries, separate worker assignment, and per-pipe visibility in the host runtime's UI.!!! note "Library Dependency"- Controllers depend on the library being loaded in the current process. `get_required_pipe()` queries the library scoped to the current workflow via `ContextVar`, which must have been populated by loading a `LibraryCrate`. In distributed execution, each `WfPipeRouter` loads the crate from the `PipeJob` into a per-workflow `Library` instance. Controllers call `get_required_pipe()` inside Temporal workflow code — accessing per-workflow state via ContextVar. The Temporal sandbox is disabled (`--is-not-sandboxed`) because library loading is a side effect incompatible with replay semantics.+ Controllers depend on the library being loaded in the current process. `get_required_pipe()` queries the library scoped to the current run via `ContextVar`, which must have been populated by loading a `LibraryCrate`. In distributed execution, each worker-side job loads the crate from the `PipeJob` into a per-run `Library` instance before resolving child pipes. (Temporal-specific detail: the backend disables the Temporal sandbox via `--no-sandbox` because library loading is a side effect incompatible with replay semantics.) See [Runtime Bridge & Transport](./runtime-bridge-and-transport.md) for how the crate and working memory cross the boundary.---## Distributed Execution-In distributed execution, the PipeJob is serialized and sent to a Temporal worker for execution.+In distributed execution, the PipeJob is serialized and sent to a host-runtime worker for execution. The concrete dispatch shape below is illustrated with Pipelex's [Temporal backend](https://pipelex.com/products#temporal); the boundary mechanics (LibraryCrate, deferred hydration, per-call isolation) are backend-neutral and documented in [Runtime Bridge & Transport](./runtime-bridge-and-transport.md).### Flow@@ -185,8 +186,8 @@ sequenceDiagramS->>S: Resolve pipe, build LibraryCrateS->>S: Create PipeJob (crate attached)- Note over S: PipeRouterTop.run()- S->>S: prepare_for_temporal()<br/>(WM → working_memory_raw)+ Note over S: TemporalPipeRouter.run() (submitter side)+ S->>S: prepare_job_for_transport()<br/>(WM → working_memory_raw)S->>T: Submit WfPipeRouter(PipeJob)T->>W: Dispatch workflow@@ -209,76 +210,36 @@ sequenceDiagramS->>S: Hydrate PipeOutput<br/>(working_memory_raw → WM)```-### Key Components+### Key components (the backend's role)-**PipeRouterTop** (`pipelex/temporal/tprl_pipe/pipe_router_top.py`)+A host-runtime orchestrator supplies the pieces that turn this generic shape into real workflows. The contract each implements is open core; the concrete classes are backend-specific. In Pipelex's Temporal backend:-Implements `PipeRouterProtocol` like `PipeRouter`, but dispatches via Temporal instead of direct execution. Uses `WorkflowExecutor` to submit `WfPipeRouter` workflows.+- a **router** implementing `PipeRouterProtocol` (`TemporalPipeRouter`) that dispatches via Temporal instead of running in-process;+- a **worker-side job** (a Temporal workflow) that receives a deserialized `PipeJob`, loads the crate, hydrates working memory, and calls `pipe.run_pipe()` — exactly like the direct router;+- a **data converter** that uses Kajson to (de)serialize Pydantic models with subclass types preserved — critical because `PipeJob.pipe` is a `PipeAbstract` subclass and `WorkingMemory` holds `Stuff` objects with concept-specific content classes;+- a **worker entry point** that calls `Pipelex.make(boot_orchestrator="temporal")` and starts the worker with its registered workflows and activities (shipped as the `pipelex-temporal worker` console command).-**WfPipeRouter** (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`)+The concrete modules live in the closed `pipelex-temporal` plugin (`pipelex_temporal/...`), documented in that repo. The open contracts they satisfy — `PipeRouterProtocol`, the boundary DTOs, serialization — are listed in the [Orchestrator SPI](./orchestrator-plugins.md#the-orchestrator-spi).-The Temporal workflow that runs on the worker. Receives a deserialized `PipeJob` and calls `pipe.run_pipe()` — exactly like the direct router.+### Content generation activities-**Kajson Data Converter** (`pipelex/temporal/temporal_data_converter.py`)--Custom Temporal payload converter that uses Kajson for serializing/deserializing Pydantic models. Preserves subclass types during transport, which is critical because `PipeJob.pipe` is a `PipeAbstract` subclass and `WorkingMemory` contains `Stuff` objects with concept-specific content classes.--**Worker CLI** (`pipelex/temporal/worker_cli.py`)--Entry point for the worker process. Calls `Pipelex.make(temporal_enabled=True)` to initialize the framework, then starts the Temporal worker with registered workflows and activities.--### Content Generation Activities--Concrete pipe operators (PipeLLM, PipeCompose, PipeExtract, PipeImgGen) dispatch dedicated Temporal activities for their actual work. Activities are invoked directly from inside `WfPipeRouter` via `ContentGeneratorInWorkflow` (`pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py`), which calls `workflow.execute_activity(act_*, ...)` — there is no intermediate child workflow per content-generation call.--| Activity | Purpose |-|----------|---------|-| `act_llm_gen_text` | LLM text generation |-| `act_llm_gen_object` | LLM structured output (single object) |-| `act_llm_gen_object_list` | LLM structured output (list) |-| `act_jinja2_gen_text` | Jinja2 template rendering |-| `act_extract_gen_extract_pages` | Document extraction (OCR) |-| `act_render_page_views` | PDF page rendering (page-views) |-| `act_img_gen_images` | Image generation |--All content-generation activities resolve dispatch via `worker_config.resolve_dispatch(...)`, so task-queue routing can be configured per activity (and, where applicable, per handle) through `[temporal.worker_config.activity_queues]`. If no routing is configured, activities fall back to the workflow's own task queue.--### Worker Environment--The `@with_conditional_worker` decorator on `PipeRouterTop._run_pipe_job()` supports two environments:--- **EXTERNAL** (production) — assumes a worker is already running, submits the workflow directly-- **INTERNAL** (testing) — spins up an embedded worker for the duration of the execution+Concrete pipe operators (PipeLLM, PipeCompose, PipeExtract, PipeImgGen) dispatch their actual leaf work as host-runtime activities, invoked from inside the worker-side job via the runtime's in-workflow content generator — no intermediate child workflow per content-generation call. A backend can route those activities to different task queues per activity (and, where applicable, per handle); in the Temporal backend that routing is configured under the plugin's own `[worker_config.activity_queues]` config (in the plugin's `temporal.toml`, not core config), falling back to the job's own task queue when unset. The concrete activity set and queue-routing reference live in the `pipelex-temporal` plugin's docs; see also [Content Generation Across Boundaries](./distributed-content-generation.md) for the backend-neutral serialization mechanism.!!! note "Future Improvement: Per-Pipe Routing"- Router selection is currently global and binary: either all pipes go through Temporal, or none do. A simple `PipeCompose` (microseconds of Jinja2 rendering) gets the same Temporal overhead as a `PipeLLM` (minutes of API call time). Per-pipe or per-type routing decisions could improve efficiency.+ Router selection is currently global and binary: either all pipes go through the distributed backend, or none do. A simple `PipeCompose` (microseconds of Jinja2 rendering) gets the same dispatch overhead as a `PipeLLM` (minutes of API call time). Per-pipe or per-type routing decisions could improve efficiency.----## LibraryCrate, Deferred Hydration, and Per-Workflow Isolation--Distributed execution introduces three mechanisms that don't exist in direct mode. These solve the fundamental challenge: the worker is a separate process that doesn't share the submitter's library, class registry, or working memory state.--For a detailed walkthrough of these mechanisms, see [Temporal Integration](./temporal-integration.md).--### LibraryCrate Propagation--The submitter builds a `LibraryCrate` — a serializable snapshot of all pipes and concepts — and attaches it to the `PipeJob`. Each `WfPipeRouter` on the worker loads the crate into a per-workflow scoped library, making all pipes resolvable via `get_required_pipe()`.--### Deferred Hydration (Input and Output)--Deferred hydration applies to both **PipeJob inputs** and **PipeOutput return values**. In both cases, `WorkingMemory` is serialized as a raw JSON dict (`working_memory_raw`) instead of a typed object, avoiding deserialization failures when dynamic concept classes aren't registered in the receiving process's class registry.--- **Input**: `PipeJob.prepare_for_temporal()` moves `working_memory` to `working_memory_raw` before dispatch. The worker hydrates it after loading the crate.-- **Output**: `PipeOutput.prepare_for_temporal()` does the same before returning from a child workflow. The parent (`PipeRouterChild` or `PipeRouterTop`) hydrates after receiving.--### Per-Workflow ClassRegistry and Library Scoping+## Crossing the boundary: crate, hydration, isolation-Each workflow creates its own `ClassRegistry` (pre-seeded from the global registry) and its own `Library` instance, both scoped via `ContextVar`. This ensures concurrent workflows with conflicting concept names (e.g., two bundles that both define `Result` with different fields) don't cross-contaminate.+Distributed execution introduces three mechanisms that don't exist in direct mode, because the worker is a separate process that shares none of the submitter's library, class registry, or working-memory state. They are backend-neutral; for the full walkthrough see [Runtime Bridge & Transport](./runtime-bridge-and-transport.md). In short:-### Known Limitation: StuffArtefact Serialization in Dry-Run+- **LibraryCrate propagation** — the submitter attaches a serializable snapshot of all pipes and concepts to the `PipeJob`; every worker-side job loads it (idempotently, by fingerprint) so `get_required_pipe()` resolves at every level.+- **Deferred hydration** — `WorkingMemory` crosses as a raw JSON dict (`working_memory_raw`) on both `PipeJob` inputs and `PipeOutput` return values, avoiding deserialization failures when a receiving process hasn't registered the bundle's dynamic concept classes. The transport-prep helpers (`prepare_job_for_transport`, `prepare_output_for_transport`) live in the closed `pipelex-transport` library; the raw field and the hydration helper stay in open core.+- **Per-call isolation** — each job gets its own `ClassRegistry` (pre-seeded from the global registry) and its own `Library`, scoped via `ContextVar`, so concurrent jobs with conflicting concept names (e.g., two bundles that both define `Result` with different fields) don't cross-contaminate.-PipeCondition and PipeCompose dispatch the `act_jinja2_gen_text` activity (via `ContentGeneratorInWorkflow.make_templated_text`) to render their expression / construct templates. The activity serializes working memory contents through the Temporal data converter. In dry-run mode, working memory contains `StuffArtefact` debug objects that are not JSON-serializable, causing these internal activities to fail. PipeBatch also fails in dry-run (likely a related serialization issue in child PipeJob creation). PipeSequence and PipeParallel are unaffected — they dispatch child pipes through `SubPipe.run_pipe()` without internal templating activities. See [Temporal Integration](./temporal-integration.md#known-limitation-stuffartefact-serialization-in-dry-run) for details and fix direction.+!!! warning "Known backend limitation: StuffArtefact in dry-run"+ On a host runtime whose data converter serializes working memory for internal templating activities (Temporal's `act_jinja2_gen_text`, dispatched by PipeCondition and PipeCompose), dry-run mode can fail: working memory then holds `StuffArtefact` debug objects that are not JSON-serializable. PipeSequence and PipeParallel are unaffected — they dispatch child pipes without internal templating activities. This is a backend-specific concern; see the `pipelex-temporal` plugin docs for details and fix direction.---@@ -296,7 +257,7 @@ flowchart TBS1 --> S2 --> S3 --> S4 --> S5 --> S6end- S6 --> Decision{temporal.is_enabled?}+ S6 --> Decision{boot_orchestrator set?}subgraph Direct["Direct Execution"]D1["PipeRouter._run_pipe_job()"]@@ -308,12 +269,12 @@ flowchart TBD3 --> D4end- subgraph Distributed["Distributed Execution (Temporal)"]- T1["PipeRouterTop: prepare_for_temporal()"]+ subgraph Distributed["Distributed Execution (Temporal example)"]+ T1["Submitter router: prepare_job_for_transport()"]T2["Kajson serialize PipeJob + crate"]- T3["Temporal Server"]+ T3["Host runtime (Temporal Server)"]T4["Worker: Kajson deserialize"]- T5["WfPipeRouter: load crate, hydrate WM"]+ T5["Worker job: load crate, hydrate WM"]T6["pipe.run_pipe()"]T1 --> T2 --> T3 --> T4 --> T5 --> T6end@@ -335,18 +296,13 @@ flowchart TB| Pipeline setup | `pipelex/pipeline/pipeline_run_setup.py` || PipeRouter (direct) | `pipelex/pipe_run/pipe_router.py` || PipeRouterProtocol | `pipelex/pipe_run/pipe_router_protocol.py` |-| PipeRouterTop (distributed, outside workflow) | `pipelex/temporal/tprl_pipe/pipe_router_top.py` |-| PipeRouterChild (distributed, inside workflow) | `pipelex/temporal/tprl_pipe/pipe_router_child.py` |-| Auto-switch utility | `pipelex/temporal/temporal_workflow_utils.py` |-| WfPipeRouter workflow | `pipelex/temporal/tprl_pipe/wf_pipe_router.py` |-| Kajson data converter | `pipelex/temporal/temporal_data_converter.py` |-| Worker CLI | `pipelex/temporal/worker_cli.py` |+| Distributed router / worker job / data converter / worker CLI | backend-specific — in the host-runtime plugin (e.g. `pipelex_temporal/...`), not open core || Library manager | `pipelex/libraries/library_manager.py` || ConceptFactory | `pipelex/core/concepts/concept_factory.py` || StructureGenerator | `pipelex/core/concepts/structure_generation/generator.py` || LibraryCrate model | `pipelex/libraries/library_crate.py` || LibraryCrate factory | `pipelex/libraries/library_crate_factory.py` |-| Deferred hydration utility | `pipelex/temporal/tprl_pipe/hydration.py` |+| Deferred hydration utility (open core) | `pipelex/runtime_bridge/primitives/hydration.py` || Hub (get_required_pipe, get_class_registry) | `pipelex/hub.py` || PipeSequence | `pipelex/pipe_controllers/sequence/pipe_sequence.py` || PipeCondition | `pipelex/pipe_controllers/condition/pipe_condition.py` |diff --git a/docs/under-the-hood/runtime-bridge-and-transport.md b/docs/under-the-hood/runtime-bridge-and-transport.mdnew file mode 100644index 000000000..ebf90afb3--- /dev/null+++ b/docs/under-the-hood/runtime-bridge-and-transport.md@@ -0,0 +1,154 @@+---+title: "Runtime Bridge & Transport"+description: "The backend-neutral seam that carries a pipe run across process boundaries — boundary DTOs, LibraryCrate propagation, deferred hydration, and per-call registry/library isolation — and where open core ends and the closed pipelex-transport layer begins."+---++# Runtime Bridge & Transport++This page is for contributors working on Pipelex internals. For the capability overview, see the user-facing [Distributed Execution](../distributed-execution/index.md) page instead.++Pipelex runs a pipe either **direct** (everything in one Python process) or **distributed** on a host runtime (a separate worker process, potentially on another machine). Both paths reach pipe execution through the same framework-agnostic **runtime bridge**. This page documents that bridge backend-neutrally: the boundary data types, how a library snapshot and working memory cross a process boundary, and how concurrent runs stay isolated. The concrete realizations — Temporal-backed durable execution and the Mistral Workflows integration — are commercial Pipelex platform capabilities; their backend-specific topology lives in their own plugin repos.++!!! info "A commercial capability rides on this seam"+ Embedding a Pipelex pipe run into a host runtime — so it executes as durable, distributed workflows — is part of Pipelex's [workflow-orchestration offer](https://pipelex.com/products#durable-execution). This page documents the **open seam** and the **open boundary types**; the cross-process plumbing that drives them lives in the closed `pipelex-transport` library (see [Where open core ends](#where-open-core-ends-and-pipelex-transport-begins) below). For the orchestrator extension point a third-party backend compiles against, see [Orchestrator Plugins](./orchestrator-plugins.md).++---++## Direct vs distributed, one bridge++| Mode | Where pipes run | How the PipeJob travels |+|------|-----------------|--------------------------|+| **Direct** | All pipes in the same process. Library, class registry, and pipe resolution are shared in-memory. | Stays in-process; no serialization. |+| **Distributed** | The PipeJob is serialized and sent to a worker — a separate process. | Crosses the boundary as structured JSON, carrying a serializable library snapshot. |++Both modes share the same pipe definitions, library loading, and controller logic. The only difference is *where* a pipe runs and how its `PipeJob` gets there. Distributing execution across separate processes introduces problems that don't exist in-process — those problems, and their solutions, are this page's subject. For the routing layer that sits above the bridge (how a controller dispatches its child pipes in each mode), see [Pipe Routing & Execution](./pipe-routing-and-execution.md).++---++## The boundary data types++Three open-core Pydantic models are everything that crosses a process boundary.++### PipeJob — the unit of execution++`PipeJob` (`pipelex/pipe_run/pipe_job.py`) carries everything needed to run a single pipe:++| Field | Purpose |+|-------|---------|+| `pipe` | The resolved pipe object (concrete operator or controller). |+| `working_memory` | Runtime data store — typed `Stuff` objects keyed by variable name. |+| `working_memory_raw` | Raw JSON dict of working memory for deferred hydration (cross-process only). |+| `pipe_run_params` | Execution config: run mode (LIVE/DRY), output multiplicity, pipe stack for cycle detection. |+| `job_metadata` | Pipeline run ID, user ID, tracing context. |+| `library_crate` | Serializable library snapshot for distributed execution. |++### PipeOutput — the result that comes back++`PipeOutput` (`pipelex/core/pipes/pipe_output.py`) carries the run result. Like `PipeJob`, it has both a typed `working_memory` and a `working_memory_raw` dict — the raw field is how a result travels back across a boundary without requiring class resolution at every hop (see [Deferred hydration](#deferred-hydration)).++### LibraryCrate — a serializable library snapshot++A `LibraryCrate` (`pipelex/libraries/library_crate.py`) is a flat, serializable snapshot of the library — the pipes and concepts a worker needs to run the job:++```python+class LibraryCrate(BaseModel):+ concepts: dict[str, ConceptBlueprint | str] # concept ref -> blueprint or description+ pipes: dict[str, PipeBlueprintUnion] # pipe ref -> blueprint+ domains: dict[str, DomainBlueprint] # domain code -> domain metadata+ source_map: dict[str, str] # ref -> source file (error traceability)+ fingerprint: str # SHA256 of serialized content+```++Domain is encoded in the dictionary keys (e.g., `scoring.WeightedScore`, `scoring.compute_score`), not in a structural container. `source_map` lets error messages trace back to origin files. The `fingerprint` enables idempotent loading — a worker that already loaded a crate with the same fingerprint skips the reload. The crate is a plain Pydantic model, so a host runtime serializes it as structured JSON in the job input — fully inspectable, not opaque bytes.++---++## The three cross-process challenges++Sending a pipe run to a separate worker process introduces three challenges that single-process execution never faces:++1. **Pipe resolution** — Controllers call `get_required_pipe()` to resolve child pipes. The worker's library must contain those pipes. → solved by **LibraryCrate propagation**.+2. **Dynamic class deserialization** — `.mthds` bundles generate Python classes at runtime (e.g., `RawText` inheriting from `TextContent`). A receiving process must have those classes registered before it can deserialize `WorkingMemory`. → solved by **deferred hydration**.+3. **Concurrent isolation** — One worker may run many jobs at once. Two jobs that each define a concept named `Result` with different structures must not share a class registry or library. → solved by **per-call scoping**.++---++## LibraryCrate propagation++The submitter builds a `LibraryCrate` from its loaded library (`pipelex/libraries/library_crate_factory.py`) and attaches it to the `PipeJob`. On the worker, the job's entry point loads the crate into a scoped library, registering the bundle's dynamic concept classes, so `get_required_pipe()` resolves every pipe at every level of nesting. Because a distributed run can fan out into nested jobs (a controller's child pipes), every nested job carries the same crate; loading is idempotent via the fingerprint, so re-receiving a crate already loaded on a worker is a no-op.++---++## Deferred hydration++### The chicken-and-egg problem++A host runtime's data converter deserializes a payload (a job input, a return value) before any bundle code has run — but reconstructing a typed `WorkingMemory` needs the bundle's dynamic concept classes registered, and those only get registered when the `LibraryCrate` is loaded *inside* the job. Two deserialization gaps follow:++1. **Input** — the `PipeJob` input contains `WorkingMemory` whose dynamic classes don't exist yet on the worker.+2. **Output** — a nested job's `PipeOutput` return value contains `WorkingMemory` with dynamic classes, and the parent's data converter typically runs outside the job's scope, so it can't reach the per-job class registry.++### The solution: carry working memory raw++Both `PipeJob` and `PipeOutput` carry a `working_memory_raw` field — a plain JSON dict — alongside the typed `working_memory`. The raw dict needs no class resolution to cross a boundary, which decouples transport from class registration.++- **Input dehydration** — before dispatch, the working memory is moved into `working_memory_raw` (a plain dict produced by `WorkingMemory.dump_for_transport()`, which stays in open core at `pipelex/core/memory/working_memory.py`). After the worker loads the crate, the job's entry point hydrates the raw dict back to a typed `WorkingMemory`, inside the job's scope.+- **Output dehydration** — before a nested job returns, its working memory is dehydrated to `working_memory_raw`. The raw dict then flows **end-to-end** — nested job → parent → delivery — without intermediate rehydration, because parents and delivery workers generally have not loaded the crate. Only the **top-level submitter** rehydrates.++The worker-side hydration helper (`pipelex/runtime_bridge/primitives/hydration.py`) is open core: it iterates the raw dict, uses `StuffContentFactory` to rebuild typed content from the registered classes, and preserves aliases. It also guards against cross-exec class-identity mismatches (`_validate_as_known_class` round-trips items through `model_dump()`): the dynamic classes a worker re-execs from the crate are new Python identities sharing a name with whatever a data converter eagerly rebuilt, and a naive `model_validate` would reject the older instance.++### Boundary-by-boundary (backend-neutral)++| Boundary | Direction | What crosses | Hydrate on receive? |+|----------|-----------|--------------|----------------------|+| Submitter → top worker job | input | `PipeJob` (`working_memory_raw` + `library_crate`) | Yes — worker loads crate, then hydrates in scope |+| Parent job → nested job | input | Same `PipeJob` shape (crate re-loaded idempotently) | Yes — in the nested job's own scope |+| Nested job → parent | output | `PipeOutput` (`working_memory_raw`) | **No** — parent forwards the raw dict |+| Parent job → delivery | result arg | `PipeOutput` (`working_memory_raw`, no crate) | **No** — delivery renders from raw, best-effort local hydration of the main stuff using only globally registered classes |+| Top worker job → submitter | output | `PipeOutput` (`working_memory_raw`) | Yes — the submitter rehydrates, opening a fresh scoped library if a crate is available |++Because delivery renders from the raw dict (`pipelex/pipe_run/delivery_executor.py` branches on which field is populated), a delivery worker never needs the crate loaded — which is the precondition for true distributed execution. Dynamic concepts lose typed rendering on the delivery worker (a generic field-walking fallback still produces readable JSON/HTML), but the worker stays crate-free.++!!! note "Where the transport prep lives"+ Moving working memory into `working_memory_raw` for a job (`prepare_job_for_transport`), dehydrating a result before return (`prepare_output_for_transport`), and the submitter-side rehydration are part of the **closed `pipelex-transport`** library — they are the cross-process plumbing. The *raw field itself*, `dump_for_transport`, and the worker-side hydration helper stay **open core**, because they are host-agnostic and are also exercised by the open `pipelex-api` runner.++---++## Per-call scoping++A single worker may process many concurrent jobs. If two jobs define a concept named `Result` with different fields, they must not share a `ClassRegistry` or `Library` — otherwise the second registration overwrites the first and silently corrupts data. Each job therefore creates its own scoped state:++1. **ClassRegistry** — a fresh `ClassRegistry` pre-seeded from the global registry (which holds the base classes from `PIPELEXPATH`). The crate's dynamic classes register here, never in the global registry.+2. **Library** — a fresh `Library` (`library_manager.open_fresh_library()`) with that `ClassRegistry` attached, set as current via a `ContextVar` keyed by the run id. Run-id keying (not job/workflow id) is deliberate: a reused workflow id across retries/resets could otherwise let a closed predecessor's cleanup tear down a live successor's library on the same worker.+3. **Lookup chain** — `hub.get_class_registry()` reads the `ContextVar`, returns the scoped library's registry, and falls back to the global registry when no library is set (e.g. a data converter running outside the job scope — which is exactly why output uses deferred hydration).+4. **Cleanup** — `library_manager.teardown(library_id)` drops the library and its registry; the `ContextVar` is reset. No manual GC needed.++The hub's class-registry and library scoping (`pipelex/hub.py`, `pipelex/libraries/library.py`, `pipelex/libraries/library_manager.py`) is open core — the same machinery serves direct execution, the open runner, and any host-runtime plugin.++---++## Where open core ends and `pipelex-transport` begins++The boundary types and the host-agnostic helpers are **open** (MIT, in the `pipelex` distribution). The cross-process *plumbing* that drives a pipe run across the boundary is **closed** (the proprietary `pipelex-transport` library, installed via git+ssh, pinned to an exact core rev). Embedding Pipelex into a host runtime is a commercial capability, so the entry point and dispatch primitives are not an open extension seam.++| Concern | Open core (`pipelex`) | Closed (`pipelex-transport`) |+|---|---|---|+| Boundary DTOs | `PipeJob`, `PipeOutput`, `LibraryCrate`, `runtime_bridge/payloads.py` | — |+| Serialization | `runtime_bridge/serialization.py` | — |+| Mode / delivery model | `orchestration_mode`, `DeliveryMode`, the orchestrator registry / SPI | — |+| Working-memory raw + dump | `working_memory_raw`, `WorkingMemory.dump_for_transport` | — |+| Worker-side hydration | `runtime_bridge/primitives/hydration.py` | — |+| Bridge entry point | — | `run_pipe_via_bridge`, `build_pipe_job_from_input` |+| Transport prep | — | `prepare_job_for_transport`, `prepare_output_for_transport` |+| Submitter rehydration, delivery, trace flush, scoped-library, pipe classification, submit-arg envelope | — | the dispatch primitives |++A third-party orchestrator compiles against the **open** rows only (the [Orchestrator SPI](./orchestrator-plugins.md#the-orchestrator-spi)); Pipelex's own host-runtime plugins (`pipelex-temporal`, `pipelex-mistralai-workflows`) additionally compile against `pipelex-transport`.++---++## Next steps++- [Orchestrator Plugins](./orchestrator-plugins.md) — the orchestration-mode seam, the Orchestrator SPI, and the open/closed boundary in contract terms.+- [Pipe Routing & Execution](./pipe-routing-and-execution.md) — how PipeJobs are created and routed in direct and distributed modes.+- [Content Generation Across Boundaries](./distributed-content-generation.md) — how dynamic Pydantic models and large binary payloads cross a worker boundary.+- [Distributed Execution](../distributed-execution/index.md) — the user-facing capability overview, and the [workflow-orchestration offer](https://pipelex.com/products#durable-execution) (Temporal-backed durable execution, the Mistral Workflows integration).diff --git a/docs/under-the-hood/temporal-integration.md b/docs/under-the-hood/temporal-integration.mddeleted file mode 100644index 1e451819a..000000000--- a/docs/under-the-hood/temporal-integration.md+++ /dev/null@@ -1,276 +0,0 @@-----title: "Temporal Integration"-description: "How Pipelex distributes pipe execution across Temporal workers — LibraryCrate propagation, deferred hydration, per-workflow isolation, and the serialization chain."------# Temporal Integration--This page is for contributors working on Pipelex internals. If you're deploying or operating Pipelex on Temporal, see the user-facing [Distributed Execution with Temporal](../distributed-execution/temporal/index.md) guide instead.--This page covers the Temporal-specific mechanisms that enable distributed pipe execution. For the general pipe routing architecture (shared between direct and distributed modes), see [Pipe Routing & Execution](./pipe-routing-and-execution.md).-------## The Three Challenges--Distributing pipe execution across separate worker processes introduces three challenges that don't exist in direct (single-process) mode:--1. **Pipe resolution** — Controllers call `get_required_pipe()` to resolve child pipes. The worker's library must contain those pipes.-2. **Dynamic class deserialization** — `.mthds` bundles generate Python classes at runtime (e.g., `RawText` inheriting from `TextContent`). The worker must have those classes registered before deserializing `WorkingMemory`.-3. **Concurrent isolation** — Multiple workflows running on the same worker may define conflicting class/pipe names. Each workflow needs its own scoped registry and library.--These are solved by three mechanisms: LibraryCrate propagation, deferred hydration, and per-workflow scoping.-------## LibraryCrate--A `LibraryCrate` is a serializable snapshot of the library — two flat dicts of blueprints:--```python-class LibraryCrate(BaseModel):- concepts: dict[str, ConceptBlueprint | str] # concept_ref -> blueprint or description- pipes: dict[str, PipeBlueprintUnion] # pipe_ref -> blueprint- domains: dict[str, DomainBlueprint] # domain_code -> domain metadata- source_map: dict[str, str] # ref -> source file (error traceability)- fingerprint: str # SHA256 of serialized content-```--Domain is encoded in the dictionary keys (e.g., `scoring.WeightedScore`, `scoring.compute_score`), not in a structural container. `domains` carries per-domain metadata (description, system prompt). `source_map` enables error messages that trace back to origin files. The `fingerprint` enables idempotent loading — if a worker already loaded a crate with the same fingerprint, it skips the reload.--### How it propagates--```mermaid-sequenceDiagram- participant S as Submitter- participant W as Worker-- S->>S: Load library, build LibraryCrate- S->>W: PipeJob (crate attached)-- Note over W: WfPipeRouter.run()- W->>W: load_from_crate(crate)- W->>W: pipe.run_pipe()-- Note over W: Controller dispatches child- W->>W: PipeRouterChild → child workflow- Note over W: Child WfPipeRouter.run()- W->>W: load_from_crate(same crate)- W->>W: child pipe.run_pipe()-- W->>S: PipeOutput (dehydrated)- S->>S: Hydrate PipeOutput-```--Every child workflow receives the crate through its `PipeJob`. Each `WfPipeRouter` loads the crate into its own scoped library. This ensures every workflow — parent or child, on any worker — can resolve all pipes.--### Visibility--The `LibraryCrate` is a Pydantic model. Temporal serializes it as structured JSON in the workflow input, making it fully visible in the Temporal dashboard — not opaque bytes.-------## Deferred Hydration--### The chicken-and-egg problem--Temporal's data converter deserializes payloads (workflow inputs and return values) using Kajson, which needs dynamic concept classes registered to reconstruct typed `WorkingMemory`. But those classes only get registered when the `LibraryCrate` is loaded inside the workflow. This creates two deserialization gaps:--1. **Input**: The `PipeJob` workflow input contains `WorkingMemory` with dynamic classes that don't exist yet on the worker.-2. **Output**: A child workflow's `PipeOutput` return value contains `WorkingMemory` with dynamic classes. The parent's data converter runs in the Temporal SDK's event processing context (outside the workflow coroutine's `ContextVar` scope), so it can't access the parent's per-workflow class registry.--### The solution--Both `PipeJob` and `PipeOutput` carry a `working_memory_raw` field for deferred hydration:--```python-class PipeJob(BaseModel):- working_memory: WorkingMemory | None = None- working_memory_raw: dict[str, Any] | None = None- library_crate: LibraryCrate | None = None--class PipeOutput(PipeOutputAbstract[WorkingMemory]):- working_memory: WorkingMemory = Field(default_factory=WorkingMemory)- working_memory_raw: dict[str, Any] | None = None-```--**Input dehydration/hydration cycle:**--- Before dispatch, `PipeJob.prepare_for_temporal()` moves `working_memory` to `working_memory_raw` (a plain dict via `dump_for_temporal()`).-- After the worker loads the crate, `WfPipeRouter` hydrates the raw dict back to typed `WorkingMemory`.--**Output dehydration/hydration cycle:**--- Before returning from a workflow, `WfPipeRouter` calls `PipeOutput.prepare_for_temporal()` to dehydrate `WorkingMemory` to `working_memory_raw`.-- The raw dict flows **end-to-end**: child workflow → parent `WfPipeRun` → `act_deliver` activity, all without intermediate rehydration. Parents do not own a per-workflow registry that would let them rehydrate the child's output, and the activity worker is typically a different process that has not loaded the crate.-- Only the **top-level submitter** rehydrates, via `rehydrate_pipe_output_with_crate()` (`pipelex/temporal/tprl_pipe/submitter_hydration.py`). When the submitter has not loaded the bundle locally (e.g., a remote API client that built the `PipeJob` from a serialized `LibraryCrate`), this function opens a fresh per-call scoped Library, loads the crate into a scoped `ClassRegistry` pre-seeded from the global, hydrates inside the scope, then tears down — leaving the submitter's global registry untouched.--The hydration function (`pipelex/temporal/tprl_pipe/hydration.py`) iterates the raw dict, uses `StuffContentFactory` to reconstruct typed content objects using the registered classes, and preserves aliases. It also includes a small but unintuitive guard (`_validate_as_known_class`) that round-trips items through `model_dump()` to defeat cross-exec class-identity mismatches: kajson may eagerly rebuild instances using one class identity, then `load_from_crate` re-execs the source and replaces the registry entry with a new identity (same name, different `id()`), which would otherwise cause Pydantic's `model_validate` to reject the older instance.--### Why output dehydration goes end-to-end--Earlier designs had the parent `WfPipeRun` rehydrate the child's `PipeOutput` before scheduling `act_deliver`, then propagated dynamic classes from the per-workflow registry into the worker's global `KajsonManager` registry so the activity worker could decode the typed payload. This worked on a single worker but broke under distributed Temporal: any worker polling the task queue can pick up the activity, and `ContextVars` / per-worker globals do not cross processes. The current design eliminates that hack — `act_deliver` receives the raw dict directly, never needs class resolution for transport, and the activity worker does not need the crate loaded.--### Activity-side rendering of the raw working memory--`act_deliver` cannot assume dynamic concept classes are registered on its worker. `DeliveryExecutor.generate_result_files()` (`pipelex/pipe_run/delivery_executor.py`) therefore branches on which field is populated:--- **Typed path** (`working_memory` set, in-process / same-worker): existing typed renderers run unchanged.-- **Raw path** (`working_memory_raw` set, cross-process Temporal): `working_memory.json` is written directly from the raw dict; for the main stuff, `try_local_hydrate_stuff()` attempts a single-Stuff hydration using only globally registered classes (built-in `StuffContent` subclasses are always present). On success, typed renderers produce `main_stuff.{md,html}` and the viewer; on miss (dynamic concept class not registered locally), a generic field-walking fallback renders JSON-in-markdown and `<pre>`-in-HTML so users still get readable output. Misses log a warning so silent regressions on built-in hydration surface.--Tradeoff: dynamic concepts lose typed rendering on the activity worker, but the activity worker no longer needs the crate, which is the precondition for true distributed execution.--### Boundary-by-boundary summary--| Boundary | Direction | What crosses | Hydration on receive? |-|----------|-----------|--------------|------------------------|-| Submitter → top `WfPipeRouter` | input | `PipeJob` with `working_memory_raw` + `library_crate` | Yes — worker loads crate, then hydrates inside scope |-| Parent `WfPipeRouter` → child `WfPipeRouter` | input | Same `PipeJob` shape (crate re-loaded idempotently via fingerprint) | Yes — child worker loads crate, hydrates inside its own scope |-| Child `WfPipeRouter` → parent `WfPipeRun` | output | `PipeOutput` with `working_memory_raw` | **No** — parent forwards raw dict |-| Parent `WfPipeRun` → `act_deliver` | activity arg | `PipeOutput` with `working_memory_raw` (no crate) | **No** — activity renders from raw, with best-effort local hydration of the main stuff using only globally registered classes |-| Top `WfPipeRouter` → submitter | output | `PipeOutput` with `working_memory_raw` | Yes — `rehydrate_pipe_output_with_crate()` opens a fresh scoped Library if a crate is available, otherwise falls back to the global registry |--### Dashboard visibility--`working_memory_raw` is a plain JSON dict — fully readable in the Temporal dashboard without needing class resolution.-------## Per-Workflow Scoping--### Why per-workflow isolation matters--A single Temporal worker may process multiple concurrent workflows. If two workflows define a concept named `Result` with different structures (e.g., `Result(score, label)` vs `Result(value, confidence, is_valid)`), they must not share the same `ClassRegistry` or `Library`. Without isolation, the second workflow's class registration would overwrite the first's, causing silent data corruption.--### How it works--Each `WfPipeRouter.run()` creates its own scoped state:--1. **ClassRegistry**: A new `ClassRegistry` pre-seeded from the global registry (which has base classes from `PIPELEXPATH`). Dynamic classes from the crate are registered here — not in the global registry.--2. **Library**: A new `Library` instance opened via `library_manager.open_fresh_library()`, with the `ClassRegistry` attached as a `PrivateAttr`. The library is set as the current library via the `_library_id` `ContextVar`. The library id is keyed by the run id (`wf_{run_id}`) — replay-stable but unique per run. Keying by workflow id would collide across runs of a reused workflow id (workflow-level retry policy, Temporal reset, resubmission of the same `pipeline_run_id`): a closed predecessor run's late eviction cleanup could tear down a live successor run's library on the same worker. With run-id keying, a pre-existing library under the id can only be this same run's own leftover (an interrupted execution whose cleanup never ran), so `open_fresh_library` tears it down before opening: reusing it would fingerprint-skip the crate load against the fresh registry — the crate's dynamic classes would never land in it. The stale teardown is best-effort: the manager forgets the entry pop-first, and a raising teardown is logged and never fails the fresh open (worker-local leak state must not decide setup success). The same run-id keying applies to the rest of the per-run worker-local state: the graph tracer key and the report-delegate event-log context key.--3. **ClassRegistry lookup chain**: `hub.get_class_registry()` reads `_library_id` from the `ContextVar`, gets the library's attached `ClassRegistry`. Falls back to the global registry if no library is set.--4. **Cleanup**: `library_manager.teardown(library_id)` deletes the library and its `ClassRegistry`. The `ContextVar` is reset via `clear_current_library()`. No manual GC needed — the `ClassRegistry` is garbage-collected with the `Library`. In the workflow's `finally` block, this worker-local cleanup runs BEFORE the awaited `act_flush_trace_events` activity: the await is a suspension point where an eviction-time `BaseException` can abort the rest of the block, so ordering cleanup first guarantees an eviction can only skip the flush itself, never the teardown.--### Kajson integration--The Temporal data converter passes the active `ClassRegistry` to `kajson.loads()`:--```python-pydantic_gizmo = kajson.loads(data, class_registry=get_class_registry())-```--Inside a workflow coroutine, `get_class_registry()` returns the per-workflow scoped registry (via `_library_id` ContextVar). However, the data converter is also called from the Temporal SDK's event processing context (e.g., when deserializing child workflow return values), where the ContextVar is not set. In that case, it falls back to the global registry. This is why PipeOutput uses deferred hydration — the `working_memory_raw` dict doesn't require class resolution during deserialization.-------## Workflow Topology--A single pipe execution can create many Temporal workflows. Each level of nesting dispatches child workflows:--```-PipeSequence (3 steps)- → 1 parent WfPipeRouter- → 3 child WfPipeRouters (one per step)- Each child step (if PipeLLM):- → 1 act_llm_gen_text activity (dispatched directly from WfPipeRouter- via ContentGeneratorInWorkflow.make_llm_text)--PipeParallel (2 branches, then summary)- → 1 parent WfPipeRouter- → 3 child WfPipeRouters (2 branches concurrent + 1 summary)--PipeBatch (3 items)- → 1 parent WfPipeRouter- → 1 child WfPipeRouter for the PipeBatch controller- → 3 child WfPipeRouters (one per item)-```--Each `WfPipeRouter` loads the `LibraryCrate` (idempotent via fingerprint), so pipe resolution works at every level.-------## Controller Dispatch Patterns on Temporal--Each controller type creates a different child workflow pattern:--| Controller | Dispatch pattern | Child workflows |-|------------|-----------------|----------------|-| **PipeSequence** | Sequential: each step via `SubPipe.run_pipe()` → `PipeRouterChild` | N children (one per step), executed sequentially |-| **PipeParallel** | Concurrent: all branches via `SubPipe.run_pipe()` + `asyncio.gather()` | N children (one per branch), executed concurrently |-| **PipeBatch** | Fan-out: each list item via `SubPipe.run_pipe()` | N children (one per item), executed concurrently |-| **PipeCondition** | Conditional: evaluates expression, dispatches selected outcome via `SubPipe.run_pipe()` | 1 child (the chosen outcome pipe), plus an internal `act_jinja2_gen_text` activity for expression evaluation |--### Internal templating activities--Some controllers dispatch internal `act_jinja2_gen_text` activities in addition to pipe routing:--- **PipeCondition**: `act_jinja2_gen_text` to evaluate the expression template against working memory-- **PipeCompose**: `act_jinja2_gen_text` to resolve construct field references (e.g., `{ from = "title_text.text" }`)--Both call `ContentGeneratorInWorkflow.make_templated_text` from inside `WfPipeRouter`, which dispatches the activity directly via `workflow.execute_activity(act_jinja2_gen_text, ...)`. These activities carry working memory contents through the Temporal data converter, which creates a serialization dependency on working memory content types.-------## Validation Dispatch: One In-Process Activity--Pipeline *execution* fans out across workflows and activities as described above — but *validation* deliberately does the opposite. When Temporal is enabled, a `/validate` job (validation sweep + graph-producing dry-run) dispatches as the one-step wrapper workflow `wf_dry_validate`, which runs a single `act_dry_validate` activity and returns everything the canonical validation report needs that must be computed worker-side, in one round-trip: the per-pipe status map, the best-effort `GraphSpec`, the library-wide `pending_signatures`, and the `pipe_io_contracts` IO contracts (built inside the worker's library window — their JSON-Schema rendering resolves bundle-defined structure classes through the loaded library's class registry, so the API side never needs to re-acquire a library).--Inside the activity everything is in-process and in-memory:--- The library is loaded once: the sweep half is `validate_bundle` itself (the same function the direct-mode route calls, so both backends share the categorized `ValidateBundleError` contract), which leaves the library loaded; the graph dry-run (`dry_run_pipe_in_process`) runs against that same library; teardown happens once in the activity's `finally`.-- ContextVar scopes pin the run in-process: `scoped_pipe_router` (nested controller sub-pipes resolve a local router, not the `TemporalPipeRouter`), `scoped_content_generator` (inference leaves resolve an inline dry generator, not `ContentGeneratorInWorkflow`), and `scoped_event_log` (the graph traces into a shared `InMemoryEventLog` — no NDJSON file, no DynamoDB round-trip; the `GraphSpec` rides back on the activity result). No usage/cost events are emitted.-- The graph is best-effort: an expected dry-run failure (the run-failure wrappers, or a mock-input mint failure) returns `graph_spec=None` with validation still successful; any other error propagates and fails the activity.-- Validation failures cross the boundary as structured `ErrorReport`s via the activity error boundary — a `ValidateBundleError` keeps its categorized per-error data (blueprint/factory/pipe/dry-run), so the API can render meaningful diagnostics. Signatures are not a failure: an unsatisfied `PipeSignature` is a runnability fact carried on the report's `pending_signatures` / `is_runnable`, not an error that crosses the boundary.--Submitters call `dispatch_dry_validate` (`pipelex/temporal/tprl_pipe/dry_validate_dispatch.py`) for the whole round-trip. The wrapper workflow exists so the dispatch works on the current `temporalio` SDK; replacing it with a true standalone activity is a deferred optimization.-------## Known Limitation: StuffArtefact Serialization in Dry-Run--In dry-run mode, PipeLLM produces `StuffArtefact` debug objects as working memory content. When controllers dispatch internal templating activities that carry working memory contents through the Temporal data converter, Kajson fails with `TypeError: Type <class 'StuffArtefact'> is not JSON serializable`.--`StuffArtefact` is not a Pydantic model — it's a plain object used for dry-run tracing that was never designed for cross-process serialization.--**Confirmed affected**:--- **PipeCondition**: expression evaluation dispatches `act_jinja2_gen_text`, which serializes working memory to evaluate the Jinja2 expression. Fails with `StuffArtefact` TypeError.-- **PipeCompose**: construct field resolution dispatches `act_jinja2_gen_text` similarly.--**Likely affected** (hangs in testing, root cause not fully diagnosed):--- **PipeBatch**: fan-out creates child PipeJobs with copies of working memory. The hang may be caused by `StuffArtefact` in the serialized child PipeJob, or by a different serialization issue in the list content extraction.--**Not affected**:--- **PipeSequence**: no internal templating activities — child pipes are dispatched via `SubPipe.run_pipe()` → `PipeRouterChild`, which creates clean child PipeJobs.-- **PipeParallel**: same dispatch pattern as PipeSequence — branches go through `SubPipe.run_pipe()` without internal templating activities.-- **All controllers in live mode**: `StuffArtefact` is only produced in dry-run. Live execution uses real content objects that serialize correctly.--**Fix direction**: Either make `StuffArtefact` serializable (implement `__json__` or convert to Pydantic), or strip non-serializable objects from working memory before passing to internal templating activities.-------## File Reference--| Component | File |-|-----------|------|-| LibraryCrate model | `pipelex/libraries/library_crate.py` |-| LibraryCrate factory | `pipelex/libraries/library_crate_factory.py` |-| PipeJob (crate + raw WM fields) | `pipelex/pipe_run/pipe_job.py` |-| PipeOutput (raw WM for deferred hydration) | `pipelex/core/pipes/pipe_output.py` |-| PipeRouterTop (submitter-side dispatch) | `pipelex/temporal/tprl_pipe/pipe_router_top.py` |-| PipeRouterChild (worker-side child dispatch) | `pipelex/temporal/tprl_pipe/pipe_router_child.py` |-| WfPipeRouter (workflow entry point) | `pipelex/temporal/tprl_pipe/wf_pipe_router.py` |-| Deferred hydration (worker-side) | `pipelex/temporal/tprl_pipe/hydration.py` |-| Submitter-side rehydration | `pipelex/temporal/tprl_pipe/submitter_hydration.py` |-| Activity-side delivery rendering (typed/raw dual path) | `pipelex/pipe_run/delivery_executor.py` |-| WorkingMemory dehydration helper | `pipelex/core/memory/working_memory.py` (`dump_for_temporal`) |-| Kajson data converter | `pipelex/temporal/temporal_data_converter.py` |-| Hub (ClassRegistry + Library scoping) | `pipelex/hub.py` |-| Library (PrivateAttr ClassRegistry) | `pipelex/libraries/library.py` |-| Worker CLI | `pipelex/temporal/worker_cli.py` |-| Temporal task manager | `pipelex/temporal/temporal_task_manager.py` |diff --git a/mkdocs.yml b/mkdocs.ymlindex 382b4c0c3..11315311c 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -58,7 +58,13 @@ plugins:separator: '[\s\-\.]+'- redirects:redirect_maps:- 'under-the-hood/temporal-deployment.md': 'distributed-execution/cluster-setup.md'+ 'under-the-hood/temporal-deployment.md': 'distributed-execution/index.md'+ 'distributed-execution/temporal/index.md': 'distributed-execution/index.md'+ 'distributed-execution/cluster-setup.md': 'distributed-execution/index.md'+ 'distributed-execution/workers.md': 'distributed-execution/index.md'+ 'distributed-execution/task-routing.md': 'distributed-execution/index.md'+ 'distributed-execution/observability.md': 'distributed-execution/index.md'+ 'under-the-hood/temporal-integration.md': 'under-the-hood/runtime-bridge-and-transport.md''home/1-features/advanced-customizations.md': 'features/advanced-customizations.md''home/1-features/claude-code-skills-plugin.md': 'features/claude-code-skills-plugin.md''home/1-features/cli.md': 'features/cli.md'@@ -310,10 +316,11 @@ plugins:- under-the-hood/test-profile-configuration.md: "Test Profile Configuration"- under-the-hood/init-cli-flows.md: "Init CLI Flows"- under-the-hood/pipe-routing-and-execution.md: "Pipe Routing & Execution"+ - under-the-hood/inference-backend-plugins.md: "Inference Backend Plugins"+ - under-the-hood/orchestrator-plugins.md: "Orchestrator Plugins"- under-the-hood/error-model.md: "Error Model"- - under-the-hood/temporal-integration.md: "Temporal Integration"- - under-the-hood/temporal-deployment.md: "Temporal Deployment"- - under-the-hood/distributed-content-generation.md: "Distributed Content Generation"+ - under-the-hood/runtime-bridge-and-transport.md: "Runtime Bridge & Transport"+ - under-the-hood/distributed-content-generation.md: "Content Generation Across Boundaries"Project:- contributing.md: "Contributing"- contribute/configuration-defaults-and-overrides.md: "Configuration Internals"@@ -368,16 +375,6 @@ not_in_nav: |!errors/platform-and-tooling.mdagents/*.md-# exclude_docs: forward-looking pages kept in the repo but NOT built or served on-# the live site yet. The Mistral Workflows backend ships as a separate package-# (pipelex-mistralai-workflows) that is not published yet, so its user docs are-# held back to avoid advertising an unreleased package. Re-publish them — delete-# this block and restore the "Mistral Workflows" nav entries under Distributed-# Execution — once the package lands. See-# wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md.-exclude_docs: |- distributed-execution/mistral-workflows/-nav:- Get Started:- Build & Run AI Methods: index.md@@ -485,14 +482,7 @@ nav:- Automatic Retries: reliability/automatic-retries.md- Failure Classification: reliability/failure-classification.md- Durable Execution: reliability/durable-execution.md- - Distributed Execution:- - Overview: distributed-execution/index.md- - Temporal:- - Overview: distributed-execution/temporal/index.md- - Cluster Setup: distributed-execution/cluster-setup.md- - Worker Deployment: distributed-execution/workers.md- - Task-Queue Routing: distributed-execution/task-routing.md- - Workflow Observability: distributed-execution/observability.md+ - Distributed Execution: distributed-execution/index.md- Reference:- Gateway Available Models: setup/gateway-models.md- CLI Reference:@@ -558,9 +548,11 @@ nav:- Test Profile Configuration: under-the-hood/test-profile-configuration.md- Init CLI Flows: under-the-hood/init-cli-flows.md- Pipe Routing & Execution: under-the-hood/pipe-routing-and-execution.md+ - Inference Backend Plugins: under-the-hood/inference-backend-plugins.md+ - Orchestrator Plugins: under-the-hood/orchestrator-plugins.md- Error Model: under-the-hood/error-model.md- - Temporal Integration: under-the-hood/temporal-integration.md- - Distributed Content Generation: under-the-hood/distributed-content-generation.md+ - Runtime Bridge & Transport: under-the-hood/runtime-bridge-and-transport.md+ - Content Generation Across Boundaries: under-the-hood/distributed-content-generation.md- Project:- Contributing: contributing.md- Configuration Internals: contribute/configuration-defaults-and-overrides.mddiff --git a/pipelex/cli/_cli.py b/pipelex/cli/_cli.pyindex e74421c52..8bb560002 100644--- a/pipelex/cli/_cli.py+++ b/pipelex/cli/_cli.py@@ -11,40 +11,40 @@ from pipelex.cli.commands.graph_cmd import graph_appfrom pipelex.cli.commands.init.command import init_cmdfrom pipelex.cli.commands.init.ui.types import InitFocusfrom pipelex.cli.commands.login.command import login_cmd+from pipelex.cli.commands.plugins_cmd import plugins_appfrom pipelex.cli.commands.run.app import run_app-from pipelex.cli.commands.setup_temporal_namespace_cmd import setup_temporal_namespace_cmdfrom pipelex.cli.commands.show_cmd import show_appfrom pipelex.cli.commands.update_cmd import update_cmdfrom pipelex.cli.commands.validate.app import validate_appfrom pipelex.cli.commands.which_cmd import which_cmd-from pipelex.cli.commands.worker_cmd import worker_cmdfrom pipelex.cli.deck_notice import warn_if_deck_stalefrom pipelex.cli.error_handlers import set_traceback_requestedfrom pipelex.cli.readiness import check_readinessfrom pipelex.hub import get_consolefrom pipelex.tools.misc.package_utils import get_package_version+# Core commands in display order (natural ordering doesn't work between Typer groups and commands).+_CORE_COMMAND_ORDER: list[str] = [+ "login",+ "init",+ "doctor",+ "update",+ "build",+ "validate",+ "run",+ "graph",+ "show",+ "which",+ "plugins",+]+class PipelexCLI(TyperGroup):"""Custom CLI group that handles global options like --no-logo."""@overridedef list_commands(self, ctx: Context) -> list[str]:- # List the commands in the proper order because natural ordering doesn't work between Typer groups and commands- return [- "login",- "init",- "doctor",- "update",- "build",- "validate",- "run",- "graph",- "show",- "which",- "worker",- "setup-temporal-namespace",- ]+ return list(_CORE_COMMAND_ORDER)@overridedef get_command(self, ctx: Context, cmd_name: str) -> Command | None:@@ -231,8 +231,4 @@ app.add_typer(run_app, name="run", help="Run a method or pipe, optionally providapp.add_typer(graph_app, name="graph", help="Generate and render execution graphs")app.add_typer(show_app, name="show", help="Show configuration, pipes, and list AI models")app.command(name="which", help="Locate where a pipe is defined, similar to 'which' for executables")(which_cmd)-app.command(name="worker", help="Start a Temporal worker for distributed workflow execution")(worker_cmd)-app.command(- name="setup-temporal-namespace",- help="Register Pipelex's custom search attributes on the configured Temporal namespace",-)(setup_temporal_namespace_cmd)+app.add_typer(plugins_app, name="plugins", help="Inspect the discovered plugins (inference backends, orchestrators) and their contributions")diff --git a/pipelex/cli/cli_factory.py b/pipelex/cli/cli_factory.pyindex 13d5d095b..1f859ab3b 100644--- a/pipelex/cli/cli_factory.py+++ b/pipelex/cli/cli_factory.py@@ -33,7 +33,7 @@ def make_pipelex_for_cli(context: ErrorContext,library_dirs: list[str] | list[Path] | None = None,needs_inference: bool = True,- temporal_enabled: bool | None = None,+ boot_orchestrator: str | None = None,needs_model_specs: bool | None = None,) -> Pipelex:"""Initialize Pipelex for CLI commands with proper error handling.@@ -45,7 +45,7 @@ def make_pipelex_for_cli(context: The CLI context for error messages.library_dirs: The library directories to use for the Pipelex instance.needs_inference: When False, skip inference setup (credentials, gateway, telemetry).- temporal_enabled: When provided, overrides the temporal.is_enabled config value.+ boot_orchestrator: When provided, boots this process under the orchestrator plugin of this name.needs_model_specs: When True, load real model specs even without inference.Returns:@@ -59,7 +59,7 @@ def make_pipelex_for_cli(integration_mode=IntegrationMode.CLI,library_dirs=library_dirs,needs_inference=needs_inference,- temporal_enabled=temporal_enabled,+ boot_orchestrator=boot_orchestrator,needs_model_specs=needs_model_specs,)except InferenceSetupRequiredError as exc:diff --git a/pipelex/cli/commands/plugins_cmd.py b/pipelex/cli/commands/plugins_cmd.pynew file mode 100644index 000000000..298790479--- /dev/null+++ b/pipelex/cli/commands/plugins_cmd.py@@ -0,0 +1,37 @@+import typer+from rich import box+from rich.table import Table++from pipelex.hub import get_console+from pipelex.plugins.discovery import build_registrar+from pipelex.system.configuration.config_loader import config_manager+from pipelex.system.configuration.configs import PipelexConfig++plugins_app = typer.Typer(no_args_is_help=True)+++@plugins_app.command(name="list", help="List discovered plugins (built-in and external), what each registered, and denylist state")+def plugins_list_command() -> None:+ """Discover plugins and print what each contributed.++ Runs the pure ``build_registrar`` against the loaded config — the same+ discovery the runtime uses at boot — so the listing reflects exactly what+ would be wired in (and what the ``plugins.disabled`` denylist turns off).+ """+ console = get_console()+ config = PipelexConfig.model_validate(config_manager.load_config())+ registrar = build_registrar(config=config)++ table = Table(title="Pipelex plugins", show_header=True, header_style="bold cyan", box=box.SQUARE_DOUBLE_HEAD)+ table.add_column("Name", style="green")+ table.add_column("Origin", style="blue")+ table.add_column("Status", style="yellow")+ table.add_column("API", justify="right")+ table.add_column("Contributions")++ for discovery in registrar.discoveries:+ contributions = "\n".join(discovery.contributions) if discovery.contributions else (discovery.detail or "—")+ targets_api = str(discovery.targets_api) if discovery.targets_api is not None else "—"+ table.add_row(discovery.name, discovery.origin, discovery.status, targets_api, contributions)++ console.print(table)diff --git a/pipelex/cli/commands/run/_run_core.py b/pipelex/cli/commands/run/_run_core.pyindex b96ce69ab..a3c0a7b0c 100644--- a/pipelex/cli/commands/run/_run_core.py+++ b/pipelex/cli/commands/run/_run_core.py@@ -359,7 +359,7 @@ def execute_run(library_dir: list[str] | None,costs: bool | None = None,telemetry_command_label: str = COMMAND,- temporal: bool | None = None,+ orchestrator: str | None = None,dynamic_output_concept_ref: str | None = None,save_csv: str | None = None,) -> None:@@ -367,7 +367,7 @@ def execute_run(Shared between the ``method`` and ``pipe`` subcommands."""- make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN, library_dirs=library_dir, temporal_enabled=temporal)+ make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN, library_dirs=library_dir, boot_orchestrator=orchestrator)try:with get_telemetry_manager().telemetry_context():diff --git a/pipelex/cli/commands/run/bundle_cmd.py b/pipelex/cli/commands/run/bundle_cmd.pyindex f9697ec4b..792e4a291 100644--- a/pipelex/cli/commands/run/bundle_cmd.py+++ b/pipelex/cli/commands/run/bundle_cmd.py@@ -77,9 +77,9 @@ def run_bundle_cmd(list[str] | None,typer.Option("--library-dir", "-L", help="Directory to search for pipe definitions (.mthds files). Can be specified multiple times."),] = None,- temporal: Annotated[- bool | None,- typer.Option("--temporal/--no-temporal", help="Override config: enable or disable Temporal workflow execution"),+ orchestrator: Annotated[+ str | None,+ typer.Option("--orchestrator", help="Boot this process under the named orchestrator plugin (e.g. 'temporal'); omit for in-process execution"),] = None,dynamic_output_concept_ref: Annotated[str | None,@@ -189,7 +189,7 @@ def run_bundle_cmd(library_dir=library_dir,costs=costs,telemetry_command_label=f"{COMMAND} bundle",- temporal=temporal,+ orchestrator=orchestrator,dynamic_output_concept_ref=dynamic_output_concept_ref,save_csv=save_csv,)diff --git a/pipelex/cli/commands/run/method_cmd.py b/pipelex/cli/commands/run/method_cmd.pyindex bb9ed0008..e0ad5e35f 100644--- a/pipelex/cli/commands/run/method_cmd.py+++ b/pipelex/cli/commands/run/method_cmd.py@@ -76,9 +76,9 @@ def run_method_cmd(list[str] | None,typer.Option("--library-dir", "-L", help="Directory to search for pipe definitions (.mthds files). Can be specified multiple times."),] = None,- temporal: Annotated[- bool | None,- typer.Option("--temporal/--no-temporal", help="Override config: enable or disable Temporal workflow execution"),+ orchestrator: Annotated[+ str | None,+ typer.Option("--orchestrator", help="Boot this process under the named orchestrator plugin (e.g. 'temporal'); omit for in-process execution"),] = None,dynamic_output_concept_ref: Annotated[str | None,@@ -159,7 +159,7 @@ def run_method_cmd(library_dir=effective_library_dir,costs=costs,telemetry_command_label=f"{COMMAND} method",- temporal=temporal,+ orchestrator=orchestrator,dynamic_output_concept_ref=dynamic_output_concept_ref,save_csv=save_csv,)diff --git a/pipelex/cli/commands/run/pipe_cmd.py b/pipelex/cli/commands/run/pipe_cmd.pyindex 7e96fab46..88878cb41 100644--- a/pipelex/cli/commands/run/pipe_cmd.py+++ b/pipelex/cli/commands/run/pipe_cmd.py@@ -73,9 +73,9 @@ def run_pipe_cmd(list[str] | None,typer.Option("--library-dir", "-L", help="Directory to search for pipe definitions (.mthds files). Can be specified multiple times."),] = None,- temporal: Annotated[- bool | None,- typer.Option("--temporal/--no-temporal", help="Override config: enable or disable Temporal workflow execution"),+ orchestrator: Annotated[+ str | None,+ typer.Option("--orchestrator", help="Boot this process under the named orchestrator plugin (e.g. 'temporal'); omit for in-process execution"),] = None,dynamic_output_concept_ref: Annotated[str | None,@@ -154,7 +154,7 @@ def run_pipe_cmd(library_dir=library_dir,costs=costs,telemetry_command_label=f"{COMMAND} pipe",- temporal=temporal,+ orchestrator=orchestrator,dynamic_output_concept_ref=dynamic_output_concept_ref,save_csv=save_csv,)diff --git a/pipelex/cli/commands/setup_temporal_namespace_cmd.py b/pipelex/cli/commands/setup_temporal_namespace_cmd.pydeleted file mode 100644index 04c54b84a..000000000--- a/pipelex/cli/commands/setup_temporal_namespace_cmd.py+++ /dev/null@@ -1,182 +0,0 @@-"""pipelex setup-temporal-namespace — Register Pipelex's custom search-attributes on the configured Temporal namespace.--Wraps the same ``ensure_required_search_attributes_registered`` helper used by-the test infrastructure so operators get a one-shot CLI fix for the otherwise-opaque ``Namespace ... has no mapping defined for search attribute ...`` error-the cluster raises when a workflow start references an unregistered attribute.--Reads the same ``[temporal.search_attributes].attributes`` block the worker-populates, so the names registered here can never drift from the names the-worker will dispatch with. ``--dry-run`` prints the equivalent raw ``temporal``-CLI invocation without touching the cluster — useful when a namespace admin-needs to register on the operator's behalf.--Optional-dependency contract: this module must not import ``pipelex.temporal``-at module level so ``pipelex --help`` works without the ``temporal`` extra-installed. Deferred imports inside the function body carry the ``PLC0415``-ruff-ignore, matching the ``worker_cmd`` precedent. We deliberately do NOT-wrap the deferred imports in ``try/except ImportError`` — that would silently-relabel real bugs in the imported chain as "install the temporal extra";-``ModuleNotFoundError: No module named 'temporalio'`` from the raw chain is-already actionable.-"""--import asyncio-from typing import Annotated--import typer--from pipelex import log-from pipelex.cli.cli_factory import make_pipelex_for_cli-from pipelex.cli.error_handlers import ErrorContext-from pipelex.config import get_config-from pipelex.pipelex import Pipelex---def setup_temporal_namespace_cmd(- dry_run: Annotated[- bool,- typer.Option(- "--dry-run",- help="Print the equivalent `temporal operator search-attribute create` command without executing.",- ),- ] = False,- server: Annotated[- str | None,- typer.Option(- "--server",- help="Target a non-default server profile from [temporal.temporal_config.temporal_server_configs] (defaults to selected_server).",- ),- ] = None,-) -> None:- """Register Pipelex's custom search attributes on the Temporal namespace.-- Examples:- pipelex setup-temporal-namespace- pipelex setup-temporal-namespace --dry-run- pipelex setup-temporal-namespace --server testing- """- # ``needs_inference=False``: namespace registration only needs the Temporal- # config; forcing inference setup (gateway/telemetry/credentials) would make- # the command fail on unrelated prerequisites that aren't required to talk- # to the Operator service.- make_pipelex_for_cli(- context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN,- temporal_enabled=True,- needs_inference=False,- )-- try:- from temporalio.service import RPCError # noqa: PLC0415-- from pipelex.temporal.temporal_connect import connect_to_temporal_selected_server # noqa: PLC0415- from pipelex.temporal.tprl.namespace_check import ( # noqa: PLC0415- RegistrationFailure,- ensure_required_search_attributes_registered,- format_tcld_cli_command,- format_temporal_cli_command,- )-- search_attributes_config = get_config().temporal.search_attributes- if not search_attributes_config.enabled:- typer.secho(- "[temporal.search_attributes].enabled = false — nothing to register. "- "Set enabled = true in your pipelex.toml to opt in to custom search attributes.",- fg=typer.colors.YELLOW,- err=True,- )- return-- configured = list(search_attributes_config.attributes)- if not configured:- typer.secho(- "[temporal.search_attributes].attributes is empty — nothing to register.",- fg=typer.colors.YELLOW,- err=True,- )- return-- temporal_config = get_config().temporal.temporal_config- selected_profile = server or temporal_config.selected_server- server_config = temporal_config.temporal_server_configs.get(selected_profile)- if server_config is None:- typer.secho(- f"Unknown server profile '{selected_profile}'. Known: {sorted(temporal_config.temporal_server_configs.keys())}",- fg=typer.colors.RED,- err=True,- )- raise typer.Exit(1)- namespace = server_config.namespace-- if dry_run:- typer.echo(f"# Equivalent Temporal CLI invocation for namespace '{namespace}' (server profile '{selected_profile}'):")- typer.echo(format_temporal_cli_command(configured, namespace=namespace))- typer.echo(- "\n# For Temporal Cloud, the namespace admin can also use:\n"- f"# {format_tcld_cli_command(configured, namespace=namespace)}\n"- "# or the Cloud UI: Namespace → Custom Search Attributes."- )- return-- async def _connect_and_register() -> RegistrationFailure | tuple[str, ...]:- # Single event loop for both awaits: the Temporal SDK binds the- # Client to the loop it was created on, so the registration call- # must run on the same loop as the connect.- temporal_client = await connect_to_temporal_selected_server(selected_server_config=selected_profile)- return await ensure_required_search_attributes_registered(- temporal_client=temporal_client,- namespace=namespace,- configured_attributes=configured,- )-- log.info(f"Connecting to Temporal server profile '{selected_profile}' (namespace='{namespace}')...")- try:- outcome = asyncio.run(_connect_and_register())- except RPCError as exc:- # The helper propagates non-PERMISSION_DENIED RPC errors (e.g.- # NOT_FOUND when the namespace doesn't exist, UNAVAILABLE when the- # control plane is down). Frame these for the operator instead of- # leaking a raw traceback.- typer.secho(- f"Could not reach Temporal namespace '{namespace}' on profile '{selected_profile}': {exc}",- fg=typer.colors.RED,- err=True,- )- raise typer.Exit(1) from exc-- if isinstance(outcome, RegistrationFailure):- typer.secho(- "Permission denied: this Temporal API key cannot register search "- "attributes on the namespace (OperatorService.ListSearchAttributes "- "or OperatorService.AddSearchAttributes). Ask a namespace admin to "- "register the missing attributes manually:\n",- fg=typer.colors.RED,- err=True,- )- typer.echo(format_temporal_cli_command(outcome.missing, namespace=namespace), err=True)- typer.echo(- "\nFor Temporal Cloud:\n"- f" {format_tcld_cli_command(outcome.missing, namespace=namespace)}\n"- " or the Cloud UI: Namespace → Custom Search Attributes.\n"- f"\n(Underlying RPC error: {outcome.rpc_error_message})",- err=True,- )- raise typer.Exit(1)-- # outcome is the tuple of newly-registered attribute names. Empty means- # the namespace already had everything — report that distinctly so the- # operator knows the command was idempotent rather than misreading the- # success count.- if len(outcome) == 0:- typer.secho(- f"All {len(configured)} configured custom search attribute(s) were already registered on namespace '{namespace}'.",- fg=typer.colors.GREEN,- )- else:- typer.secho(- f"Registered {len(outcome)} new custom search attribute(s) on namespace '{namespace}': {list(outcome)}.",- fg=typer.colors.GREEN,- )- finally:- Pipelex.teardown_if_needed()diff --git a/pipelex/cli/commands/validate/_validate_core.py b/pipelex/cli/commands/validate/_validate_core.pyindex bd28f812f..b3cd10479 100644--- a/pipelex/cli/commands/validate/_validate_core.py+++ b/pipelex/cli/commands/validate/_validate_core.py@@ -191,17 +191,17 @@ def execute_validate(library_dirs: list[Path] | None,telemetry_command_label: str = COMMAND,allow_signatures: bool = False,- temporal: bool | None = None,+ orchestrator: str | None = None,) -> None:"""Synchronous entry point wrapping the async validation with Pipelex setup/teardown.- ``temporal`` overrides ``temporal.is_enabled`` for the boot (mirroring ``run``). The- validation sweep itself always runs in-process — under a Temporal-enabled hub it scopes- its own in-process router so nested controller sub-pipes do not dispatch to Temporal — so- this flag does not change *what* validation does; it controls how Pipelex boots, which is- the lever for exercising the "validation stays in-process on a Temporal backend" contract.+ ``orchestrator`` boots this process under the named orchestrator plugin (mirroring ``run``).+ The validation sweep itself always runs in-process — under an orchestrator-backed hub it scopes+ its own in-process router so nested controller sub-pipes do not dispatch to the orchestrator+ runtime — so this flag does not change *what* validation does; it controls how Pipelex boots,+ which is the lever for exercising the "validation stays in-process on an orchestrator backend" contract."""- make_pipelex_for_cli(context=ErrorContext.VALIDATION, needs_inference=False, needs_model_specs=True, temporal_enabled=temporal)+ make_pipelex_for_cli(context=ErrorContext.VALIDATION, needs_inference=False, needs_model_specs=True, boot_orchestrator=orchestrator)try:with get_telemetry_manager().telemetry_context():diff --git a/pipelex/cli/commands/validate/bundle_cmd.py b/pipelex/cli/commands/validate/bundle_cmd.pyindex 6b780608e..791fb01ca 100644--- a/pipelex/cli/commands/validate/bundle_cmd.py+++ b/pipelex/cli/commands/validate/bundle_cmd.py@@ -30,12 +30,12 @@ def validate_bundle_cmd(help="Accept PipeSignature placeholders in the dependency graph (lenient mode).",),] = False,- temporal: Annotated[- bool | None,+ orchestrator: Annotated[+ str | None,typer.Option(- "--temporal/--no-temporal",- help="Override config temporal.is_enabled for the boot. The sweep stays in-process either way; "- "use it to verify validation does not dispatch to Temporal under a Temporal-enabled hub.",+ "--orchestrator",+ help="Boot this process under the named orchestrator plugin (e.g. 'temporal'). The validation sweep "+ "stays in-process either way; use it to verify validation does not dispatch to an orchestrator runtime.",),] = None,) -> None:@@ -104,5 +104,5 @@ def validate_bundle_cmd(library_dirs=library_dirs_paths,telemetry_command_label=f"{COMMAND} bundle",allow_signatures=allow_signatures,- temporal=temporal,+ orchestrator=orchestrator,)diff --git a/pipelex/cli/commands/validate/method_cmd.py b/pipelex/cli/commands/validate/method_cmd.pyindex cd8a46ec7..2cb68d5c3 100644--- a/pipelex/cli/commands/validate/method_cmd.py+++ b/pipelex/cli/commands/validate/method_cmd.py@@ -26,12 +26,12 @@ def validate_method_cmd(help="Directory to search for pipe definitions (.mthds files). Can be specified multiple times.",),] = None,- temporal: Annotated[- bool | None,+ orchestrator: Annotated[+ str | None,typer.Option(- "--temporal/--no-temporal",- help="Override config temporal.is_enabled for the boot. The sweep stays in-process either way; "- "use it to verify validation does not dispatch to Temporal under a Temporal-enabled hub.",+ "--orchestrator",+ help="Boot this process under the named orchestrator plugin (e.g. 'temporal'). The validation sweep "+ "stays in-process either way; use it to verify validation does not dispatch to an orchestrator runtime.",),] = None,) -> None:@@ -59,5 +59,5 @@ def validate_method_cmd(bundle_path=None,library_dirs=library_dirs_paths,telemetry_command_label=f"{COMMAND} method",- temporal=temporal,+ orchestrator=orchestrator,)diff --git a/pipelex/cli/commands/validate/pipe_cmd.py b/pipelex/cli/commands/validate/pipe_cmd.pyindex f11dd8926..5720f3a3f 100644--- a/pipelex/cli/commands/validate/pipe_cmd.py+++ b/pipelex/cli/commands/validate/pipe_cmd.py@@ -41,12 +41,12 @@ def validate_pipe_cmd(help="Accept PipeSignature placeholders in the dependency graph (lenient mode).",),] = False,- temporal: Annotated[- bool | None,+ orchestrator: Annotated[+ str | None,typer.Option(- "--temporal/--no-temporal",- help="Override config temporal.is_enabled for the boot. The sweep stays in-process either way; "- "use it to verify validation does not dispatch to Temporal under a Temporal-enabled hub.",+ "--orchestrator",+ help="Boot this process under the named orchestrator plugin (e.g. 'temporal'). The validation sweep "+ "stays in-process either way; use it to verify validation does not dispatch to an orchestrator runtime.",),] = None,) -> None:@@ -73,7 +73,7 @@ def validate_pipe_cmd(library_dirs=library_dirs_paths,needs_inference=False,needs_model_specs=True,- temporal_enabled=temporal,+ boot_orchestrator=orchestrator,)do_validate_all_libraries_and_dry_run(library_dirs=library_dirs_paths,@@ -127,5 +127,5 @@ def validate_pipe_cmd(library_dirs=library_dirs_paths,telemetry_command_label=f"{COMMAND} pipe",allow_signatures=allow_signatures,- temporal=temporal,+ orchestrator=orchestrator,)diff --git a/pipelex/cli/commands/worker_cmd.py b/pipelex/cli/commands/worker_cmd.pydeleted file mode 100644index 96a6ddf04..000000000--- a/pipelex/cli/commands/worker_cmd.py+++ /dev/null@@ -1,74 +0,0 @@-"""pipelex worker — Start a Temporal worker."""--import asyncio-from typing import Annotated--import typer--from pipelex import log-from pipelex.cli.cli_factory import make_pipelex_for_cli-from pipelex.cli.error_handlers import ErrorContext-from pipelex.pipelex import Pipelex---def worker_cmd(- is_not_sandboxed: Annotated[- bool,- typer.Option("--no-sandbox", help="Run workflows without Temporal sandbox restrictions"),- ] = False,- is_unit_testing: Annotated[- bool,- typer.Option("--unit-testing", help="Run in unit testing mode"),- ] = False,- task_queue: Annotated[- str | None,- typer.Option("--task-queue", help="Override the task queue name from config"),- ] = None,- scope: Annotated[- str | None,- typer.Option("--scope", help="Worker scope name from [temporal.worker_scopes.scopes] (defaults to default_scope)"),- ] = None,- profile: Annotated[- str | None,- typer.Option(- "--profile",- help="Worker runtime profile name from [temporal.worker_runtime_profiles.profiles] (defaults to default_profile)",- ),- ] = None,-) -> None:- """Start a Temporal worker.-- This command initializes Pipelex with Temporal enabled and starts- a worker that listens for workflow and activity tasks.-- Examples:- pipelex worker- pipelex worker --no-sandbox- pipelex worker --task-queue my_queue- pipelex worker --scope router- pipelex worker --profile anthropic-tier4 --scope runner-llm --task-queue anthropic_q- """- make_pipelex_for_cli(context=ErrorContext.VALIDATION_BEFORE_PIPE_RUN, temporal_enabled=True)-- from pipelex.temporal.temporal_hub import get_task_manager # noqa: PLC0415-- try:- log.info("Starting Temporal worker...")- asyncio.run(- get_task_manager().run_worker(- is_not_sandboxed=is_not_sandboxed,- is_unit_testing=is_unit_testing,- task_queue=task_queue,- scope_name=scope,- profile_name=profile,- )- )- except KeyboardInterrupt:- log.info("Worker stopped by user")- except Exception as exc:- # CLI command root: any unexpected failure is reported to the user and exits non-zero via typer.Exit.- log.error(f"Worker failed: {exc}")- typer.secho(f"Worker failed: {exc}", fg=typer.colors.RED, err=True)- raise typer.Exit(1) from exc- finally:- Pipelex.teardown_if_needed()diff --git a/pipelex/cogt/content_generation/search_generate.py b/pipelex/cogt/content_generation/search_generate.pyindex 649ddfe70..5756d8c0c 100644--- a/pipelex/cogt/content_generation/search_generate.py+++ b/pipelex/cogt/content_generation/search_generate.py@@ -19,7 +19,7 @@ from pipelex.cogt.search.search_job_factory import SearchJobFactoryfrom pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstractfrom pipelex.cogt.search.search_worker_factory import SearchWorkerFactoryfrom pipelex.core.stuffs.search_result_content import SearchResultContent-from pipelex.hub import get_model_deck+from pipelex.hub import get_model_deck, get_report_delegatedef _make_search_worker(search_assignment: SearchAssignment) -> SearchWorkerAbstract:@@ -28,7 +28,7 @@ def _make_search_worker(search_assignment: SearchAssignment) -> SearchWorkerAbstmodel_handle=search_assignment.search_setting.model,model_type=ModelType.SEARCH,)- return SearchWorkerFactory.make_search_worker(inference_model=inference_model)+ return SearchWorkerFactory.make_search_worker(inference_model=inference_model, reporting_delegate=get_report_delegate())def _make_search_job(search_assignment: SearchAssignment) -> SearchJob:diff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.pyindex 643825da0..2076e6b13 100644--- a/pipelex/cogt/exceptions.py+++ b/pipelex/cogt/exceptions.py@@ -438,6 +438,22 @@ class ModelManagerError(CogtError):pass+class ModelListingUnsupportedError(CogtError):+ """A registered lister cannot enumerate models for an SDK variant at runtime.++ A *soft* control signal for the ``list-models`` loop: it is caught and the SDK+ is reported as unsupported-for-remote-listing rather than failing the command.+ Same outcome as a registry miss (no lister registered for the SDK at all). A+ lister closure raises it when its client variant cannot list (e.g. a+ bedrock-backed Anthropic client), translating any vendor-specific+ "unsupported" exception so core names no integration.+ """++ def __init__(self, *, sdk: str) -> None:+ self.sdk = sdk+ super().__init__(f"The '{sdk}' SDK client does not support remote model listing.")++class ModelDeckNotFoundError(CogtError):passdiff --git a/pipelex/cogt/extract/extract_worker_factory.py b/pipelex/cogt/extract/extract_worker_factory.pyindex 1449ff80b..7bfb022c9 100644--- a/pipelex/cogt/extract/extract_worker_factory.py+++ b/pipelex/cogt/extract/extract_worker_factory.py@@ -1,10 +1,10 @@-import importlib.util+from typing import castfrom pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.exceptions import MissingDependencyError-from pipelex.hub import get_models_manager, get_plugin_manager-from pipelex.plugins.plugin_sdk_registry import Plugin+from pipelex.hub import get_inference_backend_registry, get_models_manager, get_sdk_client_manager+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.reporting.reporting_protocol import ReportingProtocol@@ -16,104 +16,16 @@ class ExtractWorkerFactory:*,reporting_delegate: ReportingProtocol | None = None,) -> ExtractWorkerAbstract:- plugin = Plugin.make_for_inference_model(inference_model=inference_model)+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)- plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry- extract_worker: ExtractWorkerAbstract- match plugin.sdk:- case "gateway_extract":- from pipelex.plugins.gateway.gateway_extract_worker import GatewayExtractWorker # noqa: PLC0415- from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415-- extract_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=GatewayFactory.make_portkey_client(backend=backend),- )-- extract_worker = GatewayExtractWorker(- sdk_instance=extract_sdk_instance,- extra_config=backend.extra_config,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "mistral":- if importlib.util.find_spec("mistralai") is None:- lib_name = "mistralai"- lib_extra_name = "mistral"- msg = "The mistralai SDK is required in order to use Mistral OCR models through the mistralai client."- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.mistral.mistral_extract_worker import MistralExtractWorker # noqa: PLC0415- from pipelex.plugins.mistral.mistral_factory import MistralFactory # noqa: PLC0415-- extract_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=MistralFactory.make_mistral_client(backend=backend),- )-- extract_worker = MistralExtractWorker(- sdk_instance=extract_sdk_instance,- extra_config=backend.extra_config,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "pypdfium2":- from pipelex.plugins.pypdfium2.pypdfium2_worker import Pypdfium2Worker # noqa: PLC0415-- extract_worker = Pypdfium2Worker(- extra_config=backend.extra_config,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "docling_sdk":- if importlib.util.find_spec("docling") is None:- lib_name = "docling"- lib_extra_name = "docling"- msg = "The docling library is required in order to use Docling for PDF and image text extraction."- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.docling.docling_extract_worker import DoclingExtractWorker # noqa: PLC0415- from pipelex.plugins.docling.docling_factory import DoclingFactory # noqa: PLC0415-- extract_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=DoclingFactory.make_docling_sdk(),- )-- extract_worker = DoclingExtractWorker(- sdk_instance=extract_sdk_instance,- extra_config=backend.extra_config,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "linkup_fetch":- if importlib.util.find_spec("linkup") is None:- lib_name = "linkup"- lib_extra_name = "linkup"- msg = "The linkup SDK is required in order to use Linkup Fetch extraction models."- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.linkup.linkup_extract_worker import LinkupExtractWorker # noqa: PLC0415-- extract_worker = LinkupExtractWorker(- extra_config=backend.extra_config,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case _:- msg = f"Plugin '{plugin}' is not supported"- raise NotImplementedError(msg)-- return extract_worker+ make_worker = get_inference_backend_registry().lookup(family=InferenceFamily.EXTRACT, sdk=model_handle.sdk)+ worker = make_worker(+ inference_model=inference_model,+ backend=backend,+ sdk_clients=get_sdk_client_manager().sdk_client_registry,+ reporting_delegate=reporting_delegate,+ )+ # The EXTRACT family registry only ever holds extract workers (the+ # (family, sdk) key guarantees it); the uniform MakeWorkerFn return type is+ # widened to InferenceWorkerAbstract, so narrow it back here.+ return cast("ExtractWorkerAbstract", worker)diff --git a/pipelex/cogt/img_gen/img_gen_worker_factory.py b/pipelex/cogt/img_gen/img_gen_worker_factory.pyindex e71eec0eb..72e2c877f 100644--- a/pipelex/cogt/img_gen/img_gen_worker_factory.py+++ b/pipelex/cogt/img_gen/img_gen_worker_factory.py@@ -1,11 +1,10 @@-import importlib.util-from typing import TYPE_CHECKING+from typing import castfrom pipelex.cogt.img_gen.img_gen_worker_abstract import ImgGenWorkerAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.exceptions import MissingDependencyError-from pipelex.hub import get_models_manager, get_plugin_manager-from pipelex.plugins.plugin_sdk_registry import Plugin+from pipelex.hub import get_inference_backend_registry, get_models_manager, get_sdk_client_manager+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.reporting.reporting_protocol import ReportingProtocol@@ -17,194 +16,16 @@ class ImgGenWorkerFactory:*,reporting_delegate: ReportingProtocol | None = None,) -> ImgGenWorkerAbstract:- plugin = Plugin.make_for_inference_model(inference_model=inference_model)+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)- plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry- img_gen_worker: ImgGenWorkerAbstract- match plugin.sdk:- case "gateway_img_gen":- from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415- from pipelex.plugins.gateway.gateway_img_gen_worker import GatewayImgGenWorker # noqa: PLC0415-- img_gen_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=GatewayFactory.make_portkey_client(backend=backend),- )-- img_gen_worker = GatewayImgGenWorker(- sdk_instance=img_gen_sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "fal":- if importlib.util.find_spec("fal_client") is None:- lib_name = "fal-client"- lib_extra_name = "fal"- msg = "The fal-client SDK is required in order to use FAL models (generation of images)."- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from fal_client import AsyncClient as FalAsyncClient # noqa: PLC0415-- from pipelex.plugins.fal.fal_img_gen_worker import FalImgGenWorker # noqa: PLC0415-- img_gen_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=FalAsyncClient(key=backend.api_key),- )-- img_gen_worker = FalImgGenWorker(- sdk_instance=img_gen_sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "huggingface_img_gen":- from huggingface_hub import AsyncInferenceClient # noqa: PLC0415-- from pipelex.plugins.huggingface.huggingface_factory import HuggingFaceFactory # noqa: PLC0415- from pipelex.plugins.huggingface.huggingface_img_gen_worker import HuggingFaceImgGenWorker # noqa: PLC0415-- if TYPE_CHECKING:- from huggingface_hub.inference._providers import PROVIDER_OR_POLICY_T # noqa: PLC0415-- provider_literal: PROVIDER_OR_POLICY_T- if provider_str := plugin.variant:- provider_literal = HuggingFaceFactory.make_huggingface_inference_provider(provider_str=provider_str)- else:- provider_literal = "auto"- img_gen_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=AsyncInferenceClient(- provider=provider_literal,- token=backend.api_key,- ),- )-- img_gen_worker = HuggingFaceImgGenWorker(- sdk_instance=img_gen_sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "openai_img_gen":- from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_img_gen_worker import OpenAIImgGenWorker # noqa: PLC0415-- img_gen_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=OpenAIClientFactory.make_openai_client(- plugin=plugin,- backend=backend,- ),- )-- img_gen_worker = OpenAIImgGenWorker(- sdk_instance=img_gen_sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "blackboxai_img_gen":- from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_completions_img_gen_worker import OpenAICompletionsImgGenWorker # noqa: PLC0415-- img_gen_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=OpenAIClientFactory.make_openai_client(- plugin=plugin,- backend=backend,- ),- )-- # Deferred import: avoid pulling heavy SDK at module-load time- from pipelex.plugins.blackboxai.blackboxai_completions_factory import BlackboxaiCompletionsFactory # noqa: PLC0415-- bbai_completions_factory = BlackboxaiCompletionsFactory(is_http_url_enabled=True)-- img_gen_worker = OpenAICompletionsImgGenWorker(- openai_completions_factory=bbai_completions_factory,- sdk_instance=img_gen_sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "azure_rest_img_gen":- from pipelex.plugins.azure_rest.azure_img_gen_worker import AzureImgGenWorker # noqa: PLC0415-- img_gen_worker = AzureImgGenWorker(- plugin=plugin,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "google":- if importlib.util.find_spec("google.genai") is None:- lib_name = "google-genai"- lib_extra_name = "google"- msg = "The google-genai SDK is required in order to use Google Gemini Image models."- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.google.google_factory import GoogleFactory # noqa: PLC0415- from pipelex.plugins.google.google_img_gen_worker import GoogleImgGenWorker # noqa: PLC0415-- img_gen_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=GoogleFactory.make_google_client(backend=backend),- )-- img_gen_worker = GoogleImgGenWorker(- sdk_instance=img_gen_sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "gateway_completions":- from pipelex.plugins.gateway.gateway_completions_factory import GatewayCompletionsFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_completions_img_gen_worker import OpenAICompletionsImgGenWorker # noqa: PLC0415-- img_gen_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=GatewayCompletionsFactory.make_portkey_openai_client_for_completions(- plugin=plugin,- backend=backend,- ),- )-- gateway_completions_factory = GatewayCompletionsFactory(is_http_url_enabled=False)-- img_gen_worker = OpenAICompletionsImgGenWorker(- openai_completions_factory=gateway_completions_factory,- sdk_instance=img_gen_sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "openrouter_img_gen":- from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_completions_img_gen_worker import OpenAICompletionsImgGenWorker # noqa: PLC0415-- img_gen_sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=OpenAIClientFactory.make_openai_client(- plugin=plugin,- backend=backend,- ),- )-- # Deferred import: avoid pulling heavy SDK at module-load time- from pipelex.plugins.openrouter.openrouter_completions_factory import OpenRouterCompletionsFactory # noqa: PLC0415-- openrouter_completions_factory = OpenRouterCompletionsFactory(is_http_url_enabled=True)-- img_gen_worker = OpenAICompletionsImgGenWorker(- openai_completions_factory=openrouter_completions_factory,- sdk_instance=img_gen_sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case _:- msg = f"Plugin '{plugin}' is not supported for image generation"- raise NotImplementedError(msg)-- return img_gen_worker+ make_worker = get_inference_backend_registry().lookup(family=InferenceFamily.IMG_GEN, sdk=model_handle.sdk)+ worker = make_worker(+ inference_model=inference_model,+ backend=backend,+ sdk_clients=get_sdk_client_manager().sdk_client_registry,+ reporting_delegate=reporting_delegate,+ )+ # The IMG_GEN family registry only ever holds image generation workers (the+ # (family, sdk) key guarantees it); the uniform MakeWorkerFn return type is+ # widened to InferenceWorkerAbstract, so narrow it back here.+ return cast("ImgGenWorkerAbstract", worker)diff --git a/pipelex/cogt/llm/llm_worker_factory.py b/pipelex/cogt/llm/llm_worker_factory.pyindex 9bf2014b8..a3eb86fda 100644--- a/pipelex/cogt/llm/llm_worker_factory.py+++ b/pipelex/cogt/llm/llm_worker_factory.py@@ -1,10 +1,10 @@-import importlib.util+from typing import castfrom pipelex.cogt.llm.llm_worker_internal_abstract import LLMWorkerInternalAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec-from pipelex.exceptions import MissingDependencyError-from pipelex.hub import get_models_manager, get_plugin_manager-from pipelex.plugins.plugin_sdk_registry import Plugin+from pipelex.hub import get_inference_backend_registry, get_models_manager, get_sdk_client_manager+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.reporting.reporting_protocol import ReportingProtocol@@ -15,241 +15,16 @@ class LLMWorkerFactory:*,reporting_delegate: ReportingProtocol | None = None,) -> LLMWorkerInternalAbstract:- plugin = Plugin.make_for_inference_model(inference_model=inference_model)+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)- plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry- llm_worker: LLMWorkerInternalAbstract- match plugin.sdk:- case "gateway_completions":- from pipelex.plugins.gateway.gateway_completions_factory import GatewayCompletionsFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=GatewayCompletionsFactory.make_portkey_openai_client_for_completions(- plugin=plugin,- backend=backend,- ),- )-- gateway_completions_factory = GatewayCompletionsFactory(is_http_url_enabled=False)-- llm_worker = OpenAICompletionsLLMWorker(- openai_completions_factory=gateway_completions_factory,- sdk_instance=sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "gateway_responses":- from pipelex.plugins.gateway.gateway_responses_factory import GatewayResponsesFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=GatewayResponsesFactory.make_portkey_openai_client_for_responses(- plugin=plugin,- backend=backend,- ),- )-- gateway_responses_factory = GatewayResponsesFactory(is_http_url_enabled=False)-- llm_worker = OpenAIResponsesLLMWorker(- openai_responses_factory=gateway_responses_factory,- sdk_instance=sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "portkey_completions":- from pipelex.plugins.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker # noqa: PLC0415- from pipelex.plugins.portkey.portkey_completions_factory import PortkeyCompletionsFactory # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=PortkeyCompletionsFactory.make_portkey_openai_client_for_completions(- plugin=plugin,- backend=backend,- ),- )-- portkey_completions_factory = PortkeyCompletionsFactory(is_http_url_enabled=False)-- llm_worker = OpenAICompletionsLLMWorker(- openai_completions_factory=portkey_completions_factory,- sdk_instance=sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "portkey_responses":- from pipelex.plugins.openai.openai_responses_factory import OpenAIResponsesFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker # noqa: PLC0415- from pipelex.plugins.portkey.portkey_responses_factory import PortkeyResponsesFactory # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=PortkeyResponsesFactory.make_portkey_openai_client_for_responses(- plugin=plugin,- backend=backend,- ),- )-- portkey_responses_factory = PortkeyResponsesFactory(is_http_url_enabled=False)-- llm_worker = OpenAIResponsesLLMWorker(- openai_responses_factory=portkey_responses_factory,- sdk_instance=sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "openai_responses" | "azure_openai_responses":- from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_responses_factory import OpenAIResponsesFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=OpenAIClientFactory.make_openai_client(- plugin=plugin,- backend=backend,- ),- )-- openai_responses_factory = OpenAIResponsesFactory(is_http_url_enabled=True)-- llm_worker = OpenAIResponsesLLMWorker(- openai_responses_factory=openai_responses_factory,- sdk_instance=sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "openai" | "azure_openai":- from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsFactory # noqa: PLC0415- from pipelex.plugins.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=OpenAIClientFactory.make_openai_client(- plugin=plugin,- backend=backend,- ),- )-- openai_completions_factory = OpenAICompletionsFactory(is_http_url_enabled=True)-- llm_worker = OpenAICompletionsLLMWorker(- openai_completions_factory=openai_completions_factory,- sdk_instance=sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "anthropic" | "bedrock_anthropic":- if importlib.util.find_spec("anthropic") is None:- lib_name = "anthropic"- lib_extra_name = "anthropic"- msg = (- "The anthropic SDK is required in order to use Anthropic models via the anthropic client. "- "However, you can use Anthropic models through bedrock directly "- "by using the 'bedrock-anthropic-claude' llm family. (eg: bedrock-anthropic-claude)"- )- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.anthropic.anthropic_factory import AnthropicFactory # noqa: PLC0415- from pipelex.plugins.anthropic.anthropic_llm_worker import AnthropicLLMWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=AnthropicFactory.make_anthropic_client(plugin=plugin, backend=backend),- )-- llm_worker = AnthropicLLMWorker(- sdk_instance=sdk_instance,- extra_config=backend.extra_config,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "mistral":- if importlib.util.find_spec("mistralai") is None:- lib_name = "mistralai"- lib_extra_name = "mistral"- msg = (- "The mistralai SDK is required in order to use Mistral models through the mistralai client. "- "However, you can use Mistral models through bedrock directly "- "by using the 'bedrock-mistral' llm family. (eg: bedrock-mistral-large)"- )- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.mistral.mistral_factory import MistralFactory # noqa: PLC0415- from pipelex.plugins.mistral.mistral_llm_worker import MistralLLMWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=MistralFactory.make_mistral_client(backend=backend),- )-- llm_worker = MistralLLMWorker(- mistral_factory=MistralFactory(),- sdk_instance=sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "bedrock_boto3" | "bedrock_aioboto3":- if importlib.util.find_spec("boto3") is None or importlib.util.find_spec("aioboto3") is None:- lib_name = "boto3,aioboto3"- lib_extra_name = "bedrock"- msg = "The boto3 and aioboto3 SDKs are required to use Bedrock models."- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.bedrock.bedrock_factory import BedrockFactory # noqa: PLC0415- from pipelex.plugins.bedrock.bedrock_llm_worker import BedrockLLMWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=BedrockFactory.make_bedrock_client(plugin=plugin, backend=backend),- )-- llm_worker = BedrockLLMWorker(- sdk_instance=sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case "google":- if importlib.util.find_spec("google.genai") is None:- lib_name = "google-genai"- lib_extra_name = "google"- msg = "The google-genai SDK is required in order to use Google Gemini API directly."- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.google.google_factory import GoogleFactory # noqa: PLC0415- from pipelex.plugins.google.google_llm_worker import GoogleLLMWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=GoogleFactory.make_google_client(backend=backend),- )-- llm_worker = GoogleLLMWorker(- sdk_instance=sdk_instance,- inference_model=inference_model,- reporting_delegate=reporting_delegate,- )- case _:- msg = f"Plugin '{plugin}' is not supported"- raise NotImplementedError(msg)- return llm_worker+ make_worker = get_inference_backend_registry().lookup(family=InferenceFamily.LLM, sdk=model_handle.sdk)+ worker = make_worker(+ inference_model=inference_model,+ backend=backend,+ sdk_clients=get_sdk_client_manager().sdk_client_registry,+ reporting_delegate=reporting_delegate,+ )+ # The LLM family registry only ever holds LLM workers (the (family, sdk)+ # key guarantees it); the uniform MakeWorkerFn return type is widened to+ # InferenceWorkerAbstract, so narrow it back here.+ return cast("LLMWorkerInternalAbstract", worker)diff --git a/pipelex/cogt/model_backends/model_lists.py b/pipelex/cogt/model_backends/model_lists.pyindex f3b659ebf..818941032 100644--- a/pipelex/cogt/model_backends/model_lists.py+++ b/pipelex/cogt/model_backends/model_lists.py@@ -1,13 +1,10 @@from __future__ import annotations-import importlib.util-from rich.markup import escapefrom pipelex.cli.exceptions import PipelexCLIError-from pipelex.cogt.exceptions import ModelManagerError-from pipelex.exceptions import MissingDependencyError-from pipelex.hub import get_console, get_models_manager+from pipelex.cogt.exceptions import ModelListingUnsupportedError, ModelManagerError+from pipelex.hub import get_console, get_model_lister_registry, get_models_managerclass ModelLister:@@ -49,135 +46,32 @@ class ModelLister:models_by_sdk[sdk] = []models_by_sdk[sdk].append(model_name)- # Process each SDK separately+ # Process each SDK separately, dispatching to the lister its backend plugin registered.+ # A missing-extra guard lives inside each lister (it raises MissingDependencyError when+ # called without its SDK installed), so the loop names no integration.+ lister_registry = get_model_lister_registry()any_listed = Falseunsupported_sdks: list[str] = []for sdk in models_by_sdk:+ lister = lister_registry.get_optional(sdk=sdk)+ if lister is None:+ # No plugin registered a lister for this SDK — it does not support remote listing.+ unsupported_sdks.append(sdk)+ continuetry:- match sdk:- case "openai" | "azure_openai" | "openai_responses" | "azure_openai_responses":- from pipelex.plugins.openai.openai_list import list_openai_models # noqa: PLC0415-- await list_openai_models(- sdk=sdk,- backend_name=backend_name,- backend=backend,- flat=flat,- any_listed=any_listed,- )- any_listed = True-- case "anthropic":- if importlib.util.find_spec("anthropic") is None:- lib_name = "anthropic"- lib_extra_name = "anthropic"- msg = (- "The anthropic SDK is required in order to use Anthropic models via the anthropic client. "- "However, you can use Anthropic models through bedrock directly "- "by using the 'bedrock-anthropic-claude' llm family. (eg: bedrock-anthropic-claude)"- )- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.anthropic.anthropic_exceptions import AnthropicSDKUnsupportedError # noqa: PLC0415- from pipelex.plugins.anthropic.anthropic_list import list_anthropic_models # noqa: PLC0415-- try:- await list_anthropic_models(- sdk=sdk,- backend_name=backend_name,- backend=backend,- flat=flat,- any_listed=any_listed,- )- any_listed = True- except AnthropicSDKUnsupportedError:- unsupported_sdks.append(sdk)- continue-- case "mistral":- if importlib.util.find_spec("mistralai") is None:- lib_name = "mistralai"- lib_extra_name = "mistral"- msg = (- "The mistralai SDK is required in order to use Mistral models through the mistralai client. "- "However, you can use Mistral models through bedrock directly "- "by using the 'bedrock-mistral' llm family. (eg: bedrock-mistral-large)"- )- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.mistral.mistral_list import list_mistral_models # noqa: PLC0415-- list_mistral_models(- sdk=sdk,- backend_name=backend_name,- flat=flat,- any_listed=any_listed,- )- any_listed = True-- case "google":- if importlib.util.find_spec("google.genai") is None:- lib_name = "google-genai"- lib_extra_name = "google"- msg = "The google-genai SDK is required to use Google GenAI models."- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.google.google_list import list_google_models # noqa: PLC0415-- await list_google_models(- sdk=sdk,- backend_name=backend_name,- backend=backend,- flat=flat,- any_listed=any_listed,- )- any_listed = True-- case "bedrock" | "bedrock_aioboto3":- if importlib.util.find_spec("boto3") is None or importlib.util.find_spec("aioboto3") is None:- lib_name = "boto3,aioboto3"- lib_extra_name = "bedrock"- msg = "The boto3 and aioboto3 SDKs are required to use Bedrock models."- raise MissingDependencyError(- lib_name,- lib_extra_name,- msg,- )-- from pipelex.plugins.bedrock.bedrock_list import list_bedrock_models # noqa: PLC0415-- list_bedrock_models(- sdk=sdk,- backend_name=backend_name,- backend=backend,- flat=flat,- any_listed=any_listed,- )- any_listed = True-- case _:- # SDK doesn't support listing- unsupported_sdks.append(sdk)- continue-+ await lister(sdk=sdk, backend_name=backend_name, backend=backend, flat=flat, any_listed=any_listed)+ any_listed = True+ except ModelListingUnsupportedError:+ # The lister's client variant cannot enumerate models (e.g. a bedrock-backed+ # Anthropic client) — the same soft outcome as a missing lister.+ unsupported_sdks.append(sdk)+ continueexcept PipelexCLIError:raiseexcept Exception as exc:- # Per-SDK boundary: SDK list calls (plugin imports, network, remote APIs) have a non-enumerable- # failure surface; any failure is converted to PipelexCLIError with context. Re-raises, never swallows.+ # Case 2: lister dispatch is an unbounded surface (lazy plugin imports, network, remote+ # provider APIs); any failure is converted to PipelexCLIError with context. Never swallows.msg = f"Error listing models for SDK '{sdk}' in backend '{backend_name}': {exc}"raise PipelexCLIError(msg) from excdiff --git a/pipelex/cogt/search/search_worker_factory.py b/pipelex/cogt/search/search_worker_factory.pyindex dc83690fa..88e0ad2d2 100644--- a/pipelex/cogt/search/search_worker_factory.py+++ b/pipelex/cogt/search/search_worker_factory.py@@ -1,7 +1,11 @@+from typing import cast+from pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract-from pipelex.hub import get_models_manager, get_plugin_manager, get_report_delegate-from pipelex.plugins.plugin import Plugin+from pipelex.hub import get_inference_backend_registry, get_models_manager, get_sdk_client_manager+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandle+from pipelex.reporting.reporting_protocol import ReportingProtocolclass SearchWorkerFactory:@@ -9,39 +13,29 @@ class SearchWorkerFactory:def make_search_worker(cls,inference_model: InferenceModelSpec,+ *,+ reporting_delegate: ReportingProtocol | None = None,) -> SearchWorkerAbstract:- """Create a search worker for the given inference model.-- Discriminates on plugin.sdk to select the appropriate implementation.+ """Create a search worker for the given inference model via the inference-backend registry.Args:inference_model: The model spec from the backend configuration.+ reporting_delegate: The reporting delegate passed through to the worker+ (supplied by the caller — the factory no longer reaches into the hub).Returns:A SearchWorkerAbstract instance."""- plugin = Plugin.make_for_inference_model(inference_model=inference_model)+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)- plugin_sdk_registry = get_plugin_manager().plugin_sdk_registry- search_worker: SearchWorkerAbstract- match plugin.sdk:- case "linkup":- from pipelex.plugins.linkup.linkup_search_worker import LinkupSearchWorker # noqa: PLC0415-- search_worker = LinkupSearchWorker(inference_model=inference_model, reporting_delegate=get_report_delegate())- case "gateway_search":- from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415- from pipelex.plugins.gateway.gateway_search_worker import GatewaySearchWorker # noqa: PLC0415-- sdk_instance = plugin_sdk_registry.get_sdk_instance(plugin=plugin) or plugin_sdk_registry.set_sdk_instance(- plugin=plugin,- sdk_instance=GatewayFactory.make_portkey_client(backend=backend),- )- search_worker = GatewaySearchWorker(- sdk_instance=sdk_instance, inference_model=inference_model, reporting_delegate=get_report_delegate()- )- case _:- msg = f"Plugin '{plugin}' is not supported for search"- raise NotImplementedError(msg)-- return search_worker+ make_worker = get_inference_backend_registry().lookup(family=InferenceFamily.SEARCH, sdk=model_handle.sdk)+ worker = make_worker(+ inference_model=inference_model,+ backend=backend,+ sdk_clients=get_sdk_client_manager().sdk_client_registry,+ reporting_delegate=reporting_delegate,+ )+ # The SEARCH family registry only ever holds search workers (the+ # (family, sdk) key guarantees it); the uniform MakeWorkerFn return type is+ # widened to InferenceWorkerAbstract, so narrow it back here.+ return cast("SearchWorkerAbstract", worker)diff --git a/pipelex/core/memory/working_memory.py b/pipelex/core/memory/working_memory.pyindex c6a10da02..72ac82496 100644--- a/pipelex/core/memory/working_memory.py+++ b/pipelex/core/memory/working_memory.py@@ -469,8 +469,8 @@ class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):"""Serialize the working memory as a dictionary."""return self.model_dump(serialize_as_any=True)- def dump_for_temporal(self) -> dict[str, Any]:- """Serialize for Temporal transit with explicit ListContent representation.+ def dump_for_transport(self) -> dict[str, Any]:+ """Serialize for cross-process transport with explicit ListContent representation.Like smart_dump(), but serializes ListContent as a plain list instead of``{"items": [...]}``. This removes the ambiguity at hydration time: adiff --git a/pipelex/core/pipes/pipe_output.py b/pipelex/core/pipes/pipe_output.pyindex 5c329617f..cfd5cc231 100644--- a/pipelex/core/pipes/pipe_output.py+++ b/pipelex/core/pipes/pipe_output.py@@ -14,7 +14,6 @@ from pipelex.core.stuffs.stuff_content import StuffContentTypefrom pipelex.core.stuffs.text_and_images_content import TextAndImagesContentfrom pipelex.core.stuffs.text_content import TextContentfrom pipelex.graph.graphspec import GraphSpec-from pipelex.libraries.library_crate import LibraryCratefrom pipelex.pipeline.pipeline_models import SpecialPipelineIdfrom pipelex.reporting.reporting_types import AnyTokensUsage@@ -32,29 +31,6 @@ class PipeOutput(PipeOutputAbstract[WorkingMemory]):tokens_usages: list[AnyTokensUsage] | None = Noneusage_assembly_error: str | None = None- def prepare_for_temporal(self, library_crate: LibraryCrate | None = None) -> "PipeOutput":- """Dehydrate WorkingMemory to raw dict for Temporal transit.-- Returns a copy with working_memory serialized to a plain dict- (no dynamic class metadata), leaving the original unchanged.- The receiving side must call hydrate_working_memory() to reconstruct- the typed WorkingMemory after dynamic classes are registered.-- Symmetric with `PipeJob.prepare_for_temporal()`: when `library_crate`- is None, dehydration is a no-op — there are no dynamic concept classes- to round-trip, so the typed WorkingMemory can travel as-is.- """- if library_crate is None:- return self- if not self.working_memory.root:- return self- return self.model_copy(- update={- "working_memory_raw": self.working_memory.dump_for_temporal(),- "working_memory": WorkingMemory(),- }- )-@propertydef main_stuff(self) -> Stuff:return self.working_memory.get_main_stuff()diff --git a/pipelex/hub.py b/pipelex/hub.pyindex 5c0be9b91..4542cd18e 100644--- a/pipelex/hub.py+++ b/pipelex/hub.py@@ -31,7 +31,7 @@ from pipelex.libraries.pipe.pipe_library_abstract import PipeLibraryAbstractfrom pipelex.observer.observer_protocol import ObserverProtocolfrom pipelex.pipeline.pipeline import Pipelinefrom pipelex.pipeline.pipeline_manager_abstract import PipelineManagerAbstract-from pipelex.plugins.plugin_manager import PluginManager+from pipelex.plugins.sdk_client_manager import SdkClientManagerfrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.system.configuration.config_loader import config_managerfrom pipelex.system.configuration.config_root import ConfigRoot@@ -49,6 +49,10 @@ if TYPE_CHECKING:from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocolfrom pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol+ from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistry+ from pipelex.plugins.inference_backend_registry import InferenceBackendRegistry+ from pipelex.plugins.model_lister_registry import ModelListerRegistry+ from pipelex.plugins.orchestrator_registry import OrchestratorRegistryfrom pipelex.tracing.event_log_protocol import EventLogProtocol@@ -71,7 +75,11 @@ class PipelexHub:self._func_registry: FuncRegistry | None = None# cogtself._models_manager: ModelManagerAbstract | None = None- self._plugin_manager: PluginManager | None = None+ self._sdk_client_manager: SdkClientManager | None = None+ self._inference_backend_registry: InferenceBackendRegistry | None = None+ self._model_lister_registry: ModelListerRegistry | None = None+ self._orchestrator_registry: OrchestratorRegistry | None = None+ self._bundle_validator_registry: BundleValidatorRegistry | None = Noneself._inference_manager: InferenceManagerProtocolself._report_delegate: ReportingProtocolself._content_generator: ContentGeneratorProtocol | None = None@@ -175,8 +183,20 @@ class PipelexHub:def set_models_manager(self, models_manager: ModelManagerAbstract):self._models_manager = models_manager- def set_plugin_manager(self, plugin_manager: PluginManager):- self._plugin_manager = plugin_manager+ def set_sdk_client_manager(self, sdk_client_manager: SdkClientManager):+ self._sdk_client_manager = sdk_client_manager++ def set_inference_backend_registry(self, inference_backend_registry: "InferenceBackendRegistry"):+ self._inference_backend_registry = inference_backend_registry++ def set_model_lister_registry(self, model_lister_registry: "ModelListerRegistry"):+ self._model_lister_registry = model_lister_registry++ def set_orchestrator_registry(self, orchestrator_registry: "OrchestratorRegistry"):+ self._orchestrator_registry = orchestrator_registry++ def set_bundle_validator_registry(self, bundle_validator_registry: "BundleValidatorRegistry"):+ self._bundle_validator_registry = bundle_validator_registrydef set_inference_manager(self, inference_manager: InferenceManagerProtocol):self._inference_manager = inference_manager@@ -286,11 +306,35 @@ class PipelexHub:raise RuntimeError(msg)return self._models_manager- def get_plugin_manager(self) -> PluginManager:- if self._plugin_manager is None:- msg = "PluginManager2 is not initialized"+ def get_sdk_client_manager(self) -> SdkClientManager:+ if self._sdk_client_manager is None:+ msg = "SdkClientManager is not initialized"+ raise RuntimeError(msg)+ return self._sdk_client_manager++ def get_inference_backend_registry(self) -> "InferenceBackendRegistry":+ if self._inference_backend_registry is None:+ msg = "InferenceBackendRegistry is not initialized"+ raise RuntimeError(msg)+ return self._inference_backend_registry++ def get_model_lister_registry(self) -> "ModelListerRegistry":+ if self._model_lister_registry is None:+ msg = "ModelListerRegistry is not initialized"+ raise RuntimeError(msg)+ return self._model_lister_registry++ def get_orchestrator_registry(self) -> "OrchestratorRegistry":+ if self._orchestrator_registry is None:+ msg = "OrchestratorRegistry is not initialized"raise RuntimeError(msg)- return self._plugin_manager+ return self._orchestrator_registry++ def get_bundle_validator_registry(self) -> "BundleValidatorRegistry":+ if self._bundle_validator_registry is None:+ msg = "BundleValidatorRegistry is not initialized"+ raise RuntimeError(msg)+ return self._bundle_validator_registrydef get_inference_manager(self) -> InferenceManagerProtocol:return self._inference_manager@@ -444,8 +488,24 @@ def get_model_deck() -> ModelDeck:return get_models_manager().get_model_deck()-def get_plugin_manager() -> PluginManager:- return get_pipelex_hub().get_plugin_manager()+def get_sdk_client_manager() -> SdkClientManager:+ return get_pipelex_hub().get_sdk_client_manager()+++def get_inference_backend_registry() -> "InferenceBackendRegistry":+ return get_pipelex_hub().get_inference_backend_registry()+++def get_model_lister_registry() -> "ModelListerRegistry":+ return get_pipelex_hub().get_model_lister_registry()+++def get_orchestrator_registry() -> "OrchestratorRegistry":+ return get_pipelex_hub().get_orchestrator_registry()+++def get_bundle_validator_registry() -> "BundleValidatorRegistry":+ return get_pipelex_hub().get_bundle_validator_registry()def get_inference_manager() -> InferenceManagerProtocol:diff --git a/pipelex/kit/agent_rules/commands.md b/pipelex/kit/agent_rules/commands.mdindex ae05e9bf9..a4ec2714d 100644--- a/pipelex/kit/agent_rules/commands.md+++ b/pipelex/kit/agent_rules/commands.md@@ -91,30 +91,6 @@```Note: `TEST=LF` (or `TEST=lf`) will use pytest's `--lf` flag instead of name filtering.-## Temporal Integration Test Options-- The Temporal integration tests support different server modes via the `--temporal-server` pytest CLI option:-- - `--temporal-server`: Which Temporal server to use- - `none` (default): in-process test server — no external dependencies, used in CI- - `time-skipping`: in-process server with deterministic time control- - A profile name from `temporal_server_configs` in `pipelex.toml` (e.g. `local`, `testing`): connects to a real Temporal server using the profile's host, namespace, and API key settings-- ```bash- # CI default: in-process server- .venv/bin/pytest tests/integration/pipelex/temporal/-- # Dev with local Temporal server- .venv/bin/pytest tests/integration/pipelex/temporal/ \- --temporal-server local-- # Dev with cloud/testing server- .venv/bin/pytest tests/integration/pipelex/temporal/ \- --temporal-server testing- ```------## Prerequisites for running command lines: use virtual environment**CRITICAL**: Before running any `pipelex` commands or `pytest`, you MUST use the appropriate Python virtual environment. The only exceptions are our `make` commands which already include the env activation.diff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.tomlindex de39d5c70..75af5d268 100644--- a/pipelex/kit/configs/pipelex.toml+++ b/pipelex/kit/configs/pipelex.toml@@ -223,130 +223,10 @@ is_dump_response_enabled = falseis_dump_error_enabled = false####################################################################################################-# Temporal Config+# Plugins####################################################################################################-[temporal]-is_enabled = false--# Custom Temporal search attributes Pipelex populates on every workflow start-# so the dashboard can group/filter by PipeCode, PipelineRunId, SessionId,-# UserId, and DomainCode. Set `enabled = false` to skip both the worker-boot-# registration check and the per-workflow attribute attachment — useful when-# the operator owns the namespace policy and can't grant search-attribute-# admin to the worker.-[temporal.search_attributes]-enabled = false-attributes = ["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"]--[temporal.payload_codec_config]-is_enabled = false-size_threshold = 1_048_576 # in bytes (~1 MB, well under Temporal's 2 MB hard limit)-storage_prefix = "temporal-payloads/"--[temporal.payload_codec_config.storage_provider_config]-method = "local"--[temporal.payload_codec_config.storage_provider_config.local]-uri_format = "{hash}"-local_storage_path = ".pipelex/temporal-payload-store"--[temporal.worker_config]-default_task_queue = "temporal_task_queue"-workflow_execution_timeout = "1:00:00" # workflow-level budget (not the activity timeout)-run_timeout = "1:00:00"-task_timeout = "0:00:10"-start_delay = "0:00:00"-rpc_timeout = "1:00:00"-# Baseline activity timeout. Per-queue and per-handle overlays in-# [temporal.queue_options] / activity_queues.*.handle_options can override.-default_activity_start_to_close_timeout = "0:10:00"-# Per-activity, per-handle routing. Empty by default — every activity lands on-# `default_task_queue` above (same as legacy behavior). Populate this to route-# activities to dedicated worker pools (e.g. OpenAI vs Anthropic, a heavy-payload-# image pool, isolated Mistral OCR), with optional per-handle queue overrides and-# rare per-handle option overrides. See pipelex/kit/configs/pipelex.toml for the-# full schema and examples.-[temporal.worker_config.activity_queues]--[temporal.worker_config.retry_policy_config]-initial_interval = "0:00:03"-backoff_coefficient = 2.0-maximum_interval = "unlimited"-maximum_attempts = 3-non_retryable_error_types = [- "DryRunMockBuildError",- "DryRunObjectFidelityError",- "ExtractHandleNotFoundError",- "FileNotFoundError",- "ImgGenHandleNotFoundError",- "LLMHandleNotFoundError",- "ModelNotFoundError",- "ValidationError",-]-# Per-queue / per-handle overlays add to the list above via the layer-specific-# `non_retryable_error_types_extra` field (see the commented example below).-# That field belongs only on overlays — the baseline schema deliberately does-# not accept it.--# Per-queue submitter options. Layer on top of worker_config defaults at-# dispatch time. Add `[temporal.queue_options.<queue>]` entries to match each-# backend pool's latency/retry profile (timeouts, retry policy, cluster-wide-# rate caps). Ships empty to match the package default — see the rationale in-# pipelex/pipelex.toml. Each deployment configures the overlay for whatever-# name it picked for `default_task_queue`.-[temporal.queue_options]--# Named worker-runtime profiles. A worker process selects one via `--profile`.-# The `default` profile below mirrors the pre-v2 hardcoded knobs, so default-# behavior is unchanged. Add more profiles to shape pools differently — e.g.-# small concurrency on a GPU pool sitting next to a high-throughput LLM pool.-# See pipelex/kit/configs/pipelex.toml for examples.-[temporal.worker_runtime_profiles]-default_profile = "default"--[temporal.worker_runtime_profiles.profiles.default]-tuning_mode = "explicit"-max_cached_workflows = 10000-max_concurrent_workflow_tasks = 1000-max_concurrent_activities = 1000-max_concurrent_local_activities = 1000-max_concurrent_workflow_task_polls = 100-max_concurrent_activity_task_polls = 100-max_activities_per_second = 1000-sticky_queue_schedule_to_start_timeout = "0:30:00"-max_heartbeat_throttle_interval = "1:00:00"-default_heartbeat_throttle_interval = "1:00:00"-graceful_shutdown_timeout = "0:30:00"--[temporal.temporal_config]-selected_server = "local"--[temporal.temporal_config.temporal_server_configs.local]-description = "default localhost"-target_host = "localhost:7233"-namespace = "default"-api_key_method = "none"-api_key_id = ""--[temporal.temporal_config.temporal_server_configs.testing]-description = "Temporal server for testing"-target_host = "${TESTING_TEMPORAL_TARGET_HOST}"-namespace = "${TESTING_TEMPORAL_NAMESPACE}"-api_key_method = "secret_provider"-api_key_id = "${TESTING_TEMPORAL_MAIN_API_KEY}"--[temporal.temporal_config.temporal_log_config]-is_workflow_info_on_message = false-is_workflow_info_on_extra = true-is_full_workflow_info_on_extra = false-is_activity_info_on_message = false-is_activity_info_on_extra = true-is_full_activity_info_on_extra = false-is_formatter_enabled = true-is_prefix_enabled = true-managed_loggers = [- "temporalio.activity",- "temporalio.workflow",- "temporalio.worker._activity",-]+[plugins]+# Names of discovered plugins to skip at startup (a denylist). Discovery is the+# source of truth for which plugins are present; this only turns one off.+disabled = []diff --git a/pipelex/pipe_operators/llm/pipe_llm.py b/pipelex/pipe_operators/llm/pipe_llm.pyindex 726497fa0..8760c89a6 100644--- a/pipelex/pipe_operators/llm/pipe_llm.py+++ b/pipelex/pipe_operators/llm/pipe_llm.py@@ -212,8 +212,8 @@ class PipeLLM(PipeOperator[PipeLLMOutput]):if (not self.llm_prompt_spec.templating_style) and (inference_model := model_deck.get_optional_inference_model(model_handle=llm_setting_main.model, model_type=ModelType.LLM)):- # Note: the case where we don't get an inference model corresponds to the use of an external LLM Plugin- # TODO: improve this by making it possible to get the inference model for external LLM Plugins+ # Note: the case where we don't get an inference model corresponds to the use of an external LLM plugin+ # TODO: improve this by making it possible to get the inference model for external LLM pluginsprompting_target = llm_setting_main.prompting_target or inference_model.prompting_targetself.llm_prompt_spec.templating_style = get_config().pipelex.prompting_config.get_prompting_style(prompting_target=prompting_target,diff --git a/pipelex/pipe_run/pipe_job.py b/pipelex/pipe_run/pipe_job.pyindex 1c533bd76..7e44dbde9 100644--- a/pipelex/pipe_run/pipe_job.py+++ b/pipelex/pipe_run/pipe_job.py@@ -30,20 +30,3 @@ class PipeJob(BaseModel):msg = "WorkingMemory is in raw form and has not been hydrated yet"raise PipeJobError(msg)return WorkingMemory()-- def prepare_for_temporal(self) -> "PipeJob":- """Return a PipeJob ready for Temporal dispatch, with WorkingMemory serialized to raw dict.-- Returns a copy when mutation is needed, leaving the original unchanged.- Returns self when no transformation is required.- """- if self.library_crate is None:- return self- if self.working_memory is not None:- return self.model_copy(- update={- "working_memory_raw": self.working_memory.dump_for_temporal(),- "working_memory": None,- }- )- return selfdiff --git a/pipelex/pipelex.py b/pipelex/pipelex.pyindex 521bb19b7..d868b6cb4 100644--- a/pipelex/pipelex.py+++ b/pipelex/pipelex.py@@ -1,7 +1,8 @@import typesimport warnings+from collections.abc import Callablefrom pathlib import Path-from typing import TYPE_CHECKING, Any, cast+from typing import TYPE_CHECKING, Any, TypeVar, castfrom kajson.class_registry import ClassRegistryfrom kajson.class_registry_abstract import ClassRegistryAbstract@@ -47,7 +48,13 @@ from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocolfrom pipelex.pipe_run.pipe_run import PipeRunfrom pipelex.pipeline.pipeline_manager import PipelineManagerfrom pipelex.pipeline.pipeline_manager_abstract import PipelineManagerAbstract-from pipelex.plugins.plugin_manager import PluginManager+from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistry+from pipelex.plugins.discovery import build_registrar+from pipelex.plugins.inference_backend_registry import InferenceBackendRegistry+from pipelex.plugins.model_lister_registry import ModelListerRegistry+from pipelex.plugins.orchestrator_registry import OrchestratorRegistry+from pipelex.plugins.registrar import HubSlot, PluginRegistrar+from pipelex.plugins.sdk_client_manager import SdkClientManagerfrom pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.reporting.reporting_protocol import ReportingNoOp, ReportingProtocolfrom pipelex.system.configuration.config_loader import config_manager@@ -92,6 +99,8 @@ if TYPE_CHECKING:PACKAGE_NAME, PACKAGE_VERSION = get_package_info()+_HubSlotImplT = TypeVar("_HubSlotImplT")+class Pipelex(metaclass=MetaSingleton):def __init__(@@ -125,9 +134,12 @@ class Pipelex(metaclass=MetaSingleton):# toolsself.class_registry: ClassRegistryAbstract | None = Noneself.func_registry: FuncRegistry | None = None+ # plugins — the registrar is built in setup() (build_registrar); held for the slot-claim /+ # CLI-command / teardown apply-points. Declared here so teardown() can guard on it.+ self._plugin_registrar: PluginRegistrar | None = None# cogt- self.plugin_manager = PluginManager()- self.pipelex_hub.set_plugin_manager(self.plugin_manager)+ self.sdk_client_manager = SdkClientManager()+ self.pipelex_hub.set_sdk_client_manager(self.sdk_client_manager)self.reporting_delegate: ReportingProtocol | None = Noneself.telemetry_manager: TelemetryManagerAbstract | None = None@@ -167,7 +179,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.*,integration_mode: IntegrationMode,needs_inference: bool = True,- temporal_enabled: bool | None = None,+ boot_orchestrator: str | None = None,needs_model_specs: bool | None = None,class_registry: ClassRegistryAbstract | None = None,secrets_provider: SecretsProviderAbstract | None = None,@@ -189,13 +201,11 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.msg = f"The base setup method does not support any additional arguments: {kwargs}"raise PipelexSetupError(msg)- # Override temporal.is_enabled if temporal_enabled is explicitly provided- if temporal_enabled is not None:- config = get_config()- updated_temporal = config.temporal.model_copy(update={"is_enabled": temporal_enabled})- config.temporal = updated_temporal-- self._temporal_task_manager: object | None = None+ # Boot this process under the named orchestrator plugin, when explicitly provided.+ # The matching boot-orchestrator plugin (e.g. Temporal) claims the hub slots in its+ # register() iff plugins.boot_orchestrator == its own name; any other value is in-process.+ if boot_orchestrator is not None:+ get_config().plugins.boot_orchestrator = boot_orchestrator# Initialize secrets provider early - needed for gateway checksecrets_provider = secrets_provider or EnvSecretsProvider()@@ -324,7 +334,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.# --- AI Models and Inference Management ------------------------------------------------- self.plugin_manager.setup()+ self.sdk_client_manager.setup()self.models_manager: ModelManagerAbstract = models_manager or ModelManager()self.pipelex_hub.set_models_manager(models_manager=self.models_manager)@@ -372,17 +382,29 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.# backend-keyed unconditionally (eng review D4) — a keyless Temporal submitter must still# dispatch activities and mock inside them, so `needs_inference` plays no role in picking the# generator. Its other boot roles (gateway/model setup, credentials, telemetry) are unchanged.+ # --- Plugin discovery -----------------------------------------------------------------+ # Build the plugin registrar from the fully-resolved config (pure and import-light:+ # registering the built-ins imports no backend SDK). Runs after the gateway/model setup+ # checks and before the hub setup points below — the family worker factories look their+ # backends up on the hub registries at run time. The slot-claim thunks, CLI commands and+ # teardown callbacks the registrar also accumulates are applied at their ordered+ # apply-points in a later phase (no orchestrator plugin contributes any of them yet).+ plugin_registrar = build_registrar(config=get_config())+ self._plugin_registrar = plugin_registrar+ self.pipelex_hub.set_inference_backend_registry(InferenceBackendRegistry(plugin_registrar.inference_backends))+ self.pipelex_hub.set_model_lister_registry(ModelListerRegistry(plugin_registrar.model_listers))+ self.pipelex_hub.set_orchestrator_registry(OrchestratorRegistry(plugin_registrar.orchestrators))+ self.pipelex_hub.set_bundle_validator_registry(BundleValidatorRegistry(plugin_registrar.bundle_validators))+self.pipelex_hub.set_dry_run_forced(not needs_inference)+ # Injection precedence (codex C8): explicit setup() param > plugin slot-claim thunk > core default.+ # A boot-orchestrator plugin (Temporal worker) claims the CONTENT_GENERATOR slot; its thunk runs+ # only here, never at register, so booting a non-worker process imports no host-runtime SDK.if content_generator is None:- if get_config().temporal.is_enabled:- from pipelex.temporal.tprl_content_generation.content_generator_in_workflow_factory import ( # noqa: PLC0415- ContentGeneratorInWorkflowFactory,- )-- content_generator = ContentGeneratorInWorkflowFactory.make_content_generator_in_workflow()- else:- generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)- content_generator = ContentGenerator(generated_content_factory=generated_content_factory)+ content_generator = self._resolve_hub_slot(+ slot=HubSlot.CONTENT_GENERATOR,+ default=lambda: ContentGenerator(generated_content_factory=GeneratedContentFactory(storage_provider=storage_provider)),+ )self.pipelex_hub.set_content_generator(content_generator)self.inference_manager = inference_manager or InferenceManager()@@ -428,54 +450,54 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.multi_observer = MultiObserver(observers=observers)self.pipelex_hub.set_observer(observer=multi_observer)- # --- Temporal --------------------------------------------------------------------------+ # --- Task Manager ----------------------------------------------------------------------+ # The TASK_MANAGER slot is claimed only by a boot-orchestrator plugin running this process as+ # its runtime (a Temporal worker). The thunk does the full wiring on the plugin's own hub and+ # is torn down via a registered teardown callback (LIFO) — no core default, no explicit param.- if get_config().temporal.is_enabled:- from pipelex.temporal.tasks import Tasks # noqa: PLC0415- from pipelex.temporal.temporal_hub import temporal_hub # noqa: PLC0415- from pipelex.temporal.temporal_task_manager import TemporalTaskManager # noqa: PLC0415-- temporal_task_manager = TemporalTaskManager()- temporal_hub.set_task_manager(temporal_task_manager)- temporal_task_manager.complement_catalog(- extra_catalog=Tasks.TASK_PACKS,- extra_workflows=[],- extra_activities=[],- )- temporal_task_manager.setup()- self._temporal_task_manager = temporal_task_manager+ task_manager_factory = plugin_registrar.slot_claims.get(HubSlot.TASK_MANAGER)+ if task_manager_factory is not None:+ task_manager_factory()# --- Pipe Router -----------------------------------------------------------------------+ # Injection precedence (codex C8): explicit setup() param > plugin slot-claim thunk > core default.if pipe_router:self.pipelex_hub.set_pipe_router(pipe_router)- elif get_config().temporal.is_enabled:- from pipelex.temporal.tprl_pipe.temporal_pipe_router import make_temporal_pipe_router # noqa: PLC0415-- self.pipelex_hub.set_pipe_router(make_temporal_pipe_router())else:- self.pipelex_hub.set_pipe_router(PipeRouter(observer=multi_observer))+ self.pipelex_hub.set_pipe_router(self._resolve_hub_slot(slot=HubSlot.PIPE_ROUTER, default=lambda: PipeRouter(observer=multi_observer)))# --- Pipe Run --------------------------------------------------------------------------+ # No explicit param for pipe_run: plugin slot-claim thunk > core default.- if get_config().temporal.is_enabled:- from pipelex.temporal.tprl_pipe.temporal_pipe_run import make_temporal_pipe_run # noqa: PLC0415-- self.pipelex_hub.set_pipe_run(make_temporal_pipe_run())- else:- self.pipelex_hub.set_pipe_run(PipeRun(pipe_router=self.pipelex_hub.get_required_pipe_router()))+ self.pipelex_hub.set_pipe_run(+ self._resolve_hub_slot(slot=HubSlot.PIPE_RUN, default=lambda: PipeRun(pipe_router=self.pipelex_hub.get_required_pipe_router()))+ )log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} setup done")- def teardown(self):- # temporal- if self._temporal_task_manager is not None:- from pipelex.temporal.temporal_hub import temporal_hub # noqa: PLC0415- from pipelex.temporal.temporal_task_manager import TemporalTaskManager # noqa: PLC0415+ def _resolve_hub_slot(self, *, slot: HubSlot, default: Callable[[], _HubSlotImplT]) -> _HubSlotImplT:+ """Resolve a process-global hub slot: a plugin's claimed thunk if present, else the core default.++ Call sites apply explicit-injection precedence first (an explicit ``setup()`` param wins),+ so this only arbitrates plugin-claim vs core-default. The claim is a thunk invoked here at the+ boot apply-point — never during ``register`` — so a non-worker boot constructs no plugin impl.+ The return type follows the core ``default`` (the claimed thunk is typed ``Any`` and adopts it).+ """+ if self._plugin_registrar is not None:+ factory = self._plugin_registrar.slot_claims.get(slot)+ if factory is not None:+ return cast("_HubSlotImplT", factory())+ return default()- if isinstance(self._temporal_task_manager, TemporalTaskManager):- self._temporal_task_manager.teardown()- temporal_hub.reset()+ def teardown(self):+ # Plugin-contributed teardown callbacks (LIFO) — e.g. a Temporal worker tears down its task+ # manager + resets its hub. Names no integration: the callbacks were registered by whichever+ # boot-orchestrator plugin claimed the runtime, and run before core teardown so a worker's+ # in-flight Temporal resources release first.+ if self._plugin_registrar is not None:+ for teardown_callback in reversed(self._plugin_registrar.teardown_callbacks):+ teardown_callback()# pipelexself.pipeline_manager.teardown()@@ -486,7 +508,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.self.inference_manager.teardown()if self.reporting_delegate:self.reporting_delegate.teardown()- self.plugin_manager.teardown()+ self.sdk_client_manager.teardown()# toolsself.kajson_manager.teardown()@@ -514,7 +536,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.*,integration_mode: IntegrationMode = IntegrationMode.PYTHON,needs_inference: bool = True,- temporal_enabled: bool | None = None,+ boot_orchestrator: str | None = None,needs_model_specs: bool | None = None,class_registry: ClassRegistryAbstract | None = None,secrets_provider: SecretsProviderAbstract | None = None,@@ -547,8 +569,10 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.deck validation. Useful for commands like validate/show that don't call inferenceAPIs. Generator selection stays backend-keyed. Submitter-side contract only: it doesnot constrain work this process executes as a Temporal worker.- temporal_enabled: When provided, overrides the temporal.is_enabled config value.- True forces Temporal workflow execution, False forces direct execution.+ boot_orchestrator: When provided, boots this process under the orchestrator plugin+ of this name (e.g. "temporal" to run pipes through the Temporal worker runtime).+ Any other value (or None) leaves execution in-process. Core names no orchestrator;+ the matching plugin gates on its own name.needs_model_specs: When True, load real model specs even if needs_inferenceis False. When None (default), follows needs_inference. Useful for validatecommands that need gateway-provided model specs without enabling full inference.@@ -588,7 +612,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.pipelex_instance.setup(integration_mode=integration_mode,needs_inference=needs_inference,- temporal_enabled=temporal_enabled,+ boot_orchestrator=boot_orchestrator,needs_model_specs=needs_model_specs,class_registry=class_registry,secrets_provider=secrets_provider,diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex 5d6fef5d6..a2e199ae7 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -446,238 +446,7 @@ preferred_agent_targets = ["claude", "agents"][migration.migration_maps.telemetry]-####################################################################################################-# Temporal Config-####################################################################################################--[temporal]-is_enabled = false--[temporal.search_attributes]-# Master toggle. When false, workflow starts don't attach any custom search-# attributes, the worker-boot check is skipped, and the dashboard view falls-# back to WorkflowType / WorkflowId / StartTime only.-#-# Default off: custom search attributes must be pre-registered on the target-# Temporal namespace, and `temporal server start-dev` does not auto-register-# them — so a default-on baseline makes every local dev loop fail on the-# first workflow start. Deployments that have registered the attributes-# (Temporal Cloud namespaces, self-hosted clusters with the migration run)-# opt in by setting `enabled = true` in their env override.-enabled = false-# Subset of the five built-in attributes to populate. Names not in this list-# are skipped at workflow-start time AND not required at worker boot. Pipelex-# only knows how to populate these five; arbitrary custom names are out of-# scope (they would require code that knows the value source).-attributes = ["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"]--[temporal.payload_codec_config]-is_enabled = false-size_threshold = 1_048_576-storage_prefix = "temporal-payloads/"--[temporal.payload_codec_config.storage_provider_config]-method = "local"--[temporal.payload_codec_config.storage_provider_config.local]-uri_format = "{hash}"-local_storage_path = ".pipelex/temporal-payload-store"--[temporal.worker_config]-default_task_queue = "temporal_task_queue"-workflow_execution_timeout = "1:00:00" # for the workflow itself (NOT used as activity timeout)-run_timeout = "1:00:00"-task_timeout = "0:00:10"-start_delay = "0:00:00"-rpc_timeout = "1:00:00"-# Baseline activity timeout used when no per-queue or per-handle overlay applies.-default_activity_start_to_close_timeout = "0:10:00"--[temporal.worker_config.activity_queues]--[temporal.worker_config.retry_policy_config]-initial_interval = "0:00:03"-backoff_coefficient = 2.0-maximum_interval = "unlimited"-maximum_attempts = 3-non_retryable_error_types = [- "DryRunMockBuildError",- "DryRunObjectFidelityError",- "ExtractHandleNotFoundError",- "FileNotFoundError",- "ImgGenHandleNotFoundError",- "LLMHandleNotFoundError",- "ModelNotFoundError",- "ValidationError",-]-# Per-queue and per-handle overlays contribute `non_retryable_error_types_extra`-# lists; the dispatch resolver unions them on top of the baseline above. That-# field belongs only on overlays — adding it here would be silently dropped by-# the resolver, so the baseline schema (RetryPolicyConfig) deliberately does-# not accept it.--# Per-queue submitter options. Operators populate this when a backend pool-# needs its own timeout or retry tuning. See pipelex/kit/configs/pipelex.toml-# for examples.-#-# The baseline ships no entries here. Each deployment is expected to add its-# own ``[temporal.queue_options.<queue>]`` block keyed by whatever name it-# chose for ``default_task_queue`` (and any extra queues referenced by-# ``activity_queues``). Shipping an opinionated entry for the baseline's-# ``temporal_task_queue`` name would create an orphan overlay the moment a-# downstream override renames the queue — and config merging cannot delete-# baseline keys, so that orphan would be unrecoverable from the override.-[temporal.queue_options]--# Named worker-runtime profiles. A worker process selects one via --profile.-# Defaults are lifted verbatim from the pre-v2 hardcoded Worker(...) construction-# so out-of-the-box behavior is unchanged.-[temporal.worker_runtime_profiles]-default_profile = "default"--[temporal.worker_runtime_profiles.profiles.default]-tuning_mode = "explicit"-max_cached_workflows = 10000-max_concurrent_workflow_tasks = 1000-max_concurrent_activities = 1000-max_concurrent_local_activities = 1000-max_concurrent_workflow_task_polls = 100-max_concurrent_activity_task_polls = 100-max_activities_per_second = 1000-sticky_queue_schedule_to_start_timeout = "0:30:00"-max_heartbeat_throttle_interval = "1:00:00"-default_heartbeat_throttle_interval = "1:00:00"-graceful_shutdown_timeout = "0:30:00"--[temporal.temporal_config]-selected_server = "local"--[temporal.temporal_config.temporal_server_configs.local]-description = "default localhost"-target_host = "localhost:7233"-namespace = "default"-api_key_method = "none"-api_key_id = ""--[temporal.temporal_config.temporal_server_configs.testing]-description = "Temporal server for testing"-target_host = "${TESTING_TEMPORAL_TARGET_HOST}"-namespace = "${TESTING_TEMPORAL_NAMESPACE}"-api_key_method = "secret_provider"-api_key_id = "${TESTING_TEMPORAL_MAIN_API_KEY}"--[temporal.temporal_config.temporal_log_config]-is_workflow_info_on_message = false-is_workflow_info_on_extra = true-is_full_workflow_info_on_extra = false-is_activity_info_on_message = false-is_activity_info_on_extra = true-is_full_activity_info_on_extra = false-is_formatter_enabled = true-is_prefix_enabled = true-managed_loggers = [- "temporalio.activity",- "temporalio.workflow",- "temporalio.worker._activity",-]-# When true, every workflow.execute_activity dispatch logs the resolved queue +-# timeout + retry along with the layer each value came from (baseline /-# queue_options / handle_options). Verbose — enable when debugging mis-tuned-# timeouts.-is_dispatch_resolution_traced = false--[temporal.worker_scopes]-default_scope = "full"--# Default — single worker handling everything.-[temporal.worker_scopes.scopes.full]-required_tasks_packs = ["crafting", "pipe"]-required_workflows = []-required_activities = []-excluded_workflows = []-excluded_activities = []-disable_all_workflows = false-disable_all_activities = false--# Workflow-only worker. Pair with `runner` on the same task queue to force-# activity dispatch to a different process (deterministic cross-worker hops).-[temporal.worker_scopes.scopes.router]-required_tasks_packs = ["crafting", "pipe"]-required_workflows = []-required_activities = []-excluded_workflows = []-excluded_activities = []-disable_all_workflows = false-disable_all_activities = true--# Activity-only worker. Counterpart to `router`.-[temporal.worker_scopes.scopes.runner]-required_tasks_packs = ["crafting", "pipe"]-required_workflows = []-required_activities = []-excluded_workflows = []-excluded_activities = []-disable_all_workflows = true-disable_all_activities = false--# Specialized activity-only scopes for deployment manifests with one worker pool-# per backend class. Each registers only the activities its name implies — paired-# with `--task-queue <q>` so the worker pulls only its class of work.-#-# Note: `act_render_page_views` is intentionally listed under BOTH `runner-img-gen`-# (used by single-image / image-list generation paths that may render a layout view)-# AND `runner-extract` (used by the `make_extract_pages` two-activity branch that-# pairs OCR with page-view rendering). Belt-and-suspenders: both scopes can render-# pages, so neither breaks if routing sends the activity to either.--[temporal.worker_scopes.scopes.runner-llm]-required_tasks_packs = []-required_workflows = []-required_activities = [- "act_llm_gen_text",- "act_llm_gen_object",- "act_llm_gen_object_list",-]-excluded_workflows = []-excluded_activities = []-disable_all_workflows = true-disable_all_activities = false--[temporal.worker_scopes.scopes.runner-img-gen]-required_tasks_packs = []-required_workflows = []-required_activities = ["act_img_gen_images", "act_render_page_views"]-excluded_workflows = []-excluded_activities = []-disable_all_workflows = true-disable_all_activities = false--[temporal.worker_scopes.scopes.runner-extract]-required_tasks_packs = []-required_workflows = []-required_activities = ["act_extract_gen_extract_pages", "act_render_page_views"]-excluded_workflows = []-excluded_activities = []-disable_all_workflows = true-disable_all_activities = false--[temporal.worker_scopes.scopes.runner-jinja2]-required_tasks_packs = []-required_workflows = []-required_activities = ["act_jinja2_gen_text"]-excluded_workflows = []-excluded_activities = []-disable_all_workflows = true-disable_all_activities = false--[temporal.worker_scopes.scopes.runner-search]-required_tasks_packs = []-required_workflows = []-required_activities = [- "act_search_gen_sourced_answer",- "act_search_gen_structured",-]-excluded_workflows = []-excluded_activities = []-disable_all_workflows = true-disable_all_activities = false+[plugins]+# Names of discovered plugins to skip at startup (a denylist). Discovery is the+# source of truth for which plugins are present; this only turns one off.+disabled = []diff --git a/pipelex/pipeline/bundle_validator.py b/pipelex/pipeline/bundle_validator.pyindex 512b2612a..07f214e28 100644--- a/pipelex/pipeline/bundle_validator.py+++ b/pipelex/pipeline/bundle_validator.py@@ -107,7 +107,7 @@ class BundleValidator:# Keep the router instance (don't discard it inside PipeRun): the sweep installs it as the active# router via scoped_pipe_router (see validate_pipes) so nested controller sub-pipes — which# dispatch through get_pipe_router() — resolve THIS in-process router instead of the hub default.- # Mirrors runtime_bridge.bridge._run_direct.+ # Mirrors runtime_bridge.direct_orchestrator.DirectOrchestrator.run.self._pipe_router = PipeRouter(observer=ObserverNoOp())self._pipe_run: PipeRunProtocol = PipeRun(pipe_router=self._pipe_router)@@ -244,7 +244,7 @@ class BundleValidator:# back to the hub default. Under a Temporal-enabled hub the default is the Temporal router, so# without the scope a controller (a PipeBatch/PipeParallel fan-out, or any PipeSequence step)# would leak its nested pipes to Temporal — turning a no-cost in-process dry run into real- # top-level workflow dispatches (HTTP 422). Mirrors runtime_bridge.bridge._run_direct. The+ # top-level workflow dispatches (HTTP 422). Mirrors runtime_bridge.direct_orchestrator.DirectOrchestrator.run. The# scope is contextvar-based, so concurrent /validate sweeps don't cross-contaminate, and the# asyncio tasks a batch fan-out spawns copy the context at creation (inside this scope) so they# inherit the override too.diff --git a/pipelex/pipeline/direct_bundle_validator.py b/pipelex/pipeline/direct_bundle_validator.pynew file mode 100644index 000000000..5b384c757--- /dev/null+++ b/pipelex/pipeline/direct_bundle_validator.py@@ -0,0 +1,56 @@+"""The core, always-on in-process bundle validator.++Validates MTHDS bundles in-process via ``validate_bundles_in_process``, the same sweep+the local protocol runtime uses. Returns the verdict as a value (decision A): the canonical+``PipelexValidationReport`` on success, or the structured ``ErrorReport`` an invalid bundle's+``ValidateBundleError`` projects. Any other exception propagates as a no-verdict infra fault.++Registered unconditionally by the core ``direct`` plugin alongside ``DirectOrchestrator``, so+the agnostic base gets in-process ``/validate`` "for free" through the same seam the Temporal+plugin contributes to. Import-light: core validation machinery only, no host-runtime SDK.+"""++from __future__ import annotations++from typing import TYPE_CHECKING++from pipelex.pipeline.exceptions import ValidateBundleError+from pipelex.pipeline.validate_in_process import validate_bundles_in_process++if TYPE_CHECKING:+ from collections.abc import Sequence+ from pathlib import Path++ from pipelex.base_exceptions import ErrorReport+ from pipelex.pipeline.validation_report import PipelexValidationReport+++class DirectBundleValidator:+ """In-process ``/validate``; loads and tears down the method library on the host.++ Satisfies ``BundleValidatorProtocol`` structurally. Returns the precise+ ``PipelexValidationReport`` on the valid arm (a covariant narrowing of the protocol's+ ``ValidationReport`` valid arm) so a caller holding the concrete class keeps full type+ information; the protocol stays widened to the leaf base to keep core import-acyclic.+ """++ async def validate_bundles(+ self,+ *,+ mthds_contents: list[str],+ mthds_sources: list[str] | None,+ allow_signatures: bool,+ library_dirs: Sequence[Path] | None,+ ) -> PipelexValidationReport | ErrorReport:+ try:+ return await validate_bundles_in_process(+ mthds_contents=mthds_contents,+ mthds_sources=mthds_sources,+ library_dirs=library_dirs,+ allow_signatures=allow_signatures,+ log_context="API validate",+ )+ except ValidateBundleError as exc:+ # An invalid bundle is a produced verdict, not a fault: surface its structured+ # report as the invalid arm. Any other exception (genuine infra fault) propagates.+ return exc.to_error_report()diff --git a/pipelex/plugins/anthropic/anthropic_factory.py b/pipelex/plugins/anthropic/anthropic_factory.pyindex c2950fa70..b00b2483c 100644--- a/pipelex/plugins/anthropic/anthropic_factory.py+++ b/pipelex/plugins/anthropic/anthropic_factory.py@@ -15,7 +15,7 @@ from pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategoryfrom pipelex.config import get_configfrom pipelex.plugins.anthropic.anthropic_exceptions import AnthropicFactoryError-from pipelex.plugins.plugin_sdk_registry import Plugin+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.tools.aws.aws_config import BedrockAccessVariantfrom pipelex.tools.uri.prepared_file import PreparedFile, PreparedFileBase64, PreparedFileHttpUrl, PreparedFileLocalPathfrom pipelex.types import StrEnum@@ -36,14 +36,14 @@ class AnthropicSdkVariant(StrEnum):class AnthropicFactory:@staticmethoddef make_anthropic_client(- plugin: Plugin,+ model_handle: ModelHandle,*,backend: InferenceBackend,) -> AsyncAnthropic | AsyncAnthropicBedrock:try:- sdk_variant = AnthropicSdkVariant(plugin.sdk)+ sdk_variant = AnthropicSdkVariant(model_handle.sdk)except ValueError as exc:- msg = f"Plugin '{plugin}' is not supported by AnthropicFactory"+ msg = f"ModelHandle '{model_handle}' is not supported by AnthropicFactory"raise AnthropicFactoryError(msg) from exc# Tier 1 transport retry: set the SDK client's retry budget explicitly from configdiff --git a/pipelex/plugins/anthropic/anthropic_list.py b/pipelex/plugins/anthropic/anthropic_list.pyindex 5d07d1426..138688fd2 100644--- a/pipelex/plugins/anthropic/anthropic_list.py+++ b/pipelex/plugins/anthropic/anthropic_list.py@@ -9,7 +9,7 @@ from rich.table import Tablefrom pipelex.cli.exceptions import PipelexCLIErrorfrom pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_console-from pipelex.plugins.plugin_sdk_registry import Plugin+from pipelex.plugins.model_handle import ModelHandleif TYPE_CHECKING:from anthropic.types import ModelInfo@@ -44,10 +44,10 @@ async def list_anthropic_models(from pipelex.plugins.anthropic.anthropic_llms import anthropic_list_available_models # noqa: PLC0415- plugin = Plugin(sdk=sdk, backend=backend_name)+ model_handle = ModelHandle(sdk=sdk, backend=backend_name)try:anthropic_models = await anthropic_list_available_models(- plugin=plugin,+ model_handle=model_handle,backend=backend,)diff --git a/pipelex/plugins/anthropic/anthropic_llms.py b/pipelex/plugins/anthropic/anthropic_llms.pyindex 621b027e7..4f1a31c93 100644--- a/pipelex/plugins/anthropic/anthropic_llms.py+++ b/pipelex/plugins/anthropic/anthropic_llms.py@@ -4,17 +4,17 @@ from anthropic.types import ModelInfofrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.plugins.anthropic.anthropic_exceptions import AnthropicModelListingError, AnthropicSDKUnsupportedErrorfrom pipelex.plugins.anthropic.anthropic_factory import AnthropicFactory-from pipelex.plugins.plugin import Plugin+from pipelex.plugins.model_handle import ModelHandle-async def anthropic_list_available_models(plugin: Plugin, *, backend: InferenceBackend) -> list[ModelInfo]:+async def anthropic_list_available_models(model_handle: ModelHandle, *, backend: InferenceBackend) -> list[ModelInfo]:"""List available Anthropic models.Returns:List[ModelInfo]: A list of Anthropic model information objects"""- anthropic_client = AnthropicFactory.make_anthropic_client(plugin=plugin, backend=backend)+ anthropic_client = AnthropicFactory.make_anthropic_client(model_handle=model_handle, backend=backend)if not hasattr(anthropic_client, "models"):msg = f"{type(anthropic_client).__name__} does not support listing models"raise AnthropicSDKUnsupportedError(msg)diff --git a/pipelex/plugins/anthropic/anthropic_plugin.py b/pipelex/plugins/anthropic/anthropic_plugin.pynew file mode 100644index 000000000..9b39e215c--- /dev/null+++ b/pipelex/plugins/anthropic/anthropic_plugin.py@@ -0,0 +1,72 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol++_ANTHROPIC_MISSING_MSG = (+ "The anthropic SDK is required in order to use Anthropic models via the anthropic client. "+ "However, you can use Anthropic models through bedrock directly "+ "by using the 'bedrock-anthropic-claude' llm family. (eg: bedrock-anthropic-claude)"+)+++def _make_anthropic_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="anthropic", extra="anthropic", msg=_ANTHROPIC_MISSING_MSG)++ from pipelex.plugins.anthropic.anthropic_factory import AnthropicFactory # noqa: PLC0415+ from pipelex.plugins.anthropic.anthropic_llm_worker import AnthropicLLMWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: AnthropicFactory.make_anthropic_client(model_handle=model_handle, backend=backend),+ )+ return AnthropicLLMWorker(+ sdk_instance=sdk_instance,+ extra_config=backend.extra_config,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++async def _list_anthropic_models(+ *,+ sdk: str,+ backend_name: str,+ backend: InferenceBackend,+ flat: bool,+ any_listed: bool,+) -> None:+ from pipelex.cogt.exceptions import ModelListingUnsupportedError # noqa: PLC0415+ from pipelex.plugins.anthropic.anthropic_exceptions import AnthropicSDKUnsupportedError # noqa: PLC0415+ from pipelex.plugins.anthropic.anthropic_list import list_anthropic_models # noqa: PLC0415++ try:+ await list_anthropic_models(sdk=sdk, backend_name=backend_name, backend=backend, flat=flat, any_listed=any_listed)+ except AnthropicSDKUnsupportedError as exc:+ # Translate the vendor "this client variant can't list" into the core soft signal the+ # list-models loop understands, so core names no Anthropic-specific exception.+ raise ModelListingUnsupportedError(sdk=sdk) from exc+++class AnthropicPlugin:+ """Built-in driver for Anthropic models via the anthropic SDK (also serves bedrock_anthropic)."""++ name = "anthropic"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="anthropic", make_worker=_make_anthropic_worker)+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="bedrock_anthropic", make_worker=_make_anthropic_worker)+ registrar.add_model_lister(sdk="anthropic", lister=_list_anthropic_models)diff --git a/pipelex/plugins/azure_rest/azure_img_gen_worker.py b/pipelex/plugins/azure_rest/azure_img_gen_worker.pyindex 5a23f2a7a..cad76f728 100644--- a/pipelex/plugins/azure_rest/azure_img_gen_worker.py+++ b/pipelex/plugins/azure_rest/azure_img_gen_worker.py@@ -23,7 +23,7 @@ from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategfrom pipelex.config import get_configfrom pipelex.hub import get_models_managerfrom pipelex.plugins.azure_rest.azure_exceptions import AzureCredentialsError-from pipelex.plugins.plugin import Plugin+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.tools.log.log import log@@ -31,17 +31,17 @@ from pipelex.tools.log.log import logclass AzureImgGenWorker(ImgGenWorkerAbstract):def __init__(self,- plugin: Plugin,+ model_handle: ModelHandle,inference_model: InferenceModelSpec,reporting_delegate: ReportingProtocol | None = None,):super().__init__(inference_model=inference_model, reporting_delegate=reporting_delegate)- if plugin.sdk != "azure_rest_img_gen":- msg = f"Plugin '{plugin}' is not supported for image generation"+ if model_handle.sdk != "azure_rest_img_gen":+ msg = f"ModelHandle '{model_handle}' is not supported for image generation"raise NotImplementedError(msg)- self.plugin = plugin- backend_name = self.plugin.backend+ self.model_handle = model_handle+ backend_name = self.model_handle.backendbackend = get_models_manager().get_required_inference_backend(backend_name)self.endpoint = backend.endpointself.api_version = backend.extra_config.get("api_version")diff --git a/pipelex/plugins/azure_rest/azure_rest_plugin.py b/pipelex/plugins/azure_rest/azure_rest_plugin.pynew file mode 100644index 000000000..962fae1b4--- /dev/null+++ b/pipelex/plugins/azure_rest/azure_rest_plugin.py@@ -0,0 +1,36 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol+++def _make_azure_rest_img_gen_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend, # noqa: ARG001 - uniform MakeWorkerFn signature; this REST worker builds its own client+ sdk_clients: SdkClientRegistry, # noqa: ARG001 - direct construction, no SDK-client caching+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.azure_rest.azure_img_gen_worker import AzureImgGenWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ return AzureImgGenWorker(+ model_handle=model_handle,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class AzureRestPlugin:+ """Built-in driver for Azure OpenAI image generation via the REST worker (no SDK client cache)."""++ name = "azure_rest"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.IMG_GEN, sdk="azure_rest_img_gen", make_worker=_make_azure_rest_img_gen_worker)diff --git a/pipelex/plugins/plugin_factory_abstract.py b/pipelex/plugins/backend_extras_factory.pysimilarity index 92%rename from pipelex/plugins/plugin_factory_abstract.pyrename to pipelex/plugins/backend_extras_factory.pyindex f122409c1..c1ec63b63 100644--- a/pipelex/plugins/plugin_factory_abstract.py+++ b/pipelex/plugins/backend_extras_factory.py@@ -5,7 +5,7 @@ from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstractfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec-class PluginFactoryAbstract(ABC):+class BackendExtrasFactory(ABC):@abstractmethoddef make_extras(self, inference_model: InferenceModelSpec, *, inference_job: InferenceJobAbstract, output_desc: strdiff --git a/pipelex/plugins/bedrock/bedrock_factory.py b/pipelex/plugins/bedrock/bedrock_factory.pyindex da869b226..a452aa490 100644--- a/pipelex/plugins/bedrock/bedrock_factory.py+++ b/pipelex/plugins/bedrock/bedrock_factory.py@@ -5,7 +5,7 @@ from pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.plugins.bedrock.bedrock_client_protocol import BedrockClientProtocolfrom pipelex.plugins.bedrock.bedrock_exceptions import BedrockFactoryErrorfrom pipelex.plugins.bedrock.bedrock_message import BedrockContentItem, BedrockMessage-from pipelex.plugins.plugin_sdk_registry import Plugin+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.types import StrEnum@@ -26,14 +26,14 @@ class BedrockFactory:@classmethoddef make_bedrock_client(cls,- plugin: Plugin,+ model_handle: ModelHandle,*,backend: InferenceBackend,) -> BedrockClientProtocol:try:- sdk_variant = BedrockSdkVariant(plugin.sdk)+ sdk_variant = BedrockSdkVariant(model_handle.sdk)except ValueError as exc:- msg = f"Plugin '{plugin}' is not supported by BedrockFactory"+ msg = f"ModelHandle '{model_handle}' is not supported by BedrockFactory"raise BedrockFactoryError(msg) from excbedrock_async_client: BedrockClientProtocoldiff --git a/pipelex/plugins/bedrock/bedrock_list.py b/pipelex/plugins/bedrock/bedrock_list.pyindex 8e1edc7d6..7c2920bab 100644--- a/pipelex/plugins/bedrock/bedrock_list.py+++ b/pipelex/plugins/bedrock/bedrock_list.py@@ -10,7 +10,7 @@ from pipelex.cli.exceptions import PipelexCLIErrorfrom pipelex.config import get_configfrom pipelex.exceptions import MissingDependencyErrorfrom pipelex.hub import get_console-from pipelex.plugins.plugin_sdk_registry import Plugin+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.tools.aws.exceptions import AwsCredentialsErrorif TYPE_CHECKING:@@ -40,7 +40,7 @@ def list_bedrock_models(from pipelex.plugins.bedrock.bedrock_llms import bedrock_list_available_models # noqa: PLC0415- plugin = Plugin(sdk=sdk, backend=backend_name)+ model_handle = ModelHandle(sdk=sdk, backend=backend_name)try:# Get AWS region for display@@ -51,9 +51,9 @@ def list_bedrock_models(raise PipelexCLIError(msg) from exctry:- # List available models using the plugin-specific function+ # List available models using the SDK-specific functionbedrock_models_list = bedrock_list_available_models(- plugin=plugin,+ model_handle=model_handle,backend=backend,)diff --git a/pipelex/plugins/bedrock/bedrock_llms.py b/pipelex/plugins/bedrock/bedrock_llms.pyindex 35c8beb17..0b3d36b7e 100644--- a/pipelex/plugins/bedrock/bedrock_llms.py+++ b/pipelex/plugins/bedrock/bedrock_llms.py@@ -3,19 +3,19 @@ from typing import Any, castfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.config import get_configfrom pipelex.exceptions import MissingDependencyError-from pipelex.plugins.plugin_sdk_registry import Plugin+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.tools.aws.exceptions import AwsCredentialsErrordef bedrock_list_available_models(- plugin: Plugin, # noqa: ARG001+ model_handle: ModelHandle, # noqa: ARG001*,backend: InferenceBackend, # noqa: ARG001) -> list[dict[str, Any]]:"""List available Bedrock foundation models.Args:- plugin: The plugin configuration (unused, kept for interface consistency)+ model_handle: The model handle (unused, kept for interface consistency)backend: The inference backend configuration (unused, kept for interface consistency)Returns:diff --git a/pipelex/plugins/bedrock/bedrock_plugin.py b/pipelex/plugins/bedrock/bedrock_plugin.pynew file mode 100644index 000000000..594587943--- /dev/null+++ b/pipelex/plugins/bedrock/bedrock_plugin.py@@ -0,0 +1,62 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol++_BEDROCK_MISSING_MSG = "The boto3 and aioboto3 SDKs are required to use Bedrock models."+++def _make_bedrock_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec=["boto3", "aioboto3"], extra="bedrock", msg=_BEDROCK_MISSING_MSG)++ from pipelex.plugins.bedrock.bedrock_factory import BedrockFactory # noqa: PLC0415+ from pipelex.plugins.bedrock.bedrock_llm_worker import BedrockLLMWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: BedrockFactory.make_bedrock_client(model_handle=model_handle, backend=backend),+ )+ return BedrockLLMWorker(+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++# Async to satisfy the uniform ListModelsFn contract (the loop awaits it) even though Bedrock lists synchronously.+async def _list_bedrock_models( # noqa: RUF029+ *,+ sdk: str,+ backend_name: str,+ backend: InferenceBackend,+ flat: bool,+ any_listed: bool,+) -> None:+ from pipelex.plugins.bedrock.bedrock_list import list_bedrock_models # noqa: PLC0415++ list_bedrock_models(sdk=sdk, backend_name=backend_name, backend=backend, flat=flat, any_listed=any_listed)+++class BedrockPlugin:+ """Built-in driver for Bedrock models via boto3/aioboto3."""++ name = "bedrock"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="bedrock_boto3", make_worker=_make_bedrock_worker)+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="bedrock_aioboto3", make_worker=_make_bedrock_worker)+ registrar.add_model_lister(sdk="bedrock", lister=_list_bedrock_models)+ registrar.add_model_lister(sdk="bedrock_aioboto3", lister=_list_bedrock_models)diff --git a/pipelex/plugins/blackboxai/blackboxai_plugin.py b/pipelex/plugins/blackboxai/blackboxai_plugin.pynew file mode 100644index 000000000..fdd183609--- /dev/null+++ b/pipelex/plugins/blackboxai/blackboxai_plugin.py@@ -0,0 +1,43 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol+++def _make_blackboxai_img_gen_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.blackboxai.blackboxai_completions_factory import BlackboxaiCompletionsFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_completions_img_gen_worker import OpenAICompletionsImgGenWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: OpenAIClientFactory.make_openai_client(model_handle=model_handle, backend=backend),+ )+ return OpenAICompletionsImgGenWorker(+ openai_completions_factory=BlackboxaiCompletionsFactory(is_http_url_enabled=True),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class BlackboxaiPlugin:+ """Built-in driver for Blackbox AI image generation via the OpenAI-completions substrate."""++ name = "blackboxai"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.IMG_GEN, sdk="blackboxai_img_gen", make_worker=_make_blackboxai_img_gen_worker)diff --git a/pipelex/plugins/builtins.py b/pipelex/plugins/builtins.pynew file mode 100644index 000000000..bbdbacaf9--- /dev/null+++ b/pipelex/plugins/builtins.py@@ -0,0 +1,52 @@+from pipelex.plugins.anthropic.anthropic_plugin import AnthropicPlugin+from pipelex.plugins.azure_rest.azure_rest_plugin import AzureRestPlugin+from pipelex.plugins.bedrock.bedrock_plugin import BedrockPlugin+from pipelex.plugins.blackboxai.blackboxai_plugin import BlackboxaiPlugin+from pipelex.plugins.contract import PipelexPlugin+from pipelex.plugins.direct.direct_plugin import DirectOrchestratorPlugin+from pipelex.plugins.docling.docling_plugin import DoclingPlugin+from pipelex.plugins.fal.fal_plugin import FalPlugin+from pipelex.plugins.gateway.gateway_plugin import GatewayPlugin+from pipelex.plugins.google.google_plugin import GooglePlugin+from pipelex.plugins.huggingface.huggingface_plugin import HuggingFacePlugin+from pipelex.plugins.linkup.linkup_plugin import LinkupPlugin+from pipelex.plugins.mistral.mistral_plugin import MistralPlugin+from pipelex.plugins.openai.openai_plugin import OpenAIPlugin+from pipelex.plugins.openrouter.openrouter_plugin import OpenRouterPlugin+from pipelex.plugins.portkey.portkey_plugin import PortkeyPlugin+from pipelex.plugins.pypdfium2.pypdfium2_plugin import Pypdfium2Plugin++# The plugins Pipelex ships with — discovered at boot ahead of any external+# entry-point plugin. Each is import-light: importing this module imports no+# backend SDK (the SDKs load lazily inside the make_worker closures).+#+# Temporal is NOT here: it ships as the external ``pipelex-temporal`` distribution,+# discovered via its ``pipelex.plugins`` entry point. Installing that dist makes the+# ``"temporal"`` orchestrator (and, when a process is booted under it via+# ``plugins.boot_orchestrator = "temporal"``, the worker runtime) available — zero+# config, no core import of ``temporalio``.+BUILTIN_PLUGINS: list[PipelexPlugin] = [+ DirectOrchestratorPlugin(),+ OpenAIPlugin(),+ GatewayPlugin(),+ PortkeyPlugin(),+ AnthropicPlugin(),+ MistralPlugin(),+ BedrockPlugin(),+ GooglePlugin(),+ FalPlugin(),+ HuggingFacePlugin(),+ BlackboxaiPlugin(),+ OpenRouterPlugin(),+ AzureRestPlugin(),+ DoclingPlugin(),+ Pypdfium2Plugin(),+ LinkupPlugin(),+]++# Built-in plugins that core requires unconditionally — naming one in+# ``plugins.disabled`` is a configuration error, not a no-op. ``direct`` owns the+# DIRECT orchestrator (you cannot boot without an in-process execution mode);+# ``openai`` is the always-on default inference driver (no optional SDK to avoid),+# so disabling it would only break the out-of-the-box experience.+CORE_UNCONDITIONAL_PLUGIN_NAMES: frozenset[str] = frozenset({"direct", "openai"})diff --git a/pipelex/plugins/bundle_validator_registry.py b/pipelex/plugins/bundle_validator_registry.pynew file mode 100644index 000000000..90701e1f1--- /dev/null+++ b/pipelex/plugins/bundle_validator_registry.py@@ -0,0 +1,89 @@+"""Per-call validate seam, mirroring the ``OrchestratorRegistry`` for ``/validate``.++A bundle validator produces a validation verdict for one orchestration mode, the way an+orchestrator runs a pipe for one mode. The seam is generic across orchestrators: the+core ``direct`` plugin contributes the in-process validator (``"direct"``), the external+``pipelex-temporal`` plugin contributes the worker-dispatched validator (under+``"temporal"``), and a Mistral validator can slot in later — none of which core or a+host runtime names.++Verdict-as-value (not raise): ``validate_bundles`` *returns* the verdict — the valid+report (a ``ValidationReport``) or the invalid ``ErrorReport`` (carrying+``validation_errors``) — and raises only for a no-verdict infra fault (which a host+runtime maps to a 5xx). This is the same valid/invalid pair the API maps to its+200-always ``/validate`` wire, so the verdict contract is backend-independent.++Seam typed at the MTHDS-protocol level (``ValidationReport``), not the concrete+``PipelexValidationReport`` envelope, for two reasons that point the same way: (1) the+concrete report's module reaches the hub, so naming it from this hub-reachable seam+would close an import cycle — the same reason the orchestrator seam returns the leaf+``PipelexPipeRunOutput`` rather than a hub-coupled type; (2) the brand boundary — the+seam is generic across orchestrators (language-standard altitude), so it speaks the+protocol report, and the Pipelex-runtime envelope is the concrete value the validators+produce and the API surfaces. Every Pipelex validator's valid arm is in fact the+canonical ``PipelexValidationReport`` (a ``ValidationReport`` subtype); the API recovers+that precise type at its edge.+"""++from typing import TYPE_CHECKING, Protocol, TypeAlias, runtime_checkable++from mthds.protocol.models import ValidationReport++from pipelex.base_exceptions import ErrorReport+from pipelex.runtime_bridge.orchestration_mode import OrchestrationMode++if TYPE_CHECKING:+ from collections.abc import Sequence+ from pathlib import Path++# The verdict a validator returns: valid arm = the protocol validation report (at runtime+# always the canonical PipelexValidationReport subtype), invalid arm = the structured+# ErrorReport. Bound at runtime (not just under TYPE_CHECKING) so an external validator+# plugin can import it to annotate its own validate_bundles. Both arms are leaf types w.r.t.+# the hub, so this hub-reachable seam stays import-acyclic even with the runtime imports.+BundleValidationVerdict: TypeAlias = ValidationReport | ErrorReport+++@runtime_checkable+class BundleValidatorProtocol(Protocol):+ """How ``/validate`` produces a verdict under one orchestration mode.++ A validator plugin registers one of these per ``orchestration_mode`` token it serves+ (``"direct"`` in core; ``"temporal"`` from ``pipelex-temporal``;+ ``"mistralai-workflows"`` later). The API resolves the mode and dispatches through the+ ``BundleValidatorRegistry`` instead of branching on a backend. Validation is inherently+ blocking, so there is no ``delivery`` axis here (unlike ``OrchestratorProtocol.run``).++ ``library_dirs`` is host context the in-process arm needs to load the method library;+ a dispatched arm (Temporal) ignores it — its worker loads its own library.+ """++ async def validate_bundles(+ self,+ *,+ mthds_contents: list[str],+ mthds_sources: list[str] | None,+ allow_signatures: bool,+ library_dirs: "Sequence[Path] | None",+ ) -> BundleValidationVerdict: ...+++class BundleValidatorRegistry:+ """Read view over the bundle validators contributed by discovered plugins.++ Keyed by the open ``OrchestrationMode`` token (a ``str``). Built once at boot from+ the registrar's accumulated validators and stored on the hub.+ """++ def __init__(self, validators: dict[OrchestrationMode, BundleValidatorProtocol]):+ self._validators: dict[OrchestrationMode, BundleValidatorProtocol] = dict(validators)++ def get_optional(self, *, mode: OrchestrationMode) -> BundleValidatorProtocol | None:+ return self._validators.get(mode)++ def has(self, *, mode: OrchestrationMode) -> bool:+ return mode in self._validators++ @property+ def modes(self) -> list[OrchestrationMode]:+ return list(self._validators)diff --git a/pipelex/plugins/contract.py b/pipelex/plugins/contract.pynew file mode 100644index 000000000..6fdec6553--- /dev/null+++ b/pipelex/plugins/contract.py@@ -0,0 +1,37 @@+from typing import TYPE_CHECKING, Protocol, runtime_checkable++if TYPE_CHECKING:+ from pipelex.plugins.registrar import PluginRegistrar++# The single coarse plugin-API version. A discovered plugin declares the version+# it targets via ``targets_api``; discovery fails loud on a mismatch (see+# ``PluginApiVersionMismatchError``). Bump this only on a breaking change to the+# registrar menu or the plugin contract.+#+# v2 added the optional ``add_http_error_mapper`` capability (a framework-agnostic+# transport-fault → ``ErrorReport`` mapping a host runtime renders into its own+# HTTP error response).+PLUGIN_API_VERSION: int = 2+++@runtime_checkable+class PipelexPlugin(Protocol):+ """A unit of optional capability discovered at startup.++ A plugin contributes inference backends, model listers, orchestrators,+ hub-slot claims, HTTP-error mappers and teardown callbacks by calling the menu+ methods on the ``PluginRegistrar`` it is handed.++ **Invariant — ``register`` is side-effect-free.** It may *only* call+ registrar menu methods: no hub access, no I/O, no SDK/client construction.+ This is what makes ``build_registrar`` safe to run more than once (it runs at+ CLI-build to harvest commands *and* again at boot). Anything heavy — importing+ a backend SDK, constructing a client, importing ``temporalio`` — happens lazily+ inside the ``make_worker`` closures and the hub-slot-claim thunks, never in+ ``register`` itself.+ """++ name: str+ targets_api: int++ def register(self, registrar: "PluginRegistrar") -> None: ...diff --git a/pipelex/temporal/__init__.py b/pipelex/plugins/direct/__init__.pysimilarity index 100%rename from pipelex/temporal/__init__.pyrename to pipelex/plugins/direct/__init__.pydiff --git a/pipelex/plugins/direct/direct_plugin.py b/pipelex/plugins/direct/direct_plugin.pynew file mode 100644index 000000000..24450a467--- /dev/null+++ b/pipelex/plugins/direct/direct_plugin.py@@ -0,0 +1,22 @@+from pipelex.pipeline.direct_bundle_validator import DirectBundleValidator+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.runtime_bridge.direct_orchestrator import DirectOrchestrator+from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE+++class DirectOrchestratorPlugin:+ """Core, always-on plugin contributing the in-process ``"direct"`` orchestrator and validator.++ ``"direct"`` is the default orchestration mode and must always be available; this+ plugin is core-unconditional (denylisting it is a startup error). It contributes both+ the in-process pipe orchestrator and the in-process ``/validate`` bundle validator —+ constructing either imports no host-runtime SDK.+ """++ name = "direct"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_orchestrator(mode=DIRECT_ORCHESTRATION_MODE, orchestrator=DirectOrchestrator())+ registrar.add_bundle_validator(mode=DIRECT_ORCHESTRATION_MODE, validator=DirectBundleValidator())diff --git a/pipelex/plugins/discovery.py b/pipelex/plugins/discovery.pynew file mode 100644index 000000000..cef78cd30--- /dev/null+++ b/pipelex/plugins/discovery.py@@ -0,0 +1,114 @@+import importlib.metadata+from typing import TYPE_CHECKING, cast++from pipelex import log+from pipelex.plugins.builtins import BUILTIN_PLUGINS, CORE_UNCONDITIONAL_PLUGIN_NAMES+from pipelex.plugins.contract import PLUGIN_API_VERSION, PipelexPlugin+from pipelex.plugins.exceptions import (+ BrokenPluginError,+ CoreUnconditionalPluginDisabledError,+ PluginApiVersionMismatchError,+ PluginError,+)+from pipelex.plugins.registrar import PluginDiscovery, PluginOrigin, PluginRegistrar, PluginStatus++if TYPE_CHECKING:+ from pipelex.system.configuration.configs import PipelexConfig++# External plugins advertise themselves under this entry-point group:+# [project.entry-points."pipelex.plugins"]+# my_plugin = "my_package:make_plugin"+ENTRY_POINT_GROUP = "pipelex.plugins"+++def build_registrar(*, config: "PipelexConfig") -> PluginRegistrar:+ """Discover every plugin and let each register itself into a fresh registrar.++ A **pure function**: it touches no global state and constructs no client/SDK,+ so it is safe to call more than once (D3 runs it at CLI-build to harvest+ commands and again at boot). Iterates ``BUILTIN_PLUGINS`` first, then external+ ``pipelex.plugins`` entry points; version-checks each; skips (and logs) any+ plugin named in ``config.plugins.disabled`` — externals are denylisted by their+ entry-point name *before* ``load()`` so a broken installed plugin can still be+ disabled; and is fail-loud on every conflict (duplicate backend/mode/slot,+ version mismatch, broken plugin).+ """+ registrar = PluginRegistrar(config=config)+ disabled = set(config.plugins.disabled)++ # Built-ins are already instantiated, so their name is known up front.+ for plugin in BUILTIN_PLUGINS:+ if _skip_if_disabled(registrar, name=plugin.name, origin=PluginOrigin.BUILTIN, targets_api=plugin.targets_api, disabled=disabled):+ continue+ _register_plugin(registrar, plugin=plugin, origin=PluginOrigin.BUILTIN)++ # External entry points: denylist by entry-point name *before* load(), so a+ # broken/dependency-missing installed plugin can still be recovered by disabling+ # it. A working external plugin may set a ``.name`` that differs from its+ # entry-point name; the post-load check honors the denylist on that too, so+ # either name disables it.+ for entry_point in _external_entry_points():+ if _skip_if_disabled(registrar, name=entry_point.name, origin=PluginOrigin.EXTERNAL, targets_api=None, disabled=disabled):+ continue+ plugin = _load_external_plugin(entry_point)+ if _skip_if_disabled(registrar, name=plugin.name, origin=PluginOrigin.EXTERNAL, targets_api=plugin.targets_api, disabled=disabled):+ continue+ _register_plugin(registrar, plugin=plugin, origin=PluginOrigin.EXTERNAL)++ return registrar+++def _skip_if_disabled(registrar: PluginRegistrar, *, name: str, origin: PluginOrigin, targets_api: int | None, disabled: set[str]) -> bool:+ """Record and skip a denylisted plugin; return ``True`` if it was skipped.++ Fail-loud when a core-unconditional plugin is denylisted — that is a+ configuration error, never a silent no-op.+ """+ if name not in disabled:+ return False+ if name in CORE_UNCONDITIONAL_PLUGIN_NAMES:+ raise CoreUnconditionalPluginDisabledError(plugin_name=name)+ log.info(f"Plugin '{name}' is disabled via plugins.disabled; skipping.")+ registrar.discoveries.append(+ PluginDiscovery(+ name=name,+ origin=origin,+ status=PluginStatus.DISABLED,+ targets_api=targets_api,+ detail="disabled via plugins.disabled",+ )+ )+ return True+++def _external_entry_points() -> list[importlib.metadata.EntryPoint]:+ return list(importlib.metadata.entry_points(group=ENTRY_POINT_GROUP))+++def _load_external_plugin(entry_point: importlib.metadata.EntryPoint) -> PipelexPlugin:+ try:+ loaded = entry_point.load()+ # The entry point resolves to a plugin instance, or to a zero-arg factory+ # (a class or function) returning one.+ return cast("PipelexPlugin", loaded() if callable(loaded) else loaded)+ except Exception as exc:+ # Case 2: loading/constructing a third-party plugin from an entry point is an unbounded surface.+ raise BrokenPluginError(plugin_name=entry_point.name, reason=f"failed to load entry point: {exc}") from exc+++def _register_plugin(registrar: PluginRegistrar, *, plugin: PipelexPlugin, origin: PluginOrigin) -> None:+ targets_api = getattr(plugin, "targets_api", None)+ if targets_api != PLUGIN_API_VERSION:+ raise PluginApiVersionMismatchError(+ plugin_name=getattr(plugin, "name", "<unknown>"), targets_api=targets_api, supported_api=PLUGIN_API_VERSION+ )+ registrar.begin_plugin(name=plugin.name, origin=origin, targets_api=PLUGIN_API_VERSION)+ try:+ plugin.register(registrar)+ except PluginError:+ # Our own structured, fail-loud conflicts (duplicate backend/mode/slot)+ # already name both contributors — propagate them verbatim.+ raise+ except Exception as exc:+ # Case 2: plugin.register is unbounded third-party dispatch.+ raise BrokenPluginError(plugin_name=plugin.name, reason=str(exc)) from excdiff --git a/pipelex/plugins/docling/docling_plugin.py b/pipelex/plugins/docling/docling_plugin.pynew file mode 100644index 000000000..81a7dad0b--- /dev/null+++ b/pipelex/plugins/docling/docling_plugin.py@@ -0,0 +1,46 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol++_DOCLING_MISSING_MSG = "The docling library is required in order to use Docling for PDF and image text extraction."+++def _make_docling_extract_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="docling", extra="docling", msg=_DOCLING_MISSING_MSG)++ from pipelex.plugins.docling.docling_extract_worker import DoclingExtractWorker # noqa: PLC0415+ from pipelex.plugins.docling.docling_factory import DoclingFactory # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=DoclingFactory.make_docling_sdk,+ )+ return DoclingExtractWorker(+ sdk_instance=sdk_instance,+ extra_config=backend.extra_config,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class DoclingPlugin:+ """Built-in driver for Docling PDF/image text extraction via the docling library."""++ name = "docling"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.EXTRACT, sdk="docling_sdk", make_worker=_make_docling_extract_worker)diff --git a/pipelex/plugins/exceptions.py b/pipelex/plugins/exceptions.pynew file mode 100644index 000000000..6acdf7e64--- /dev/null+++ b/pipelex/plugins/exceptions.py@@ -0,0 +1,142 @@+from pipelex.base_exceptions import ErrorDomain, PipelexError+++class PluginError(PipelexError):+ """Base for plugin-discovery and plugin-registration failures."""++ error_domain = ErrorDomain.CONFIG+ _declared_title = "Plugin error"+++class PluginApiVersionMismatchError(PluginError):+ """A discovered plugin targets a plugin-API version this Pipelex does not support."""++ def __init__(self, *, plugin_name: str, targets_api: object, supported_api: int):+ self.plugin_name = plugin_name+ self.targets_api = targets_api+ self.supported_api = supported_api+ message = (+ f"Plugin '{plugin_name}' targets plugin API version {targets_api!r}, "+ f"but this Pipelex supports version {supported_api}. "+ "Upgrade Pipelex or install a plugin version that targets the supported API."+ )+ super().__init__(message)+++class InferenceBackendNotFoundError(PluginError):+ """No inference backend is registered for a requested (family, sdk).++ Raised by the family worker factories on a registry-lookup miss — typically a+ model whose backend plugin is not installed or was disabled via+ ``plugins.disabled``.+ """++ def __init__(self, *, family: str, sdk: str):+ self.family = family+ self.sdk = sdk+ message = f"No inference backend registered for sdk '{sdk}' in the {family} family. Is its plugin installed and enabled?"+ super().__init__(message)+++class DuplicateInferenceBackendError(PluginError):+ """Two plugins registered an inference backend for the same (family, sdk)."""++ def __init__(self, *, family: str, sdk: str, first_plugin: str, second_plugin: str):+ self.family = family+ self.sdk = sdk+ self.first_plugin = first_plugin+ self.second_plugin = second_plugin+ message = (+ f"Inference backend ({family}, '{sdk}') is registered by both plugin "+ f"'{first_plugin}' and plugin '{second_plugin}'. Each (family, sdk) must have a single backend."+ )+ super().__init__(message)+++class DuplicateModelListerError(PluginError):+ """Two plugins registered a model lister for the same sdk."""++ def __init__(self, *, sdk: str, first_plugin: str, second_plugin: str):+ self.sdk = sdk+ self.first_plugin = first_plugin+ self.second_plugin = second_plugin+ message = (+ f"Model lister for sdk '{sdk}' is registered by both plugin "+ f"'{first_plugin}' and plugin '{second_plugin}'. Each sdk must have a single lister."+ )+ super().__init__(message)+++class DuplicateOrchestratorError(PluginError):+ """Two plugins registered an orchestrator for the same orchestration mode."""++ def __init__(self, *, mode: str, first_plugin: str, second_plugin: str):+ self.mode = mode+ self.first_plugin = first_plugin+ self.second_plugin = second_plugin+ message = (+ f"Orchestrator for orchestration mode '{mode}' is registered by both plugin "+ f"'{first_plugin}' and plugin '{second_plugin}'. Each mode must have a single orchestrator."+ )+ super().__init__(message)+++class DuplicateBundleValidatorError(PluginError):+ """Two plugins registered a bundle validator for the same orchestration mode."""++ def __init__(self, *, mode: str, first_plugin: str, second_plugin: str):+ self.mode = mode+ self.first_plugin = first_plugin+ self.second_plugin = second_plugin+ message = (+ f"Bundle validator for orchestration mode '{mode}' is registered by both plugin "+ f"'{first_plugin}' and plugin '{second_plugin}'. Each mode must have a single validator."+ )+ super().__init__(message)+++class DuplicateHttpErrorMapperError(PluginError):+ """Two plugins registered an HTTP-error mapper for the same exception type."""++ def __init__(self, *, exc_type: str, first_plugin: str, second_plugin: str):+ self.exc_type = exc_type+ self.first_plugin = first_plugin+ self.second_plugin = second_plugin+ message = (+ f"HTTP-error mapper for exception type '{exc_type}' is registered by both plugin "+ f"'{first_plugin}' and plugin '{second_plugin}'. Each exception type must have a single mapper."+ )+ super().__init__(message)+++class HubSlotAlreadyClaimedError(PluginError):+ """Two plugins claimed the same hub slot."""++ def __init__(self, *, slot: str, first_plugin: str, second_plugin: str):+ self.slot = slot+ self.first_plugin = first_plugin+ self.second_plugin = second_plugin+ message = (+ f"Hub slot '{slot}' is claimed by both plugin '{first_plugin}' and plugin "+ f"'{second_plugin}'. Each slot can be claimed by at most one plugin."+ )+ super().__init__(message)+++class CoreUnconditionalPluginDisabledError(PluginError):+ """A plugin that core requires unconditionally was named in plugins.disabled."""++ def __init__(self, *, plugin_name: str):+ self.plugin_name = plugin_name+ message = f"Plugin '{plugin_name}' is required by core and cannot be disabled via plugins.disabled. Remove it from the denylist."+ super().__init__(message)+++class BrokenPluginError(PluginError):+ """A discovered plugin failed while loading or registering itself."""++ def __init__(self, *, plugin_name: str, reason: str):+ self.plugin_name = plugin_name+ self.reason = reason+ message = f"Plugin '{plugin_name}' failed to register: {reason}"+ super().__init__(message)diff --git a/pipelex/plugins/fal/fal_plugin.py b/pipelex/plugins/fal/fal_plugin.pynew file mode 100644index 000000000..dcfcc0f9c--- /dev/null+++ b/pipelex/plugins/fal/fal_plugin.py@@ -0,0 +1,46 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol++_FAL_MISSING_MSG = "The fal-client SDK is required in order to use FAL models (generation of images)."+++def _make_fal_img_gen_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="fal_client", dependency_name="fal-client", extra="fal", msg=_FAL_MISSING_MSG)++ from fal_client import AsyncClient as FalAsyncClient # noqa: PLC0415++ from pipelex.plugins.fal.fal_img_gen_worker import FalImgGenWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: FalAsyncClient(key=backend.api_key),+ )+ return FalImgGenWorker(+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class FalPlugin:+ """Built-in driver for FAL image generation models via the fal-client SDK."""++ name = "fal"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.IMG_GEN, sdk="fal", make_worker=_make_fal_img_gen_worker)diff --git a/pipelex/plugins/gateway/gateway_completions_factory.py b/pipelex/plugins/gateway/gateway_completions_factory.pyindex c24c2bc35..4c1653582 100644--- a/pipelex/plugins/gateway/gateway_completions_factory.py+++ b/pipelex/plugins/gateway/gateway_completions_factory.py@@ -35,7 +35,7 @@ if TYPE_CHECKING:from pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec- from pipelex.plugins.plugin_sdk_registry import Plugin+ from pipelex.plugins.model_handle import ModelHandleclass GatewayCompletionsFactory(OpenAICompletionsFactory):@@ -103,7 +103,7 @@ class GatewayCompletionsFactory(OpenAICompletionsFactory):@classmethoddef make_portkey_openai_client_for_completions(cls,- plugin: Plugin,+ model_handle: ModelHandle,*,backend: InferenceBackend,) -> openai.AsyncOpenAI:@@ -114,8 +114,8 @@ class GatewayCompletionsFactory(OpenAICompletionsFactory):endpoint = GatewayFactory.get_endpoint(backend=backend)api_key = GatewayFactory.get_api_key(backend=backend)- if not GatewayOpenAISdkVariant.is_completions(plugin.sdk):- msg = f"Plugin '{plugin}' is not supported by '{cls.__name__}'"+ if not GatewayOpenAISdkVariant.is_completions(model_handle.sdk):+ msg = f"ModelHandle '{model_handle}' is not supported by '{cls.__name__}'"raise GatewayFactoryError(msg)return openai.AsyncOpenAI(diff --git a/pipelex/plugins/gateway/gateway_plugin.py b/pipelex/plugins/gateway/gateway_plugin.pynew file mode 100644index 000000000..d5e9c724f--- /dev/null+++ b/pipelex/plugins/gateway/gateway_plugin.py@@ -0,0 +1,162 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol+++def _make_gateway_completions_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.gateway.gateway_completions_factory import GatewayCompletionsFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: GatewayCompletionsFactory.make_portkey_openai_client_for_completions(model_handle=model_handle, backend=backend),+ )+ return OpenAICompletionsLLMWorker(+ openai_completions_factory=GatewayCompletionsFactory(is_http_url_enabled=False),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_gateway_responses_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.gateway.gateway_responses_factory import GatewayResponsesFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: GatewayResponsesFactory.make_portkey_openai_client_for_responses(model_handle=model_handle, backend=backend),+ )+ return OpenAIResponsesLLMWorker(+ openai_responses_factory=GatewayResponsesFactory(is_http_url_enabled=False),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_gateway_img_gen_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415+ from pipelex.plugins.gateway.gateway_img_gen_worker import GatewayImgGenWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: GatewayFactory.make_portkey_client(backend=backend),+ )+ return GatewayImgGenWorker(+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_gateway_completions_img_gen_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.gateway.gateway_completions_factory import GatewayCompletionsFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_completions_img_gen_worker import OpenAICompletionsImgGenWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: GatewayCompletionsFactory.make_portkey_openai_client_for_completions(model_handle=model_handle, backend=backend),+ )+ return OpenAICompletionsImgGenWorker(+ openai_completions_factory=GatewayCompletionsFactory(is_http_url_enabled=False),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_gateway_extract_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.gateway.gateway_extract_worker import GatewayExtractWorker # noqa: PLC0415+ from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: GatewayFactory.make_portkey_client(backend=backend),+ )+ return GatewayExtractWorker(+ sdk_instance=sdk_instance,+ extra_config=backend.extra_config,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_gateway_search_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.gateway.gateway_factory import GatewayFactory # noqa: PLC0415+ from pipelex.plugins.gateway.gateway_search_worker import GatewaySearchWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: GatewayFactory.make_portkey_client(backend=backend),+ )+ return GatewaySearchWorker(+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class GatewayPlugin:+ """Built-in driver for the Pipelex Gateway (OpenAI-compatible substrate), serving all inference families."""++ name = "gateway"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="gateway_completions", make_worker=_make_gateway_completions_worker)+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="gateway_responses", make_worker=_make_gateway_responses_worker)+ registrar.add_inference_backend(family=InferenceFamily.IMG_GEN, sdk="gateway_img_gen", make_worker=_make_gateway_img_gen_worker)+ registrar.add_inference_backend(+ family=InferenceFamily.IMG_GEN, sdk="gateway_completions", make_worker=_make_gateway_completions_img_gen_worker+ )+ registrar.add_inference_backend(family=InferenceFamily.EXTRACT, sdk="gateway_extract", make_worker=_make_gateway_extract_worker)+ registrar.add_inference_backend(family=InferenceFamily.SEARCH, sdk="gateway_search", make_worker=_make_gateway_search_worker)diff --git a/pipelex/plugins/gateway/gateway_responses_factory.py b/pipelex/plugins/gateway/gateway_responses_factory.pyindex a5d18963a..1fa9f04ee 100644--- a/pipelex/plugins/gateway/gateway_responses_factory.py+++ b/pipelex/plugins/gateway/gateway_responses_factory.py@@ -18,22 +18,22 @@ if TYPE_CHECKING:from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstractfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec- from pipelex.plugins.plugin_sdk_registry import Plugin+ from pipelex.plugins.model_handle import ModelHandleclass GatewayResponsesFactory(OpenAIResponsesFactory):@classmethoddef make_portkey_openai_client_for_responses(cls,- plugin: Plugin,+ model_handle: ModelHandle,*,backend: InferenceBackend,) -> openai.AsyncOpenAI:is_debug_enabled = GatewayFactory.is_debug_enabled(backend=backend)endpoint = GatewayFactory.get_endpoint(backend=backend)api_key = GatewayFactory.get_api_key(backend=backend)- if not GatewayOpenAISdkVariant.is_responses(plugin.sdk):- msg = f"Plugin '{plugin}' is not supported by '{cls.__name__}'"+ if not GatewayOpenAISdkVariant.is_responses(model_handle.sdk):+ msg = f"ModelHandle '{model_handle}' is not supported by '{cls.__name__}'"raise GatewayFactoryError(msg)return openai.AsyncOpenAI(diff --git a/pipelex/plugins/google/google_plugin.py b/pipelex/plugins/google/google_plugin.pynew file mode 100644index 000000000..c30e834f1--- /dev/null+++ b/pipelex/plugins/google/google_plugin.py@@ -0,0 +1,85 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol++_GOOGLE_MISSING_MSG = "The google-genai SDK is required in order to use Google Gemini API directly."+_GOOGLE_IMG_GEN_MISSING_MSG = "The google-genai SDK is required in order to use Google Gemini Image models."+++def _make_google_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="google.genai", dependency_name="google-genai", extra="google", msg=_GOOGLE_MISSING_MSG)++ from pipelex.plugins.google.google_factory import GoogleFactory # noqa: PLC0415+ from pipelex.plugins.google.google_llm_worker import GoogleLLMWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: GoogleFactory.make_google_client(backend=backend),+ )+ return GoogleLLMWorker(+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_google_img_gen_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="google.genai", dependency_name="google-genai", extra="google", msg=_GOOGLE_IMG_GEN_MISSING_MSG)++ from pipelex.plugins.google.google_factory import GoogleFactory # noqa: PLC0415+ from pipelex.plugins.google.google_img_gen_worker import GoogleImgGenWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: GoogleFactory.make_google_client(backend=backend),+ )+ return GoogleImgGenWorker(+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++async def _list_google_models(+ *,+ sdk: str,+ backend_name: str,+ backend: InferenceBackend,+ flat: bool,+ any_listed: bool,+) -> None:+ from pipelex.plugins.google.google_list import list_google_models # noqa: PLC0415++ await list_google_models(sdk=sdk, backend_name=backend_name, backend=backend, flat=flat, any_listed=any_listed)+++class GooglePlugin:+ """Built-in driver for Google Gemini models (LLM + image generation) via the google-genai SDK."""++ name = "google"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="google", make_worker=_make_google_worker)+ registrar.add_inference_backend(family=InferenceFamily.IMG_GEN, sdk="google", make_worker=_make_google_img_gen_worker)+ registrar.add_model_lister(sdk="google", lister=_list_google_models)diff --git a/pipelex/plugins/huggingface/huggingface_plugin.py b/pipelex/plugins/huggingface/huggingface_plugin.pynew file mode 100644index 000000000..acb136133--- /dev/null+++ b/pipelex/plugins/huggingface/huggingface_plugin.py@@ -0,0 +1,57 @@+from typing import TYPE_CHECKING++from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol++if TYPE_CHECKING:+ from huggingface_hub.inference._providers import PROVIDER_OR_POLICY_T++_HUGGINGFACE_MISSING_MSG = "The huggingface_hub SDK is required in order to use HuggingFace image generation models."+++def _make_huggingface_img_gen_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="huggingface_hub", extra="huggingface", msg=_HUGGINGFACE_MISSING_MSG)++ from huggingface_hub import AsyncInferenceClient # noqa: PLC0415++ from pipelex.plugins.huggingface.huggingface_factory import HuggingFaceFactory # noqa: PLC0415+ from pipelex.plugins.huggingface.huggingface_img_gen_worker import HuggingFaceImgGenWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ provider_literal: PROVIDER_OR_POLICY_T+ if provider_str := model_handle.variant:+ provider_literal = HuggingFaceFactory.make_huggingface_inference_provider(provider_str=provider_str)+ else:+ provider_literal = "auto"+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: AsyncInferenceClient(provider=provider_literal, token=backend.api_key),+ )+ return HuggingFaceImgGenWorker(+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class HuggingFacePlugin:+ """Built-in driver for HuggingFace image generation models via the huggingface_hub SDK."""++ name = "huggingface"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.IMG_GEN, sdk="huggingface_img_gen", make_worker=_make_huggingface_img_gen_worker)diff --git a/pipelex/plugins/inference_backend_registry.py b/pipelex/plugins/inference_backend_registry.pynew file mode 100644index 000000000..017d03669--- /dev/null+++ b/pipelex/plugins/inference_backend_registry.py@@ -0,0 +1,84 @@+import importlib.util+from collections.abc import Callable, Sequence+from typing import TypeAlias++from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.plugins.exceptions import InferenceBackendNotFoundError+from pipelex.system.exceptions import MissingDependencyError+from pipelex.types import StrEnum+++class InferenceFamily(StrEnum):+ LLM = "llm"+ IMG_GEN = "img_gen"+ EXTRACT = "extract"+ SEARCH = "search"+++# The uniform inference-backend factory. A backend plugin registers one of these+# per (family, sdk) it serves. It is import-light to *reference* (a plain+# callable) and only imports its SDK lazily when *called*.+#+# Called by the family worker factory as:+# make_worker(*, inference_model, backend, sdk_clients, reporting_delegate)+# where ``sdk_clients`` is the process-wide ``SdkClientRegistry`` (for client+# caching) and ``reporting_delegate`` is the per-call reporting delegate.+MakeWorkerFn: TypeAlias = Callable[..., InferenceWorkerAbstract]+++def require_sdk(*, spec: str | Sequence[str], extra: str, msg: str, dependency_name: str | None = None) -> None:+ """Raise ``MissingDependencyError`` if any of ``spec`` is not importable.++ The DRY replacement for the repeated ``find_spec(...) is None`` guard that+ used to sit in every dispatch arm. Called *inside* ``make_worker`` so a+ missing optional extra fails when the backend is actually used, not at boot.++ - ``spec``: the import name(s) to probe (e.g. ``"anthropic"`` or+ ``["boto3", "aioboto3"]`` when several are required together).+ - ``dependency_name``: the human-facing package name shown in the error;+ defaults to the joined names of the *missing* specs only (override when the+ import name differs from the distribution name, e.g. spec+ ``"google.genai"`` / dependency ``"google-genai"``).+ - ``extra``: the pip extra to install (drives the ``pipelex[<extra>]`` hint).+ """+ specs = [spec] if isinstance(spec, str) else list(spec)+ missing: list[str] = []+ for one_spec in specs:+ try:+ if importlib.util.find_spec(one_spec) is None:+ missing.append(one_spec)+ except ModuleNotFoundError:+ # ``find_spec`` imports the parent of a dotted spec (e.g. ``google`` for+ # ``google.genai``); an entirely absent parent raises ModuleNotFoundError+ # rather than returning None, so treat that as "missing" too.+ missing.append(one_spec)+ if missing:+ # Name only the specs that are actually absent, so a user who already has+ # one of several required SDKs is not told to (re)install it.+ raise MissingDependencyError(dependency_name or ",".join(missing), extra, msg)+++class InferenceBackendRegistry:+ """Read view over the inference backends contributed by discovered plugins.++ Keyed by ``(family, sdk)``. Built once at boot from the registrar's+ accumulated backends and stored on the hub; the family worker factories look+ up their ``make_worker`` here instead of branching on a ``match`` over SDK+ strings.+ """++ def __init__(self, backends: dict[tuple[InferenceFamily, str], MakeWorkerFn]):+ self._backends: dict[tuple[InferenceFamily, str], MakeWorkerFn] = dict(backends)++ def lookup(self, *, family: InferenceFamily, sdk: str) -> MakeWorkerFn:+ make_worker = self._backends.get((family, sdk))+ if make_worker is None:+ raise InferenceBackendNotFoundError(family=family, sdk=sdk)+ return make_worker++ def has(self, *, family: InferenceFamily, sdk: str) -> bool:+ return (family, sdk) in self._backends++ @property+ def keys(self) -> list[tuple[InferenceFamily, str]]:+ return list(self._backends)diff --git a/pipelex/plugins/linkup/linkup_plugin.py b/pipelex/plugins/linkup/linkup_plugin.pynew file mode 100644index 000000000..5af3427ca--- /dev/null+++ b/pipelex/plugins/linkup/linkup_plugin.py@@ -0,0 +1,57 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol++_LINKUP_MISSING_MSG = "The linkup SDK is required in order to use Linkup Fetch extraction models."+_LINKUP_SEARCH_MISSING_MSG = "The linkup SDK is required in order to use Linkup search models."+++def _make_linkup_extract_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry, # noqa: ARG001 - stateless worker, no SDK-client caching+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="linkup", extra="linkup", msg=_LINKUP_MISSING_MSG)++ from pipelex.plugins.linkup.linkup_extract_worker import LinkupExtractWorker # noqa: PLC0415++ return LinkupExtractWorker(+ extra_config=backend.extra_config,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_linkup_search_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend, # noqa: ARG001 - uniform MakeWorkerFn signature; the search worker builds its own client+ sdk_clients: SdkClientRegistry, # noqa: ARG001 - stateless worker, no SDK-client caching+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="linkup", extra="linkup", msg=_LINKUP_SEARCH_MISSING_MSG)++ from pipelex.plugins.linkup.linkup_search_worker import LinkupSearchWorker # noqa: PLC0415++ return LinkupSearchWorker(+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class LinkupPlugin:+ """Built-in driver for Linkup, serving both the Extract (fetch) and Search families via the linkup SDK."""++ name = "linkup"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.EXTRACT, sdk="linkup_fetch", make_worker=_make_linkup_extract_worker)+ registrar.add_inference_backend(family=InferenceFamily.SEARCH, sdk="linkup", make_worker=_make_linkup_search_worker)diff --git a/pipelex/plugins/mistral/mistral_plugin.py b/pipelex/plugins/mistral/mistral_plugin.pynew file mode 100644index 000000000..79ad433c0--- /dev/null+++ b/pipelex/plugins/mistral/mistral_plugin.py@@ -0,0 +1,92 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol++_MISTRAL_MISSING_MSG = (+ "The mistralai SDK is required in order to use Mistral models through the mistralai client. "+ "However, you can use Mistral models through bedrock directly "+ "by using the 'bedrock-mistral' llm family. (eg: bedrock-mistral-large)"+)+_MISTRAL_EXTRACT_MISSING_MSG = "The mistralai SDK is required in order to use Mistral OCR models through the mistralai client."+++def _make_mistral_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="mistralai", extra="mistral", msg=_MISTRAL_MISSING_MSG)++ from pipelex.plugins.mistral.mistral_factory import MistralFactory # noqa: PLC0415+ from pipelex.plugins.mistral.mistral_llm_worker import MistralLLMWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: MistralFactory.make_mistral_client(backend=backend),+ )+ return MistralLLMWorker(+ mistral_factory=MistralFactory(),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_mistral_extract_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ require_sdk(spec="mistralai", extra="mistral", msg=_MISTRAL_EXTRACT_MISSING_MSG)++ from pipelex.plugins.mistral.mistral_extract_worker import MistralExtractWorker # noqa: PLC0415+ from pipelex.plugins.mistral.mistral_factory import MistralFactory # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: MistralFactory.make_mistral_client(backend=backend),+ )+ return MistralExtractWorker(+ sdk_instance=sdk_instance,+ extra_config=backend.extra_config,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++# Async to satisfy the uniform ListModelsFn contract (the loop awaits it) even though Mistral lists synchronously.+async def _list_mistral_models( # noqa: RUF029+ *,+ sdk: str,+ backend_name: str,+ backend: InferenceBackend, # noqa: ARG001 — uniform lister signature; Mistral lists without a backend client+ flat: bool,+ any_listed: bool,+) -> None:+ from pipelex.plugins.mistral.mistral_list import list_mistral_models # noqa: PLC0415++ list_mistral_models(sdk=sdk, backend_name=backend_name, flat=flat, any_listed=any_listed)+++class MistralPlugin:+ """Built-in driver for Mistral models (LLM + OCR extraction) via the mistralai SDK."""++ name = "mistral"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="mistral", make_worker=_make_mistral_worker)+ registrar.add_inference_backend(family=InferenceFamily.EXTRACT, sdk="mistral", make_worker=_make_mistral_extract_worker)+ registrar.add_model_lister(sdk="mistral", lister=_list_mistral_models)diff --git a/pipelex/plugins/plugin.py b/pipelex/plugins/model_handle.pysimilarity index 87%rename from pipelex/plugins/plugin.pyrename to pipelex/plugins/model_handle.pyindex ca555040d..d7045e819 100644--- a/pipelex/plugins/plugin.py+++ b/pipelex/plugins/model_handle.py@@ -3,7 +3,7 @@ from pydantic import BaseModelfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec-class Plugin(BaseModel):+class ModelHandle(BaseModel):sdk: strbackend: strvariant: str | None = None@@ -16,8 +16,8 @@ class Plugin(BaseModel):return f"{self.sdk}@{self.backend}"@classmethod- def make_for_inference_model(cls, inference_model: InferenceModelSpec) -> "Plugin":- return Plugin(+ def make_for_inference_model(cls, inference_model: InferenceModelSpec) -> "ModelHandle":+ return ModelHandle(sdk=inference_model.sdk,backend=inference_model.backend_name,variant=inference_model.variant,diff --git a/pipelex/plugins/model_lister_registry.py b/pipelex/plugins/model_lister_registry.pynew file mode 100644index 000000000..a572711ec--- /dev/null+++ b/pipelex/plugins/model_lister_registry.py@@ -0,0 +1,38 @@+from collections.abc import Awaitable, Callable+from typing import TypeAlias++# The uniform model-listing callable. A backend plugin registers one of these per+# ``sdk`` it can enumerate models for, alongside its worker factory. Like+# ``MakeWorkerFn`` it is import-light to *reference* (a plain callable) and only+# imports its SDK lazily when *called*.+#+# Called by ``ModelLister.list_models`` as:+# await lister(sdk=sdk, backend_name=backend_name, backend=backend, flat=flat, any_listed=any_listed)+ListModelsFn: TypeAlias = Callable[..., Awaitable[None]]+++class ModelListerRegistry:+ """Read view over the model listers contributed by discovered plugins.++ Keyed by ``sdk``. Built once at boot from the registrar's accumulated listers+ and stored on the hub; ``ModelLister.list_models`` looks up its lister here+ instead of branching on a ``match`` over SDK strings.++ A lookup miss is a *soft* outcome — the SDK simply has no remote-listing+ capability — so the lookup is ``get_optional`` returning ``None`` rather than+ raising (mirroring ``OrchestratorRegistry``). The caller reports such an SDK as+ unsupported-for-listing instead of failing.+ """++ def __init__(self, listers: dict[str, ListModelsFn]):+ self._listers: dict[str, ListModelsFn] = dict(listers)++ def get_optional(self, *, sdk: str) -> ListModelsFn | None:+ return self._listers.get(sdk)++ def has(self, *, sdk: str) -> bool:+ return sdk in self._listers++ @property+ def sdks(self) -> list[str]:+ return list(self._listers)diff --git a/pipelex/plugins/openai/openai_client_factory.py b/pipelex/plugins/openai/openai_client_factory.pyindex c100cf42a..dfd3f9b69 100644--- a/pipelex/plugins/openai/openai_client_factory.py+++ b/pipelex/plugins/openai/openai_client_factory.py@@ -3,8 +3,8 @@ import openaifrom pipelex import logfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.config import get_config+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.plugins.openai.openai_exceptions import OpenAIClientFactoryError-from pipelex.plugins.plugin_sdk_registry import Pluginfrom pipelex.types import StrEnum@@ -26,14 +26,14 @@ class OpenAIClientFactory:@classmethoddef make_openai_client(cls,- plugin: Plugin,+ model_handle: ModelHandle,*,backend: InferenceBackend,) -> openai.AsyncClient:try:- sdk_variant = OpenAISdkVariant(plugin.sdk)+ sdk_variant = OpenAISdkVariant(model_handle.sdk)except ValueError as exc:- msg = f"Plugin '{plugin}' is not supported by '{cls.__name__}'"+ msg = f"ModelHandle '{model_handle}' is not supported by '{cls.__name__}'"raise OpenAIClientFactoryError(msg) from exc# We have a workaround here:diff --git a/pipelex/plugins/openai/openai_completions_factory.py b/pipelex/plugins/openai/openai_completions_factory.pyindex 0e11dd2a5..8af1f0185 100644--- a/pipelex/plugins/openai/openai_completions_factory.py+++ b/pipelex/plugins/openai/openai_completions_factory.py@@ -10,7 +10,7 @@ from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstractfrom pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory-from pipelex.plugins.plugin_factory_abstract import PluginFactoryAbstract+from pipelex.plugins.backend_extras_factory import BackendExtrasFactoryfrom pipelex.tools.uri.prepared_file import PreparedFileBase64, PreparedFileHttpUrl, PreparedFileLocalPathif TYPE_CHECKING:@@ -19,7 +19,7 @@ if TYPE_CHECKING:from openai.types.completion_usage import CompletionUsage-class OpenAICompletionsFactory(PluginFactoryAbstract):+class OpenAICompletionsFactory(BackendExtrasFactory):def __init__(self, is_http_url_enabled: bool):super().__init__()self.is_http_url_enabled = is_http_url_enableddiff --git a/pipelex/plugins/openai/openai_list.py b/pipelex/plugins/openai/openai_list.pyindex b651c682a..b8ae6129b 100644--- a/pipelex/plugins/openai/openai_list.py+++ b/pipelex/plugins/openai/openai_list.py@@ -7,8 +7,8 @@ from rich import boxfrom rich.table import Tablefrom pipelex.hub import get_console+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.plugins.openai.openai_llms import openai_list_available_models-from pipelex.plugins.plugin_sdk_registry import Pluginif TYPE_CHECKING:from openai.types import Model@@ -25,9 +25,9 @@ async def list_openai_models(any_listed: bool,) -> None:"""List OpenAI models."""- plugin = Plugin(sdk=sdk, backend=backend_name)+ model_handle = ModelHandle(sdk=sdk, backend=backend_name)openai_models = await openai_list_available_models(- plugin=plugin,+ model_handle=model_handle,backend=backend,)diff --git a/pipelex/plugins/openai/openai_llms.py b/pipelex/plugins/openai/openai_llms.pyindex 299812b0f..700f9f85b 100644--- a/pipelex/plugins/openai/openai_llms.py+++ b/pipelex/plugins/openai/openai_llms.py@@ -1,17 +1,17 @@from openai.types import Modelfrom pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory-from pipelex.plugins.plugin_sdk_registry import Pluginasync def openai_list_available_models(- plugin: Plugin,+ model_handle: ModelHandle,*,backend: InferenceBackend,) -> list[Model]:openai_client_async = OpenAIClientFactory.make_openai_client(- plugin=plugin,+ model_handle=model_handle,backend=backend,)diff --git a/pipelex/plugins/openai/openai_plugin.py b/pipelex/plugins/openai/openai_plugin.pynew file mode 100644index 000000000..b45b949bc--- /dev/null+++ b/pipelex/plugins/openai/openai_plugin.py@@ -0,0 +1,110 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol+++def _make_openai_completions_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: OpenAIClientFactory.make_openai_client(model_handle=model_handle, backend=backend),+ )+ return OpenAICompletionsLLMWorker(+ openai_completions_factory=OpenAICompletionsFactory(is_http_url_enabled=True),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_openai_responses_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_responses_factory import OpenAIResponsesFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: OpenAIClientFactory.make_openai_client(model_handle=model_handle, backend=backend),+ )+ return OpenAIResponsesLLMWorker(+ openai_responses_factory=OpenAIResponsesFactory(is_http_url_enabled=True),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_openai_img_gen_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_img_gen_worker import OpenAIImgGenWorker # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: OpenAIClientFactory.make_openai_client(model_handle=model_handle, backend=backend),+ )+ return OpenAIImgGenWorker(+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++async def _list_openai_models(+ *,+ sdk: str,+ backend_name: str,+ backend: InferenceBackend,+ flat: bool,+ any_listed: bool,+) -> None:+ from pipelex.plugins.openai.openai_list import list_openai_models # noqa: PLC0415++ await list_openai_models(sdk=sdk, backend_name=backend_name, backend=backend, flat=flat, any_listed=any_listed)+++class OpenAIPlugin:+ """Always-on built-in driver for OpenAI and Azure OpenAI (no optional SDK)."""++ name = "openai"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="openai", make_worker=_make_openai_completions_worker)+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="azure_openai", make_worker=_make_openai_completions_worker)+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="openai_responses", make_worker=_make_openai_responses_worker)+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="azure_openai_responses", make_worker=_make_openai_responses_worker)+ registrar.add_inference_backend(family=InferenceFamily.IMG_GEN, sdk="openai_img_gen", make_worker=_make_openai_img_gen_worker)+ registrar.add_model_lister(sdk="openai", lister=_list_openai_models)+ registrar.add_model_lister(sdk="azure_openai", lister=_list_openai_models)+ registrar.add_model_lister(sdk="openai_responses", lister=_list_openai_models)+ registrar.add_model_lister(sdk="azure_openai_responses", lister=_list_openai_models)diff --git a/pipelex/plugins/openai/openai_responses_factory.py b/pipelex/plugins/openai/openai_responses_factory.pyindex 680408272..81a0c662e 100644--- a/pipelex/plugins/openai/openai_responses_factory.py+++ b/pipelex/plugins/openai/openai_responses_factory.py@@ -11,7 +11,7 @@ from pipelex.cogt.image.prompt_image import PromptImageDetailfrom pipelex.cogt.image.prompt_image_utils import prep_prompt_imagesfrom pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory-from pipelex.plugins.plugin_factory_abstract import PluginFactoryAbstract+from pipelex.plugins.backend_extras_factory import BackendExtrasFactoryfrom pipelex.tools.uri.prepared_file import PreparedFileBase64, PreparedFileHttpUrl, PreparedFileLocalPathif TYPE_CHECKING:@@ -28,7 +28,7 @@ if TYPE_CHECKING:from pipelex.cogt.model_backends.model_spec import InferenceModelSpec-class OpenAIResponsesFactory(PluginFactoryAbstract):+class OpenAIResponsesFactory(BackendExtrasFactory):def __init__(self, is_http_url_enabled: bool):super().__init__()self.is_http_url_enabled = is_http_url_enableddiff --git a/pipelex/plugins/openrouter/openrouter_plugin.py b/pipelex/plugins/openrouter/openrouter_plugin.pynew file mode 100644index 000000000..d71a64f91--- /dev/null+++ b/pipelex/plugins/openrouter/openrouter_plugin.py@@ -0,0 +1,43 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol+++def _make_openrouter_img_gen_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory # noqa: PLC0415+ from pipelex.plugins.openai.openai_completions_img_gen_worker import OpenAICompletionsImgGenWorker # noqa: PLC0415+ from pipelex.plugins.openrouter.openrouter_completions_factory import OpenRouterCompletionsFactory # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: OpenAIClientFactory.make_openai_client(model_handle=model_handle, backend=backend),+ )+ return OpenAICompletionsImgGenWorker(+ openai_completions_factory=OpenRouterCompletionsFactory(is_http_url_enabled=True),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class OpenRouterPlugin:+ """Built-in driver for OpenRouter image generation via the OpenAI-completions substrate."""++ name = "openrouter"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.IMG_GEN, sdk="openrouter_img_gen", make_worker=_make_openrouter_img_gen_worker)diff --git a/pipelex/plugins/orchestrator_registry.py b/pipelex/plugins/orchestrator_registry.pynew file mode 100644index 000000000..110d17181--- /dev/null+++ b/pipelex/plugins/orchestrator_registry.py@@ -0,0 +1,61 @@+from typing import TYPE_CHECKING, Protocol, runtime_checkable++from pipelex.runtime_bridge.delivery_mode import DeliveryMode+from pipelex.runtime_bridge.orchestration_mode import OrchestrationMode++if TYPE_CHECKING:+ from pipelex.pipe_run.delivery_assignment import DeliveryAssignment+ from pipelex.pipe_run.pipe_job import PipeJob+ from pipelex.runtime_bridge.payloads import PipelexPipeRunOutput+++@runtime_checkable+class OrchestratorProtocol(Protocol):+ """How a pipe job runs under one orchestration mode.++ An orchestrator plugin registers one of these per ``orchestration_mode`` token it+ serves (``"direct"`` in core; ``"temporal"`` from ``pipelex-temporal``,+ ``"mistralai-workflows"`` from ``pipelex-mistralai-workflows``). The runtime bridge+ dispatches by token through the ``OrchestratorRegistry`` instead of branching on a+ ``match``.++ ``run`` takes the endpoint-chosen ``delivery`` (the wait-semantics axis) and honors+ it per the orchestrator's nature: an in-process orchestrator always blocks; a+ distributed one awaits completion for ``BLOCKING`` and returns a workflow id for+ ``FIRE_AND_FORGET``.++ ``supports_fire_and_forget`` is the capability the runner reads *before* dispatch:+ ``/start`` rejects (honestly, with a 4xx) when the resolved mode's orchestrator+ cannot do genuine async, rather than silently running blocking and acking.+ """++ supports_fire_and_forget: bool++ async def run(+ self,+ *,+ pipe_job: "PipeJob",+ delivery_assignment: "DeliveryAssignment | None",+ delivery: DeliveryMode,+ ) -> "PipelexPipeRunOutput": ...+++class OrchestratorRegistry:+ """Read view over the orchestrators contributed by discovered plugins.++ Keyed by the open ``OrchestrationMode`` token (a ``str``). Built once at boot from+ the registrar's accumulated orchestrators and stored on the hub.+ """++ def __init__(self, orchestrators: dict[OrchestrationMode, OrchestratorProtocol]):+ self._orchestrators: dict[OrchestrationMode, OrchestratorProtocol] = dict(orchestrators)++ def get_optional(self, *, mode: OrchestrationMode) -> OrchestratorProtocol | None:+ return self._orchestrators.get(mode)++ def has(self, *, mode: OrchestrationMode) -> bool:+ return mode in self._orchestrators++ @property+ def modes(self) -> list[OrchestrationMode]:+ return list(self._orchestrators)diff --git a/pipelex/plugins/plugin_manager.py b/pipelex/plugins/plugin_manager.pydeleted file mode 100644index 0f6fd8946..000000000--- a/pipelex/plugins/plugin_manager.py+++ /dev/null@@ -1,12 +0,0 @@-from pipelex.plugins.plugin_sdk_registry import PluginSdkRegistry---class PluginManager:- def __init__(self):- self.plugin_sdk_registry = PluginSdkRegistry()-- def setup(self):- pass-- def teardown(self):- self.plugin_sdk_registry.teardown()diff --git a/pipelex/plugins/plugin_sdk_registry.py b/pipelex/plugins/plugin_sdk_registry.pydeleted file mode 100644index d316560ad..000000000--- a/pipelex/plugins/plugin_sdk_registry.py+++ /dev/null@@ -1,24 +0,0 @@-from typing import Any--from pydantic import Field, RootModel--from pipelex.plugins.plugin import Plugin--PluginSdkRegistryRoot = dict[str, Any]---class PluginSdkRegistry(RootModel[PluginSdkRegistryRoot]):- root: PluginSdkRegistryRoot = Field(default_factory=dict)-- def teardown(self):- for sdk_instance in self.root.values():- if hasattr(sdk_instance, "teardown"):- sdk_instance.teardown()- self.root = {}-- def get_sdk_instance(self, plugin: Plugin) -> Any | None:- return self.root.get(plugin.sdk_handle)-- def set_sdk_instance(self, *, plugin: Plugin, sdk_instance: Any) -> Any:- self.root[plugin.sdk_handle] = sdk_instance- return sdk_instancediff --git a/pipelex/plugins/portkey/portkey_completions_factory.py b/pipelex/plugins/portkey/portkey_completions_factory.pyindex 4e82073b5..402e27f3f 100644--- a/pipelex/plugins/portkey/portkey_completions_factory.py+++ b/pipelex/plugins/portkey/portkey_completions_factory.py@@ -26,7 +26,7 @@ if TYPE_CHECKING:from pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec- from pipelex.plugins.plugin_sdk_registry import Plugin+ from pipelex.plugins.model_handle import ModelHandleclass PortkeyCompletionsFactory(OpenAICompletionsFactory):@@ -94,7 +94,7 @@ class PortkeyCompletionsFactory(OpenAICompletionsFactory):@classmethoddef make_portkey_openai_client_for_completions(cls,- plugin: Plugin,+ model_handle: ModelHandle,*,backend: InferenceBackend,) -> openai.AsyncOpenAI:@@ -105,8 +105,8 @@ class PortkeyCompletionsFactory(OpenAICompletionsFactory):endpoint = PortkeyFactory.get_endpoint(backend=backend)api_key = PortkeyFactory.get_api_key(backend=backend)- if not PortkeyOpenAISdkVariant.is_completions(plugin.sdk):- msg = f"Plugin '{plugin}' is not supported by '{cls.__name__}'"+ if not PortkeyOpenAISdkVariant.is_completions(model_handle.sdk):+ msg = f"ModelHandle '{model_handle}' is not supported by '{cls.__name__}'"raise PortkeyFactoryError(msg)return openai.AsyncOpenAI(diff --git a/pipelex/plugins/portkey/portkey_plugin.py b/pipelex/plugins/portkey/portkey_plugin.pynew file mode 100644index 000000000..a47a476fd--- /dev/null+++ b/pipelex/plugins/portkey/portkey_plugin.py@@ -0,0 +1,66 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.model_handle import ModelHandle+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol+++def _make_portkey_completions_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker # noqa: PLC0415+ from pipelex.plugins.portkey.portkey_completions_factory import PortkeyCompletionsFactory # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: PortkeyCompletionsFactory.make_portkey_openai_client_for_completions(model_handle=model_handle, backend=backend),+ )+ return OpenAICompletionsLLMWorker(+ openai_completions_factory=PortkeyCompletionsFactory(is_http_url_enabled=False),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++def _make_portkey_responses_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker # noqa: PLC0415+ from pipelex.plugins.portkey.portkey_responses_factory import PortkeyResponsesFactory # noqa: PLC0415++ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(+ handle=model_handle,+ build=lambda: PortkeyResponsesFactory.make_portkey_openai_client_for_responses(model_handle=model_handle, backend=backend),+ )+ return OpenAIResponsesLLMWorker(+ openai_responses_factory=PortkeyResponsesFactory(is_http_url_enabled=False),+ sdk_instance=sdk_instance,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class PortkeyPlugin:+ """Built-in driver for Portkey (OpenAI-compatible substrate)."""++ name = "portkey"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="portkey_completions", make_worker=_make_portkey_completions_worker)+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="portkey_responses", make_worker=_make_portkey_responses_worker)diff --git a/pipelex/plugins/portkey/portkey_responses_factory.py b/pipelex/plugins/portkey/portkey_responses_factory.pyindex ead67cd51..51a255779 100644--- a/pipelex/plugins/portkey/portkey_responses_factory.py+++ b/pipelex/plugins/portkey/portkey_responses_factory.py@@ -18,22 +18,22 @@ if TYPE_CHECKING:from pipelex.cogt.inference.inference_job_abstract import InferenceJobAbstractfrom pipelex.cogt.model_backends.backend import InferenceBackendfrom pipelex.cogt.model_backends.model_spec import InferenceModelSpec- from pipelex.plugins.plugin_sdk_registry import Plugin+ from pipelex.plugins.model_handle import ModelHandleclass PortkeyResponsesFactory(OpenAIResponsesFactory):@classmethoddef make_portkey_openai_client_for_responses(cls,- plugin: Plugin,+ model_handle: ModelHandle,*,backend: InferenceBackend,) -> openai.AsyncOpenAI:is_debug_enabled = PortkeyFactory.is_debug_enabled(backend=backend)endpoint = PortkeyFactory.get_endpoint(backend=backend)api_key = PortkeyFactory.get_api_key(backend=backend)- if not PortkeyOpenAISdkVariant.is_responses(plugin.sdk):- msg = f"Plugin '{plugin}' is not supported by '{cls.__name__}'"+ if not PortkeyOpenAISdkVariant.is_responses(model_handle.sdk):+ msg = f"ModelHandle '{model_handle}' is not supported by '{cls.__name__}'"raise PortkeyFactoryError(msg)return openai.AsyncOpenAI(diff --git a/pipelex/plugins/pypdfium2/pypdfium2_plugin.py b/pipelex/plugins/pypdfium2/pypdfium2_plugin.pynew file mode 100644index 000000000..4cb690346--- /dev/null+++ b/pipelex/plugins/pypdfium2/pypdfium2_plugin.py@@ -0,0 +1,34 @@+from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+from pipelex.cogt.model_backends.backend import InferenceBackend+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+from pipelex.reporting.reporting_protocol import ReportingProtocol+++def _make_pypdfium2_extract_worker(+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry, # noqa: ARG001 - stateless worker, no SDK-client caching+ reporting_delegate: ReportingProtocol | None,+) -> InferenceWorkerAbstract:+ from pipelex.plugins.pypdfium2.pypdfium2_worker import Pypdfium2Worker # noqa: PLC0415++ return Pypdfium2Worker(+ extra_config=backend.extra_config,+ inference_model=inference_model,+ reporting_delegate=reporting_delegate,+ )+++class Pypdfium2Plugin:+ """Built-in driver for local PDF text extraction via pypdfium2 (stateless, no remote SDK)."""++ name = "pypdfium2"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.EXTRACT, sdk="pypdfium2", make_worker=_make_pypdfium2_extract_worker)diff --git a/pipelex/plugins/registrar.py b/pipelex/plugins/registrar.pynew file mode 100644index 000000000..35af549cc--- /dev/null+++ b/pipelex/plugins/registrar.py@@ -0,0 +1,288 @@+from collections.abc import Callable+from typing import TYPE_CHECKING, Any, NamedTuple, TypeVar++from pydantic import BaseModel, Field++from pipelex.base_exceptions import ErrorReport+from pipelex.plugins.bundle_validator_registry import BundleValidatorProtocol+from pipelex.plugins.exceptions import (+ DuplicateBundleValidatorError,+ DuplicateHttpErrorMapperError,+ DuplicateInferenceBackendError,+ DuplicateModelListerError,+ DuplicateOrchestratorError,+ HubSlotAlreadyClaimedError,+)+from pipelex.plugins.inference_backend_registry import InferenceFamily, MakeWorkerFn+from pipelex.plugins.model_lister_registry import ListModelsFn+from pipelex.plugins.orchestrator_registry import OrchestratorProtocol+from pipelex.runtime_bridge.orchestration_mode import OrchestrationMode+from pipelex.types import StrEnum++if TYPE_CHECKING:+ from pipelex.system.configuration.configs import PipelexConfig+++_RegistryKeyT = TypeVar("_RegistryKeyT")+_RegistryValueT = TypeVar("_RegistryValueT")++# A plugin's framework-agnostic mapping from one transport/runtime exception to a+# structured ``ErrorReport``. A host runtime (``pipelex-api``) renders the report+# into its own HTTP error response (RFC 7807 + disclosure) — so core and the plugin+# name no web framework.+HttpErrorMapperFn = Callable[[Exception], ErrorReport]++# A thunk returning the concrete exception class a mapper applies to. The mapper is+# registered with this *provider* rather than the bare type so a plugin whose+# exception lives in a heavy orchestrator SDK (``temporalio``) can keep its+# ``register`` import-light: the provider — and any SDK import it performs to name+# the class — is invoked only when a host runtime resolves the mappers via+# ``get_http_error_mappers`` (i.e. at app construction), never at registration.+HttpErrorTypeProviderFn = Callable[[], type[Exception]]+++class _HttpErrorMapperContribution(NamedTuple):+ """One plugin's deferred HTTP-error-mapper contribution.++ A ``NamedTuple`` (not a model) because it holds two callables and is never+ serialized. ``exc_type_provider`` is resolved — and any SDK import it incurs+ paid — only by ``get_http_error_mappers``, which is what keeps a contributing+ plugin's ``register`` import-light.+ """++ exc_type_provider: HttpErrorTypeProviderFn+ to_error_report: HttpErrorMapperFn+ source_plugin: str+++class HubSlot(StrEnum):+ """A process-global capability slot a boot-orchestrator plugin may claim."""++ CONTENT_GENERATOR = "content_generator"+ PIPE_ROUTER = "pipe_router"+ PIPE_RUN = "pipe_run"+ TASK_MANAGER = "task_manager"+++class PluginOrigin(StrEnum):+ BUILTIN = "builtin"+ EXTERNAL = "external"+++class PluginStatus(StrEnum):+ REGISTERED = "registered"+ DISABLED = "disabled"+ BROKEN = "broken"+++class PluginDiscovery(BaseModel):+ """Observability record of one discovered plugin and what it contributed.++ Populated by ``build_registrar`` (origin/status/targets_api) and the registrar+ menu methods (one ``contributions`` line per contribution). Read by+ ``pipelex plugins list``.+ """++ name: str+ origin: PluginOrigin = Field(strict=False)+ status: PluginStatus = Field(strict=False)+ targets_api: int | None = None+ contributions: list[str] = Field(default_factory=list)+ detail: str | None = None+++class PluginRegistrar:+ """The accumulator a plugin's ``register`` writes into.++ A plugin only ever calls the menu methods below. ``build_registrar`` drives+ one plugin at a time (setting the "active" discovery so contributions are+ attributed and duplicate conflicts can name both contributors), then boot+ turns the accumulated ``inference_backends`` / ``orchestrators`` into the two+ keyed registries and applies the slot-claim thunks / teardown+ callbacks at their ordered apply-points.++ All duplicate detection is fail-loud and names both contributing plugins.+ """++ def __init__(self, *, config: "PipelexConfig"):+ self.config = config+ self.inference_backends: dict[tuple[InferenceFamily, str], MakeWorkerFn] = {}+ self.model_listers: dict[str, ListModelsFn] = {}+ self.orchestrators: dict[OrchestrationMode, OrchestratorProtocol] = {}+ self.bundle_validators: dict[OrchestrationMode, BundleValidatorProtocol] = {}+ # Ordered list (not a type-keyed dict) because the exception types are+ # resolved lazily — only ``get_http_error_mappers`` invokes the providers,+ # so duplicate-by-type detection is deferred to resolution time too.+ self.http_error_mappers: list[_HttpErrorMapperContribution] = []+ self.slot_claims: dict[HubSlot, Callable[[], Any]] = {}+ self.teardown_callbacks: list[Callable[[], None]] = []+ self.discoveries: list[PluginDiscovery] = []+ self._inference_sources: dict[tuple[InferenceFamily, str], str] = {}+ self._model_lister_sources: dict[str, str] = {}+ self._orchestrator_sources: dict[OrchestrationMode, str] = {}+ self._bundle_validator_sources: dict[OrchestrationMode, str] = {}+ self._slot_sources: dict[HubSlot, str] = {}+ # Reassigned per plugin by build_registrar; the floating default keeps the+ # menu methods safe to call outside a registration loop (e.g. a focused unit test).+ self._active = PluginDiscovery(name="(unregistered)", origin=PluginOrigin.BUILTIN, status=PluginStatus.REGISTERED)++ # ------------------------------------------------------------------ #+ # Driven by build_registrar (not by plugins)+ # ------------------------------------------------------------------ #++ def begin_plugin(self, *, name: str, origin: PluginOrigin, targets_api: int) -> PluginDiscovery:+ discovery = PluginDiscovery(name=name, origin=origin, status=PluginStatus.REGISTERED, targets_api=targets_api)+ self.discoveries.append(discovery)+ self._active = discovery+ return discovery++ # ------------------------------------------------------------------ #+ # Menu methods — the only surface a plugin's register() may call+ # ------------------------------------------------------------------ #++ def add_inference_backend(self, *, family: InferenceFamily, sdk: str, make_worker: MakeWorkerFn) -> None:+ self._add(+ store=self.inference_backends,+ sources=self._inference_sources,+ key=(family, sdk),+ value=make_worker,+ contribution=f"inference backend {family}:{sdk}",+ on_duplicate=lambda first_plugin, second_plugin: DuplicateInferenceBackendError(+ family=family, sdk=sdk, first_plugin=first_plugin, second_plugin=second_plugin+ ),+ )++ def add_model_lister(self, *, sdk: str, lister: ListModelsFn) -> None:+ self._add(+ store=self.model_listers,+ sources=self._model_lister_sources,+ key=sdk,+ value=lister,+ contribution=f"model lister {sdk}",+ on_duplicate=lambda first_plugin, second_plugin: DuplicateModelListerError(+ sdk=sdk, first_plugin=first_plugin, second_plugin=second_plugin+ ),+ )++ def add_orchestrator(self, *, mode: OrchestrationMode, orchestrator: OrchestratorProtocol) -> None:+ self._add(+ store=self.orchestrators,+ sources=self._orchestrator_sources,+ key=mode,+ value=orchestrator,+ contribution=f"orchestrator {mode}",+ on_duplicate=lambda first_plugin, second_plugin: DuplicateOrchestratorError(+ mode=mode, first_plugin=first_plugin, second_plugin=second_plugin+ ),+ )++ def add_bundle_validator(self, *, mode: OrchestrationMode, validator: BundleValidatorProtocol) -> None:+ self._add(+ store=self.bundle_validators,+ sources=self._bundle_validator_sources,+ key=mode,+ value=validator,+ contribution=f"bundle validator {mode}",+ on_duplicate=lambda first_plugin, second_plugin: DuplicateBundleValidatorError(+ mode=mode, first_plugin=first_plugin, second_plugin=second_plugin+ ),+ )++ def add_http_error_mapper(self, *, exc_type_provider: HttpErrorTypeProviderFn, to_error_report: HttpErrorMapperFn) -> None:+ """Contribute a mapping from a transport/runtime exception to a structured ``ErrorReport``.++ ``exc_type_provider`` is a thunk returning the concrete exception class the+ mapper applies to. It is resolved *lazily* by ``get_http_error_mappers`` (at a+ host runtime's app-construction time), never here — which is what keeps a+ plugin's ``register`` import-light: a plugin whose exception type lives in a+ heavy orchestrator SDK (``temporalio``) passes ``lambda: TemporalError`` and the+ SDK import is deferred until a host runtime actually consumes the mappers. The+ ``to_error_report`` closure is likewise uninvoked until an error is rendered.++ A host runtime (``pipelex-api``) iterates the resolved mappers at app+ construction and wraps each into one framework error handler (FastAPI, …) using+ its own RFC 7807 + disclosure rendering — so core and the plugin name no web+ framework. Duplicate-by-type detection is deferred to ``get_http_error_mappers``+ (the providers must run first); it stays fail-loud and names both plugins.+ """+ self.http_error_mappers.append(+ _HttpErrorMapperContribution(exc_type_provider=exc_type_provider, to_error_report=to_error_report, source_plugin=self._active.name)+ )+ self._active.contributions.append("http error mapper")++ def claim_content_generator(self, factory: Callable[[], Any]) -> None:+ self._claim(slot=HubSlot.CONTENT_GENERATOR, factory=factory)++ def claim_pipe_router(self, factory: Callable[[], Any]) -> None:+ self._claim(slot=HubSlot.PIPE_ROUTER, factory=factory)++ def claim_pipe_run(self, factory: Callable[[], Any]) -> None:+ self._claim(slot=HubSlot.PIPE_RUN, factory=factory)++ def claim_task_manager(self, factory: Callable[[], Any]) -> None:+ self._claim(slot=HubSlot.TASK_MANAGER, factory=factory)++ def add_teardown(self, callback: Callable[[], None]) -> None:+ self.teardown_callbacks.append(callback)+ self._active.contributions.append("teardown callback")++ # ------------------------------------------------------------------ #+ # Read accessors (for host runtimes consuming plugin contributions)+ # ------------------------------------------------------------------ #++ def get_http_error_mappers(self) -> dict[type[Exception], HttpErrorMapperFn]:+ """Resolve every contributed exc-type provider into a ``{exc_type: mapper}`` dict.++ The read view a host runtime (``pipelex-api``) iterates at app construction to+ register one framework error handler per exception type. Resolving the+ providers *here* (never at registration) is what lets a contributing plugin's+ ``register`` stay import-light: any orchestrator-SDK import a provider performs+ to name its concrete exception class is deferred to this call — which a host+ runtime makes only when it actually has the plugin (and therefore the SDK)+ installed, so the import always resolves. A freshly built dict, so a consumer+ cannot mutate the registrar's state. Fail-loud naming both plugins when two map+ the same resolved exception type.+ """+ resolved: dict[type[Exception], HttpErrorMapperFn] = {}+ sources: dict[type[Exception], str] = {}+ for contribution in self.http_error_mappers:+ exc_type = contribution.exc_type_provider()+ if exc_type in resolved:+ raise DuplicateHttpErrorMapperError(+ exc_type=exc_type.__qualname__, first_plugin=sources[exc_type], second_plugin=contribution.source_plugin+ )+ resolved[exc_type] = contribution.to_error_report+ sources[exc_type] = contribution.source_plugin+ return resolved++ # ------------------------------------------------------------------ #++ def _add(+ self,+ *,+ store: dict[_RegistryKeyT, _RegistryValueT],+ sources: dict[_RegistryKeyT, str],+ key: _RegistryKeyT,+ value: _RegistryValueT,+ contribution: str,+ on_duplicate: Callable[[str, str], Exception],+ ) -> None:+ """Shared body for the keyed registration menu methods (mirrors ``_claim`` for the slot menu).++ Fail-loud duplicate detection, store, source attribution, and contribution+ recording in one place; each ``add_*`` method supplies its keyed store, the+ parallel sources dict, and a factory that builds its distinctly-typed+ ``Duplicate*Error`` naming both contributing plugins.+ """+ if key in store:+ raise on_duplicate(sources[key], self._active.name)+ store[key] = value+ sources[key] = self._active.name+ self._active.contributions.append(contribution)++ def _claim(self, *, slot: HubSlot, factory: Callable[[], Any]) -> None:+ if slot in self.slot_claims:+ raise HubSlotAlreadyClaimedError(slot=slot, first_plugin=self._slot_sources[slot], second_plugin=self._active.name)+ self.slot_claims[slot] = factory+ self._slot_sources[slot] = self._active.name+ self._active.contributions.append(f"hub slot {slot}")diff --git a/pipelex/plugins/sdk_client_manager.py b/pipelex/plugins/sdk_client_manager.pynew file mode 100644index 000000000..20b9c7556--- /dev/null+++ b/pipelex/plugins/sdk_client_manager.py@@ -0,0 +1,12 @@+from pipelex.plugins.sdk_client_registry import SdkClientRegistry+++class SdkClientManager:+ def __init__(self):+ self.sdk_client_registry = SdkClientRegistry()++ def setup(self):+ pass++ def teardown(self):+ self.sdk_client_registry.teardown()diff --git a/pipelex/plugins/sdk_client_registry.py b/pipelex/plugins/sdk_client_registry.pynew file mode 100644index 000000000..887ecf35a--- /dev/null+++ b/pipelex/plugins/sdk_client_registry.py@@ -0,0 +1,37 @@+from collections.abc import Callable+from typing import Any++from pydantic import Field, RootModel++from pipelex.plugins.model_handle import ModelHandle++SdkClientRegistryRoot = dict[str, Any]+++class SdkClientRegistry(RootModel[SdkClientRegistryRoot]):+ root: SdkClientRegistryRoot = Field(default_factory=dict)++ def teardown(self):+ for sdk_instance in self.root.values():+ if hasattr(sdk_instance, "teardown"):+ sdk_instance.teardown()+ self.root = {}++ def get(self, model_handle: ModelHandle) -> Any | None:+ return self.root.get(model_handle.sdk_handle)++ def set(self, *, model_handle: ModelHandle, sdk_instance: Any) -> Any:+ self.root[model_handle.sdk_handle] = sdk_instance+ return sdk_instance++ def get_or_create(self, *, handle: ModelHandle, build: Callable[[], Any]) -> Any:+ """Return the cached SDK client for ``handle``, building and caching it on a miss.++ The DRY replacement for the ``get(...) or set(...)`` dance that used to be+ repeated in every dispatch arm. ``build`` is only called on a cache miss,+ so the heavy SDK client construction stays lazy.+ """+ existing = self.get(handle)+ if existing is not None:+ return existing+ return self.set(model_handle=handle, sdk_instance=build())diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex 1928ef1d2..d6ccd0694 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -28,7 +28,7 @@ def _is_in_temporal_activity() -> bool:imported in this process, no activity context can exist — the context is set by temporalio'sown machinery, which requires the module to be imported. Importing it here would put theentire temporalio extra (Rust bridge, protobuf) on every boot's critical path wherever the- ``pipelex[temporal]`` extra is installed, even for processes that never touch Temporal.+ ``pipelex-temporal`` plugin is installed, even for processes that never touch Temporal.Worker processes are unaffected: by the time any activity runs, ``temporalio.activity`` isnecessarily in ``sys.modules``."""diff --git a/pipelex/runtime_bridge/bridge.py b/pipelex/runtime_bridge/bridge.pydeleted file mode 100644index f1a932abf..000000000--- a/pipelex/runtime_bridge/bridge.py+++ /dev/null@@ -1,435 +0,0 @@-"""Framework-agnostic Pipelex runtime-bridge surface for host runtimes.--This module contains the boundary types (``PipelexPipeRunInput`` /-``PipelexPipeRunOutput``) and the dispatch entry-point-(``run_pipe_via_bridge``) used by host runtimes (Mistral Workflows, raw-Temporal, future plugins) to invoke Pipelex pipes from inside their own-activities. It deliberately does NOT import any host-runtime-specific-modules at module top-level so that callers can use the bridge directly-(Tier 3 usage) and so that unit tests can exercise it without optional-host-runtime deps installed.--The Temporal extra is lazy-imported only inside the temporal-mode branches.-"""--from __future__ import annotations--from typing import TYPE_CHECKING, Any, Callable, cast--import shortuuid-from pydantic import BaseModel, ConfigDict, Field, ValidationError--from pipelex.core.memory.working_memory import MAIN_STUFF_NAME-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.hub import (- get_required_pipe,- scoped_pipe_router,-)-from pipelex.libraries.library_crate import LibraryCrate-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment-from pipelex.pipe_run.exceptions import PipeJobError, PipeRouterError, PipeRunError-from pipelex.pipe_run.pipe_job_factory import PipeJobFactory-from pipelex.pipe_run.pipe_router import PipeRouter-from pipelex.pipe_run.pipe_run import PipeRun-from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory-from pipelex.pipeline.exceptions import PipeExecutionError, PipelineExecutionError-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.runtime_bridge.bootstrap import ensure_pipelex_booted-from pipelex.runtime_bridge.exceptions import (- MissingMistralWorkflowsPluginError,- MissingPipelexTemporalExtraError,- PipelexBridgeDispatchError,-)-from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode-from pipelex.runtime_bridge.primitives.scoped_library import scoped_library_for_crate-from pipelex.system.telemetry.otel_constants import OTelConstants--if TYPE_CHECKING:- from pipelex.base_exceptions import PipelexError- from pipelex.core.memory.working_memory import WorkingMemory- from pipelex.core.pipes.pipe_output import PipeOutput- from pipelex.graph.trace_context import TraceContext- from pipelex.pipe_run.pipe_job import PipeJob- from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol---# Pipe-execution failures the bridge converts into PipelexBridgeDispatchError so a host can catch a-# single error type regardless of execution mode. The Temporal branches additionally catch-# WorkflowExecutionError (lazy-imported there to keep temporal off the module import path); wrapping-# it with ``from exc`` loses no signal — the structured ErrorReport stays reachable via ``__cause__``-# and is surfaced by PipelexBridgeDispatchError.to_error_report()'s cause-chain enrichment.-_PIPE_DISPATCH_ERRORS: tuple[type[PipelexError], ...] = (- PipeRunError,- PipeJobError,- PipeRouterError,- PipeExecutionError,- PipelineExecutionError,-)---class PipelexPipeRunInput(BaseModel):- """JSON-safe input crossing the host-runtime / Temporal boundary."""-- model_config = ConfigDict(extra="forbid")-- pipe_code: str- inputs: dict[str, Any] = Field(default_factory=dict)- output_name: str | None = None- pipeline_run_id: str | None = None- user_id: str | None = None- library_crate_dump: dict[str, Any] | None = None- execution_mode: PipelexExecutionMode = PipelexExecutionMode.DIRECT- delivery_assignment_dump: dict[str, Any] | None = None---class PipelexPipeRunOutput(BaseModel):- """JSON-safe output crossing the host-runtime / Temporal boundary."""-- model_config = ConfigDict(extra="forbid")-- output_dict: dict[str, Any]- main_stuff_name: str | None = None- pipeline_run_id: str- workflow_id: str | None = None- is_completed: bool- graph_spec_dump: dict[str, Any] | None = None- # graph_assembly_error / usage_assembly_error mirror the same fields on PipeOutput: a non-None- # value means assembly of the graph / token usage failed, which a host must be able to tell- # apart from "assembly was off" (a None graph_spec_dump / tokens_usages_dump). tokens_usages_dump- # is the JSON-safe dump of the AnyTokensUsage discriminated union so a host can render the- # end-of-run cost report: None when cost reporting was off, [] when on but no inference happened.- graph_assembly_error: str | None = None- tokens_usages_dump: list[dict[str, Any]] | None = None- usage_assembly_error: str | None = None---async def run_pipe_via_bridge(- input_payload: PipelexPipeRunInput,- *,- trace_context: TraceContext | None = None,-) -> PipelexPipeRunOutput:- """Run a Pipelex pipe from inside a host-runtime activity.-- Booting Pipelex on first call (no-op if already initialized); validating- the input; opening a per-call scoped library if a ``library_crate_dump``- is provided; then dispatching to the requested execution mode.-- The optional ``trace_context`` is plumbed into ``JobMetadata`` so callers- (e.g. a streaming activity) that already opened a ``GraphTracerManager``- tracer for this pipeline run get per-step graph/usage trace events flowing- through the configured event log. ``trace_context`` is honored for ``DIRECT``- mode only — it is deliberately nulled for the Temporal modes, which have- their own event-log infrastructure via ``pipeline_run_setup``. Forwarding- a host ``trace_context`` to a Temporal mode would make Pipelex's Temporal- workflow open its tracer under the host's graph id and merge its trace- events into the host's graph, so the bridge does not thread it through.- """- ensure_pipelex_booted()-- library_crate = _decode_library_crate(input_payload.library_crate_dump)- delivery_assignment = _decode_delivery_assignment(input_payload.delivery_assignment_dump)- _validate_input(input_payload, delivery_assignment=delivery_assignment)-- with scoped_library_for_crate(library_crate, library_id_prefix="runtime_bridge"):- # trace_context is honored for DIRECT only. The Temporal modes have their- # own event-log infrastructure (via pipeline_run_setup); forwarding a host- # trace_context there would make WfPipeRouter open its tracer under the- # host's graph_id and merge Pipelex's Temporal trace events into the host- # graph — exactly the cross-contamination the contract forbids. Null it- # for the non-DIRECT modes.- is_direct = input_payload.execution_mode is PipelexExecutionMode.DIRECT- pipe_job = build_pipe_job_from_input(- input_payload=input_payload,- library_crate=library_crate,- trace_context=trace_context if is_direct else None,- )-- match input_payload.execution_mode:- case PipelexExecutionMode.DIRECT:- return await _run_direct(pipe_job=pipe_job, delivery_assignment=delivery_assignment)- case PipelexExecutionMode.TEMPORAL_BLOCKING:- _require_pipelex_temporal_extra()- return await _run_temporal_blocking(pipe_job=pipe_job, delivery_assignment=delivery_assignment)- case PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:- _require_pipelex_temporal_extra()- return await _run_temporal_fire_and_forget(pipe_job=pipe_job, delivery_assignment=delivery_assignment)- case PipelexExecutionMode.MISTRAL_NATIVE:- return await _run_mistral_native(pipe_job=pipe_job, delivery_assignment=delivery_assignment)---def build_pipe_job_from_input(- input_payload: PipelexPipeRunInput,- *,- library_crate: LibraryCrate | None,- trace_context: TraceContext | None = None,-) -> PipeJob:- """Hydrate a PipeJob from JSON-safe input.-- Looks up the pipe in the active library; the caller is responsible for- making sure the active library contains the pipe (by passing a- ``library_crate_dump`` or pre-loading the library at boot).-- The optional ``trace_context`` is plumbed into ``JobMetadata`` so a- caller (e.g. a streaming activity) that has already opened a- ``GraphTracerManager`` tracer for this pipeline run can have per-step- ``PipeStartEvent`` / ``PipeEndSuccessEvent`` events flow through the- pipe execution. When ``None``, no tracing happens (current default).- """- pipe = get_required_pipe(pipe_code=input_payload.pipe_code)-- # When the caller supplies a trace_context but no explicit pipeline_run_id, adopt the- # trace_context's lookup_key (= tracer_key or graph_id) as the run id. lookup_key is the key the- # caller's already-open tracer is registered under in GraphTracerManager AND the key PipeRun.run- # closes it under (close_tracer takes pipeline_run_id); minting a fresh id would leak the tracer.- #- # On every reachable path lookup_key == graph_id, so it is also the partition tracing_assembly- # reads under: only the Temporal child path (wf_pipe_router) opens a tracer with tracer_key !=- # graph_id, and it never routes through the bridge — non-DIRECT modes null trace_context, and it- # builds its PipeJob directly, not via this function. A host that hand-built a divergent keyed- # trace_context and forced it through DIRECT would split event emission (under graph_id) from- # assembly (under tracer_key); nothing constructs that, and it is documented as unsupported in- # wip/runtime-bridge/bridge-keyed-tracer-unsupported.md (graph_id "is typically the pipeline run id").- if input_payload.pipeline_run_id is not None:- pipeline_run_id = input_payload.pipeline_run_id- elif trace_context is not None:- pipeline_run_id = trace_context.lookup_key- else:- pipeline_run_id = shortuuid.uuid()-- working_memory: WorkingMemory- if input_payload.inputs:- working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(- pipeline_inputs=input_payload.inputs,- search_domain_codes=[pipe.domain_code],- )- else:- working_memory = WorkingMemoryFactory.make_empty()-- job_metadata = JobMetadata(- user_id=input_payload.user_id or OTelConstants.DEFAULT_USER_ID,- pipeline_run_id=pipeline_run_id,- trace_context=trace_context,- )- pipe_run_params = PipeRunParamsFactory.make_run_params()-- return PipeJobFactory.make_pipe_job(- pipe=pipe,- pipe_run_params=pipe_run_params,- job_metadata=job_metadata,- working_memory=working_memory,- output_name=input_payload.output_name,- library_crate=library_crate,- )---def serialize_pipe_output(pipe_output: PipeOutput) -> dict[str, Any]:- """Dehydrate a PipeOutput's working memory to a JSON-safe dict.-- Always uses ``WorkingMemory.dump_for_temporal()`` — the same format Pipelex- uses internally for Temporal transit. The shape is stable regardless of- whether a ``library_crate`` was attached:- ``{"root": {stuff_name: {"content": {...}, ...}}, "aliases": {...}}``.-- Type metadata embedded by ``dump_for_temporal`` lets callers reconstruct a- typed ``WorkingMemory`` when they have the matching class registry in- scope (e.g. via ``hydrate_working_memory``).- """- return pipe_output.working_memory.dump_for_temporal()---def _validate_input(input_payload: PipelexPipeRunInput, *, delivery_assignment: DeliveryAssignment | None) -> None:- if input_payload.execution_mode is PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:- if delivery_assignment is None or not delivery_assignment.has_delivery_target:- msg = (- "PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET requires a delivery_assignment_dump with at least one "- "delivery target (storage or a webhook); otherwise the pipe completion would be silently dropped."- )- raise PipelexBridgeDispatchError(msg)---def _decode_library_crate(library_crate_dump: dict[str, Any] | None) -> LibraryCrate | None:- if library_crate_dump is None:- return None- try:- return LibraryCrate.model_validate(library_crate_dump)- except ValidationError as exc:- msg = f"Invalid library_crate_dump passed to the runtime bridge: {exc}"- raise PipelexBridgeDispatchError(msg) from exc---def _decode_delivery_assignment(delivery_assignment_dump: dict[str, Any] | None) -> DeliveryAssignment | None:- if delivery_assignment_dump is None:- return None- try:- return DeliveryAssignment.model_validate(delivery_assignment_dump)- except ValidationError as exc:- msg = f"Invalid delivery_assignment_dump passed to the runtime bridge: {exc}"- raise PipelexBridgeDispatchError(msg) from exc---async def _run_direct(- pipe_job: PipeJob,- *,- delivery_assignment: DeliveryAssignment | None,-) -> PipelexPipeRunOutput:- # DIRECT mode forces in-process execution even inside a Temporal-enabled- # worker. Scope the in-process router as the active router for the WHOLE- # run so nested controller sub-pipes — which dispatch through- # get_pipe_router() — resolve THIS router rather than falling back to the- # hub default. Without the scope, the hub default in a Temporal-enabled- # worker is the Temporal router, so a DIRECT-mode sequence/batch would leak- # its nested pipes to Temporal, defeating the point of DIRECT.- direct_router = PipeRouter()- with scoped_pipe_router(direct_router):- pipe_run = PipeRun(pipe_router=direct_router)- try:- pipe_output = await pipe_run.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)- except _PIPE_DISPATCH_ERRORS as exc:- msg = f"Pipe execution failed in DIRECT mode for pipe '{pipe_job.pipe.code}': {exc}"- raise PipelexBridgeDispatchError(msg) from exc-- return _serialize_completed_output(- pipe_output=pipe_output,- workflow_id=None,- )---async def _run_temporal_blocking(- pipe_job: PipeJob,- *,- delivery_assignment: DeliveryAssignment | None,-) -> PipelexPipeRunOutput:- from pipelex.temporal.exceptions import WorkflowExecutionError # noqa: PLC0415- from pipelex.temporal.tprl_pipe.temporal_pipe_run import make_temporal_pipe_run # noqa: PLC0415-- # A pipe failure inside the Temporal workflow surfaces as WorkflowExecutionError (a- # TemporalFlowError, not in _PIPE_DISPATCH_ERRORS); catch it too so a Temporal-mode failure is- # wrapped just like DIRECT/mistral. `from exc` keeps its structured ErrorReport reachable.- dispatch_errors: tuple[type[PipelexError], ...] = (*_PIPE_DISPATCH_ERRORS, WorkflowExecutionError)- temporal_pipe_run = make_temporal_pipe_run()- try:- pipe_output = await temporal_pipe_run.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)- except dispatch_errors as exc:- msg = f"Pipe execution failed in TEMPORAL_BLOCKING mode for pipe '{pipe_job.pipe.code}': {exc}"- raise PipelexBridgeDispatchError(msg) from exc-- # Report the actual Temporal workflow id, not the bare pipeline_run_id. run() started the- # workflow with make_workflow_id(pipeline_run_id), which prefixes the id in non-NORMAL run modes- # (ut-/ci-/cc-/cct- — see temporal_manager.make_top_workflow_id); the bare pipeline_run_id would- # not resolve there. Recomputing via the same make_workflow_id keeps a single source of truth and- # matches the id fire-and-forget returns from start(). In production (RunMode.NORMAL) the prefix- # is empty, so this equals pipeline_run_id.- workflow_id = temporal_pipe_run.make_workflow_id(pipeline_run_id=pipe_job.job_metadata.pipeline_run_id)-- return _serialize_completed_output(- pipe_output=pipe_output,- workflow_id=workflow_id,- )---async def _run_temporal_fire_and_forget(- pipe_job: PipeJob,- *,- delivery_assignment: DeliveryAssignment | None,-) -> PipelexPipeRunOutput:- from pipelex.temporal.exceptions import WorkflowExecutionError # noqa: PLC0415- from pipelex.temporal.tprl_pipe.temporal_pipe_run import make_temporal_pipe_run # noqa: PLC0415-- # start() raises WorkflowExecutionError on a dispatch failure (a TemporalFlowError, not in- # _PIPE_DISPATCH_ERRORS); catch it too so fire-and-forget dispatch failures wrap uniformly.- dispatch_errors: tuple[type[PipelexError], ...] = (*_PIPE_DISPATCH_ERRORS, WorkflowExecutionError)- temporal_pipe_run = make_temporal_pipe_run()- try:- workflow_id, _handle = await temporal_pipe_run.start(pipe_job=pipe_job, delivery_assignment=delivery_assignment)- except dispatch_errors as exc:- msg = f"Pipe dispatch failed in TEMPORAL_FIRE_AND_FORGET mode for pipe '{pipe_job.pipe.code}': {exc}"- raise PipelexBridgeDispatchError(msg) from exc-- return PipelexPipeRunOutput(- output_dict={},- main_stuff_name=None,- pipeline_run_id=pipe_job.job_metadata.pipeline_run_id,- workflow_id=workflow_id,- is_completed=False,- graph_spec_dump=None,- )---def _serialize_completed_output(- pipe_output: PipeOutput,- *,- workflow_id: str | None,-) -> PipelexPipeRunOutput:- output_dict = serialize_pipe_output(pipe_output=pipe_output)-- main_stuff_name = _resolve_main_stuff_root_key(pipe_output=pipe_output)-- graph_spec_dump = pipe_output.graph_spec.model_dump(mode="json") if pipe_output.graph_spec is not None else None- tokens_usages_dump = [usage.model_dump(mode="json") for usage in pipe_output.tokens_usages] if pipe_output.tokens_usages is not None else None-- return PipelexPipeRunOutput(- output_dict=output_dict,- main_stuff_name=main_stuff_name,- pipeline_run_id=pipe_output.pipeline_run_id,- workflow_id=workflow_id,- is_completed=True,- graph_spec_dump=graph_spec_dump,- graph_assembly_error=pipe_output.graph_assembly_error,- tokens_usages_dump=tokens_usages_dump,- usage_assembly_error=pipe_output.usage_assembly_error,- )---def _resolve_main_stuff_root_key(pipe_output: PipeOutput) -> str | None:- """Return the actual ``root`` dict key under which the main stuff lives.-- The main stuff can either sit directly at ``root[MAIN_STUFF_NAME]`` or be- referenced via ``aliases[MAIN_STUFF_NAME]`` pointing at its real name.- Callers indexing the output_dict need the actual root key, not the- stuff's display ``stuff_name``.- """- working_memory = pipe_output.working_memory- if MAIN_STUFF_NAME in working_memory.root:- return MAIN_STUFF_NAME- aliased_target = working_memory.aliases.get(MAIN_STUFF_NAME)- if aliased_target is not None and aliased_target in working_memory.root:- return aliased_target- return None---def _require_pipelex_temporal_extra() -> None:- try:- import temporalio # noqa: F401, PLC0415- except ImportError as exc:- msg = "TEMPORAL_* execution modes require the pipelex[temporal] extra. Install with: pip install 'pipelex[temporal]'"- raise MissingPipelexTemporalExtraError(msg) from exc---async def _run_mistral_native(- pipe_job: PipeJob,- *,- delivery_assignment: DeliveryAssignment | None,-) -> PipelexPipeRunOutput:- try:- from pipelex_mistralai_workflows.primitives.pipe_run import ( # type: ignore[import-not-found] # noqa: PLC0415 # pyright: ignore[reportMissingImports]- make_mistral_workflows_pipe_run as _make_pipe_run_untyped, # pyright: ignore[reportUnknownVariableType]- )- except ImportError as exc:- msg = (- "PipelexExecutionMode.MISTRAL_NATIVE requires the pipelex-mistralai-workflows "- "package. Install with: pip install pipelex-mistralai-workflows"- )- raise MissingMistralWorkflowsPluginError(msg) from exc-- make_pipe_run = cast("Callable[[], PipeRunProtocol]", _make_pipe_run_untyped)- pipe_run = make_pipe_run()- try:- pipe_output = await pipe_run.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)- except (PipeRunError, PipeJobError, PipeRouterError, PipeExecutionError, PipelineExecutionError) as exc:- msg = f"Pipe execution failed in MISTRAL_NATIVE mode for pipe '{pipe_job.pipe.code}': {exc}"- raise PipelexBridgeDispatchError(msg) from exc-- return _serialize_completed_output(- pipe_output=pipe_output,- workflow_id=pipe_output.pipeline_run_id,- )diff --git a/pipelex/runtime_bridge/delivery_mode.py b/pipelex/runtime_bridge/delivery_mode.pynew file mode 100644index 000000000..432e2c604--- /dev/null+++ b/pipelex/runtime_bridge/delivery_mode.py@@ -0,0 +1,32 @@+"""The delivery axis: *whether the caller waits* for a pipe to complete.++``DeliveryMode`` is a **closed** core ``StrEnum`` and is **endpoint-intrinsic** — set by+the endpoint, never received from a caller and never carried in config. ``/execute`` and+``/validate`` are synchronous (``BLOCKING``); ``/start`` is fire-and-forget+(``FIRE_AND_FORGET``). Delivery is passed as a parameter to ``OrchestratorProtocol.run``;+each orchestrator honors it per its nature (an in-process orchestrator always blocks; a+Temporal orchestrator awaits completion vs returns a workflow id).++This axis is genuinely closed, so the StrEnum-everywhere standard applies in full:+exhaustive ``match``/``case`` with no default arm. A future ``STREAMING`` delivery would+be core's call to add here, never a plugin's — which is exactly why it stays an enum,+unlike the open :mod:`pipelex.runtime_bridge.orchestration_mode` token.+"""++from pipelex.types import StrEnum+++class DeliveryMode(StrEnum):+ """Whether a pipe run blocks until completion or returns immediately.++ BLOCKING: the call awaits the pipe to completion and returns the full output.+ Set by the synchronous endpoints (``/execute``, ``/validate``).+ FIRE_AND_FORGET: the call returns immediately (with a workflow id when the+ orchestrator is async-capable); completion is signalled out-of-band via the+ DeliveryAssignment (webhook / storage). Set by ``/start``. Only an+ async-capable orchestrator (``supports_fire_and_forget``) can honor it+ genuinely; the endpoint checks capability before dispatching.+ """++ BLOCKING = "blocking"+ FIRE_AND_FORGET = "fire_and_forget"diff --git a/pipelex/runtime_bridge/direct_orchestrator.py b/pipelex/runtime_bridge/direct_orchestrator.pynew file mode 100644index 000000000..787392929--- /dev/null+++ b/pipelex/runtime_bridge/direct_orchestrator.py@@ -0,0 +1,55 @@+"""The core, always-on DIRECT orchestrator.++Runs a pipe in-process. Extracted verbatim from the bridge's old ``_run_direct``+arm so the bridge dispatches by mode through the ``OrchestratorRegistry`` instead+of a ``match``. Registered unconditionally by the core ``direct`` plugin.++Import-light: core pipe-run + the shared serialization helpers; no host-runtime SDK.+"""++from __future__ import annotations++from typing import TYPE_CHECKING++from pipelex.hub import scoped_pipe_router+from pipelex.pipe_run.pipe_router import PipeRouter+from pipelex.pipe_run.pipe_run import PipeRun+from pipelex.runtime_bridge.exceptions import PipelexBridgeDispatchError+from pipelex.runtime_bridge.serialization import PIPE_DISPATCH_ERRORS, serialize_completed_output++if TYPE_CHECKING:+ from pipelex.pipe_run.delivery_assignment import DeliveryAssignment+ from pipelex.pipe_run.pipe_job import PipeJob+ from pipelex.runtime_bridge.delivery_mode import DeliveryMode+ from pipelex.runtime_bridge.payloads import PipelexPipeRunOutput+++class DirectOrchestrator:+ """In-process execution; no Temporal involved on Pipelex's side."""++ # In-process execution has no genuine async path — it always blocks until the pipe+ # completes — so it cannot honor FIRE_AND_FORGET. ``/start`` reads this and rejects+ # rather than running blocking and falsely acking.+ supports_fire_and_forget = False++ async def run(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None, delivery: DeliveryMode) -> PipelexPipeRunOutput: # noqa: ARG002 — delivery accepted for protocol uniformity; in-process always blocks+ # DIRECT mode forces in-process execution even inside a Temporal-enabled+ # worker. Scope the in-process router as the active router for the WHOLE+ # run so nested controller sub-pipes — which dispatch through+ # get_pipe_router() — resolve THIS router rather than falling back to the+ # hub default. Without the scope, the hub default in a Temporal-enabled+ # worker is the Temporal router, so a DIRECT-mode sequence/batch would leak+ # its nested pipes to Temporal, defeating the point of DIRECT.+ direct_router = PipeRouter()+ with scoped_pipe_router(direct_router):+ pipe_run = PipeRun(pipe_router=direct_router)+ try:+ pipe_output = await pipe_run.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)+ except PIPE_DISPATCH_ERRORS as exc:+ msg = f"Pipe execution failed in DIRECT mode for pipe '{pipe_job.pipe.code}': {exc}"+ raise PipelexBridgeDispatchError(msg) from exc++ return serialize_completed_output(+ pipe_output=pipe_output,+ workflow_id=None,+ )diff --git a/pipelex/runtime_bridge/exceptions.py b/pipelex/runtime_bridge/exceptions.pyindex bf2bca3ca..0a3b4b59d 100644--- a/pipelex/runtime_bridge/exceptions.py+++ b/pipelex/runtime_bridge/exceptions.py@@ -1,23 +1,58 @@from pipelex.base_exceptions import PipelexError+from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE, OrchestrationModeclass PipelexRuntimeBridgeError(PipelexError):"""Base for errors raised by the Pipelex runtime-bridge surface."""-class MissingPipelexTemporalExtraError(PipelexRuntimeBridgeError):- """Raised when a TEMPORAL_* execution mode is requested without the pipelex[temporal] extra."""+class MissingOrchestratorError(PipelexRuntimeBridgeError):+ """Raised when no orchestrator is registered for a requested orchestration mode.- # The message carries the actionable pip-install hint; keep it under STRICT disclosure.+ ``orchestration_mode`` is an open token: ``"direct"`` is contributed by core, and every+ other token by the plugin that owns its orchestrator (``"temporal"`` →+ ``pipelex-temporal``, ``"mistralai-workflows"`` → ``pipelex-mistralai-workflows``). A+ lookup miss therefore means *that mode's plugin is not installed* — the message is+ generic and names no orchestrator, so core stays fully decoupled from its plugins. The+ one special case is ``"direct"``: its orchestrator is core and always present, so a miss+ is a boot/discovery fault, not a missing plugin.+ """++ # The message is caller-actionable ("is its plugin installed?"); keep it under STRICT disclosure._authors_caller_facing_message = True+ def __init__(self, *, mode: OrchestrationMode):+ self.mode = mode+ super().__init__(_build_missing_message(noun="orchestrator", mode=mode))+-class MissingMistralWorkflowsPluginError(PipelexRuntimeBridgeError):- """Raised when MISTRAL_NATIVE mode is requested without ``pipelex-mistralai-workflows`` installed."""+class MissingBundleValidatorError(PipelexRuntimeBridgeError):+ """Raised when no bundle validator is registered for a requested orchestration mode.- # The message carries the actionable pip-install hint; keep it under STRICT disclosure.+ The ``/validate`` counterpart of ``MissingOrchestratorError``: ``"direct"`` is contributed+ by core and every other token by its owning plugin, so a lookup miss means that mode's+ plugin is not installed. The message is generic and names no orchestrator; ``"direct"`` is+ the boot/discovery special case (its validator is core and always present).+ """++ # The message is caller-actionable ("is its plugin installed?"); keep it under STRICT disclosure._authors_caller_facing_message = True+ def __init__(self, *, mode: OrchestrationMode):+ self.mode = mode+ super().__init__(_build_missing_message(noun="bundle validator", mode=mode))+++def _build_missing_message(*, noun: str, mode: OrchestrationMode) -> str:+ """Generic, plugin-decoupled message shared by both Missing* errors (D-F).++ A plain string compare on ``"direct"`` (not an enum ``match``) — core's one built-in+ token — singles out the boot/discovery fault from the missing-plugin case.+ """+ if mode == DIRECT_ORCHESTRATION_MODE:+ return f"No {noun} is registered for the core '{DIRECT_ORCHESTRATION_MODE}' mode; this indicates a boot or plugin-discovery problem."+ return f"No {noun} is registered for orchestration mode '{mode}'; is its plugin installed?"+class PipelexBridgeDispatchError(PipelexRuntimeBridgeError):"""Raised when a pipe dispatched through the bridge fails (validation or execution)."""diff --git a/pipelex/runtime_bridge/execution_mode.py b/pipelex/runtime_bridge/execution_mode.pydeleted file mode 100644index 4f2dd4396..000000000--- a/pipelex/runtime_bridge/execution_mode.py+++ /dev/null@@ -1,51 +0,0 @@-from pipelex.types import StrEnum---class PipelexExecutionMode(StrEnum):- """How a Pipelex pipe runs inside a host runtime activity.-- DIRECT: in-process; no Temporal involved on Pipelex's side; activity blocks- until the pipe completes. Fastest feedback, simplest ops.- TEMPORAL_BLOCKING: dispatch the pipe as a Pipelex Temporal workflow; the- activity awaits completion. Pipe runs durably on Pipelex's worker- fleet. Requires the pipelex[temporal] extra.- TEMPORAL_FIRE_AND_FORGET: dispatch the pipe as a Pipelex Temporal workflow- and return immediately with the workflow_id. Activity does NOT wait;- completion is signalled out-of-band via DeliveryAssignment (webhook /- storage). Same dep requirements as TEMPORAL_BLOCKING.- ``delivery_assignment_dump`` is required.- MISTRAL_NATIVE: decompose the pipe into native Mistral Workflows primitives- on the same worker — controllers as child workflows, leaves as- activities. Surfaces per-step retry / signals / cancellation through- the host-runtime APIs. Requires the- ``pipelex-mistralai-workflows`` package.- """-- DIRECT = "direct"- TEMPORAL_BLOCKING = "temporal_blocking"- TEMPORAL_FIRE_AND_FORGET = "temporal_fire_and_forget"- MISTRAL_NATIVE = "mistral_native"-- @property- def requires_pipelex_temporal(self) -> bool:- match self:- case PipelexExecutionMode.DIRECT | PipelexExecutionMode.MISTRAL_NATIVE:- return False- case PipelexExecutionMode.TEMPORAL_BLOCKING | PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:- return True-- @property- def requires_mistral_workflows_extra(self) -> bool:- match self:- case PipelexExecutionMode.MISTRAL_NATIVE:- return True- case PipelexExecutionMode.DIRECT | PipelexExecutionMode.TEMPORAL_BLOCKING | PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:- return False-- @property- def is_fire_and_forget(self) -> bool:- match self:- case PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:- return True- case PipelexExecutionMode.DIRECT | PipelexExecutionMode.TEMPORAL_BLOCKING | PipelexExecutionMode.MISTRAL_NATIVE:- return Falsediff --git a/pipelex/runtime_bridge/orchestration_mode.py b/pipelex/runtime_bridge/orchestration_mode.pynew file mode 100644index 000000000..5e3c86724--- /dev/null+++ b/pipelex/runtime_bridge/orchestration_mode.py@@ -0,0 +1,35 @@+"""The orchestration-mode axis: *which* orchestrator runs a pipe.++``orchestration_mode`` is an **open string token**, not a closed enum. Core owns+only ``"direct"`` (its single built-in, in-process orchestrator); every other token+is contributed by the plugin that owns the orchestrator — ``pipelex-temporal`` owns+``"temporal"``, ``pipelex-mistralai-workflows`` owns ``"mistralai-workflows"``. The+orchestrator/validator registries are keyed by this token and validation is a+registry lookup (*is this mode registered?* → ``MissingOrchestratorError`` on a miss),+never an enum-membership check.++This is the deliberate, justified exception to the repo's StrEnum-everywhere standard:+that standard exists to make the linter scream when a *closed* value is added and a+branch forgotten, but a plugin-contributed set is genuinely open — exhaustive matching+is impossible, and the dict-lookup-or-raise *is* the correct unknown-handling. It is+also the only choice faithful to "the public base names no orchestrator": a closed+``{DIRECT, TEMPORAL, MISTRAL}`` enum would re-introduce the very core→plugin coupling+the plugin-system externalization removed.++``OrchestrationMode`` is a plain ``str`` alias (not a ``NewType``) so plugins pass raw+string tokens with no casts at the registry boundary — the registry is the validator,+so a ``NewType``'s casts would buy zero validation. The wait-semantics axis lives+separately in :mod:`pipelex.runtime_bridge.delivery_mode` (a closed+``DeliveryMode`` enum) — the two are orthogonal.+"""++from typing import Final, TypeAlias++# A registered orchestration token. A semantic alias documenting intent at registry /+# protocol signatures (``dict[OrchestrationMode, ...]``); assignment-compatible with+# plain ``str`` so a plugin registers under a bare string literal with no cast.+OrchestrationMode: TypeAlias = str++# Core's one built-in token. Core code references this constant rather than the bare+# ``"direct"`` literal; every other token is owned by the plugin that registers it.+DIRECT_ORCHESTRATION_MODE: Final[str] = "direct"diff --git a/pipelex/runtime_bridge/payloads.py b/pipelex/runtime_bridge/payloads.pynew file mode 100644index 000000000..1553fdc59--- /dev/null+++ b/pipelex/runtime_bridge/payloads.py@@ -0,0 +1,57 @@+"""JSON-safe boundary payloads for the runtime bridge.++The SPI boundary contract between core's ``DirectOrchestrator`` (and the open+``pipelex-api`` runner) and the cross-process dispatch entry-point that now lives+in the closed ``pipelex-transport`` library. Kept import-light and separate from+any dispatch logic so they can be referenced (e.g. by the orchestrator SPI /+``OrchestratorProtocol``) without pulling the bootstrap path — which would form an+import cycle: pydantic, the orchestration/delivery types, stdlib typing only.+"""++from typing import Any++from pydantic import BaseModel, ConfigDict, Field++from pipelex.runtime_bridge.delivery_mode import DeliveryMode+from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE+++class PipelexPipeRunInput(BaseModel):+ """JSON-safe input crossing the host-runtime / Temporal boundary."""++ model_config = ConfigDict(extra="forbid")++ pipe_code: str+ inputs: dict[str, Any] = Field(default_factory=dict)+ output_name: str | None = None+ pipeline_run_id: str | None = None+ user_id: str | None = None+ library_crate_dump: dict[str, Any] | None = None+ # Two orthogonal axes: which orchestrator runs the pipe (open token, defaults to the+ # core in-process orchestrator) and whether the caller waits (endpoint-set; defaults+ # to BLOCKING). ``orchestration_mode`` is a plain ``str`` because the set of tokens is+ # open (plugins contribute their own); validation is the registry lookup at dispatch.+ orchestration_mode: str = DIRECT_ORCHESTRATION_MODE+ delivery: DeliveryMode = DeliveryMode.BLOCKING+ delivery_assignment_dump: dict[str, Any] | None = None+++class PipelexPipeRunOutput(BaseModel):+ """JSON-safe output crossing the host-runtime / Temporal boundary."""++ model_config = ConfigDict(extra="forbid")++ output_dict: dict[str, Any]+ main_stuff_name: str | None = None+ pipeline_run_id: str+ workflow_id: str | None = None+ is_completed: bool+ graph_spec_dump: dict[str, Any] | None = None+ # graph_assembly_error / usage_assembly_error mirror the same fields on PipeOutput: a non-None+ # value means assembly of the graph / token usage failed, which a host must be able to tell+ # apart from "assembly was off" (a None graph_spec_dump / tokens_usages_dump). tokens_usages_dump+ # is the JSON-safe dump of the AnyTokensUsage discriminated union so a host can render the+ # end-of-run cost report: None when cost reporting was off, [] when on but no inference happened.+ graph_assembly_error: str | None = None+ tokens_usages_dump: list[dict[str, Any]] | None = None+ usage_assembly_error: str | None = Nonediff --git a/pipelex/runtime_bridge/primitives/__init__.py b/pipelex/runtime_bridge/primitives/__init__.pyindex 31fa96856..fb95709fb 100644--- a/pipelex/runtime_bridge/primitives/__init__.py+++ b/pipelex/runtime_bridge/primitives/__init__.py@@ -1,6 +1,8 @@-"""Framework-agnostic primitives shared across host-runtime integrations.+"""Hydration helpers for the runtime bridge.-These helpers are used by both ``pipelex.temporal`` (raw Temporal cluster-mode) and ``pipelex_mistralai_workflows.primitives`` (Mistral-native mode).-They contain no host-runtime-specific imports — only Pipelex core types.+After the cross-process transport plumbing moved to the closed ``pipelex-transport``+library, this package holds only ``hydration.py`` — the working-memory hydration+helpers used by core's ``delivery_executor`` and the open ``pipelex-api`` runner, and+imported across the boundary by ``pipelex-transport`` (they are on the allowed import+surface). It contains no host-runtime-specific imports — only Pipelex core types."""diff --git a/pipelex/runtime_bridge/primitives/delivery.py b/pipelex/runtime_bridge/primitives/delivery.pydeleted file mode 100644index 46a834e30..000000000--- a/pipelex/runtime_bridge/primitives/delivery.py+++ /dev/null@@ -1,38 +0,0 @@-"""Framework-agnostic delivery execution (storage + webhook).--Wraps ``DeliveryExecutor.execute`` so host-runtime activities (Temporal /-Mistral) can keep their decorators thin while sharing the same body.-"""--from pipelex.base_exceptions import ErrorReport-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus-from pipelex.pipe_run.delivery_executor import DeliveryExecutor---async def execute_delivery(- *,- pipe_output: PipeOutput | None,- user_id: str,- pipeline_run_id: str,- delivery_assignment: DeliveryAssignment,- status: DeliveryStatus = DeliveryStatus.COMPLETED,- error_report: ErrorReport | None = None,- request_id: str | None = None,-) -> None:- """Execute the full delivery (storage + webhooks).-- ``error_report`` carries the worker-side classification on the failure path so the- webhook payload includes a structured ``error`` object. ``request_id`` is the- originating API request id, threaded into the delivery log lines for correlation.- """- executor = DeliveryExecutor()- await executor.execute(- pipe_output=pipe_output,- user_id=user_id,- pipeline_run_id=pipeline_run_id,- delivery_assignment=delivery_assignment,- status=status,- error_report=error_report,- request_id=request_id,- )diff --git a/pipelex/runtime_bridge/primitives/hydration.py b/pipelex/runtime_bridge/primitives/hydration.pyindex 7b94ff2fe..2b06e8804 100644--- a/pipelex/runtime_bridge/primitives/hydration.py+++ b/pipelex/runtime_bridge/primitives/hydration.py@@ -19,7 +19,7 @@ def _validate_as_known_class(*, item_class: type[StuffContent], raw_item: StuffCThe Temporal hot path now ships ListContent items as plain dicts withpipelex-private ``__pipelex_class__`` / ``__pipelex_module__`` markers- (see ``WorkingMemory.dump_for_temporal``), so kajson never eagerly rehydrates+ (see ``WorkingMemory.dump_for_transport``), so kajson never eagerly rehydratesthem — the ``isinstance(raw_item, StuffContent)`` branch below is defensive,kept for direct (non-Temporal) callers and for any future path where kajsonmight still produce an instance whose class identity drifted across execs.@@ -40,7 +40,7 @@ def _hydrate_list_item(raw_item: dict[str, Any] | str | StuffContent) -> StuffCo"""Hydrate a single list item for Anything[] results.Resolves the item class from the pipelex-private ``__pipelex_class__`` /- ``__pipelex_module__`` markers written by ``dump_for_temporal``. Falls back+ ``__pipelex_module__`` markers written by ``dump_for_transport``. Falls backto TextContent for plain text or dicts with a 'text' key (the common casefor Anything outputs without explicit type metadata)."""@@ -76,7 +76,7 @@ def hydrate_content(raw_content: list[Any] | dict[str, Any] | str, *, concept: C"""Hydrate a single StuffContent from a raw value.Handles both plain content and ListContent. The Temporal serialization- format (produced by ``WorkingMemory.dump_for_temporal()``) encodes+ format (produced by ``WorkingMemory.dump_for_transport()``) encodesListContent as a plain JSON list and single StuffContent as a dict,so the type check is unambiguous — no heuristic required. ListContentitems carry pipelex-private ``__pipelex_class__`` / ``__pipelex_module__``diff --git a/pipelex/runtime_bridge/primitives/pipe_classification.py b/pipelex/runtime_bridge/primitives/pipe_classification.pydeleted file mode 100644index b39880b35..000000000--- a/pipelex/runtime_bridge/primitives/pipe_classification.py+++ /dev/null@@ -1,35 +0,0 @@-"""Pipe classification helpers for per-step-decomposition host runtimes.--Framework-agnostic — answers the single question "should this pipe be-dispatched as a child workflow (controller) or as an activity (leaf)?".-This is bridge-side plumbing for the ``MISTRAL_NATIVE`` mode: the-``pipelex_mistralai_workflows`` host package is the intended consumer of-this controller/leaf split. There is no in-tree caller yet — the Temporal-path makes the same decision elsewhere (``temporal_pipe_router``) — so the-helpers live here, framework-agnostic, ready to be shared once the Mistral-host integration lands.-"""--from pipelex.core.pipes.pipe_abstract import PipeAbstract-from pipelex.pipe_controllers.pipe_controller import PipeController---def is_controller_pipe(pipe: PipeAbstract) -> bool:- """Return True if ``pipe`` is a controller (PipeSequence, PipeBatch, …).-- Controllers iterate over sub-pipes and are the right unit to dispatch as- a child workflow under per-step decomposition modes (e.g. MISTRAL_NATIVE).- Leaf / operator pipes (PipeLLM, PipeImg, PipePython, …) should run inside- activities, not workflows, because the controller boundary is exactly- where signals / timers / cancellation / per-step retry need to attach.- """- return isinstance(pipe, PipeController)---def is_leaf_pipe(pipe: PipeAbstract) -> bool:- """Return True if ``pipe`` is a leaf operator (PipeLLM, PipeImg, …).-- Convenience inverse of ``is_controller_pipe``. A leaf pipe maps to a- single I/O burst that fits inside one activity invocation.- """- return not is_controller_pipe(pipe)diff --git a/pipelex/runtime_bridge/primitives/pipe_run_arg.py b/pipelex/runtime_bridge/primitives/pipe_run_arg.pydeleted file mode 100644index e4519ca86..000000000--- a/pipelex/runtime_bridge/primitives/pipe_run_arg.py+++ /dev/null@@ -1,14 +0,0 @@-from pydantic import BaseModel--from pipelex.pipe_run.delivery_assignment import DeliveryAssignment-from pipelex.pipe_run.pipe_job import PipeJob---class PipeRunArg(BaseModel):- """Workflow input for WfPipeRun: bundles PipeJob + delivery assignment for Temporal serialization."""-- pipe_job: PipeJob- delivery_assignment: DeliveryAssignment | None = None-- def prepare_for_temporal(self) -> "PipeRunArg":- return self.model_copy(update={"pipe_job": self.pipe_job.prepare_for_temporal()})diff --git a/pipelex/runtime_bridge/primitives/scoped_library.py b/pipelex/runtime_bridge/primitives/scoped_library.pydeleted file mode 100644index b63578efe..000000000--- a/pipelex/runtime_bridge/primitives/scoped_library.py+++ /dev/null@@ -1,65 +0,0 @@-"""Per-call scoped library for crate-based execution/hydration outside Temporal workers."""--import uuid-from collections.abc import Generator-from contextlib import contextmanager--from kajson.class_registry import ClassRegistry-from kajson.kajson_manager import KajsonManager--from pipelex import log-from pipelex.hub import get_library_manager, scoped_current_library-from pipelex.libraries.library_crate import LibraryCrate---@contextmanager-def scoped_library_for_crate(library_crate: LibraryCrate | None, *, library_id_prefix: str) -> Generator[str | None, None, None]:- """Open a per-call scoped library loaded from ``library_crate`` for the duration of the block.-- When ``library_crate`` is None this is a no-op (yields None): callers fall back to the- library that was loaded into the active class registry at boot. Otherwise: opens a- fresh library under a uuid-suffixed id, attaches a scoped ``ClassRegistry`` pre-seeded- from the global one (so classes generated from the crate's inline structured concepts- register into the scoped registry — discarded on teardown — rather than leaking into- or colliding in the global Kajson registry), sets it as the current library, loads the- crate, and tears everything down on the way out.-- The per-call uuid keying makes leaked-predecessor collisions structurally impossible,- so plain ``open_library`` is correct here — ``open_fresh_library`` is only needed on- the Temporal worker path, whose library ids are deterministic per run. This helper is- deliberately NOT used by ``WfPipeRouter``: its teardown sequencing relative to the- flush await IS the eviction-safety fix (H2), and folding it into a context manager- would fight that ordering.- """- if library_crate is None:- yield None- return-- library_manager = get_library_manager()- library_id = f"{library_id_prefix}_{uuid.uuid4().hex}"- # open_library + set_class_registry live inside the try so a throw between opening- # the library and reaching the yield still tears it down (the entry is registered in- # the manager the moment open_library returns). scoped_current_library captures and- # restores the prior current-library ContextVar, so a call made from within an- # already-scoped library doesn't clobber the caller's context.- library_opened = False- try:- _opened_library_id, library = library_manager.open_library(library_id=library_id)- library_opened = True- global_registry = KajsonManager.get_class_registry()- scoped_registry = ClassRegistry()- scoped_registry.register_classes_dict(global_registry.get_classes_dict())- library.set_class_registry(scoped_registry)- with scoped_current_library(library_id=library_id):- library_manager.load_from_crate(library_id=library_id, crate=library_crate)- yield library_id- finally:- if library_opened:- try:- library_manager.teardown(library_id=library_id)- except Exception as teardown_exc: # noqa: BLE001- # Best-effort: teardown in the finally must neither mask the body's in-flight- # exception nor fail an otherwise-successful call. LibraryManager forgets the- # entry pop-first even when the library's own teardown raises, so no poisoned- # state remains.- log.warning(f"scoped_library_for_crate: failed to tear down scoped library '{library_id}': {teardown_exc}")diff --git a/pipelex/runtime_bridge/primitives/submitter_hydration.py b/pipelex/runtime_bridge/primitives/submitter_hydration.pydeleted file mode 100644index 83bf367c5..000000000--- a/pipelex/runtime_bridge/primitives/submitter_hydration.py+++ /dev/null@@ -1,53 +0,0 @@-"""Submitter-side rehydration of PipeOutput received from a Temporal workflow.--The workflow side (`WfPipeRouter`) dehydrates `PipeOutput.working_memory` into-`working_memory_raw` for transit. The submitter must rebuild the typed-`WorkingMemory` after the workflow returns.--When the submitter has loaded the bundle locally (via `PIPELEXPATH` or by-explicit setup), the dynamic concept classes live in the global class registry-and a plain `hydrate_working_memory(...)` call works. But when the submitter is-a remote API client that built the `PipeJob` from a serialized `LibraryCrate`-without loading it into the global registry, hydration would fail (or worse,-silently degrade) on dynamic concept fields.--`rehydrate_pipe_output_with_crate` mirrors what `WfPipeRouter` does on the-worker side: open a per-call scoped Library, pre-seed a scoped `ClassRegistry`-from the global, load the crate into it, hydrate inside the scope, then tear-down — leaving the submitter's global registry untouched.-"""--from typing import TYPE_CHECKING--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.runtime_bridge.primitives.hydration import hydrate_working_memory-from pipelex.runtime_bridge.primitives.scoped_library import scoped_library_for_crate--if TYPE_CHECKING:- from pipelex.libraries.library_crate import LibraryCrate---def rehydrate_pipe_output_with_crate(- pipe_output: PipeOutput,- *,- library_crate: "LibraryCrate | None",-) -> PipeOutput:- """Rehydrate `pipe_output.working_memory_raw` into a typed WorkingMemory.-- When `library_crate` is provided, opens a fresh per-call scoped Library,- loads the crate into it, hydrates inside the scope, and tears down — so- callers that did not load the bundle into their global registry still get- a correct typed WorkingMemory. When `library_crate` is None, falls back to- the active class registry (built-ins + whatever PIPELEXPATH loaded).-- Mutates `pipe_output` in place: sets `working_memory`, clears- `working_memory_raw`. Returns the same instance for call-chain ergonomics.- """- if pipe_output.working_memory_raw is None:- return pipe_output-- with scoped_library_for_crate(library_crate, library_id_prefix="rehydrate"):- pipe_output.working_memory = hydrate_working_memory(pipe_output.working_memory_raw)- pipe_output.working_memory_raw = None-- return pipe_outputdiff --git a/pipelex/runtime_bridge/primitives/trace_flush.py b/pipelex/runtime_bridge/primitives/trace_flush.pydeleted file mode 100644index d287b18c4..000000000--- a/pipelex/runtime_bridge/primitives/trace_flush.py+++ /dev/null@@ -1,32 +0,0 @@-"""Framework-agnostic flush of buffered trace events to the configured backend.--Called from a host-runtime activity (Temporal / Mistral) after a workflow's-``BufferingEventLog`` has drained, since synchronous I/O (boto3, file-writes) cannot run inside a workflow thread.-"""--from pipelex import log-from pipelex.config import get_config-from pipelex.tracing.event_log_factory import make_event_log-from pipelex.tracing.trace_events import TraceEvent---async def flush_trace_events_to_backend(events: list[TraceEvent]) -> None: # noqa: RUF029- """Persist buffered trace events to the configured backend.-- No-op when the events list is empty or tracing is disabled.- """- if not events:- return-- tracing_config = get_config().pipelex.tracing_config- if not tracing_config.is_enabled:- return-- event_log = make_event_log(tracing_config)- try:- for event in events:- event_log.emit(event)- log.debug(f"Flushed {len(events)} trace events to backend={tracing_config.backend}")- finally:- event_log.close()diff --git a/pipelex/runtime_bridge/serialization.py b/pipelex/runtime_bridge/serialization.pynew file mode 100644index 000000000..fe180d19c--- /dev/null+++ b/pipelex/runtime_bridge/serialization.py@@ -0,0 +1,94 @@+"""Boundary (de)serialization shared by the orchestrators.++Kept separate from the dispatch entry-point (``run_pipe_via_bridge``, now in the+closed ``pipelex-transport`` library) so the orchestrator implementations (DIRECT+in core, the distributed modes in the plugins) can serialize their ``PipeOutput``+into the JSON-safe ``PipelexPipeRunOutput`` without pulling the bootstrap path+(which would form an import cycle). This module is deliberately import-light: core+working-memory + the pipe-execution error types + ``payloads``; no bootstrap, no+plugin discovery, no host-runtime SDK.+"""++from __future__ import annotations++from typing import TYPE_CHECKING, Any++from pipelex.core.memory.working_memory import MAIN_STUFF_NAME+from pipelex.pipe_run.exceptions import PipeJobError, PipeRouterError, PipeRunError+from pipelex.pipeline.exceptions import PipeExecutionError, PipelineExecutionError+from pipelex.runtime_bridge.payloads import PipelexPipeRunOutput++if TYPE_CHECKING:+ from pipelex.base_exceptions import PipelexError+ from pipelex.core.pipes.pipe_output import PipeOutput+++# Pipe-execution failures an orchestrator converts into PipelexBridgeDispatchError so a host can catch a+# single error type regardless of execution mode. The Temporal orchestrators additionally catch+# WorkflowExecutionError (lazy-imported there to keep temporal off the module import path); wrapping+# it with ``from exc`` loses no signal — the structured ErrorReport stays reachable via ``__cause__``+# and is surfaced by PipelexBridgeDispatchError.to_error_report()'s cause-chain enrichment.+PIPE_DISPATCH_ERRORS: tuple[type[PipelexError], ...] = (+ PipeRunError,+ PipeJobError,+ PipeRouterError,+ PipeExecutionError,+ PipelineExecutionError,+)+++def serialize_pipe_output(pipe_output: PipeOutput) -> dict[str, Any]:+ """Dehydrate a PipeOutput's working memory to a JSON-safe dict.++ Always uses ``WorkingMemory.dump_for_transport()`` — the same format Pipelex+ uses internally for cross-process transit. The shape is stable regardless of+ whether a ``library_crate`` was attached:+ ``{"root": {stuff_name: {"content": {...}, ...}}, "aliases": {...}}``.++ Type metadata embedded by ``dump_for_transport`` lets callers reconstruct a+ typed ``WorkingMemory`` when they have the matching class registry in+ scope (e.g. via ``hydrate_working_memory``).+ """+ return pipe_output.working_memory.dump_for_transport()+++def serialize_completed_output(+ pipe_output: PipeOutput,+ *,+ workflow_id: str | None,+) -> PipelexPipeRunOutput:+ output_dict = serialize_pipe_output(pipe_output=pipe_output)++ main_stuff_name = resolve_main_stuff_root_key(pipe_output=pipe_output)++ graph_spec_dump = pipe_output.graph_spec.model_dump(mode="json") if pipe_output.graph_spec is not None else None+ tokens_usages_dump = [usage.model_dump(mode="json") for usage in pipe_output.tokens_usages] if pipe_output.tokens_usages is not None else None++ return PipelexPipeRunOutput(+ output_dict=output_dict,+ main_stuff_name=main_stuff_name,+ pipeline_run_id=pipe_output.pipeline_run_id,+ workflow_id=workflow_id,+ is_completed=True,+ graph_spec_dump=graph_spec_dump,+ graph_assembly_error=pipe_output.graph_assembly_error,+ tokens_usages_dump=tokens_usages_dump,+ usage_assembly_error=pipe_output.usage_assembly_error,+ )+++def resolve_main_stuff_root_key(pipe_output: PipeOutput) -> str | None:+ """Return the actual ``root`` dict key under which the main stuff lives.++ The main stuff can either sit directly at ``root[MAIN_STUFF_NAME]`` or be+ referenced via ``aliases[MAIN_STUFF_NAME]`` pointing at its real name.+ Callers indexing the output_dict need the actual root key, not the+ stuff's display ``stuff_name``.+ """+ working_memory = pipe_output.working_memory+ if MAIN_STUFF_NAME in working_memory.root:+ return MAIN_STUFF_NAME+ aliased_target = working_memory.aliases.get(MAIN_STUFF_NAME)+ if aliased_target is not None and aliased_target in working_memory.root:+ return aliased_target+ return Nonediff --git a/pipelex/system/configuration/config_loader.py b/pipelex/system/configuration/config_loader.pyindex 4f3624a36..af3ba71ed 100644--- a/pipelex/system/configuration/config_loader.py+++ b/pipelex/system/configuration/config_loader.py@@ -1,11 +1,15 @@import shutilfrom pathlib import Path-from typing import Any+from typing import Any, TypeVar++from pydantic import BaseModelfrom pipelex.system.runtime import runtime_managerfrom pipelex.tools.misc.json_utils import deep_updatefrom pipelex.tools.misc.toml_utils import load_toml_from_path_and_merge_with_overrides+_PluginConfigT = TypeVar("_PluginConfigT", bound=BaseModel)+CONFIG_DIR_NAME = ".pipelex"CONFIG_NAME = "pipelex.toml"@@ -192,6 +196,78 @@ class ConfigLoader:files.append(config_dir / "pipelex_temporary_override.toml")return files+ @classmethod+ def _plugin_override_files_for_dir(cls, config_dir: Path, *, name: str) -> list[Path]:+ """Build the plugin-config override sequence for one ``.pipelex`` dir.++ Order matters: later files win on key collisions during deep-merge. Mirrors+ ``_override_files_for_dir`` but keyed on a plugin's config ``name`` rather+ than ``"pipelex"``, and intentionally narrower — a plugin config carries no+ local / run_mode / temporary tiers (it is env-selected and deployment-baked,+ not developer-scratch-layered).++ Returns:+ Ordered list of candidate file paths (missing files are ignored at load time).+ """+ return [+ config_dir / f"{name}_{runtime_manager.environment}.toml",+ config_dir / f"{name}_override.toml",+ ]++ def load_plugin_config(+ self,+ *,+ name: str,+ package_dir: Path,+ schema: type[_PluginConfigT],+ extra_overrides: dict[str, Any] | None = None,+ ) -> _PluginConfigT:+ """Load, deep-merge and validate a plugin's config with env layering.++ Mirrors ``load_config``'s env-keyed layering for an arbitrary plugin whose+ config base name is ``name`` (e.g. ``"temporal"``). Every discovered plugin+ self-loads its config through this one helper so they all inherit identical+ env semantics: one image bakes every env file and ``PIPELEX_ENV``+ (``runtime_manager.environment``) selects which ``{name}_{env}.toml`` wins+ at runtime.++ Layers, deep-merged in order (later wins per leaf key):++ 1. Packaged default: ``{package_dir}/{name}.toml`` — the plugin's bundled+ default, shipped inside its own distribution.+ 2. Global override sequence from ``~/.pipelex/``:+ ``{name}_{environment}.toml`` then ``{name}_override.toml``.+ 3. Project override sequence from ``{project_root}/.pipelex/`` (same two+ files), when a project dir is found and distinct from the global dir.+ 4. Programmatic ``extra_overrides``, if any.++ Missing files at any tier are skipped, so the packaged default alone is a+ valid, fully-resolved config. Unlike ``load_config`` this never creates the+ global config dir or copies kit templates — a plugin config is purely+ additive layering over its own packaged default.++ Args:+ name: The plugin config base name, used for both the packaged default+ filename (``{name}.toml``) and the ``.pipelex`` override filenames.+ package_dir: Directory holding the plugin's packaged ``{name}.toml``+ (typically ``Path(__file__).parent`` in the plugin).+ schema: The pydantic model the merged config is validated into.+ extra_overrides: Optional dict deep-merged on top as the final layer.++ Returns:+ The merged config validated into ``schema``.+ """+ list_of_configs: list[Path] = [package_dir / f"{name}.toml"]+ list_of_configs.extend(self._plugin_override_files_for_dir(self.global_config_dir, name=name))+ project_dir = self.project_config_dir+ if project_dir is not None and project_dir != self.global_config_dir:+ list_of_configs.extend(self._plugin_override_files_for_dir(project_dir, name=name))++ merged = load_toml_from_path_and_merge_with_overrides(paths=list_of_configs)+ if extra_overrides:+ deep_update(merged, updates=extra_overrides)+ return schema.model_validate(merged)+def load_config(self, *, extra_overrides: dict[str, Any] | None = None, config_dir: Path | None = None) -> dict[str, Any]:"""Load and merge configurations from pipelex and local config files.diff --git a/pipelex/system/configuration/configs.py b/pipelex/system/configuration/configs.pyindex ec250eb5f..419b5198b 100644--- a/pipelex/system/configuration/configs.py+++ b/pipelex/system/configuration/configs.py@@ -11,7 +11,6 @@ from pipelex.graph.graph_config import GraphConfigfrom pipelex.language.mthds_config import MthdsConfigfrom pipelex.system.configuration.config_model import ConfigModelfrom pipelex.system.configuration.config_root import ConfigRoot-from pipelex.temporal.config_temporal import Temporalfrom pipelex.tools.aws.aws_config import AwsConfigfrom pipelex.tools.log.log_config import LogConfigfrom pipelex.tools.storage.storage_config import StorageConfig@@ -244,9 +243,24 @@ class MigrationConfig(ConfigModel):return [(key, value) for key, value in renaming_map.items() if text in value]+class PluginsConfig(ConfigModel):+ # Names of discovered plugins to skip at startup (a denylist; discovery is the+ # source of truth for presence). Denylisting a plugin core requires+ # unconditionally is a startup error. There is intentionally no allowlist.+ disabled: list[str]++ # Boot *this process* under the orchestrator plugin of this name. A boot-orchestrator+ # plugin (e.g. the Temporal plugin) claims the process-global hub slots in its+ # ``register`` iff ``plugins.boot_orchestrator == <its own name>``; any other value+ # (or ``None``) leaves execution in-process. Core names no orchestrator — the gate is+ # a plain name match, set programmatically (CLI ``--orchestrator`` / ``Pipelex.setup``),+ # not from ``pipelex.toml``. Optional, so it stays absent from the TOML defaults.+ boot_orchestrator: str | None = None++class PipelexConfig(ConfigRoot):session_id: str = shortuuid.uuid()cogt: Cogt- temporal: Temporalpipelex: Pipelex+ plugins: PluginsConfigmigration: MigrationConfigdiff --git a/pipelex/system/exceptions.py b/pipelex/system/exceptions.pyindex b61e54503..694ef6eb1 100644--- a/pipelex/system/exceptions.py+++ b/pipelex/system/exceptions.py@@ -66,5 +66,5 @@ class ConfigValidationError(FatalError):pass-class ConfigModelError(ValueError, FatalError):+class ConfigModelError(FatalError, ValueError):passdiff --git a/pipelex/temporal/codec/__init__.py b/pipelex/temporal/codec/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/pipelex/temporal/codec/codec_factory.py b/pipelex/temporal/codec/codec_factory.pydeleted file mode 100644index b9f3d94c6..000000000--- a/pipelex/temporal/codec/codec_factory.py+++ /dev/null@@ -1,23 +0,0 @@-"""Factory for building a :class:`StoragePayloadCodec` from the Pipelex config."""--from __future__ import annotations--from pipelex.config import get_config-from pipelex.temporal.codec.storage_payload_codec import StoragePayloadCodec-from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_config---def make_codec_from_config() -> StoragePayloadCodec:- """Build a :class:`StoragePayloadCodec` from the current Pipelex config.-- Returns:- A fully configured codec backed by the storage provider specified in- ``temporal.payload_codec_config``.- """- payload_codec_config = get_config().temporal.payload_codec_config- storage_provider = make_storage_provider_from_config(payload_codec_config.storage_provider_config)- return StoragePayloadCodec(- storage_provider=storage_provider,- size_threshold=payload_codec_config.size_threshold,- storage_prefix=payload_codec_config.storage_prefix,- )diff --git a/pipelex/temporal/codec/codec_server.py b/pipelex/temporal/codec/codec_server.pydeleted file mode 100644index d2e51700f..000000000--- a/pipelex/temporal/codec/codec_server.py+++ /dev/null@@ -1,110 +0,0 @@-"""HTTP codec server for the Temporal Web UI.--Exposes ``POST /encode`` and ``POST /decode`` endpoints that the Temporal Web UI-can call to transparently decode payloads offloaded by :class:`StoragePayloadCodec`.-Without this server, the UI shows opaque ``binary/storage-ref`` blobs instead of-the actual workflow data.--Usage::-- python -m pipelex.temporal.codec.codec_server_cli-"""--from __future__ import annotations--from functools import partial-from typing import TYPE_CHECKING, Awaitable, Callable--from aiohttp import hdrs, web-from google.protobuf import json_format-from temporalio.api.common.v1 import Payload, Payloads--from pipelex import log-from pipelex.tools.storage.exceptions import StorageFileNotFoundError--if TYPE_CHECKING:- from collections.abc import Sequence-- from pipelex.temporal.codec.storage_payload_codec import StoragePayloadCodec---def _set_cors_headers(response: web.Response, *, request_origin: str, cors_origins: list[str]) -> None:- """Set CORS headers on the response if the request origin is allowed."""- if request_origin in cors_origins:- response.headers[hdrs.ACCESS_CONTROL_ALLOW_ORIGIN] = request_origin- response.headers[hdrs.ACCESS_CONTROL_ALLOW_METHODS] = "POST"- response.headers[hdrs.ACCESS_CONTROL_ALLOW_HEADERS] = "content-type,x-namespace"---def _make_cors_handler(cors_origins: list[str]) -> Callable[[web.Request], Awaitable[web.Response]]:- """Create a CORS preflight handler for the given origins."""-- async def cors_options(request: web.Request) -> web.Response: # noqa: RUF029 — must be async for aiohttp- response = web.Response()- _set_cors_headers(response, request_origin=request.headers.get(hdrs.ORIGIN, ""), cors_origins=cors_origins)- return response-- return cors_options---def _error_response(request: web.Request, *, cors_origins: list[str], status: int, text: str) -> web.Response:- """Build an error response with CORS headers so browsers see the diagnostic message."""- response = web.Response(status=status, text=text)- _set_cors_headers(response, request_origin=request.headers.get(hdrs.ORIGIN, ""), cors_origins=cors_origins)- return response---async def _apply( # kw-only: ignore — aiohttp route handler; aiohttp passes `request` positionally via partial(_apply, codec_fn, cors_origins)- codec_fn: Callable[[Sequence[Payload]], Awaitable[list[Payload]]],- cors_origins: list[str],- request: web.Request,-) -> web.Response:- """Generic handler: parse Payloads JSON, apply codec function, return Payloads JSON."""- if request.content_type != "application/json":- return _error_response(request, cors_origins=cors_origins, status=415, text="Expected application/json")-- try:- payloads_proto = json_format.Parse(await request.read(), Payloads())- except json_format.ParseError:- return _error_response(request, cors_origins=cors_origins, status=400, text="Malformed Payloads JSON")-- try:- result_payloads = await codec_fn(payloads_proto.payloads)- except StorageFileNotFoundError as exc:- log.error(f"Codec storage lookup failed: {exc}")- return _error_response(request, cors_origins=cors_origins, status=404, text="Storage object not found")- except (OSError, FileNotFoundError) as exc:- log.error(f"Codec I/O error: {exc}")- return _error_response(request, cors_origins=cors_origins, status=502, text="Storage I/O error")-- result_proto = Payloads(payloads=result_payloads)-- response = web.Response()- _set_cors_headers(response, request_origin=request.headers.get(hdrs.ORIGIN, ""), cors_origins=cors_origins)- response.content_type = "application/json"- response.text = json_format.MessageToJson(result_proto)- return response---def build_codec_server(codec: StoragePayloadCodec, *, cors_origins: list[str]) -> web.Application:- """Build an aiohttp application implementing the Temporal codec server protocol.-- Args:- codec: The payload codec to use for encode/decode operations.- cors_origins: Allowed CORS origins (e.g. ``["http://localhost:8233"]``).-- Returns:- A configured aiohttp web application ready to run.- """- cors_handler = _make_cors_handler(cors_origins)- application = web.Application()- application.add_routes(- [- web.post("/encode", partial(_apply, codec.encode, cors_origins)),- web.post("/decode", partial(_apply, codec.decode, cors_origins)),- web.options("/encode", cors_handler),- web.options("/decode", cors_handler),- ]- )- log.info(f"Codec server configured — CORS origins: {cors_origins}")- return applicationdiff --git a/pipelex/temporal/codec/codec_server_cli.py b/pipelex/temporal/codec/codec_server_cli.pydeleted file mode 100644index 415f85553..000000000--- a/pipelex/temporal/codec/codec_server_cli.py+++ /dev/null@@ -1,55 +0,0 @@-"""CLI entry point for the Temporal codec server.--Usage::-- python -m pipelex.temporal.codec.codec_server_cli- python -m pipelex.temporal.codec.codec_server_cli --port 8082- python -m pipelex.temporal.codec.codec_server_cli --cors-origin http://localhost:8233 --cors-origin https://cloud.temporal.io-"""--from __future__ import annotations--from typing import Annotated--import typer-from aiohttp import web--from pipelex import log-from pipelex.config import get_config-from pipelex.pipelex import Pipelex-from pipelex.temporal.codec.codec_factory import make_codec_from_config-from pipelex.temporal.codec.codec_server import build_codec_server-from pipelex.temporal.exceptions import TemporalConfigError--app = typer.Typer()--DEFAULT_CORS_ORIGINS = ["http://localhost:8233"]---def start(- host: Annotated[str, typer.Option(help="Host to bind the codec server to")] = "127.0.0.1",- port: Annotated[int, typer.Option(help="Port to bind the codec server to")] = 8081,- cors_origin: Annotated[list[str] | None, typer.Option(help="Allowed CORS origin (repeatable)")] = None,-) -> None:- """Start the Temporal codec server for Web UI payload decoding."""- Pipelex.make(temporal_enabled=True, needs_inference=False)-- payload_codec_config = get_config().temporal.payload_codec_config- if not payload_codec_config.is_enabled:- msg = (- "Payload codec is not enabled in the config (temporal.payload_codec_config.is_enabled = false). "- "Enable it in your pipelex.toml before starting the codec server."- )- raise TemporalConfigError(msg)-- codec = make_codec_from_config()- cors_origins = cors_origin or DEFAULT_CORS_ORIGINS- application = build_codec_server(codec=codec, cors_origins=cors_origins)-- log.info(f"Starting Temporal codec server on {host}:{port}")- web.run_app(application, host=host, port=port)---if __name__ == "__main__":- app()diff --git a/pipelex/temporal/codec/storage_payload_codec.py b/pipelex/temporal/codec/storage_payload_codec.pydeleted file mode 100644index 48e4be421..000000000--- a/pipelex/temporal/codec/storage_payload_codec.py+++ /dev/null@@ -1,123 +0,0 @@-from __future__ import annotations--import json-import re-from hashlib import sha256-from typing import TYPE_CHECKING, Any, cast--from temporalio.api.common.v1 import Payload-from temporalio.converter import PayloadCodec-from typing_extensions import override--from pipelex import log-from pipelex.tools.storage.exceptions import StorageInvalidUriError-from pipelex.tools.storage.storage_provider_abstract import PIPELEX_STORAGE_SCHEME--if TYPE_CHECKING:- from collections.abc import Sequence-- from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract--STORAGE_REF_ENCODING = b"binary/storage-ref"-JSON_PLAIN_ENCODING = b"json/plain"---class StoragePayloadCodec(PayloadCodec):- """Codec that offloads large Temporal payloads to external storage.-- Payloads whose serialized size meets or exceeds ``size_threshold`` are- stored via the provided :class:`StorageProviderAbstract` and replaced with- a lightweight reference payload. Payloads below the threshold pass through- unchanged. Content-addressed keys (SHA-256) give natural deduplication.-- Storage keys are structured as ``{user_id}/{pipeline_run_id}/{hash}`` when- the payload contains ``job_metadata``, enabling per-client cleanup.- """-- def __init__(- self,- storage_provider: StorageProviderAbstract,- size_threshold: int,- storage_prefix: str,- ) -> None:- self._storage = storage_provider- self._size_threshold = size_threshold- self._storage_prefix = f"{storage_prefix}/" if storage_prefix and not storage_prefix.endswith("/") else storage_prefix- self._expected_uri_prefix = f"{PIPELEX_STORAGE_SCHEME}{self._storage_prefix}"-- @staticmethod- def _extract_job_routing(payload: Payload) -> tuple[str, str] | None:- """Extract user_id and pipeline_run_id from a JSON payload's job_metadata.-- Returns:- A (user_id, pipeline_run_id) tuple, or None if the payload is not- JSON or does not contain job_metadata at the top level.- """- if payload.metadata.get("encoding") != JSON_PLAIN_ENCODING:- return None- try:- data = json.loads(payload.data)- except (json.JSONDecodeError, UnicodeDecodeError):- return None- if not isinstance(data, dict):- return None- typed_data = cast("dict[str, Any]", data)- job_metadata = typed_data.get("job_metadata")- if not isinstance(job_metadata, dict):- return None- user_id: str | None = job_metadata.get("user_id") # type: ignore[assignment]- pipeline_run_id: str | None = job_metadata.get("pipeline_run_id") # type: ignore[assignment]- if isinstance(user_id, str) and isinstance(pipeline_run_id, str):- return user_id, pipeline_run_id- return None-- @staticmethod- def _sanitize_path_segment(segment: str) -> str:- """Sanitize a path segment to prevent traversal and encoding issues."""- return re.sub(r"[^A-Za-z0-9_\-]", "_", segment)-- def _build_storage_key(self, payload: Payload, *, hash_hex: str) -> str:- """Build a storage key, structured by job routing when available."""- routing = self._extract_job_routing(payload)- if routing:- user_id, pipeline_run_id = routing- safe_user_id = self._sanitize_path_segment(user_id)- safe_run_id = self._sanitize_path_segment(pipeline_run_id)- return f"{self._storage_prefix}{safe_user_id}/{safe_run_id}/{hash_hex}"- return f"{self._storage_prefix}{hash_hex}"-- @override- async def encode(self, payloads: Sequence[Payload]) -> list[Payload]:- encoded: list[Payload] = []- for payload in payloads:- serialized = payload.SerializeToString()- if len(serialized) < self._size_threshold:- encoded.append(payload)- continue- hash_hex = sha256(serialized).hexdigest()- key = self._build_storage_key(payload, hash_hex=hash_hex)- uri = await self._storage.store(data=serialized, key=key)- log.dev(f"Payload offloaded to storage: key='{key}'")- ref_payload = Payload(- metadata={"encoding": STORAGE_REF_ENCODING},- data=uri.encode(),- )- encoded.append(ref_payload)- return encoded-- @override- async def decode(self, payloads: Sequence[Payload]) -> list[Payload]:- decoded: list[Payload] = []- for payload in payloads:- if payload.metadata.get("encoding") != STORAGE_REF_ENCODING:- decoded.append(payload)- continue- uri = payload.data.decode()- if not uri.startswith(self._expected_uri_prefix):- msg = f"Refusing to load storage reference '{uri}': URI does not start with expected prefix '{self._expected_uri_prefix}'"- raise StorageInvalidUriError(msg)- original_bytes = await self._storage.load(uri)- original_payload = Payload()- original_payload.ParseFromString(original_bytes)- decoded.append(original_payload)- return decodeddiff --git a/pipelex/temporal/config_temporal.py b/pipelex/temporal/config_temporal.pydeleted file mode 100644index f17424302..000000000--- a/pipelex/temporal/config_temporal.py+++ /dev/null@@ -1,759 +0,0 @@-import difflib-from datetime import timedelta-from typing import TYPE_CHECKING, Annotated, Any, Literal, Union--from pydantic import Field, model_validator-from pydantic.dataclasses import dataclass--from pipelex import log-from pipelex.system.configuration.config_model import ConfigModel-from pipelex.temporal.exceptions import TemporalConfigError, WorkerTaskQueueUnknownError-from pipelex.tools.storage.storage_config import StorageProviderConfig-from pipelex.types import Self, StrEnum--if TYPE_CHECKING:- from temporalio.common import RetryPolicy-else:- # Runtime placeholder so Pydantic resolves the ``DispatchOptions.retry_policy``- # field annotation to ``Any`` (no validation, no isinstance) WITHOUT importing- # ``temporalio`` — keeping this module importable on installs that skipped the- # ``temporal`` extra. Type checkers see the real ``RetryPolicy`` above. With a- # bare forward ref instead, the model imports (deferred schema) but raises- # ``PydanticUserError`` the moment a ``DispatchOptions`` is constructed.- RetryPolicy = Any---# Names of the five custom search attributes Pipelex knows how to populate.-# Listed here (rather than in ``pipelex.temporal.tprl.namespace_check``) so the-# ``SearchAttributesConfig`` validator can reference them without pulling-# ``temporalio`` into the config-load path. Pipelex only knows how to populate-# these five; arbitrary custom names are out of scope (they would require code-# that knows the value source).-BUILTIN_SEARCH_ATTRIBUTES: tuple[str, ...] = (- "PipeCode",- "PipelineRunId",- "SessionId",- "UserId",- "DomainCode",-)---class SearchAttributesConfig(ConfigModel):- """Configuration for the five custom Temporal search attributes Pipelex populates.-- ``enabled`` is a master toggle:-- - ``true`` → workflow starts attach the subset listed in ``attributes`` and- the worker-boot check requires those same attributes to be registered on- the namespace.- - ``false`` → no custom search attributes are attached, the worker-boot- check is skipped, and the dashboard view degrades to- WorkflowType / WorkflowId / StartTime only.-- ``attributes`` is the opt-in/opt-out subset of the five built-ins. Names not- in this list are neither attached at workflow start nor required at worker- boot. Custom attribute names that are not built-ins are rejected by the- validator — Pipelex only knows how to populate the five built-ins.- """-- enabled: bool- attributes: list[str]-- @model_validator(mode="after")- def validate_attribute_names(self) -> Self:- unknown = [name for name in self.attributes if name not in BUILTIN_SEARCH_ATTRIBUTES]- if unknown:- msg = (- f"[temporal.search_attributes] attributes contains unknown name(s) {unknown}. "- f"Pipelex only knows how to populate the five built-in attributes: "- f"{list(BUILTIN_SEARCH_ATTRIBUTES)}."- )- raise TemporalConfigError(msg)- return self---class SecretMethod(StrEnum):- NONE = "none"- ENV_VAR = "env_var"- SECRET_PROVIDER = "secret_provider"---class TemporalServerConfig(ConfigModel):- """Configuration model for Temporal server settings."""-- description: str- target_host: str- namespace: str- api_key_method: SecretMethod = Field(strict=False)- api_key_id: str-- @property- def full_description(self) -> str:- """Generate a full description of the Temporal server configuration.-- Returns:- str: A detailed description including non-default target host and namespace.- """- desc = self.description- if self.target_host != "localhost:7233":- desc += f", target host: {self.target_host}"- if self.namespace != "default":- desc += f", namespace: {self.namespace}"- return desc---class TemporalLogConfig(ConfigModel):- """Configuration model for Temporal logging settings."""-- is_workflow_info_on_message: bool- is_workflow_info_on_extra: bool- is_full_workflow_info_on_extra: bool- is_activity_info_on_message: bool- is_activity_info_on_extra: bool- is_full_activity_info_on_extra: bool- is_formatter_enabled: bool- is_prefix_enabled: bool- managed_loggers: list[str]- is_dispatch_resolution_traced: bool---class WorkerScope(ConfigModel):- """Defines the subset of workflows and activities a worker registers.-- Resolution order: pack contents → required_* additions → excluded_* subtractions →- disable_all_* clearing. Names match TaskPack keys, workflow class __name__, and- activity function __name__.-- `disable_all_workflows` / `disable_all_activities` are useful to split workflow-only- and activity-only workers on the same task queue (Temporal SDK rejects multiple- workers with overlapping task types on the same queue).- """-- required_tasks_packs: list[str]- required_workflows: list[str]- required_activities: list[str]- excluded_workflows: list[str]- excluded_activities: list[str]- disable_all_workflows: bool- disable_all_activities: bool---class WorkerScopesConfig(ConfigModel):- """Named worker scopes selectable via --scope on the worker CLI."""-- default_scope: str- scopes: dict[str, WorkerScope]-- @model_validator(mode="after")- def validate_default_scope(self) -> Self:- if self.default_scope not in self.scopes:- msg = f"default_scope '{self.default_scope}' not found in scopes (known: {sorted(self.scopes.keys())})"- raise TemporalConfigError(msg)- return self---class TemporalConfig(ConfigModel):- """Configuration model for overall Temporal settings."""-- temporal_server_configs: dict[str, TemporalServerConfig]- selected_server: str- temporal_log_config: TemporalLogConfig-- @model_validator(mode="after")- def validate_selected_server(self) -> Self:- """Validate that the selected server exists in the server configurations.-- Raises:- TemporalConfigError: If the selected server is not found in the configurations.-- Returns:- Self: The validated instance.- """- if self.selected_server not in self.temporal_server_configs:- msg = f"Selected server '{self.selected_server}' not found in temporal_server_configs"- raise TemporalConfigError(msg)- return self---class RetryPolicyConfigBase(ConfigModel):- """Shared retry policy scalars. Layer-specific subclasses add the appropriate- non-retryable-types field: ``RetryPolicyConfig`` (baseline) owns the main- list, ``RetryPolicyConfigOverlay`` (queue / handle) owns the additive- ``_extra`` list. ``ConfigModel`` is ``extra="forbid"``, so the layer- asymmetry is enforced by Pydantic at config load — no runtime validator- needed.- """-- initial_interval: timedelta = Field(strict=False, gt=timedelta(0))- backoff_coefficient: float = Field(ge=1.0)- maximum_interval: Union[Annotated[timedelta, Field(strict=False, gt=timedelta(0))], Literal["unlimited"]]- maximum_attempts: Union[Annotated[int, Field(gt=0)], Literal["unlimited"]]-- def make_retry_policy(self, merged_non_retryable_types: list[str]) -> "RetryPolicy":- """Create a RetryPolicy instance from these scalars + the merged- non-retryable list assembled by the dispatch resolver across layers.- """- from temporalio.common import RetryPolicy as _RetryPolicy # noqa: PLC0415-- maximum_attempts: int- if self.maximum_attempts == "unlimited":- # This is according to the Temporal SDK's documentation- maximum_attempts = 0- else:- maximum_attempts = self.maximum_attempts-- maximum_interval: timedelta | None- # this test is in two steps because timedelta's are actually read from the config as strings- if isinstance(self.maximum_interval, str) and self.maximum_interval == "unlimited":- maximum_interval = None- else:- maximum_interval = self.maximum_interval-- return _RetryPolicy(- initial_interval=self.initial_interval,- backoff_coefficient=self.backoff_coefficient,- maximum_interval=maximum_interval,- maximum_attempts=maximum_attempts,- non_retryable_error_types=merged_non_retryable_types,- )---class RetryPolicyConfig(RetryPolicyConfigBase):- """Baseline retry policy on ``worker_config.retry_policy_config``. Owns the- main non-retryable list — the dispatch resolver seeds the merged set from- here and extends with overlay ``_extra`` lists.- """-- non_retryable_error_types: list[str] = Field(default_factory=list)- """Class-name fallback list for the retry decision.-- Retryability for a ``CogtError`` carrying an ``InferenceErrorCategory`` is- decided by ``category.is_retryable`` (see ``TemporalError.from_message_exception``)- — *not* by this list. The class-name list applies to:-- - Non-``CogtError`` ``PipelexError`` subclasses, which carry no category.- - Any ``CogtError`` raised without a category set.- - Special cases that must override the category default (e.g. forcing a- type non-retryable on a specific queue via ``non_retryable_error_types_extra``).- """---class RetryPolicyConfigOverlay(RetryPolicyConfigBase):- """Per-queue / per-handle overlay retry policy. Contributes additively via- ``non_retryable_error_types_extra``; the baseline main list always rides- through. The baseline class's ``non_retryable_error_types`` field is- deliberately absent here — setting it on an overlay would silently bypass- the additive composition rule. ``ConfigModel``'s ``extra="forbid"`` raises- ``ValidationError`` if a user supplies the disallowed field.- """-- non_retryable_error_types_extra: list[str] = Field(default_factory=list)- """Additive class-name overrides layered onto the baseline non-retryable list.-- Same fallback-and-override semantics as ``RetryPolicyConfig.non_retryable_error_types``:- category-carrying ``CogtError`` retryability is decided by category, not by- name. Use this list to force a type non-retryable for a specific queue or- handle, or to cover category-less exceptions on that route.- """---class QueueOptions(ConfigModel):- """Per-queue submitter options + queue-level rate limit.-- Resolution order at dispatch (for timeouts and retry):- per-handle override (handle_options.<handle>) →- this (queue_options[resolved_queue]) →- worker_config defaults.-- Note: ``heartbeat_timeout`` lives here (queue scope) because heartbeat cadence- is a property of the backend on the other side of the queue. It is NOT on- ``HandleOptions``. If a single model on a backend ever needs a different- cadence, add the field to ``HandleOptions`` then — schema change is one line.- """-- start_to_close_timeout: timedelta | None = Field(default=None, strict=False, gt=timedelta(0))- schedule_to_close_timeout: timedelta | None = Field(default=None, strict=False, gt=timedelta(0))- schedule_to_start_timeout: timedelta | None = Field(default=None, strict=False, gt=timedelta(0))- heartbeat_timeout: timedelta | None = Field(default=None, strict=False, gt=timedelta(0))- retry_policy_config: RetryPolicyConfigOverlay | None = None- # Cluster-wide queue rate limit, conveyed to the Temporal server by every- # worker on this queue. The latest value to be sent by a worker wins.- max_task_queue_activities_per_second: float | None = Field(default=None, gt=0)---class HandleOptions(ConfigModel):- """Per-handle option overrides. Layers on top of QueueOptions for the resolved queue.-- Deliberately narrow: only timeout and retry. Heartbeat is queue-level.- Other fields will be added on demand when a real case surfaces.- """-- start_to_close_timeout: timedelta | None = Field(default=None, strict=False, gt=timedelta(0))- retry_policy_config: RetryPolicyConfigOverlay | None = None---class ActivityRouteConfig(ConfigModel):- """Per-activity routing entry.-- Args:- default: The task queue used when no per-handle override matches.- by_handle: Mapping from runtime handle (e.g. ``llm_handle``,- ``img_gen_handle``, ``extract_handle``) to a dedicated task queue.- handle_options: Rare per-handle option overrides for a single handle- that needs different timeout/retry from its queue baseline (e.g.- a long-context model variant). Layered on top of the resolved- ``QueueOptions`` at dispatch time.- """-- default: str- by_handle: dict[str, str] = Field(default_factory=dict)- handle_options: dict[str, HandleOptions] = Field(default_factory=dict)---class WorkerTuningMode(StrEnum):- """How a worker scales its slot counts.-- Only EXPLICIT is implemented in v2. RESOURCE_BASED is reserved for a future- iteration (Temporal SDK's ``WorkerTuner.create_resource_based``). Defined now- so the profile schema doesn't break when we add it.- """-- EXPLICIT = "explicit"- RESOURCE_BASED = "resource_based"---class WorkerRuntimeProfile(ConfigModel):- """Bundle of ``Worker(...)`` constructor tuning. One worker process selects one profile.-- ``tuning_mode`` MUST be ``"explicit"`` in v2. A model_validator rejects- ``"resource_based"`` with a clear "not implemented yet" message until the- follow-up iteration ships the resource-based path.- """-- tuning_mode: WorkerTuningMode = Field(strict=False)- # ge=0 on the workflow-cache-and-activity-slot knobs lets operators stand- # up workflow-only workers (max_concurrent_activities=0) for the router- # profile pattern. The poll/workflow knobs need gt=0 — a worker with zero- # workflow-task polls can't make progress.- max_cached_workflows: int = Field(ge=0)- max_concurrent_workflow_tasks: int = Field(gt=0)- max_concurrent_activities: int = Field(ge=0)- max_concurrent_local_activities: int = Field(ge=0)- max_concurrent_workflow_task_polls: int = Field(gt=0)- max_concurrent_activity_task_polls: int = Field(ge=0)- max_activities_per_second: float = Field(gt=0)- sticky_queue_schedule_to_start_timeout: timedelta = Field(strict=False, gt=timedelta(0))- max_heartbeat_throttle_interval: timedelta = Field(strict=False, gt=timedelta(0))- default_heartbeat_throttle_interval: timedelta = Field(strict=False, gt=timedelta(0))- graceful_shutdown_timeout: timedelta = Field(strict=False, gt=timedelta(0))-- @model_validator(mode="after")- def reject_unimplemented_tuning_mode(self) -> Self:- match self.tuning_mode:- case WorkerTuningMode.EXPLICIT:- return self- case WorkerTuningMode.RESOURCE_BASED:- msg = "tuning_mode='resource_based' is reserved but not implemented in v2; use 'explicit'."- raise TemporalConfigError(msg)---class WorkerRuntimeProfilesConfig(ConfigModel):- """Named worker-runtime profiles selectable via ``--profile`` on the worker CLI."""-- default_profile: str- profiles: dict[str, WorkerRuntimeProfile]-- @model_validator(mode="after")- def validate_default_profile(self) -> Self:- if self.default_profile not in self.profiles:- msg = f"default_profile '{self.default_profile}' not in profiles (known: {sorted(self.profiles.keys())})"- raise TemporalConfigError(msg)- return self---@dataclass(frozen=True)-class DispatchOptions:- """Resolved per-call dispatch bundle returned by ``WorkerConfig.resolve_dispatch``.-- Splat ``to_execute_kwargs()`` into ``workflow.execute_activity(...)`` to- set queue + timeouts + retry consistently. ``task_queue`` is ``None`` when- no routing is configured (``activity_queues`` empty) — ``to_execute_kwargs``- then omits the key so Temporal routes to the workflow's own queue. Optional- timeouts are likewise omitted when unset so the Temporal SDK applies its- own defaults (rather than ``None``, which the SDK rejects).-- Frozen, build-once-read-only: constructed only inside ``resolve_dispatch``,- immediately splatted via ``to_execute_kwargs``, and never serialized across- the Temporal wire. ``retry_policy`` is ``Any`` at runtime (see the- ``RetryPolicy`` binding above) — type checkers still see the real- ``temporalio`` ``RetryPolicy``.- """-- task_queue: str | None- start_to_close_timeout: timedelta- retry_policy: RetryPolicy- schedule_to_close_timeout: timedelta | None = None- schedule_to_start_timeout: timedelta | None = None- heartbeat_timeout: timedelta | None = None-- def to_execute_kwargs(self) -> dict[str, Any]:- """Return a kwargs dict to splat into ``workflow.execute_activity``."""- kwargs: dict[str, Any] = {- "start_to_close_timeout": self.start_to_close_timeout,- "retry_policy": self.retry_policy,- }- if self.task_queue is not None:- kwargs["task_queue"] = self.task_queue- if self.schedule_to_close_timeout is not None:- kwargs["schedule_to_close_timeout"] = self.schedule_to_close_timeout- if self.schedule_to_start_timeout is not None:- kwargs["schedule_to_start_timeout"] = self.schedule_to_start_timeout- if self.heartbeat_timeout is not None:- kwargs["heartbeat_timeout"] = self.heartbeat_timeout- return kwargs---class WorkerConfig(ConfigModel):- """Submitter-side defaults plus the worker's home queue.-- ``default_task_queue`` is the fallback used by ``resolve_queue`` when no- ``activity_queues`` entry matches. ``default_activity_start_to_close_timeout``- is the baseline activity timeout used when no per-queue or per-handle- overlay applies (Phase 2 resolver).- """-- default_task_queue: str- activity_queues: dict[str, ActivityRouteConfig]- workflow_execution_timeout: timedelta = Field(strict=False)- run_timeout: timedelta | None = Field(default=None, strict=False)- task_timeout: timedelta | None = Field(default=None, strict=False)- start_delay: timedelta | None = Field(default=None, strict=False)- rpc_timeout: timedelta | None = Field(default=None, strict=False)- default_activity_start_to_close_timeout: timedelta = Field(strict=False)- default_activity_heartbeat_timeout: timedelta | None = Field(default=None, strict=False)- retry_policy_config: RetryPolicyConfig-- @property- def retry_policy(self) -> "RetryPolicy":- """Create a baseline RetryPolicy from the worker config.-- Returns:- RetryPolicy: A configured RetryPolicy object built from the baseline.- """- return self.retry_policy_config.make_retry_policy(merged_non_retryable_types=list(self.retry_policy_config.non_retryable_error_types))-- def all_non_retryable_error_types(- self,- queue_options_by_queue: dict[str, "QueueOptions"],- ) -> set[str]:- """Union of every ``non_retryable_error_types`` declared anywhere in the- temporal config: baseline main list + every queue overlay's ``_extra``- + every handle overlay's ``_extra``.-- Used by ``TemporalError.from_app_error`` for log-severity classification- — that error-handling site sees only the error type string, not the- dispatch that produced it, so it can't know which overlay contributed- the entry. Retry behavior itself is correct via the dispatch-time- ``retry_policy``; this helper only powers the log-severity decision.- """- result: set[str] = set(self.retry_policy_config.non_retryable_error_types)- for queue_opts in queue_options_by_queue.values():- if queue_opts.retry_policy_config is not None:- result.update(queue_opts.retry_policy_config.non_retryable_error_types_extra)- for route in self.activity_queues.values():- for handle_opts in route.handle_options.values():- if handle_opts.retry_policy_config is not None:- result.update(handle_opts.retry_policy_config.non_retryable_error_types_extra)- return result-- def resolve_queue(self, activity_name: str, *, routing_key: str | None = None) -> str | None:- """Resolve which task queue an activity should dispatch to.-- Hybrid fallback semantic. When ``activity_queues`` is fully empty- (default config, no routing configured), returns ``None`` — the dispatch- path then omits ``task_queue`` so Temporal routes the activity to the- workflow's own queue. This preserves the ``with_conditional_worker``- test pattern (workflow on a random queue, activities ride along) and- the pre-v1 default behavior for installs that haven't opted into- per-activity routing.-- When ``activity_queues`` has any entry, the operator has opted into- routing topology and unmapped activities still fall back explicitly to- ``default_task_queue``.-- Args:- activity_name: The Temporal activity ``__name__`` (e.g. ``"act_llm_gen_text"``).- routing_key: Optional per-activity handle (model handle, img-gen- handle, extract handle). Activities without a meaningful- routing dimension pass ``None``.-- Returns:- The task queue name, or ``None`` when no routing is configured.- Resolution order when ``activity_queues`` is non-empty:- 1. ``activity_queues[activity_name].by_handle[routing_key]``- 2. ``activity_queues[activity_name].default``- 3. ``self.default_task_queue`` (worker-wide default)- """- if not self.activity_queues:- return None- activity_route = self.activity_queues.get(activity_name)- if activity_route is None:- return self.default_task_queue- if routing_key is not None:- per_handle = activity_route.by_handle.get(routing_key)- if per_handle is not None:- return per_handle- return activity_route.default-- def resolve_dispatch(- self,- activity_name: str,- *,- routing_key: str | None = None,- queue_options_by_queue: dict[str, "QueueOptions"] | None = None,- is_traced: bool = False,- ) -> DispatchOptions:- """Resolve the full per-call dispatch bundle for an activity.-- Layers, last-wins for scalars (None means "no contribution"):- 1. worker_config baseline (``default_activity_start_to_close_timeout``,- ``default_activity_heartbeat_timeout``, ``retry_policy_config``).- 2. ``queue_options[resolved_queue]`` if present.- 3. ``activity_queues[activity_name].handle_options[routing_key]`` if present.-- Dispatch / queue_options asymmetry: when ``activity_queues`` is empty- the hybrid fallback returns ``resolved_queue=None`` so dispatch omits- the ``task_queue`` kwarg (activities ride the workflow's own queue).- But ``queue_options[default_task_queue]`` STILL applies in that case —- single-queue deployments can tune timeouts/retry/rate without opting- into the ``activity_queues`` routing topology.-- ``non_retryable_error_types`` composes **additively** across all three- layers (baseline list + queue ``_extra`` + handle ``_extra``) — never- substituted. This is a safety lean: per-queue layers can add to the- no-retry list but not remove from it.-- Args:- activity_name: The Temporal activity ``__name__``.- routing_key: Optional per-activity handle.- queue_options_by_queue: The full ``queue_options`` map (from- ``Temporal.queue_options``). Explicit dependency rather than a- ``get_config()`` reach so this method stays unit-testable.- is_traced: When true, emit one INFO log line per call with the- resolved queue + timeout + retry attempts + heartbeat and the- layer each scalar came from (baseline / queue_options /- handle_options). Off by default — verbose; turn on when- debugging mis-tuned timeouts.-- Returns:- A ``DispatchOptions`` ready to be splatted into- ``workflow.execute_activity(...)``.- """- resolved_queue = self.resolve_queue(activity_name=activity_name, routing_key=routing_key)-- queue_opts: QueueOptions | None = None- if queue_options_by_queue is not None:- if resolved_queue is not None:- queue_opts = queue_options_by_queue.get(resolved_queue)- else:- # Empty-routing hybrid fallback: dispatch still emits- # ``task_queue=None`` so the activity rides the workflow's own- # queue, but apply ``queue_options[default_task_queue]``- # overlays when configured so single-queue deployments can- # tune timeouts/retry/rate without opting into the- # ``activity_queues`` routing topology.- queue_opts = queue_options_by_queue.get(self.default_task_queue)-- handle_opts: HandleOptions | None = None- activity_route = self.activity_queues.get(activity_name)- if activity_route is not None and routing_key is not None:- handle_opts = activity_route.handle_options.get(routing_key)-- start_to_close = self.default_activity_start_to_close_timeout- start_to_close_source = "baseline"- schedule_to_close: timedelta | None = None- schedule_to_start: timedelta | None = None- heartbeat: timedelta | None = self.default_activity_heartbeat_timeout- heartbeat_source = "baseline"-- if queue_opts is not None:- if queue_opts.start_to_close_timeout is not None:- start_to_close = queue_opts.start_to_close_timeout- start_to_close_source = "queue_options"- if queue_opts.schedule_to_close_timeout is not None:- schedule_to_close = queue_opts.schedule_to_close_timeout- if queue_opts.schedule_to_start_timeout is not None:- schedule_to_start = queue_opts.schedule_to_start_timeout- if queue_opts.heartbeat_timeout is not None:- heartbeat = queue_opts.heartbeat_timeout- heartbeat_source = "queue_options"-- if handle_opts is not None and handle_opts.start_to_close_timeout is not None:- start_to_close = handle_opts.start_to_close_timeout- start_to_close_source = "handle_options"-- # Pick the deepest retry policy config to seed the build (intervals,- # backoff, attempts come from the most-specific layer). Then compose- # non_retryable_error_types additively across all three layers. The- # common ancestor RetryPolicyConfigBase lets us type the seed uniformly- # while keeping baseline vs overlay separated at the schema level.- retry_base: RetryPolicyConfigBase = self.retry_policy_config- retry_source = "baseline"- if queue_opts is not None and queue_opts.retry_policy_config is not None:- retry_base = queue_opts.retry_policy_config- retry_source = "queue_options"- if handle_opts is not None and handle_opts.retry_policy_config is not None:- retry_base = handle_opts.retry_policy_config- retry_source = "handle_options"-- merged_non_retryable: list[str] = list(self.retry_policy_config.non_retryable_error_types)- if queue_opts is not None and queue_opts.retry_policy_config is not None:- merged_non_retryable.extend(queue_opts.retry_policy_config.non_retryable_error_types_extra)- if handle_opts is not None and handle_opts.retry_policy_config is not None:- merged_non_retryable.extend(handle_opts.retry_policy_config.non_retryable_error_types_extra)- # Dedupe while preserving order so the resulting list is deterministic- # and small even when layers overlap.- seen: set[str] = set()- deduped_non_retryable: list[str] = []- for error_type in merged_non_retryable:- if error_type not in seen:- seen.add(error_type)- deduped_non_retryable.append(error_type)-- retry_policy = retry_base.make_retry_policy(merged_non_retryable_types=deduped_non_retryable)-- if is_traced:- parts = [- f"temporal.dispatch act={activity_name} handle={routing_key}",- f"-> queue={resolved_queue}",- f"timeout={start_to_close.total_seconds()}s (from={start_to_close_source})",- f"retry_attempts={retry_base.maximum_attempts} (from={retry_source})",- ]- if heartbeat is not None:- parts.append(f"heartbeat={heartbeat.total_seconds()}s (from={heartbeat_source})")- log.info(" ".join(parts))-- return DispatchOptions(- task_queue=resolved_queue,- start_to_close_timeout=start_to_close,- schedule_to_close_timeout=schedule_to_close,- schedule_to_start_timeout=schedule_to_start,- heartbeat_timeout=heartbeat,- retry_policy=retry_policy,- )---class PayloadCodecConfig(ConfigModel):- """Configuration for the storage-based payload codec that offloads large payloads."""-- is_enabled: bool- size_threshold: int- storage_prefix: str- storage_provider_config: StorageProviderConfig---class Temporal(ConfigModel):- """Main configuration model for Temporal."""-- is_enabled: bool- temporal_config: TemporalConfig- worker_config: WorkerConfig- queue_options: dict[str, QueueOptions]- worker_runtime_profiles: WorkerRuntimeProfilesConfig- worker_scopes: WorkerScopesConfig- payload_codec_config: PayloadCodecConfig- search_attributes: SearchAttributesConfig-- @model_validator(mode="after")- def validate_no_orphan_queue_references(self) -> Self:- """Fail config-load on two orphan patterns:-- 1. ``activity_queues`` references a queue with no ``queue_options``- entry and not equal to ``default_task_queue`` — almost always a typo.- 2. ``queue_options`` declares a queue that nothing routes to and isn't- ``default_task_queue`` — overlay will never apply.-- Every queue named under ``activity_queues`` must have a matching- ``[temporal.queue_options.<q>]`` entry. An empty stanza is fine and- explicitly means "use worker_config defaults for this queue" — it is- the one-line declaration of intent that closes the typo loop.- ``default_task_queue`` is implicitly known and never needs a- ``queue_options`` entry.- """- known_queues = set(self.queue_options.keys())- known_queues.add(self.worker_config.default_task_queue)- for activity_name, route in self.worker_config.activity_queues.items():- if route.default not in known_queues:- msg = (- f"temporal: queue {route.default!r} is referenced by "- f"activity_queues.{activity_name}.default but has no "- f"[temporal.queue_options.{route.default}] entry and is not default_task_queue. "- f"Fix: either (a) add an entry '[temporal.queue_options.{route.default}]' "- f"to your config (an empty stanza is fine — it means 'use worker_config "- f"defaults for this queue'), or (b) fix the typo in "- f"activity_queues.{activity_name}.default."- )- raise TemporalConfigError(msg)- for handle, handle_queue in route.by_handle.items():- if handle_queue not in known_queues:- msg = (- f"temporal: queue {handle_queue!r} is referenced by "- f'activity_queues.{activity_name}.by_handle["{handle}"] but has no '- f"[temporal.queue_options.{handle_queue}] entry and is not default_task_queue. "- f"Fix: either (a) add an entry '[temporal.queue_options.{handle_queue}]' "- f"to your config (an empty stanza is fine — it means 'use worker_config "- f"defaults for this queue'), or (b) fix the typo in "- f'activity_queues.{activity_name}.by_handle["{handle}"].'- )- raise TemporalConfigError(msg)-- # Symmetric check: queue_options entries that nothing routes to are- # silently no-op overlays. ``default_task_queue`` is considered- # "routed" because the hybrid fallback applies its queue_options when- # ``activity_queues`` is empty.- routed_queues: set[str] = {self.worker_config.default_task_queue}- for route in self.worker_config.activity_queues.values():- routed_queues.add(route.default)- routed_queues.update(route.by_handle.values())- for queue_name in self.queue_options:- if queue_name not in routed_queues:- msg = (- f"temporal: [temporal.queue_options.{queue_name}] declares queue "- f"{queue_name!r} but no activity_queues route references it and "- f"{queue_name!r} is not default_task_queue. The overlay will never apply. "- f"Fix: either (a) add a route under [temporal.worker_config.activity_queues.*] "- f"that points at {queue_name!r}, or (b) remove the "- f"[temporal.queue_options.{queue_name}] entry."- )- raise TemporalConfigError(msg)- return self-- def validate_task_queue_known(self, task_queue: str) -> None:- """Raise ``WorkerTaskQueueUnknownError`` when ``task_queue`` is not- declared anywhere in the temporal config — neither as- ``default_task_queue``, nor in any ``activity_queues`` entry, nor in- ``queue_options``.-- Strict counterpart to the lenient warn that fires at config load time:- a worker polling a queue that nothing routes to is almost always a typo,- and the runtime can't detect it later — the worker would sit idle- forever. Fast-fail at startup with a "did you mean?" suggestion when a- close match exists.-- Called from both ``worker_cli.configure`` (CLI fast-fail before library- load) and ``TemporalTaskManager.run_worker`` (so programmatic callers- get the check too).- """- known_queues: set[str] = {self.worker_config.default_task_queue}- for route in self.worker_config.activity_queues.values():- known_queues.add(route.default)- known_queues.update(route.by_handle.values())- known_queues.update(self.queue_options.keys())- if task_queue in known_queues:- return- sorted_known = sorted(known_queues)- suggestions = difflib.get_close_matches(task_queue, sorted_known, n=1, cutoff=0.7)- msg = f"--task-queue '{task_queue}' is not referenced by any routing or options entry. Known queues: {sorted_known}."- if suggestions:- msg += f" Did you mean '{suggestions[0]}'?"- raise WorkerTaskQueueUnknownError(msg)diff --git a/pipelex/temporal/exceptions.py b/pipelex/temporal/exceptions.pydeleted file mode 100644index 0a52bc051..000000000--- a/pipelex/temporal/exceptions.py+++ /dev/null@@ -1,93 +0,0 @@-from typing_extensions import override--from pipelex.base_exceptions import ErrorDomain, ErrorReport, PipelexError---class TemporalFlowError(PipelexError):- pass---class UnrecoverableWorkflowFailureError(TemporalFlowError):- """Synthesized when a Temporal workflow failure carries no recoverable ``ErrorReport``.-- ``recover_error_report`` is total — every ``BaseException`` produces an- ``ErrorReport``. When the failure has no embedded report (a non-Pipelex- worker exception, a Temporal infra error, a heartbeat timeout), we- synthesize this exception and return its ``to_error_report()``. That- gives consumers a uniform shape with a- stable identity (``error_type`` / ``title`` / ``type_uri``) and the- ``RUNTIME`` classification, plus the most informative message we could- recover from the failure chain.- """-- error_domain = ErrorDomain.RUNTIME- _declared_title = "Unrecoverable workflow failure"---class WorkflowInputError(TemporalFlowError):- pass---class WorkflowExecutionError(TemporalFlowError):- """A Temporal workflow failure observed on the submitter side.-- When the failure crossed the activity -> workflow -> submitter boundary- carrying a structured ``ErrorReport`` (packed into ``ApplicationError.details``- by the activity bridge), the submitter recovers it and passes it as- ``error_report``. ``to_error_report()`` then surfaces the original- classification — ``error_category`` / ``retryable`` / ``model`` / ``provider``- / ``user_action`` / ``provider_metadata`` — that the Temporal serialization- boundary would otherwise drop, leaving only a generic ``RUNTIME`` error.-- When no report was recovered (``error_report=None`` — a non-Pipelex failure,- or a ``WorkflowAlreadyStartedError`` / ``RPCError``), ``to_error_report()``- falls through to the base ``__cause__``-chain enrichment.- """-- def __init__(self, message: str, error_report: ErrorReport | None = None):- super().__init__(message)- self.error_report = error_report-- @override- def to_error_report(self) -> ErrorReport:- # A recovered report already carries the worker-side classification in- # full; return it verbatim. Upstream wrappers (``PipelineExecutionError``)- # re-apply their own ``error_type`` / ``message`` via cause-chain- # enrichment. With no recovered report, fall through to the base- # behavior, which ends with ``_enrich_error_report_from_cause``.- if self.error_report is not None:- return self.error_report- return super().to_error_report()---class ContentGenerationError(TemporalFlowError):- pass---class TemporalConfigError(ValueError, TemporalFlowError):- pass---class TemporalServerError(TemporalFlowError):- pass---class WorkerScopeConfigError(TemporalConfigError):- pass---class WorkerProfileConfigError(TemporalConfigError):- pass---class WorkerTaskQueueUnknownError(TemporalConfigError):- pass---class SearchAttributeRegistrationError(TemporalConfigError):- """Raised at worker boot when the namespace is reachable but missing a- configured custom search attribute. The error message includes both the- ``pipelex setup-temporal-namespace`` invocation and the equivalent raw- ``temporal operator search-attribute create`` command so operators on- either side of the fence can fix the gap.- """diff --git a/pipelex/temporal/log_formatter.py b/pipelex/temporal/log_formatter.pydeleted file mode 100644index 1cfcc26c1..000000000--- a/pipelex/temporal/log_formatter.py+++ /dev/null@@ -1,52 +0,0 @@-import logging-from typing import Any--from typing_extensions import override--from pipelex.config import get_config-from pipelex.tools.log.log_formatter import LevelAndEmojiLogFormatter, log_level_color, log_level_tag-from pipelex.tools.misc.terminal_utils import RESET_FONT, TerminalColor---class TemporalLogFormatter(LevelAndEmojiLogFormatter):- @override- def format(self, record: logging.LogRecord):- temporal_log_config = get_config().temporal.temporal_config.temporal_log_config- if record.name not in temporal_log_config.managed_loggers:- return super().format(record)-- # these are prints because we are configuring logs- # they are commmented but remain as a practical solution to check what temporal injects in logs in various contexts- # print(f"\n\n{record.name=}")- # print(f"\n\n{json.dumps(vars(record), default=str)}\n\n")-- prefix = ""-- if temporal_log_config.is_prefix_enabled:- wf_color = TerminalColor.MAGENTA- act_color = TerminalColor.CYAN- if temporal_workflow := getattr(record, "temporal_workflow", None):- workflow_dict: dict[str, Any] = temporal_workflow- if workflow_type := workflow_dict.get("workflow_type"):- prefix += f"{wf_color}[{workflow_type}]{RESET_FONT}"-- if temporal_activity := getattr(record, "temporal_activity", None):- activity_dict: dict[str, Any] = temporal_activity- if workflow_type := activity_dict.get("workflow_type"):- prefix += f"{wf_color}[{workflow_type}]{RESET_FONT}"- if activity_type := activity_dict.get("activity_type"):- prefix += f"{act_color}[{activity_type}]{RESET_FONT}"-- if prefix:- prefix += " "-- color = log_level_color.get(record.levelno, RESET_FONT)- tag = log_level_tag.get(record.levelno, "?????")-- if record.levelno in {logging.WARNING, logging.ERROR, logging.CRITICAL}:- record.msg = f"{color}{record.msg}{RESET_FONT}"-- log_fmt = f"{color}{tag}:{prefix}{RESET_FONT} %(message)s"-- formatter = logging.Formatter(log_fmt)- return formatter.format(record)diff --git a/pipelex/temporal/log_temporal.py b/pipelex/temporal/log_temporal.pydeleted file mode 100644index a2d37b3a9..000000000--- a/pipelex/temporal/log_temporal.py+++ /dev/null@@ -1,116 +0,0 @@-import logging-from typing import Any, ClassVar, Union--from temporalio import activity, workflow--from pipelex import log-from pipelex.config import get_config-from pipelex.temporal.log_formatter import TemporalLogFormatter-from pipelex.tools.log.log_levels import LOGGING_LEVEL_DEV, LOGGING_LEVEL_VERBOSE---def configure_temporal_logs():- """Configure Temporal logging settings based on the application configuration.-- This function sets up the log formatter, safety callable, and various logging options- for both workflow and activity loggers.- """- temporal_log_config = get_config().temporal.temporal_config.temporal_log_config- log.set_poor_log_formatter(TemporalLogFormatter())-- workflow.logger.workflow_info_on_message = temporal_log_config.is_workflow_info_on_message- workflow.logger.workflow_info_on_extra = temporal_log_config.is_workflow_info_on_extra- workflow.logger.full_workflow_info_on_extra = temporal_log_config.is_full_workflow_info_on_extra-- activity.logger.activity_info_on_message = temporal_log_config.is_activity_info_on_message- activity.logger.activity_info_on_extra = temporal_log_config.is_activity_info_on_extra- activity.logger.full_activity_info_on_extra = temporal_log_config.is_full_activity_info_on_extra---class _RequestIdLog:- """Base for the Temporal log helpers — carries the per-invocation ``request_id``- and the seven severity methods that route through the bound logger.-- A :class:`WorkflowLog` / :class:`ActivityLog` is built once per workflow or- activity invocation, bound to that invocation's ``job_metadata.request_id``- (``None`` when the run carries no inbound API request id). Every severity- method packs that ``request_id`` into the Temporal log record via- ``extra={"request_id": ...}``, so downstream log shippers can correlate the- record back to the originating API request — the call site never threads the- id itself.-- Subclasses point ``_logger`` at the right Temporal logger- (``workflow.logger`` or ``activity.logger``).- """-- # Set on each subclass to the right Temporal logger. The two loggers are- # ``LoggerAdapter`` instances initialized at module import time; a ClassVar- # reference holds the same instance the call sites and tests target.- _logger: ClassVar[Any]-- def __init__(self, request_id: str | None = None) -> None:- self._request_id = request_id-- def _build_extra(self) -> dict[str, str] | None:- """Pack the bound ``request_id`` into the logger ``extra`` dict, omitting when absent.-- The Temporal logger merges ``extra`` into each log record so downstream- formatters (and JSON-shipping consumers) can read ``record.request_id``.- """- if self._request_id is None:- return None- return {"request_id": self._request_id}-- def verbose(self, content: Union[str, Any]) -> None:- self._logger.log(level=LOGGING_LEVEL_VERBOSE, msg=content, extra=self._build_extra())-- def debug(self, content: Union[str, Any]) -> None:- self._logger.log(level=logging.DEBUG, msg=content, extra=self._build_extra())-- def dev(self, content: Union[str, Any]) -> None:- self._logger.log(level=LOGGING_LEVEL_DEV, msg=content, extra=self._build_extra())-- def info(self, content: Union[str, Any]) -> None:- self._logger.log(level=logging.INFO, msg=content, extra=self._build_extra())-- def warning(self, content: Union[str, Any]) -> None:- self._logger.log(level=logging.WARNING, msg=content, extra=self._build_extra())-- def error(self, content: Union[str, Any]) -> None:- self._logger.log(level=logging.ERROR, msg=content, extra=self._build_extra())-- def critical(self, content: Union[str, Any]) -> None:- self._logger.log(level=logging.CRITICAL, msg=content, extra=self._build_extra())---class WorkflowLog(_RequestIdLog):- """Logs messages in Temporal workflows at different severity levels.-- Build one per ``@workflow.run`` invocation, bound to that run's- ``job_metadata.request_id`` — every record it emits then carries that- ``request_id`` (see :class:`_RequestIdLog`). The module-level- :data:`workflow_log` singleton is unbound (``request_id is None``); it is- used by call sites that have no ``job_metadata`` in scope.- """-- _logger: ClassVar[Any] = workflow.logger---class ActivityLog(_RequestIdLog):- """Logs messages in Temporal activities at different severity levels.-- Build one per activity invocation, bound to that activity's- ``job_metadata.request_id`` — every record it emits then carries that- ``request_id`` (see :class:`_RequestIdLog`). The module-level- :data:`activity_log` singleton is unbound (``request_id is None``); it is- used by call sites that have no ``job_metadata`` in scope.- """-- _logger: ClassVar[Any] = activity.logger---# Unbound singletons for call sites with no ``job_metadata`` in scope (e.g. the-# ``TemporalError`` severity logs). Entry points that do have ``job_metadata``-# build their own bound instance instead — see ``WfPipeRun`` / ``WfPipeRouter``.-workflow_log = WorkflowLog()-activity_log = ActivityLog()diff --git a/pipelex/temporal/sandbox_manager.py b/pipelex/temporal/sandbox_manager.pydeleted file mode 100644index 38059d4ae..000000000--- a/pipelex/temporal/sandbox_manager.py+++ /dev/null@@ -1,29 +0,0 @@-from typing import Callable--from pydantic import BaseModel---class SandboxManager(BaseModel):- is_in_sandbox_callable: Callable[[], bool] | None = None-- def set_sandbox_callable(self, sandbox_callable: Callable[[], bool]):- """Sets a callable function to determine if we're in a sandbox, which would prevent importing unsafe modules,- and in particular prevent logging with default methods.-- Args:- sandbox_callable: A function that returns True if we're in a sandbox, False otherwise.- """- self.is_in_sandbox_callable = sandbox_callable-- def is_in_sandbox(self) -> bool:- """Returns:- bool: True if we're in a sandbox, False otherwise.- """- if self.is_in_sandbox_callable is None:- # by default, we are not in a sandbox- return False- else:- return self.is_in_sandbox_callable()---sandbox_manager = SandboxManager()diff --git a/pipelex/temporal/task_manager.py b/pipelex/temporal/task_manager.pydeleted file mode 100644index 046d9b4dd..000000000--- a/pipelex/temporal/task_manager.py+++ /dev/null@@ -1,57 +0,0 @@-from __future__ import annotations--from typing import TYPE_CHECKING, Protocol--if TYPE_CHECKING:- from temporalio.client import Client as TemporalClient- from temporalio.worker import Worker-- from pipelex.temporal.config_temporal import WorkerRuntimeProfile, WorkerScope- from pipelex.temporal.temporal_tasks import TaskPack- from pipelex.temporal.temporal_types import ActivityList, ActivityType, WorkflowList, WorkflowType---class TaskManager(Protocol):- def complement_catalog(- self,- extra_catalog: dict[str, TaskPack],- *,- extra_workflows: list[WorkflowType],- extra_activities: list[ActivityType],- ): ...-- def make_worker(- self,- temporal_client: TemporalClient,- *,- task_queue: str,- is_not_sandboxed: bool = False,- scope: WorkerScope | None = None,- runtime_profile: WorkerRuntimeProfile | None = None,- substitute_activities: dict[ActivityType, ActivityType] | None = None,- test_workflows: WorkflowList | None = None,- test_activities: ActivityList | None = None,- ) -> Worker: ...-- async def run_worker(- self,- *,- is_not_sandboxed: bool,- is_unit_testing: bool,- task_queue: str | None = None,- scope_name: str | None = None,- profile_name: str | None = None,- ): ...-- def task_packs(self) -> list[str]: ...-- def workflows_and_activities(- self,- scope: WorkerScope | None = None,- *,- test_workflows: WorkflowList | None = None,- test_activities: ActivityList | None = None,- substitute_activities: dict[ActivityType, ActivityType] | None = None,- ) -> tuple[list[WorkflowType], list[ActivityType]]: ...-- def workflows_and_activities_str(self) -> tuple[list[str], list[str]]: ...diff --git a/pipelex/temporal/tasks.py b/pipelex/temporal/tasks.pydeleted file mode 100644index 45f168e39..000000000--- a/pipelex/temporal/tasks.py+++ /dev/null@@ -1,54 +0,0 @@-from typing import ClassVar--from pipelex.temporal.temporal_tasks import TaskPack-from pipelex.temporal.tprl_content_generation.act_extract_generate import act_extract_gen_extract_pages-from pipelex.temporal.tprl_content_generation.act_img_gen_generate import act_img_gen_images-from pipelex.temporal.tprl_content_generation.act_jinja2_generate import act_jinja2_gen_text-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_object, act_llm_gen_object_list, act_llm_gen_text-from pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views-from pipelex.temporal.tprl_content_generation.act_search_generate import act_search_gen_sourced_answer, act_search_gen_structured-from pipelex.temporal.tprl_pipe.act_assemble_tracing import act_assemble_tracing-from pipelex.temporal.tprl_pipe.act_deliver import act_deliver-from pipelex.temporal.tprl_pipe.act_dry_validate import act_dry_validate-from pipelex.temporal.tprl_pipe.act_flush_trace_events import act_flush_trace_events-from pipelex.temporal.tprl_pipe.wf_dry_validate import WfDryValidate-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun-from pipelex.types import StrEnum---class PackName(StrEnum):- CRAFTING = "crafting"- PIPE = "pipe"---class Tasks:- TASK_PACKS: ClassVar[dict[str, TaskPack]] = {- PackName.CRAFTING: TaskPack(- workflow_list=[],- activity_list=[- act_llm_gen_object,- act_llm_gen_text,- act_llm_gen_object_list,- act_img_gen_images,- act_jinja2_gen_text,- act_extract_gen_extract_pages,- act_render_page_views,- act_search_gen_sourced_answer,- act_search_gen_structured,- ],- ),- PackName.PIPE: TaskPack(- workflow_list=[- WfPipeRouter,- WfPipeRun,- WfDryValidate,- ],- activity_list=[- act_assemble_tracing,- act_deliver,- act_dry_validate,- act_flush_trace_events,- ],- ),- }diff --git a/pipelex/temporal/temporal_connect.py b/pipelex/temporal/temporal_connect.pydeleted file mode 100644index 27359eeb5..000000000--- a/pipelex/temporal/temporal_connect.py+++ /dev/null@@ -1,103 +0,0 @@-from typing import TYPE_CHECKING, Mapping--from temporalio.client import Client as TemporalClient--from pipelex import log-from pipelex.config import get_config-from pipelex.hub import get_secret-from pipelex.system.environment import get_required_env-from pipelex.temporal.codec.codec_factory import make_codec_from_config-from pipelex.temporal.config_temporal import SecretMethod, TemporalConfigError, TemporalServerConfig-from pipelex.temporal.exceptions import TemporalServerError-from pipelex.temporal.temporal_data_converter import data_converter, make_data_converter--if TYPE_CHECKING:- from temporalio.converter import DataConverter---async def connect_to_temporal_server(server_config: TemporalServerConfig, *, name: str | None = None) -> TemporalClient:- """Connect to Temporal using the provided server config."""- api_key: str | None- match server_config.api_key_method:- case SecretMethod.NONE:- api_key = None- case SecretMethod.ENV_VAR:- if api_key_id := server_config.api_key_id:- api_key = get_required_env(api_key_id)- else:- msg = "api_key_id is required for api_key_method=ENV_VAR"- raise TemporalConfigError(msg)- case SecretMethod.SECRET_PROVIDER:- if api_key_id := server_config.api_key_id:- api_key = get_secret(secret_id=api_key_id)- else:- msg = "api_key_id is required for api_key_method=SECRET_PROVIDER"- raise TemporalConfigError(msg)-- tls: bool- rpc_metadata: Mapping[str, str]- if api_key:- rpc_metadata = {"temporal-namespace": server_config.namespace}- tls = True- else:- rpc_metadata = {}- tls = False-- # Build the data converter, optionally with a storage-based payload codec.- payload_codec_config = get_config().temporal.payload_codec_config- converter: DataConverter- if payload_codec_config.is_enabled:- payload_codec = make_codec_from_config()- converter = make_data_converter(payload_codec=payload_codec)- log.info(f"Payload codec enabled — threshold={payload_codec_config.size_threshold}, prefix='{payload_codec_config.storage_prefix}'")- else:- converter = data_converter-- log.info(f"Connecting to Temporal server: {server_config.full_description}")- log.info(- f"""Establishing connection to Temporal...- - Selected Server Config : {name}- - Target Host : {server_config.target_host}- - Namespace : {server_config.namespace}- - TLS Enabled : {"Yes" if tls else "No"}- - API Key : {"Provided" if api_key else "Not Provided"}- - RPC Metadata : {rpc_metadata or "None"}- """,- )-- try:- temporal_client: TemporalClient = await TemporalClient.connect(- target_host=server_config.target_host,- namespace=server_config.namespace,- rpc_metadata=rpc_metadata,- api_key=api_key,- data_converter=converter,- tls=tls,- )- except RuntimeError as exc:- msg = f"Failed to connect to Temporal server using config: {server_config.full_description}\nError: {exc}"- raise TemporalServerError(msg) from exc- return temporal_client---async def connect_to_temporal_selected_server(- selected_server_config: str,-) -> TemporalClient:- """Connect to Temporal using the server selected by argument."""- temporal_config = get_config().temporal.temporal_config- log.dev(f"Using Temporal server config named: '{selected_server_config}'")- server_config = temporal_config.temporal_server_configs.get(selected_server_config)- if not server_config:- msg = f"Server config not found for selected server: '{selected_server_config}'"- raise TemporalConfigError(msg)- temporal_client = await connect_to_temporal_server(server_config=server_config, name=selected_server_config)- log.info(f"Connected to Temporal server: '{selected_server_config}' = {server_config.full_description}")- return temporal_client---async def connect_to_temporal() -> TemporalClient:- """Connect to Temporal using the server selected from the config."""- temporal_config = get_config().temporal.temporal_config- return await connect_to_temporal_selected_server(- selected_server_config=temporal_config.selected_server,- )diff --git a/pipelex/temporal/temporal_data_converter.py b/pipelex/temporal/temporal_data_converter.pydeleted file mode 100644index 7cd8132f7..000000000--- a/pipelex/temporal/temporal_data_converter.py+++ /dev/null@@ -1,212 +0,0 @@-"""This module was entirely rewritten based on Temporal's sample "pydantic_converter":-https://github.com/temporalio/samples-python/tree/main/pydantic_converter--The purpose of this converter is to serialize and deserialize inputs and outputs of Temporal activities and workflows.--Our implementation includes several key improvements:-- Compatibility with Pydantic v2-- Support for universal JSON serialization/deserialization via `pipelex.tools.serde.kajson`-- Preservation of subclass types--Please note that, while Temporal does not officially support Generics,-this converter serves as a reliable workaround for Pydantic models.-It also supports lists such as `List[BaseModel]` and `List[BaseModelSubclass]`.--For examples, see the tests in `test_top_crafter.py`:--1. `make ti TEST=test_make_object`- - This test uses `WfCraftObject`, which returns a `BaseModel`.- - However, the actual class of the returned object is determined by `object_assignment.object_class`,- allowing the calling method to cast it to the required generic type with `cast(BaseModelType, obj)`.--2. `make ti TEST=test_make_object_list`- - This test uses `WfCraftObjectList`, which returns a `List[BaseModel]`.- - Similarly, the items in the returned list use the class specified in `object_assignment.object_class`,- allowing the calling method to cast the list to the required generic type with `cast(List[BaseModelType], obj_list)`.-"""--from typing import Any, cast--from kajson import kajson-from kajson.class_registry import ClassRegistry-from pydantic import BaseModel-from pydantic.dataclasses import is_pydantic_dataclass-from temporalio.api.common.v1 import Payload-from temporalio.converter import (- CompositePayloadConverter,- DataConverter,- DefaultPayloadConverter,- EncodingPayloadConverter,- JSONPlainPayloadConverter,- PayloadCodec,-)-from typing_extensions import override--from pipelex import log-from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory-from pipelex.hub import get_class_registry---class BaseModelPayloadConverter(JSONPlainPayloadConverter):- """Universal Pydantic JSON payload converter.-- This extends the :py:class:`JSONPlainPayloadConverter` to override :py:meth:`to_payload`.- """-- @classmethod- def _is_kajson_wire_value(cls, value: object) -> bool:- """A value kajson serializes with type preservation: a BaseModel instance or a pydantic dataclass instance.-- The BaseModel check is first so it short-circuits on the hot path before the pydantic-dataclass probe.- """- return isinstance(value, BaseModel) or is_pydantic_dataclass(type(value))-- def _kajson_to_payload(self, value: object, *, source_type_holder: object) -> Payload:- """Serialize a kajson wire value (or list of them) to a Payload.-- ``source_type_holder`` is the object whose type may carry a dynamic-class ``__kajson_class_source__``- (the value itself for a scalar, or the first element for a list). That attribute is absent on pydantic- dataclasses — only dynamically-built BaseModel classes set it — so the lookup is harmlessly ``None`` there.- """- payload_str: str = kajson.dumps(value)- metadata: dict[str, bytes] = {"encoding": self.encoding.encode()}- class_source = getattr(type(source_type_holder), "__kajson_class_source__", None)- if class_source is not None:- metadata["kajson_class_source"] = class_source.encode()- return Payload(metadata=metadata, data=payload_str.encode())-- @classmethod- def _first_kajson_list_element(cls, value: object) -> object | None:- """Return the first element if value is a non-empty list whose head is a kajson wire value, else None."""- if not isinstance(value, list):- return None- items = cast("list[object]", value)- if items and cls._is_kajson_wire_value(items[0]):- return items[0]- return None-- @override- def to_payload(self, value: Any) -> Payload | None:- if self._is_kajson_wire_value(value):- return self._kajson_to_payload(value, source_type_holder=value)- list_head = self._first_kajson_list_element(value)- if list_head is not None:- return self._kajson_to_payload(value, source_type_holder=list_head)- return super().to_payload(value)-- def _restore_class_source(self, value: BaseModel, *, class_source_code: str) -> None:- value_class = cast("Any", type(value))- value_class.__kajson_class_source__ = class_source_code-- def _kajson_deserialize_from_payload(self, payload: Payload) -> Any:- data = payload.data.decode()- log.verbose(f"unijson_deserialize_payload — data: {data}")- source_bytes = payload.metadata.get("kajson_class_source")- class_source_code = source_bytes.decode() if source_bytes else None-- global_registry = get_class_registry()- if class_source_code is not None:- # Build a per-call scoped registry overlaying the global one with the- # source-derived dynamic classes (BaseModel + Enum). Per-call so dynamic- # classes never persist across payloads — different schemas may reuse the- # same class name without collision.- source_types = SchemaToModelFactory.make_types_from_source(class_source_code)- scoped_registry = ClassRegistry()- scoped_registry.register_classes_dict(global_registry.get_classes_dict())- for type_name, type_obj in source_types.items():- scoped_registry.register_class(type_obj, name=type_name, should_warn_if_already_registered=False)- pydantic_gizmo = kajson.loads(data, class_registry=scoped_registry)- else:- pydantic_gizmo = kajson.loads(data, class_registry=global_registry)-- if class_source_code is not None:- if isinstance(pydantic_gizmo, BaseModel):- self._restore_class_source(pydantic_gizmo, class_source_code=class_source_code)- elif isinstance(pydantic_gizmo, list):- for item in cast("list[Any]", pydantic_gizmo):- if isinstance(item, BaseModel):- self._restore_class_source(item, class_source_code=class_source_code)- log.verbose(f"unijson_deserialize_payload — pydantic_gizmo: {pydantic_gizmo}")- return cast("Any", pydantic_gizmo)-- @classmethod- def _is_kajson_type(cls, type_hint: Any) -> bool:- """A type kajson reconstructs with type preservation: a BaseModel subclass or a pydantic dataclass.-- The inner args of ``Optional`` / ``list`` hints may be typing constructs rather than classes;- ``is_pydantic_dataclass`` returns ``False`` for those without raising, so no guard is needed.- """- if isinstance(type_hint, type) and issubclass(type_hint, BaseModel):- return True- # cast collapses the post-isinstance ``Any | type[Unknown]`` back to ``Any`` for the typed probe.- return is_pydantic_dataclass(cast("Any", type_hint))-- @classmethod- def _unwrap_optional_kajson_type(cls, type_hint: type[Any] | None) -> bool:- """Check if type_hint is Optional[X] / X | None where X is a kajson wire type."""- if type_hint is None:- return False- args = getattr(type_hint, "__args__", None)- if args is None:- return False- # Optional[X] is Union[X, None] — check if any non-None arg is a kajson wire type- for arg in args:- if arg is type(None):- continue- if cls._is_kajson_type(arg):- return True- return False-- @override- def from_payload(- self,- payload: Payload,- type_hint: type[Any] | None = None,- ) -> Any:- # BaseModel / pydantic-dataclass scalar case- if self._is_kajson_type(type_hint):- return self._kajson_deserialize_from_payload(payload=payload)-- # Optional case (e.g. GraphSpec | None)- if self._unwrap_optional_kajson_type(type_hint):- return self._kajson_deserialize_from_payload(payload=payload)-- # list case- origin = getattr(type_hint, "__origin__", None)- args = getattr(type_hint, "__args__", None)- log.verbose(f"Type hint origin: {origin}, args: {args}")- if origin is list and args and len(args) == 1 and self._is_kajson_type(args[0]):- log.debug(f"Type hint is a list of kajson wire type: {args[0]}")- return self._kajson_deserialize_from_payload(payload=payload)-- log.verbose(f"The type hint '{type_hint}' is neither a kajson wire type nor a list of kajson wire types")- return super().from_payload(payload, type_hint) # pyright: ignore[reportUnknownMemberType]---class PydanticCompositePayloadConverter(CompositePayloadConverter):- """Payload converter that replaces Temporal JSON conversion without BaseModelPayloadConverter JSON conversion."""-- def __init__(self) -> None:- converters: list[EncodingPayloadConverter] = []- for converter in DefaultPayloadConverter.default_encoding_payload_converters:- if isinstance(converter, JSONPlainPayloadConverter):- converters.append(BaseModelPayloadConverter())- else:- converters.append(converter)-- super().__init__(*converters)---def make_data_converter(payload_codec: PayloadCodec | None = None) -> DataConverter:- """Create a DataConverter with our Pydantic-aware payload converter and an optional codec.-- Args:- payload_codec: Optional codec for offloading large payloads to external storage.- """- return DataConverter(- payload_converter_class=PydanticCompositePayloadConverter,- payload_codec=payload_codec,- )---data_converter = make_data_converter()diff --git a/pipelex/temporal/temporal_hub.py b/pipelex/temporal/temporal_hub.pydeleted file mode 100644index 956363f1d..000000000--- a/pipelex/temporal/temporal_hub.py+++ /dev/null@@ -1,41 +0,0 @@-from __future__ import annotations--from typing import TYPE_CHECKING--if TYPE_CHECKING:- from pipelex.temporal.task_manager import TaskManager---class TemporalHub:- """TemporalHub serves as a central dependency manager to break cyclic imports between components.- It provides access to core providers and factories through a singleton instance,- allowing components to retrieve dependencies based on protocols without direct imports that could create cycles.- """-- def __init__(self):- self._task_manager: TaskManager | None = None-- # Setters-- def set_task_manager(self, task_manager: TaskManager):- self._task_manager = task_manager-- def reset(self):- self._task_manager = None-- # Getters-- def get_required_task_manager(self) -> TaskManager:- if self._task_manager is None:- msg = "TaskManager instance is not set. You must initialize Pipelex first."- raise RuntimeError(msg)- return self._task_manager---temporal_hub = TemporalHub()--# root convenience functions---def get_task_manager() -> TaskManager:- return temporal_hub.get_required_task_manager()diff --git a/pipelex/temporal/temporal_manager.py b/pipelex/temporal/temporal_manager.pydeleted file mode 100644index d4b54e532..000000000--- a/pipelex/temporal/temporal_manager.py+++ /dev/null@@ -1,116 +0,0 @@-from typing import ClassVar, Optional--from temporalio.client import Client as TemporalClient--from pipelex import log-from pipelex.system.runtime import RunMode, runtime_manager-from pipelex.temporal.config_temporal import TemporalServerConfig-from pipelex.temporal.temporal_connect import connect_to_temporal, connect_to_temporal_selected_server, connect_to_temporal_server-from pipelex.types import StrEnum---class TemporalWorkerEnvironment(StrEnum):- INTERNAL = "internal"- EXTERNAL = "external"---class TemporalManager:- _shared_instance: ClassVar[Optional["TemporalManager"]] = None-- def __init__(self, session_id: str) -> None:- if TemporalManager._shared_instance is not None:- msg = "TemporalManager is a singleton. Use get_instance() to access it."- raise RuntimeError(msg)- super().__init__()- self.session_id = session_id- self.temporal_client: TemporalClient | None = None- TemporalManager._shared_instance = self-- @classmethod- def get_instance(cls) -> "TemporalManager":- if cls._shared_instance is None:- msg = "Shared instance is not set. You must call TemporalManager.setup() once."- raise RuntimeError(msg)- return cls._shared_instance-- @classmethod- def setup(cls, session_id: str) -> None:- if cls._shared_instance is not None:- cls.teardown()- cls._shared_instance = cls(session_id=session_id)- log.debug(f"TemporalManager setup done, session_id={session_id}")-- @classmethod- def teardown(cls) -> None:- cls._shared_instance = None-- async def connect_temporal(- self,- temporal_client: TemporalClient | None = None,- *,- temporal_server_config: TemporalServerConfig | None = None,- temporal_selected_server: str | None = None,- ) -> TemporalClient:- """One remark first: this method is async only because Temporal's Client.connect() method is async.-- This method is only passing your settings or using defaults from pipelex.temporal's config.- To init the temporal client, it will use one of the provided arguments if available, in the following order:- - temporal_client- - temporal_server_config- - temporal_selected_server- ... otherwise it will use the default connection set by temporal's config.- """- # select temporal client based on the available arguments- the_temporal_client: TemporalClient- if temporal_client:- log.dev("Connecting temporal using provided Temporal client")- the_temporal_client = temporal_client- elif temporal_server_config:- log.dev(f"Connecting temporal using provided Temporal server config: {temporal_server_config.description}")- the_temporal_client = await connect_to_temporal_server(- server_config=temporal_server_config,- )- elif temporal_selected_server:- log.dev(f"Connecting temporal using selected Temporal server: {temporal_selected_server}")- the_temporal_client = await connect_to_temporal_selected_server(- selected_server_config=temporal_selected_server,- )- else:- log.dev("Connecting temporal using default Temporal client")- # use automatic default connection- the_temporal_client = await connect_to_temporal()- self.temporal_client = the_temporal_client- log.debug(f"Temporal client connected, session_id={self.session_id}")- return the_temporal_client-- async def get_temporal_client(self, should_auto_connect: bool) -> TemporalClient:- if self.temporal_client is not None:- return self.temporal_client- elif should_auto_connect:- return await self.connect_temporal()- else:- msg = "Temporal client not connected. Enable should_auto_connect or call TemporalManager.connect_temporal() first."- raise RuntimeError(msg)-- def make_top_workflow_id(self, pipeline_run_id: str) -> str:- prefix: str- match runtime_manager.run_mode:- case RunMode.UNIT_TEST:- prefix = "ut-"- case RunMode.NORMAL:- prefix = ""- case RunMode.CI_TEST:- prefix = "ci-"- case RunMode.CODEX_CLOUD:- prefix = "cc-"- case RunMode.CODEX_CLOUD_TEST:- prefix = "cct-"- return f"{prefix}{pipeline_run_id}"---def get_temporal_manager() -> TemporalManager:- return TemporalManager.get_instance()---async def get_temporal_client(should_auto_connect: bool) -> TemporalClient:- return await get_temporal_manager().get_temporal_client(should_auto_connect=should_auto_connect)diff --git a/pipelex/temporal/temporal_registry_models.py b/pipelex/temporal/temporal_registry_models.pydeleted file mode 100644index ecead90a1..000000000--- a/pipelex/temporal/temporal_registry_models.py+++ /dev/null@@ -1,7 +0,0 @@-from typing import ClassVar--from pipelex.system.registries.registry_base import ModelType, RegistryModels---class TemporalRegistryModels(RegistryModels):- GENERIC: ClassVar[list[ModelType]] = []diff --git a/pipelex/temporal/temporal_task_manager.py b/pipelex/temporal/temporal_task_manager.pydeleted file mode 100644index 51597f45a..000000000--- a/pipelex/temporal/temporal_task_manager.py+++ /dev/null@@ -1,275 +0,0 @@-import asyncio--from temporalio import workflow-from temporalio.client import Client as TemporalClient-from temporalio.worker import UnsandboxedWorkflowRunner, Worker, WorkflowRunner-from temporalio.worker.workflow_sandbox import (- SandboxedWorkflowRunner,- SandboxRestrictions,-)-from typing_extensions import override--from pipelex import log-from pipelex.base_exceptions import PipelexError-from pipelex.config import get_config-from pipelex.hub import get_class_registry-from pipelex.system.runtime import WorkerMode, runtime_manager-from pipelex.temporal.config_temporal import WorkerRuntimeProfile, WorkerScope-from pipelex.temporal.exceptions import WorkerProfileConfigError, WorkerScopeConfigError, WorkflowExecutionError-from pipelex.temporal.log_temporal import configure_temporal_logs-from pipelex.temporal.sandbox_manager import sandbox_manager-from pipelex.temporal.task_manager import TaskManager-from pipelex.temporal.temporal_connect import connect_to_temporal-from pipelex.temporal.temporal_manager import TemporalManager-from pipelex.temporal.temporal_registry_models import TemporalRegistryModels-from pipelex.temporal.temporal_tasks import TaskPack, TemporalTasks-from pipelex.temporal.temporal_types import (- ActivityList,- ActivityType,- WorkflowList,- WorkflowType,-)-from pipelex.temporal.test_extras.temporal_registry_test_models import TemporalTestModels-from pipelex.temporal.tprl.namespace_check import check_required_search_attributes---def is_in_temporal_sandbox() -> bool:- """Check if we're in a Temporal workflow sandbox.-- Returns:- bool: True if in a workflow sandbox, False otherwise.- """- return workflow.unsafe.in_sandbox()---class TemporalTaskManager(TaskManager):- def __init__(self):- self.temporal_tasks = self.__class__.create_temporal_tasks()-- @classmethod- def create_temporal_tasks(cls) -> TemporalTasks:- return TemporalTasks()-- def setup(self):- get_class_registry().register_classes(TemporalRegistryModels.get_all_models())- if runtime_manager.is_unit_testing:- log.debug("Registering test models for unit testing")- get_class_registry().register_classes(TemporalTestModels.get_all_models())- configure_temporal_logs()- # TODO: use direct tweaking of settings in Pipelex to apply the sandbox restrictions to loggers etc.- sandbox_manager.set_sandbox_callable(sandbox_callable=is_in_temporal_sandbox)- TemporalManager.setup(session_id=get_config().session_id)- log.info("TemporalTaskManager setup done")-- def teardown(self):- TemporalManager.teardown()- log.info("TemporalTaskManager teardown done")-- @override- def complement_catalog(- self,- extra_catalog: dict[str, TaskPack],- *,- extra_workflows: list[WorkflowType],- extra_activities: list[ActivityType],- ):- self.temporal_tasks.complement_catalog(- extra_catalog=extra_catalog,- extra_workflows=extra_workflows,- extra_activities=extra_activities,- )-- @override- def make_worker(- self,- temporal_client: TemporalClient,- *,- task_queue: str,- is_not_sandboxed: bool = False,- scope: WorkerScope | None = None,- runtime_profile: WorkerRuntimeProfile | None = None,- substitute_activities: dict[ActivityType, ActivityType] | None = None,- test_workflows: WorkflowList | None = None,- test_activities: ActivityList | None = None,- ) -> Worker:- workflows, activities = self.temporal_tasks.workflows_and_activities(- scope=scope,- test_activities=test_activities,- test_workflows=test_workflows,- substitute_activities=substitute_activities,- )- workflow_runner: WorkflowRunner- if is_not_sandboxed:- workflow_runner = UnsandboxedWorkflowRunner()- else:- workflow_runner = SandboxedWorkflowRunner(- restrictions=SandboxRestrictions.default.with_passthrough_modules(- "pydantic",- "pydantic.root_model",- "bson",- "pymongo",- "networkx",- "google.auth",- "google.cloud.logging",- "cogt.oai.azure_openai_config",- "pipelex.tools.log.log",- "pipelex.tools.log.log_config",- "pipelex.tools.log.log_dispatch",- "pipelex.tools.environment",- "citadel.mdb.models",- "citadel.mdb.connector",- "citadel.gcp.gsecret_helpers",- )- )-- # Resolve to the default profile when the caller does not specify one.- # Most internal/test call sites don't care about profile tuning and want- # the default knobs; the worker CLI path always passes an explicit profile.- profile = runtime_profile or self._resolve_runtime_profile_by_name(profile_name=None)-- # Queue-level cluster-wide rate cap is attached to the queue, not the- # profile. Every worker on this queue sends it; the server enforces it.- queue_options = get_config().temporal.queue_options.get(task_queue)- max_task_queue_activities_per_second = queue_options.max_task_queue_activities_per_second if queue_options is not None else None-- return Worker(- temporal_client,- task_queue=task_queue,- workflows=workflows,- activities=activities,- workflow_runner=workflow_runner,- # Make an escaping domain error a terminal workflow failure instead of Temporal's- # default "unhandled exception = workflow-task failure", which retries indefinitely —- # a silent hang (see docs/under-the-hood/error-model.md "Workflow-Level Fail-Safe- # Floor"). PipelexError is the floor: any domain error that slips past the workflow-- # level catch-alls in WfPipeRouter / WfPipeRun still fails terminally (degraded to a- # synthesized report) rather than hanging. WorkflowExecutionError — the child-dispatch- # wrapper raised by those workflows — is listed explicitly for intent. Scoped to- # PipelexError, not Exception: transient Temporal/infra errors keep their task-retry.- workflow_failure_exception_types=[WorkflowExecutionError, PipelexError],- max_cached_workflows=profile.max_cached_workflows,- max_concurrent_workflow_tasks=profile.max_concurrent_workflow_tasks,- max_concurrent_activities=profile.max_concurrent_activities,- max_concurrent_local_activities=profile.max_concurrent_local_activities,- max_concurrent_workflow_task_polls=profile.max_concurrent_workflow_task_polls,- max_concurrent_activity_task_polls=profile.max_concurrent_activity_task_polls,- sticky_queue_schedule_to_start_timeout=profile.sticky_queue_schedule_to_start_timeout,- max_heartbeat_throttle_interval=profile.max_heartbeat_throttle_interval,- default_heartbeat_throttle_interval=profile.default_heartbeat_throttle_interval,- graceful_shutdown_timeout=profile.graceful_shutdown_timeout,- max_activities_per_second=profile.max_activities_per_second,- max_task_queue_activities_per_second=max_task_queue_activities_per_second,- )-- @override- async def run_worker(- self,- *,- is_not_sandboxed: bool,- is_unit_testing: bool,- task_queue: str | None = None,- scope_name: str | None = None,- profile_name: str | None = None,- ):- try:- test_workflows: WorkflowList | None = None- test_activities: ActivityList | None = None- if is_unit_testing:- log.debug(f"is_unit_testing={is_unit_testing} Registering test models")- runtime_manager.set_worker_mode(worker_mode=WorkerMode.UNIT_TEST)- from pipelex.temporal.test_extras.temporal_test_tasks import ( # noqa: PLC0415- TEMPORAL_TEST_ACTIVITIES,- TEMPORAL_TEST_WORKFLOWS,- )-- test_workflows = TEMPORAL_TEST_WORKFLOWS- test_activities = TEMPORAL_TEST_ACTIVITIES- else:- log.debug(f"is_unit_testing={is_unit_testing} Setting worker mode to NORMAL")- runtime_manager.set_worker_mode(worker_mode=WorkerMode.NORMAL)- temporal_client = await connect_to_temporal()- worker_config = get_config().temporal.worker_config- task_queue = task_queue or worker_config.default_task_queue- # Strict check also runs at the worker CLI startup; repeated here- # so programmatic callers of ``run_worker`` (tests, library code)- # also fast-fail on typos rather than polling an idle queue.- get_config().temporal.validate_task_queue_known(task_queue)- # Hard-fail check: when the workflow-start side is configured to- # populate custom search attributes, abort worker boot loudly if- # any are missing on a reachable namespace. The cluster would- # otherwise reject every workflow start that references an- # unregistered attribute with a much less actionable error.- # ``[temporal.search_attributes].enabled = false`` skips both this- # check and the attribute attachment in- # ``build_search_attributes``.- search_attributes_config = get_config().temporal.search_attributes- if search_attributes_config.enabled:- await check_required_search_attributes(- temporal_client=temporal_client,- namespace=temporal_client.namespace,- configured_attributes=search_attributes_config.attributes,- )- scope = self._resolve_scope_by_name(scope_name=scope_name)- runtime_profile = self._resolve_runtime_profile_by_name(profile_name=profile_name)- effective_scope_name = scope_name or get_config().temporal.worker_scopes.default_scope- effective_profile_name = profile_name or get_config().temporal.worker_runtime_profiles.default_profile- log.info(f"Temporal Worker starting: profile='{effective_profile_name}' scope='{effective_scope_name}' task_queue='{task_queue}'")- async with self.make_worker(- temporal_client=temporal_client,- task_queue=task_queue,- is_not_sandboxed=is_not_sandboxed,- scope=scope,- runtime_profile=runtime_profile,- test_workflows=test_workflows,- test_activities=test_activities,- ):- log.info(f"Temporal Worker started for '{task_queue}'")- if is_not_sandboxed:- log.warning("Worker is running without sandbox")- await asyncio.Event().wait()- except asyncio.CancelledError:- log.info("Main coroutine was cancelled.")- finally:- log.info("Shutting down")-- @staticmethod- def _resolve_scope_by_name(scope_name: str | None) -> WorkerScope:- worker_scopes = get_config().temporal.worker_scopes- effective_name = scope_name or worker_scopes.default_scope- if effective_name not in worker_scopes.scopes:- msg = f"Unknown worker scope '{effective_name}' (known: {sorted(worker_scopes.scopes.keys())})"- raise WorkerScopeConfigError(msg)- return worker_scopes.scopes[effective_name]-- @staticmethod- def _resolve_runtime_profile_by_name(profile_name: str | None) -> WorkerRuntimeProfile:- profiles_config = get_config().temporal.worker_runtime_profiles- effective_name = profile_name or profiles_config.default_profile- if effective_name not in profiles_config.profiles:- msg = f"Unknown worker runtime profile '{effective_name}' (known: {sorted(profiles_config.profiles.keys())})"- raise WorkerProfileConfigError(msg)- return profiles_config.profiles[effective_name]-- @override- def task_packs(self) -> list[str]:- return self.temporal_tasks.task_packs()-- @override- def workflows_and_activities(- self,- scope: WorkerScope | None = None,- *,- test_workflows: WorkflowList | None = None,- test_activities: ActivityList | None = None,- substitute_activities: dict[ActivityType, ActivityType] | None = None,- ) -> tuple[list[WorkflowType], list[ActivityType]]:- return self.temporal_tasks.workflows_and_activities(- scope=scope,- test_workflows=test_workflows,- test_activities=test_activities,- substitute_activities=substitute_activities,- )-- @override- def workflows_and_activities_str(self) -> tuple[list[str], list[str]]:- return self.temporal_tasks.workflows_and_activities_str()diff --git a/pipelex/temporal/temporal_tasks.py b/pipelex/temporal/temporal_tasks.pydeleted file mode 100644index 06d0d82d2..000000000--- a/pipelex/temporal/temporal_tasks.py+++ /dev/null@@ -1,182 +0,0 @@-from pipelex import log-from pipelex.temporal.config_temporal import WorkerScope-from pipelex.temporal.exceptions import TemporalConfigError, WorkerScopeConfigError-from pipelex.temporal.temporal_types import ActivityList, ActivityType, WorkflowList, WorkflowType---class TaskPack:- def __init__(self, workflow_list: WorkflowList, activity_list: ActivityList):- self.workflow_list = workflow_list- self.activity_list = activity_list---class TemporalTasks:- def __init__(self):- self.tasks_catalog: dict[str, TaskPack] = {}- self.extra_workflows: list[WorkflowType] = []- self.extra_activities: list[ActivityType] = []-- def complement_catalog(- self,- extra_catalog: dict[str, TaskPack],- *,- extra_workflows: list[WorkflowType],- extra_activities: list[ActivityType],- ):- log.verbose(extra_catalog, title="Complementing catalog with extra_catalog")- for pack_name, task_pack in extra_catalog.items():- if pack_name in self.tasks_catalog:- msg = f"Task pack with the same name '{pack_name}' already exists, no merging or overriding allowed for task packs"- raise RuntimeError(msg)- self.tasks_catalog[pack_name] = task_pack-- log.verbose(extra_workflows, title="Complementing catalog with extra_workflows")- self.extra_workflows = list(set(self.extra_workflows + extra_workflows))- log.verbose(extra_activities, title="Complementing catalog with extra_activities")- self.extra_activities = list(set(self.extra_activities + extra_activities))-- def replace_catalog(- self,- new_catalog: dict[str, TaskPack],- *,- new_workflows: list[WorkflowType],- new_activities: list[ActivityType],- ):- self.tasks_catalog = new_catalog- self.extra_workflows = new_workflows- self.extra_activities = new_activities-- def task_packs(self) -> list[str]:- return list(self.tasks_catalog.keys())-- def _full_catalog(self) -> tuple[dict[str, WorkflowType], dict[str, ActivityType]]:- all_workflows: dict[str, WorkflowType] = {}- all_activities: dict[str, ActivityType] = {}- for pack_name, task_pack in self.tasks_catalog.items():- source = f"task pack '{pack_name}'"- for workflow in task_pack.workflow_list:- self._register_workflow(all_workflows=all_workflows, workflow=workflow, source=source)- for activity in task_pack.activity_list:- self._register_activity(all_activities=all_activities, activity=activity, source=source)- for workflow in self.extra_workflows:- self._register_workflow(all_workflows=all_workflows, workflow=workflow, source="extra_workflows")- for activity in self.extra_activities:- self._register_activity(all_activities=all_activities, activity=activity, source="extra_activities")- return all_workflows, all_activities-- @classmethod- def _register_workflow(cls, all_workflows: dict[str, WorkflowType], *, workflow: WorkflowType, source: str) -> None:- existing = all_workflows.get(workflow.__name__)- if existing is not None and existing is not workflow:- msg = (- f"Workflow name collision: '{workflow.__name__}' is registered as two different callables "- f"(latest source: {source}). Temporal requires a unique name per workflow."- )- raise TemporalConfigError(msg)- all_workflows[workflow.__name__] = workflow-- @classmethod- def _register_activity(cls, all_activities: dict[str, ActivityType], *, activity: ActivityType, source: str) -> None:- existing = all_activities.get(activity.__name__)- if existing is not None and existing is not activity:- msg = (- f"Activity name collision: '{activity.__name__}' is registered as two different callables "- f"(latest source: {source}). Temporal requires a unique name per activity."- )- raise TemporalConfigError(msg)- all_activities[activity.__name__] = activity-- def workflows_and_activities(- self,- scope: WorkerScope | None = None,- *,- test_workflows: WorkflowList | None = None,- test_activities: ActivityList | None = None,- substitute_activities: dict[ActivityType, ActivityType] | None = None,- ) -> tuple[list[WorkflowType], list[ActivityType]]:- all_workflows, all_activities = self._full_catalog()-- the_workflows: set[WorkflowType]- the_activities: set[ActivityType]- if scope is None:- the_workflows = set(all_workflows.values())- the_activities = set(all_activities.values())- else:- the_workflows, the_activities = self._resolve_scope(- scope=scope,- all_workflows=all_workflows,- all_activities=all_activities,- )-- if test_workflows:- the_workflows.update(test_workflows)- if test_activities:- the_activities.update(test_activities)- if substitute_activities:- for old_activity, new_activity in substitute_activities.items():- if old_activity in the_activities:- the_activities.remove(old_activity)- the_activities.add(new_activity)- return list(the_workflows), list(the_activities)-- def _resolve_scope(- self,- scope: WorkerScope,- *,- all_workflows: dict[str, WorkflowType],- all_activities: dict[str, ActivityType],- ) -> tuple[set[WorkflowType], set[ActivityType]]:- known_packs = set(self.tasks_catalog.keys())- unknown_packs = [pack for pack in scope.required_tasks_packs if pack not in known_packs]- if unknown_packs:- msg = f"Unknown task pack(s) in scope: {unknown_packs} (known: {sorted(known_packs)})"- raise WorkerScopeConfigError(msg)-- unknown_required_workflows = [name for name in scope.required_workflows if name not in all_workflows]- if unknown_required_workflows:- msg = f"Unknown workflow(s) in scope.required_workflows: {unknown_required_workflows} (known: {sorted(all_workflows.keys())})"- raise WorkerScopeConfigError(msg)- unknown_required_activities = [name for name in scope.required_activities if name not in all_activities]- if unknown_required_activities:- msg = f"Unknown activity(ies) in scope.required_activities: {unknown_required_activities} (known: {sorted(all_activities.keys())})"- raise WorkerScopeConfigError(msg)- unknown_excluded_workflows = [name for name in scope.excluded_workflows if name not in all_workflows]- if unknown_excluded_workflows:- msg = f"Unknown workflow(s) in scope.excluded_workflows: {unknown_excluded_workflows} (known: {sorted(all_workflows.keys())})"- raise WorkerScopeConfigError(msg)- unknown_excluded_activities = [name for name in scope.excluded_activities if name not in all_activities]- if unknown_excluded_activities:- msg = f"Unknown activity(ies) in scope.excluded_activities: {unknown_excluded_activities} (known: {sorted(all_activities.keys())})"- raise WorkerScopeConfigError(msg)-- the_workflows: set[WorkflowType] = set()- the_activities: set[ActivityType] = set()- for pack_name in scope.required_tasks_packs:- task_pack = self.tasks_catalog[pack_name]- the_workflows.update(task_pack.workflow_list)- the_activities.update(task_pack.activity_list)-- the_workflows.update(all_workflows[name] for name in scope.required_workflows)- the_activities.update(all_activities[name] for name in scope.required_activities)-- the_workflows.difference_update(all_workflows[name] for name in scope.excluded_workflows)- the_activities.difference_update(all_activities[name] for name in scope.excluded_activities)-- if scope.disable_all_workflows:- the_workflows.clear()- if scope.disable_all_activities:- the_activities.clear()-- if not the_workflows and not the_activities:- msg = (- f"Worker scope resolves to an empty set (packs={scope.required_tasks_packs}, "- f"required_workflows={scope.required_workflows}, required_activities={scope.required_activities}, "- f"excluded_workflows={scope.excluded_workflows}, excluded_activities={scope.excluded_activities})"- )- raise WorkerScopeConfigError(msg)-- return the_workflows, the_activities-- def workflows_and_activities_str(self) -> tuple[list[str], list[str]]:- workflows, activities = self.workflows_and_activities(test_workflows=None, test_activities=None)- return [str(w.__name__) for w in workflows], [str(a.__name__) for a in activities]diff --git a/pipelex/temporal/temporal_types.py b/pipelex/temporal/temporal_types.pydeleted file mode 100644index 014c3eb35..000000000--- a/pipelex/temporal/temporal_types.py+++ /dev/null@@ -1,6 +0,0 @@-from typing import Any, Callable--WorkflowType = type[Any]-WorkflowList = list[WorkflowType]-ActivityType = Callable[[Any], Any]-ActivityList = list[ActivityType]diff --git a/pipelex/temporal/temporal_workflow_utils.py b/pipelex/temporal/temporal_workflow_utils.pydeleted file mode 100644index fabe3844e..000000000--- a/pipelex/temporal/temporal_workflow_utils.py+++ /dev/null@@ -1,16 +0,0 @@-from temporalio import workflow-from temporalio.exceptions import TemporalError---def is_in_temporal_workflow() -> bool:- """Check if the current code is running inside a Temporal workflow.-- Uses workflow.info() which raises if not inside a workflow context.- Used by `TemporalPipeRouter` and `TemporalPipeRun` to decide between top-level dispatch- (`execute_workflow`) and child dispatch (`execute_child_workflow`).- """- try:- workflow.info()- return True- except (TemporalError, RuntimeError):- return Falsediff --git a/pipelex/temporal/test_extras/__init__.py b/pipelex/temporal/test_extras/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/pipelex/temporal/test_extras/models_for_tests.py b/pipelex/temporal/test_extras/models_for_tests.pydeleted file mode 100644index 44b559620..000000000--- a/pipelex/temporal/test_extras/models_for_tests.py+++ /dev/null@@ -1,5 +0,0 @@-from pipelex.core.stuffs.structured_content import StructuredContent---class Mock(StructuredContent):- passdiff --git a/pipelex/temporal/test_extras/temporal_registry_test_models.py b/pipelex/temporal/test_extras/temporal_registry_test_models.pydeleted file mode 100644index 1293b78e2..000000000--- a/pipelex/temporal/test_extras/temporal_registry_test_models.py+++ /dev/null@@ -1,48 +0,0 @@-from typing import ClassVar--from pydantic import BaseModel, ConfigDict, Field--from pipelex.system.registries.registry_base import ModelType, RegistryModels-from pipelex.tools.typing.pydantic_utils import empty_list_factory_of---# for testing & examples-class Person(BaseModel):- model_config = ConfigDict(arbitrary_types_allowed=True)- name: str- age: int- job: str---class FixtureLineItem(BaseModel):- """Inner-most leaf type of FixtureInvoice. Designed to exercise type preservation- across the Temporal data-converter boundary for ``act_llm_gen_object`` /- ``act_llm_gen_object_list``. Named ``Fixture*`` (not ``Test*``) so pytest- does not try to collect it as a test class.- """-- product_name: str- quantity: int- unit_price: float---class FixtureCustomer(BaseModel):- name: str- email: str---class FixtureInvoice(BaseModel):- """Test-only nested structure used to validate that ``make_object`` /- ``make_object_list`` preserves nested fields through the cross-process- JSON round-trip (``model_dump(mode="json", serialize_as_any=True)`` →- Temporal data converter → ``model_validate(...)``).- """-- invoice_number: str- customer: FixtureCustomer- line_items: list[FixtureLineItem] = Field(default_factory=empty_list_factory_of(FixtureLineItem))- total_amount: float---class TemporalTestModels(RegistryModels):- TEST_MODELS: ClassVar[list[ModelType]] = [Person, FixtureCustomer, FixtureLineItem, FixtureInvoice]diff --git a/pipelex/temporal/test_extras/temporal_test_tasks.py b/pipelex/temporal/test_extras/temporal_test_tasks.pydeleted file mode 100644index cfed82bfe..000000000--- a/pipelex/temporal/test_extras/temporal_test_tasks.py+++ /dev/null@@ -1,16 +0,0 @@-from pipelex.temporal.temporal_types import ActivityList, WorkflowList-from pipelex.temporal.test_extras.wf_test_child_pipe import WfTestChildPipeLLMObject, WfTestChildPipeLLMText-from pipelex.temporal.test_extras.wf_test_content_generator_child import WfTestContentGeneratorChild-from pipelex.temporal.test_extras.wf_test_content_generator_pdf_page_views import WfTestContentGeneratorPdfPageViews-from pipelex.temporal.test_extras.wf_test_structured_output_cross_process import WfTestStructuredOutputCrossProcess--TEMPORAL_TEST_WORKFLOWS: WorkflowList = [- WfTestContentGeneratorChild,- WfTestContentGeneratorPdfPageViews,- WfTestStructuredOutputCrossProcess,-]-TEMPORAL_TEST_ACTIVITIES: ActivityList = []---PIPELEX_TEMPORAL_TEST_WORKFLOWS: WorkflowList = [WfTestChildPipeLLMText, WfTestChildPipeLLMObject]-PIPELEX_TEMPORAL_TEST_ACTIVITIES: ActivityList = []diff --git a/pipelex/temporal/test_extras/wf_test_child_pipe.py b/pipelex/temporal/test_extras/wf_test_child_pipe.pydeleted file mode 100644index 8343a9d9b..000000000--- a/pipelex/temporal/test_extras/wf_test_child_pipe.py+++ /dev/null@@ -1,87 +0,0 @@-from temporalio import workflow--with workflow.unsafe.imports_passed_through():- from pipelex.core.concepts.native.concept_native import NativeConceptCode- from pipelex.core.pipes.pipe_factory import PipeFactory- from pipelex.core.pipes.pipe_output import PipeOutput # noqa: TC001- from pipelex.hub import get_pipe_library, get_pipe_router- from pipelex.pipe_operators.llm.pipe_llm import PipeLLM- from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint- from pipelex.pipe_run.pipe_job_factory import PipeJobFactory- from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory- from pipelex.pipeline.job_metadata import JobMetadata- from pipelex.temporal.log_temporal import workflow_log- from pipelex.temporal.test_extras.models_for_tests import Mock---class WfTestChildPipeTestCases:- SYSTEM_PROMPT = "You are a pirate, you always talk like a pirate."- USER_PROMPT = "In 3 sentences, tell me about the sea."- USER_TEXT_TRICKY_1 = """-When my son was 7 he was 3ft tall. When he was 8 he was 4ft tall. When he was 9 he was 5ft tall.-How tall do you think he was when he was 12? and at 15?-"""--[email protected](name="wf_test_child_pipe_llm_text")-class WfTestChildPipeLLMText:- @workflow.run- async def run(self) -> str | None:- workflow_log.debug("Workflow start")- pipe_llm_blueprint = PipeLLMBlueprint(- description="LLM test for basic text generation",- output=NativeConceptCode.TEXT,- system_prompt=WfTestChildPipeTestCases.SYSTEM_PROMPT,- prompt=WfTestChildPipeTestCases.USER_PROMPT,- )- pipe = PipeFactory[PipeLLM].make_from_blueprint(- domain_code="adhoc",- pipe_code="adhoc_for_test_child_pipe_llm_text",- blueprint=pipe_llm_blueprint,- )- pipe_library = get_pipe_library()- pipe_library.add_new_pipe(pipe)-- pipe_job = PipeJobFactory.make_pipe_job(- pipe=pipe,- pipe_run_params=PipeRunParamsFactory.make_run_params(),- job_metadata=JobMetadata(- user_id="temporal_test",- pipeline_run_id=workflow.info().workflow_id,- ),- )- pipe_output: PipeOutput = await get_pipe_router().run(pipe_job=pipe_job)- return pipe_output.main_stuff_as_text.text--[email protected](name="wf_test_child_pipe_llm_object")-class WfTestChildPipeLLMObject:- @workflow.run- async def run(self) -> Mock:- workflow_log.debug("Workflow start")- pipe_llm_blueprint = PipeLLMBlueprint(- description="LLM test for object generation",- output="tests.Mock",- prompt=WfTestChildPipeTestCases.USER_TEXT_TRICKY_1,- model_to_structure="llm_to_reason",- )- pipe = PipeFactory[PipeLLM].make_from_blueprint(- domain_code="adhoc",- pipe_code="adhoc_for_test_child_pipe_llm_object",- blueprint=pipe_llm_blueprint,- )- pipe_library = get_pipe_library()- pipe_library.add_new_pipe(pipe)-- pipe_job = PipeJobFactory.make_pipe_job(- pipe=pipe,- pipe_run_params=PipeRunParamsFactory.make_run_params(),- job_metadata=JobMetadata(- user_id="temporal_test",- pipeline_run_id=workflow.info().workflow_id,- ),- )- pipe_output: PipeOutput = await get_pipe_router().run(pipe_job=pipe_job)- llm_generated_object = pipe_output.main_stuff_as(content_type=Mock)- assert isinstance(llm_generated_object, Mock)- return llm_generated_objectdiff --git a/pipelex/temporal/test_extras/wf_test_content_generator_child.py b/pipelex/temporal/test_extras/wf_test_content_generator_child.pydeleted file mode 100644index a3987a3e8..000000000--- a/pipelex/temporal/test_extras/wf_test_content_generator_child.py+++ /dev/null@@ -1,118 +0,0 @@-from temporalio import workflow--with workflow.unsafe.imports_passed_through():- from pipelex import pretty_print- from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams- from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol # noqa: TC001- from pipelex.cogt.extract.extract_input import ExtractInput- from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams- from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt- from pipelex.cogt.llm.llm_prompt import LLMPrompt- from pipelex.hub import get_model_deck- from pipelex.pipe_run.pipe_run_mode import PipeRunMode- from pipelex.pipeline.job_metadata import JobMetadata- from pipelex.temporal.log_temporal import workflow_log- from pipelex.temporal.test_extras.temporal_registry_test_models import Person- from pipelex.temporal.tprl_content_generation.content_generator_in_workflow_factory import ContentGeneratorInWorkflowFactory- from tests.integration.pipelex.temporal.test_data import PipeTestCases---USER_TEXT_FOR_BASE = """-Write a detailed description of a woman's clothing in the style of a 19th-century novel.-Keep it short: 3 sentences max-"""--USER_TEXT_FOR_SINGLE_PERSON = "name: John, age: 30, job: bank teller"-MULTIPLE_USER_TEXTS_FOR_PEOPLE = [- "name: Bob, age: 25, job: banker",- "name: Maria, age: 35, job: consultant",- "name: SLartiblfastikur, age: 30, job: fizzy buzzer",- "name: Alice, age: 40, job: developer",- "name: Tom, age: 45, job: TV presenter",- "name: Jerry, age: 50, job: nurse",-]-USER_TEXTS_FOR_PEOPLE_STR = "\n".join(MULTIPLE_USER_TEXTS_FOR_PEOPLE)--USER_TEXT_FOR_HAIKU = """-Write a haiku about the meaning of life-"""--POSITIVE_TEXT_FOR_IMAGE = "A group of dogs with sunglasses playing beer pong with drinks and snacks"--[email protected](name="wf_test_content_generator_child")-class WfTestContentGeneratorChild:- @workflow.run- async def run(self, is_dry_run: bool = False):- workflow_log.debug("Workflow start")- # One generator for both arms: under is_dry_run the activities are still dispatched and- # the cogt leaf mocks inside them (run_mode=DRY rides cogt_run_params — Part B).- content_generator: ContentGeneratorProtocol = ContentGeneratorInWorkflowFactory.make_content_generator_in_workflow()-- job_metadata = JobMetadata(- user_id="temporal-test",- pipeline_run_id=workflow.info().workflow_id,- )- run_mode = PipeRunMode.DRY if is_dry_run else PipeRunMode.LIVE- cogt_run_params = CogtRunParams(run_mode=run_mode)-- llm_setting_for_text = get_model_deck().get_llm_setting(llm_choice="$testing-text")- crafted_text = await content_generator.make_llm_text(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- llm_setting_main=llm_setting_for_text,- llm_prompt_for_text=LLMPrompt(user_text=USER_TEXT_FOR_BASE),- )- pretty_print(crafted_text, title="make_llm_text")-- llm_setting_for_object = get_model_deck().get_llm_setting(llm_choice="$testing-structured")- crafted_object_direct = await content_generator.make_object(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- object_class=Person,- llm_setting_for_object=llm_setting_for_object,- llm_prompt_for_object=LLMPrompt(user_text=USER_TEXT_FOR_SINGLE_PERSON),- )- pretty_print(crafted_object_direct, title="make_object")-- crafted_object_list_direct = await content_generator.make_object_list(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- object_class=Person,- llm_setting_for_object_list=llm_setting_for_object,- llm_prompt_for_object_list=LLMPrompt(user_text=USER_TEXTS_FOR_PEOPLE_STR),- )- pretty_print(crafted_object_list_direct, title="make_object_list")-- crafted_image = await content_generator.make_single_image(- img_gen_handle="gpt-image-1-mini",- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- img_gen_prompt=ImgGenPrompt(positive_text=POSITIVE_TEXT_FOR_IMAGE),- )- pretty_print(crafted_image, title="make_single_image")-- context = {- "the_answer": "elementary, my dear Watson",- }- jinja2_text = await content_generator.make_templated_text(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- context=context,- template="♦️♦️ {{ the_answer }} ♦️♦️",- )- pretty_print(jinja2_text, title="templated_text")-- page_contents = await content_generator.make_extract_pages(- extract_input=ExtractInput(- image_uri=PipeTestCases.IMG_EXPENSE_REPORT_1,- ),- cogt_run_params=cogt_run_params,- extract_handle="azure-document-intelligence",- job_metadata=job_metadata,- extract_job_params=ExtractJobParams.make_default_extract_job_params(),- extract_job_config=ExtractJobConfig(),- )- pretty_print(page_contents, title="make_extract_pages")-- workflow_log.debug("Workflow complete")diff --git a/pipelex/temporal/test_extras/wf_test_content_generator_pdf_page_views.py b/pipelex/temporal/test_extras/wf_test_content_generator_pdf_page_views.pydeleted file mode 100644index dd8aa33b3..000000000--- a/pipelex/temporal/test_extras/wf_test_content_generator_pdf_page_views.py+++ /dev/null@@ -1,70 +0,0 @@-from temporalio import workflow--with workflow.unsafe.imports_passed_through():- from pipelex import pretty_print- from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams- from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol # noqa: TC001- from pipelex.cogt.extract.extract_input import ExtractInput- from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams- from pipelex.pipe_run.pipe_run_mode import PipeRunMode- from pipelex.pipeline.job_metadata import JobMetadata- from pipelex.temporal.exceptions import ContentGenerationError- from pipelex.temporal.log_temporal import workflow_log- from pipelex.temporal.tprl_content_generation.content_generator_in_workflow_factory import ContentGeneratorInWorkflowFactory- from tests.integration.pipelex.temporal.test_data import PipeTestCases--[email protected](name="wf_test_content_generator_pdf_page_views")-class WfTestContentGeneratorPdfPageViews:- """Exercise the ``document_uri`` + ``should_include_page_views=True`` branch of- ``ContentGeneratorInWorkflow.make_extract_pages`` end-to-end.-- Validates the wiring that unit tests can only mock: a real PDF is dispatched to- ``act_extract_gen_extract_pages`` for OCR AND to ``act_render_page_views`` for- pypdfium2 rendering, then the in-workflow attachment loop pairs each rendered- page image with its corresponding ``PageContent.page_view``.- """-- @workflow.run- async def run(self, is_dry_run: bool = False) -> None:- workflow_log.debug("Workflow start")- # One generator for both arms: under is_dry_run the activities are still dispatched and- # the cogt leaf mocks inside them (run_mode=DRY rides cogt_run_params — Part B).- content_generator: ContentGeneratorProtocol = ContentGeneratorInWorkflowFactory.make_content_generator_in_workflow()-- job_metadata = JobMetadata(- user_id="temporal-test",- pipeline_run_id=workflow.info().workflow_id,- )- run_mode = PipeRunMode.DRY if is_dry_run else PipeRunMode.LIVE- cogt_run_params = CogtRunParams(run_mode=run_mode)-- page_contents = await content_generator.make_extract_pages(- extract_input=ExtractInput(- document_uri=PipeTestCases.JOB_OFFER_PDF_LOCAL,- ),- cogt_run_params=cogt_run_params,- extract_handle="azure-document-intelligence",- job_metadata=job_metadata,- extract_job_params=ExtractJobParams(- max_nb_images=0,- image_min_size=None,- should_caption_images=False,- should_include_page_views=True,- page_views_dpi=72,- ),- extract_job_config=ExtractJobConfig(),- )- pretty_print(page_contents, title="make_extract_pages with page_views (document_uri)")-- if not page_contents:- msg = "make_extract_pages returned no pages"- raise ContentGenerationError(msg)- for index_page, page_content in enumerate(page_contents):- if page_content.page_view is None:- msg = (- f"Page {index_page} has no page_view; the document_uri + should_include_page_views branch did not attach the rendered page image"- )- raise ContentGenerationError(msg)-- workflow_log.debug("Workflow complete")diff --git a/pipelex/temporal/test_extras/wf_test_structured_output_cross_process.py b/pipelex/temporal/test_extras/wf_test_structured_output_cross_process.pydeleted file mode 100644index b805e2c47..000000000--- a/pipelex/temporal/test_extras/wf_test_structured_output_cross_process.py+++ /dev/null@@ -1,116 +0,0 @@-from temporalio import workflow--# All imports below run inside ``workflow.unsafe.imports_passed_through()``: they-# must stay at runtime (not under ``TYPE_CHECKING``) so the workflow body can-# reference them when Temporal replays history. The ``noqa: TC001`` on the-# protocol is therefore deliberate — moving it under ``TYPE_CHECKING`` would-# break replay because the symbol is type-annotated on a runtime variable.-with workflow.unsafe.imports_passed_through():- from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams- from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol # noqa: TC001- from pipelex.cogt.llm.llm_prompt import LLMPrompt- from pipelex.cogt.llm.llm_setting import LLMSetting- from pipelex.pipe_run.pipe_run_mode import PipeRunMode- from pipelex.pipeline.job_metadata import JobMetadata- from pipelex.temporal.exceptions import ContentGenerationError- from pipelex.temporal.log_temporal import workflow_log- from pipelex.temporal.test_extras.temporal_registry_test_models import FixtureCustomer, FixtureInvoice, FixtureLineItem- from pipelex.temporal.tprl_content_generation.content_generator_in_workflow_factory import ContentGeneratorInWorkflowFactory--[email protected](name="wf_test_structured_output_cross_process")-class WfTestStructuredOutputCrossProcess:- """Exercise ``ContentGeneratorInWorkflow.make_object`` / ``make_object_list``- end-to-end across the Temporal data converter, with a nested- ``FixtureInvoice`` structure (``FixtureCustomer`` + ``list[FixtureLineItem]``).-- Pair with substitute activities (``act_llm_gen_object`` /- ``act_llm_gen_object_list``) that return canonical ``FixtureInvoice``- instances so the assertion checks the serialization round-trip rather- than LLM output. The single-vs-list branch is selected by ``is_list``.- """-- @workflow.run- async def run(self, is_list: bool) -> None:- workflow_log.debug("Workflow start")- content_generator: ContentGeneratorProtocol = ContentGeneratorInWorkflowFactory.make_content_generator_in_workflow()-- job_metadata = JobMetadata(user_id="temporal-test", pipeline_run_id=workflow.info().workflow_id)- cogt_run_params = CogtRunParams(run_mode=PipeRunMode.LIVE)- llm_setting = LLMSetting(model="$testing-structured", temperature=0.0)- prompt = LLMPrompt(user_text="round-trip nested structured output")-- if is_list:- invoices = await content_generator.make_object_list(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- object_class=FixtureInvoice,- llm_setting_for_object_list=llm_setting,- llm_prompt_for_object_list=prompt,- )- _assert_invoice_list_round_trip(invoices)- else:- invoice = await content_generator.make_object(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- object_class=FixtureInvoice,- llm_setting_for_object=llm_setting,- llm_prompt_for_object=prompt,- )- _assert_invoice_round_trip(invoice)-- workflow_log.debug("Workflow complete")---def _assert_invoice_round_trip(invoice: FixtureInvoice) -> None:- """Verify the nested fields survived the cross-process serialization.-- The substitute activity returns canonical fixtures (see test file); these- assertions pin the round-trip contract: nested ``FixtureCustomer`` keeps- name and email, the ``line_items`` list preserves order and field types,- and numeric fields stay typed (``int`` for quantity, ``float`` for- unit_price). A regression in ``ContentGeneratorInWorkflow.make_object`` or- in the Temporal data converter would surface here as an ``AssertionError``- rather than a silent type drop.- """- # Runtime isinstance guards: while pyright sees the declared types as- # sufficient, the value crossed a Temporal data converter boundary —- # if the round-trip silently produced a generic dict or a wrong subclass,- # we want a clear error here, not a downstream attribute error.- if not isinstance(invoice, FixtureInvoice): # pyright: ignore[reportUnnecessaryIsInstance]- msg = f"Expected FixtureInvoice, got {type(invoice).__name__}"- raise ContentGenerationError(msg)- if not isinstance(invoice.customer, FixtureCustomer): # pyright: ignore[reportUnnecessaryIsInstance]- msg = f"Expected nested FixtureCustomer, got {type(invoice.customer).__name__}"- raise ContentGenerationError(msg)- if invoice.invoice_number != "INV-RT-001":- msg = f"Unexpected invoice_number after round-trip: {invoice.invoice_number!r}"- raise ContentGenerationError(msg)- if invoice.customer.name != "Alice Chen" or invoice.customer.email != "[email protected]":- msg = f"Customer fields not preserved: {invoice.customer!r}"- raise ContentGenerationError(msg)- if len(invoice.line_items) != 2:- msg = f"Expected 2 line items, got {len(invoice.line_items)}"- raise ContentGenerationError(msg)- for line_item in invoice.line_items:- if not isinstance(line_item, FixtureLineItem): # pyright: ignore[reportUnnecessaryIsInstance]- msg = f"Expected FixtureLineItem in list, got {type(line_item).__name__}"- raise ContentGenerationError(msg)---def _assert_invoice_list_round_trip(invoices: list[FixtureInvoice]) -> None:- if len(invoices) != 2:- msg = f"Expected 2 invoices, got {len(invoices)}"- raise ContentGenerationError(msg)- expected_numbers = {"INV-RT-A", "INV-RT-B"}- for invoice in invoices:- if not isinstance(invoice, FixtureInvoice): # pyright: ignore[reportUnnecessaryIsInstance]- msg = f"Expected FixtureInvoice, got {type(invoice).__name__}"- raise ContentGenerationError(msg)- if not invoice.line_items or not isinstance(invoice.line_items[0], FixtureLineItem): # pyright: ignore[reportUnnecessaryIsInstance]- msg = f"Invoice {invoice.invoice_number!r} missing nested FixtureLineItem"- raise ContentGenerationError(msg)- seen_numbers = {invoice.invoice_number for invoice in invoices}- if seen_numbers != expected_numbers:- msg = f"Invoice numbers not preserved: got {seen_numbers}, expected {expected_numbers}"- raise ContentGenerationError(msg)diff --git a/pipelex/temporal/test_helpers/__init__.py b/pipelex/temporal/test_helpers/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/pipelex/temporal/test_helpers/temporal_pytest_plugins.py b/pipelex/temporal/test_helpers/temporal_pytest_plugins.pydeleted file mode 100644index 4d37f67fd..000000000--- a/pipelex/temporal/test_helpers/temporal_pytest_plugins.py+++ /dev/null@@ -1,17 +0,0 @@-from pytest import Parser--from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment-from pipelex.types import StrEnum---class TemporalPytestOption(StrEnum):- WORKER_ENVIRONMENT = "--worker"---def pytest_addoption(parser: Parser):- parser.addoption(- TemporalPytestOption.WORKER_ENVIRONMENT,- type=TemporalWorkerEnvironment,- default=TemporalWorkerEnvironment.INTERNAL,- help="Which temporal worker environment to use ('internal', 'external')",- )diff --git a/pipelex/temporal/tprl/__init__.py b/pipelex/temporal/tprl/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/pipelex/temporal/tprl/activity_error_boundary.py b/pipelex/temporal/tprl/activity_error_boundary.pydeleted file mode 100644index 66a1b11ac..000000000--- a/pipelex/temporal/tprl/activity_error_boundary.py+++ /dev/null@@ -1,54 +0,0 @@-"""Activity-side half of the Temporal error bridge.--A Temporal activity raises Pipelex exceptions; the workflow side re-wraps them-with ``TemporalError.from_app_error``. In between, Temporal's default failure-converter would auto-wrap a raw ``PipelexError`` without packing our structured-``ErrorReport`` into ``ApplicationError.details`` and without deriving-``non_retryable`` from the error's ``InferenceErrorCategory``.--``convert_pipelex_errors`` closes that gap: it converts a ``PipelexError`` to a-``TemporalError`` at the activity boundary so the category-aware retry decision-and the structured report survive into workflow code.-"""--import functools-from collections.abc import Awaitable, Callable-from typing import ParamSpec, TypeVar--from pipelex.base_exceptions import PipelexError-from pipelex.temporal.tprl.temporal_error import TemporalError--_ActivityParams = ParamSpec("_ActivityParams")-_ActivityReturn = TypeVar("_ActivityReturn")---def convert_pipelex_errors(- func: Callable[_ActivityParams, Awaitable[_ActivityReturn]],-) -> Callable[_ActivityParams, Awaitable[_ActivityReturn]]:- """Convert any ``PipelexError`` raised by a Temporal activity into a ``TemporalError``.-- ``TemporalError.from_message_exception`` derives ``non_retryable`` from the- error's ``InferenceErrorCategory`` and packs ``to_error_report()`` into- ``ApplicationError.details`` — the structured data the workflow-side- ``TemporalError.from_app_error`` then recovers.-- Apply this decorator *below* ``@activity.defn`` so Temporal registers the- wrapped function; ``functools.wraps`` preserves the ``__name__`` and- annotations Temporal inspects. Only ``PipelexError`` is caught — a non-Pipelex- exception propagates untouched and Temporal's default converter handles it.-- Args:- func: The async activity function to wrap.-- Returns:- The wrapped async activity function.- """-- @functools.wraps(func)- async def wrapper(*args: _ActivityParams.args, **kwargs: _ActivityParams.kwargs) -> _ActivityReturn:- try:- return await func(*args, **kwargs)- except PipelexError as exc:- raise TemporalError.from_message_exception(exc=exc) from exc-- return wrapperdiff --git a/pipelex/temporal/tprl/conditional_worker.py b/pipelex/temporal/tprl/conditional_worker.pydeleted file mode 100644index 91a241880..000000000--- a/pipelex/temporal/tprl/conditional_worker.py+++ /dev/null@@ -1,63 +0,0 @@-from functools import wraps-from typing import Any, Awaitable, Callable, TypeVar, cast--import shortuuid--from pipelex import log-from pipelex.config import get_config-from pipelex.pipe_run.exceptions import AsyncExecutionNotEnabledError-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment-from pipelex.temporal.temporal_workflow_utils import is_in_temporal_workflow-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor, WorkflowInput, WorkflowOutput--FuncExecuteWorkflow = TypeVar("FuncExecuteWorkflow", bound=Callable[..., Awaitable[Any]])---def with_conditional_worker(execute_workflow: FuncExecuteWorkflow) -> FuncExecuteWorkflow:- """Decorate a ``WorkflowExecutor`` dispatch method with the per-environment- worker-bootstrap behaviour, gated by the deployment-level async-enabled- guard.-- The guard fires here (before the ``match``) rather than inside the wrapped- method body so it precedes the ``INTERNAL`` branch's ``temporal_client()``- call: if the check ran inside the body it would be bypassed on the- ``INTERNAL`` path, leaving only ``temporal_client()``'s lower-level check- as the failsafe and silently inverting the facade docstring contract.-- The guard is skipped when the wrapped method is invoked from inside a- Temporal workflow (the child-dispatch path through- ``TemporalPipeRouter._run_pipe_job``): being inside a workflow proves the- backend is already running, and ``get_config()`` is unsafe to read from- the Temporal sandbox.- """-- @wraps(execute_workflow)- async def wrapper(- self: WorkflowExecutor[WorkflowInput, WorkflowOutput],- *args: Any,- **kwargs: Any,- ) -> Any:- if not is_in_temporal_workflow() and not get_config().temporal.is_enabled:- raise AsyncExecutionNotEnabledError.with_default_message()- log.debug(f"worker_environment: {self.worker_environment}")- match self.worker_environment:- case TemporalWorkerEnvironment.INTERNAL:- # add some random to task_queue to avoid this worker to take over other (possibly failed) tasks scheduled by preceding tests- original_task_queue = self.task_queue- self.task_queue = f"{self.task_queue}-{shortuuid.uuid()[:5]}"-- temporal_client = await self.temporal_client()- try:- async with get_task_manager().make_worker(- temporal_client=temporal_client,- task_queue=self.task_queue,- is_not_sandboxed=True,- ):- return await execute_workflow(self, *args, **kwargs)- finally:- self.task_queue = original_task_queue- case TemporalWorkerEnvironment.EXTERNAL:- return await execute_workflow(self, *args, **kwargs)-- return cast("FuncExecuteWorkflow", wrapper)diff --git a/pipelex/temporal/tprl/namespace_check.py b/pipelex/temporal/tprl/namespace_check.pydeleted file mode 100644index 551dfcbfe..000000000--- a/pipelex/temporal/tprl/namespace_check.py+++ /dev/null@@ -1,218 +0,0 @@-"""Worker-startup hard-fail check for the configured custom search attributes.--The five built-in attributes (``PipeCode``, ``PipelineRunId``, ``SessionId``,-``UserId``, ``DomainCode``) are listed in-``pipelex.temporal.config_temporal.BUILTIN_SEARCH_ATTRIBUTES``. The runtime-check operates on the configured subset declared in-``[temporal.search_attributes].attributes``, not on the full built-in set.--Two entry points:--- ``check_required_search_attributes`` — hard-fail audit at worker boot. Raises- ``SearchAttributeRegistrationError`` (subclass of ``TemporalConfigError``)- when any *configured* attribute is missing on a reachable namespace; this- crashes worker boot loudly with the exact registration commands. ``RPCError``- from the cluster-metadata call stays a soft fail (warn and continue) — the- namespace was unreachable, not misconfigured. Anything else propagates.-- ``ensure_required_search_attributes_registered`` — auto-register the missing- attributes. Used by test infrastructure to set up the in-process Temporal- server before any tests run, and by the ``pipelex setup-temporal-namespace``- CLI command. Catches ``RPCError(PERMISSION_DENIED)`` on either RPC- (``ListSearchAttributes`` or ``AddSearchAttributes``) and returns a- structured ``RegistrationFailure`` instead of raising so the CLI can format- a fallback runbook for operators whose API key lacks namespace-admin- permissions. Returns the tuple of newly-registered names (possibly empty,- for the idempotent "everything already present" no-op) on success so callers- can report the actual delta instead of the configured-set size.-"""--from collections.abc import Sequence-from typing import Final--from pydantic import BaseModel, ConfigDict-from temporalio.api.enums.v1 import IndexedValueType-from temporalio.api.operatorservice.v1 import (- AddSearchAttributesRequest,- ListSearchAttributesRequest,-)-from temporalio.client import Client as TemporalClient-from temporalio.service import RPCError, RPCStatusCode--from pipelex import log-from pipelex.temporal.exceptions import SearchAttributeRegistrationError--# The Pipelex CLI invocation operators run when their config already points at-# the right server. ``--dry-run`` prints the raw ``temporal`` CLI fallback-# instead of executing.-PIPELEX_SETUP_CLI_COMMAND: Final[str] = "pipelex setup-temporal-namespace"---class RegistrationFailure(BaseModel):- """Structured permission-denied outcome from- ``ensure_required_search_attributes_registered``. The CLI command formats- this into the fallback runbook so operators whose worker API key lacks- ``OperatorService.AddSearchAttributes`` permission know which raw- ``temporal`` / ``tcld`` invocations to run instead.-- Modeled (not a ``NamedTuple``) so it stays type-distinct from the- ``tuple[str, ...]`` success arm of the- ``RegistrationFailure | tuple[str, ...]`` return — callers discriminate via- ``isinstance(x, RegistrationFailure)`` and a plain success tuple must never- satisfy that check.- """-- model_config = ConfigDict(frozen=True)-- namespace: str- missing: tuple[str, ...]- rpc_error_message: str---def format_temporal_cli_command(missing: Sequence[str], *, namespace: str) -> str:- """Format the exact ``temporal operator search-attribute create`` invocation- so an operator with raw ``temporal`` CLI access can copy-paste it.- """- parts = [- "temporal operator search-attribute create",- f" --namespace {namespace}",- ]- for name in missing:- parts.append(f" --name {name} --type Keyword")- return " \\\n".join(parts)---def format_tcld_cli_command(missing: Sequence[str], *, namespace: str) -> str:- """Format the Temporal Cloud ``tcld`` invocation a namespace admin runs- when the worker API key lacks ``OperatorService.AddSearchAttributes``- permission. Used both as a hint in the dry-run output and as part of the- permission-denied fallback runbook.- """- attribute_flags = " ".join(f"--search-attribute {name}=Keyword" for name in missing)- return f"tcld namespace search-attributes add --namespace {namespace} {attribute_flags}"---async def check_required_search_attributes(- temporal_client: TemporalClient,- *,- namespace: str,- configured_attributes: Sequence[str],-) -> None:- """Hard-fail at worker boot if any configured custom search attribute is- missing on a reachable namespace.-- Failure model:-- - All configured attributes present → silent.- - Some missing on a reachable namespace → raise- ``SearchAttributeRegistrationError`` with both the ``pipelex- setup-temporal-namespace`` invocation and the equivalent raw ``temporal- operator search-attribute create`` command in the message.- - ``RPCError`` from the operator service (namespace unreachable, transient- cluster issue) → log a warning, continue. Fast-failing here would block- every worker boot during a control-plane outage; the cluster will reject- workflow starts that reference unregistered attributes anyway.- - Any other exception → propagates and crashes worker boot.-- Args:- temporal_client: The client used to talk to the operator service.- namespace: Namespace to inspect; usually ``temporal_client.namespace``.- configured_attributes: The subset of attribute names the worker is- configured to populate. Pass an empty sequence to skip the check- entirely (the caller normally does this when- ``[temporal.search_attributes].enabled = false``).- """- if not configured_attributes:- return- try:- response = await temporal_client.operator_service.list_search_attributes(- ListSearchAttributesRequest(namespace=namespace),- )- except RPCError as exc:- log.warning(- f"Temporal search-attribute check could not reach namespace '{namespace}' "- f"(RPCError: {exc}). Worker boot continues; workflow starts will fail at "- f"dispatch time if the configured attributes are not registered."- )- return-- present = set(response.custom_attributes.keys())- missing = tuple(name for name in configured_attributes if name not in present)- if not missing:- return-- temporal_cli_command = format_temporal_cli_command(missing, namespace=namespace)- msg = (- f"Temporal namespace '{namespace}' is missing required custom search attributes: "- f"{list(missing)}. Workflow starts that reference these attributes are rejected "- f"by the cluster, so worker boot is aborted. Register them with either:\n\n"- f" {PIPELEX_SETUP_CLI_COMMAND}\n\n"- f"or directly via the Temporal CLI:\n\n"- f"{temporal_cli_command}\n"- )- raise SearchAttributeRegistrationError(msg)---async def ensure_required_search_attributes_registered(- temporal_client: TemporalClient,- *,- namespace: str,- configured_attributes: Sequence[str],-) -> RegistrationFailure | tuple[str, ...]:- """Register any missing configured search attributes as Keyword.-- Idempotent: calls ``ListSearchAttributes`` first and only adds the ones- that are absent. Used by test infrastructure to set up the in-process- Temporal server and by the ``pipelex setup-temporal-namespace`` CLI- command.-- Returns the tuple of newly-registered attribute names on success — an- empty tuple in the idempotent "everything already registered" no-op case,- otherwise the names that were just added. Returns a ``RegistrationFailure``- when the namespace is reachable but the client's API key lacks read- (``ListSearchAttributes``) or write (``AddSearchAttributes``) permission;- the CLI command formats it into the fallback runbook. Other ``RPCError``- codes (``UNAVAILABLE``, ``NOT_FOUND``) propagate.-- Args:- temporal_client: The client used to talk to the operator service.- namespace: Namespace to register against.- configured_attributes: Subset of attribute names to ensure are- registered. Pass ``BUILTIN_SEARCH_ATTRIBUTES`` for the full set.- """- if not configured_attributes:- return ()- try:- response = await temporal_client.operator_service.list_search_attributes(- ListSearchAttributesRequest(namespace=namespace),- )- except RPCError as list_exc:- if list_exc.status == RPCStatusCode.PERMISSION_DENIED:- # Without read access we cannot compute the actual missing set, so- # surface the full configured set as the operator's work item.- return RegistrationFailure(- namespace=namespace,- missing=tuple(configured_attributes),- rpc_error_message=str(list_exc),- )- raise- present = set(response.custom_attributes.keys())- missing = tuple(name for name in configured_attributes if name not in present)- if not missing:- return ()-- try:- await temporal_client.operator_service.add_search_attributes(- AddSearchAttributesRequest(- namespace=namespace,- search_attributes=dict.fromkeys(missing, IndexedValueType.INDEXED_VALUE_TYPE_KEYWORD),- ),- )- except RPCError as add_exc:- if add_exc.status == RPCStatusCode.PERMISSION_DENIED:- return RegistrationFailure(- namespace=namespace,- missing=missing,- rpc_error_message=str(add_exc),- )- raise- return missingdiff --git a/pipelex/temporal/tprl/observability.py b/pipelex/temporal/tprl/observability.pydeleted file mode 100644index 9aeb9a42f..000000000--- a/pipelex/temporal/tprl/observability.py+++ /dev/null@@ -1,185 +0,0 @@-"""Helpers that build the Temporal observability surface (search attributes,-static summary/details, per-activity summary) from Pipelex identity (`PipeJob`,-`JobMetadata`, `PipeAbstract`).--Centralizing the formatting policy keeps the call sites — in-``content_generator_in_workflow.py``, ``temporal_pipe_run.py``,-``temporal_pipe_router.py``, ``wf_pipe_run.py`` — thin and trivially-unit-testable. Length caps follow Temporal's documented limits-(200-byte summary, 20 KB details).-"""--from collections.abc import Mapping-from typing import Final--from temporalio.common import SearchAttributeKey, SearchAttributePair, TypedSearchAttributes--from pipelex.config import get_config-from pipelex.core.pipes.pipe_abstract import PipeAbstract-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.temporal_manager import get_temporal_manager--_MAX_SUMMARY_BYTES: Final[int] = 200-_MAX_DETAILS_BYTES: Final[int] = 20 * 1024-_ELLIPSIS: Final[str] = "…"--# Typed search-attribute keys. Defined once at module level so call sites and-# tests can share the exact ``SearchAttributeKey`` instances (key equality is-# by name + type, so identity is not required, but keeping a single instance-# avoids accidental drift between the builder and any future readers).-PIPE_CODE_KEY: Final[SearchAttributeKey[str]] = SearchAttributeKey.for_keyword("PipeCode")-PIPELINE_RUN_ID_KEY: Final[SearchAttributeKey[str]] = SearchAttributeKey.for_keyword("PipelineRunId")-SESSION_ID_KEY: Final[SearchAttributeKey[str]] = SearchAttributeKey.for_keyword("SessionId")-USER_ID_KEY: Final[SearchAttributeKey[str]] = SearchAttributeKey.for_keyword("UserId")-DOMAIN_CODE_KEY: Final[SearchAttributeKey[str]] = SearchAttributeKey.for_keyword("DomainCode")---def _truncate_utf8(text: str, *, max_bytes: int) -> str:- """Return ``text`` truncated so its UTF-8 encoding fits in ``max_bytes``.-- A trailing ellipsis is appended when truncation occurs. The slice is decoded- with ``errors="ignore"`` to drop any partial multi-byte sequence at the cut.- """- encoded = text.encode("utf-8")- if len(encoded) <= max_bytes:- return text- ellipsis_bytes = len(_ELLIPSIS.encode("utf-8"))- safe = encoded[: max_bytes - ellipsis_bytes].decode("utf-8", errors="ignore")- return f"{safe}{_ELLIPSIS}"---_LIBRARY_CRATE_ID_LEN: Final[int] = 12---_KEY_BY_NAME: Final[dict[str, SearchAttributeKey[str]]] = {- "PipeCode": PIPE_CODE_KEY,- "PipelineRunId": PIPELINE_RUN_ID_KEY,- "SessionId": SESSION_ID_KEY,- "UserId": USER_ID_KEY,- "DomainCode": DOMAIN_CODE_KEY,-}---def stamp_submitter_session_id(pipe_job: PipeJob) -> PipeJob:- """Stamp the current ``TemporalManager.session_id`` onto ``pipe_job.job_metadata``- when it is not already set. Called at every top-level Temporal dispatch- boundary (``TemporalPipeRun.run``, ``TemporalPipeRun.start``,- ``TemporalPipeRouter._run_pipe_job`` top-level branch) so the value flows- into child workflows through the workflow input.-- Why this matters: ``build_search_attributes`` reads ``session_id`` off- ``pipe_job.job_metadata``. Inside workflow code (child-workflow starts in- ``WfPipeRun`` and ``TemporalPipeRouter``) that read must be deterministic- — Temporal verifies replayed ``StartChildWorkflowExecution`` commands- against the recorded ones, and ``TemporalManager.session_id`` differs- across worker processes. Reading it once at the submitter boundary and- threading it through the workflow input is the deterministic path.- """- if pipe_job.job_metadata.session_id is not None:- return pipe_job- stamped_metadata = pipe_job.job_metadata.model_copy(update={"session_id": get_temporal_manager().session_id})- return pipe_job.model_copy(update={"job_metadata": stamped_metadata})---def build_search_attributes(pipe_job: PipeJob) -> TypedSearchAttributes:- """Build the typed search attributes for a workflow start, filtered by the- configured subset.-- Returns an empty ``TypedSearchAttributes`` when- ``[temporal.search_attributes].enabled = false`` so workflow starts attach- no custom attributes (the dashboard view degrades to- WorkflowType / WorkflowId / StartTime). Otherwise emits the subset of pairs- declared in ``[temporal.search_attributes].attributes``.-- Used identically at top-level dispatch (submitter side) and at child- dispatch (inside a workflow): the child's ``pipe_job`` already carries the- inherited ``PipelineRunId`` / ``UserId`` / ``SessionId`` from its parent,- and ``PipeCode`` / ``DomainCode`` correctly reflect the child's own pipe.- Reading every field off ``pipe_job`` (the workflow input) keeps the helper- a pure function — child workflow start commands stay byte-equal across- replays even when the worker process restarts with a fresh- ``TemporalManager.session_id``.- """- config = get_config().temporal.search_attributes- if not config.enabled:- return TypedSearchAttributes([])- enabled_names = set(config.attributes)- # ``session_id`` is stamped at the submitter dispatch boundary; if a caller- # somehow forgot to stamp it, fall back to the empty string so we emit a- # well-formed (if uninformative) attribute rather than ``None`` — Keyword- # search attributes don't accept ``None``.- session_id = pipe_job.job_metadata.session_id or ""- value_by_name: dict[str, str] = {- "PipeCode": pipe_job.pipe.code,- "PipelineRunId": pipe_job.job_metadata.pipeline_run_id,- "SessionId": session_id,- "UserId": pipe_job.job_metadata.user_id,- "DomainCode": pipe_job.pipe.domain_code,- }- pairs = [SearchAttributePair(_KEY_BY_NAME[name], value_by_name[name]) for name in _KEY_BY_NAME if name in enabled_names]- return TypedSearchAttributes(pairs)---def build_static_summary(pipe: PipeAbstract) -> str:- """Return ``{pipe_code} — {description}`` truncated to 200 bytes.-- ``pipe.description`` is a required-but-can-be-empty Pydantic field; when- empty the dash-and-tail are omitted entirely.- """- if pipe.description:- text = f"{pipe.code} — {pipe.description}"- else:- text = pipe.code- return _truncate_utf8(text, max_bytes=_MAX_SUMMARY_BYTES)---def build_static_details(pipe_job: PipeJob) -> str:- """Return a Markdown table of identity fields for the workflow's static details.-- The "Library crate" and "Input" rows are best-effort — emitted when the- relevant data is available on ``pipe_job``, omitted otherwise.- """- pipe = pipe_job.pipe- metadata = pipe_job.job_metadata- rows: list[tuple[str, str]] = [- ("Pipe", f"`{pipe.code}`"),- ("Domain", f"`{pipe.domain_code}`"),- ("Pipeline run", f"`{metadata.pipeline_run_id}`"),- ("User", f"`{metadata.user_id}`"),- ("Session", f"`{metadata.session_id or ''}`"),- ]- if pipe_job.library_crate is not None and pipe_job.library_crate.fingerprint:- rows.append(("Library crate", f"`{pipe_job.library_crate.fingerprint[:_LIBRARY_CRATE_ID_LEN]}`"))- input_names = list(pipe.inputs.root.keys())- if input_names:- rows.append(("Input", ", ".join(f"`{name}`" for name in input_names)))- lines: list[str] = ["| Field | Value |", "|---|---|"]- for field, value in rows:- lines.append(f"| {field} | {value} |")- return _truncate_utf8("\n".join(lines), max_bytes=_MAX_DETAILS_BYTES)---def build_activity_summary(- method_label: str,- *,- job_metadata: JobMetadata,- extras: Mapping[str, str] | None = None,-) -> str:- """Return ``{method_label} · pipe={pipe_code} · {extras...}`` truncated to 200 bytes.-- The ``pipe=`` segment is omitted when ``job_metadata.pipe_code`` is unset so- the format degrades gracefully for tests / fixtures with no pipe context.-- ``extras`` is a regular mapping (not ``**kwargs``) so callers can use Python- reserved words as keys — the design table specifies ``class={class_name}``- verbatim, and ``extras={"class": ...}`` lets the call site stay readable.- """- parts: list[str] = [method_label]- if job_metadata.pipe_code:- parts.append(f"pipe={job_metadata.pipe_code}")- if extras:- for key, value in extras.items():- parts.append(f"{key}={value}")- text = " · ".join(parts)- return _truncate_utf8(text, max_bytes=_MAX_SUMMARY_BYTES)diff --git a/pipelex/temporal/tprl/temporal_error.py b/pipelex/temporal/tprl/temporal_error.pydeleted file mode 100644index 5dc919977..000000000--- a/pipelex/temporal/tprl/temporal_error.py+++ /dev/null@@ -1,284 +0,0 @@-from collections.abc import Sequence-from typing import Any, cast--from pydantic import ValidationError-from temporalio import activity-from temporalio.exceptions import ApplicationError--from pipelex.base_exceptions import ErrorReport, PipelexError, iter_cause_chain-from pipelex.cogt.exceptions import find_inference_error_category_in_chain-from pipelex.config import get_config-from pipelex.temporal.exceptions import UnrecoverableWorkflowFailureError-from pipelex.temporal.log_temporal import activity_log, workflow_log-from pipelex.types import Self--_ERROR_REPORT_REQUIRED_KEYS: frozenset[str] = frozenset({"error_type", "message", "title", "type_uri"})--# Suffixed onto the synthesized ``UnrecoverableWorkflowFailureError`` message when-# the bridge-side schema-validation fallback in :func:`recover_error_report` fires.-# Detectors (tests, ops dashboards, the documentation in-# ``docs/under-the-hood/error-model.md``) reference this marker as a stable-# contract — import the constant instead of duplicating the literal.-_ERROR_REPORT_VALIDATION_FAILED_MARKER: str = "[error report failed schema validation]"---def error_report_dict_from_details(details: Sequence[Any]) -> dict[str, Any] | None:- """Recover the ``ErrorReport`` dict packed into ``ApplicationError.details``.-- The bridge packs ``exc.to_error_report().to_dict()`` as the first details- entry. After Temporal serialization the dict comes back as a plain mapping;- we identify it by the presence of every ``ErrorReport``-mandatory key- (``error_type`` / ``message`` / ``title`` / ``type_uri``) so an unrelated- details payload that happens to carry only two of them is not picked up- and routed into the schema-validation fallback.- """- for entry in details:- if isinstance(entry, dict) and all(key in entry for key in _ERROR_REPORT_REQUIRED_KEYS):- return cast("dict[str, Any]", entry)- return None---def _find_error_report_dict(exc: BaseException) -> dict[str, Any] | None:- """Return the first details-packed ``ErrorReport`` dict in the ``__cause__`` chain of ``exc``.-- Temporal sets ``__cause__`` on the ``WorkflowFailureError`` raised by- ``client.execute_workflow`` to the deserialized failure; the structured- ``ErrorReport`` packed by the activity bridge rides on an- ``ApplicationError``'s ``details``. A workflow that wraps a failed child- workflow re-raises its own failure — Temporal serializes that re-raised- ``WorkflowExecutionError`` as an outer ``ApplicationError`` whose ``details``- are empty, with the report-carrying child ``ApplicationError`` deeper in the- chain. The walk therefore continues past a report-less ``ApplicationError``- rather than stopping at the first one. The child-workflow boundary exposes- its failure via ``ChildWorkflowError.cause`` rather than ``__cause__``, so- its caller passes ``exc.cause`` straight in.- """- for node in iter_cause_chain(exc):- if isinstance(node, ApplicationError):- report_dict = error_report_dict_from_details(node.details)- if report_dict is not None:- return report_dict- return None---def _message_from_exc(exc: BaseException) -> str:- """Return the most informative message available from a Temporal failure chain.-- ``WorkflowFailureError`` carries the generic outer text ``"Workflow execution failed"``;- the underlying ``__cause__`` (a Temporal ``ApplicationError`` carrying the- worker exception, or a non-Temporal exception) holds the real detail. We walk- the ``__cause__`` chain and surface the deepest non-empty message, falling- back to ``repr(exc)`` when every message in the chain is unset.-- A whitespace-only ``str(node)`` is treated the same as empty — without the- ``strip()`` guard, a node whose message is purely spaces or newlines would- win and the synthesized preamble would be visually broken.- """- deepest_message = ""- for node in iter_cause_chain(exc):- text = str(node)- if text.strip():- deepest_message = text- return deepest_message or repr(exc)---def recover_error_report(exc: BaseException) -> ErrorReport:- """Recover the structured ``ErrorReport`` from a Temporal failure.-- Walks the ``__cause__`` chain for an ``ApplicationError`` carrying a- details-packed report (see :func:`error_report_dict_from_details`) and- rebuilds an :class:`ErrorReport` from it. When no such payload is present- in the chain — a non-Pipelex exception, a worker crash, a heartbeat- timeout, or anything else that crossed the boundary without going through- the activity bridge — synthesizes an :class:`UnrecoverableWorkflowFailureError`- report so callers in the error-recovery path always have a structured- report to surface. The synthesized report carries the most informative- message recoverable from the failure chain (see :func:`_message_from_exc`)- and a stable identity / classification- (``error_type="UnrecoverableWorkflowFailureError"``,- ``error_domain=RUNTIME``).-- A report dict that is found but fails :meth:`ErrorReport.from_dict`- validation is an internal contract violation within one deploy — the- activity bridge and the submitter share the schema. Raising there would- abort the caller before it can deliver the failure webhook, leaving the- receiver with no notification at all, so this synthesizes the same- :class:`UnrecoverableWorkflowFailureError` fallback — carrying the- recovered ``message`` and an ``[error report failed schema validation]``- marker. The failed run is still reported to the receiver and the contract- bug stays visible: a bug to fix, but not at the cost of a silent run.- """- report_dict = _find_error_report_dict(exc)- if report_dict is not None:- try:- return ErrorReport.from_dict(report_dict)- except ValidationError:- # The details payload looked like a report — it carried error_type- # and message — but failed the ErrorReport schema. Synthesize the- # fallback instead of raising, so the caller still delivers the- # failure webhook; the recovered message plus marker keep the- # contract bug visible on the wire.- # A whitespace-only ``report_dict["message"]`` is treated the same- # as missing — without the ``strip()`` guard, the fallback would not- # fire and the synthesized preamble would render as a visually- # broken `` [error report failed schema validation]``.- raw_message = report_dict.get("message")- if isinstance(raw_message, str) and raw_message.strip():- recovered_message = raw_message- else:- recovered_message = _message_from_exc(exc)- fallback_message = f"{recovered_message} {_ERROR_REPORT_VALIDATION_FAILED_MARKER}"- return UnrecoverableWorkflowFailureError(fallback_message).to_error_report()- return UnrecoverableWorkflowFailureError(_message_from_exc(exc)).to_error_report()---class TemporalError(ApplicationError):- """A Pipelex error crossing the Temporal activity → workflow boundary.-- Two things travel with the error across the boundary:-- - ``non_retryable``: when the exception's ``__cause__`` chain carries a- ``CogtError`` with an ``InferenceErrorCategory`` the flag is derived from- ``category.is_retryable`` — recovered even under the ``PipeRunError`` /- ``PipeRouterError`` / ``PipelineExecutionError`` wrappers, by walking the- ``__cause__`` chain. For a chain carrying no category the bridge falls- back to the configured ``non_retryable_error_types`` class-name list.- - ``error_report``: the structured ``ErrorReport`` dict is packed into- ``ApplicationError.details`` so workflow code keeps ``error_category``,- ``user_action``, ``model`` and ``provider`` — not just the message string.- """-- def __init__(- self,- message: str,- error_type: str | None,- non_retryable: bool = False,- error_report: dict[str, Any] | None = None,- ):- details: tuple[dict[str, Any], ...] = (error_report,) if error_report is not None else ()- super().__init__(- message,- *details,- type=error_type,- non_retryable=non_retryable,- )- self.error_report = error_report-- @classmethod- def _log_critical(cls, message: str) -> None:- """Log a non-retryable error at critical severity, in the active Temporal context.-- ``from_message_exception`` runs activity-side and ``from_app_error``- workflow-side; ``workflow.logger`` raises ``_NotInWorkflowEventLoopError``- outside a workflow event loop, so the logger must match the context.- """- if activity.in_activity():- activity_log.critical(message)- else:- workflow_log.critical(message)-- @classmethod- def _log_error(cls, message: str) -> None:- """Log a retryable error at error severity, in the active Temporal context.-- See ``_log_critical`` for why the logger must match the Temporal context.- """- if activity.in_activity():- activity_log.error(message)- else:- workflow_log.error(message)-- @classmethod- def _error_type_in_name_list(cls, error_type: str | None) -> bool:- """Check ``error_type`` against the configured ``non_retryable_error_types`` list.-- The fallback retry signal for exceptions that carry no- ``InferenceErrorCategory`` — see ``RetryPolicyConfig.non_retryable_error_types``.- """- temporal_config = get_config().temporal- all_non_retryable = temporal_config.worker_config.all_non_retryable_error_types(- queue_options_by_queue=temporal_config.queue_options,- )- return error_type in all_non_retryable-- @classmethod- def from_app_error(cls, exc: ApplicationError) -> Self:- """Re-wrap an ``ApplicationError`` observed in workflow code.-- The incoming error has already crossed Temporal serialization, so its- ``non_retryable`` flag and details payload reflect the decision the- activity-side bridge made. We preserve both. Only when neither is- present (a plain ``ApplicationError`` that never went through this- bridge) do we fall back to the class-name list for the severity decision.- """- message = exc.message- error_type = exc.type- error_report = error_report_dict_from_details(exc.details)- non_retryable = exc.non_retryable- if not non_retryable and error_report is None:- non_retryable = cls._error_type_in_name_list(error_type)- if non_retryable:- cls._log_critical(f"Non retryable error from ApplicationError[{error_type}]: {message}")- else:- cls._log_error(f"Error from ApplicationError[{error_type}]: {message}")- return cls(- message=message,- error_type=error_type,- non_retryable=non_retryable,- error_report=error_report,- )-- @classmethod- def from_message_exception(cls, exc: PipelexError, *, force_non_retryable: bool = False) -> Self:- """Convert a Pipelex exception into a ``TemporalError``.-- **Activity-side (default).** Retryability is derived from the ``__cause__``- chain: a ``CogtError`` carrying an ``InferenceErrorCategory`` sets it from- ``category.is_retryable`` — recovered even when wrapper exceptions- (``PipeRunError``, ``PipeRouterError``, ``PipelineExecutionError``) sit on- top. A chain with no categorized ``CogtError`` falls back to the configured- ``non_retryable_error_types`` class-name list.-- **Workflow-side inline (``force_non_retryable=True``).** The inline fail-safe- in ``WfPipeRouter`` / ``WfPipeRun`` converts a domain error raised directly in- workflow code. It forces ``non_retryable=True`` for two reasons: an inline- error must fail the workflow *terminally* rather than trigger a blunt whole-- workflow retry that re-runs already-completed inline work (retry belongs at- the activity boundary, where it is scoped and observable); and forcing the- flag short-circuits ``_is_non_retryable`` — the only ``get_config()`` read in- this method — so the conversion stays deterministic inside workflow code (a- config-derived decision is replay-fragile across deploys). The report's own- ``retryable`` field is config-free and preserved either way, as informational- metadata.- """- message = exc.message- error_type = exc.__class__.__name__- error_report = exc.to_error_report().to_dict()- non_retryable = force_non_retryable or cls._is_non_retryable(exc=exc, error_type=error_type)- if non_retryable:- cls._log_critical(f"Non retryable error from PipelexError[{error_type}]: {message}")- else:- cls._log_error(f"Retryable error from PipelexError[{error_type}]: {message}")- return cls(- message=message,- error_type=error_type,- non_retryable=non_retryable,- error_report=error_report,- )-- @classmethod- def _is_non_retryable(cls, exc: PipelexError, *, error_type: str) -> bool:- """Decide retryability — category-aware for ``CogtError``, name-list fallback otherwise.-- A categorized ``CogtError`` is usually wrapped — ``PipeRunError`` ->- ``PipeRouterError`` -> ``PipelineExecutionError`` — by the time it reaches the- activity boundary, so the ``InferenceErrorCategory`` is recovered by walking the- ``__cause__`` chain rather than inspecting only the outer exception. This keeps- ``non_retryable`` consistent with the chain-enriched ``retryable`` field of the- ``ErrorReport`` packed alongside it. A chain carrying no category falls back to- the configured ``non_retryable_error_types`` class-name list.- """- error_category = find_inference_error_category_in_chain(exc)- if error_category is not None:- return not error_category.is_retryable- return cls._error_type_in_name_list(error_type)diff --git a/pipelex/temporal/tprl/workflow_caller.py b/pipelex/temporal/tprl/workflow_caller.pydeleted file mode 100644index a41053760..000000000--- a/pipelex/temporal/tprl/workflow_caller.py+++ /dev/null@@ -1,326 +0,0 @@-from collections.abc import Mapping, Sequence-from datetime import timedelta-from typing import Any, Callable, Coroutine, Generic, Protocol, TypeVar, Union, cast--from pydantic import BaseModel-from temporalio import workflow-from temporalio.client import Callback, WorkflowFailureError, WorkflowHandle-from temporalio.client import Client as TemporalClient-from temporalio.common import RetryPolicy, TypedSearchAttributes-from temporalio.exceptions import ChildWorkflowError, WorkflowAlreadyStartedError-from temporalio.service import RPCError-from temporalio.workflow import ChildWorkflowHandle--from pipelex import log-from pipelex.config import get_config-from pipelex.pipe_run.exceptions import AsyncExecutionNotEnabledError-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment, get_temporal_client, get_temporal_manager-from pipelex.temporal.tprl.temporal_error import recover_error_report--T = TypeVar("T")-WorkflowOutput = TypeVar("WorkflowOutput", covariant=True)-WorkflowInput = TypeVar("WorkflowInput", bound=Union[BaseModel, dict[Any, Any]], contravariant=True)---class WorkflowCaller:- def __init__(- self,- task_queue: str | None = None,- workflow_execution_timeout: timedelta | None = None,- run_timeout: timedelta | None = None,- task_timeout: timedelta | None = None,- retry_policy: RetryPolicy | None = None,- start_delay: timedelta | None = None,- rpc_timeout: timedelta | None = None,- ):- self.task_queue: str | None = task_queue- self.execution_timeout: timedelta | None = workflow_execution_timeout- self.run_timeout: timedelta | None = run_timeout- self.task_timeout: timedelta | None = task_timeout- self.retry_policy: RetryPolicy | None = retry_policy- self.start_delay: timedelta | None = start_delay- self.rpc_timeout: timedelta | None = rpc_timeout---class WorkflowClass(Protocol[WorkflowInput, WorkflowOutput]):- @workflow.run- async def run(self, workflow_arg: WorkflowInput) -> WorkflowOutput: ...---class WorkflowExecutor(WorkflowCaller, Generic[WorkflowInput, WorkflowOutput]):- """A class that provides methods to execute and start workflows, both top-level and child workflows."""-- def __init__(- self,- temporal_client: TemporalClient | None = None,- should_auto_connect_temporal: bool = False,- worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,- **kwargs: Any,- ):- super().__init__(**kwargs)- self._temporal_client = temporal_client- self.should_auto_connect_temporal = should_auto_connect_temporal- self.worker_environment = worker_environment-- log.dev(f"{self.__class__.__name__} initialized for task queue '{self.task_queue}'")- log.dev(f"worker_environment='{self.worker_environment}', auto_connect_temporal={self.should_auto_connect_temporal}")-- @property- def class_name(self) -> str:- return self.__class__.__name__-- async def temporal_client(self) -> TemporalClient:- """Get the temporal client, raising an error if not set."""- if not get_config().temporal.is_enabled:- raise AsyncExecutionNotEnabledError.with_default_message()- if self._temporal_client is None:- if self.should_auto_connect_temporal:- log.debug(f"{self.class_name} auto-connecting to Temporal server")- self._temporal_client = await get_temporal_client(should_auto_connect=self.should_auto_connect_temporal)- else:- msg = "temporal_client is not connected and is not allowed to auto-connect"- raise WorkflowExecutionError(msg)- return self._temporal_client-- def make_workflow_id(self, pipeline_run_id: str) -> str:- workflow_id = get_temporal_manager().make_top_workflow_id(pipeline_run_id=pipeline_run_id)- log.debug(f"Top workflow_id: {workflow_id}")- return workflow_id-- async def execute_workflow(- self,- workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]],- *,- workflow_arg: WorkflowInput,- workflow_id: str,- search_attributes: TypedSearchAttributes | None = None,- static_summary: str | None = None,- static_details: str | None = None,- memo: Mapping[str, Any] | None = None,- ) -> WorkflowOutput:- """Execute a workflow and wait for its completion."""- try:- client = await self.temporal_client()- return await cast(- "Callable[..., Coroutine[Any, Any, WorkflowOutput]]",- client.execute_workflow, # pyright: ignore[reportUnknownMemberType]- )(- workflow=workflow_class.run,- arg=workflow_arg,- id=workflow_id,- task_queue=self.task_queue,- execution_timeout=self.execution_timeout,- retry_policy=self.retry_policy,- run_timeout=self.run_timeout,- task_timeout=self.task_timeout,- start_delay=self.start_delay,- rpc_timeout=self.rpc_timeout,- search_attributes=search_attributes,- static_summary=static_summary,- static_details=static_details,- memo=memo,- )- except WorkflowFailureError as exc:- # The activity bridge packs the structured ErrorReport into the- # failed workflow's ApplicationError.details; recover it so the- # classification survives the workflow -> submitter hop instead of- # flooring to a generic RUNTIME error. ``recover_error_report`` is- # total — a failure carrying no recoverable report yields a- # synthesized ``UnrecoverableWorkflowFailureError`` report whose- # message preserves the underlying exception detail.- error_report = recover_error_report(exc)- log.error(f"Failed to execute workflow {workflow_class.__name__}: {error_report.message}")- raise WorkflowExecutionError(error_report.message, error_report=error_report) from exc- except (WorkflowAlreadyStartedError, RPCError) as exc:- log.error(f"Failed to execute workflow {workflow_class.__name__}: {exc}")- msg = f"Failed to execute workflow {workflow_class.__name__}"- raise WorkflowExecutionError(msg) from exc-- async def start_workflow(- self,- workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]],- *,- workflow_arg: WorkflowInput,- workflow_id: str,- callbacks: Sequence[Callback] | None = None,- search_attributes: TypedSearchAttributes | None = None,- static_summary: str | None = None,- static_details: str | None = None,- memo: Mapping[str, Any] | None = None,- ) -> WorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]:- """Start a workflow without waiting for its completion."""- try:- client = await self.temporal_client()- return await cast(- "Callable[..., Coroutine[Any, Any, WorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]]]",- client.start_workflow, # pyright: ignore[reportUnknownMemberType]- )(- workflow=workflow_class.run,- arg=workflow_arg,- id=workflow_id,- task_queue=self.task_queue,- execution_timeout=self.execution_timeout,- retry_policy=self.retry_policy,- run_timeout=self.run_timeout,- task_timeout=self.task_timeout,- start_delay=self.start_delay,- rpc_timeout=self.rpc_timeout,- callbacks=callbacks or [],- search_attributes=search_attributes,- static_summary=static_summary,- static_details=static_details,- memo=memo,- )- except (WorkflowAlreadyStartedError, RPCError) as exc:- log.error(f"Failed to start workflow {workflow_class.__name__}: {exc}")- msg = f"Failed to start workflow {workflow_class.__name__}"- raise WorkflowExecutionError(msg) from exc-- async def execute_child_workflow(- self,- workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]],- *,- workflow_arg: WorkflowInput,- workflow_id: str,- child_task_queue: str | None = None,- search_attributes: TypedSearchAttributes | None = None,- static_summary: str | None = None,- static_details: str | None = None,- memo: Mapping[str, Any] | None = None,- ) -> WorkflowOutput:- """Execute a child workflow and wait for its completion.-- NOT used by Pipelex's in-workflow child-spawn sites — call- ``workflow.execute_child_workflow(...)`` directly from inside a- workflow. See the replay-determinism note below.-- Pipelex's own in-workflow child-spawn sites- (``pipelex.temporal.tprl_pipe.wf_pipe_run`` and- ``pipelex.temporal.tprl_pipe.temporal_pipe_router``) deliberately do- NOT use this wrapper. ``WorkflowExecutorFactory.create_executor``- reads ``get_config().temporal.worker_config`` to seed- ``execution_timeout`` / ``retry_policy`` / ``run_timeout`` /- ``task_timeout`` / ``start_delay`` / ``rpc_timeout`` for the- executor instance, and forwarding those into a- ``StartChildWorkflowExecution`` command from inside a workflow would- bake config-derived values into the recorded history — replay after a- config edit would then re-derive different values and Temporal would- reject the replay with a non-determinism mismatch. The submitter-side- entry points (``execute_workflow`` / ``start_workflow``) do not have- this problem because they run outside the workflow sandbox. If you- need to start a child workflow from inside a workflow, call- ``workflow.execute_child_workflow(...)`` directly and pass only- replay-deterministic options (workflow input, ids derived from the- input, ``search_attributes`` / ``static_summary`` built by the pure- helpers in ``pipelex.temporal.tprl.observability``).- """- try:- return await cast(- "Callable[..., Coroutine[Any, Any, WorkflowOutput]]",- workflow.execute_child_workflow, # pyright: ignore[reportUnknownMemberType]- )(- workflow=workflow_class.run,- arg=workflow_arg,- id=workflow_id,- task_queue=child_task_queue,- execution_timeout=self.execution_timeout,- retry_policy=self.retry_policy,- run_timeout=self.run_timeout,- task_timeout=self.task_timeout,- search_attributes=search_attributes,- static_summary=static_summary,- static_details=static_details,- memo=memo,- )- except ChildWorkflowError as exc:- log.error(f"ChildWorkflowError in {workflow_class.__name__} caused by: {exc.cause}")- # Mirror ``execute_workflow``'s recovery path — see its rationale comment.- # ``recover_error_report`` is total, so a non-``ApplicationError`` cause (a- # plain worker exception) still yields a synthesized report instead of a- # generic error. ``ChildWorkflowError`` exposes the underlying failure via- # ``exc.cause``; fall back to the error itself when it carries no cause.- error_report = recover_error_report(exc.cause if exc.cause is not None else exc)- raise WorkflowExecutionError(error_report.message, error_report=error_report) from exc-- async def start_child_workflow(- self,- workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]],- *,- workflow_arg: WorkflowInput,- workflow_id: str,- child_task_queue: str | None = None,- search_attributes: TypedSearchAttributes | None = None,- static_summary: str | None = None,- static_details: str | None = None,- memo: Mapping[str, Any] | None = None,- ) -> ChildWorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]:- """Start a child workflow without waiting for its completion.-- Same replay-determinism caveat as ``execute_child_workflow``: Pipelex's- in-workflow child-spawn sites bypass this wrapper because the- config-derived options it carries would be baked into the- ``StartChildWorkflowExecution`` command and break replay after any- config change. Call ``workflow.start_child_workflow(...)`` directly- from inside a workflow.- """- try:- return await cast(- "Callable[..., Coroutine[Any, Any, ChildWorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]]]",- workflow.start_child_workflow, # pyright: ignore[reportUnknownMemberType]- )(- workflow=workflow_class.run,- arg=workflow_arg,- id=workflow_id,- task_queue=child_task_queue,- execution_timeout=self.execution_timeout,- retry_policy=self.retry_policy,- run_timeout=self.run_timeout,- task_timeout=self.task_timeout,- search_attributes=search_attributes,- static_summary=static_summary,- static_details=static_details,- memo=memo,- )- except ChildWorkflowError as exc:- log.error(f"ChildWorkflowError in {workflow_class.__name__} caused by: {exc.cause}")- # Mirror ``execute_workflow``'s recovery path — see its rationale comment.- # ``recover_error_report`` is total, so a non-``ApplicationError`` cause (a- # plain worker exception) still yields a synthesized report instead of a- # generic error. ``ChildWorkflowError`` exposes the underlying failure via- # ``exc.cause``; fall back to the error itself when it carries no cause.- error_report = recover_error_report(exc.cause if exc.cause is not None else exc)- raise WorkflowExecutionError(error_report.message, error_report=error_report) from exc---class WorkflowExecutorFactory(Generic[WorkflowInput, WorkflowOutput]):- @classmethod- def create_executor(- cls,- *,- task_queue: str | None = None,- workflow_execution_timeout: timedelta | None = None,- retry_policy: RetryPolicy | None = None,- run_timeout: timedelta | None = None,- task_timeout: timedelta | None = None,- start_delay: timedelta | None = None,- rpc_timeout: timedelta | None = None,- temporal_client: TemporalClient | None = None,- should_auto_connect_temporal: bool = False,- worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,- ) -> WorkflowExecutor[WorkflowInput, WorkflowOutput]:- """Creates a WorkflowExecutor with configuration from pipelex.temporal's config if not provided."""- config = get_config().temporal.worker_config-- return WorkflowExecutor[WorkflowInput, WorkflowOutput](- task_queue=task_queue or config.default_task_queue,- workflow_execution_timeout=workflow_execution_timeout or config.workflow_execution_timeout,- retry_policy=retry_policy or config.retry_policy,- run_timeout=run_timeout or config.run_timeout,- task_timeout=task_timeout or config.task_timeout,- start_delay=start_delay or config.start_delay,- rpc_timeout=rpc_timeout or config.rpc_timeout,- temporal_client=temporal_client,- should_auto_connect_temporal=should_auto_connect_temporal,- worker_environment=worker_environment,- )diff --git a/pipelex/temporal/tprl_content_generation/__init__.py b/pipelex/temporal/tprl_content_generation/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/pipelex/temporal/tprl_content_generation/act_extract_generate.py b/pipelex/temporal/tprl_content_generation/act_extract_generate.pydeleted file mode 100644index 4112a4e52..000000000--- a/pipelex/temporal/tprl_content_generation/act_extract_generate.py+++ /dev/null@@ -1,26 +0,0 @@-from temporalio import activity--from pipelex import log-from pipelex.cogt.content_generation.assignment_models import ExtractAssignment-from pipelex.cogt.content_generation.extract_generate import extract_gen_pages_and_store-from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory-from pipelex.core.stuffs.page_content import PageContent-from pipelex.hub import get_storage_provider-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors---@convert_pipelex_errors-async def act_extract_gen_extract_pages(extract_assignment: ExtractAssignment) -> list[PageContent]:- """Extract pages and store extracted images, returning lightweight PageContent references.-- Large binary data (extracted images) is stored within the activity and never crosses- the Temporal workflow boundary — only URLs are returned.- """- log.dev("act_extract_gen_extract_pages")- storage_provider = get_storage_provider()- generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)- return await extract_gen_pages_and_store(- extract_assignment=extract_assignment,- generated_content_factory=generated_content_factory,- )diff --git a/pipelex/temporal/tprl_content_generation/act_img_gen_generate.py b/pipelex/temporal/tprl_content_generation/act_img_gen_generate.pydeleted file mode 100644index 0f55a3576..000000000--- a/pipelex/temporal/tprl_content_generation/act_img_gen_generate.py+++ /dev/null@@ -1,32 +0,0 @@-from temporalio import activity--from pipelex import log-from pipelex.cogt.content_generation.assignment_models import ImgGenAssignment-from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory-from pipelex.cogt.content_generation.img_gen_generate import img_gen_image_list_and_store, img_gen_single_image_and_store-from pipelex.core.stuffs.image_content import ImageContent-from pipelex.hub import get_storage_provider-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors---@convert_pipelex_errors-async def act_img_gen_images(img_gen_assignment: ImgGenAssignment) -> list[ImageContent]:- """Generate images and store them, returning lightweight ImageContent references.-- Large binary data (base64/bytes) is stored within the activity and never crosses- the Temporal workflow boundary — only URLs are returned.- """- log.dev("act_img_gen_images")- storage_provider = get_storage_provider()- generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)- if img_gen_assignment.nb_images > 1:- return await img_gen_image_list_and_store(- img_gen_assignment=img_gen_assignment,- generated_content_factory=generated_content_factory,- )- image_content = await img_gen_single_image_and_store(- img_gen_assignment=img_gen_assignment,- generated_content_factory=generated_content_factory,- )- return [image_content]diff --git a/pipelex/temporal/tprl_content_generation/act_jinja2_generate.py b/pipelex/temporal/tprl_content_generation/act_jinja2_generate.pydeleted file mode 100644index 3743ef3fb..000000000--- a/pipelex/temporal/tprl_content_generation/act_jinja2_generate.py+++ /dev/null@@ -1,13 +0,0 @@-from temporalio import activity--from pipelex import log-from pipelex.cogt.content_generation.assignment_models import TemplatingAssignment-from pipelex.cogt.content_generation.templating_generate import templating_gen_text-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors---@convert_pipelex_errors-async def act_jinja2_gen_text(jinja2_assignment: TemplatingAssignment) -> str:- log.dev("act_jinja2_gen_text")- return await templating_gen_text(templating_assignment=jinja2_assignment)diff --git a/pipelex/temporal/tprl_content_generation/act_llm_generate.py b/pipelex/temporal/tprl_content_generation/act_llm_generate.pydeleted file mode 100644index 741c48f53..000000000--- a/pipelex/temporal/tprl_content_generation/act_llm_generate.py+++ /dev/null@@ -1,28 +0,0 @@-from pydantic import BaseModel-from temporalio import activity--from pipelex import log-from pipelex.cogt.content_generation.assignment_models import LLMAssignment, ObjectAssignment-from pipelex.cogt.content_generation.llm_generate import llm_gen_object, llm_gen_object_list, llm_gen_text-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors---@convert_pipelex_errors-async def act_llm_gen_text(llm_assignment: LLMAssignment) -> str:- log.dev("act_llm_gen_text")- return await llm_gen_text(llm_assignment=llm_assignment)---@convert_pipelex_errors-async def act_llm_gen_object(object_assignment: ObjectAssignment) -> BaseModel:- log.dev("act_llm_gen_object")- return await llm_gen_object(object_assignment=object_assignment)---@convert_pipelex_errors-async def act_llm_gen_object_list(object_assignment: ObjectAssignment) -> list[BaseModel]:- log.dev("act_llm_gen_object_list")- return await llm_gen_object_list(object_assignment=object_assignment)diff --git a/pipelex/temporal/tprl_content_generation/act_render_page_views.py b/pipelex/temporal/tprl_content_generation/act_render_page_views.pydeleted file mode 100644index 52bd08674..000000000--- a/pipelex/temporal/tprl_content_generation/act_render_page_views.py+++ /dev/null@@ -1,27 +0,0 @@-from temporalio import activity--from pipelex import log-from pipelex.cogt.content_generation.assignment_models import RenderPageViewsAssignment-from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory-from pipelex.cogt.content_generation.render_generate import render_page_views_and_store-from pipelex.core.stuffs.image_content import ImageContent-from pipelex.hub import get_storage_provider-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors---@convert_pipelex_errors-async def act_render_page_views(render_assignment: RenderPageViewsAssignment) -> list[ImageContent]:- """Render PDF pages as images, store them, and return lightweight ImageContent references.-- Performs pypdfium2 rendering and storage within the activity so that large binary data- never crosses a Temporal workflow boundary. The rendering + storing logic lives in the- framework-agnostic ``render_page_views_and_store`` leaf, shared with the direct backend.- """- log.dev("act_render_page_views")- storage_provider = get_storage_provider()- generated_content_factory = GeneratedContentFactory(storage_provider=storage_provider)- return await render_page_views_and_store(- render_assignment=render_assignment,- generated_content_factory=generated_content_factory,- )diff --git a/pipelex/temporal/tprl_content_generation/act_search_generate.py b/pipelex/temporal/tprl_content_generation/act_search_generate.pydeleted file mode 100644index 5df84925c..000000000--- a/pipelex/temporal/tprl_content_generation/act_search_generate.py+++ /dev/null@@ -1,23 +0,0 @@-from typing import Any--from temporalio import activity--from pipelex import log-from pipelex.cogt.content_generation.assignment_models import SearchAssignment, SearchObjectAssignment-from pipelex.cogt.content_generation.search_generate import search_gen_sourced_answer, search_gen_structured-from pipelex.core.stuffs.search_result_content import SearchResultContent-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors---@convert_pipelex_errors-async def act_search_gen_sourced_answer(search_assignment: SearchAssignment) -> SearchResultContent:- log.dev("act_search_gen_sourced_answer")- return await search_gen_sourced_answer(search_assignment=search_assignment)---@convert_pipelex_errors-async def act_search_gen_structured(search_object_assignment: SearchObjectAssignment) -> dict[str, Any]:- log.dev("act_search_gen_structured")- return await search_gen_structured(search_object_assignment=search_object_assignment)diff --git a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py b/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.pydeleted file mode 100644index d292211f3..000000000--- a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py+++ /dev/null@@ -1,559 +0,0 @@-from typing import Any--from pydantic import BaseModel, ValidationError-from temporalio import workflow-from temporalio.exceptions import ActivityError, ApplicationError-from typing_extensions import override--from pipelex import log-from pipelex.cogt.content_generation.assignment_models import (- ExtractAssignment,- ImgGenAssignment,- LLMAssignment,- ObjectAssignment,- RenderPageViewsAssignment,- SearchAssignment,- SearchObjectAssignment,- TemplatingAssignment,-)-from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams-from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata-from pipelex.cogt.content_generation.exceptions import DryRunObjectFidelityError-from pipelex.cogt.extract.extract_input import ExtractInput-from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams-from pipelex.cogt.img_gen.img_gen_job_components import ImgGenJobConfig, ImgGenJobParams-from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt-from pipelex.cogt.llm.llm_prompt import LLMPrompt-from pipelex.cogt.llm.llm_setting import LLMSetting-from pipelex.cogt.templating.template_category import TemplateCategory-from pipelex.cogt.templating.templating_style import TemplatingStyle-from pipelex.config import get_config-from pipelex.core.stuffs.image_content import ImageContent-from pipelex.core.stuffs.page_content import PageContent-from pipelex.core.stuffs.search_result_content import SearchResultContent-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.exceptions import ContentGenerationError-from pipelex.temporal.tprl.observability import build_activity_summary-from pipelex.temporal.tprl.temporal_error import TemporalError-from pipelex.temporal.tprl_content_generation.act_extract_generate import act_extract_gen_extract_pages-from pipelex.temporal.tprl_content_generation.act_img_gen_generate import act_img_gen_images-from pipelex.temporal.tprl_content_generation.act_jinja2_generate import act_jinja2_gen_text-from pipelex.temporal.tprl_content_generation.act_llm_generate import (- act_llm_gen_object,- act_llm_gen_object_list,- act_llm_gen_text,-)-from pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views-from pipelex.temporal.tprl_content_generation.act_search_generate import act_search_gen_sourced_answer, act_search_gen_structured-from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar---def _revalidate_against_object_class(- raw_obj: BaseModel,- *,- object_class: type[BaseModelTypeVar],- is_mock_built: bool,-) -> BaseModelTypeVar:- """Re-validate an activity-boundary object against the original ``object_class``.-- The Temporal arm of ``ContentGenerator._revalidate_against_object_class``: round-trips the- activity-boundary ``BaseModel`` through json (``mode="json"`` is required for fields that need json-mode- serialization to round-trip cleanly) into the caller's original concrete class (e.g. ``StructuredContent``).-- Under the dry-run leaf mock (``run_mode=DRY``) the object was built from the- schema-reconstructed class inside ``act_llm_gen_object*``, which can drop invariants the original class- enforces (custom validators, ``json_schema_extra`` format/pattern hints datamodel-code-generator omits- on round-trip). That re-validation failure is re-raised as a clear- :class:`DryRunObjectFidelityError` rather than an opaque pydantic crash mid-workflow (review F2).- Scoped to the dry path only — a LIVE provider's invalid output keeps its existing ``ValidationError``.- """- raw_data = raw_obj.model_dump(mode="json", serialize_as_any=True)- if not is_mock_built:- return object_class.model_validate(raw_data)- try:- return object_class.model_validate(raw_data)- except ValidationError as exc:- raise DryRunObjectFidelityError.for_object_class(object_class.__name__) from exc---class ContentGeneratorInWorkflow(ContentGeneratorProtocol):- """ContentGenerator that dispatches each operation as a Temporal activity directly.-- The ``InWorkflow`` suffix signals the load-bearing constraint: every method calls- ``workflow.execute_activity(...)``, which only works when invoked from inside a- workflow's ``run()``. Calling these methods outside of a workflow context will fail.-- Activity IDs are never customized — the Temporal SDK assigns deterministic- sequential integers per workflow run, which both guarantees uniqueness and- is replay-safe by construction. Per-call meaning is carried in ``summary=``.-- Holds no storage factory: all storage happens inside the activities (the- ``*_and_store`` leaves), never on the workflow side.- """-- @override- @update_job_metadata- async def make_llm_text(- self,- *,- job_metadata: JobMetadata,- cogt_run_params: CogtRunParams,- llm_setting_main: LLMSetting,- llm_prompt_for_text: LLMPrompt,- ) -> str:- log.debug(f"ContentGeneratorInWorkflow make_llm_text: {llm_prompt_for_text}")- log.verbose(f"llm_setting_main: {llm_setting_main}")- worker_config = get_config().temporal.worker_config- llm_assignment = LLMAssignment(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- llm_setting=llm_setting_main,- llm_prompt=llm_prompt_for_text,- )- log.verbose(llm_assignment.desc, title="llm_assignment")- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_llm_gen_text.__name__,- routing_key=llm_assignment.llm_handle,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- generated_text: str = await workflow.execute_activity(- act_llm_gen_text,- arg=llm_assignment,- summary=build_activity_summary("LLM text", job_metadata=job_metadata, extras={"model": llm_assignment.llm_handle}),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- log.verbose(f"ContentGeneratorInWorkflow generated text: {generated_text}")- return generated_text-- @override- @update_job_metadata- async def make_object(- self,- *,- job_metadata: JobMetadata,- cogt_run_params: CogtRunParams,- object_class: type[BaseModelTypeVar],- llm_setting_for_object: LLMSetting,- llm_prompt_for_object: LLMPrompt,- ) -> BaseModelTypeVar:- log.verbose(f"ContentGeneratorInWorkflow make_object: {llm_prompt_for_object}")- worker_config = get_config().temporal.worker_config- llm_assignment_for_object = LLMAssignment(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- llm_setting=llm_setting_for_object,- llm_prompt=llm_prompt_for_object,- )- object_assignment = ObjectAssignment.make_for_class(- object_class=object_class,- llm_assignment=llm_assignment_for_object,- )- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_llm_gen_object.__name__,- routing_key=llm_assignment_for_object.llm_handle,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- obj: BaseModel = await workflow.execute_activity(- act_llm_gen_object,- arg=object_assignment,- summary=build_activity_summary("LLM object", job_metadata=job_metadata, extras={"class": object_class.__name__}),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- log.verbose(f"ContentGeneratorInWorkflow generated object direct: {obj}")- return _revalidate_against_object_class(obj, object_class=object_class, is_mock_built=cogt_run_params.run_mode.is_dry)-- @override- @update_job_metadata- async def make_object_list(- self,- *,- job_metadata: JobMetadata,- cogt_run_params: CogtRunParams,- object_class: type[BaseModelTypeVar],- llm_setting_for_object_list: LLMSetting,- llm_prompt_for_object_list: LLMPrompt,- nb_items: int | None = None,- ) -> list[BaseModelTypeVar]:- worker_config = get_config().temporal.worker_config- llm_assignment_for_object = LLMAssignment(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- llm_setting=llm_setting_for_object_list,- llm_prompt=llm_prompt_for_object_list,- )- object_assignment = ObjectAssignment.make_for_class(- object_class=object_class,- llm_assignment=llm_assignment_for_object,- nb_items=nb_items,- )- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_llm_gen_object_list.__name__,- routing_key=llm_assignment_for_object.llm_handle,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- obj_list: list[BaseModel] = await workflow.execute_activity(- act_llm_gen_object_list,- arg=object_assignment,- summary=build_activity_summary("LLM object list", job_metadata=job_metadata, extras={"class": object_class.__name__}),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- log.verbose(f"ContentGeneratorInWorkflow generated object list direct: {obj_list}")- return [- _revalidate_against_object_class(raw_obj, object_class=object_class, is_mock_built=cogt_run_params.run_mode.is_dry)- for raw_obj in obj_list- ]-- @override- @update_job_metadata- async def make_single_image(- self,- *,- job_metadata: JobMetadata,- cogt_run_params: CogtRunParams,- img_gen_handle: str,- img_gen_prompt: ImgGenPrompt,- img_gen_job_params: ImgGenJobParams | None = None,- img_gen_job_config: ImgGenJobConfig | None = None,- ) -> ImageContent:- worker_config = get_config().temporal.worker_config- img_gen_config = get_config().cogt.img_gen_config- img_gen_assignment = ImgGenAssignment(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- img_gen_handle=img_gen_handle,- img_gen_prompt=img_gen_prompt,- img_gen_job_params=img_gen_job_params or img_gen_config.make_default_img_gen_job_params(),- img_gen_job_config=img_gen_job_config or img_gen_config.img_gen_job_config,- nb_images=1,- )- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_img_gen_images.__name__,- routing_key=img_gen_assignment.img_gen_handle,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- image_content_list: list[ImageContent] = await workflow.execute_activity(- act_img_gen_images,- arg=img_gen_assignment,- summary=build_activity_summary("Img gen 1×", job_metadata=job_metadata, extras={"model": img_gen_handle}),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- if len(image_content_list) != 1:- msg = f"Expected 1 image, got {len(image_content_list)}"- raise ContentGenerationError(msg)- image_content = image_content_list[0]- log.verbose(f"ContentGeneratorInWorkflow generated image: {image_content}")- return image_content-- @override- @update_job_metadata- async def make_image_list(- self,- *,- job_metadata: JobMetadata,- cogt_run_params: CogtRunParams,- img_gen_handle: str,- img_gen_prompt: ImgGenPrompt,- nb_images: int,- img_gen_job_params: ImgGenJobParams | None = None,- img_gen_job_config: ImgGenJobConfig | None = None,- ) -> list[ImageContent]:- worker_config = get_config().temporal.worker_config- img_gen_config = get_config().cogt.img_gen_config- img_gen_assignment = ImgGenAssignment(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- img_gen_handle=img_gen_handle,- img_gen_prompt=img_gen_prompt,- img_gen_job_params=img_gen_job_params or img_gen_config.make_default_img_gen_job_params(),- img_gen_job_config=img_gen_job_config or img_gen_config.img_gen_job_config,- nb_images=nb_images,- )- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_img_gen_images.__name__,- routing_key=img_gen_assignment.img_gen_handle,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- image_content_list: list[ImageContent] = await workflow.execute_activity(- act_img_gen_images,- arg=img_gen_assignment,- summary=build_activity_summary("Img gen N×", job_metadata=job_metadata, extras={"model": img_gen_handle, "n": str(nb_images)}),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- log.verbose(f"ContentGeneratorInWorkflow generated image list: {image_content_list}")- return image_content_list-- @override- @update_job_metadata- async def make_templated_text(- self,- *,- job_metadata: JobMetadata,- cogt_run_params: CogtRunParams,- context: dict[str, Any],- template: str,- templating_style: TemplatingStyle | None = None,- template_category: TemplateCategory | None = None,- ) -> str:- worker_config = get_config().temporal.worker_config- templating_assignment = TemplatingAssignment(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- context=context,- template=template,- templating_style=templating_style,- category=template_category or TemplateCategory.BASIC,- )- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_jinja2_gen_text.__name__,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- jinja2_text: str = await workflow.execute_activity(- act_jinja2_gen_text,- arg=templating_assignment,- summary=build_activity_summary("Templated text", job_metadata=job_metadata),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- log.verbose(f"ContentGeneratorInWorkflow templated text: {jinja2_text}")- return jinja2_text-- @override- @update_job_metadata- async def make_render_page_views(- self,- *,- job_metadata: JobMetadata,- cogt_run_params: CogtRunParams,- extract_input: ExtractInput,- extract_handle: str,- extract_job_params: ExtractJobParams | None = None,- extract_job_config: ExtractJobConfig | None = None,- ) -> list[ImageContent]:- if not extract_input.document_uri:- msg = "PDF URI is required to render page views"- raise ValueError(msg)- worker_config = get_config().temporal.worker_config- job_params = extract_job_params or ExtractJobParams.make_default_extract_job_params()- page_views_dpi = job_params.page_views_dpi or get_config().cogt.extract_config.default_page_views_dpi- render_assignment = RenderPageViewsAssignment(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- document_uri=extract_input.document_uri,- page_views_dpi=page_views_dpi,- )- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_render_page_views.__name__,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- image_content_list: list[ImageContent] = await workflow.execute_activity(- act_render_page_views,- arg=render_assignment,- summary=build_activity_summary("Render page views", job_metadata=job_metadata),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- log.verbose(f"ContentGeneratorInWorkflow rendered page views: {image_content_list}")- return image_content_list-- @override- @update_job_metadata- async def make_extract_pages(- self,- *,- job_metadata: JobMetadata,- cogt_run_params: CogtRunParams,- extract_input: ExtractInput,- extract_handle: str,- extract_job_params: ExtractJobParams,- extract_job_config: ExtractJobConfig,- ) -> list[PageContent]:- worker_config = get_config().temporal.worker_config- extract_assignment = ExtractAssignment(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- extract_handle=extract_handle,- extract_input=extract_input,- extract_job_params=extract_job_params,- extract_job_config=extract_job_config,- )- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_extract_gen_extract_pages.__name__,- routing_key=extract_assignment.extract_handle,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- page_contents: list[PageContent] = await workflow.execute_activity(- act_extract_gen_extract_pages,- arg=extract_assignment,- summary=build_activity_summary("Extract pages", job_metadata=job_metadata, extras={"handle": extract_handle}),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- log.verbose(f"ContentGeneratorInWorkflow generated page contents: {page_contents}")-- if extract_job_params.should_include_page_views:- page_view_contents: list[ImageContent] = []- if extract_input.document_uri:- page_views_dpi = extract_job_params.page_views_dpi or get_config().cogt.extract_config.default_page_views_dpi- render_assignment = RenderPageViewsAssignment(- job_metadata=job_metadata,- cogt_run_params=cogt_run_params,- document_uri=extract_input.document_uri,- page_views_dpi=page_views_dpi,- )- render_dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_render_page_views.__name__,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- page_view_contents = await workflow.execute_activity(- act_render_page_views,- arg=render_assignment,- summary=build_activity_summary("Render page views (extract)", job_metadata=job_metadata),- **render_dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- elif extract_input.image_uri:- page_view_contents = [ImageContent(url=extract_input.image_uri)]- if len(page_view_contents) != len(page_contents):- msg = f"Number of page view contents ({len(page_view_contents)}) does not match number of page contents ({len(page_contents)})"- raise ContentGenerationError(msg)- for page_content in page_contents:- page_content.page_view = page_view_contents.pop(0)-- return page_contents-- @override- async def make_search_sourced_answer(- self,- search_assignment: SearchAssignment,- ) -> SearchResultContent:- worker_config = get_config().temporal.worker_config- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_search_gen_sourced_answer.__name__,- routing_key=search_assignment.search_handle,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- search_result: SearchResultContent = await workflow.execute_activity(- act_search_gen_sourced_answer,- arg=search_assignment,- summary=build_activity_summary(- "Search sourced answer", job_metadata=search_assignment.job_metadata, extras={"model": search_assignment.search_handle}- ),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- log.verbose(f"ContentGeneratorInWorkflow generated search result: {search_result}")- return search_result-- @override- async def make_search_structured(- self,- *,- output_structure_class: type[BaseModelTypeVar],- search_assignment: SearchAssignment,- ) -> BaseModelTypeVar:- worker_config = get_config().temporal.worker_config- search_object_assignment = SearchObjectAssignment.make_for_class(- output_class=output_structure_class,- search_assignment=search_assignment,- )- dispatch_kwargs = worker_config.resolve_dispatch(- activity_name=act_search_gen_structured.__name__,- routing_key=search_assignment.search_handle,- queue_options_by_queue=get_config().temporal.queue_options,- is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,- ).to_execute_kwargs()- try:- result_dict: dict[str, Any] = await workflow.execute_activity(- act_search_gen_structured,- arg=search_object_assignment,- summary=build_activity_summary(- "Search structured", job_metadata=search_assignment.job_metadata, extras={"class": output_structure_class.__name__}- ),- **dispatch_kwargs,- )- except ActivityError as exc:- log.error(f"ActivityError caused by: {exc.cause}")- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- # Re-validate on the submitter (workflow) side against the original class — pure and- # deterministic, so the dynamic output class never has to cross the activity boundary. The- # activity returns the provider's raw dict unvalidated (e.g. the gateway worker returns- # json.loads(...)), so a malformed structured response raises a bare ValidationError here in- # workflow code. Left raw it is neither WorkflowExecutionError nor PipelexError, so Temporal- # treats it as a workflow-task failure and retries forever, hanging the submitter — the exact- # failure mode this seam exists to prevent. Convert it to a typed PipelexError so the workflow- # fails and surfaces as an ErrorReport, matching the direct path: under a leaf mock the failure- # is the known schema-round-trip fidelity gap (D6), otherwise it is a malformed provider response.- try:- return output_structure_class.model_validate(result_dict)- except ValidationError as exc:- if search_assignment.cogt_run_params.run_mode.is_dry:- raise DryRunObjectFidelityError.for_object_class(output_structure_class.__name__) from exc- msg = f"Structured search result failed validation against {output_structure_class.__name__}: {exc}"- raise ContentGenerationError(msg) from excdiff --git a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow_factory.py b/pipelex/temporal/tprl_content_generation/content_generator_in_workflow_factory.pydeleted file mode 100644index 80d373979..000000000--- a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow_factory.py+++ /dev/null@@ -1,7 +0,0 @@-from pipelex.temporal.tprl_content_generation.content_generator_in_workflow import ContentGeneratorInWorkflow---class ContentGeneratorInWorkflowFactory:- @classmethod- def make_content_generator_in_workflow(cls) -> ContentGeneratorInWorkflow:- return ContentGeneratorInWorkflow()diff --git a/pipelex/temporal/tprl_pipe/__init__.py b/pipelex/temporal/tprl_pipe/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/pipelex/temporal/tprl_pipe/act_assemble_tracing.py b/pipelex/temporal/tprl_pipe/act_assemble_tracing.pydeleted file mode 100644index d3fdd3d8a..000000000--- a/pipelex/temporal/tprl_pipe/act_assemble_tracing.py+++ /dev/null@@ -1,50 +0,0 @@-"""Temporal activity for assembling tracing artifacts from the trace-event stream.--Reads trace events from the configured backend (DynamoDB / NDJSON) once and-assembles the full GraphSpec (for ``--graph``) and the aggregated token usage-(for ``--costs``). Runs as an activity because backend reads are I/O that cannot-happen inside a workflow thread.-"""--from pydantic import BaseModel-from temporalio import activity--from pipelex.pipe_run.tracing_assembly import TracingAssembly, assemble_tracing---class AssembleTracingArg(BaseModel):- """Input for the tracing assembly activity."""-- pipeline_run_id: str- domain_code: str | None = None- main_pipe_code: str | None = None- # Which artifacts to assemble (mirror of the run's TraceContext emit flags). The submitter gates- # the dispatch on these too; passing them through keeps the activity a pure function of its input.- assemble_graph: bool = True- assemble_usage: bool = True---# NOT decorated with @convert_pipelex_errors: tracing assembly is observability, not a pipe step.-# `assemble_tracing` already catches the EXPECTED best-effort failures (backend read errors, malformed-# events, broken tracing infra) and returns them on the *_assembly_error fields — those never raise here.-# A genuinely unexpected error (a programming bug in assembly) is deliberately allowed to propagate so the-# activity fails and `WfPipeRun` records it on usage_assembly_error / graph_assembly_error (its existing-# `except ActivityError` branch), mirroring DIRECT mode where assemble_tracing_on_output lets the same-# errors surface. Swallowing them here would make the workflow see success and silently produce no cost-# report and no diagnostic. The plain ActivityError carries the message WfPipeRun stringifies; no-# ErrorReport classification is needed for an observability artifact.[email protected](name="act_assemble_tracing")-async def act_assemble_tracing(arg: AssembleTracingArg) -> TracingAssembly: # noqa: RUF029- """Read trace events and assemble graph + usage.-- Returns an empty result when no events are found or tracing is disabled. Expected best-effort- failures are caught inside :func:`assemble_tracing` and returned on the ``*_assembly_error`` fields;- unexpected errors propagate so the workflow surfaces them (see the module-level note above).- """- return assemble_tracing(- pipeline_run_id=arg.pipeline_run_id,- assemble_graph=arg.assemble_graph,- assemble_usage=arg.assemble_usage,- domain_code=arg.domain_code,- main_pipe_code=arg.main_pipe_code,- )diff --git a/pipelex/temporal/tprl_pipe/act_deliver.py b/pipelex/temporal/tprl_pipe/act_deliver.pydeleted file mode 100644index 688e85222..000000000--- a/pipelex/temporal/tprl_pipe/act_deliver.py+++ /dev/null@@ -1,45 +0,0 @@-"""Temporal activity wrapper for delivery (storage + webhook).--Thin ``@activity.defn`` wrapper around the framework-agnostic core in-``pipelex.runtime_bridge.primitives.delivery``.-"""--from pydantic import BaseModel-from temporalio import activity--from pipelex.base_exceptions import ErrorReport-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus-from pipelex.runtime_bridge.primitives.delivery import execute_delivery-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors---class DeliveryActivityArg(BaseModel):- """Input for the act_deliver activity; ``error_report`` carries the worker-side classification on failure.-- ``request_id`` carries the originating ``JobMetadata.request_id`` so the delivery- phase's logs (storage / webhook) can be correlated with the workflow logs and the- inbound API request that started the run.- """-- pipe_output: PipeOutput | None = None- user_id: str- pipeline_run_id: str- delivery_assignment: DeliveryAssignment- status: DeliveryStatus = DeliveryStatus.COMPLETED- error_report: ErrorReport | None = None- request_id: str | None = None--[email protected](name="act_deliver")-@convert_pipelex_errors-async def act_deliver(arg: DeliveryActivityArg) -> None:- await execute_delivery(- pipe_output=arg.pipe_output,- user_id=arg.user_id,- pipeline_run_id=arg.pipeline_run_id,- delivery_assignment=arg.delivery_assignment,- status=arg.status,- error_report=arg.error_report,- request_id=arg.request_id,- )diff --git a/pipelex/temporal/tprl_pipe/act_dry_validate.py b/pipelex/temporal/tprl_pipe/act_dry_validate.pydeleted file mode 100644index 4906034ee..000000000--- a/pipelex/temporal/tprl_pipe/act_dry_validate.py+++ /dev/null@@ -1,170 +0,0 @@-"""Temporal activity running the whole /validate job — validation sweep + graph dry-run — in one process.--When Temporal is enabled, the API dispatches the whole ``/validate`` job to a worker as ONE-activity (via the one-step wrapper workflow ``WfDryValidate``) instead of running the sweep-API-side and the graph dry-run as a top-level worker workflow. The body is fully in-process:--- the **sweep** is :func:`pipelex.pipeline.validate_bundle.validate_bundle` — the exact function- the direct-mode ``/validate`` route calls, so both backends surface the SAME error contract- (the categorized ``ValidateBundleError`` cascade) and the same status map. It is already- Temporal-safe: ``BundleValidator.validate_pipes`` scopes its own in-process router and dry- content generator and runs a locally-constructed ``PipeRun``.-- the **graph dry-run** is ``dry_run_pipe_in_process`` — in-process by construction (local- ``PipeRun`` under ``scoped_event_log`` / ``scoped_pipe_router`` / ``scoped_content_generator``),- tracing into an in-memory event log; the ``GraphSpec`` rides back on the activity result. No- NDJSON file, no DynamoDB round-trip, no usage/cost reporting.--Library lifecycle (D4): the library is loaded ONCE — ``validate_bundle`` loads it and leaves it-current on success (its D6 loaded-on-success contract; it owns teardown on failure), the graph-runs against the same library, and this activity tears it down once in its ``finally``.--The activity result carries everything the canonical validation report needs that must be-computed worker-side, against the worker's loaded library (D10): the success-only status map,-the best-effort ``GraphSpec``, the library-wide ``pending_signatures``, and the-``pipe_io_contracts`` IO contracts (built inside the library window — their JSON-Schema rendering-needs the loaded library's class registry, so the API side never needs to re-acquire one).--Error contract (D3): validation failures (blueprint/factory/wiring errors, unexpected pipe failures,-strict-mode signature refusals) RAISE out of ``validate_bundle`` as ``ValidateBundleError`` and-cross the boundary as structured ``ErrorReport``s via ``convert_pipelex_errors`` — the API-renders them as the same RFC 7807 422 the direct path produces (identical ``error_type``,-``error_domain=input``, caller-facing message carrying the offending refs).--Graph best-effort (D5, widened to cross-backend parity): any ``PipelexError`` raised while-resolving or dry-running the graph pipe — plus pydantic ``ValidationError`` / polyfactory-``FactoryException`` (the mock-input mint shapes) — degrades to ``graph_spec=None``. This is-the exact contract the direct-mode route applies around ``dry_run_pipeline`` and the same-catch ``BundleValidator._classify_pipe`` uses, so both backends answer identically for the-same bundle. Only non-Pipelex programming bugs propagate and fail the activity, mirroring-``assemble_tracing``'s bug-propagation policy.-"""--from pydantic import BaseModel-from temporalio import activity--from pipelex import log-from pipelex.base_exceptions import PipelexError-from pipelex.graph.graphspec import GraphSpec-from pipelex.hub import (- clear_current_library,- get_current_library_id_or_none,- get_library_manager,- set_current_library,-)-from pipelex.pipe_run.dry_run_in_process import best_effort_graph_spec-from pipelex.pipeline.blueprint_selection import select_primary_blueprint-from pipelex.pipeline.bundle_validator import DryRunOutput-from pipelex.pipeline.pipe_io_contracts import PipeIOContract, build_pipe_io_contracts-from pipelex.pipeline.validate_bundle import validate_bundle-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors---class DryValidateArg(BaseModel):- """Input for the dry-run+validation activity (serializable across the Temporal boundary)."""-- mthds_contents: list[str] | None = None- # Optional per-content sources threaded onto `blueprint.source` so the worker's- # categorized `ValidateBundleError` carries a real `source` (length must match- # `mthds_contents` — `validate_bundle` enforces it). `None` keeps `source=None`.- mthds_sources: list[str] | None = None- allow_signatures: bool = False- # Optional explicit pipe selection for the graph arm; defaults to the bundle's declared- # main_pipe (qualified from the first blueprint declaring one). The sweep always covers the- # whole bundle; no graph is produced when neither this nor a main_pipe is set.- pipe_code: str | None = None---class DryValidateResult(BaseModel):- """Output of the dry-run+validation activity — everything the canonical validation report- needs that must be computed worker-side, against the worker's loaded library (D10):-- - ``dry_run_outputs``: the success-only per-pipe status map (D3) that- ``build_validated_pipes`` projects into ``validated_pipes``.- - ``graph_spec``: the best-effort graph of the main pipe.- - ``pending_signatures``: qualified refs of pipes still declared as ``PipeSignature`` in- the assembled library — the runnability verdict's input.- - ``pipe_io_contracts``: per-pipe IO contracts keyed by ``pipe_ref``, built via- ``build_pipe_io_contracts`` inside the worker's library window (JSON-Schema rendering- resolves bundle-defined structure classes through the loaded library's class registry,- so the API side never needs to re-acquire a library).- """-- dry_run_outputs: dict[str, DryRunOutput]- graph_spec: GraphSpec | None = None- # Deliberately REQUIRED (no wire default): a version-skewed worker that doesn't emit- # these fields must fail loudly at deserialization, not default to "nothing pending"- # and yield a silently wrong is_runnable verdict.- pending_signatures: list[str]- pipe_io_contracts: dict[str, PipeIOContract]--[email protected](name="act_dry_validate")-@convert_pipelex_errors-async def act_dry_validate(arg: DryValidateArg) -> DryValidateResult:- """Run the validation sweep + the in-memory graph dry-run against one library, in-process.-- Raises (crossing as structured ``ErrorReport``s via ``convert_pipelex_errors``):- ValidateBundleError: any validation failure — blueprint/factory/wiring errors and unexpected- dry-run pipe failures (the same categorized cascade the direct-mode route surfaces).- Signatures are NOT a failure (D-B): an unsatisfied ``PipeSignature`` rides the result's- ``pending_signatures`` / ``is_runnable``, it does not raise.- PipeIOContractError: a JSON-Schema rendering failure while building `pipe_io_contracts`- (non-retryable at the workflow tier, like ValidateBundleError — deterministic).- PipelexError: non-validation failures (config, library, tracing infra).- """- prev_library_id = get_current_library_id_or_none()- # Failure → validate_bundle owns its teardown and the error propagates through the boundary.- # Success → the library is left loaded + current (D6) and THIS activity owns the teardown.- validate_result = await validate_bundle(- mthds_contents=arg.mthds_contents,- mthds_sources=arg.mthds_sources,- allow_signatures=arg.allow_signatures,- )- library_id = get_current_library_id_or_none()- # Explicit body-success flag — NOT sys.exc_info() in the finally, which also sees an- # exception the CALLER is currently handling and would wrongly suppress a genuine- # teardown failure after a successful body.- body_succeeded = False- try:- # Pipe IO contracts: built INSIDE the library window (D10) — the JSON-Schema rendering- # resolves bundle-defined structure classes through the loaded library's class registry.- pipe_io_contracts = build_pipe_io_contracts(validate_result.pipes)-- # Graph: best-effort (D5), against the SAME loaded library, via the ONE shared- # graph-arm implementation — an unknown explicit pipe_code degrades to- # graph_spec=None just like any other graph-arm domain failure.- graph_spec: GraphSpec | None = await best_effort_graph_spec(- pipe_ref=arg.pipe_code or select_primary_blueprint(validate_result.blueprints).main_pipe_ref,- library_id=library_id,- log_context="act_dry_validate",- )-- result = DryValidateResult(- dry_run_outputs=validate_result.dry_run_result,- graph_spec=graph_spec,- pending_signatures=validate_result.pending_signatures,- pipe_io_contracts=pipe_io_contracts,- )- body_succeeded = True- return result- finally:- # Restore the caller's outer current-library FIRST so the guarantee survives a teardown- # raise, then tear the validated library down once — mirrors acquire_and_validate.- if prev_library_id is not None and prev_library_id != library_id:- set_current_library(library_id=prev_library_id)- else:- clear_current_library()- if library_id is not None:- try:- get_library_manager().teardown(library_id=library_id)- except PipelexError as teardown_error:- # A teardown failure must not REPLACE the body's in-flight error (the caller's- # 422 would name the teardown instead of the user's actual problem) — suppress- # it and let the primary propagate; raise it only when the body succeeded.- # Mirrors PipeRun.run's close_tracer handling.- if body_succeeded:- raise- log.error(- f"act_dry_validate: library teardown also failed after a body error; "- f"raising the original error. Suppressed teardown error: {teardown_error}"- )diff --git a/pipelex/temporal/tprl_pipe/act_flush_trace_events.py b/pipelex/temporal/tprl_pipe/act_flush_trace_events.pydeleted file mode 100644index 5c09d647f..000000000--- a/pipelex/temporal/tprl_pipe/act_flush_trace_events.py+++ /dev/null@@ -1,25 +0,0 @@-"""Temporal activity wrapper for flushing buffered trace events.--Thin ``@activity.defn`` wrapper around the framework-agnostic core in-``pipelex.runtime_bridge.primitives.trace_flush``.-"""--from pydantic import BaseModel, Field-from temporalio import activity--from pipelex.runtime_bridge.primitives.trace_flush import flush_trace_events_to_backend-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors-from pipelex.tools.typing.pydantic_utils import empty_list_factory_of-from pipelex.tracing.trace_events import TraceEvent---class FlushTraceEventsArg(BaseModel):- """Input for the act_flush_trace_events activity."""-- events: list[TraceEvent] = Field(default_factory=empty_list_factory_of(TraceEvent))--[email protected](name="act_flush_trace_events")-@convert_pipelex_errors-async def act_flush_trace_events(arg: FlushTraceEventsArg) -> None:- await flush_trace_events_to_backend(arg.events)diff --git a/pipelex/temporal/tprl_pipe/dry_validate_dispatch.py b/pipelex/temporal/tprl_pipe/dry_validate_dispatch.pydeleted file mode 100644index b03c03cb4..000000000--- a/pipelex/temporal/tprl_pipe/dry_validate_dispatch.py+++ /dev/null@@ -1,66 +0,0 @@-"""Submitter-side dispatch for the dry-run+validation wrapper workflow.--The single round-trip a Temporal-enabled ``/validate`` caller awaits: dispatch-``WfDryValidate`` (which runs the one ``act_dry_validate`` activity in-process on a worker)-and get back the full worker-computed ``DryValidateResult`` — per-pipe status map,-best-effort ``GraphSpec``, ``pending_signatures``, and the ``pipe_io_contracts`` IO contracts-(D10). Used by the API route (cross-repo, ``pipelex-api``) and by the Tier-2d e2e-submitter script.-"""--from datetime import timedelta--from temporalio.common import RetryPolicy--from pipelex.config import get_config-from pipelex.pipeline.pipeline_factory import PipelineFactory-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutorFactory-from pipelex.temporal.tprl_pipe.act_dry_validate import DryValidateArg, DryValidateResult-from pipelex.temporal.tprl_pipe.wf_dry_validate import WfDryValidate---async def dispatch_dry_validate(- arg: DryValidateArg,- *,- task_queue: str | None = None,- should_auto_connect_temporal: bool = True,-) -> DryValidateResult:- """Dispatch ``WfDryValidate`` to a worker and await the full ``DryValidateResult``.-- A validation failure (strict-mode signature refusal, unexpected pipe failure) comes back- as ``WorkflowExecutionError`` carrying the structured ``ErrorReport`` recovered by- ``WorkflowExecutor.execute_workflow`` — the same error shape every other Temporal dispatch- surfaces, which the API renders as an RFC 7807 422.-- Args:- arg: The serializable activity input (mthds_contents, allow_signatures, pipe_code).- task_queue: Optional task-queue override; defaults to the configured default queue.- should_auto_connect_temporal: Whether the executor may auto-connect the Temporal client.-- Returns:- The activity result: per-pipe status map, best-effort GraphSpec, pending_signatures,- and the pipe_io_contracts IO contracts — everything the canonical validation report- needs that must be computed worker-side (D10).- """- worker_config = get_config().temporal.worker_config- executor = WorkflowExecutorFactory[DryValidateArg, DryValidateResult]().create_executor(- task_queue=task_queue or worker_config.default_task_queue,- should_auto_connect_temporal=should_auto_connect_temporal,- # Interactive round-trip: bound the whole workflow to the activity retry budget- # (2 × 5 min attempts + dispatch slack) instead of the config default — a batch-tuned- # 1-hour window during which a worker outage would queue stale validation work that no- # HTTP caller is still waiting for. With this bound, an outage fails the caller fast.- workflow_execution_timeout=timedelta(minutes=12),- # No workflow-level retry (D-C5): validation is deterministic, so re-running the whole- # wrapper workflow on failure is pure waste — the config default policy would re-run a- # failed validation up to 3 times (its non_retryable list doesn't know ValidateBundleError).- # Transient infra failures are already retried at the ACTIVITY tier by WfDryValidate's own- # bounded policy.- retry_policy=RetryPolicy(maximum_attempts=1),- )- workflow_id = executor.make_workflow_id(pipeline_run_id=f"dry_validate_{PipelineFactory.make_pipeline_run_id()}")- return await executor.execute_workflow(- workflow_class=WfDryValidate,- workflow_arg=arg,- workflow_id=workflow_id,- )diff --git a/pipelex/temporal/tprl_pipe/temporal_pipe_router.py b/pipelex/temporal/tprl_pipe/temporal_pipe_router.pydeleted file mode 100644index eb2a03027..000000000--- a/pipelex/temporal/tprl_pipe/temporal_pipe_router.py+++ /dev/null@@ -1,143 +0,0 @@-from datetime import timedelta--from temporalio import workflow-from temporalio.common import RetryPolicy-from temporalio.exceptions import ChildWorkflowError-from typing_extensions import override--from pipelex import log-from pipelex.config import get_config-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.observer.observer_protocol import ObserverNoOp-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol-from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment-from pipelex.temporal.temporal_workflow_utils import is_in_temporal_workflow-from pipelex.temporal.tprl.conditional_worker import with_conditional_worker-from pipelex.temporal.tprl.observability import (- build_search_attributes,- build_static_details,- build_static_summary,- stamp_submitter_session_id,-)-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor, WorkflowExecutorFactory-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter---class TemporalPipeRouter(WorkflowExecutor[PipeJob, PipeOutput], PipeRouterProtocol):- """Temporal pipe router: auto-detects whether to dispatch as top-level or child workflow.-- When called outside a Temporal workflow, dispatches via execute_workflow (top-level).- When called inside a Temporal workflow, dispatches via execute_child_workflow (child).- """-- def __init__(- self,- task_queue: str,- workflow_execution_timeout: timedelta | None = None,- retry_policy: RetryPolicy | None = None,- should_auto_connect_temporal: bool = False,- worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,- ):- log.debug(f"TemporalPipeRouter init with worker_environment: {worker_environment}")- super().__init__(- workflow_execution_timeout=workflow_execution_timeout,- retry_policy=retry_policy,- task_queue=task_queue,- should_auto_connect_temporal=should_auto_connect_temporal,- worker_environment=worker_environment,- )- self.observer = ObserverNoOp()-- @override- @with_conditional_worker- async def _run_pipe_job(- self,- pipe_job: PipeJob,- ) -> PipeOutput:- pipe_job = pipe_job.prepare_for_temporal()-- if is_in_temporal_workflow():- # Child workflow dispatch (inside a Temporal workflow).- # The child id is built off the parent's workflow id with an underscore- # separator, a pipe-code segment for readability and an 8-hex-char- # disambiguator from ``workflow.uuid4()`` (replay-safe — Temporal's uuid4- # is deterministic). The separator is ``_``, never ``/``: workflow ids must- # stay free of path separators so they can be reused verbatim as S3 keys or- # file names without spawning spurious directory segments.- log.debug("TemporalPipeRouter: child workflow dispatch")- parent_workflow_id = workflow.info().workflow_id- child_workflow_id = f"{parent_workflow_id}_{pipe_job.pipe.code}-{str(workflow.uuid4())[:8]}"- # Dispatch via ``workflow.execute_child_workflow`` directly. The- # ``WorkflowExecutorFactory`` is only safe at the submitter boundary- # because it reads ``get_config().temporal.worker_config`` to seed- # execution_timeout / retry_policy / run_timeout / etc., which would- # bake config-dependent options into the recorded- # ``StartChildWorkflowExecution`` command and break determinism on- # replay after any config change. Wrap raw ``ChildWorkflowError``- # as ``WorkflowExecutionError`` so the worker's- # ``workflow_failure_exception_types`` contract still ends the- # workflow terminally instead of triggering task-failure retry.- try:- pipe_output = await workflow.execute_child_workflow(- WfPipeRouter.run,- arg=pipe_job,- id=child_workflow_id,- search_attributes=build_search_attributes(pipe_job),- static_summary=build_static_summary(pipe_job.pipe),- static_details=build_static_details(pipe_job),- )- except ChildWorkflowError as exc:- msg = f"Failed to execute child workflow WfPipeRouter for pipe '{pipe_job.pipe.code}'"- raise WorkflowExecutionError(msg) from exc- else:- # Top-level dispatch (outside a Temporal workflow). Stamp the submitter's- # session_id onto the pipe_job's metadata so child workflows started later- # from inside Temporal code can read it without touching worker-local state.- pipe_job = stamp_submitter_session_id(pipe_job)- log.debug(f"TemporalPipeRouter: top-level dispatch, task_queue={self.task_queue}")- executor = WorkflowExecutorFactory[PipeJob, PipeOutput]().create_executor(- task_queue=self.task_queue,- workflow_execution_timeout=self.execution_timeout,- retry_policy=self.retry_policy,- run_timeout=self.run_timeout,- task_timeout=self.task_timeout,- start_delay=self.start_delay,- rpc_timeout=self.rpc_timeout,- should_auto_connect_temporal=self.should_auto_connect_temporal,- worker_environment=self.worker_environment,- )- pipe_output = await executor.execute_workflow(- workflow_class=WfPipeRouter,- workflow_id=self.make_workflow_id(pipeline_run_id=pipe_job.job_metadata.pipeline_run_id),- workflow_arg=pipe_job,- search_attributes=build_search_attributes(pipe_job),- static_summary=build_static_summary(pipe_job.pipe),- static_details=build_static_details(pipe_job),- )-- # Rehydrate PipeOutput: reconstruct typed WorkingMemory from raw dict.- # Uses a per-call scoped library when the pipe_job carries a crate so the- # submitter does not need to have pre-loaded the bundle into its global registry.- return rehydrate_pipe_output_with_crate(pipe_output, library_crate=pipe_job.library_crate)---def make_temporal_pipe_router(- *,- task_queue: str | None = None,- workflow_execution_timeout: timedelta | None = None,- retry_policy: RetryPolicy | None = None,- should_auto_connect_temporal: bool = True,- worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,-) -> TemporalPipeRouter:- """Factory: creates a TemporalPipeRouter from config defaults."""- worker_config = get_config().temporal.worker_config- return TemporalPipeRouter(- task_queue=task_queue or worker_config.default_task_queue,- workflow_execution_timeout=workflow_execution_timeout or worker_config.workflow_execution_timeout,- retry_policy=retry_policy or worker_config.retry_policy,- should_auto_connect_temporal=should_auto_connect_temporal,- worker_environment=worker_environment,- )diff --git a/pipelex/temporal/tprl_pipe/temporal_pipe_run.py b/pipelex/temporal/tprl_pipe/temporal_pipe_run.pydeleted file mode 100644index e4b874c61..000000000--- a/pipelex/temporal/tprl_pipe/temporal_pipe_run.py+++ /dev/null@@ -1,147 +0,0 @@-from datetime import timedelta--from temporalio.client import WorkflowHandle-from temporalio.common import RetryPolicy-from typing_extensions import override--from pipelex import log-from pipelex.config import get_config-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol-from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg-from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate-from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment-from pipelex.temporal.tprl.conditional_worker import with_conditional_worker-from pipelex.temporal.tprl.observability import (- build_search_attributes,- build_static_details,- build_static_summary,- stamp_submitter_session_id,-)-from pipelex.temporal.tprl.workflow_caller import WorkflowClass, WorkflowExecutor, WorkflowExecutorFactory-from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun---class TemporalPipeRun(WorkflowExecutor[PipeRunArg, PipeOutput], PipeRunProtocol):- """Temporal-mode PipeRun: dispatches WfPipeRun workflow which orchestrates execution + delivery."""-- def __init__(- self,- task_queue: str,- workflow_execution_timeout: timedelta | None = None,- retry_policy: RetryPolicy | None = None,- should_auto_connect_temporal: bool = False,- worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,- ) -> None:- log.debug(f"TemporalPipeRun init with worker_environment: {worker_environment}")- super().__init__(- workflow_execution_timeout=workflow_execution_timeout,- retry_policy=retry_policy,- task_queue=task_queue,- should_auto_connect_temporal=should_auto_connect_temporal,- worker_environment=worker_environment,- )-- @override- @with_conditional_worker- async def run(- self,- pipe_job: PipeJob,- *,- delivery_assignment: DeliveryAssignment | None = None,- ) -> PipeOutput:- """Execute a pipe run via Temporal (blocking — waits for completion).-- The deployment-level async-enabled guard fires inside- ``with_conditional_worker`` (above), before the worker setup on the- ``INTERNAL`` path and before this body on ``EXTERNAL`` — so- ``stamp_submitter_session_id`` (which reads the ``TemporalManager``- singleton) is unreachable on a disabled deployment.- """- # Stamp submitter session_id before the workflow input is built so the- # value flows through to every child workflow via the workflow arg.- pipe_job = stamp_submitter_session_id(pipe_job)- pipe_run_arg = PipeRunArg(- pipe_job=pipe_job,- delivery_assignment=delivery_assignment,- )- pipe_run_arg = pipe_run_arg.prepare_for_temporal()-- executor = WorkflowExecutorFactory[PipeRunArg, PipeOutput]().create_executor(- task_queue=self.task_queue,- should_auto_connect_temporal=self.should_auto_connect_temporal,- worker_environment=self.worker_environment,- )- pipe_output = await executor.execute_workflow(- workflow_class=WfPipeRun,- workflow_id=self.make_workflow_id(pipeline_run_id=pipe_job.job_metadata.pipeline_run_id),- workflow_arg=pipe_run_arg,- search_attributes=build_search_attributes(pipe_job),- static_summary=build_static_summary(pipe_job.pipe),- static_details=build_static_details(pipe_job),- )-- # Rehydrate PipeOutput on the submitter. When the pipe_job carries a crate,- # the helper opens a per-call scoped library so the submitter does not need- # the bundle pre-loaded in its global registry.- return rehydrate_pipe_output_with_crate(pipe_output, library_crate=pipe_job.library_crate)-- @with_conditional_worker- async def start(- self,- pipe_job: PipeJob,- *,- delivery_assignment: DeliveryAssignment | None = None,- ) -> tuple[str, WorkflowHandle[WorkflowClass[PipeRunArg, PipeOutput], PipeOutput]]:- """Start a pipe run without waiting for completion.-- Returns the workflow_id and a WorkflowHandle that can be awaited later.- The deployment-level async-enabled guard fires inside- ``with_conditional_worker``; see :meth:`run` for the rationale.- """- log.debug(f"TemporalPipeRun start using task_queue: {self.task_queue}")- # Stamp submitter session_id before building the workflow input so it- # flows through every child workflow via the workflow arg.- pipe_job = stamp_submitter_session_id(pipe_job)- pipe_run_arg = PipeRunArg(- pipe_job=pipe_job,- delivery_assignment=delivery_assignment,- )- pipe_run_arg = pipe_run_arg.prepare_for_temporal()-- executor = WorkflowExecutorFactory[PipeRunArg, PipeOutput]().create_executor(- task_queue=self.task_queue,- should_auto_connect_temporal=self.should_auto_connect_temporal,- worker_environment=self.worker_environment,- )- workflow_id = self.make_workflow_id(pipeline_run_id=pipe_job.job_metadata.pipeline_run_id)- handle = await executor.start_workflow(- workflow_class=WfPipeRun,- workflow_id=workflow_id,- workflow_arg=pipe_run_arg,- search_attributes=build_search_attributes(pipe_job),- static_summary=build_static_summary(pipe_job.pipe),- static_details=build_static_details(pipe_job),- )- return workflow_id, handle---def make_temporal_pipe_run(- *,- task_queue: str | None = None,- workflow_execution_timeout: timedelta | None = None,- retry_policy: RetryPolicy | None = None,- should_auto_connect_temporal: bool = True,- worker_environment: TemporalWorkerEnvironment = TemporalWorkerEnvironment.EXTERNAL,-) -> TemporalPipeRun:- """Factory: creates a TemporalPipeRun from config defaults."""- worker_config = get_config().temporal.worker_config- return TemporalPipeRun(- task_queue=task_queue or worker_config.default_task_queue,- workflow_execution_timeout=workflow_execution_timeout or worker_config.workflow_execution_timeout,- retry_policy=retry_policy or worker_config.retry_policy,- should_auto_connect_temporal=should_auto_connect_temporal,- worker_environment=worker_environment,- )diff --git a/pipelex/temporal/tprl_pipe/wf_dry_validate.py b/pipelex/temporal/tprl_pipe/wf_dry_validate.pydeleted file mode 100644index ebbbacd94..000000000--- a/pipelex/temporal/tprl_pipe/wf_dry_validate.py+++ /dev/null@@ -1,46 +0,0 @@-"""One-step wrapper workflow dispatching the dry-run+validation activity.--The dispatch unit the API awaits for a Temporal-enabled ``/validate``: it runs the single-``act_dry_validate`` activity and returns its result — functionally identical to a standalone-activity call, but it works on the current ``temporalio`` SDK with no bump (the pre-flight-dispatch decision). Replacing it with a true standalone activity is the deferred Phase-G0-optimization.-"""--from datetime import timedelta--from temporalio import workflow-from temporalio.common import RetryPolicy-from typing_extensions import override--with workflow.unsafe.imports_passed_through():- from pipelex.temporal.tprl.workflow_caller import WorkflowClass- from pipelex.temporal.tprl_pipe.act_dry_validate import DryValidateArg, DryValidateResult, act_dry_validate--[email protected](name="wf_dry_validate")-class WfDryValidate(WorkflowClass[DryValidateArg, DryValidateResult]):- """Runs ``act_dry_validate`` once and returns its result."""-- @override- @workflow.run- async def run(self, workflow_arg: DryValidateArg) -> DryValidateResult:- # D-C5: explicit timeout + retry bounds, deterministic in workflow code (no config reads).- # Validation is deterministic — its failures must NOT retry. The deterministic-failure- # surface crosses the boundary as TWO types: validate_bundle wraps- # DryRunError / PipeRunError / PipelexInterpreterError into- # ValidateBundleError before they escape, and build_pipe_io_contracts wraps a- # JSON-Schema rendering failure into PipeIOContractError; the activity-boundary- # TemporalError carries the OUTER PipelexError class name as the ApplicationError- # type (which is what non_retryable_error_types matches). The graph arm never raises- # domain errors (it degrades to graph_spec=None). Everything else (worker crash,- # transient infra) gets one retry.- return await workflow.execute_activity(- act_dry_validate,- arg=workflow_arg,- start_to_close_timeout=timedelta(minutes=5),- retry_policy=RetryPolicy(- maximum_attempts=2,- non_retryable_error_types=["ValidateBundleError", "PipeIOContractError"],- ),- )diff --git a/pipelex/temporal/tprl_pipe/wf_pipe_router.py b/pipelex/temporal/tprl_pipe/wf_pipe_router.pydeleted file mode 100644index eb580d69e..000000000--- a/pipelex/temporal/tprl_pipe/wf_pipe_router.py+++ /dev/null@@ -1,309 +0,0 @@-from datetime import timedelta--from temporalio import workflow-from temporalio.common import RetryPolicy-from temporalio.exceptions import ActivityError, ApplicationError, FailureError-from typing_extensions import override--with workflow.unsafe.imports_passed_through():- from kajson.class_registry import ClassRegistry- from kajson.kajson_manager import KajsonManager-- from pipelex.base_exceptions import PipelexError, iter_cause_chain- from pipelex.core.pipes.pipe_output import PipeOutput- from pipelex.graph.graph_tracer_manager import GraphTracerManager- from pipelex.hub import clear_current_library, get_library_manager, get_report_delegate, set_current_library- from pipelex.pipe_run.pipe_job import PipeJob- from pipelex.runtime_bridge.primitives.hydration import hydrate_working_memory- from pipelex.temporal.log_temporal import WorkflowLog- from pipelex.temporal.tprl.temporal_error import TemporalError- from pipelex.temporal.tprl.workflow_caller import WorkflowClass- from pipelex.temporal.tprl_pipe.act_flush_trace_events import FlushTraceEventsArg, act_flush_trace_events- from pipelex.tracing.buffering_event_log import BufferingEventLog---def _carries_temporal_failure(exc: BaseException) -> bool:- """True when ``exc`` or any error in its ``__cause__`` chain is a Temporal ``FailureError``.-- This is the hinge of the inline fail-safe: ``True`` ⇒ leave ``exc`` untouched,- ``False`` ⇒ convert it with ``from_message_exception``. Carrying a Temporal- ``FailureError`` *is* the definition of "already a terminal Temporal failure",- so it is the predicate, not a proxy for a narrower type.-- The reachable case it protects: a controller pipe (e.g. ``PipeSequence``) runs a- sub-pipe as a child workflow; the sub-pipe fails; ``TemporalPipeRouter`` wraps the- ``ChildWorkflowError`` as ``WorkflowExecutionError`` (``temporal_pipe_router.py``),- which escapes the parent's ``pipe.run_pipe`` into the ``except PipelexError`` below.- That ``WorkflowExecutionError`` carries no report of its own — the rich leaf- classification lives deeper, in the child's ``ApplicationError.details``, reachable- only by ``recover_error_report``'s ``__cause__`` walk at the submitter (which- normalizes ``ChildWorkflowError.cause`` into the chain). A worker-side- ``from_message_exception`` here would flatten it, because ``to_error_report``'s- enrichment stops at the non-``PipelexError`` ``ChildWorkflowError``. So we propagate.-- Invariant this relies on: disciplined ``raise … from`` usage. The one way it can- misattribute is "recover-then-rechain" — pipe code that handles a side failure and- then raises a *fresh, unrelated* error ``from`` the one it already recovered. The- fresh error would then carry that side ``FailureError`` and propagate untouched,- surfacing the side failure's classification instead of its own. Don't do that: a- fresh error must only be chained ``from`` the error that actually caused it.-- Walks via ``iter_cause_chain`` (single cyclic-guard). See- ``docs/under-the-hood/error-model.md`` "Workflow-Level Fail-Safe Floor" for the- design narrative.- """- return any(isinstance(node, FailureError) for node in iter_cause_chain(exc))--[email protected](name="wf_pipe_router")-class WfPipeRouter(WorkflowClass[PipeJob, PipeOutput]):- @override- @workflow.run- async def run(- self,- workflow_arg: PipeJob,- ) -> PipeOutput:- # Bound once per invocation: every record below carries this run's- # request_id (None when the run carries no inbound API request id).- workflow_log = WorkflowLog(request_id=workflow_arg.job_metadata.request_id)- workflow_log.debug("Workflow start")-- pipe = workflow_arg.pipe- workflow_log.verbose(f"Routing {pipe.__class__.__name__} pipe '{workflow_arg.pipe.code}': {pipe.description}")-- # Run-scoped identity for ALL per-run worker-local state (library id, tracer key,- # event-log context key) and for event/node-id stamps: run_id is replay-stable but- # unique per run, so a reused workflow id (workflow-level retry_policy, Temporal- # reset, resubmission of the same pipeline_run_id) can never collide with a live- # successor run's state on the same worker.- wf_run_id = workflow.info().run_id-- # Set up per-workflow library if a library crate is present- library_crate = workflow_arg.library_crate- wf_library_id: str | None = None-- # Per-workflow tracing state (declared before try for finally block access).- # schedule_flush is the payload-pure flush gate, computed in the tracing setup- # block below (see the determinism comments there and in the finally).- event_log = None- wf_tracer_key: str | None = None- schedule_flush: bool = False- trace_context = workflow_arg.job_metadata.trace_context-- pipe_output: PipeOutput | None = None-- try:- if library_crate is not None:- # 1. Create per-workflow ClassRegistry pre-seeded from global- global_registry = KajsonManager.get_class_registry()- workflow_registry = ClassRegistry()- workflow_registry.register_classes_dict(global_registry.get_classes_dict())-- # 2. Open library and attach registry to it. The library id is keyed by run_id —- # replay-stable (a replay of this run sees the same run_id) but unique per run.- # Keying by workflow_id would collide across runs: workflow ids are reused by- # workflow-level retry_policy, Temporal reset, and resubmission of the same- # pipeline_run_id (make_workflow_id is deterministic), so a closed predecessor- # run's late eviction cleanup could tear down a live successor run's library.- # With run_id keying, a pre-existing library under this id can only be THIS run's- # own leftover — an evicted/interrupted execution whose finally never ran (H2):- # reusing it would fingerprint-skip the crate load below against the fresh- # registry, so the crate's dynamic classes never land in it and inline hydration- # fails where history recorded success. open_fresh_library tears any such- # leftover down — pure in-memory, no commands emitted, so it is replay-safe.- library_manager = get_library_manager()- wf_library_id = f"wf_{wf_run_id}"- wf_library = library_manager.open_fresh_library(library_id=wf_library_id)- wf_library.set_class_registry(workflow_registry)- set_current_library(library_id=wf_library_id)-- # 3. Load crate (registers dynamic classes into workflow_registry via hub.get_class_registry())- library_manager.load_from_crate(library_id=wf_library_id, crate=library_crate)-- # 4. Hydrate WorkingMemory if needed- if workflow_arg.working_memory_raw is not None:- workflow_arg.working_memory = hydrate_working_memory(workflow_arg.working_memory_raw)- workflow_arg.working_memory_raw = None-- # Set up per-workflow graph tracing if enabled.- pipeline_run_id = workflow_arg.job_metadata.pipeline_run_id-- # Determinism: whether tracing is set up — and therefore whether the finally- # block schedules act_flush_trace_events — must be a pure function of the- # workflow payload, never of worker-local state (which can differ between- # the worker that recorded the history and the one replaying it, causing a- # [TMPRL1100] nondeterminism error). The presence of trace_context IS the- # submitter's decision; the worker-local tracing_config.is_enabled check- # lives in the flush activity, where reading local config is legal. No- # best-effort guard around this block: every step is pure in-memory state- # (no I/O, no config reads) and open_tracer is collision-proof against- # tracer keys leaked by a prior interrupted execution, so a failure here- # is a real bug that must surface — and being inline-deterministic, it- # re-fires identically on replay.- if trace_context is not None:- # Use BufferingEventLog inside workflows (no I/O allowed).- # Events are flushed to the real backend via act_flush_trace_events.- event_log = BufferingEventLog()- wf_tracer_key = wf_run_id- # Payload-pure flush gate: the buffer is populated exclusively by inline- # (workflow-thread) emissions, and since the unified dry run moved leaf- # mocking inside the activities, the only inline source left is graph- # events from the tracer when emit_graph_events is on. Every usage- # emission — LIVE, DRY, or mock-usage — happens activity-side via the- # per-process runner fallback, so with graph events off the buffer is- # deterministically empty and the flush round-trip is skipped. The gate- # input rides in the payload, so the decision replays identically.- schedule_flush = trace_context.emit_graph_events- wf_trace_context = GraphTracerManager.get_or_create_instance().open_tracer(- graph_id=trace_context.graph_id,- data_inclusion=trace_context.data_inclusion,- # D5: only feed the event log to the tracer when graph events are wanted. In- # costs-only mode the tracer still mints node ids but emits no graph events;- # usage events flow via the report delegate's set_event_log below.- event_log=event_log if trace_context.emit_graph_events else None,- workflow_id=wf_run_id,- pipeline_run_id=pipeline_run_id,- tracer_key=wf_tracer_key,- # Threaded in so the returned context is born with the correct flags (no emit-flag- # footgun in the model_copy below).- emit_graph_events=trace_context.emit_graph_events,- emit_usage_events=trace_context.emit_usage_events,- )- # Update job_metadata with the per-workflow trace_context (carries tracer_key + emit- # flags), but preserve parent_node_id from the incoming context so CONTAINS edges link- # back to the parent workflow's controller node.- wf_trace_context = wf_trace_context.model_copy(- update={"parent_node_id": trace_context.parent_node_id},- )- workflow_arg.job_metadata = workflow_arg.job_metadata.model_copy(- update={"trace_context": wf_trace_context},- )- # Configure the report delegate for usage event emission — only when cost reporting- # is on. In graph-only mode no usage context is registered, so usage events are- # suppressed (the runner fallback also gates on emit_usage_events). Only emissions- # from the workflow thread itself would land in this buffer — and since the unified- # dry run moved leaf mocking inside the activities, no such inline source currently- # exists: ReportingManager routes activity-side emissions to its per-process- # fallback even when the activity runs co-located, so the buffer content stays a- # deterministic function of inline execution and re-fires identically on replay.- if trace_context.emit_usage_events:- get_report_delegate().set_event_log(- context_key=wf_tracer_key,- event_log=event_log,- workflow_id=wf_run_id,- pipeline_run_id=pipeline_run_id,- )-- working_memory = workflow_arg.get_working_memory()- pipe_output = await pipe.run_pipe(- job_metadata=workflow_arg.job_metadata,- working_memory=working_memory,- output_name=workflow_arg.output_name,- pipe_run_params=workflow_arg.pipe_run_params,- library_crate=library_crate,- )- except ActivityError as exc:- if isinstance(exc.cause, ApplicationError):- raise TemporalError.from_app_error(exc=exc.cause) from exc- raise- except PipelexError as exc:- # Inline fail-safe floor: a domain error raised inline in workflow code (never via an- # activity) is neither an ActivityError nor an ApplicationError, so without this clause- # it escapes as a non-terminal workflow-task failure and retries indefinitely — a silent- # hang (see docs/under-the-hood/error-model.md "Workflow-Level Fail-Safe Floor"). Convert- # a genuine inline error to a terminal, classified TemporalError; leave one that already- # carries a Temporal failure untouched for the submitter to recover. The propagate-vs-- # convert decision and the invariant it relies on live in _carries_temporal_failure.- # Scoped to PipelexError: transient Temporal/infra errors keep Temporal's task-retry.- # force_non_retryable: an inline error must fail terminally, not trigger a blunt whole-- # workflow retry that re-runs completed inline work — retry belongs at the activity- # boundary. It also keeps this workflow-side conversion config-free (deterministic).- if _carries_temporal_failure(exc):- raise- raise TemporalError.from_message_exception(exc=exc, force_non_retryable=True) from exc- finally:- # ALL worker-local cleanup in this finally is synchronous and runs before the- # awaited flush activity at the end. Eviction safety (H2): the flush await is a- # suspension point where a BaseException can be raised (_WorkflowBeingEvictedError- # on sticky-cache eviction) — it escapes the `except Exception` around the flush- # and aborts the rest of this finally block. Worker-local cleanup (tracer,- # event-log context, per-workflow library + crate fingerprint) must therefore run- # BEFORE that await, so an eviction can no longer leak state keyed by the- # deterministic wf_{run_id} into the worker-local singletons and poison a- # same-worker replay. (Workflow cancellation is NOT such an interruption: at an- # activity await the SDK surfaces it as an ActivityError — an Exception — which- # the best-effort except around the flush swallows.)-- # Close per-workflow graph tracer (collects in-memory graph spec). In costs-only- # mode close_tracer returns None by contract (teardown skips the spec build), so- # the graph_spec gate alone keeps costs-only outputs spec-free.- if wf_tracer_key is not None:- try:- graph_spec = GraphTracerManager.get_or_create_instance().close_tracer(wf_tracer_key)- if graph_spec is not None and pipe_output is not None:- pipe_output.graph_spec = graph_spec- except Exception as tracer_exc: # noqa: BLE001- # Best-effort: tracer close in the finally block must never fail the workflow — log and continue.- workflow_log.warning(f"Failed to close per-workflow tracer: {tracer_exc}")- # Clear stale event log state from the report delegate, under the SAME key- # set_event_log registered it with (clear_event_log is a no-op pop when no- # context was registered, e.g. graph-only mode).- get_report_delegate().clear_event_log(context_key=wf_tracer_key)-- if wf_library_id is not None:- try:- get_library_manager().teardown(library_id=wf_library_id)- except Exception as teardown_exc: # noqa: BLE001- # Best-effort: library teardown in the finally block must never fail the- # workflow or skip the remaining worker-local cleanup below — log and- # continue. LibraryManager forgets the entry pop-first even when the- # library's own teardown raises, so no poisoned state remains.- workflow_log.warning(f"Failed to tear down per-workflow library: {teardown_exc}")- finally:- clear_current_library()-- if event_log is not None:- buffered_events = event_log.drain()- event_log.close()-- # The awaited flush comes LAST: an interruption here can only skip the flush- # itself, never the worker-local cleanup above — and on replay the buffer is- # rebuilt deterministically from inline execution, so the flush re-fires with- # the same content.- #- # Determinism (H1): the schedule is a pure function of the payload — the- # event_log sentinel (trace_context presence) and schedule_flush- # (emit_graph_events, a payload field). It must NOT be gated on the buffer- # CONTENT: anything written into the buffer from outside the workflow- # thread would not be re-written on replay (activities do not re-execute),- # so a content-gated schedule recorded in history could vanish from the- # replayed command stream after a routine sticky-cache eviction- # ([TMPRL1100]). The payload-pure schedule_flush gate only skips- # graph-events-off runs, where the buffer is deterministically empty (see- # the tracing setup block). The activity no-ops on an empty list either way.- if schedule_flush:- try:- await workflow.execute_activity(- act_flush_trace_events,- arg=FlushTraceEventsArg(events=buffered_events),- start_to_close_timeout=timedelta(seconds=30),- retry_policy=RetryPolicy(maximum_attempts=3),- )- except Exception as flush_exc: # noqa: BLE001- # Best-effort: trace-event flush in the finally block must never fail the workflow — log and continue.- workflow_log.warning(f"Failed to flush trace events: {flush_exc}")- elif buffered_events:- # Tripwire, not a guard: with graph events off the buffer is deterministically- # empty by the invariant above, so this branch is unreachable today. If a- # future inline emission path breaks that invariant, these events would be- # silently discarded — turn that into a log line instead.- workflow_log.warning(f"Discarding {len(buffered_events)} buffered trace events: flush skipped by the graph-events-off gate")-- # Dehydrate PipeOutput for Temporal transit: serialize WorkingMemory to- # raw dict so the parent's data converter can deserialize without needing- # dynamic concept classes in its ClassRegistry.- assert pipe_output is not None- pipe_output = pipe_output.prepare_for_temporal(library_crate=library_crate)-- workflow_log.debug("Workflow complete")- return pipe_outputdiff --git a/pipelex/temporal/tprl_pipe/wf_pipe_run.py b/pipelex/temporal/tprl_pipe/wf_pipe_run.pydeleted file mode 100644index 108d15876..000000000--- a/pipelex/temporal/tprl_pipe/wf_pipe_run.py+++ /dev/null@@ -1,187 +0,0 @@-from datetime import timedelta--from temporalio import workflow-from temporalio.common import RetryPolicy-from temporalio.exceptions import ActivityError, ChildWorkflowError-from typing_extensions import override--with workflow.unsafe.imports_passed_through():- from pipelex.base_exceptions import ErrorReport, PipelexError # must traverse the workflow sandbox- from pipelex.core.pipes.pipe_output import PipeOutput- from pipelex.pipe_run.delivery_assignment import DeliveryStatus- from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg- from pipelex.temporal.exceptions import WorkflowExecutionError- from pipelex.temporal.log_temporal import WorkflowLog- from pipelex.temporal.tprl.observability import build_search_attributes, build_static_summary- from pipelex.temporal.tprl.temporal_error import TemporalError, recover_error_report- from pipelex.temporal.tprl.workflow_caller import WorkflowClass- from pipelex.temporal.tprl_pipe.act_assemble_tracing import AssembleTracingArg, act_assemble_tracing- from pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg, act_deliver- from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter--[email protected](name="wf_pipe_run")-class WfPipeRun(WorkflowClass[PipeRunArg, PipeOutput]):- """Parent workflow that orchestrates pipe execution + delivery.-- 1. Runs WfPipeRouter as a child workflow to execute the pipe- 2. Runs a delivery activity — always runs, even on failure (to notify the completion Lambda)- """-- @override- @workflow.run- async def run(self, workflow_arg: PipeRunArg) -> PipeOutput:- pipe_job = workflow_arg.pipe_job- # Bound once per invocation: every record below carries this run's- # request_id (None when the run carries no inbound API request id).- workflow_log = WorkflowLog(request_id=pipe_job.job_metadata.request_id)- workflow_log.debug("WfPipeRun start")-- delivery_assignment = workflow_arg.delivery_assignment- pipeline_run_id: str = pipe_job.job_metadata.pipeline_run_id-- # Step 1: Execute the pipe via child workflow- workflow_log.debug(f"Starting child WfPipeRouter for pipe '{pipe_job.pipe.code}'")- status: DeliveryStatus = DeliveryStatus.COMPLETED- pipe_output: PipeOutput | None = None- execution_error: WorkflowExecutionError | None = None- error_report: ErrorReport | None = None-- # The wf_pipe_router child runs the same pipe as wf_pipe_run, so its- # search attributes and static summary are identical — re-derive them- # from the same pipe_job. Dispatched via ``workflow.execute_child_workflow``- # directly so the recorded ``StartChildWorkflowExecution`` command is a- # pure function of the workflow input: no config-derived- # execution_timeout / retry_policy / task_queue smuggled in via the- # ``WorkflowExecutorFactory``, which would change across deploys and- # break determinism on replay after a config edit.- try:- pipe_output = await workflow.execute_child_workflow(- WfPipeRouter.run,- arg=pipe_job,- id=f"{workflow.info().workflow_id}_pipe-router",- search_attributes=build_search_attributes(pipe_job),- static_summary=build_static_summary(pipe_job.pipe),- )- workflow_log.debug("WfPipeRouter completed successfully")- except ChildWorkflowError as exc:- status = DeliveryStatus.FAILED- # Wrap the raw ``ChildWorkflowError`` as ``WorkflowExecutionError`` so the rest of- # this function and the outer ``execute_workflow`` caller continue to see the same- # Pipelex error type as before — the integration test ``test_wf_pipe_run_failure_path``- # pins the workflow_failure_exception_types contract on ``WorkflowExecutionError``.- # ``recover_error_report`` lifts the structured classification out of the child failure- # (``ChildWorkflowError`` exposes it via ``exc.cause``) so the FAILED webhook carries it.- #- # Manual ``__cause__`` wire (not ``raise X from exc``): we hold the wrapped error for a- # deferred re-raise in the post-delivery block below so ``act_deliver`` still fires on- # the failure path. Raising here would short-circuit delivery.- error_report = recover_error_report(exc.cause if exc.cause is not None else exc)- execution_error = WorkflowExecutionError("WfPipeRouter failed", error_report=error_report)- execution_error.__cause__ = exc- workflow_log.error(f"WfPipeRouter failed: {exc}")- except PipelexError as exc:- # Inline fail-safe floor, parent side: a domain error raised inline in this workflow- # (e.g. building the child's search attributes / static summary, evaluated as arguments- # to execute_child_workflow) is not a ChildWorkflowError, so without this clause it- # escapes as a non-terminal workflow-task failure and hangs (see error-model.md- # "Workflow-Level Fail-Safe Floor"). Route it through the same deferred re-raise as a- # child failure so act_deliver still fires the FAILED webhook on the failure path, then- # re-raise terminally below. Scoped to PipelexError: transient infra errors keep task-retry.- status = DeliveryStatus.FAILED- error_report = exc.to_error_report()- execution_error = WorkflowExecutionError(exc.message, error_report=error_report)- # Mint a details-carrying TemporalError as the cause so the classification survives the- # workflow -> submitter serialization. The child-failure path gets this for free (the- # child's TemporalError rides the ChildWorkflowError chain); the inline path has no such- # child failure, so without this carrier the submitter would floor to a synthesized- # UnrecoverableWorkflowFailureError. error_report (above) still feeds the webhook directly.- # force_non_retryable: here the flag does not change WfPipeRun's own retry (the raised- # WorkflowExecutionError drives that, like the child-failure path) — it is set to keep- # this workflow-side conversion config-free (deterministic), matching the router.- execution_error.__cause__ = TemporalError.from_message_exception(exc=exc, force_non_retryable=True)- workflow_log.error(f"WfPipeRun inline failure: {exc}")-- # Step 2: Assemble full graph + usage from trace events (cross-worker)- # Runs as an activity because DynamoDB reads are I/O forbidden in workflows. The dispatch is- # gated on the run's emit flags (F1): a costs-only run assembles usage but no graph, so- # graph_spec stays None — matching DIRECT mode and preserving the --no-graph contract.- trace_context = pipe_job.job_metadata.trace_context- if pipe_output is not None and trace_context is not None and (trace_context.emit_graph_events or trace_context.emit_usage_events):- try:- tracing_assembly = await workflow.execute_activity(- act_assemble_tracing,- arg=AssembleTracingArg(- pipeline_run_id=pipeline_run_id,- domain_code=pipe_job.pipe.domain_code,- main_pipe_code=pipe_job.pipe.code,- assemble_graph=trace_context.emit_graph_events,- assemble_usage=trace_context.emit_usage_events,- ),- start_to_close_timeout=timedelta(seconds=30),- retry_policy=RetryPolicy(maximum_attempts=3),- )- # Mirror DIRECT's assemble_tracing_on_output: copy every populated field, including the- # best-effort *_assembly_error fields. assemble_tracing catches the expected read/assemble- # failures internally and returns them on these fields (no exception raised), so without- # this the failure would be observable in DIRECT but silently lost in TEMPORAL.- if tracing_assembly.graph_spec is not None:- pipe_output.graph_spec = tracing_assembly.graph_spec- if tracing_assembly.graph_assembly_error is not None:- pipe_output.graph_assembly_error = tracing_assembly.graph_assembly_error- if tracing_assembly.tokens_usages is not None:- pipe_output.tokens_usages = tracing_assembly.tokens_usages- if tracing_assembly.usage_assembly_error is not None:- pipe_output.usage_assembly_error = tracing_assembly.usage_assembly_error- except ActivityError as assembly_exc:- workflow_log.warning(f"Tracing assembly failed, continuing with delivery: {assembly_exc}")- # Record the failure only on the concern(s) actually requested, so a costs-only run never- # surfaces a graph_assembly_error (and vice versa).- if trace_context.emit_graph_events:- pipe_output.graph_assembly_error = str(assembly_exc)- if trace_context.emit_usage_events:- pipe_output.usage_assembly_error = str(assembly_exc)-- # Step 3: Run delivery activity if requested — notifies the completion- # Lambda of success or failure when a delivery_assignment was provided.- # No assignment → no delivery (matches PipeRun direct-mode semantics).- delivery_error: ActivityError | None = None- if delivery_assignment is not None:- workflow_log.debug(f"Running delivery: pipeline_run_id={pipeline_run_id}, status={status}")- activity_arg = DeliveryActivityArg(- user_id=pipe_job.job_metadata.user_id,- pipeline_run_id=pipeline_run_id,- delivery_assignment=delivery_assignment,- status=status,- error_report=error_report,- request_id=pipe_job.job_metadata.request_id,- )- # Include pipe_output only on success- if pipe_output is not None:- activity_arg = activity_arg.model_copy(update={"pipe_output": pipe_output})-- try:- await workflow.execute_activity(- act_deliver,- arg=activity_arg,- start_to_close_timeout=timedelta(seconds=60),- retry_policy=RetryPolicy(maximum_attempts=3),- )- workflow_log.debug("Delivery completed")- except ActivityError as activity_error:- # Capture but do not raise yet: when both router and delivery fail- # (e.g., pipe failure + webhook outage) we must preserve the original- # execution_error for failure attribution, matching direct-mode PipeRun.- delivery_error = activity_error- workflow_log.error(f"Delivery activity failed: {activity_error}")-- # Re-raise: prefer the original execution error so failure attribution stays- # correct when delivery also fails. Fall back to the delivery error otherwise.- if execution_error is not None:- raise execution_error- if delivery_error is not None:- raise delivery_error-- assert pipe_output is not None- workflow_log.debug("WfPipeRun complete")- return pipe_outputdiff --git a/pipelex/temporal/worker_cli.py b/pipelex/temporal/worker_cli.pydeleted file mode 100644index e388b6c58..000000000--- a/pipelex/temporal/worker_cli.py+++ /dev/null@@ -1,111 +0,0 @@-"""python -m pipelex.temporal.worker_cli-python -m pipelex.temporal.worker_cli --is-unit-testing-python -m pipelex.temporal.worker_cli --is-not-sandboxed-python -m pipelex.temporal.worker_cli --task-queue my_task_queue-python -m pipelex.temporal.worker_cli --scope router-python -m pipelex.temporal.worker_cli --profile anthropic-tier4 --scope runner-llm --task-queue anthropic_q-"""--import asyncio-import os-from typing import Annotated--import typer--from pipelex import log-from pipelex.config import get_config-from pipelex.pipelex import Pipelex-from pipelex.system.runtime import RunMode, runtime_manager-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.tools.misc.toml_utils import load_toml_from_path--app = typer.Typer()---async def run_worker(- *,- project: str | None = None,- is_not_sandboxed: bool = False,- is_unit_testing: bool = False,- task_queue: str | None = None,- scope_name: str | None = None,- profile_name: str | None = None,-):- if project is None:- log.info(f"Starting worker for current project, from {os.path.relpath(__file__)}")- else:- log.info(f"Starting worker for chosen project '{project}', from {os.path.relpath(__file__)}")- await get_task_manager().run_worker(- is_not_sandboxed=is_not_sandboxed,- is_unit_testing=is_unit_testing,- task_queue=task_queue,- scope_name=scope_name,- profile_name=profile_name,- )---def configure(- project: Annotated[str | None, typer.Argument(help="The project name if you don't want to get it from pyproject.toml")] = None,- is_not_sandboxed: Annotated[bool, typer.Option(help="Flag to run without sandbox")] = False,- is_unit_testing: Annotated[bool, typer.Option(help="Flag to indicate if running unit tests")] = False,- task_queue: Annotated[str | None, typer.Option(help="The task queue to use")] = None,- scope: Annotated[str | None, typer.Option(help="Worker scope name from [temporal.worker_scopes.scopes] (defaults to default_scope)")] = None,- profile: Annotated[- str | None,- typer.Option(help="Worker runtime profile name from [temporal.worker_runtime_profiles.profiles] (defaults to default_profile)"),- ] = None,-):- if is_unit_testing:- runtime_manager.set_run_mode(RunMode.UNIT_TEST)- if project is None:- pyproject = load_toml_from_path(path="pyproject.toml")- project = pyproject.get("project", {}).get("name") or pyproject.get("tool", {}).get("poetry", {}).get("name")- if not project:- msg = "Project name not found in pyproject.toml"- raise ValueError(msg)-- Pipelex.make(temporal_enabled=True)-- # Fast-fail on an unknown --task-queue before doing any heavy boot work- # (library load, concept-class generation, kajson registration). Catches- # operator typos in seconds instead of after multi-second library init.- # The same check runs again inside ``TemporalTaskManager.run_worker`` so- # programmatic callers (tests, library code) also benefit.- effective_task_queue = task_queue or get_config().temporal.worker_config.default_task_queue- get_config().temporal.validate_task_queue_known(effective_task_queue)-- # Load base library from PIPELEXPATH at worker startup.- # This generates dynamic concept classes and registers them with Kajson (fixing deserialization)- # and populates the pipe library (fixing get_required_pipe() for controllers).- from pipelex.hub import get_library_manager, resolve_library_dirs, set_current_library # noqa: PLC0415-- library_manager = get_library_manager()- library_manager.open_library(library_id="worker_base")- set_current_library(library_id="worker_base")- effective_dirs, source_label = resolve_library_dirs(library_dirs=None)- if effective_dirs:- log.info(f"Worker loading base library from {len(effective_dirs)} directory(ies) ({source_label})")- library_manager.load_libraries(library_id="worker_base", library_dirs=effective_dirs)- else:- log.info("No library directories configured for worker (PIPELEXPATH not set)")-- if not get_config().temporal.is_enabled:- log.warning("temporal.is_enabled is false in config, but forcing it on for worker mode")- updated_temporal = get_config().temporal.model_copy(update={"is_enabled": True})- get_config().temporal = updated_temporal-- asyncio.run(- run_worker(- project=project,- is_not_sandboxed=is_not_sandboxed,- is_unit_testing=is_unit_testing,- task_queue=task_queue,- scope_name=scope,- profile_name=profile,- )- )---if __name__ == "__main__":- app()diff --git a/pipelex/test_extras/error_report_parity.py b/pipelex/test_extras/error_report_parity.pynew file mode 100644index 000000000..2e6bef1de--- /dev/null+++ b/pipelex/test_extras/error_report_parity.py@@ -0,0 +1,106 @@+"""Single source of truth for the local / Temporal ``ErrorReport`` parity contract.++This module is **shipped in the open-core wheel on purpose**. The parity contract spans two+repos: open core asserts that a failing pipe surfaces a fully classified ``ErrorReport`` on the+local execution path, and the closed ``pipelex-temporal`` plugin asserts the *same*+classification survives the activity -> workflow -> submitter boundary. Both arms import the+fixtures below, so the classification a failing pipe surfaces cannot silently diverge between the+two execution paths — there is one definition, not two hand-synced copies.++Shipping it from core also gives third-party orchestrator-plugin authors a ready-made fixture to+prove their own backend preserves the ``ErrorReport`` across its boundary.++The only genuinely per-repo bit — the ``.mthds`` bundle each repo ships to drive the failing pipe+— is deliberately NOT here. Each arm subclasses these classes and adds its own ``BUNDLE_FILE``.+"""++from typing import ClassVar++from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError, SearchJobFailureError+from pipelex.cogt.inference.error_classification import UserAction, UserActionKind+++class ErrorReportParityTestData:+ """Constants + factory driving both arms of the local / Temporal ``ErrorReport`` parity test.++ A single deliberately-failing ``PipeLLM`` call is the source of truth for both arms: each runs+ the ``native_text_sequence`` pipe with the LLM call mocked to raise ``make_failing_llm_error()``,+ then asserts the resulting ``ErrorReport`` against the same ``EXPECTED_*`` classification.+ """++ PIPE_CODE: ClassVar[str] = "native_text_sequence"++ # The worker-side failure injected into the LLM call. CONFIGURATION is+ # non-retryable, so the Temporal activity is not retried and the workflow+ # fails on the first attempt — the test stays fast and deterministic.+ FAILURE_MESSAGE: ClassVar[str] = "LLM provider rejected the request: invalid API key"+ FAILURE_MODEL: ClassVar[str] = "gpt-4o-mini"+ FAILURE_PROVIDER: ClassVar[str] = "openai"+ FAILURE_CATEGORY: ClassVar[InferenceErrorCategory] = InferenceErrorCategory.CONFIGURATION+ FAILURE_USER_ACTION_DETAIL: ClassVar[str] = "Verify the API key configured for this provider."++ # The classification both arms must surface on the recovered ErrorReport.+ EXPECTED_RETRYABLE: ClassVar[bool] = False+ EXPECTED_USER_ACTION_KIND: ClassVar[UserActionKind] = UserActionKind.CHECK_CREDENTIALS++ @classmethod+ def make_failing_llm_error(cls) -> LLMCompletionError:+ """Build a fresh classified ``LLMCompletionError``.++ Each arm builds its own instance — an exception carries traceback state,+ so a single shared instance must not be reused across runs.+ """+ error = LLMCompletionError(+ cls.FAILURE_MESSAGE,+ error_category=cls.FAILURE_CATEGORY,+ user_action=UserAction(kind=cls.EXPECTED_USER_ACTION_KIND, detail=cls.FAILURE_USER_ACTION_DETAIL),+ )+ # model_handle / backend_name are declared on CogtError; a real worker fills them+ # at its public-method chokepoint. This test mocks above the worker, so set them+ # directly (mirrors tests/integration/pipelex/cli/agent_cli/test_run_error_chain.py).+ error.model_handle = cls.FAILURE_MODEL+ error.backend_name = cls.FAILURE_PROVIDER+ return error+++class SearchErrorReportParityTestData:+ """Constants + factory driving both arms of the local / Temporal ``ErrorReport`` parity test for web search.++ The search counterpart of :class:`ErrorReportParityTestData`. A single deliberately-failing+ ``PipeSearch`` leaf is the source of truth for both arms: the local arm runs the ``native_search``+ pipe through the direct ``ContentGenerator`` with ``make_search_sourced_answer`` mocked to fail; the+ Temporal arm runs the same pipe through ``WfPipeRouter`` with the activity-side ``search_gen_sourced_answer``+ mocked to fail. Both assert the same ``EXPECTED_*`` classification.+ """++ PIPE_CODE: ClassVar[str] = "native_search"++ # The worker-side failure injected into the search call. CONFIGURATION is non-retryable, so the+ # Temporal activity is not retried and the workflow fails on the first attempt — fast and deterministic.+ FAILURE_MESSAGE: ClassVar[str] = "Search provider rejected the request: connection error"+ FAILURE_MODEL: ClassVar[str] = "linkup/standard"+ FAILURE_PROVIDER: ClassVar[str] = "linkup"+ FAILURE_CATEGORY: ClassVar[InferenceErrorCategory] = InferenceErrorCategory.CONFIGURATION+ FAILURE_USER_ACTION_DETAIL: ClassVar[str] = "Verify the API key configured for this search provider."++ # The classification both arms must surface on the recovered ErrorReport.+ EXPECTED_RETRYABLE: ClassVar[bool] = False+ EXPECTED_USER_ACTION_KIND: ClassVar[UserActionKind] = UserActionKind.CHECK_CREDENTIALS++ @classmethod+ def make_failing_search_error(cls) -> SearchJobFailureError:+ """Build a fresh classified ``SearchJobFailureError``.++ Each arm builds its own instance — an exception carries traceback state, so a single shared+ instance must not be reused across runs.+ """+ error = SearchJobFailureError(+ cls.FAILURE_MESSAGE,+ error_category=cls.FAILURE_CATEGORY,+ user_action=UserAction(kind=cls.EXPECTED_USER_ACTION_KIND, detail=cls.FAILURE_USER_ACTION_DETAIL),+ )+ # model_handle / backend_name are declared on CogtError; a real search worker fills them at its+ # public-method chokepoint. This test mocks above the worker, so set them directly.+ error.model_handle = cls.FAILURE_MODEL+ error.backend_name = cls.FAILURE_PROVIDER+ return errordiff --git a/pyproject.toml b/pyproject.tomlindex dc1870e3e..9b7fdc4fe 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -94,7 +94,6 @@ linkup = ["linkup-sdk>=0.12.0"]mistralai = ["mistralai>=1.12.0"]dynamodb = ["boto3>=1.34.131"]s3 = ["boto3>=1.34.131", "aioboto3>=13.4.0"]-temporal = ["temporalio==1.24.0", "aiohttp>=3.14.0"]docs = ["mike>=2.1.3","mkdocs>=1.6.1",@@ -309,7 +308,6 @@ markers = ["codex_disabled: tests that should not run in Codex","dry_runnable: tests that can be run in dry-run mode","pipelex_api: tests that require access to the Pipelex API",- "temporal: tests that require a Temporal server",][tool.coverage.run]diff --git a/tests/CLAUDE.md b/tests/CLAUDE.mdindex 71b215cd7..9e89258e7 100644--- a/tests/CLAUDE.md+++ b/tests/CLAUDE.md@@ -31,7 +31,6 @@ Replace `<paths>` with the test directories from the mapping below. Concatenate| `pipelex/pipeline/` | `tests/integration/pipelex/pipeline/` || `pipelex/plugins/` | `tests/unit/pipelex/plugins/ tests/integration/pipelex/plugins/` || `pipelex/system/` | `tests/unit/pipelex/system/ tests/integration/pipelex/system/` |-| `pipelex/temporal/` | `tests/unit/pipelex/temporal/ tests/integration/pipelex/temporal/` || `pipelex/tools/` | `tests/unit/pipelex/tools/ tests/integration/pipelex/tools/` |Note: `pipe_controllers/`, `pipe_operators/`, and `pipe_run/` share integration tests at `tests/integration/pipelex/pipes/` -- deduplicate the path when multiple of these areas are touched.diff --git a/tests/integration/pipelex/temporal/library_crate/cv_batch_screening.mthds b/tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening.mthdssimilarity index 100%rename from tests/integration/pipelex/temporal/library_crate/cv_batch_screening.mthdsrename to tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening.mthdsdiff --git a/tests/integration/pipelex/temporal/library_crate/cv_batch_screening_inputs.json b/tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening_inputs.jsonsimilarity index 100%rename from tests/integration/pipelex/temporal/library_crate/cv_batch_screening_inputs.jsonrename to tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening_inputs.jsondiff --git a/tests/e2e/pipelex/cv_batch_screening/test_cv_batch_screening.py b/tests/e2e/pipelex/cv_batch_screening/test_cv_batch_screening.pyindex f51a65441..bb9d9c889 100644--- a/tests/e2e/pipelex/cv_batch_screening/test_cv_batch_screening.py+++ b/tests/e2e/pipelex/cv_batch_screening/test_cv_batch_screening.py@@ -1,9 +1,9 @@"""Direct-mode e2e test for the CV batch screening pipeline.Runs the pipelex-demos example 21 pipeline through ``PipelexMTHDSProtocol`` without any-Temporal layer. The same bundle is also exercised by the temporal in-process test-(``tests/integration/pipelex/temporal/library_crate/test_wf_cv_batch_screening.py``)-and by the ``/temporal-e2e-validate`` skill (distributed 3-process validation).+Temporal layer. The same bundle is also exercised by the temporal in-process test in the+``pipelex-temporal`` package and by the ``/temporal-e2e-validate`` skill (distributed+3-process validation). Each repo keeps its own copy of the crate (see ``test_data.py``)."""from pathlib import Path@@ -17,7 +17,7 @@ from pipelex.core.stuffs.list_content import ListContentfrom pipelex.pipe_run.pipe_run_mode import PipeRunModefrom pipelex.pipeline.runner import PipelexMTHDSProtocolfrom tests.cases.documents import DocumentTestCases-from tests.integration.pipelex.temporal.test_data import CvBatchScreeningTemporalTestData+from tests.e2e.pipelex.cv_batch_screening.test_data import CvBatchScreeningTemporalTestData@pytest.mark.dry_runnablediff --git a/tests/e2e/pipelex/cv_batch_screening/test_data.py b/tests/e2e/pipelex/cv_batch_screening/test_data.pynew file mode 100644index 000000000..4d6c49fdc--- /dev/null+++ b/tests/e2e/pipelex/cv_batch_screening/test_data.py@@ -0,0 +1,43 @@+from typing import ClassVar++# The CV batch screening bundle is shared test data: the direct-mode e2e here, the+# Temporal integration suite (in pipelex-temporal), and the /temporal-e2e-validate skill+# all exercise the same .mthds. Each repo keeps its own copy of the crate so neither+# depends on the other's test tree.+_CRATE_DIR: str = "tests/e2e/pipelex/cv_batch_screening/library_crate"+++class CvBatchScreeningTemporalTestData:+ """Nested-controller CV batch screening pipeline sourced from pipelex-demos example 21.++ Exercises PipeSequence -> PipeSequence -> PipeBatch -> PipeSequence with PipeExtract ++ PipeLLM operators in both the inner job-offer branch and the per-CV batch branch.+ """++ BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/cv_batch_screening.mthds"+ INPUTS_FILE: ClassVar[str] = f"{_CRATE_DIR}/cv_batch_screening_inputs.json"+ PIPE_CODE: ClassVar[str] = "batch_analyze_cvs_for_job_offer"+ DOMAIN: ClassVar[str] = "cv_batch_screening"++ EXPECTED_PIPE_REFS: ClassVar[list[str]] = [+ "cv_batch_screening.batch_analyze_cvs_for_job_offer",+ "cv_batch_screening.prepare_job_offer",+ "cv_batch_screening.extract_one_job_offer",+ "cv_batch_screening.analyze_job_requirements",+ "cv_batch_screening.process_cv",+ "cv_batch_screening.extract_one_cv",+ "cv_batch_screening.analyze_one_cv",+ "cv_batch_screening.analyze_match",+ ]++ EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [+ "job_requirements",+ "match_analyses",+ ]++ EXPECTED_CANDIDATE_MATCH_FIELDS: ClassVar[list[str]] = [+ "match_score",+ "strengths",+ "gaps",+ "overall_assessment",+ ]diff --git a/tests/integration/pipelex/conftest.py b/tests/integration/pipelex/conftest.pyindex 9a66f22d5..fc2be53ce 100644--- a/tests/integration/pipelex/conftest.py+++ b/tests/integration/pipelex/conftest.py@@ -21,7 +21,7 @@ from .fixtures.generator_fixtures import content_generator, generated_content_fafrom .fixtures.img_gen_fixtures import img_gen_job_paramsfrom .fixtures.llm_fixtures import llm_job_params, llm_preset_idfrom .fixtures.model_combo import ModelCombo-from .fixtures.plugin_fixtures import plugin_for_anthropic, plugin_for_openai+from .fixtures.model_handle_fixtures import model_handle_for_anthropic, model_handle_for_openai# Make fixtures available (prevent unused import warnings)__all__ = [@@ -35,9 +35,9 @@ __all__ = ["img_gen_combo","extract_combo","search_combo",- # Plugin fixtures- "plugin_for_openai",- "plugin_for_anthropic",+ # Model handle fixtures+ "model_handle_for_openai",+ "model_handle_for_anthropic",# Image generation fixtures"img_gen_job_params",# Generator fixturesdiff --git a/tests/integration/pipelex/temporal/library_crate/native_search.mthds b/tests/integration/pipelex/error_handling/bundles/native_search.mthdssimilarity index 100%rename from tests/integration/pipelex/temporal/library_crate/native_search.mthdsrename to tests/integration/pipelex/error_handling/bundles/native_search.mthdsdiff --git a/tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds b/tests/integration/pipelex/error_handling/bundles/native_text_sequence.mthdssimilarity index 100%rename from tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthdsrename to tests/integration/pipelex/error_handling/bundles/native_text_sequence.mthdsdiff --git a/tests/integration/pipelex/error_handling/test_data.py b/tests/integration/pipelex/error_handling/test_data.pyindex c9e5f0421..b3ecf356f 100644--- a/tests/integration/pipelex/error_handling/test_data.py+++ b/tests/integration/pipelex/error_handling/test_data.py@@ -1,107 +1,25 @@-"""Shared test data for the local / Temporal ``ErrorReport`` parity pair.+"""Per-repo bundle paths for the local / Temporal ``ErrorReport`` parity pair.-A single deliberately-failing ``PipeLLM`` call is the source of truth for both-arms of the parity check:--- the Temporal full-chain test- (``tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py``),-- the local full-chain test- (``tests/integration/pipelex/error_handling/test_error_report_local_full_chain.py``).--Both run the same ``native_text_sequence`` pipe with the LLM call mocked to raise-``make_failing_llm_error()``, then assert the resulting ``ErrorReport`` against the-same ``EXPECTED_*`` classification constants. Because both arms are pinned to the-same constants, local / Temporal parity holds by construction: if either path-drops a classification field, that arm fails.+The parity-invariant fixtures (the injected failure + its expected classification) and the error+factories are the single shared source of truth in ``pipelex.test_extras.error_report_parity`` —+shipped in the open-core wheel so this repo's local arm and the closed ``pipelex-temporal``+plugin's Temporal arm import the *same* data and cannot silently diverge. Only the ``.mthds``+bundle each repo ships is genuinely per-repo, so that is all these subclasses add."""from typing import ClassVar-from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError, SearchJobFailureError-from pipelex.cogt.inference.error_classification import UserAction, UserActionKind---class ErrorReportParityTestData:- """Constants driving both arms of the local / Temporal ``ErrorReport`` parity test."""-- # The failing pipe: an existing native-Text PipeSequence whose first step is a- # PipeLLM. Reused as-is — no new .mthds bundle needed.- BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds"- PIPE_CODE: ClassVar[str] = "native_text_sequence"-- # The worker-side failure injected into the LLM call. CONFIGURATION is- # non-retryable, so the Temporal activity is not retried and the workflow- # fails on the first attempt — the test stays fast and deterministic.- FAILURE_MESSAGE: ClassVar[str] = "LLM provider rejected the request: invalid API key"- FAILURE_MODEL: ClassVar[str] = "gpt-4o-mini"- FAILURE_PROVIDER: ClassVar[str] = "openai"- FAILURE_CATEGORY: ClassVar[InferenceErrorCategory] = InferenceErrorCategory.CONFIGURATION- FAILURE_USER_ACTION_DETAIL: ClassVar[str] = "Verify the API key configured for this provider."-- # The classification both arms must surface on the recovered ErrorReport.- EXPECTED_RETRYABLE: ClassVar[bool] = False- EXPECTED_USER_ACTION_KIND: ClassVar[UserActionKind] = UserActionKind.CHECK_CREDENTIALS-- @classmethod- def make_failing_llm_error(cls) -> LLMCompletionError:- """Build a fresh classified ``LLMCompletionError``.-- Each arm builds its own instance — an exception carries traceback state,- so a single shared instance must not be reused across runs.- """- error = LLMCompletionError(- cls.FAILURE_MESSAGE,- error_category=cls.FAILURE_CATEGORY,- user_action=UserAction(kind=cls.EXPECTED_USER_ACTION_KIND, detail=cls.FAILURE_USER_ACTION_DETAIL),- )- # model_handle / backend_name are declared on CogtError; a real worker fills them- # at its public-method chokepoint. This test mocks above the worker, so set them- # directly (mirrors tests/integration/pipelex/cli/agent_cli/test_run_error_chain.py).- error.model_handle = cls.FAILURE_MODEL- error.backend_name = cls.FAILURE_PROVIDER- return error---class SearchErrorReportParityTestData:- """Constants driving both arms of the local / Temporal ``ErrorReport`` parity test for web search.+from pipelex.test_extras.error_report_parity import ErrorReportParityTestData as SharedErrorReportParityTestData+from pipelex.test_extras.error_report_parity import SearchErrorReportParityTestData as SharedSearchErrorReportParityTestData- The search counterpart of :class:`ErrorReportParityTestData`. A single deliberately-failing- ``PipeSearch`` leaf is the source of truth for both arms: the local arm runs the ``native_search``- pipe through the direct ``ContentGenerator`` with ``make_search_sourced_answer`` mocked to fail; the- Temporal arm runs the same pipe through ``WfPipeRouter`` with the activity-side ``search_gen_sourced_answer``- mocked to fail. Both assert the same ``EXPECTED_*`` classification, so parity holds by construction.- """- # A single-step PipeSearch bundle over the native SearchResult concept — no inputs, no dynamic classes.- BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate/native_search.mthds"- PIPE_CODE: ClassVar[str] = "native_search"+class ErrorReportParityTestData(SharedErrorReportParityTestData):+ """Local-arm view of the parity fixtures, pointing at this repo's ``native_text_sequence`` bundle."""- # The worker-side failure injected into the search call. CONFIGURATION is non-retryable, so the- # Temporal activity is not retried and the workflow fails on the first attempt — fast and deterministic.- FAILURE_MESSAGE: ClassVar[str] = "Search provider rejected the request: connection error"- FAILURE_MODEL: ClassVar[str] = "linkup/standard"- FAILURE_PROVIDER: ClassVar[str] = "linkup"- FAILURE_CATEGORY: ClassVar[InferenceErrorCategory] = InferenceErrorCategory.CONFIGURATION- FAILURE_USER_ACTION_DETAIL: ClassVar[str] = "Verify the API key configured for this search provider."+ BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/error_handling/bundles/native_text_sequence.mthds"- # The classification both arms must surface on the recovered ErrorReport.- EXPECTED_RETRYABLE: ClassVar[bool] = False- EXPECTED_USER_ACTION_KIND: ClassVar[UserActionKind] = UserActionKind.CHECK_CREDENTIALS- @classmethod- def make_failing_search_error(cls) -> SearchJobFailureError:- """Build a fresh classified ``SearchJobFailureError``.+class SearchErrorReportParityTestData(SharedSearchErrorReportParityTestData):+ """Local-arm view of the search parity fixtures, pointing at this repo's ``native_search`` bundle."""- Each arm builds its own instance — an exception carries traceback state, so a single shared- instance must not be reused across runs.- """- error = SearchJobFailureError(- cls.FAILURE_MESSAGE,- error_category=cls.FAILURE_CATEGORY,- user_action=UserAction(kind=cls.EXPECTED_USER_ACTION_KIND, detail=cls.FAILURE_USER_ACTION_DETAIL),- )- # model_handle / backend_name are declared on CogtError; a real search worker fills them at its- # public-method chokepoint. This test mocks above the worker, so set them directly.- error.model_handle = cls.FAILURE_MODEL- error.backend_name = cls.FAILURE_PROVIDER- return error+ BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/error_handling/bundles/native_search.mthds"diff --git a/tests/integration/pipelex/fixtures/model_handle_fixtures.py b/tests/integration/pipelex/fixtures/model_handle_fixtures.pynew file mode 100644index 000000000..49d33a5f0--- /dev/null+++ b/tests/integration/pipelex/fixtures/model_handle_fixtures.py@@ -0,0 +1,27 @@+"""Model-handle test fixtures."""++import pytest++from pipelex.plugins.model_handle import ModelHandle+++ params=[+ ModelHandle(sdk="openai", backend="openai"),+ ModelHandle(sdk="azure_openai", backend="azure_openai"),+ ],+)+def model_handle_for_openai(request: pytest.FixtureRequest) -> ModelHandle:+ assert isinstance(request.param, ModelHandle)+ return request.param+++ params=[+ ModelHandle(sdk="anthropic", backend="anthropic"),+ ModelHandle(sdk="bedrock_anthropic", backend="bedrock_anthropic"),+ ],+)+def model_handle_for_anthropic(request: pytest.FixtureRequest) -> ModelHandle:+ assert isinstance(request.param, ModelHandle)+ return request.paramdiff --git a/tests/integration/pipelex/fixtures/plugin_fixtures.py b/tests/integration/pipelex/fixtures/plugin_fixtures.pydeleted file mode 100644index 5a89d56e7..000000000--- a/tests/integration/pipelex/fixtures/plugin_fixtures.py+++ /dev/null@@ -1,27 +0,0 @@-"""Plugin-related test fixtures."""--import pytest--from pipelex.plugins.plugin_sdk_registry import Plugin--- params=[- Plugin(sdk="openai", backend="openai"),- Plugin(sdk="azure_openai", backend="azure_openai"),- ],-)-def plugin_for_openai(request: pytest.FixtureRequest) -> Plugin:- assert isinstance(request.param, Plugin)- return request.param--- params=[- Plugin(sdk="anthropic", backend="anthropic"),- Plugin(sdk="bedrock_anthropic", backend="bedrock_anthropic"),- ],-)-def plugin_for_anthropic(request: pytest.FixtureRequest) -> Plugin:- assert isinstance(request.param, Plugin)- return request.paramdiff --git a/tests/integration/pipelex/plugins/test_anthropic.py b/tests/integration/pipelex/plugins/test_anthropic.pyindex 1de1ea625..bffac6a6f 100644--- a/tests/integration/pipelex/plugins/test_anthropic.py+++ b/tests/integration/pipelex/plugins/test_anthropic.py@@ -6,7 +6,7 @@ from rich.table import Tablefrom pipelex.hub import get_console, get_models_managerfrom pipelex.plugins.anthropic.anthropic_exceptions import AnthropicSDKUnsupportedErrorfrom pipelex.plugins.anthropic.anthropic_llms import anthropic_list_available_models-from pipelex.plugins.plugin_sdk_registry import Plugin+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.system.environment import all_env_vars_are_set, any_env_var_is_placeholderfrom tests.integration.pipelex.plugins.conftest import is_backend_available@@ -24,18 +24,18 @@ class TestAnthropic:async def test_anthropic_list_models(self,pytestconfig: pytest.Config,- plugin_for_anthropic: Plugin,+ model_handle_for_anthropic: ModelHandle,):- if not is_backend_available(plugin_for_anthropic.backend):- pytest.skip(f"Backend '{plugin_for_anthropic.backend}' is not available or enabled")+ if not is_backend_available(model_handle_for_anthropic.backend):+ pytest.skip(f"Backend '{model_handle_for_anthropic.backend}' is not available or enabled")if not all_env_vars_are_set(keys=REQUIRED_ENV_VARS):pytest.skip(f"Some key(s) missing amongst {REQUIRED_ENV_VARS}")if any_env_var_is_placeholder(REQUIRED_ENV_VARS):pytest.skip(f"Some key(s) among {REQUIRED_ENV_VARS} are a placeholder, can't be used to test listing models")try:- backend = get_models_manager().get_required_inference_backend(plugin_for_anthropic.backend)+ backend = get_models_manager().get_required_inference_backend(model_handle_for_anthropic.backend)anthropic_models_list = await anthropic_list_available_models(- plugin=plugin_for_anthropic,+ model_handle=model_handle_for_anthropic,backend=backend,)except AuthenticationError as auth_exc:diff --git a/tests/integration/pipelex/plugins/test_openai.py b/tests/integration/pipelex/plugins/test_openai.pyindex 8e91ee774..047901a58 100644--- a/tests/integration/pipelex/plugins/test_openai.py+++ b/tests/integration/pipelex/plugins/test_openai.py@@ -2,8 +2,8 @@ import pytestfrom pipelex import pretty_printfrom pipelex.hub import get_models_manager+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.plugins.openai.openai_llms import openai_list_available_models-from pipelex.plugins.plugin_sdk_registry import Pluginfrom pipelex.system.environment import all_env_vars_are_set, any_env_var_is_placeholderfrom tests.integration.pipelex.plugins.conftest import is_backend_available@@ -17,29 +17,29 @@ class TestOpenAI:async def test_openai_list_available_models(self,pytestconfig: pytest.Config,- plugin_for_openai: Plugin,+ model_handle_for_openai: ModelHandle,):- if not is_backend_available(plugin_for_openai.backend):- pytest.skip(f"Backend '{plugin_for_openai.backend}' is not available or enabled")- match plugin_for_openai.backend:+ if not is_backend_available(model_handle_for_openai.backend):+ pytest.skip(f"Backend '{model_handle_for_openai.backend}' is not available or enabled")+ match model_handle_for_openai.backend:case "openai":required_env_vars = ["OPENAI_API_KEY"]case "azure_openai":required_env_vars = ["AZURE_API_KEY", "AZURE_API_BASE", "AZURE_API_VERSION"]case _:- msg = f"Plugin {plugin_for_openai} is not supported in this test"+ msg = f"Model handle {model_handle_for_openai} is not supported in this test"raise ValueError(msg)if not all_env_vars_are_set(keys=required_env_vars):pytest.skip(f"Some key(s) missing amongst {required_env_vars}")if any_env_var_is_placeholder(required_env_vars):pytest.skip(f"Some key(s) among {required_env_vars} are a placeholder, can't be used to test listing models")- backend = get_models_manager().get_required_inference_backend(plugin_for_openai.backend)+ backend = get_models_manager().get_required_inference_backend(model_handle_for_openai.backend)openai_models_list = await openai_list_available_models(- plugin=plugin_for_openai,+ model_handle=model_handle_for_openai,backend=backend,)if pytestconfig.get_verbosity() >= 2:list_of_ids = [model.id for model in openai_models_list]- pretty_print(list_of_ids, title=f"models available for {plugin_for_openai}")+ pretty_print(list_of_ids, title=f"models available for {model_handle_for_openai}")pretty_print(openai_models_list)diff --git a/tests/integration/pipelex/runtime_bridge/conftest.py b/tests/integration/pipelex/runtime_bridge/conftest.pydeleted file mode 100644index f37bf871e..000000000--- a/tests/integration/pipelex/runtime_bridge/conftest.py+++ /dev/null@@ -1,40 +0,0 @@-from collections.abc import Generator-from pathlib import Path--import pytest--from pipelex.hub import get_func_registry, get_library_manager, set_current_library-from tests.integration.pipelex.runtime_bridge.test_data.bridge_funcs import (- mistralai_workflows_bridge_echo,- mistralai_workflows_bridge_raise,-)--TEST_DATA_DIR = Path(__file__).parent / "test_data"--[email protected](scope="class")-def bridge_test_library() -> Generator[str, None, None]:- """Open a class-scoped library populated with the bridge test pipe.-- The pipe ``mistralai_workflows_bridge_test.bridge_func_pipe`` is registered- in the active library, and the matching Python function is registered in- the FuncRegistry. Both are torn down on exit.- """- func_registry = get_func_registry()- func_registry.register_function(mistralai_workflows_bridge_echo)- func_registry.register_function(mistralai_workflows_bridge_raise)-- library_manager = get_library_manager()- library_id, _ = library_manager.open_library()- set_current_library(library_id=library_id)- library_manager.load_libraries(- library_id=library_id,- library_dirs=[TEST_DATA_DIR],- )- try:- yield library_id- finally:- library_manager.teardown(library_id=library_id)- for func_name in ("mistralai_workflows_bridge_echo", "mistralai_workflows_bridge_raise"):- if func_registry.has_function(func_name):- func_registry.unregister_function_by_name(func_name)diff --git a/tests/integration/pipelex/runtime_bridge/test_bridge_direct.py b/tests/integration/pipelex/runtime_bridge/test_bridge_direct.pydeleted file mode 100644index 628035411..000000000--- a/tests/integration/pipelex/runtime_bridge/test_bridge_direct.py+++ /dev/null@@ -1,272 +0,0 @@-"""Layer-1 integration tests for the runtime-bridge in DIRECT mode.--These tests do NOT depend on any host-runtime optional package — they-exercise only the framework-agnostic core (``run_pipe_via_bridge`` with a-real loaded pipe). The Mistral Workflows activity wrapper is covered-separately in the ``pipelex-mistralai-workflows`` package, which DOES-require the optional dep.-"""--import json-from typing import Any--import pytest-from kajson.kajson_manager import KajsonManager--from pipelex.graph.graph_config import DataInclusionConfig-from pipelex.graph.trace_context import TraceContext-from pipelex.hub import get_current_library_id_or_none, get_library_manager-from pipelex.runtime_bridge.bridge import (- PipelexPipeRunInput,- build_pipe_job_from_input,- run_pipe_via_bridge,-)-from pipelex.runtime_bridge.exceptions import PipelexBridgeDispatchError-from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode--PIPE_REF = "mistralai_workflows_bridge_test.bridge_func_pipe"-ENVELOPE_PIPE_REF = "mistralai_workflows_bridge_test.bridge_envelope_pipe"-RAISE_PIPE_REF = "mistralai_workflows_bridge_test.bridge_raise_pipe"--[email protected](loop_scope="class")-class TestBridgeDirect:- async def test_direct_mode_with_globally_loaded_library(- self,- bridge_test_library: str, # noqa: ARG002- ) -> None:- """Bridge runs a pipe found in the active library when no crate is provided."""- result = await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code=PIPE_REF,- inputs={"input_text": "hello world"},- execution_mode=PipelexExecutionMode.DIRECT,- )- )-- assert result.is_completed is True- assert result.workflow_id is None- assert result.main_stuff_name is not None- main_stuff_dump = result.output_dict["root"][result.main_stuff_name]- assert main_stuff_dump["content"]["text"] == "hello world"- # Usage fields exist on the boundary DTO (None here — this run does no inference- # and opens no tracer, so the cost-assembly path produces nothing).- assert result.tokens_usages_dump is None- assert result.usage_assembly_error is None-- async def test_direct_mode_with_library_crate_dump(- self,- bridge_test_library: str,- ) -> None:- """Bridge round-trips through ``library_crate_dump`` end-to-end.-- Captures a LibraryCrate from the loaded library, pipes it through the- bridge as a JSON-safe dict, and verifies the pipe still resolves and- runs against the per-call scoped library that the bridge opens.- """- crate = get_library_manager().get_crate(library_id=bridge_test_library)- assert crate is not None- crate_dump: dict[str, Any] = crate.model_dump(mode="json")-- result = await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code=PIPE_REF,- inputs={"input_text": "via crate"},- library_crate_dump=crate_dump,- execution_mode=PipelexExecutionMode.DIRECT,- )- )-- assert result.is_completed is True- assert result.main_stuff_name is not None- main_stuff_dump = result.output_dict["root"][result.main_stuff_name]- assert main_stuff_dump["content"]["text"] == "via crate"-- async def test_direct_mode_uses_caller_pipeline_run_id(- self,- bridge_test_library: str, # noqa: ARG002- ) -> None:- """Caller-supplied ``pipeline_run_id`` propagates to the PipeJob."""- caller_run_id = "caller-supplied-run-id"- result = await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code=PIPE_REF,- inputs={"input_text": "trace me"},- pipeline_run_id=caller_run_id,- execution_mode=PipelexExecutionMode.DIRECT,- )- )-- assert result.is_completed is True- assert result.pipeline_run_id == caller_run_id-- async def test_direct_mode_dynamic_concept_round_trips_via_library_crate_dump(- self,- bridge_test_library: str,- ) -> None:- """A concept with an inline structure round-trips through ``library_crate_dump``.-- ``EchoEnvelope`` is defined inline in the bridge_test bundle. The bridge- dehydrates the library to a JSON-safe crate dump, opens a per-call- scoped library on the receiving side, and re-hydrates the concept so- ``PipeCompose`` can construct a ``StructuredContent`` matching the- dynamic shape.- """- envelope_pipe_ref = "mistralai_workflows_bridge_test.bridge_envelope_pipe"- crate = get_library_manager().get_crate(library_id=bridge_test_library)- assert crate is not None- crate_dump: dict[str, Any] = crate.model_dump(mode="json")-- result = await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code=envelope_pipe_ref,- inputs={"input_text": "wrapped"},- library_crate_dump=crate_dump,- execution_mode=PipelexExecutionMode.DIRECT,- )- )-- assert result.is_completed is True- assert result.main_stuff_name is not None- main_stuff = result.output_dict["root"][result.main_stuff_name]- content = main_stuff["content"]- assert content["text"] == "wrapped"- assert content["origin"] == "mistralai_workflows_bridge"-- async def test_crate_run_restores_prior_current_library(- self,- bridge_test_library: str,- ) -> None:- """A crate-scoped bridge run restores the caller's current library on exit.-- The bridge opens a per-call scoped library for the crate. When it- returns, the previously active library (the fixture's) must be restored,- not cleared — otherwise a later no-crate bridge call in the same async- context could no longer resolve pipes.- """- assert get_current_library_id_or_none() == bridge_test_library-- crate = get_library_manager().get_crate(library_id=bridge_test_library)- assert crate is not None- crate_dump: dict[str, Any] = crate.model_dump(mode="json")-- await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code=PIPE_REF,- inputs={"input_text": "restore me"},- library_crate_dump=crate_dump,- execution_mode=PipelexExecutionMode.DIRECT,- )- )-- assert get_current_library_id_or_none() == bridge_test_library-- async def test_run_id_derives_from_trace_context_when_pipeline_run_id_omitted(- self,- bridge_test_library: str, # noqa: ARG002- ) -> None:- """With a trace_context but no pipeline_run_id, the run id adopts the lookup_key.-- The caller's tracer is registered (and later closed) under trace_context.lookup_key- (tracer_key or graph_id), and tracing assembly reads events by job_metadata.pipeline_run_id.- Minting a fresh id would split the two — dropping the run's graph + cost data and leaking- the tracer — so the bridge adopts the lookup_key. On every reachable path lookup_key ==- graph_id (keyed tracers are a Temporal-child construct that never reaches the bridge — see- wip/runtime-bridge/bridge-keyed-tracer-unsupported.md). An explicit pipeline_run_id still- wins; absent both, a fresh id is minted.- """- data_inclusion = DataInclusionConfig(- stuff_json_content=False,- stuff_text_content=False,- stuff_html_content=False,- error_stack_traces=False,- pipe_and_concept_registry=False,- )- trace_context = TraceContext(graph_id="host-graph-id", data_inclusion=data_inclusion)-- # No tracer_key: lookup_key == graph_id.- job_from_trace = build_pipe_job_from_input(- input_payload=PipelexPipeRunInput(pipe_code=PIPE_REF, inputs={"input_text": "x"}),- library_crate=None,- trace_context=trace_context,- )- assert job_from_trace.job_metadata.pipeline_run_id == "host-graph-id"-- # tracer_key set: run id is the lookup_key (tracer_key), so tracer open/close stay aligned.- keyed_trace_context = TraceContext(graph_id="host-graph-id", tracer_key="host-tracer-key", data_inclusion=data_inclusion)- job_from_tracer_key = build_pipe_job_from_input(- input_payload=PipelexPipeRunInput(pipe_code=PIPE_REF, inputs={"input_text": "x"}),- library_crate=None,- trace_context=keyed_trace_context,- )- assert job_from_tracer_key.job_metadata.pipeline_run_id == "host-tracer-key"-- job_explicit = build_pipe_job_from_input(- input_payload=PipelexPipeRunInput(pipe_code=PIPE_REF, inputs={"input_text": "x"}, pipeline_run_id="explicit-id"),- library_crate=None,- trace_context=trace_context,- )- assert job_explicit.job_metadata.pipeline_run_id == "explicit-id"-- job_minted = build_pipe_job_from_input(- input_payload=PipelexPipeRunInput(pipe_code=PIPE_REF, inputs={"input_text": "x"}),- library_crate=None,- trace_context=None,- )- assert job_minted.job_metadata.pipeline_run_id- assert job_minted.job_metadata.pipeline_run_id != "host-graph-id"-- async def test_direct_mode_wraps_user_code_exception_in_dispatch_error(- self,- bridge_test_library: str, # noqa: ARG002- ) -> None:- """A non-Pipelex exception from user pipe code surfaces as PipelexBridgeDispatchError.-- The PipeFunc operator wraps the user function's ValueError into a- PipeRunError, which the bridge converts to its boundary error type — so a- raw ValueError never escapes the bridge across the host-runtime seam.- """- with pytest.raises(PipelexBridgeDispatchError):- await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code=RAISE_PIPE_REF,- inputs={"input_text": "boom"},- execution_mode=PipelexExecutionMode.DIRECT,- )- )-- async def test_crate_inline_concept_does_not_leak_into_global_registry(- self,- bridge_test_library: str,- ) -> None:- """Classes generated from a crate's inline concepts stay out of the global registry.-- The bridge loads the crate into a per-call scoped ``ClassRegistry`` so a- long-lived / multi-tenant host doesn't accumulate (or collide on) dynamic- concept classes across crate versions. We rename the bundle's inline- ``EchoEnvelope`` concept to a probe name that has never been loaded- globally, so any presence in the global registry afterwards is- unambiguously a leak from the bridge's crate path.- """- crate = get_library_manager().get_crate(library_id=bridge_test_library)- assert crate is not None- crate_dump: dict[str, Any] = crate.model_dump(mode="json")- probe_dump: dict[str, Any] = json.loads(json.dumps(crate_dump).replace("EchoEnvelope", "EchoEnvelopeProbe"))-- global_registry = KajsonManager.get_class_registry()- before = set(global_registry.get_classes_dict().keys())- assert not any("EchoEnvelopeProbe" in name for name in before), "Probe class must not be globally loaded before the run"-- result = await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code=ENVELOPE_PIPE_REF,- inputs={"input_text": "probe"},- library_crate_dump=probe_dump,- execution_mode=PipelexExecutionMode.DIRECT,- )- )-- assert result.is_completed is True- after = set(global_registry.get_classes_dict().keys())- leaked = {name for name in (after - before) if "EchoEnvelopeProbe" in name}- assert not leaked, f"Crate-generated class(es) leaked into the global Kajson registry: {leaked}"diff --git a/tests/integration/pipelex/runtime_bridge/test_data/bridge_funcs.py b/tests/integration/pipelex/runtime_bridge/test_data/bridge_funcs.pydeleted file mode 100644index 492caebde..000000000--- a/tests/integration/pipelex/runtime_bridge/test_data/bridge_funcs.py+++ /dev/null@@ -1,25 +0,0 @@-"""Test functions registered for the runtime-bridge integration tests."""--from pipelex.core.memory.working_memory import WorkingMemory-from pipelex.core.stuffs.text_content import TextContent---def mistralai_workflows_bridge_echo(working_memory: WorkingMemory) -> TextContent:- """Echo the ``input_text`` stuff back as a TextContent output.-- Used by tests/integration/pipelex/runtime_bridge to validate end-to-end- pipe execution through the bridge without invoking inference.- """- input_text = working_memory.get_stuff_as_str("input_text")- return TextContent(text=input_text)---def mistralai_workflows_bridge_raise(working_memory: WorkingMemory) -> TextContent: # noqa: ARG001- """Raise a plain ``ValueError`` to exercise the bridge's error boundary.-- The PipeFunc operator wraps any exception from user code into a- ``PipeRunError``, which the bridge converts to ``PipelexBridgeDispatchError`` —- so a raw ``ValueError`` never escapes the bridge across the host-runtime seam.- """- msg = "intentional failure from bridge test func"- raise ValueError(msg)diff --git a/tests/integration/pipelex/runtime_bridge/test_data/bridge_test.mthds b/tests/integration/pipelex/runtime_bridge/test_data/bridge_test.mthdsdeleted file mode 100644index 28d390eb1..000000000--- a/tests/integration/pipelex/runtime_bridge/test_data/bridge_test.mthds+++ /dev/null@@ -1,62 +0,0 @@-domain = "mistralai_workflows_bridge_test"-description = "Test pipes for the mistralai_workflows plugin bridge"--[concept.EchoEnvelope]-description = "Custom dynamic concept used to exercise library_crate_dump round-trip for inline-structured concepts."--[concept.EchoEnvelope.structure]-text = { type = "text", required = true, description = "The echoed text" }-origin = { type = "text", required = true, description = "Origin marker for the echo" }--[pipe.bridge_func_pipe]-type = "PipeFunc"-description = "Echoes the input text back as output (DIRECT mode only — PipeFunc is not Temporal-compatible)"-output = "Text"-function_name = "mistralai_workflows_bridge_echo"--[pipe.bridge_raise_pipe]-type = "PipeFunc"-description = "Raises a plain ValueError to exercise the bridge error boundary (DIRECT mode only)"-output = "Text"-function_name = "mistralai_workflows_bridge_raise"--[pipe.bridge_compose_pipe]-type = "PipeCompose"-description = "Echoes the input_text via a Jinja2 template (Temporal-compatible)"-inputs = { input_text = "Text" }-output = "Text"-template = "{{ input_text.text }}"--[pipe.bridge_envelope_pipe]-type = "PipeCompose"-description = "Composes the input_text into a structured EchoEnvelope, exercising dynamic-concept round-trip via library_crate_dump"-inputs = { input_text = "Text" }-output = "EchoEnvelope"--[pipe.bridge_envelope_pipe.construct]-text = { from = "input_text.text" }-origin = "mistralai_workflows_bridge"--[pipe.bridge_seq_step_one]-type = "PipeCompose"-description = "First step of bridge_sequence_pipe — uppercases the input text"-inputs = { input_text = "Text" }-output = "Text"-template = "{{ input_text.text | upper }}"--[pipe.bridge_seq_step_two]-type = "PipeCompose"-description = "Second step of bridge_sequence_pipe — wraps the result with markers"-inputs = { uppercased = "Text" }-output = "Text"-template = "[STEP2:{{ uppercased.text }}]"--[pipe.bridge_sequence_pipe]-type = "PipeSequence"-description = "Two-step sequence used by Phase 2.1 streaming tests to assert per-step events"-inputs = { input_text = "Text" }-output = "Text"-steps = [- { pipe = "bridge_seq_step_one", result = "uppercased" },- { pipe = "bridge_seq_step_two", result = "final_text" },-]diff --git a/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.pynew file mode 100644index 000000000..2d929bcb7--- /dev/null+++ b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py@@ -0,0 +1,95 @@+"""Boot-time hub-slot resolution: injection precedence (codex C8) + teardown LIFO.++At each process-global hub slot, ``Pipelex.setup`` resolves: explicit ``setup()`` param > plugin+slot-claim thunk > core default. The slot claim must never silently override an explicit injection.+Teardown runs the plugin-registered teardown callbacks LIFO. These are exercised here with a fake+registrar (no Temporal needed); the real Temporal boot-via-slots path is covered end-to-end by+``test_keyless_boot_forced_dry`` and the Temporal integration suite.+"""++from collections.abc import Generator+from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest+from pytest_mock import MockerFixture++from pipelex.hub import get_content_generator, get_pipe_router+from pipelex.pipelex import Pipelex+from pipelex.plugins.registrar import HubSlot, PluginRegistrar+from pipelex.system.runtime import IntegrationMode, runtime_manager++if TYPE_CHECKING:+ from pipelex.system.configuration.configs import PipelexConfig++[email protected](autouse=True)+def reset_pipelex_config_fixture() -> Generator[None, None, None]:+ """Override the global module fixture: this module boots per test and tears down."""+ Pipelex.teardown_if_needed()+ yield+ Pipelex.teardown_if_needed()+++def _test_integration_mode() -> IntegrationMode:+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST+++def _fake_registrar(mocker: MockerFixture) -> PluginRegistrar:+ """A registrar with empty registries (boot stores them but doesn't consume them during setup)."""+ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(temporal=SimpleNamespace(is_enabled=False))))+ mocker.patch("pipelex.pipelex.build_registrar", return_value=registrar)+ return registrar+++class TestHubSlotInjectionPrecedence:+ def test_slot_claim_provides_content_generator_when_no_explicit_param(self, mocker: MockerFixture) -> None:+ """With no explicit param, a claimed slot's thunk result wins over the core default."""+ claimed = mocker.sentinel.claimed_content_generator+ registrar = _fake_registrar(mocker)+ registrar.slot_claims[HubSlot.CONTENT_GENERATOR] = lambda: claimed++ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)++ assert get_content_generator() is claimed++ def test_explicit_content_generator_wins_over_slot_claim(self, mocker: MockerFixture) -> None:+ """An explicit setup() param wins over a plugin slot claim — the claim must not override it."""+ claimed = mocker.sentinel.claimed_content_generator+ explicit = mocker.sentinel.explicit_content_generator+ registrar = _fake_registrar(mocker)+ registrar.slot_claims[HubSlot.CONTENT_GENERATOR] = lambda: claimed++ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, content_generator=explicit) # pyright: ignore[reportArgumentType]++ assert get_content_generator() is explicit++ def test_explicit_pipe_router_wins_over_slot_claim(self, mocker: MockerFixture) -> None:+ """The explicit pipe_router param wins over a PIPE_ROUTER slot claim."""+ claimed_router = mocker.sentinel.claimed_router+ explicit_router = mocker.sentinel.explicit_router+ registrar = _fake_registrar(mocker)+ registrar.slot_claims[HubSlot.PIPE_ROUTER] = lambda: claimed_router+ # Claim PIPE_RUN too so boot does not build a default PipeRun from the sentinel router.+ registrar.slot_claims[HubSlot.PIPE_RUN] = lambda: mocker.sentinel.claimed_run++ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, pipe_router=explicit_router) # pyright: ignore[reportArgumentType]++ assert get_pipe_router() is explicit_router++ def test_teardown_callbacks_run_lifo(self, mocker: MockerFixture) -> None:+ """Plugin-registered teardown callbacks run last-registered-first."""+ order: list[str] = []+ registrar = _fake_registrar(mocker)+ registrar.teardown_callbacks.extend(+ [+ lambda: order.append("first"),+ lambda: order.append("second"),+ lambda: order.append("third"),+ ]+ )++ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)+ Pipelex.teardown_if_needed()++ assert order == ["third", "second", "first"]diff --git a/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py b/tests/integration/pipelex/system/test_keyless_boot_forced_dry.pyindex 8e12523c5..977b7ccf4 100644--- a/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py+++ b/tests/integration/pipelex/system/test_keyless_boot_forced_dry.py@@ -23,7 +23,6 @@ from pipelex.pipe_run.pipe_run_mode import PipeRunModefrom pipelex.pipelex import Pipelexfrom pipelex.pipeline.execution_seams import load_libraries_and_activate, prepare_pipe_jobfrom pipelex.system.runtime import IntegrationMode, runtime_manager-from pipelex.temporal.tprl_content_generation.content_generator_in_workflow import ContentGeneratorInWorkflow@pytest.fixture(scope="module", autouse=True)@@ -39,19 +38,18 @@ def _test_integration_mode() -> IntegrationMode:class TestKeylessBootForcedDry:- def _boot_keyless(self, *, temporal_enabled: bool) -> None:+ def _boot_keyless(self) -> None:Pipelex.teardown_if_needed()Pipelex.make(integration_mode=_test_integration_mode(),needs_inference=False,- temporal_enabled=temporal_enabled,)@pytest.mark.asyncioasync def test_keyless_direct_boot_inline_generator_and_forced_dry(self) -> None:"""Keyless + direct: inline generator, forced-DRY flag set, and a LIVE-requested job prepared as DRY."""try:- self._boot_keyless(temporal_enabled=False)+ self._boot_keyless()assert is_dry_run_forced()assert isinstance(get_content_generator(), ContentGenerator)@@ -78,15 +76,6 @@ class TestKeylessBootForcedDry:finally:Pipelex.teardown_if_needed()- def test_keyless_temporal_boot_in_workflow_generator_and_forced_dry(self) -> None:- """Keyless + Temporal-enabled: the in-workflow generator is still selected (backend-keyed, D4)."""- try:- self._boot_keyless(temporal_enabled=True)- assert is_dry_run_forced()- assert isinstance(get_content_generator(), ContentGeneratorInWorkflow)- finally:- Pipelex.teardown_if_needed()-def test_keyed_boot_does_not_force_dry(self) -> None:"""Control arm: a normal boot leaves the flag unset."""try:diff --git a/tests/integration/pipelex/temporal/conftest.py b/tests/integration/pipelex/temporal/conftest.pydeleted file mode 100644index 75d23e027..000000000--- a/tests/integration/pipelex/temporal/conftest.py+++ /dev/null@@ -1,270 +0,0 @@-import faulthandler-import logging-import os-from pathlib import Path-from typing import AsyncGenerator, Generator, cast--import pytest-import pytest_asyncio-from pytest import FixtureRequest, Parser-from temporalio.client import Client as TemporalClient-from temporalio.testing import WorkflowEnvironment--from pipelex import log-from pipelex.pipelex import Pipelex-from pipelex.system.runtime import IntegrationMode, runtime_manager-from pipelex.temporal.config_temporal import BUILTIN_SEARCH_ATTRIBUTES-from pipelex.temporal.tasks import Tasks-from pipelex.temporal.temporal_connect import connect_to_temporal_selected_server-from pipelex.temporal.temporal_data_converter import data_converter-from pipelex.temporal.temporal_hub import temporal_hub-from pipelex.temporal.temporal_task_manager import TemporalTaskManager-from pipelex.temporal.tprl.namespace_check import RegistrationFailure, ensure_required_search_attributes_registered-from pipelex.test_extras.shared_pytest_plugins import ClassRegistryMode-from pipelex.tracing.activity_event_log import ActivityEventLogCache--TEMPORAL_SERVER_NONE = "none"-TEMPORAL_SERVER_TIME_SKIPPING = "time-skipping"--HANG_DUMP_DIR_ENV_VAR = "PIPELEX_HANG_DUMP_DIR"-HANG_DUMP_DELAY_SECONDS = 150.0--[email protected](scope="session", autouse=True)-def temporal_failure_log_capture() -> Generator[None, None, None]:- """Mirror temporalio WARNING+ logs (with tracebacks) to a file under the hang-dump dir.-- Enabled only when ``PIPELEX_HANG_DUMP_DIR`` is set (CI hang debugging). A payload-conversion- error inside a workflow task makes Temporal retry the task forever — the test then hangs- until pytest-timeout's thread-method kill, which destroys pytest's captured logs along with- the process. temporalio logs each failed workflow activation WITH the offending traceback,- so mirroring its logger to a file preserves the actual error across the kill.- """- dump_dir_value = os.environ.get(HANG_DUMP_DIR_ENV_VAR)- if not dump_dir_value:- yield- return- dump_dir = Path(dump_dir_value)- dump_dir.mkdir(parents=True, exist_ok=True)- handler = logging.FileHandler(dump_dir / f"temporalio-pid{os.getpid()}.log", encoding="utf-8")- handler.setLevel(logging.WARNING)- handler.setFormatter(logging.Formatter("%(asctime)s %(name)s %(levelname)s %(message)s"))- temporalio_logger = logging.getLogger("temporalio")- temporalio_logger.addHandler(handler)- try:- yield- finally:- temporalio_logger.removeHandler(handler)- handler.close()--[email protected](autouse=True)-def temporal_hang_watchdog(request: FixtureRequest) -> Generator[None, None, None]:- """Dump every thread's stack to a file when a temporal test exceeds the watchdog delay.-- Enabled only when the ``PIPELEX_HANG_DUMP_DIR`` env var is set (CI hang debugging).- The delay sits below pytest-timeout's ``--timeout`` so the dump lands BEFORE the- thread-method kill (``os._exit``) destroys the process — pytest-xdist swallows worker- stderr on "node down", so a file under the dump dir is the only way to see where a- hung test was stuck on a remote runner. One file per worker process; each test appends- an armed/disarmed marker, so the hung test is the last armed entry without a disarm.- """- dump_dir_value = os.environ.get(HANG_DUMP_DIR_ENV_VAR)- if not dump_dir_value:- yield- return- test_id = cast("str", request.node.nodeid) # pyright: ignore[reportUnknownMemberType]- dump_dir = Path(dump_dir_value)- dump_dir.mkdir(parents=True, exist_ok=True)- dump_path = dump_dir / f"hang-pid{os.getpid()}.txt"- with dump_path.open("a", encoding="utf-8") as dump_file:- dump_file.write(f"\n===== armed: {test_id} =====\n")- dump_file.flush()- faulthandler.dump_traceback_later(HANG_DUMP_DELAY_SECONDS, file=dump_file)- try:- yield- finally:- faulthandler.cancel_dump_traceback_later()- dump_file.write(f"===== disarmed: {test_id} =====\n")---def pytest_addoption(parser: Parser) -> None:- parser.addoption(- "--temporal-server",- default=TEMPORAL_SERVER_NONE,- help="Which temporal server to use ('none' for in-process, 'time-skipping', or a profile name from temporal_server_configs)",- )--[email protected](scope="module", autouse=True)-def reset_pipelex_config_fixture() -> Generator[None, None, None]:- """Override root conftest's fixture for Temporal tests.-- Always initializes Pipelex per module so that logging and config are available- for boot_temporal and other fixtures, regardless of --temporal-server mode.- """- Pipelex.make(- integration_mode=IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST,- )- yield- Pipelex.teardown_if_needed()--[email protected](scope="module", autouse=True)-def boot_temporal(reset_pipelex_config_fixture: None) -> Generator[None, None, None]: # noqa: ARG001- """Boot the temporal layer for temporal tests.-- Depends on reset_pipelex_config_fixture to ensure Pipelex is initialized- before we access logging or config. Creates a TemporalTaskManager, populates- the task catalog, and registers it on the temporal_hub so that- get_task_manager() works.- """- manager = TemporalTaskManager()- temporal_hub.set_task_manager(manager)- manager.complement_catalog(- extra_catalog=Tasks.TASK_PACKS,- extra_workflows=[],- extra_activities=[],- )- manager.setup()-- # Install Temporal-aware pipe routers and content generator so that:- # 1. Sub-pipes dispatch as child workflows (not inline in the sandbox)- # 2. Inference calls (LLM, img_gen, etc.) dispatch as activities (not inline)- # This mirrors what a full Temporal-enabled Pipelex.make() would set up.- from pipelex.hub import get_pipelex_hub # noqa: PLC0415- from pipelex.temporal.tprl_content_generation.content_generator_in_workflow_factory import ( # noqa: PLC0415- ContentGeneratorInWorkflowFactory,- )- from pipelex.temporal.tprl_pipe.temporal_pipe_router import make_temporal_pipe_router # noqa: PLC0415-- pipelex_hub = get_pipelex_hub()- pipelex_hub.set_pipe_router(make_temporal_pipe_router())-- pipelex_hub.set_content_generator(ContentGeneratorInWorkflowFactory.make_content_generator_in_workflow())-- yield- manager.teardown()- temporal_hub.reset()-- # Clear cached inference workers and SDK instances between modules.- # When session owns Pipelex (--temporal-server <profile>), the InferenceManager- # persists across modules. Each module's test class uses its own event loop- # (loop_scope="class"), so cached workers hold httpx connections bound to the- # previous module's (now-closed) event loop. Clearing forces fresh workers/clients- # on the next module's event loop.- from pipelex.hub import get_inference_manager, get_plugin_manager # noqa: PLC0415-- get_inference_manager().teardown()- get_plugin_manager().plugin_sdk_registry.teardown()--[email protected](autouse=True)-def reset_activity_event_log_cache() -> Generator[None, None, None]:- """Reset the per-process activity event log cache around every temporal test.-- ReportingManager routes every activity-side usage emission through the per-process- fallback (even co-located activities — see audit finding H1), and the fallback's- event log is process-cached on ``ActivityEventLogCache`` with the traces_dir it was- first built with. Without this reset, a cache created by an earlier test would keep- writing into that test's directory and the current test's events would silently- land elsewhere. Hoisted to this conftest because any temporal test whose activities- report usage engages the cache, not just the ones under ``tracing/``.- """- ActivityEventLogCache.reset_for_tests()- yield- ActivityEventLogCache.reset_for_tests()---@pytest_asyncio.fixture(scope="session") # pyright: ignore[reportUntypedFunctionDecorator, reportUnknownMemberType]-async def env(request: FixtureRequest) -> AsyncGenerator[WorkflowEnvironment, None]:- """Temporal test environment, shared across all temporal tests.-- Uses an in-process server by default (--temporal-server none).- Pass a profile name from temporal_server_configs to connect to a real server.-- For real server modes, Pipelex is temporarily bootstrapped here to read- connection config (session fixtures run before module-scoped ones), then- torn down so reset_pipelex_config_fixture can re-initialize per module.- The gRPC TemporalClient stays alive independently of the Pipelex lifecycle.- """- server_option: str = cast("str", request.config.getoption("--temporal-server"))- workflow_env: WorkflowEnvironment- if server_option == TEMPORAL_SERVER_NONE:- workflow_env = await WorkflowEnvironment.start_local(data_converter=data_converter) # pyright: ignore[reportUnknownMemberType]- elif server_option == TEMPORAL_SERVER_TIME_SKIPPING:- workflow_env = await WorkflowEnvironment.start_time_skipping(data_converter=data_converter)- else:- # Bootstrap Pipelex temporarily to read server connection config.- # If a module-scoped fixture already initialized Pipelex (happens when- # env is lazily triggered after reset_pipelex_config_fixture), skip init- # and just read the config from the existing instance.-- # TODO: Pipelex.make() is heavy — it initializes inference, registries, etc.- # We only need the config here. Design a lightweight boot path (e.g.- # Pipelex.load_config_only()) that parses pipelex.toml without full init.- needs_teardown = False- if Pipelex.get_optional_instance() is None:- Pipelex.make(- integration_mode=IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST,- )- needs_teardown = True- temporal_client = await connect_to_temporal_selected_server(selected_server_config=server_option)- if needs_teardown:- Pipelex.teardown_if_needed()- workflow_env = WorkflowEnvironment.from_client(temporal_client)- # Every Temporal namespace — in-process, time-skipping, or real-server profile —- # starts with no custom search attributes registered, so the cluster's- # StartWorkflowExecution RPC rejects every workflow that sets one. Register- # them here so all temporal tests can start workflows without per-test- # bootstrap. Idempotent: only adds attributes that are missing.- # ``RegistrationFailure`` is returned (not raised) when the API key lacks- # ``AddSearchAttributes`` permission (Temporal Cloud read-only key); surface- # it as a warning so the operator runs ``pipelex setup-temporal-namespace``- # with an admin key before rerunning the tests. For in-process and- # time-skipping servers the caller is always admin, so this branch is- # unreachable there — but handling it uniformly keeps the call site honest- # against the function's full return type.- registration_result = await ensure_required_search_attributes_registered(- temporal_client=workflow_env.client,- namespace=workflow_env.client.namespace,- configured_attributes=BUILTIN_SEARCH_ATTRIBUTES,- )- if isinstance(registration_result, RegistrationFailure):- msg = (- f"Could not auto-register Temporal search attributes on namespace "- f"'{registration_result.namespace}' (missing: {list(registration_result.missing)}). "- f"Workflow starts will fail at dispatch. Run `pipelex setup-temporal-namespace` "- f"with an admin API key, then rerun the tests. RPC error: "- f"{registration_result.rpc_error_message}"- )- log.warning(msg)- yield workflow_env- await workflow_env.shutdown()---@pytest_asyncio.fixture # pyright: ignore[reportUntypedFunctionDecorator, reportUnknownMemberType]-async def temporal_client(env: WorkflowEnvironment) -> TemporalClient: # noqa: RUF029- """Temporal client connected to the test server."""- return env.client---def pytest_generate_tests(metafunc: pytest.Metafunc) -> None:- """Auto-parametrize is_class_registry_isolated based on --class-registry.-- By default (``--class-registry both``), every test class that requests- ``is_class_registry_isolated`` runs twice: once with classes in the global- registry (shared) and once scoped to the library (isolated). Pass- ``--class-registry shared`` or ``--class-registry isolated`` to force a- single mode (useful for debugging).- """- if "is_class_registry_isolated" not in metafunc.fixturenames:- return- mode = ClassRegistryMode(metafunc.config.getoption("--class-registry"))- match mode:- case ClassRegistryMode.BOTH:- metafunc.parametrize("is_class_registry_isolated", [False, True], ids=["shared", "isolated"], scope="class")- case ClassRegistryMode.ISOLATED:- metafunc.parametrize("is_class_registry_isolated", [True], ids=["isolated"], scope="class")- case ClassRegistryMode.SHARED:- metafunc.parametrize("is_class_registry_isolated", [False], ids=["shared"], scope="class")diff --git a/tests/integration/pipelex/temporal/content_generation/conftest.py b/tests/integration/pipelex/temporal/content_generation/conftest.pydeleted file mode 100644index 71df3dd0d..000000000--- a/tests/integration/pipelex/temporal/content_generation/conftest.py+++ /dev/null@@ -1,44 +0,0 @@-from typing import cast--import pytest-from pytest import FixtureRequest--from pipelex.config import get_config-from pipelex.hub import get_class_registry-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.test_extras.temporal_registry_test_models import Person-from tests.integration.pipelex.temporal.test_utils import rprint--[email protected](autouse=True)-def enable_temporal_config():- """Enable temporal in config for content_generation tests that go through WorkflowExecutor."""- config = get_config()- original_enabled = config.temporal.is_enabled- config.temporal = config.temporal.model_copy(update={"is_enabled": True})- yield- config.temporal = config.temporal.model_copy(update={"is_enabled": original_enabled})--[email protected](autouse=True)-def register_test_temporal_models():- # Code to run before each test- rprint("\n[magenta]Temporal classes registration[/magenta]")- get_class_registry().register_class(class_type=Person)- yield- # Code to run after each test- rprint("\n[magenta]Temporal classes unregistration[/magenta]")- get_class_registry().unregister_class(class_type=Person)---def tprl_job_metadata(request: FixtureRequest) -> JobMetadata:- """Provide a JobMetadata instance.-- Uses the test function name as pipeline_run_id for stable test identification.- """- pipeline_run_id = cast("str", request.node.originalname) # pyright: ignore[reportUnknownMemberType]- return JobMetadata(- user_id="test",- pipeline_run_id=pipeline_run_id,- )diff --git a/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_child.py b/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_child.pydeleted file mode 100644index 43b0b1d39..000000000--- a/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_child.py+++ /dev/null@@ -1,33 +0,0 @@-import uuid--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.test_extras.temporal_test_tasks import TEMPORAL_TEST_ACTIVITIES, TEMPORAL_TEST_WORKFLOWS-from pipelex.temporal.test_extras.wf_test_content_generator_child import WfTestContentGeneratorChild--[email protected]_runnable[email protected](loop_scope="class")-class TestTprlContentGeneratorChild:- async def test_tprl_content_generator_child(self, temporal_client: TemporalClient, tprl_job_metadata: JobMetadata, pipe_run_mode: PipeRunMode):- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- test_workflows=TEMPORAL_TEST_WORKFLOWS,- test_activities=TEMPORAL_TEST_ACTIVITIES,- ):- await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfTestContentGeneratorChild.run,- arg=pipe_run_mode.is_dry,- id=tprl_job_metadata.pipeline_run_id,- task_queue=task_queue,- )diff --git a/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_pdf_page_views.py b/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_pdf_page_views.pydeleted file mode 100644index 0eb4d6f5c..000000000--- a/tests/integration/pipelex/temporal/content_generation/test_tprl_content_generator_pdf_page_views.py+++ /dev/null@@ -1,50 +0,0 @@-import uuid--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.test_extras.temporal_test_tasks import TEMPORAL_TEST_ACTIVITIES, TEMPORAL_TEST_WORKFLOWS-from pipelex.temporal.test_extras.wf_test_content_generator_pdf_page_views import WfTestContentGeneratorPdfPageViews--[email protected]_runnable[email protected](loop_scope="class")-class TestTprlContentGeneratorPdfPageViews:- async def test_tprl_content_generator_pdf_page_views(- self,- temporal_client: TemporalClient,- tprl_job_metadata: JobMetadata,- pipe_run_mode: PipeRunMode,- ):- """End-to-end coverage for ContentGeneratorInWorkflow.make_extract_pages with- document_uri + should_include_page_views=True.-- The unit suite mocks both activities; this test runs them for real (Azure- Document Intelligence + pypdfium2 page rendering) through Temporal. Failure- modes this catches that mocked unit tests cannot:- - RenderPageViewsAssignment serialization mismatch across the activity boundary- - page_views_dpi config plumbing dropped before the activity payload is built- - The in-workflow page_view attachment loop running in the wrong order or- on a stale page_contents reference- - Activity registration miss for act_render_page_views in the test worker- """- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- test_workflows=TEMPORAL_TEST_WORKFLOWS,- test_activities=TEMPORAL_TEST_ACTIVITIES,- ):- await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfTestContentGeneratorPdfPageViews.run,- arg=pipe_run_mode.is_dry,- id=tprl_job_metadata.pipeline_run_id,- task_queue=task_queue,- )diff --git a/tests/integration/pipelex/temporal/data_converter/conftest.py b/tests/integration/pipelex/temporal/data_converter/conftest.pydeleted file mode 100644index eea1aae7d..000000000--- a/tests/integration/pipelex/temporal/data_converter/conftest.py+++ /dev/null@@ -1,17 +0,0 @@-import pytest--from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter---def payload_converter() -> BaseModelPayloadConverter:- return BaseModelPayloadConverter()---class CraftingTestCases:- USER_TEXT_FOR_BASE = """- Write a detailed description of a woman's clothing in the style of a 19th-century novel.- Keep it short: 3 sentences max- """-- USER_TEXT_FOR_SINGLE_PERSON = "name: John, age: 30, job: bank teller"diff --git a/tests/integration/pipelex/temporal/data_converter/test_data_conv_craft.py b/tests/integration/pipelex/temporal/data_converter/test_data_conv_craft.pydeleted file mode 100644index 25935aeef..000000000--- a/tests/integration/pipelex/temporal/data_converter/test_data_conv_craft.py+++ /dev/null@@ -1,66 +0,0 @@-import pytest--from pipelex import pretty_print-from pipelex.cogt.content_generation.assignment_models import LLMAssignment, ObjectAssignment-from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams-from pipelex.cogt.llm.llm_prompt import LLMPrompt-from pipelex.hub import get_model_deck-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter-from pipelex.temporal.test_extras.temporal_registry_test_models import Person--from .conftest import CraftingTestCases---class TestDataConverterForCrafting:- def test_data_converter_for_make_llm_text(- self,- payload_converter: BaseModelPayloadConverter,- ):- user_text = CraftingTestCases.USER_TEXT_FOR_BASE- llm_setting = get_model_deck().get_llm_setting(llm_choice="$testing-text")- llm_prompt_for_text = LLMPrompt(user_text=user_text)- llm_assignment = LLMAssignment(- job_metadata=JobMetadata(user_id="test", pipeline_run_id="test"),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),- llm_setting=llm_setting,- llm_prompt=llm_prompt_for_text,- )- pretty_print(llm_assignment, title="llm_assignment")- payload = payload_converter.to_payload(llm_assignment)- pretty_print(payload, title="payload")- assert payload- restored: LLMAssignment = payload_converter.from_payload(payload, type_hint=LLMAssignment)- pretty_print(restored, title="restored LLMAssignment")- assert restored- assert llm_assignment == restored- # The run-mode contract must survive the wire: the leaf inside the activity keys on it.- assert restored.cogt_run_params.run_mode.is_dry-- def test_data_converter_for_make_object(- self,- payload_converter: BaseModelPayloadConverter,- ):- user_text = CraftingTestCases.USER_TEXT_FOR_SINGLE_PERSON- llm_setting_for_object = get_model_deck().get_llm_setting(llm_choice="$testing-structured")- llm_prompt_for_object = LLMPrompt(user_text=user_text)- llm_assignment_for_object = LLMAssignment(- job_metadata=JobMetadata(user_id="test", pipeline_run_id="test"),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- llm_setting=llm_setting_for_object,- llm_prompt=llm_prompt_for_object,- )- object_assignment = ObjectAssignment.make_for_class(- object_class=Person,- llm_assignment=llm_assignment_for_object,- )- pretty_print(object_assignment, title="object_assignment")- payload = payload_converter.to_payload(object_assignment)- pretty_print(payload, title="payload")- assert payload- restored: ObjectAssignment = payload_converter.from_payload(payload, type_hint=ObjectAssignment)- pretty_print(restored, title="restored ObjectAssignment")- assert restored- assert object_assignment == restoreddiff --git a/tests/integration/pipelex/temporal/data_converter/test_data_conv_dataclass.py b/tests/integration/pipelex/temporal/data_converter/test_data_conv_dataclass.pydeleted file mode 100644index c942ea543..000000000--- a/tests/integration/pipelex/temporal/data_converter/test_data_conv_dataclass.py+++ /dev/null@@ -1,84 +0,0 @@-"""Direct converter round-trip tests for pydantic dataclasses.--Phase 0 widened ``BaseModelPayloadConverter`` so a pydantic dataclass (and ``Optional`` /-``list`` of them) routes through kajson with type preservation, instead of falling through-to Temporal's stock JSON. These call ``to_payload`` / ``from_payload`` directly and assert-both value and concrete type survive. A regression guard locks the untouched ``BaseModel``-path to byte-identical output.-"""--from datetime import timedelta-from typing import Any, cast--import pytest-from kajson import kajson-from pydantic import BaseModel-from pydantic.dataclasses import dataclass as pydantic_dataclass-from pydantic.dataclasses import is_pydantic_dataclass--from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter---class ConvInner(BaseModel):- label: str- count: int---@pydantic_dataclass-class ConvDataclass:- name: str- delay: timedelta- inner: ConvInner- note: str | None = None---class ConvPlainModel(BaseModel):- field_a: str- field_b: int---class TestDataConverterForPydanticDataclass:- def test_scalar_dataclass_roundtrip(self, payload_converter: BaseModelPayloadConverter) -> None:- original = ConvDataclass(name="x", delay=timedelta(seconds=30), inner=ConvInner(label="L", count=2))- payload = payload_converter.to_payload(original)- assert payload- restored = cast("ConvDataclass", payload_converter.from_payload(payload, type_hint=ConvDataclass))- assert is_pydantic_dataclass(type(restored))- assert isinstance(restored.inner, ConvInner)- assert restored == original-- def test_optional_dataclass_roundtrip(self, payload_converter: BaseModelPayloadConverter) -> None:- original = ConvDataclass(name="opt", delay=timedelta(seconds=5), inner=ConvInner(label="O", count=1), note="set")- payload = payload_converter.to_payload(original)- assert payload- optional_hint: Any = ConvDataclass | None- restored = cast("ConvDataclass", payload_converter.from_payload(payload, type_hint=optional_hint))- assert is_pydantic_dataclass(type(restored))- assert restored == original-- def test_list_of_dataclass_roundtrip(self, payload_converter: BaseModelPayloadConverter) -> None:- original = [- ConvDataclass(name="a", delay=timedelta(seconds=1), inner=ConvInner(label="A", count=1)),- ConvDataclass(name="b", delay=timedelta(seconds=2), inner=ConvInner(label="B", count=2)),- ]- payload = payload_converter.to_payload(original)- assert payload- list_hint: Any = list[ConvDataclass]- restored = cast("list[ConvDataclass]", payload_converter.from_payload(payload, type_hint=list_hint))- assert all(is_pydantic_dataclass(type(item)) for item in restored)- assert all(isinstance(item.inner, ConvInner) for item in restored)- assert restored == original-- def test_base_model_payload_is_byte_identical(self, payload_converter: BaseModelPayloadConverter) -> None:- # Regression guard: the dataclass widening must leave the BaseModel path byte-identical.- # A plain BaseModel (no __kajson_class_source__) yields exactly kajson.dumps bytes plus the- # encoding metadata key — nothing the dataclass branch could perturb.- model = ConvPlainModel(field_a="hello", field_b=7)- payload = payload_converter.to_payload(model)- assert payload- assert set(payload.metadata.keys()) == {"encoding"}- assert payload.metadata["encoding"] == payload_converter.encoding.encode()- assert payload.data == kajson.dumps(model).encode()- restored = cast("ConvPlainModel", payload_converter.from_payload(payload, type_hint=ConvPlainModel))- assert restored == modeldiff --git a/tests/integration/pipelex/temporal/data_converter/test_data_conv_enum_roundtrip.py b/tests/integration/pipelex/temporal/data_converter/test_data_conv_enum_roundtrip.pydeleted file mode 100644index 26e0fbf61..000000000--- a/tests/integration/pipelex/temporal/data_converter/test_data_conv_enum_roundtrip.py+++ /dev/null@@ -1,54 +0,0 @@-from typing import cast--import pytest-from pydantic import BaseModel--from pipelex import pretty_print-from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory-from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter-from pipelex.types import StrEnum---class PetSpecies(StrEnum):- DOG = "dog"- CAT = "cat"---class Pet(BaseModel):- name: str- species: PetSpecies---class TestEnumRoundTrip:- def test_dynamic_class_with_enum_field_round_trips(- self,- payload_converter: BaseModelPayloadConverter,- ):- """A class with an enum-shaped field generated via SchemaToModelFactory must- survive a full payload round-trip. Exercises the receiver-side exec path that- rebuilds the dynamic class from `__kajson_class_source__` — without it the- deserializer cannot resolve the dynamic class and the round-trip fails.- """- schema = Pet.model_json_schema()- dynamic_pet_cls = SchemaToModelFactory.make_from_json_schema(schema, class_name="Pet")- instance = dynamic_pet_cls(name="Rex", species=PetSpecies.DOG)- pretty_print(instance, title="instance")-- payload = payload_converter.to_payload(instance)- pretty_print(payload, title="payload")- assert payload is not None- assert payload.metadata.get("kajson_class_source")-- restored = cast("BaseModel", payload_converter.from_payload(payload, type_hint=BaseModel))- pretty_print(restored, title="restored")-- restored_class: type[BaseModel] = type(restored)- assert restored_class.__name__ == "Pet"- # datamodel-code-generator now emits enum-shaped `$defs` as- # `RootModel[Literal[...]]` (since `enum_field_as_literal=LiteralType.All` is- # set in `_generate_source_from_schema`), so `restored.species` is a- # `RootModel` wrapping the value, not a Python `Enum` instance. Assert on the- # serialized data — that is what actually crosses the Temporal payload boundary.- assert restored.model_dump() == {"name": "Rex", "species": "dog"}- assert getattr(restored_class, "__kajson_class_source__", None)diff --git a/tests/integration/pipelex/temporal/data_converter/test_data_conv_pipe_run_params.py b/tests/integration/pipelex/temporal/data_converter/test_data_conv_pipe_run_params.pydeleted file mode 100644index cc353baf5..000000000--- a/tests/integration/pipelex/temporal/data_converter/test_data_conv_pipe_run_params.py+++ /dev/null@@ -1,37 +0,0 @@-"""Wire round-trip for ``PipeRunParams`` with the nested ``CogtRunParams`` carrier (eng review D2).--``WfPipeRouter`` receives ``pipe_run_params`` inside its workflow arg, so the nested carrier must-survive the Temporal payload converter — and ``run_mode`` (a delegating property, deliberately not-a serialized field) must read back correctly from the restored nested model.-"""--import pytest--from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipe_run.pipe_run_params import PipeRunParams-from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter---class TestDataConverterForPipeRunParams:- def test_cogt_run_params_round_trip(- self,- payload_converter: BaseModelPayloadConverter,- ):- pipe_run_params = PipeRunParams(- run_mode=PipeRunMode.DRY,- is_mock_usage=True,- pipe_stack_limit=20,- pipe_stack=["root", "child"],- )-- payload = payload_converter.to_payload(pipe_run_params)- assert payload- restored: PipeRunParams = payload_converter.from_payload(payload, type_hint=PipeRunParams)-- assert restored == pipe_run_params- assert restored.run_mode.is_dry- assert restored.is_mock_usage- # The derived carrier reads the restored fields — what generators stamp on assignments.- assert restored.cogt_run_params == CogtRunParams(run_mode=PipeRunMode.DRY, is_mock_usage=True)diff --git a/tests/integration/pipelex/temporal/data_converter/test_data_conv_pydantic.py b/tests/integration/pipelex/temporal/data_converter/test_data_conv_pydantic.pydeleted file mode 100644index 9ecc129c1..000000000--- a/tests/integration/pipelex/temporal/data_converter/test_data_conv_pydantic.py+++ /dev/null@@ -1,28 +0,0 @@-from typing import Any--import pytest-from pydantic import RootModel--from pipelex import pretty_print-from pipelex.temporal.temporal_data_converter import BaseModelPayloadConverter---class MyRootModel(RootModel[dict[str, Any]]):- pass---class TestDataConverterForRootModel:- def test_data_converter_for_root_model_with_dict(- self,- payload_converter: BaseModelPayloadConverter,- ):- my_root_model_1 = MyRootModel(root={"a": 1, "b": 2})- pretty_print(my_root_model_1, title="my_root_model_1")- payload = payload_converter.to_payload(my_root_model_1)- pretty_print(payload, title="payload")- assert payload- restored: MyRootModel = payload_converter.from_payload(payload, type_hint=MyRootModel)- pretty_print(restored, title="restored MyRootModel")- assert restored- assert my_root_model_1 == restoreddiff --git a/tests/integration/pipelex/temporal/library_crate/conflict_concept_alpha.mthds b/tests/integration/pipelex/temporal/library_crate/conflict_concept_alpha.mthdsdeleted file mode 100644index 1ab24701a..000000000--- a/tests/integration/pipelex/temporal/library_crate/conflict_concept_alpha.mthds+++ /dev/null@@ -1,38 +0,0 @@-domain = "conflict_alpha"-description = "PipeSequence with concept 'Result' having score+label fields (conflicts with beta's Result)"--[concept.Result]-description = "A scored result with label"--[concept.Result.structure]-score = { type = "integer", required = true, description = "Numeric score" }-label = { type = "text", required = true, description = "Category label" }--[pipe]-[pipe.alpha_pipeline]-type = "PipeSequence"-description = "Two-step pipeline producing a scored Result then summarizing it"-output = "Text"-steps = [- { pipe = "alpha_generate", result = "alpha_result" },- { pipe = "alpha_summarize", result = "alpha_summary" },-]--[pipe.alpha_generate]-type = "PipeLLM"-description = "Generate a scored result"-output = "Result"-model = "$testing-text"-prompt = "Generate a result with score=42 and label='excellent'. Respond with exactly those values."--[pipe.alpha_summarize]-type = "PipeLLM"-description = "Summarize the scored result"-inputs = { alpha_result = "Result" }-output = "Text"-model = "$testing-text"-prompt = """-Summarize this result:-Score: $alpha_result.score-Label: $alpha_result.label-"""diff --git a/tests/integration/pipelex/temporal/library_crate/conflict_concept_beta.mthds b/tests/integration/pipelex/temporal/library_crate/conflict_concept_beta.mthdsdeleted file mode 100644index 5ca4141b8..000000000--- a/tests/integration/pipelex/temporal/library_crate/conflict_concept_beta.mthds+++ /dev/null@@ -1,40 +0,0 @@-domain = "conflict_beta"-description = "PipeSequence with concept 'Result' having value+confidence+is_valid fields (conflicts with alpha's Result)"--[concept.Result]-description = "A confidence-rated result with validity"--[concept.Result.structure]-value = { type = "text", required = true, description = "The result value" }-confidence = { type = "number", required = true, description = "Confidence score 0-1" }-is_valid = { type = "text", required = true, description = "Whether the result is valid" }--[pipe]-[pipe.beta_pipeline]-type = "PipeSequence"-description = "Two-step pipeline producing a confidence-rated Result then summarizing it"-output = "Text"-steps = [- { pipe = "beta_generate", result = "beta_result" },- { pipe = "beta_summarize", result = "beta_summary" },-]--[pipe.beta_generate]-type = "PipeLLM"-description = "Generate a confidence-rated result"-output = "Result"-model = "$testing-text"-prompt = "Generate a result with value='positive', confidence=0.95, is_valid='true'. Respond with exactly those values."--[pipe.beta_summarize]-type = "PipeLLM"-description = "Summarize the confidence-rated result"-inputs = { beta_result = "Result" }-output = "Text"-model = "$testing-text"-prompt = """-Summarize this result:-Value: $beta_result.value-Confidence: $beta_result.confidence-Valid: $beta_result.is_valid-"""diff --git a/tests/integration/pipelex/temporal/library_crate/conflict_pipe_alpha.mthds b/tests/integration/pipelex/temporal/library_crate/conflict_pipe_alpha.mthdsdeleted file mode 100644index cdb8fcc89..000000000--- a/tests/integration/pipelex/temporal/library_crate/conflict_pipe_alpha.mthds+++ /dev/null@@ -1,31 +0,0 @@-domain = "pipe_conflict_alpha"-description = "PipeSequence with alpha_shared_step PipeLLM about colors — tests per-workflow pipe_ref isolation"--[pipe]-[pipe.pipe_alpha_pipeline]-type = "PipeSequence"-description = "Two-step pipeline about colors"-output = "Text"-steps = [- { pipe = "alpha_shared_step", result = "step_result" },- { pipe = "alpha_finalize", result = "final_result" },-]--[pipe.alpha_shared_step]-type = "PipeLLM"-description = "Generate text about colors"-output = "Text"-model = "$testing-text"-prompt = "List three primary colors in a single sentence."--[pipe.alpha_finalize]-type = "PipeLLM"-description = "Finalize the color text"-inputs = { step_result = "Text" }-output = "Text"-model = "$testing-text"-prompt = """-Rewrite this text in uppercase:--@step_result.text-"""diff --git a/tests/integration/pipelex/temporal/library_crate/conflict_pipe_beta.mthds b/tests/integration/pipelex/temporal/library_crate/conflict_pipe_beta.mthdsdeleted file mode 100644index 945e06f71..000000000--- a/tests/integration/pipelex/temporal/library_crate/conflict_pipe_beta.mthds+++ /dev/null@@ -1,31 +0,0 @@-domain = "pipe_conflict_beta"-description = "PipeSequence with beta_shared_step PipeLLM about animals — tests per-workflow pipe_ref isolation"--[pipe]-[pipe.pipe_beta_pipeline]-type = "PipeSequence"-description = "Two-step pipeline about animals"-output = "Text"-steps = [- { pipe = "beta_shared_step", result = "step_result" },- { pipe = "beta_finalize", result = "final_result" },-]--[pipe.beta_shared_step]-type = "PipeLLM"-description = "Generate text about animals"-output = "Text"-model = "$testing-text"-prompt = "List three farm animals in a single sentence."--[pipe.beta_finalize]-type = "PipeLLM"-description = "Finalize the animal text"-inputs = { step_result = "Text" }-output = "Text"-model = "$testing-text"-prompt = """-Rewrite this text in a formal tone:--@step_result.text-"""diff --git a/tests/integration/pipelex/temporal/library_crate/conftest.py b/tests/integration/pipelex/temporal/library_crate/conftest.pydeleted file mode 100644index 6cacfd4dc..000000000--- a/tests/integration/pipelex/temporal/library_crate/conftest.py+++ /dev/null@@ -1,221 +0,0 @@-from collections.abc import Generator-from pathlib import Path--import pytest--from pipelex.core.interpreter.interpreter import PipelexInterpreter-from pipelex.core.memory.working_memory import WorkingMemory-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.pipe_abstract import PipeAbstract-from pipelex.hub import get_library_manager-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle, pipe_job_from_library-from tests.integration.pipelex.temporal.test_data import (- CombinedPipelineTemporalTestData,- ConflictConceptAlphaTestData,- ConflictConceptBetaTestData,- ConflictPipeAlphaTestData,- ConflictPipeBetaTestData,- CvBatchScreeningTemporalTestData,- LibraryCrateTestData,- MultiConceptAlphaTestData,- MultiConceptBetaTestData,- PipeBatchTemporalTestData,- PipeComposeTemporalTestData,- PipeConditionTemporalTestData,- PipeParallelTemporalTestData,-)--# --- Existing fixtures (Phase 2) -----[email protected](scope="class")-def pipe_job_from_library_dirs(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """Build a PipeJob by loading a specific bundle file (simulates PIPELEXPATH-style loading).-- Uses library_file_paths instead of library_dirs to avoid picking up unrelated- .mthds bundles (conflict/multi test bundles) that share the same directory.- """-- def _load(library_id: str) -> None:- get_library_manager().load_libraries(- library_id=library_id,- library_file_paths=[Path(LibraryCrateTestData.BUNDLE_FILE)],- )-- yield from pipe_job_from_library(_load, pipe_code=LibraryCrateTestData.PIPE_CODE, isolated_registry=is_class_registry_isolated)--[email protected](scope="class")-def pipe_job_from_mthds_content(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """Build a PipeJob by loading the test bundle from mthds_content string."""- yield from pipe_job_from_bundle(- bundle_file=LibraryCrateTestData.BUNDLE_FILE,- pipe_code=LibraryCrateTestData.PIPE_CODE,- isolated_registry=is_class_registry_isolated,- )---# --- Conflict concept fixtures (Scenario 1) -----[email protected](scope="class")-def alpha_concept_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with concept Result(score, label)."""- yield from pipe_job_from_bundle(- bundle_file=ConflictConceptAlphaTestData.BUNDLE_FILE,- pipe_code=ConflictConceptAlphaTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )--[email protected](scope="class")-def beta_concept_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with concept Result(value, confidence, is_valid)."""- yield from pipe_job_from_bundle(- bundle_file=ConflictConceptBetaTestData.BUNDLE_FILE,- pipe_code=ConflictConceptBetaTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )---# --- Conflict pipe fixtures (Scenario 2) -----[email protected](scope="class")-def alpha_pipe_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with pipe 'shared_step' about colors."""- yield from pipe_job_from_bundle(- bundle_file=ConflictPipeAlphaTestData.BUNDLE_FILE,- pipe_code=ConflictPipeAlphaTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )--[email protected](scope="class")-def beta_pipe_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with pipe 'shared_step' about animals."""- yield from pipe_job_from_bundle(- bundle_file=ConflictPipeBetaTestData.BUNDLE_FILE,- pipe_code=ConflictPipeBetaTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )---# --- Multi-concept fixtures (Scenario 3) -----[email protected](scope="class")-def multi_alpha_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with Profile(name, age) + Summary(headline, body)."""- yield from pipe_job_from_bundle(- bundle_file=MultiConceptAlphaTestData.BUNDLE_FILE,- pipe_code=MultiConceptAlphaTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )--[email protected](scope="class")-def multi_beta_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with Profile(title, department, level) + Summary(content)."""- yield from pipe_job_from_bundle(- bundle_file=MultiConceptBetaTestData.BUNDLE_FILE,- pipe_code=MultiConceptBetaTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )---# --- Controller type coverage fixtures -----[email protected](scope="class")-def condition_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with PipeCondition inside a PipeSequence."""- yield from pipe_job_from_bundle(- bundle_file=PipeConditionTemporalTestData.BUNDLE_FILE,- pipe_code=PipeConditionTemporalTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )--[email protected](scope="class")-def parallel_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with PipeParallel inside a PipeSequence."""- yield from pipe_job_from_bundle(- bundle_file=PipeParallelTemporalTestData.BUNDLE_FILE,- pipe_code=PipeParallelTemporalTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )--[email protected](scope="class")-def batch_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with PipeBatch inside a PipeSequence."""- yield from pipe_job_from_bundle(- bundle_file=PipeBatchTemporalTestData.BUNDLE_FILE,- pipe_code=PipeBatchTemporalTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )--[email protected](scope="class")-def compose_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with PipeCompose operator and dynamic Report concept."""- yield from pipe_job_from_bundle(- bundle_file=PipeComposeTemporalTestData.BUNDLE_FILE,- pipe_code=PipeComposeTemporalTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )--[email protected](scope="class")-def combined_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob with PipeParallel + PipeCondition in a single PipeSequence."""- yield from pipe_job_from_bundle(- bundle_file=CombinedPipelineTemporalTestData.BUNDLE_FILE,- pipe_code=CombinedPipelineTemporalTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- )---# --- CV batch screening fixture (deeply-nested controller + PipeExtract + PipeLLM) -----[email protected](scope="class")-def cv_batch_screening_job(pipe_run_mode: PipeRunMode, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob for the CV batch screening pipeline, pre-populated with mock cvs + job_offer_pdf.-- Mirrors the demos example 21 pipeline (PipeSequence -> PipeSequence -> PipeBatch).- The top-level pipe requires ``cvs: Document[]`` and ``job_offer_pdf: Document``- inputs; we synthesize them via ``make_mock_inputs`` so the fixture works in dry- mode (the default for the temporal in-process test).- """-- def _load(library_id: str) -> None:- mthds_content = Path(CvBatchScreeningTemporalTestData.BUNDLE_FILE).read_text(encoding="utf-8")- blueprint = PipelexInterpreter.make_pipelex_bundle_blueprint(mthds_content=mthds_content)- get_library_manager().load_from_blueprints(library_id=library_id, blueprints=[blueprint])-- def _build_working_memory(pipe: PipeAbstract) -> WorkingMemory:- needed_inputs = WorkingMemoryFactory.convert_to_working_memory_format(needed_inputs_spec=pipe.needed_inputs())- return WorkingMemoryFactory.make_mock_inputs(needed_inputs=needed_inputs)-- yield from pipe_job_from_library(- load_fn=_load,- pipe_code=CvBatchScreeningTemporalTestData.PIPE_CODE,- pipe_run_mode=pipe_run_mode,- isolated_registry=is_class_registry_isolated,- working_memory_builder=_build_working_memory,- )diff --git a/tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds b/tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthdsdeleted file mode 100644index 2a137826f..000000000--- a/tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds+++ /dev/null@@ -1,38 +0,0 @@-domain = "dynamic_concept_test"-description = "PipeSequence with dynamic concept (inline structure) for deferred hydration tests"--[concept.Greeting]-description = "A greeting with message and language"--[concept.Greeting.structure]-message = { type = "text", required = true, description = "The greeting message" }-language = { type = "text", required = true, description = "The language of the greeting" }--[pipe]-[pipe.dynamic_greeting_sequence]-type = "PipeSequence"-description = "Two-step pipeline producing a dynamic Greeting concept"-output = "Text"-steps = [- { pipe = "generate_greeting", result = "greeting_result" },- { pipe = "summarize_greeting", result = "summary_result" },-]--[pipe.generate_greeting]-type = "PipeLLM"-description = "Generate a greeting in a specific language"-output = "Greeting"-model = "$testing-text"-prompt = "Generate a greeting in French. Respond with exactly: message='Bonjour le monde', language='French'"--[pipe.summarize_greeting]-type = "PipeLLM"-description = "Summarize the greeting"-inputs = { greeting_result = "Greeting" }-output = "Text"-model = "$testing-text"-prompt = """-Summarize this greeting:-Message: $greeting_result.message-Language: $greeting_result.language-"""diff --git a/tests/integration/pipelex/temporal/library_crate/helpers.py b/tests/integration/pipelex/temporal/library_crate/helpers.pydeleted file mode 100644index 0eb165673..000000000--- a/tests/integration/pipelex/temporal/library_crate/helpers.py+++ /dev/null@@ -1,101 +0,0 @@-"""Shared test helpers for Temporal LibraryCrate integration tests."""--import uuid-from typing import Any--from temporalio.client import Client as TemporalClient--from pipelex.core.memory.working_memory import WorkingMemory-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.core.stuffs.stuff import Stuff-from pipelex.core.stuffs.stuff_content_factory import StuffContentFactory-from pipelex.core.stuffs.text_content import TextContent-from pipelex.hub import get_required_concept-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData---def make_prepared_greeting_job(pipe_job: PipeJob, stuff_code: str) -> PipeJob:- """Copy ``pipe_job`` with an input WorkingMemory carrying a dynamic-concept Greeting- stuff, dehydrated for Temporal transit via ``prepare_for_temporal``.-- The Greeting concept comes from the deferred-hydration bundle, so the receiving- workflow must hydrate the stuff inline after loading the crate — the deferred- hydration path.- """- greeting_concept = get_required_concept(concept_ref=f"{DeferredHydrationTestData.DOMAIN}.Greeting")- greeting_content = StuffContentFactory.make_stuff_content_from_concept_required(- concept=greeting_concept,- value={"message": "Bonjour le monde", "language": "French"},- )- greeting_stuff = Stuff(- stuff_code=stuff_code,- stuff_name="greeting_result",- concept=greeting_concept,- content=greeting_content,- )- input_memory = WorkingMemory()- input_memory.root["greeting_result"] = greeting_stuff- return pipe_job.model_copy(update={"working_memory": input_memory}).prepare_for_temporal()---def rehydrate_pipe_output(pipe_output: PipeOutput, pipe_job: PipeJob | None = None) -> PipeOutput:- """Rehydrate a deferred PipeOutput received from a Temporal workflow.-- Thin wrapper around the production helper `rehydrate_pipe_output_with_crate`- so the integration tests share the exact same scoped-registry plumbing- that production submitters use.- """- library_crate = pipe_job.library_crate if pipe_job is not None else None- return rehydrate_pipe_output_with_crate(pipe_output, library_crate=library_crate)---async def execute_workflow(- pipe_job: PipeJob,- temporal_client: TemporalClient,- task_queue: str,- **kwargs: Any,-) -> PipeOutput:- """Execute WfPipeRouter and return PipeOutput, rehydrating if needed."""- workflow_id = str(uuid.uuid4())- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=pipe_job,- id=workflow_id,- task_queue=task_queue,- **kwargs,- )- return rehydrate_pipe_output(pipe_output, pipe_job)---def assert_stuff_names(pipe_output: PipeOutput, expected_names: list[str]) -> None:- """Assert that all expected stuffs exist in the output with non-None content."""- working_memory = pipe_output.working_memory- assert working_memory is not None- for stuff_name in expected_names:- assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"- stuff = working_memory.get_stuff(stuff_name)- assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"---def assert_text_stuff_names(pipe_output: PipeOutput, expected_names: list[str]) -> None:- """Assert that all expected stuffs exist and contain TextContent."""- assert_stuff_names(pipe_output, expected_names)- working_memory = pipe_output.working_memory- assert working_memory is not None- for stuff_name in expected_names:- stuff = working_memory.get_stuff(stuff_name)- assert isinstance(stuff.content, TextContent), f"Expected TextContent for '{stuff_name}', got {type(stuff.content).__name__}"---def assert_structured_fields(pipe_output: PipeOutput, stuff_name: str, expected_fields: list[str]) -> None:- """Assert that a stuff's content is StructuredContent with the expected fields."""- working_memory = pipe_output.working_memory- assert working_memory is not None- stuff = working_memory.get_stuff(stuff_name)- assert isinstance(stuff.content, StructuredContent), f"Expected StructuredContent for '{stuff_name}', got {type(stuff.content).__name__}"- for field in expected_fields:- assert hasattr(stuff.content, field), f"StructuredContent for '{stuff_name}' missing field '{field}'"diff --git a/tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds b/tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthdsdeleted file mode 100644index c2f4544fc..000000000--- a/tests/integration/pipelex/temporal/library_crate/large_payload_sequence.mthds+++ /dev/null@@ -1,47 +0,0 @@-domain = "large_payload_test"-description = "Multi-step pipeline that accumulates large WorkingMemory for codec stress testing"--[pipe.large_payload_sequence]-type = "PipeSequence"-description = "Three-step pipeline accumulating verbose text output"-output = "Text"-steps = [- { pipe = "verbose_step_one", result = "step_one_result" },- { pipe = "verbose_step_two", result = "step_two_result" },- { pipe = "verbose_step_three", result = "step_three_result" },-]--[pipe.verbose_step_one]-type = "PipeLLM"-description = "First step: generate a detailed story"-output = "Text"-model = "$testing-text"-prompt = "Write a very detailed, multi-paragraph story about a space adventure. Include character names, planet descriptions, and dialogue."--[pipe.verbose_step_two]-type = "PipeLLM"-description = "Second step: expand with analysis"-inputs = { step_one_result = "Text" }-output = "Text"-model = "$testing-text"-prompt = """-Analyze the following story in great detail. Cover themes, character development, plot structure, and writing style.--@step_one_result.text-"""--[pipe.verbose_step_three]-type = "PipeLLM"-description = "Third step: create a comprehensive summary"-inputs = { step_one_result = "Text", step_two_result = "Text" }-output = "Text"-model = "$testing-text"-prompt = """-Based on this story and analysis, write a comprehensive literary review.--Story:-@step_one_result.text--Analysis:-@step_two_result.text-"""diff --git a/tests/integration/pipelex/temporal/library_crate/multi_concept_alpha.mthds b/tests/integration/pipelex/temporal/library_crate/multi_concept_alpha.mthdsdeleted file mode 100644index 73dbe4304..000000000--- a/tests/integration/pipelex/temporal/library_crate/multi_concept_alpha.mthds+++ /dev/null@@ -1,60 +0,0 @@-domain = "multi_alpha"-description = "PipeSequence with Profile(name,age) and Summary(headline,body) — conflicts with beta's Profile and Summary"--[concept.Profile]-description = "A personal profile with name and age"--[concept.Profile.structure]-name = { type = "text", required = true, description = "Person's name" }-age = { type = "integer", required = true, description = "Person's age" }--[concept.Summary]-description = "A summary with headline and body"--[concept.Summary.structure]-headline = { type = "text", required = true, description = "Summary headline" }-body = { type = "text", required = true, description = "Summary body text" }--[pipe]-[pipe.multi_alpha_pipeline]-type = "PipeSequence"-description = "Three-step pipeline: generate personal Profile, then Summary, then multi_alpha_finalize as Text"-output = "Text"-steps = [- { pipe = "multi_alpha_generate_profile", result = "profile_result" },- { pipe = "multi_alpha_generate_summary", result = "summary_result" },- { pipe = "multi_alpha_finalize", result = "final_result" },-]--[pipe.multi_alpha_generate_profile]-type = "PipeLLM"-description = "Generate a personal profile"-output = "Profile"-model = "$testing-text"-prompt = "Generate a profile with name='Alice' and age=30. Respond with exactly those values."--[pipe.multi_alpha_generate_summary]-type = "PipeLLM"-description = "Summarize the profile"-inputs = { profile_result = "Profile" }-output = "Summary"-model = "$testing-text"-prompt = """-Create a summary for this person:-Name: $profile_result.name-Age: $profile_result.age--Respond with a headline and body.-"""--[pipe.multi_alpha_finalize]-type = "PipeLLM"-description = "Finalize as plain text"-inputs = { summary_result = "Summary" }-output = "Text"-model = "$testing-text"-prompt = """-Format this summary as plain text:-Headline: $summary_result.headline-Body: $summary_result.body-"""diff --git a/tests/integration/pipelex/temporal/library_crate/multi_concept_beta.mthds b/tests/integration/pipelex/temporal/library_crate/multi_concept_beta.mthdsdeleted file mode 100644index f5204839c..000000000--- a/tests/integration/pipelex/temporal/library_crate/multi_concept_beta.mthds+++ /dev/null@@ -1,58 +0,0 @@-domain = "multi_beta"-description = "PipeSequence with Profile(title,department,level) and Summary(content) — conflicts with alpha's Profile and Summary"--[concept.Profile]-description = "A professional profile with title, department, and level"--[concept.Profile.structure]-title = { type = "text", required = true, description = "Job title" }-department = { type = "text", required = true, description = "Department name" }-level = { type = "integer", required = true, description = "Seniority level" }--[concept.Summary]-description = "A compact summary with just content"--[concept.Summary.structure]-content = { type = "text", required = true, description = "Summary content" }--[pipe]-[pipe.multi_beta_pipeline]-type = "PipeSequence"-description = "Three-step pipeline: generate professional Profile, then Summary, then multi_beta_finalize as Text"-output = "Text"-steps = [- { pipe = "multi_beta_generate_profile", result = "profile_result" },- { pipe = "multi_beta_generate_summary", result = "summary_result" },- { pipe = "multi_beta_finalize", result = "final_result" },-]--[pipe.multi_beta_generate_profile]-type = "PipeLLM"-description = "Generate a professional profile"-output = "Profile"-model = "$testing-text"-prompt = "Generate a profile with title='Engineer', department='Platform', level=3. Respond with exactly those values."--[pipe.multi_beta_generate_summary]-type = "PipeLLM"-description = "Summarize the professional profile"-inputs = { profile_result = "Profile" }-output = "Summary"-model = "$testing-text"-prompt = """-Create a one-line summary for this employee:-Title: $profile_result.title-Department: $profile_result.department-Level: $profile_result.level-"""--[pipe.multi_beta_finalize]-type = "PipeLLM"-description = "Finalize as plain text"-inputs = { summary_result = "Summary" }-output = "Text"-model = "$testing-text"-prompt = """-Format this summary as plain text:-@summary_result.content-"""diff --git a/tests/integration/pipelex/temporal/library_crate/pdf_extract_inputs.json b/tests/integration/pipelex/temporal/library_crate/pdf_extract_inputs.jsondeleted file mode 100644index 4f557e25c..000000000--- a/tests/integration/pipelex/temporal/library_crate/pdf_extract_inputs.json+++ /dev/null@@ -1,8 +0,0 @@-{- "source_pdf": {- "concept": "native.Document",- "content": {- "url": "../../../../data/documents/Job-Offer.pdf"- }- }-}diff --git a/tests/integration/pipelex/temporal/library_crate/pdf_extract_page_views.mthds b/tests/integration/pipelex/temporal/library_crate/pdf_extract_page_views.mthdsdeleted file mode 100644index f8d0b329b..000000000--- a/tests/integration/pipelex/temporal/library_crate/pdf_extract_page_views.mthds+++ /dev/null@@ -1,11 +0,0 @@-domain = "pdf_extract_routing_test"-description = "PDF extraction with page_views — exercises act_extract_gen_extract_pages + act_render_page_views"--[pipe]-[pipe.pdf_extract_with_page_views]-type = "PipeExtract"-description = "Extract pages from a PDF and render each page as an image (two activities dispatched)"-inputs = { source_pdf = "Document" }-output = "Page[]"-model = "@default-extract-document"-page_views = truediff --git a/tests/integration/pipelex/temporal/library_crate/per_handle_routing.mthds b/tests/integration/pipelex/temporal/library_crate/per_handle_routing.mthdsdeleted file mode 100644index 956286e92..000000000--- a/tests/integration/pipelex/temporal/library_crate/per_handle_routing.mthds+++ /dev/null@@ -1,31 +0,0 @@-domain = "per_handle_routing_test"-description = "Two LLM calls using distinct concrete model handles, exercising by_handle routing"--[pipe]-[pipe.per_handle_routing_sequence]-type = "PipeSequence"-description = "Step 1 dispatched to claude-4.6-sonnet, step 2 dispatched to gemini-flash-latest"-output = "Text"-steps = [- { pipe = "step_handle_a", result = "step_handle_a_result" },- { pipe = "step_handle_b", result = "step_handle_b_result" },-]--[pipe.step_handle_a]-type = "PipeLLM"-description = "Dispatches via handle_a (claude-4.6-sonnet) — routes to q_handle_a"-output = "Text"-model = "claude-4.6-sonnet"-prompt = "Write a single short sentence about the color blue."--[pipe.step_handle_b]-type = "PipeLLM"-description = "Dispatches via handle_b (gemini-flash-latest) — routes to q_handle_b"-inputs = { step_handle_a_result = "Text" }-output = "Text"-model = "gemini-flash-latest"-prompt = """-Rewrite the following text in a formal tone:--@step_handle_a_result.text-"""diff --git a/tests/integration/pipelex/temporal/library_crate/structured_output_sequence.mthds b/tests/integration/pipelex/temporal/library_crate/structured_output_sequence.mthdsdeleted file mode 100644index 96f3d6344..000000000--- a/tests/integration/pipelex/temporal/library_crate/structured_output_sequence.mthds+++ /dev/null@@ -1,49 +0,0 @@-domain = "structured_output_test"-description = "Pipelines exercising act_llm_gen_object / act_llm_gen_object_list across the Temporal data converter (custom Invoice concept with nested Customer + LineItem list)"--[concept.LineItem]-description = "A single line item in an invoice"--[concept.LineItem.structure]-product_name = { type = "text", description = "Name of the product", required = true }-quantity = { type = "integer", description = "Quantity ordered", required = true }-unit_price = { type = "number", description = "Price per unit", required = true }--[concept.Customer]-description = "A customer for an invoice"--[concept.Customer.structure]-name = { type = "text", description = "Customer's full name", required = true }-email = { type = "text", description = "Customer's email address", required = true }--[concept.Invoice]-description = "A structured invoice with nested customer + line items"--[concept.Invoice.structure]-invoice_number = { type = "text", description = "Unique invoice identifier", required = true }-customer = { type = "concept", concept_ref = "structured_output_test.Customer", description = "The customer for this invoice", required = true }-line_items = { type = "list", item_type = "concept", item_concept_ref = "structured_output_test.LineItem", description = "List of line items in the invoice", required = true }-total_amount = { type = "number", description = "Total invoice amount", required = true }--[pipe]-[pipe.generate_invoice_single]-type = "PipeLLM"-description = "Generate a single Invoice (exercises act_llm_gen_object cross-process round-trip)"-output = "Invoice"-model = "$testing-structured"-prompt = """-Generate a single complete invoice for customer 'Alice Chen' ([email protected])-with two line items: 'Widget' x3 at 9.99 and 'Gadget' x1 at 49.50.-Use invoice_number INV-2026-001 and total_amount 79.47.-"""--[pipe.generate_invoice_list]-type = "PipeLLM"-description = "Generate multiple Invoices (exercises act_llm_gen_object_list cross-process round-trip)"-output = "Invoice[2]"-model = "$testing-structured"-prompt = """-Generate 2 distinct invoices, each with at least one nested LineItem.-Use unique invoice_number values (INV-2026-A, INV-2026-B) so the round-trip-test can distinguish them.-"""diff --git a/tests/integration/pipelex/temporal/library_crate/structured_search.mthds b/tests/integration/pipelex/temporal/library_crate/structured_search.mthdsdeleted file mode 100644index c0243003f..000000000--- a/tests/integration/pipelex/temporal/library_crate/structured_search.mthds+++ /dev/null@@ -1,16 +0,0 @@-domain = "structured_search_test"-description = "Single PipeSearch step over a custom structured TopicSummary concept (exercises act_search_gen_structured across the Temporal boundary)"--[concept.TopicSummary]-description = "A structured summary of web-search findings for a topic"--[concept.TopicSummary.structure]-title = { type = "text", description = "Topic title", required = true }-summary = { type = "text", description = "Summary of the findings", required = true }-key_points = { type = "list", item_type = "text", description = "Key points distilled from the search", required = true }--[pipe.structured_search]-type = "PipeSearch"-description = "Search the web and return a structured TopicSummary"-output = "TopicSummary"-prompt = "Summarize the latest developments in AI safety research."diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_batch.mthdsdeleted file mode 100644index 27ca9f726..000000000--- a/tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds+++ /dev/null@@ -1,40 +0,0 @@-domain = "temporal_batch_test"-description = "PipeBatch within a PipeSequence for Temporal fan-out child workflow dispatch testing"--[concept]-Topic = "A subject for content generation"-TopicNote = "A short note about a topic"--[pipe]-[pipe.temporal_batch_sequence]-type = "PipeSequence"-description = "Generate topics then process each via batch"-output = "TopicNote[]"-steps = [- { pipe = "temporal_generate_topics", result = "topics" },- { pipe = "batch_temporal_describe_topics", result = "notes" },-]--[pipe.temporal_generate_topics]-type = "PipeLLM"-description = "Generate a list of topics"-output = "Topic[3]"-model = "$testing-text"-prompt = "List exactly 3 topics: science, history, music"--[pipe.batch_temporal_describe_topics]-type = "PipeBatch"-description = "Describe each topic via fan-out child workflows"-inputs = { topics = "Topic[]" }-output = "TopicNote[]"-branch_pipe_code = "temporal_describe_topic"-input_list_name = "topics"-input_item_name = "topic"--[pipe.temporal_describe_topic]-type = "PipeLLM"-description = "Write a short note about a topic"-inputs = { topic = "Topic" }-output = "TopicNote"-model = "$testing-text"-prompt = "Write one sentence about: $topic"diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_combined.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_combined.mthdsdeleted file mode 100644index d3240f73a..000000000--- a/tests/integration/pipelex/temporal/library_crate/temporal_combined.mthds+++ /dev/null@@ -1,76 +0,0 @@-domain = "temporal_combined_test"-description = "Combined PipeParallel + PipeCondition in a PipeSequence for nested Temporal dispatch testing"--[concept.QualityReport]-description = "A quality assessment report"--[concept.QualityReport.structure]-assessment = { type = "text", required = true, description = "Quality assessment" }-confidence = { type = "text", required = true, description = "Confidence level" }--[concept]-PartContent = "Content generated by a parallel branch"--[pipe]-[pipe.temporal_combined_pipeline]-type = "PipeSequence"-description = "Combines parallel and condition dispatch patterns"-output = "QualityReport"-steps = [- { pipe = "parallel_generate", result = "parallel_done" },- { pipe = "quality_gate", result = "gate_result" },- { pipe = "produce_report", result = "final_report" },-]--[pipe.parallel_generate]-type = "PipeParallel"-description = "Generate two pieces of content in parallel"-output = "Text"-add_each_output = true-branches = [- { pipe = "generate_part_a", result = "part_a" },- { pipe = "generate_part_b", result = "part_b" },-]--[pipe.generate_part_a]-type = "PipeLLM"-description = "Generate part A"-output = "PartContent"-model = "$testing-text"-prompt = "Output exactly: Part A content"--[pipe.generate_part_b]-type = "PipeLLM"-description = "Generate part B"-output = "PartContent"-model = "$testing-text"-prompt = "Output exactly: Part B content"--[pipe.quality_gate]-type = "PipeCondition"-description = "Quality check on generated content"-output = "Text"-expression_template = "pass"-default_outcome = "handle_failure"--[pipe.quality_gate.outcomes]-pass = "continue"--[pipe.handle_failure]-type = "PipeLLM"-description = "Handle quality check failure"-output = "Text"-model = "$testing-text"-prompt = "Output exactly: quality_failed"--[pipe.produce_report]-type = "PipeLLM"-description = "Produce the final quality report"-inputs = { part_a = "PartContent", part_b = "PartContent" }-output = "QualityReport"-model = "$testing-text"-prompt = """-Create a quality report from:-Part A: $part_a-Part B: $part_b-"""diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_compose.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_compose.mthdsdeleted file mode 100644index dddd53046..000000000--- a/tests/integration/pipelex/temporal/library_crate/temporal_compose.mthds+++ /dev/null@@ -1,44 +0,0 @@-domain = "temporal_compose_test"-description = "PipeCompose operator within a PipeSequence for Temporal testing with deferred hydration"--[concept.Report]-description = "A simple report with title and body"--[concept.Report.structure]-title = { type = "text", required = true, description = "Report title" }-body = { type = "text", required = true, description = "Report body" }--[pipe]-[pipe.temporal_compose_sequence]-type = "PipeSequence"-description = "Generate title and body then compose into a Report"-output = "Report"-steps = [- { pipe = "generate_title", result = "title_text" },- { pipe = "generate_body", result = "body_text" },- { pipe = "compose_report", result = "final_report" },-]--[pipe.generate_title]-type = "PipeLLM"-description = "Generate a report title"-output = "Text"-model = "$testing-text"-prompt = "Output exactly: Quarterly Review"--[pipe.generate_body]-type = "PipeLLM"-description = "Generate a report body"-output = "Text"-model = "$testing-text"-prompt = "Output exactly: Revenue increased by 15 percent."--[pipe.compose_report]-type = "PipeCompose"-description = "Compose title and body into a structured Report"-inputs = { title_text = "Text", body_text = "Text" }-output = "Report"--[pipe.compose_report.construct]-title = { from = "title_text.text" }-body = { from = "body_text.text" }diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_condition.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_condition.mthdsdeleted file mode 100644index 977804f92..000000000--- a/tests/integration/pipelex/temporal/library_crate/temporal_condition.mthds+++ /dev/null@@ -1,55 +0,0 @@-domain = "temporal_condition_test"-description = "PipeCondition within a PipeSequence for Temporal child workflow dispatch testing"--[concept]-CategoryLabel = "A category label for routing"--[pipe]-[pipe.temporal_condition_sequence]-type = "PipeSequence"-description = "Sequence that generates a category then routes via condition"-output = "Text"-steps = [- { pipe = "generate_category", result = "category_text" },- { pipe = "route_by_category", result = "routed_result" },-]--[pipe.generate_category]-type = "PipeLLM"-description = "Generate a category label"-output = "CategoryLabel"-model = "$testing-text"-prompt = "Output exactly one word: alpha"--[pipe.route_by_category]-type = "PipeCondition"-description = "Route based on category text content"-inputs = { category_text = "CategoryLabel" }-output = "Text"-expression_template = "{{ category_text.text }}"-default_outcome = "handle_default"--[pipe.route_by_category.outcomes]-alpha = "handle_alpha"-beta = "handle_beta"--[pipe.handle_alpha]-type = "PipeLLM"-description = "Handle alpha category"-output = "Text"-model = "$testing-text"-prompt = "Output exactly: handled_alpha"--[pipe.handle_beta]-type = "PipeLLM"-description = "Handle beta category"-output = "Text"-model = "$testing-text"-prompt = "Output exactly: handled_beta"--[pipe.handle_default]-type = "PipeLLM"-description = "Handle default category"-output = "Text"-model = "$testing-text"-prompt = "Output exactly: handled_default"diff --git a/tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds b/tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthdsdeleted file mode 100644index 2d140014a..000000000--- a/tests/integration/pipelex/temporal/library_crate/temporal_parallel.mthds+++ /dev/null@@ -1,52 +0,0 @@-domain = "temporal_parallel_test"-description = "PipeParallel within a PipeSequence for Temporal concurrent child workflow dispatch testing"--[concept]-ToneAnalysis = "Analysis of text tone"-LengthAnalysis = "Analysis of text length"--[pipe]-[pipe.temporal_parallel_sequence]-type = "PipeSequence"-description = "Sequence wrapping a parallel dispatch then a summary"-output = "Text"-steps = [- { pipe = "analyze_in_parallel", result = "parallel_done" },- { pipe = "summarize_results", result = "summary" },-]--[pipe.analyze_in_parallel]-type = "PipeParallel"-description = "Two branches running concurrently as separate child workflows"-output = "Text"-add_each_output = true-branches = [- { pipe = "branch_tone", result = "tone_result" },- { pipe = "branch_length", result = "length_result" },-]--[pipe.branch_tone]-type = "PipeLLM"-description = "Analyze tone"-output = "ToneAnalysis"-model = "$testing-text"-prompt = "Output exactly: formal_tone"--[pipe.branch_length]-type = "PipeLLM"-description = "Analyze length"-output = "LengthAnalysis"-model = "$testing-text"-prompt = "Output exactly: medium_length"--[pipe.summarize_results]-type = "PipeLLM"-description = "Summarize parallel results"-inputs = { tone_result = "ToneAnalysis", length_result = "LengthAnalysis" }-output = "Text"-model = "$testing-text"-prompt = """-Combine these results:-Tone: $tone_result-Length: $length_result-"""diff --git a/tests/integration/pipelex/temporal/library_crate/test_distributed_scopes.py b/tests/integration/pipelex/temporal/library_crate/test_distributed_scopes.pydeleted file mode 100644index 2b85464a6..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_distributed_scopes.py+++ /dev/null@@ -1,87 +0,0 @@-"""Two scoped workers on the same task queue: router (workflow-only) + runner (activity-only).--Validates that the worker scopes mechanism resolves correctly and that two scoped-workers can coexist on the same task queue (Temporal SDK rejects overlapping task-types — workflow-only + activity-only is the only viable split on one queue).--NOTE: this in-process test cannot reproduce the cross-process registry-propagation-bug described in TODOS.md, because both workers share the same Python process and-the same global `KajsonManager`. The deterministic cross-process regression lives-in the `/temporal-e2e-validate` skill, which spawns separate worker processes via-the `--scope router`/`--scope runner` CLI flags.-"""--import uuid-from collections.abc import Generator--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.config import get_config-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData--[email protected](scope="class")-def pipe_job_with_dynamic_concept(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- yield from pipe_job_from_bundle(- bundle_file=DeferredHydrationTestData.BUNDLE_FILE,- pipe_code=DeferredHydrationTestData.PIPE_CODE,- isolated_registry=is_class_registry_isolated,- )--[email protected](loop_scope="class")-class TestDistributedScopes:- async def test_dynamic_concept_across_router_and_runner_workers(- self,- pipe_job_with_dynamic_concept: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Two scoped workers on the same queue: WfPipeRun must land on a different- worker than WfPipeRouter. Dynamic concept must still resolve end-to-end.- """- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- worker_scopes = get_config().temporal.worker_scopes- router_scope = worker_scopes.scopes["router"]- runner_scope = worker_scopes.scopes["runner"]-- async with (- get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- scope=router_scope,- ),- get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- scope=runner_scope,- ),- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=pipe_job_with_dynamic_concept,- id=workflow_id,- task_queue=task_queue,- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output, pipe_job=pipe_job_with_dynamic_concept)- working_memory = pipe_output.working_memory- assert working_memory is not None-- greeting_stuff = working_memory.get_stuff("greeting_result")- assert isinstance(greeting_stuff.content, StructuredContent)- assert hasattr(greeting_stuff.content, "message")- assert hasattr(greeting_stuff.content, "language")diff --git a/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.py b/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.pydeleted file mode 100644index 16636579e..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.py+++ /dev/null@@ -1,205 +0,0 @@-"""Integration test for submitter-side rehydration without bundle pre-loaded.--Validates that `rehydrate_pipe_output_with_crate` succeeds against an empty-submitter-side global registry by spinning up its own per-call scoped library-from the `LibraryCrate` carried by the `PipeJob`.--This is the production guarantee for remote API clients that built the-`PipeJob` from a serialized `LibraryCrate` and never loaded the bundle into-their global `KajsonManager` registry.-"""--import uuid-from collections.abc import Generator--import pytest-from kajson.kajson_manager import KajsonManager-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.hub import clear_current_library, get_current_library, set_current_library-from pipelex.pipe_run.exceptions import PipeJobError-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.runtime_bridge.primitives.hydration import hydrate_working_memory-from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData--GREETING_CLASS_NAME = f"{DeferredHydrationTestData.DOMAIN}__Greeting"---def _current_library_id_or_none() -> str | None:- """Read current_library without raising when none is set."""- try:- return get_current_library()- except RuntimeError:- return None--[email protected](scope="class")-def pipe_job_isolated_dynamic_concept() -> Generator[PipeJob, None, None]:- """PipeJob built from the dynamic-concept bundle with a scoped (isolated) ClassRegistry.-- Forces ``isolated_registry=True`` so the dynamic Greeting class lands in- the fixture's scoped registry and the global ``KajsonManager`` registry- stays clean — the precondition for the "submitter without bundle- pre-loaded" scenario this module exercises.- """- yield from pipe_job_from_bundle(- bundle_file=DeferredHydrationTestData.BUNDLE_FILE,- pipe_code=DeferredHydrationTestData.PIPE_CODE,- isolated_registry=True,- )--[email protected](loop_scope="class")-class TestSubmitterWithoutBundleLoaded:- async def _dispatch_and_get_deferred_pipe_output(- self,- pipe_job: PipeJob,- temporal_client: TemporalClient,- ) -> PipeOutput:- """Run the workflow and return the PipeOutput as-received (with working_memory_raw populated)."""- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=pipe_job,- id=workflow_id,- task_queue=task_queue,- )-- assert isinstance(pipe_output, PipeOutput)- assert pipe_output.working_memory_raw is not None, "Worker should have returned a deferred PipeOutput (working_memory_raw populated)"- return pipe_output-- async def test_helper_rehydrates_with_clean_global_registry(- self,- pipe_job_isolated_dynamic_concept: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """rehydrate_pipe_output_with_crate succeeds when the submitter never loaded the bundle.-- The fixture runs in isolated-registry mode, so the dynamic Greeting class- lives only in the fixture's scoped library, never in the global registry.- We additionally clear the current_library ContextVar before calling the- helper so the active class registry falls back to the (clean) global —- which simulates a remote submitter that received the PipeJob+crate over- the wire and never touched a Pipelex library at all.- """- pipe_output = await self._dispatch_and_get_deferred_pipe_output(- pipe_job_isolated_dynamic_concept,- temporal_client,- )-- global_registry = KajsonManager.get_class_registry()- assert global_registry.get_class(name=GREETING_CLASS_NAME) is None, (- f"Precondition violated: '{GREETING_CLASS_NAME}' is in the global KajsonManager registry; "- "the test cannot prove the submitter-without-bundle path."- )-- previous_library_id = _current_library_id_or_none()- clear_current_library()- try:- rehydrated = rehydrate_pipe_output_with_crate(- pipe_output,- library_crate=pipe_job_isolated_dynamic_concept.library_crate,- )-- assert rehydrated is pipe_output, "Helper should mutate in place and return the same instance"- assert rehydrated.working_memory_raw is None, "working_memory_raw should be cleared after rehydration"-- greeting_stuff = rehydrated.working_memory.get_stuff("greeting_result")- assert isinstance(greeting_stuff.content, StructuredContent), (- f"Expected StructuredContent for 'greeting_result' after rehydration, got {type(greeting_stuff.content).__name__}"- )- assert hasattr(greeting_stuff.content, "message"), "Hydrated Greeting should have 'message' field"- assert hasattr(greeting_stuff.content, "language"), "Hydrated Greeting should have 'language' field"-- assert global_registry.get_class(name=GREETING_CLASS_NAME) is None, (- f"Helper leaked '{GREETING_CLASS_NAME}' into the global KajsonManager registry"- )- assert _current_library_id_or_none() is None, "Helper should have restored the previous (None) current_library after teardown"- finally:- if previous_library_id is not None:- set_current_library(library_id=previous_library_id)-- async def test_no_crate_path_fails_without_bundle(- self,- pipe_job_isolated_dynamic_concept: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Counterpart: rehydrating without the crate fails when the submitter has no bundle.-- Demonstrates the exact silent-failure surface the helper closes — a- plain `hydrate_working_memory` call against a clean global registry- cannot resolve the dynamic Greeting class. The new helper avoids this- by opening a per-call scoped library from the crate.- """- pipe_output = await self._dispatch_and_get_deferred_pipe_output(- pipe_job_isolated_dynamic_concept,- temporal_client,- )- assert pipe_output.working_memory_raw is not None-- previous_library_id = _current_library_id_or_none()- clear_current_library()- try:- with pytest.raises(PipeJobError):- hydrate_working_memory(pipe_output.working_memory_raw)- finally:- if previous_library_id is not None:- set_current_library(library_id=previous_library_id)-- async def test_repeated_rehydrations_do_not_leak(- self,- pipe_job_isolated_dynamic_concept: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Repeated back-to-back submitter rehydrations leave no residual state.-- Dispatches one workflow, then invokes the helper twice in sequence on- independent copies of the deferred PipeOutput. Catches per-call- scoped-library leaks (open libraries piling up, dynamic classes- bleeding into the global registry, current_library not restored).- Sequential rather than `asyncio.gather`-parallel because the helper- is synchronous — gathering it would only run sequentially anyway.- """- pipe_output_template = await self._dispatch_and_get_deferred_pipe_output(- pipe_job_isolated_dynamic_concept,- temporal_client,- )- raw_payload = pipe_output_template.working_memory_raw- assert raw_payload is not None-- global_registry = KajsonManager.get_class_registry()- assert global_registry.get_class(name=GREETING_CLASS_NAME) is None-- previous_library_id = _current_library_id_or_none()- clear_current_library()- try:- crate = pipe_job_isolated_dynamic_concept.library_crate- for _index in range(2):- pipe_output = pipe_output_template.model_copy(update={"working_memory_raw": dict(raw_payload)})- rehydrated = rehydrate_pipe_output_with_crate(pipe_output, library_crate=crate)- greeting_stuff = rehydrated.working_memory.get_stuff("greeting_result")- assert isinstance(greeting_stuff.content, StructuredContent)- assert hasattr(greeting_stuff.content, "message")- assert hasattr(greeting_stuff.content, "language")- assert _current_library_id_or_none() is None, "Helper should leave current_library back at None after each call"- assert global_registry.get_class(name=GREETING_CLASS_NAME) is None, (- f"Iteration leaked '{GREETING_CLASS_NAME}' into the global KajsonManager registry"- )- finally:- if previous_library_id is not None:- set_current_library(library_id=previous_library_id)diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_combined_pipeline.py b/tests/integration/pipelex/temporal/library_crate/test_wf_combined_pipeline.pydeleted file mode 100644index 2ae9f64dd..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_combined_pipeline.py+++ /dev/null@@ -1,79 +0,0 @@-"""Integration tests for combined controller types through Temporal workflows.--Validates that PipeParallel + PipeCondition nested within a single PipeSequence execute-correctly on Temporal workers via LibraryCrate propagation. This exercises the deepest-child workflow dispatch chain: sequence → parallel children → condition child → outcome child.-Also exercises Phase 3 deferred hydration via the QualityReport inline structure concept.-"""--import uuid-from datetime import timedelta--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import CombinedPipelineTemporalTestData--[email protected](loop_scope="class")-class TestWfCombinedPipeline:- @staticmethod- def _assert_crate_structure(pipe_job: PipeJob) -> None:- crate = pipe_job.library_crate- assert crate is not None, "library_crate should be set on PipeJob"- assert crate.fingerprint, "fingerprint should be non-empty"-- assert len(crate.pipes) == len(CombinedPipelineTemporalTestData.EXPECTED_PIPE_REFS), (- f"Expected {len(CombinedPipelineTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"- )- for pipe_ref in CombinedPipelineTemporalTestData.EXPECTED_PIPE_REFS:- assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"-- async def test_crate_contains_combined_pipes(self, combined_job: PipeJob):- """Verify the crate has all pipe_refs from parallel, condition, and LLM pipes."""- self._assert_crate_structure(combined_job)-- async def test_combined_pipeline_via_temporal(- self,- combined_job: PipeJob,- temporal_client: TemporalClient,- ):- """Nested parallel + condition dispatch produces a structured QualityReport on the worker."""- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=combined_job,- id=workflow_id,- task_queue=task_queue,- execution_timeout=timedelta(seconds=30),- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- for stuff_name in CombinedPipelineTemporalTestData.EXPECTED_STUFF_NAMES:- assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"-- # Verify QualityReport has expected structured fields- report_stuff = working_memory.get_stuff("final_report")- assert isinstance(report_stuff.content, StructuredContent), (- f"Expected StructuredContent for final_report, got {type(report_stuff.content).__name__}"- )- for field_name in CombinedPipelineTemporalTestData.EXPECTED_REPORT_FIELDS:- assert hasattr(report_stuff.content, field_name), f"QualityReport missing field '{field_name}' — deferred hydration may have failed"diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_concept_isolation.py b/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_concept_isolation.pydeleted file mode 100644index c6d02547d..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_concept_isolation.py+++ /dev/null@@ -1,134 +0,0 @@-"""Concurrent concept isolation tests for Temporal workflows.--Validates that two workflows running simultaneously on the same worker can each-define a concept called 'Result' with incompatible structures, and the per-workflow-ClassRegistry scoping (via ContextVar) keeps them isolated.--Without proper scoping, the bare class name 'Result' would collide in the registry,-causing one workflow to deserialize with the wrong field structure — a silent data-corruption bug.--┌──────────────────────────────────────────────────────────────┐-│ Temporal Test Server (in-process) │-├──────────────────────────────────────────────────────────────┤-│ ┌──────────────── Worker (shared) ────────────────────────┐ │-│ │ ┌─ Workflow A ─────────────┐ ┌─ Workflow B ─────────┐ │ │-│ │ │ concept "Result": │ │ concept "Result": │ │ │-│ │ │ score: int │ │ value: text │ │ │-│ │ │ label: text │ │ confidence: number │ │ │-│ │ │ │ │ is_valid: text │ │ │-│ │ │ ClassRegistry A (scoped) │ │ ClassRegistry B │ │ │-│ │ └──────────────────────────┘ └───────────────────────┘ │ │-│ └─────────────────────────────────────────────────────────┘ │-└──────────────────────────────────────────────────────────────┘-"""--import asyncio-import uuid--import pytest-from temporalio.client import Client as TemporalClient--from pipelex import log-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.temporal_hub import get_task_manager-from tests.integration.pipelex.temporal.library_crate.helpers import (- assert_structured_fields,- assert_stuff_names,- assert_text_stuff_names,- execute_workflow,-)-from tests.integration.pipelex.temporal.test_data import ConflictConceptAlphaTestData, ConflictConceptBetaTestData--[email protected]_runnable[email protected](loop_scope="class")-class TestWfConcurrentConceptIsolation:- async def test_alpha_solo(- self,- pipe_run_mode: PipeRunMode,- alpha_concept_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Alpha pipeline alone: Result should have score and label fields."""- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- output = await execute_workflow(alpha_concept_job, temporal_client, task_queue)-- assert_stuff_names(output, ConflictConceptAlphaTestData.EXPECTED_STUFF_NAMES)- assert_structured_fields(output, "alpha_result", ConflictConceptAlphaTestData.EXPECTED_RESULT_FIELDS)- log.info(f"Alpha solo passed (mode={pipe_run_mode})")-- async def test_beta_solo(- self,- pipe_run_mode: PipeRunMode,- beta_concept_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Beta pipeline alone: Result should have value, confidence, and is_valid fields."""- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- output = await execute_workflow(beta_concept_job, temporal_client, task_queue)-- assert_stuff_names(output, ConflictConceptBetaTestData.EXPECTED_STUFF_NAMES)- assert_structured_fields(output, "beta_result", ConflictConceptBetaTestData.EXPECTED_RESULT_FIELDS)- log.info(f"Beta solo passed (mode={pipe_run_mode})")-- async def test_concurrent_different_results(- self,- pipe_run_mode: PipeRunMode,- alpha_concept_job: PipeJob,- beta_concept_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Both pipelines run simultaneously on the same worker.-- Each defines concept 'Result' with different fields. Without per-workflow- ClassRegistry scoping, one would overwrite the other in the registry.- """- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- alpha_output, beta_output = await asyncio.gather(- execute_workflow(alpha_concept_job, temporal_client, task_queue),- execute_workflow(beta_concept_job, temporal_client, task_queue),- )-- # Alpha: Result has score + label- assert_stuff_names(alpha_output, ConflictConceptAlphaTestData.EXPECTED_STUFF_NAMES)- assert_structured_fields(alpha_output, "alpha_result", ConflictConceptAlphaTestData.EXPECTED_RESULT_FIELDS)-- # Beta: Result has value + confidence + is_valid- assert_stuff_names(beta_output, ConflictConceptBetaTestData.EXPECTED_STUFF_NAMES)- assert_structured_fields(beta_output, "beta_result", ConflictConceptBetaTestData.EXPECTED_RESULT_FIELDS)-- # Verify summary outputs are TextContent- assert_text_stuff_names(alpha_output, ["alpha_summary"])- assert_text_stuff_names(beta_output, ["beta_summary"])-- log.info(f"Concurrent concept isolation passed (mode={pipe_run_mode})")-- async def test_repeated_concurrent(- self,- pipe_run_mode: PipeRunMode,- alpha_concept_job: PipeJob,- beta_concept_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Run the concurrent test 5 times to catch intermittent ContextVar race conditions."""- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- for round_index in range(5):- alpha_output, beta_output = await asyncio.gather(- execute_workflow(alpha_concept_job, temporal_client, task_queue),- execute_workflow(beta_concept_job, temporal_client, task_queue),- )-- assert_structured_fields(alpha_output, "alpha_result", ConflictConceptAlphaTestData.EXPECTED_RESULT_FIELDS)- assert_structured_fields(beta_output, "beta_result", ConflictConceptBetaTestData.EXPECTED_RESULT_FIELDS)- log.verbose(f"Round {round_index + 1}/5 passed")-- log.info(f"Repeated concurrent concept isolation passed (mode={pipe_run_mode})")diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_pipe_isolation.py b/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_pipe_isolation.pydeleted file mode 100644index 98058d133..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_concurrent_pipe_isolation.py+++ /dev/null@@ -1,115 +0,0 @@-"""Concurrent pipe isolation tests for Temporal workflows.--Validates that two workflows running simultaneously on the same worker each load-their own LibraryCrate and resolve pipe_refs through per-workflow library scoping-(via ContextVar). Alpha's pipes (about colors) and beta's pipes (about animals)-live in separate domains and are resolved independently on each workflow.--┌──────────────────────────────────────────────────────────────┐-│ Temporal Test Server (in-process) │-├──────────────────────────────────────────────────────────────┤-│ ┌──────────────── Worker (shared) ────────────────────────┐ │-│ │ ┌─ Workflow A ─────────────┐ ┌─ Workflow B ──────────┐ │ │-│ │ │ domain: pipe_conflict_a │ │ domain: pipe_conflict_b│ │ │-│ │ │ alpha_shared_step: │ │ beta_shared_step: │ │ │-│ │ │ prompt: about colors │ │ prompt: about animals│ │-│ │ │ │ │ │ │ │-│ │ │ Library A (ContextVar) │ │ Library B (ContextVar) │ │ │-│ │ └──────────────────────────┘ └────────────────────────┘ │ │-│ └─────────────────────────────────────────────────────────┘ │-└──────────────────────────────────────────────────────────────┘-"""--import asyncio-import uuid--import pytest-from temporalio.client import Client as TemporalClient--from pipelex import log-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.temporal_hub import get_task_manager-from tests.integration.pipelex.temporal.library_crate.helpers import assert_text_stuff_names, execute_workflow-from tests.integration.pipelex.temporal.test_data import ConflictPipeAlphaTestData, ConflictPipeBetaTestData--[email protected]_runnable[email protected](loop_scope="class")-class TestWfConcurrentPipeIsolation:- async def test_pipe_alpha_solo(- self,- pipe_run_mode: PipeRunMode,- alpha_pipe_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Alpha pipeline alone: shared_step should generate text about colors."""- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- output = await execute_workflow(alpha_pipe_job, temporal_client, task_queue)-- assert_text_stuff_names(output, ConflictPipeAlphaTestData.EXPECTED_STUFF_NAMES)- log.info(f"Pipe alpha solo passed (mode={pipe_run_mode})")-- async def test_pipe_beta_solo(- self,- pipe_run_mode: PipeRunMode,- beta_pipe_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Beta pipeline alone: shared_step should generate text about animals."""- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- output = await execute_workflow(beta_pipe_job, temporal_client, task_queue)-- assert_text_stuff_names(output, ConflictPipeBetaTestData.EXPECTED_STUFF_NAMES)- log.info(f"Pipe beta solo passed (mode={pipe_run_mode})")-- async def test_concurrent_pipe_resolution(- self,- pipe_run_mode: PipeRunMode,- alpha_pipe_job: PipeJob,- beta_pipe_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Both pipelines run simultaneously on the same worker.-- Each defines pipe 'shared_step' with a different prompt. Without per-workflow- library scoping, get_required_pipe('shared_step') could resolve the wrong one.- """- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- alpha_output, beta_output = await asyncio.gather(- execute_workflow(alpha_pipe_job, temporal_client, task_queue),- execute_workflow(beta_pipe_job, temporal_client, task_queue),- )-- assert_text_stuff_names(alpha_output, ConflictPipeAlphaTestData.EXPECTED_STUFF_NAMES)- assert_text_stuff_names(beta_output, ConflictPipeBetaTestData.EXPECTED_STUFF_NAMES)-- log.info(f"Concurrent pipe isolation passed (mode={pipe_run_mode})")-- async def test_repeated_concurrent_pipe_resolution(- self,- pipe_run_mode: PipeRunMode,- alpha_pipe_job: PipeJob,- beta_pipe_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Run the concurrent test 5 times to catch intermittent race conditions."""- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- for round_index in range(5):- alpha_output, beta_output = await asyncio.gather(- execute_workflow(alpha_pipe_job, temporal_client, task_queue),- execute_workflow(beta_pipe_job, temporal_client, task_queue),- )-- assert_text_stuff_names(alpha_output, ConflictPipeAlphaTestData.EXPECTED_STUFF_NAMES)- assert_text_stuff_names(beta_output, ConflictPipeBetaTestData.EXPECTED_STUFF_NAMES)- log.verbose(f"Round {round_index + 1}/5 passed")-- log.info(f"Repeated concurrent pipe isolation passed (mode={pipe_run_mode})")diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_cv_batch_screening.py b/tests/integration/pipelex/temporal/library_crate/test_wf_cv_batch_screening.pydeleted file mode 100644index 2632fab78..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_cv_batch_screening.py+++ /dev/null@@ -1,92 +0,0 @@-"""Integration test for the CV batch screening pipeline routed through Temporal.--Validates that a deeply-nested controller stack (PipeSequence -> PipeSequence + PipeBatch ->-PipeSequence -> PipeExtract + PipeLLM) executes correctly when the LibraryCrate ships to-the worker via the PipeJob. Mirrors the pipelex-demos example 21 pipeline.--Runs in dry mode against the in-process Temporal server bundled with the test conftest-(``--temporal-server none``). For live distributed validation, use the-``/temporal-e2e-validate`` skill which dispatches the same bundle through the CLI-against a real Temporal server with split router/runner workers.-"""--import uuid-from datetime import timedelta--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import CvBatchScreeningTemporalTestData--[email protected](loop_scope="class")-class TestWfCvBatchScreening:- @staticmethod- def _assert_crate_structure(pipe_job: PipeJob) -> None:- crate = pipe_job.library_crate- assert crate is not None, "library_crate should be set on PipeJob"- assert crate.fingerprint, "fingerprint should be non-empty"-- assert len(crate.pipes) == len(CvBatchScreeningTemporalTestData.EXPECTED_PIPE_REFS), (- f"Expected {len(CvBatchScreeningTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"- )- for pipe_ref in CvBatchScreeningTemporalTestData.EXPECTED_PIPE_REFS:- assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"-- async def test_crate_contains_full_pipe_tree(self, cv_batch_screening_job: PipeJob):- """All controller and operator pipes (outer + nested PipeSequence + batch branch) are in the crate."""- self._assert_crate_structure(cv_batch_screening_job)-- @pytest.mark.xfail(- strict=False,- reason=(- "PipeBatch dispatching child workflows for the per-CV branch can race under "- "pytest-xdist when concurrent test classes load overlapping dynamic concepts. "- "Passes in isolation and via the /temporal-e2e-validate skill in distributed mode."- ),- )- async def test_cv_batch_screening_via_temporal_dry_run(- self,- cv_batch_screening_job: PipeJob,- temporal_client: TemporalClient,- ):- """End-to-end dry-run dispatch through WfPipeRouter on an in-process worker.-- Exercises:-- - LibraryCrate propagation to the worker for deeply-nested controllers- - Dynamic concept hydration for CandidateProfile, JobRequirements, CandidateMatch- - PipeExtract + PipeLLM operator round-trip through the activity boundary- - PipeBatch fan-out to per-CV child workflows- """- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=cv_batch_screening_job,- id=workflow_id,- task_queue=task_queue,- execution_timeout=timedelta(seconds=60),- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- for stuff_name in CvBatchScreeningTemporalTestData.EXPECTED_STUFF_NAMES:- assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"- stuff = working_memory.get_stuff(stuff_name)- assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py b/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.pydeleted file mode 100644index 2a8cbae48..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_deferred_hydration.py+++ /dev/null@@ -1,141 +0,0 @@-"""Integration tests for deferred WorkingMemory hydration through Temporal workflows.--Validates Phase 3: PipeSequence controllers with dynamic concept classes execute-on Temporal workers. The WorkingMemory is serialized as a raw dict before dispatch,-then hydrated on the worker after load_from_crate() registers the dynamic classes.-"""--import uuid-from collections.abc import Generator--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.core.stuffs.text_content import TextContent-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import make_prepared_greeting_job, rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData--[email protected](scope="class")-def pipe_job_with_dynamic_concept(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """Build a PipeJob from the dynamic concept test bundle (mthds_content with inline structure)."""- yield from pipe_job_from_bundle(- bundle_file=DeferredHydrationTestData.BUNDLE_FILE,- pipe_code=DeferredHydrationTestData.PIPE_CODE,- isolated_registry=is_class_registry_isolated,- )--[email protected](loop_scope="class")-class TestWfDeferredHydration:- async def test_crate_contains_dynamic_concept(self, pipe_job_with_dynamic_concept: PipeJob) -> None:- """The crate should contain the Greeting concept with its structure."""- crate = pipe_job_with_dynamic_concept.library_crate- assert crate is not None-- assert len(crate.pipes) == len(DeferredHydrationTestData.EXPECTED_PIPE_REFS)- for pipe_ref in DeferredHydrationTestData.EXPECTED_PIPE_REFS:- assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"-- # Verify the Greeting concept with structure is in the crate- greeting_ref = f"{DeferredHydrationTestData.DOMAIN}.Greeting"- assert greeting_ref in crate.concepts, f"Expected concept '{greeting_ref}' not found in crate"-- async def test_dynamic_concept_sequence_via_temporal(- self,- pipe_job_with_dynamic_concept: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Full round-trip: dynamic concept classes are registered on the worker via deferred hydration."""- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=pipe_job_with_dynamic_concept,- id=workflow_id,- task_queue=task_queue,- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- for stuff_name in DeferredHydrationTestData.EXPECTED_STUFF_NAMES:- assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"- stuff = working_memory.get_stuff(stuff_name)- assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"-- # Verify greeting_result has StructuredContent with the right fields- greeting_stuff = working_memory.get_stuff("greeting_result")- assert isinstance(greeting_stuff.content, StructuredContent), f"Expected StructuredContent, got {type(greeting_stuff.content).__name__}"- assert hasattr(greeting_stuff.content, "message"), "Greeting content should have 'message' field"- assert hasattr(greeting_stuff.content, "language"), "Greeting content should have 'language' field"-- # Verify summary_result is TextContent- summary_stuff = working_memory.get_stuff("summary_result")- assert isinstance(summary_stuff.content, TextContent)-- async def test_input_wm_deferred_hydration_round_trip(- self,- pipe_job_with_dynamic_concept: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Exercises the full deferred hydration path: input WM with dynamic Greeting concept- is serialized via prepare_for_temporal(), sent through Temporal, and correctly- hydrated on the worker before the pipe sequence runs.- """- # Build a WorkingMemory with a dynamic Greeting stuff pre-populated, then- # prepare the pipe job for Temporal dispatch- prepared_job = make_prepared_greeting_job(pipe_job_with_dynamic_concept, stuff_code="test_input")-- # Verify prepare_for_temporal converted WM to raw dict- assert prepared_job.working_memory is None, "prepare_for_temporal should set working_memory to None"- assert prepared_job.working_memory_raw is not None, "prepare_for_temporal should populate working_memory_raw"- assert "greeting_result" in prepared_job.working_memory_raw.get("root", {}), "Raw WM should contain greeting_result"-- # Dispatch through Temporal — the worker will hydrate working_memory_raw back to typed WM- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=prepared_job,- id=workflow_id,- task_queue=task_queue,- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- # The greeting_result should still exist (was in input WM, and also produced by the first pipe step)- assert working_memory.is_stuff_exists("greeting_result"), "greeting_result should be in output WM"- greeting_stuff_out = working_memory.get_stuff("greeting_result")- assert isinstance(greeting_stuff_out.content, StructuredContent), (- f"Expected StructuredContent after hydration, got {type(greeting_stuff_out.content).__name__}"- )- assert hasattr(greeting_stuff_out.content, "message"), "Hydrated Greeting should have 'message' field"- assert hasattr(greeting_stuff_out.content, "language"), "Hydrated Greeting should have 'language' field"-- # summary_result should also exist (produced by the second pipe step)- assert working_memory.is_stuff_exists("summary_result"), "summary_result should be in output WM"diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_library_crate.py b/tests/integration/pipelex/temporal/library_crate/test_wf_library_crate.pydeleted file mode 100644index 7e8211727..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_library_crate.py+++ /dev/null@@ -1,115 +0,0 @@-"""Integration tests for LibraryCrate propagation through Temporal workflows.--Validates Phase 2 of the master plan: PipeSequence controllers execute on Temporal workers-because the LibraryCrate ships inside the PipeJob and gets loaded on the worker.-Uses a bundle with only native Text concepts to avoid the Layer 1 Kajson issue (Phase 3).-"""--import uuid-from datetime import timedelta--import pytest-from temporalio.client import Client as TemporalClient-from temporalio.client import WorkflowFailureError--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import LibraryCrateTestData--[email protected](loop_scope="class")-class TestWfLibraryCrate:- @staticmethod- def _assert_crate_structure(pipe_job: PipeJob) -> None:- crate = pipe_job.library_crate- assert crate is not None, "library_crate should be set on PipeJob"- assert crate.fingerprint, "fingerprint should be non-empty"-- assert len(crate.pipes) == len(LibraryCrateTestData.EXPECTED_PIPE_REFS), (- f"Expected {len(LibraryCrateTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"- )- for pipe_ref in LibraryCrateTestData.EXPECTED_PIPE_REFS:- assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"-- json_str = crate.model_dump_json()- assert len(json_str) > 0- roundtripped = crate.model_validate_json(json_str)- assert roundtripped.fingerprint == crate.fingerprint-- async def _execute_workflow_and_assert(self, pipe_job: PipeJob, temporal_client: TemporalClient) -> None:- """Execute WfPipeRouter on an inline worker and assert PipeOutput is returned."""- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=pipe_job,- id=workflow_id,- task_queue=task_queue,- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- for stuff_name in LibraryCrateTestData.EXPECTED_STUFF_NAMES:- assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"- stuff = working_memory.get_stuff(stuff_name)- assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"-- async def test_crate_structure_from_dirs(self, pipe_job_from_library_dirs: PipeJob):- self._assert_crate_structure(pipe_job_from_library_dirs)-- async def test_crate_structure_from_mthds_content(self, pipe_job_from_mthds_content: PipeJob):- self._assert_crate_structure(pipe_job_from_mthds_content)-- async def test_pipe_sequence_via_temporal_dry_run(- self,- pipe_job_from_library_dirs: PipeJob,- temporal_client: TemporalClient,- ):- """Proves get_required_pipe() works on the worker for child pipes via LibraryCrate."""- await self._execute_workflow_and_assert(pipe_job_from_library_dirs, temporal_client)-- async def test_pipe_sequence_via_temporal_from_mthds_content(- self,- pipe_job_from_mthds_content: PipeJob,- temporal_client: TemporalClient,- ):- """Same as above but the crate was built from an inline mthds_content string."""- await self._execute_workflow_and_assert(pipe_job_from_mthds_content, temporal_client)-- async def test_missing_crate_fails_pipe_resolution(- self,- pipe_job_from_mthds_content: PipeJob,- temporal_client: TemporalClient,- ):- """A PipeJob without a crate should fail: child pipes can't resolve on worker."""- pipe_job_without_crate = pipe_job_from_mthds_content.model_copy(update={"library_crate": None})-- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- with pytest.raises(WorkflowFailureError):- await temporal_client.execute_workflow(- workflow=WfPipeRouter.run,- arg=pipe_job_without_crate,- id=workflow_id,- task_queue=task_queue,- execution_timeout=timedelta(seconds=10),- )diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_multi_concept_isolation.py b/tests/integration/pipelex/temporal/library_crate/test_wf_multi_concept_isolation.pydeleted file mode 100644index 2f679f061..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_multi_concept_isolation.py+++ /dev/null@@ -1,153 +0,0 @@-"""Multi-concept isolation stress tests for Temporal workflows.--Validates the worst-case scenario: two workflows running simultaneously on the same-worker, each defining MULTIPLE dynamic concepts with the SAME names but incompatible-structures. This exercises the full ClassRegistry scoping path under maximum pressure.--Both workflows define 'Profile' and 'Summary' concepts but with completely different-field structures. If the per-workflow ClassRegistry scoping (via ContextVar) fails,-one workflow's Profile class could be used to deserialize the other's — silently-returning wrong data without any error.--┌──────────────────────────────────────────────────────────────┐-│ Temporal Test Server (in-process) │-├──────────────────────────────────────────────────────────────┤-│ ┌──────────────── Worker (shared) ────────────────────────┐ │-│ │ ┌─ Workflow A ─────────────┐ ┌─ Workflow B ─────────┐ │ │-│ │ │ "Profile": │ │ "Profile": │ │ │-│ │ │ name: text │ │ title: text │ │ │-│ │ │ age: integer │ │ department: text │ │ │-│ │ │ │ │ level: integer │ │ │-│ │ │ "Summary": │ │ "Summary": │ │ │-│ │ │ headline: text │ │ content: text │ │ │-│ │ │ body: text │ │ │ │ │-│ │ └──────────────────────────┘ └───────────────────────┘ │ │-│ └─────────────────────────────────────────────────────────┘ │-└──────────────────────────────────────────────────────────────┘-"""--import asyncio-import uuid--import pytest-from temporalio.client import Client as TemporalClient--from pipelex import log-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.temporal_hub import get_task_manager-from tests.integration.pipelex.temporal.library_crate.helpers import (- assert_structured_fields,- assert_stuff_names,- assert_text_stuff_names,- execute_workflow,-)-from tests.integration.pipelex.temporal.test_data import MultiConceptAlphaTestData, MultiConceptBetaTestData---def _assert_alpha_output(output: PipeOutput) -> None:- """Assert alpha pipeline produced Profile(name, age) + Summary(headline, body) + Text."""- assert_stuff_names(output, MultiConceptAlphaTestData.EXPECTED_STUFF_NAMES)- assert_structured_fields(output, "profile_result", MultiConceptAlphaTestData.EXPECTED_PROFILE_FIELDS)- assert_structured_fields(output, "summary_result", MultiConceptAlphaTestData.EXPECTED_SUMMARY_FIELDS)- assert_text_stuff_names(output, ["final_result"])---def _assert_beta_output(output: PipeOutput) -> None:- """Assert beta pipeline produced Profile(title, department, level) + Summary(content) + Text."""- assert_stuff_names(output, MultiConceptBetaTestData.EXPECTED_STUFF_NAMES)- assert_structured_fields(output, "profile_result", MultiConceptBetaTestData.EXPECTED_PROFILE_FIELDS)- assert_structured_fields(output, "summary_result", MultiConceptBetaTestData.EXPECTED_SUMMARY_FIELDS)- assert_text_stuff_names(output, ["final_result"])--[email protected]_runnable[email protected](loop_scope="class")-class TestWfMultiConceptIsolation:- async def test_multi_alpha_solo(- self,- pipe_run_mode: PipeRunMode,- multi_alpha_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Alpha pipeline alone: Profile(name, age) + Summary(headline, body)."""- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- output = await execute_workflow(multi_alpha_job, temporal_client, task_queue)-- _assert_alpha_output(output)- log.info(f"Multi-concept alpha solo passed (mode={pipe_run_mode})")-- async def test_multi_beta_solo(- self,- pipe_run_mode: PipeRunMode,- multi_beta_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Beta pipeline alone: Profile(title, department, level) + Summary(content)."""- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- output = await execute_workflow(multi_beta_job, temporal_client, task_queue)-- _assert_beta_output(output)- log.info(f"Multi-concept beta solo passed (mode={pipe_run_mode})")-- async def test_concurrent_multi_concept(- self,- pipe_run_mode: PipeRunMode,- multi_alpha_job: PipeJob,- multi_beta_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Both pipelines run simultaneously with conflicting Profile and Summary classes.-- This is the worst-case scenario: two dynamic classes with the same name ('Profile')- and two more ('Summary') coexist on the same worker. Without scoping, alpha's- Profile(name, age) could be used to deserialize beta's Profile(title, department, level).- """- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- alpha_output, beta_output = await asyncio.gather(- execute_workflow(multi_alpha_job, temporal_client, task_queue),- execute_workflow(multi_beta_job, temporal_client, task_queue),- )-- _assert_alpha_output(alpha_output)- _assert_beta_output(beta_output)-- log.info(f"Concurrent multi-concept isolation passed (mode={pipe_run_mode})")-- async def test_high_concurrency_multi_concept(- self,- pipe_run_mode: PipeRunMode,- multi_alpha_job: PipeJob,- multi_beta_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Run 3 pairs concurrently (6 workflows) to stress ContextVar scoping.-- Exercises the scenario where many workflow coroutines are interleaved on the- same worker, maximizing the chance of ContextVar leaking between them.- """- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- results = await asyncio.gather(- execute_workflow(multi_alpha_job, temporal_client, task_queue),- execute_workflow(multi_beta_job, temporal_client, task_queue),- execute_workflow(multi_alpha_job, temporal_client, task_queue),- execute_workflow(multi_beta_job, temporal_client, task_queue),- execute_workflow(multi_alpha_job, temporal_client, task_queue),- execute_workflow(multi_beta_job, temporal_client, task_queue),- )-- # results[0], [2], [4] are alpha; [1], [3], [5] are beta- for index in (0, 2, 4):- _assert_alpha_output(results[index])- for index in (1, 3, 5):- _assert_beta_output(results[index])-- log.info(f"High concurrency multi-concept isolation passed (mode={pipe_run_mode})")diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_batch.py b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_batch.pydeleted file mode 100644index d4513d72f..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_batch.py+++ /dev/null@@ -1,77 +0,0 @@-"""Integration tests for PipeBatch dispatch through Temporal workflows.--Validates that PipeBatch controllers execute on Temporal workers via LibraryCrate propagation.-Each batch item is dispatched as a separate child workflow through TemporalPipeRouter (fan-out).-"""--import uuid-from datetime import timedelta--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import PipeBatchTemporalTestData--[email protected](loop_scope="class")-class TestWfPipeBatch:- @staticmethod- def _assert_crate_structure(pipe_job: PipeJob) -> None:- crate = pipe_job.library_crate- assert crate is not None, "library_crate should be set on PipeJob"- assert crate.fingerprint, "fingerprint should be non-empty"-- assert len(crate.pipes) == len(PipeBatchTemporalTestData.EXPECTED_PIPE_REFS), (- f"Expected {len(PipeBatchTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"- )- for pipe_ref in PipeBatchTemporalTestData.EXPECTED_PIPE_REFS:- assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"-- async def test_crate_contains_batch_pipes(self, batch_job: PipeJob):- """Verify the crate has all pipe_refs including the batch branch pipe."""- self._assert_crate_structure(batch_job)-- @pytest.mark.xfail(- strict=False,- reason=(- "Flaky under pytest-xdist: KajsonDecoderError for temporal_batch_test__Topic "- "when class registration races with sibling workers. Passes in isolation."- ),- )- async def test_batch_sequence_via_temporal(- self,- batch_job: PipeJob,- temporal_client: TemporalClient,- ):- """PipeBatch dispatches one child workflow per list item (fan-out) on the worker."""- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=batch_job,- id=workflow_id,- task_queue=task_queue,- execution_timeout=timedelta(seconds=30),- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- for stuff_name in PipeBatchTemporalTestData.EXPECTED_STUFF_NAMES:- assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"- stuff = working_memory.get_stuff(stuff_name)- assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_compose.py b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_compose.pydeleted file mode 100644index 2a7dcf4bd..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_compose.py+++ /dev/null@@ -1,82 +0,0 @@-"""Integration tests for PipeCompose operator through Temporal workflows.--Validates that PipeCompose operators execute on Temporal workers via LibraryCrate propagation.-Also exercises Phase 3 deferred hydration — the Report concept has an inline structure that-requires dynamic class registration on the worker before PipeCompose can construct it.-"""--import uuid-from datetime import timedelta--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import PipeComposeTemporalTestData--[email protected](loop_scope="class")-class TestWfPipeCompose:- @staticmethod- def _assert_crate_structure(pipe_job: PipeJob) -> None:- crate = pipe_job.library_crate- assert crate is not None, "library_crate should be set on PipeJob"- assert crate.fingerprint, "fingerprint should be non-empty"-- assert len(crate.pipes) == len(PipeComposeTemporalTestData.EXPECTED_PIPE_REFS), (- f"Expected {len(PipeComposeTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"- )- for pipe_ref in PipeComposeTemporalTestData.EXPECTED_PIPE_REFS:- assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"-- async def test_crate_contains_compose_pipes(self, compose_job: PipeJob):- """Verify the crate has all pipe_refs and the Report concept."""- self._assert_crate_structure(compose_job)- crate = compose_job.library_crate- assert crate is not None- report_ref = f"{PipeComposeTemporalTestData.DOMAIN}.Report"- assert report_ref in crate.concepts, f"Expected concept_ref '{report_ref}' not found in crate"-- async def test_compose_sequence_via_temporal(- self,- compose_job: PipeJob,- temporal_client: TemporalClient,- ):- """PipeCompose constructs a structured Report on the worker after deferred hydration."""- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=compose_job,- id=workflow_id,- task_queue=task_queue,- execution_timeout=timedelta(seconds=30),- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- for stuff_name in PipeComposeTemporalTestData.EXPECTED_STUFF_NAMES:- assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"-- # Verify the Report has the expected structured fields (deferred hydration worked)- report_stuff = working_memory.get_stuff("final_report")- assert isinstance(report_stuff.content, StructuredContent), (- f"Expected StructuredContent for final_report, got {type(report_stuff.content).__name__}"- )- for field_name in PipeComposeTemporalTestData.EXPECTED_REPORT_FIELDS:- assert hasattr(report_stuff.content, field_name), f"Report missing field '{field_name}' — deferred hydration may have failed"diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_condition.py b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_condition.pydeleted file mode 100644index 5def39404..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_condition.py+++ /dev/null@@ -1,71 +0,0 @@-"""Integration tests for PipeCondition dispatch through Temporal workflows.--Validates that PipeCondition controllers execute on Temporal workers via LibraryCrate propagation.-The condition routes to different outcome pipes based on an expression — each outcome is a-child workflow dispatched through TemporalPipeRouter.-"""--import uuid-from datetime import timedelta--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import PipeConditionTemporalTestData--[email protected](loop_scope="class")-class TestWfPipeCondition:- @staticmethod- def _assert_crate_structure(pipe_job: PipeJob) -> None:- crate = pipe_job.library_crate- assert crate is not None, "library_crate should be set on PipeJob"- assert crate.fingerprint, "fingerprint should be non-empty"-- assert len(crate.pipes) == len(PipeConditionTemporalTestData.EXPECTED_PIPE_REFS), (- f"Expected {len(PipeConditionTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"- )- for pipe_ref in PipeConditionTemporalTestData.EXPECTED_PIPE_REFS:- assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"-- async def test_crate_contains_condition_pipes(self, condition_job: PipeJob):- """Verify the crate has all pipe_refs including condition outcomes."""- self._assert_crate_structure(condition_job)-- async def test_condition_sequence_via_temporal(- self,- condition_job: PipeJob,- temporal_client: TemporalClient,- ):- """PipeCondition dispatches the selected outcome as a child workflow on the worker."""- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=condition_job,- id=workflow_id,- task_queue=task_queue,- execution_timeout=timedelta(seconds=30),- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- for stuff_name in PipeConditionTemporalTestData.EXPECTED_STUFF_NAMES:- assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"- stuff = working_memory.get_stuff(stuff_name)- assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"diff --git a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_parallel.py b/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_parallel.pydeleted file mode 100644index b078c0dd6..000000000--- a/tests/integration/pipelex/temporal/library_crate/test_wf_pipe_parallel.py+++ /dev/null@@ -1,68 +0,0 @@-"""Integration tests for PipeParallel dispatch through Temporal workflows.--Validates that PipeParallel controllers execute on Temporal workers via LibraryCrate propagation.-Each parallel branch is dispatched as a concurrent child workflow through TemporalPipeRouter.-"""--import uuid--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import PipeParallelTemporalTestData--[email protected](loop_scope="class")-class TestWfPipeParallel:- @staticmethod- def _assert_crate_structure(pipe_job: PipeJob) -> None:- crate = pipe_job.library_crate- assert crate is not None, "library_crate should be set on PipeJob"- assert crate.fingerprint, "fingerprint should be non-empty"-- assert len(crate.pipes) == len(PipeParallelTemporalTestData.EXPECTED_PIPE_REFS), (- f"Expected {len(PipeParallelTemporalTestData.EXPECTED_PIPE_REFS)} pipes, got {len(crate.pipes)}"- )- for pipe_ref in PipeParallelTemporalTestData.EXPECTED_PIPE_REFS:- assert pipe_ref in crate.pipes, f"Expected pipe_ref '{pipe_ref}' not found in crate"-- async def test_crate_contains_parallel_pipes(self, parallel_job: PipeJob):- """Verify the crate has all pipe_refs including parallel branches."""- self._assert_crate_structure(parallel_job)-- async def test_parallel_sequence_via_temporal(- self,- parallel_job: PipeJob,- temporal_client: TemporalClient,- ):- """PipeParallel dispatches branches as concurrent child workflows on the worker."""- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=parallel_job,- id=workflow_id,- task_queue=task_queue,- )-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- for stuff_name in PipeParallelTemporalTestData.EXPECTED_STUFF_NAMES:- assert working_memory.is_stuff_exists(stuff_name), f"Expected stuff '{stuff_name}' missing from output"- stuff = working_memory.get_stuff(stuff_name)- assert stuff.content is not None, f"Stuff '{stuff_name}' has no content"diff --git a/tests/integration/pipelex/temporal/test_activity_error_boundary.py b/tests/integration/pipelex/temporal/test_activity_error_boundary.pydeleted file mode 100644index cde348e3b..000000000--- a/tests/integration/pipelex/temporal/test_activity_error_boundary.py+++ /dev/null@@ -1,245 +0,0 @@-"""Integration test for the Temporal activity error boundary (Followup 5).--Drives a real activity through a real Temporal worker, makes its inner generate-call raise a real ``CogtError``, and asserts what ``TemporalError.from_app_error``-observes on the workflow side.--Each in-scope activity is decorated with ``@convert_pipelex_errors``, so a raised-``PipelexError`` becomes a ``TemporalError`` that packs ``to_error_report()`` into-``ApplicationError.details`` and derives ``non_retryable`` from the error's-``InferenceErrorCategory``. Without that boundary, Temporal's default failure-converter auto-wraps the raw error, ``from_app_error`` lands in its-``error_report is None`` fallback, and the category-aware retry decision never runs.--Two probe workflows exercise the boundary: one over an LLM activity-(``act_llm_gen_text``) and one over a non-LLM activity-(``act_extract_gen_extract_pages``), proving the wiring is not LLM-specific.-"""--import uuid-from datetime import timedelta-from typing import Any--import pytest-from pydantic import BaseModel-from pytest_mock import MockerFixture-from temporalio import workflow-from temporalio.client import Client as TemporalClient-from temporalio.common import RetryPolicy-from temporalio.exceptions import ActivityError, ApplicationError-from temporalio.worker import UnsandboxedWorkflowRunner, Worker--from pipelex import log-from pipelex.cogt.content_generation.assignment_models import ExtractAssignment, LLMAssignment-from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams-from pipelex.cogt.exceptions import CogtError, InferenceErrorCategory-from pipelex.cogt.extract.extract_input import ExtractInput-from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams-from pipelex.cogt.llm.llm_prompt import LLMPrompt-from pipelex.hub import get_model_deck-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.tprl.temporal_error import TemporalError-from pipelex.temporal.tprl_content_generation.act_extract_generate import act_extract_gen_extract_pages-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text--ACTIVITY_FAILURE_MESSAGE = "simulated activity failure"---class ErrorBoundaryProbeResult(BaseModel):- """What the workflow side observed after re-wrapping the activity failure.-- Returned (not re-raised) by the probe workflow so the test can assert on the- payload that survived the activity → workflow boundary.- """-- non_retryable: bool- error_report: dict[str, Any] | None = None---def _probe_result_from_activity_error(exc: ActivityError, activity_name: str) -> ErrorBoundaryProbeResult:- """Run the workflow-side bridge on an ``ActivityError`` and capture the conversion.-- Temporal wraps an activity failure's cause as an ``ApplicationError``; anything- else (a timeout, a cancellation) means the probe never reached the bridge and- is a hard test error.- """- cause = exc.cause- if not isinstance(cause, ApplicationError):- unexpected_cause_msg = f"{activity_name} should fail with an ApplicationError cause, got {type(cause).__name__}: {cause}"- raise TypeError(unexpected_cause_msg) from exc- temporal_error = TemporalError.from_app_error(exc=cause)- return ErrorBoundaryProbeResult(- non_retryable=temporal_error.non_retryable,- error_report=temporal_error.error_report,- )--[email protected](name="wf_error_boundary_probe")-class WfErrorBoundaryProbe:- """Executes the real ``act_llm_gen_text`` activity, expects it to fail, and reports the conversion.-- ``maximum_attempts=1`` on the activity retry policy is mandatory: a failure- that is (wrongly) classified retryable would otherwise loop until the timeout- and hang the test. The probe only cares about the first hop.- """-- @workflow.run- async def run(self, llm_assignment: LLMAssignment) -> ErrorBoundaryProbeResult:- try:- await workflow.execute_activity(- act_llm_gen_text,- arg=llm_assignment,- start_to_close_timeout=timedelta(seconds=30),- retry_policy=RetryPolicy(maximum_attempts=1),- )- except ActivityError as exc:- return _probe_result_from_activity_error(exc=exc, activity_name="act_llm_gen_text")- unreachable_msg = "act_llm_gen_text was expected to fail"- raise AssertionError(unreachable_msg)--[email protected](name="wf_extract_error_boundary_probe")-class WfExtractErrorBoundaryProbe:- """Same probe over the non-LLM ``act_extract_gen_extract_pages`` activity.-- Proves the error boundary is wired on every in-scope activity, not just the- LLM ones. See ``WfErrorBoundaryProbe`` for why ``maximum_attempts=1`` is set.- """-- @workflow.run- async def run(self, extract_assignment: ExtractAssignment) -> ErrorBoundaryProbeResult:- try:- await workflow.execute_activity(- act_extract_gen_extract_pages,- arg=extract_assignment,- start_to_close_timeout=timedelta(seconds=30),- retry_policy=RetryPolicy(maximum_attempts=1),- )- except ActivityError as exc:- return _probe_result_from_activity_error(exc=exc, activity_name="act_extract_gen_extract_pages")- unreachable_msg = "act_extract_gen_extract_pages was expected to fail"- raise AssertionError(unreachable_msg)--[email protected](loop_scope="class")-class TestActivityErrorBoundary:- @pytest.mark.parametrize(- ("error_category", "expected_non_retryable"),- [- pytest.param(InferenceErrorCategory.CONFIGURATION, True, id="configuration-non-retryable"),- pytest.param(InferenceErrorCategory.TRANSIENT, False, id="transient-retryable"),- pytest.param(None, False, id="category-less-fallback"),- ],- )- async def test_real_activity_failure_surfaces_error_report_on_workflow_side(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- error_category: InferenceErrorCategory | None,- expected_non_retryable: bool,- ) -> None:- """A real ``CogtError`` raised inside ``act_llm_gen_text`` must reach the- workflow side as a ``TemporalError`` carrying a populated ``ErrorReport``- and the category-aware ``non_retryable`` flag.- """- # Make the activity's inner generate call raise a real CogtError. The- # worker runs in-process, so patching the activity module's imported- # name reaches the activity when it executes.- raised_error = CogtError(ACTIVITY_FAILURE_MESSAGE, error_category=error_category)- mocker.patch(- "pipelex.temporal.tprl_content_generation.act_llm_generate.llm_gen_text",- new=mocker.AsyncMock(side_effect=raised_error),- )-- llm_assignment = LLMAssignment(- job_metadata=JobMetadata(user_id="test", pipeline_run_id="test"),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- llm_setting=get_model_deck().get_llm_setting(llm_choice="$testing-text"),- llm_prompt=LLMPrompt(user_text="never reaches the model — llm_gen_text is mocked"),- )-- task_queue = f"q_err_boundary_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_err_boundary_{uuid.uuid4().hex[:8]}"-- async with Worker(- temporal_client,- task_queue=task_queue,- workflows=[WfErrorBoundaryProbe],- activities=[act_llm_gen_text],- workflow_runner=UnsandboxedWorkflowRunner(),- ):- result: ErrorBoundaryProbeResult = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- WfErrorBoundaryProbe.run,- arg=llm_assignment,- id=workflow_id,- task_queue=task_queue,- )-- log.info(f"ErrorBoundaryProbeResult: {result}")-- assert result.non_retryable is expected_non_retryable, (- f"non_retryable should be {expected_non_retryable} for category={error_category}, got {result.non_retryable}"- )- assert result.error_report is not None, "the structured ErrorReport must survive the activity → workflow boundary"- assert result.error_report["error_type"] == "CogtError"- assert result.error_report["message"] == ACTIVITY_FAILURE_MESSAGE- if error_category is not None:- assert result.error_report["error_category"] == error_category- assert result.error_report["retryable"] is (not expected_non_retryable)- else:- # A category-less CogtError has no retryability signal — to_error_report()- # drops the None-valued field, so the key is absent from the report.- assert "retryable" not in result.error_report, "a category-less CogtError must carry no retryability signal in its report"-- async def test_real_non_llm_activity_failure_surfaces_error_report_on_workflow_side(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- ) -> None:- """The boundary is wired on non-LLM activities too: a ``CogtError`` raised- inside ``act_extract_gen_extract_pages`` must reach the workflow side as a- ``TemporalError`` with a populated ``ErrorReport`` and category-aware- ``non_retryable``.- """- raised_error = CogtError(ACTIVITY_FAILURE_MESSAGE, error_category=InferenceErrorCategory.CONFIGURATION)- mocker.patch(- "pipelex.temporal.tprl_content_generation.act_extract_generate.extract_gen_pages_and_store",- new=mocker.AsyncMock(side_effect=raised_error),- )-- extract_assignment = ExtractAssignment(- job_metadata=JobMetadata(user_id="test", pipeline_run_id="test"),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- extract_handle="extract-handle-never-reached",- extract_input=ExtractInput(document_uri="file://never-reached.pdf"),- extract_job_params=ExtractJobParams.make_default_extract_job_params(),- extract_job_config=ExtractJobConfig(),- )-- task_queue = f"q_err_boundary_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_err_boundary_{uuid.uuid4().hex[:8]}"-- async with Worker(- temporal_client,- task_queue=task_queue,- workflows=[WfExtractErrorBoundaryProbe],- activities=[act_extract_gen_extract_pages],- workflow_runner=UnsandboxedWorkflowRunner(),- ):- result: ErrorBoundaryProbeResult = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- WfExtractErrorBoundaryProbe.run,- arg=extract_assignment,- id=workflow_id,- task_queue=task_queue,- )-- log.info(f"ErrorBoundaryProbeResult: {result}")-- assert result.non_retryable is True, "a CONFIGURATION CogtError must be non-retryable on the workflow side"- assert result.error_report is not None, "the structured ErrorReport must survive the activity → workflow boundary"- assert result.error_report["error_type"] == "CogtError"- assert result.error_report["message"] == ACTIVITY_FAILURE_MESSAGE- assert result.error_report["error_category"] == InferenceErrorCategory.CONFIGURATION- assert result.error_report["retryable"] is Falsediff --git a/tests/integration/pipelex/temporal/test_data.py b/tests/integration/pipelex/temporal/test_data.pydeleted file mode 100644index d9674edb8..000000000--- a/tests/integration/pipelex/temporal/test_data.py+++ /dev/null@@ -1,507 +0,0 @@-from typing import ClassVar--from pipelex.core.concepts.concept_factory import ConceptFactory-from pipelex.core.concepts.native.concept_native import NativeConceptCode-from pipelex.core.stuffs.image_content import ImageContent-from pipelex.core.stuffs.list_content import ListContent-from pipelex.core.stuffs.stuff import Stuff-from pipelex.core.stuffs.stuff_factory import StuffFactory-from pipelex.core.stuffs.text_content import TextContent---class PipeTestCases:- SYSTEM_PROMPT = "You are a pirate, you always talk like a pirate."- USER_PROMPT = "In 3 sentences, tell me about the sea."- USER_TEXT_TRICKY_1 = """- When my son was 7 he was 3ft tall. When he was 8 he was 4ft tall. When he was 9 he was 5ft tall.- How tall do you think he was when he was 12? and at 15?- """-- # Create simple Stuff objects- SIMPLE_STUFF = StuffFactory.make_stuff(- name="text",- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),- content=TextContent(text="Describe a t-shirt in 2 sentences"),- )-- IMG_EXPENSE_REPORT_1 = "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/invoices/invoice_1.png"- IMG_FASHION_PHOTO_1 = "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/fashion/fashion_photo_1.jpg"- JOB_OFFER_PDF_LOCAL = "tests/data/documents/Job-Offer.pdf"- SIMPLE_STUFF_TEXT = StuffFactory.make_stuff(- name="text",- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),- content=TextContent(text="Describe a t-shirt in 2 sentences"),- )- SIMPLE_STUFF_IMAGE = StuffFactory.make_stuff(- name="image",- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.IMAGE),- content=ImageContent(url=IMG_FASHION_PHOTO_1),- )-- BATCH_TEST: ClassVar[list[tuple[str, Stuff, str, str]]] = [ # pipe_code, stuff, input_list_stuff_name, input_item_stuff_name- (- "batch_test",- StuffFactory.make_stuff(- concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),- name="colors",- content=ListContent(- items=[- TextContent(text="blue"),- TextContent(text="red"),- TextContent(text="green"),- ]- ),- ),- "colors",- "color",- ),- ]- STUFF_AND_PIPE: ClassVar[list[tuple[str, Stuff, str]]] = [ # topic, stuff, pipe_code- # TODO: fix testing implict concept- # (- # "Process Simple Text",- # SIMPLE_STUFF,- # "test_implicit_concept",- # ),- # (- # "Process Simple Text",- # SIMPLE_STUFF_TEXT,- # "simple_llm_test_from_text",- # ),- (- "Process Simple Image",- SIMPLE_STUFF_IMAGE,- "simple_llm_test_from_image",- ),- ]-- NO_INPUT: ClassVar[list[tuple[str, str]]] = [ # topic, pipe- (- "Test with no input",- "test_no_input",- ),- (- "Imagine nature products of different colors",- "imagine_nature_product_list",- ),- (- "Create characters",- "create_characters",- ),- ]---class LibraryCrateTestData:- """Test constants for LibraryCrate integration tests.-- Uses a PipeSequence bundle with only native Text concepts (no dynamic classes)- to avoid the Layer 1 Kajson deserialization issue (Phase 3). This still fully- tests Layer 2 (pipe resolution via get_required_pipe on the worker).- """-- BUNDLE_DIR: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate"- BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate/native_text_sequence.mthds"- PIPE_CODE: ClassVar[str] = "native_text_sequence"- DOMAIN: ClassVar[str] = "native_text_test"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "native_text_test.native_text_sequence",- "native_text_test.step_one",- "native_text_test.step_two",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "step_one_result",- "step_two_result",- ]---class DeferredHydrationTestData:- """Test constants for Phase 3 deferred hydration integration tests.-- Uses a PipeSequence bundle with a custom concept (Greeting) that has an inline- structure — this triggers dynamic class generation and exercises the full- deferred hydration + scoped ClassRegistry path.- """-- BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate/dynamic_concept_sequence.mthds"- PIPE_CODE: ClassVar[str] = "dynamic_greeting_sequence"- DOMAIN: ClassVar[str] = "dynamic_concept_test"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "dynamic_concept_test.dynamic_greeting_sequence",- "dynamic_concept_test.generate_greeting",- "dynamic_concept_test.summarize_greeting",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "greeting_result",- "summary_result",- ]---_CRATE_DIR: str = "tests/integration/pipelex/temporal/library_crate"---class ConflictConceptAlphaTestData:- """Concept 'Result' with score (integer) + label (text).-- Paired with ConflictConceptBetaTestData which defines a different 'Result'.- Tests that per-workflow ClassRegistry scoping keeps them isolated.- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/conflict_concept_alpha.mthds"- PIPE_CODE: ClassVar[str] = "alpha_pipeline"- DOMAIN: ClassVar[str] = "conflict_alpha"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "conflict_alpha.alpha_pipeline",- "conflict_alpha.alpha_generate",- "conflict_alpha.alpha_summarize",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "alpha_result",- "alpha_summary",- ]-- EXPECTED_RESULT_FIELDS: ClassVar[list[str]] = ["score", "label"]---class ConflictConceptBetaTestData:- """Concept 'Result' with value (text) + confidence (number) + is_valid (text).-- Paired with ConflictConceptAlphaTestData which defines a different 'Result'.- Tests that per-workflow ClassRegistry scoping keeps them isolated.- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/conflict_concept_beta.mthds"- PIPE_CODE: ClassVar[str] = "beta_pipeline"- DOMAIN: ClassVar[str] = "conflict_beta"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "conflict_beta.beta_pipeline",- "conflict_beta.beta_generate",- "conflict_beta.beta_summarize",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "beta_result",- "beta_summary",- ]-- EXPECTED_RESULT_FIELDS: ClassVar[list[str]] = ["value", "confidence", "is_valid"]---class ConflictPipeAlphaTestData:- """Pipe 'alpha_shared_step' as PipeLLM about colors.-- Paired with ConflictPipeBetaTestData. Tests that per-workflow library scoping- via ContextVar resolves the correct pipe_ref for each workflow.- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/conflict_pipe_alpha.mthds"- PIPE_CODE: ClassVar[str] = "pipe_alpha_pipeline"- DOMAIN: ClassVar[str] = "pipe_conflict_alpha"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "pipe_conflict_alpha.pipe_alpha_pipeline",- "pipe_conflict_alpha.alpha_shared_step",- "pipe_conflict_alpha.alpha_finalize",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "step_result",- "final_result",- ]---class ConflictPipeBetaTestData:- """Pipe 'beta_shared_step' as PipeLLM about animals.-- Paired with ConflictPipeAlphaTestData. Tests that per-workflow library scoping- via ContextVar resolves the correct pipe_ref for each workflow.- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/conflict_pipe_beta.mthds"- PIPE_CODE: ClassVar[str] = "pipe_beta_pipeline"- DOMAIN: ClassVar[str] = "pipe_conflict_beta"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "pipe_conflict_beta.pipe_beta_pipeline",- "pipe_conflict_beta.beta_shared_step",- "pipe_conflict_beta.beta_finalize",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "step_result",- "final_result",- ]---class MultiConceptAlphaTestData:- """Profile(name, age) + Summary(headline, body).-- Paired with MultiConceptBetaTestData which defines different Profile and Summary.- Tests worst-case: multiple same-named dynamic classes across concurrent workflows.- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/multi_concept_alpha.mthds"- PIPE_CODE: ClassVar[str] = "multi_alpha_pipeline"- DOMAIN: ClassVar[str] = "multi_alpha"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "multi_alpha.multi_alpha_pipeline",- "multi_alpha.multi_alpha_generate_profile",- "multi_alpha.multi_alpha_generate_summary",- "multi_alpha.multi_alpha_finalize",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "profile_result",- "summary_result",- "final_result",- ]-- EXPECTED_PROFILE_FIELDS: ClassVar[list[str]] = ["name", "age"]- EXPECTED_SUMMARY_FIELDS: ClassVar[list[str]] = ["headline", "body"]---class MultiConceptBetaTestData:- """Profile(title, department, level) + Summary(content).-- Paired with MultiConceptAlphaTestData which defines different Profile and Summary.- Tests worst-case: multiple same-named dynamic classes across concurrent workflows.- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/multi_concept_beta.mthds"- PIPE_CODE: ClassVar[str] = "multi_beta_pipeline"- DOMAIN: ClassVar[str] = "multi_beta"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "multi_beta.multi_beta_pipeline",- "multi_beta.multi_beta_generate_profile",- "multi_beta.multi_beta_generate_summary",- "multi_beta.multi_beta_finalize",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "profile_result",- "summary_result",- "final_result",- ]-- EXPECTED_PROFILE_FIELDS: ClassVar[list[str]] = ["title", "department", "level"]- EXPECTED_SUMMARY_FIELDS: ClassVar[list[str]] = ["content"]---class PipeConditionTemporalTestData:- """PipeCondition within a PipeSequence for Temporal child workflow dispatch testing."""-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_condition.mthds"- PIPE_CODE: ClassVar[str] = "temporal_condition_sequence"- DOMAIN: ClassVar[str] = "temporal_condition_test"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "temporal_condition_test.temporal_condition_sequence",- "temporal_condition_test.generate_category",- "temporal_condition_test.route_by_category",- "temporal_condition_test.handle_alpha",- "temporal_condition_test.handle_beta",- "temporal_condition_test.handle_default",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "category_text",- "routed_result",- ]---class PipeParallelTemporalTestData:- """PipeParallel within a PipeSequence for Temporal concurrent child workflow dispatch testing."""-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_parallel.mthds"- PIPE_CODE: ClassVar[str] = "temporal_parallel_sequence"- DOMAIN: ClassVar[str] = "temporal_parallel_test"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "temporal_parallel_test.temporal_parallel_sequence",- "temporal_parallel_test.analyze_in_parallel",- "temporal_parallel_test.branch_tone",- "temporal_parallel_test.branch_length",- "temporal_parallel_test.summarize_results",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "tone_result",- "length_result",- "summary",- ]---class PipeBatchTemporalTestData:- """PipeBatch within a PipeSequence for Temporal fan-out child workflow dispatch testing."""-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_batch.mthds"- PIPE_CODE: ClassVar[str] = "temporal_batch_sequence"- DOMAIN: ClassVar[str] = "temporal_batch_test"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "temporal_batch_test.temporal_batch_sequence",- "temporal_batch_test.temporal_generate_topics",- "temporal_batch_test.batch_temporal_describe_topics",- "temporal_batch_test.temporal_describe_topic",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "topics",- "notes",- ]---class PipeComposeTemporalTestData:- """PipeCompose operator within a PipeSequence with deferred hydration of Report concept."""-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_compose.mthds"- PIPE_CODE: ClassVar[str] = "temporal_compose_sequence"- DOMAIN: ClassVar[str] = "temporal_compose_test"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "temporal_compose_test.temporal_compose_sequence",- "temporal_compose_test.generate_title",- "temporal_compose_test.generate_body",- "temporal_compose_test.compose_report",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "title_text",- "body_text",- "final_report",- ]-- EXPECTED_REPORT_FIELDS: ClassVar[list[str]] = ["title", "body"]---class CombinedPipelineTemporalTestData:- """Combined PipeParallel + PipeCondition in a PipeSequence for nested Temporal dispatch testing."""-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_combined.mthds"- PIPE_CODE: ClassVar[str] = "temporal_combined_pipeline"- DOMAIN: ClassVar[str] = "temporal_combined_test"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "temporal_combined_test.temporal_combined_pipeline",- "temporal_combined_test.parallel_generate",- "temporal_combined_test.generate_part_a",- "temporal_combined_test.generate_part_b",- "temporal_combined_test.quality_gate",- "temporal_combined_test.handle_failure",- "temporal_combined_test.produce_report",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "part_a",- "part_b",- "final_report",- ]-- EXPECTED_REPORT_FIELDS: ClassVar[list[str]] = ["assessment", "confidence"]---class PayloadCodecPipelineTestData:- """Test constants for StoragePayloadCodec integration tests with real pipe execution.-- Reuses existing bundles to prove the codec is transparent — same pipelines- work unchanged when payloads are offloaded to storage.- """-- # Reuse native_text_sequence for single/multi-step codec tests- NATIVE_BUNDLE_FILE: ClassVar[str] = LibraryCrateTestData.BUNDLE_FILE- NATIVE_PIPE_CODE: ClassVar[str] = LibraryCrateTestData.PIPE_CODE- NATIVE_EXPECTED_STUFF_NAMES: ClassVar[list[str]] = LibraryCrateTestData.EXPECTED_STUFF_NAMES-- # Reuse dynamic_concept_sequence for dynamic concept + codec test- DYNAMIC_BUNDLE_FILE: ClassVar[str] = DeferredHydrationTestData.BUNDLE_FILE- DYNAMIC_PIPE_CODE: ClassVar[str] = DeferredHydrationTestData.PIPE_CODE- DYNAMIC_EXPECTED_STUFF_NAMES: ClassVar[list[str]] = DeferredHydrationTestData.EXPECTED_STUFF_NAMES-- # Codec config- SIZE_THRESHOLD: ClassVar[int] = 1024- STORAGE_PREFIX: ClassVar[str] = "test-codec-pipeline/"---class LargePayloadTestData:- """Test constants for large payload stress testing through Temporal with codec."""-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/large_payload_sequence.mthds"- PIPE_CODE: ClassVar[str] = "large_payload_sequence"- DOMAIN: ClassVar[str] = "large_payload_test"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "large_payload_test.large_payload_sequence",- "large_payload_test.verbose_step_one",- "large_payload_test.verbose_step_two",- "large_payload_test.verbose_step_three",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "step_one_result",- "step_two_result",- "step_three_result",- ]-- SIZE_THRESHOLD: ClassVar[int] = 1024- STORAGE_PREFIX: ClassVar[str] = "test-large-payload/"---class CvBatchScreeningTemporalTestData:- """Nested-controller CV batch screening pipeline sourced from pipelex-demos example 21.-- Exercises PipeSequence -> PipeSequence -> PipeBatch -> PipeSequence with PipeExtract +- PipeLLM operators in both the inner job-offer branch and the per-CV batch branch.- Used by the e2e suite (direct mode), the temporal integration suite (in-process- server + worker), and the `/temporal-e2e-validate` skill (distributed 3-process).- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/cv_batch_screening.mthds"- INPUTS_FILE: ClassVar[str] = f"{_CRATE_DIR}/cv_batch_screening_inputs.json"- PIPE_CODE: ClassVar[str] = "batch_analyze_cvs_for_job_offer"- DOMAIN: ClassVar[str] = "cv_batch_screening"-- EXPECTED_PIPE_REFS: ClassVar[list[str]] = [- "cv_batch_screening.batch_analyze_cvs_for_job_offer",- "cv_batch_screening.prepare_job_offer",- "cv_batch_screening.extract_one_job_offer",- "cv_batch_screening.analyze_job_requirements",- "cv_batch_screening.process_cv",- "cv_batch_screening.extract_one_cv",- "cv_batch_screening.analyze_one_cv",- "cv_batch_screening.analyze_match",- ]-- EXPECTED_STUFF_NAMES: ClassVar[list[str]] = [- "job_requirements",- "match_analyses",- ]-- EXPECTED_CANDIDATE_MATCH_FIELDS: ClassVar[list[str]] = [- "match_score",- "strengths",- "gaps",- "overall_assessment",- ]---class PipeOcrTestCases:- PIPE_OCR_IMAGE_TEST_CASES: ClassVar[list[str]] = [- # LOCAL- "tests/data/documents/solar_system.png",- # REMOTE- "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/documents/solar_system.png",- ]- PIPE_OCR_PDF_TEST_CASES: ClassVar[list[str]] = [- # LOCAL- "tests/data/documents/solar_system.pdf",- # REMOTE- "https://storage.googleapis.com/public_test_files_7fa6_4277_9ab/documents/solar_system.pdf",- ]diff --git a/tests/integration/pipelex/temporal/test_dry_run_graph_in_process.py b/tests/integration/pipelex/temporal/test_dry_run_graph_in_process.pydeleted file mode 100644index 0026ba652..000000000--- a/tests/integration/pipelex/temporal/test_dry_run_graph_in_process.py+++ /dev/null@@ -1,116 +0,0 @@-"""Mode-1 Temporal guard: the in-process graph dry-run must stay in-process under a Temporal hub.--Phase-2 companion to ``test_validate_sweep_stays_in_process``: where that test proves the-*validation sweep* never dispatches to Temporal, this one proves the *graph-producing dry-run*-(``dry_run_pipe_in_process``) runs fully in-process and traces into an in-memory event log even-when the hub default router is the real ``TemporalPipeRouter`` and the hub default content-generator is ``ContentGeneratorInWorkflow`` (wired by this suite's ``boot_temporal`` fixture,-exactly as a Temporal-enabled worker/API process is wired).--Contract asserted (TODOS.md Phase 2):--- (a) a correct, non-empty ``GraphSpec`` comes back, covering the whole controller topology;-- (b) ZERO workflows/activities are dispatched (spy on ``WorkflowExecutor.execute_workflow``);-- (c) no file/DDB transport is touched (``make_event_log`` is forbidden — the scoped in-memory- instance is the single transport);-- the zero-dispatch guarantee holds with the DRY mock at the LEAF (Part B): DRY routes through- the hub-resolved content generator, so the scoped inline generator is what keeps the run- in-process — this is what ``scoped_content_generator`` exists for;-- tracer-key alignment: emit and assemble share the ``pipeline_run_id`` partition by- construction (a divergent key would yield an empty graph, so the non-empty assertion pins it).-"""--from pathlib import Path--import pytest-from pytest_mock import MockerFixture--from pipelex.cogt.content_generation.content_generator import ContentGenerator-from pipelex.graph.graphspec import GraphSpec-from pipelex.hub import get_library_manager, get_pipelex_hub, get_required_pipe, scoped_current_library-from pipelex.pipe_run.dry_run_in_process import dry_run_pipe_in_process-from pipelex.pipeline.execution_seams import acquire_library-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor-from pipelex.temporal.tprl_content_generation.content_generator_in_workflow import ContentGeneratorInWorkflow-from pipelex.temporal.tprl_pipe.temporal_pipe_router import TemporalPipeRouter-from pipelex.tracing.in_memory_event_log import InMemoryEventLog-from pipelex.tracing.trace_events import UsageReportEvent-from tests.integration.pipelex.temporal.test_data import PipeParallelTemporalTestData--_PARALLEL_MAIN_PIPE_REF = f"{PipeParallelTemporalTestData.DOMAIN}.{PipeParallelTemporalTestData.PIPE_CODE}"--[email protected](loop_scope="class")-class TestDryRunGraphInProcess:- def _assert_temporal_hub_preconditions(self) -> None:- """The hub defaults must be the REAL Temporal ones — that's the environment being guarded against."""- hub = get_pipelex_hub()- assert isinstance(hub.get_required_pipe_router(), TemporalPipeRouter), (- "Temporal suite precondition: the hub default router must be the real TemporalPipeRouter"- )- assert isinstance(hub.get_required_content_generator(), ContentGeneratorInWorkflow), (- "Temporal suite precondition: the hub default content generator must be ContentGeneratorInWorkflow"- )-- def _forbid_event_log_factory(self, mocker: MockerFixture) -> None:- factory_error = AssertionError("make_event_log must not be called: the in-process graph dry-run traces in memory only")- mocker.patch("pipelex.pipeline.pipeline_run_setup.make_event_log", side_effect=factory_error)- mocker.patch("pipelex.pipe_run.tracing_assembly.make_event_log", side_effect=factory_error)-- async def _run_in_process_graph_dry_run(self, library_id: str) -> GraphSpec:- mthds_content = Path(PipeParallelTemporalTestData.BUNDLE_FILE).read_text(encoding="utf-8")- # scoped_current_library restores any outer current-library binding on exit instead of- # clobbering it (acquire_library sets the same id internally; the scope owns the restore).- with scoped_current_library(library_id):- acquire_library(library_id=library_id, mthds_contents=[mthds_content])- try:- main_pipe = get_required_pipe(pipe_code=_PARALLEL_MAIN_PIPE_REF)- return await dry_run_pipe_in_process(pipe=main_pipe, library_id=library_id)- finally:- get_library_manager().teardown(library_id=library_id)-- async def test_graph_dry_run_zero_dispatch_in_memory(self, mocker: MockerFixture) -> None:- self._assert_temporal_hub_preconditions()- execute_workflow_spy = mocker.spy(WorkflowExecutor, "execute_workflow")- self._forbid_event_log_factory(mocker)- emit_spy = mocker.spy(InMemoryEventLog, "emit")-- graph_spec = await self._run_in_process_graph_dry_run(library_id="dry_run_graph_in_process_lib")-- # (a) correct, non-empty GraphSpec covering the whole controller topology (sequence,- # parallel fan-out, both branches, summary). Tracer-key alignment is pinned by the same- # assertion: a divergent emit/assemble key would have produced an empty graph.- traced_pipe_codes = {node.pipe_code for node in graph_spec.nodes if node.pipe_code}- expected_pipe_codes = {pipe_ref.split(".")[-1] for pipe_ref in PipeParallelTemporalTestData.EXPECTED_PIPE_REFS}- assert expected_pipe_codes <= traced_pipe_codes- assert graph_spec.edges-- # (b) zero Temporal dispatch during the whole dry-run.- execute_workflow_spy.assert_not_called()-- # (d) usage-event isolation (pre-flight decision 3): the dry leaves report synthetic- # zero-token jobs, but the run's trace context has emit_usage_events=False, so no- # UsageReportEvent ever reaches the (scoped, in-memory) transport — nothing leaks to any- # ambient registry, and the events die with the run.- emitted_events = [call.args[1] for call in emit_spy.call_args_list]- assert emitted_events, "the dry-run must emit graph events through the scoped log"- assert not [event for event in emitted_events if isinstance(event, UsageReportEvent)]-- async def test_leaf_level_mock_stays_in_process(self, mocker: MockerFixture) -> None:- """Leaf-level DRY (Part B, now the real path): DRY routes through the hub-resolved- generator and the cogt leaf mocks, so the scoped inline generator must keep the run- in-process — without ``scoped_content_generator`` the leaf would reach the hub's- ``ContentGeneratorInWorkflow`` and try to dispatch activities.- """- self._assert_temporal_hub_preconditions()- execute_workflow_spy = mocker.spy(WorkflowExecutor, "execute_workflow")- self._forbid_event_log_factory(mocker)- # Prove the run actually used the scoped inline generator (not the hub default).- inline_text_spy = mocker.spy(ContentGenerator, "make_llm_text")-- graph_spec = await self._run_in_process_graph_dry_run(library_id="dry_run_graph_leaf_mock_lib")-- assert graph_spec.nodes- assert inline_text_spy.call_count >= 1- execute_workflow_spy.assert_not_called()diff --git a/tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py b/tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.pydeleted file mode 100644index 80d88d162..000000000--- a/tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py+++ /dev/null@@ -1,403 +0,0 @@-"""Mode-1 isolation tests for the dry-run+validation Temporal activity (``act_dry_validate``).--The whole ``/validate`` job — validation sweep **+** graph-producing dry-run — runs as ONE-in-process activity dispatched via the one-step wrapper workflow ``WfDryValidate``. A real worker-runs the wrapper-workflow→activity against the in-process Temporal server, and the tests pin the-Phase-3 contract (TODOS.md):--- the per-pipe status map and the ``GraphSpec`` are correct, in one round-trip;-- ZERO nested dispatch: the workflow history contains exactly one scheduled activity- (``act_dry_validate``) and no child workflows (asserted at the Temporal-history level, per D6 —- not on the ``WorkflowExecutor`` wrapper);-- tracing stays in memory: no NDJSON partition appears, ``make_event_log`` is never called;-- the graph is best-effort (D5): an expected dry-run failure yields ``graph_spec=None`` with- validation still successful, while a non-Pipelex programming bug propagates and fails the run;-- a validation failure crosses back as a structured ``ErrorReport`` whose per-error- ``validation_errors`` data survives the crossing (D3/T3);-- concurrent invocations don't cross-contaminate the scoped overrides.-"""--import asyncio-import uuid-from pathlib import Path--import pytest-from pytest_mock import MockerFixture-from temporalio.api.enums.v1 import EventType-from temporalio.client import Client as TemporalClient-from temporalio.client import WorkflowFailureError-from temporalio.common import RetryPolicy-from temporalio.worker import UnsandboxedWorkflowRunner, Worker--from pipelex.base_exceptions import ValidationErrorCategory-from pipelex.config import get_config-from pipelex.hub import clear_current_library, get_current_library_id_or_none, get_library_manager-from pipelex.pipe_run.exceptions import DryRunError-from pipelex.pipeline.bundle_validator import DryRunStatus-from pipelex.pipeline.exceptions import PipeIOContractError-from pipelex.pipeline.execution_seams import acquire_library-from pipelex.pipeline.validate_bundle import validate_bundle-from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend-from pipelex.temporal.tprl.temporal_error import recover_error_report-from pipelex.temporal.tprl_pipe.act_dry_validate import DryValidateArg, DryValidateResult, act_dry_validate-from pipelex.temporal.tprl_pipe.wf_dry_validate import WfDryValidate-from tests.integration.pipelex.temporal.test_data import PipeParallelTemporalTestData--_PARALLEL_MAIN_PIPE_REF = f"{PipeParallelTemporalTestData.DOMAIN}.{PipeParallelTemporalTestData.PIPE_CODE}"--_ALPHA_MTHDS = """-domain = "dry_validate_alpha"-description = "Alpha bundle for act_dry_validate isolation tests"-main_pipe = "alpha_sequence"--[pipe.alpha_sequence]-type = "PipeSequence"-description = "Two-step alpha sequence"-inputs = { subject = "Text" }-output = "Text"-steps = [- { pipe = "alpha_first", result = "first_take" },- { pipe = "alpha_second", result = "second_take" },-]--[pipe.alpha_first]-type = "PipeLLM"-description = "First alpha step"-inputs = { subject = "Text" }-output = "Text"-prompt = "Describe $subject"--[pipe.alpha_second]-type = "PipeLLM"-description = "Second alpha step"-inputs = { first_take = "Text" }-output = "Text"-prompt = "Refine $first_take"-"""--_BETA_MTHDS = """-domain = "dry_validate_beta"-description = "Beta bundle for act_dry_validate isolation tests"-main_pipe = "beta_single"--[pipe.beta_single]-type = "PipeLLM"-description = "Single beta pipe"-inputs = { topic = "Text" }-output = "Text"-prompt = "Expand on $topic"-"""--_NO_MAIN_MTHDS = """-domain = "dry_validate_no_main"-description = "Bundle with no main_pipe, for the no-graph arm"--[pipe.lone_pipe]-type = "PipeLLM"-description = "Single pipe, no main_pipe declared"-inputs = { topic = "Text" }-output = "Text"-prompt = "Expand on $topic"-"""--_SIGNATURE_MTHDS = """-domain = "dry_validate_sig"-description = "Bundle carrying an unimplemented PipeSignature"-main_pipe = "sig_caller"--[pipe.sig_caller]-type = "PipeSequence"-description = "Sequence that reaches a signature"-inputs = { doc = "Text" }-output = "Text"-steps = [- { pipe = "unimplemented_sig", result = "sig_result" },-]--[pipe.unimplemented_sig]-type = "PipeSignature"-description = "A contract-only placeholder"-inputs = { doc = "Text" }-output = "Text"-"""--_INVALID_MTHDS = """-domain = "dry_validate_invalid"-description = "Bundle with a real validation error: an invalid main_pipe code fails blueprint validation"-main_pipe = "Not A Valid Pipe Code!"--[concept.InvalidDoc]-description = "A document for the invalid-bundle test"-"""--[email protected](loop_scope="class")-class TestDryValidateActivityInMemory:- def _forbid_event_log_factory(self, mocker: MockerFixture) -> None:- factory_error = AssertionError("make_event_log must not be called: act_dry_validate traces in memory only")- mocker.patch("pipelex.pipeline.pipeline_run_setup.make_event_log", side_effect=factory_error)- mocker.patch("pipelex.pipe_run.tracing_assembly.make_event_log", side_effect=factory_error)-- def _point_ndjson_tracing_at(self, mocker: MockerFixture, traces_dir: Path) -> None:- cfg = get_config().pipelex.tracing_config- mocker.patch.object(cfg, "is_enabled", True)- mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)- mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=str(traces_dir)))-- async def _execute(- self,- temporal_client: TemporalClient,- arg: DryValidateArg,- *,- workflow_id: str | None = None,- task_queue: str | None = None,- ) -> DryValidateResult:- workflow_id = workflow_id or f"wf_dry_validate_{uuid.uuid4().hex[:8]}"- task_queue = task_queue or f"q_dry_validate_{uuid.uuid4().hex[:8]}"- async with Worker(- temporal_client,- task_queue=task_queue,- workflows=[WfDryValidate],- activities=[act_dry_validate],- workflow_runner=UnsandboxedWorkflowRunner(),- ):- return await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfDryValidate.run,- arg=arg,- id=workflow_id,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )-- async def test_status_map_and_graph_in_one_round_trip_zero_nested_dispatch(- self, temporal_client: TemporalClient, tmp_path: Path, mocker: MockerFixture- ) -> None:- traces_dir = tmp_path / "traces"- traces_dir.mkdir()- self._point_ndjson_tracing_at(mocker, traces_dir)- self._forbid_event_log_factory(mocker)-- workflow_id = f"wf_dry_validate_{uuid.uuid4().hex[:8]}"- mthds_content = Path(PipeParallelTemporalTestData.BUNDLE_FILE).read_text(encoding="utf-8")- result = await self._execute(- temporal_client,- DryValidateArg(mthds_contents=[mthds_content], pipe_code=_PARALLEL_MAIN_PIPE_REF),- workflow_id=workflow_id,- )-- # Status map: every pipe in the bundle swept SUCCESS.- for pipe_ref in PipeParallelTemporalTestData.EXPECTED_PIPE_REFS:- assert result.dry_run_outputs[pipe_ref].status == DryRunStatus.SUCCESS-- # GraphSpec: non-empty, covering the whole controller topology.- assert result.graph_spec is not None- traced_pipe_codes = {node.pipe_code for node in result.graph_spec.nodes if node.pipe_code}- expected_pipe_codes = {pipe_ref.split(".")[-1] for pipe_ref in PipeParallelTemporalTestData.EXPECTED_PIPE_REFS}- assert expected_pipe_codes <= traced_pipe_codes-- # Zero nested dispatch, asserted on the Temporal history itself (D6): exactly one scheduled- # activity — act_dry_validate — and no child workflows.- history = await temporal_client.get_workflow_handle(workflow_id).fetch_history()- scheduled_activities = [- event.activity_task_scheduled_event_attributes.activity_type.name- for event in history.events- if event.event_type == EventType.EVENT_TYPE_ACTIVITY_TASK_SCHEDULED- ]- assert scheduled_activities == ["act_dry_validate"]- child_workflow_initiations = [- event for event in history.events if event.event_type == EventType.EVENT_TYPE_START_CHILD_WORKFLOW_EXECUTION_INITIATED- ]- assert child_workflow_initiations == []-- # In-memory tracing: no NDJSON partition appeared (make_event_log was also forbidden above).- assert list(traces_dir.iterdir()) == []-- async def test_main_pipe_auto_resolution_from_bundle(self, temporal_client: TemporalClient, mocker: MockerFixture) -> None:- """Without an explicit pipe_code, the graph arm targets the bundle's declared main_pipe."""- self._forbid_event_log_factory(mocker)-- result = await self._execute(temporal_client, DryValidateArg(mthds_contents=[_ALPHA_MTHDS]))-- assert result.dry_run_outputs["dry_validate_alpha.alpha_sequence"].status == DryRunStatus.SUCCESS- assert result.graph_spec is not None- traced_pipe_codes = {node.pipe_code for node in result.graph_spec.nodes if node.pipe_code}- assert {"alpha_sequence", "alpha_first", "alpha_second"} <= traced_pipe_codes-- # D10: the worker-computed wire fields cross the boundary — pipe_io_contracts keyed by- # namespaced pipe_ref with typed entries surviving (de)serialization, and a complete- # bundle reporting nothing pending.- assert result.pending_signatures == []- assert set(result.pipe_io_contracts) == {- "dry_validate_alpha.alpha_sequence",- "dry_validate_alpha.alpha_first",- "dry_validate_alpha.alpha_second",- }- sequence_contract = result.pipe_io_contracts["dry_validate_alpha.alpha_sequence"]- assert sequence_contract.inputs["subject"].concept_ref == "native.Text"- assert sequence_contract.output.concept_ref == "native.Text"-- async def test_graph_failure_is_best_effort(self, temporal_client: TemporalClient, mocker: MockerFixture) -> None:- """D5: an expected dry-run failure in the graph arm yields graph_spec=None, validation still OK."""- mocker.patch(- "pipelex.pipe_run.dry_run_in_process.dry_run_pipe_in_process",- side_effect=DryRunError("simulated graph dry-run failure"),- )-- result = await self._execute(temporal_client, DryValidateArg(mthds_contents=[_ALPHA_MTHDS]))-- assert result.graph_spec is None- assert result.dry_run_outputs["dry_validate_alpha.alpha_sequence"].status == DryRunStatus.SUCCESS-- async def test_graph_bug_propagates(self, temporal_client: TemporalClient, mocker: MockerFixture) -> None:- """D5 (bug-propagates arm): a non-Pipelex programming bug in the graph arm fails the run."""- mocker.patch(- "pipelex.pipe_run.dry_run_in_process.dry_run_pipe_in_process",- side_effect=KeyError("simulated programming bug"),- )-- with pytest.raises(WorkflowFailureError):- await self._execute(temporal_client, DryValidateArg(mthds_contents=[_ALPHA_MTHDS]))-- async def test_validation_failure_crosses_as_structured_error_report(self, temporal_client: TemporalClient) -> None:- """D3/T3: a genuine validation failure raises, and the structured per-error ``validation_errors``- data survives the ErrorReport crossing (it must not degrade to a plain message).-- Signatures are never an error (D-B), so the trigger is a real validation fault — an invalid- ``main_pipe`` code that fails blueprint validation. The sweep goes through ``validate_bundle``'s- cascade, so the crossing report carries the SAME identity the direct-mode route's body carries —- ``ValidateBundleError``, ``error_domain=input`` — with the per-error items in- ``validation_errors`` (the API ``InvalidReport`` is built from exactly these ErrorReport fields).- """- with pytest.raises(WorkflowFailureError) as exc_info:- await self._execute(temporal_client, DryValidateArg(mthds_contents=[_INVALID_MTHDS]))-- error_report = recover_error_report(exc_info.value)- assert error_report.error_type == "ValidateBundleError"- assert error_report.error_domain == "input"- # The structured per-error data survives the crossing (not a bare message).- assert error_report.validation_errors is not None- assert any(item.category == ValidationErrorCategory.BLUEPRINT_VALIDATION for item in error_report.validation_errors)-- async def test_signatures_allowed_in_lenient_mode(self, temporal_client: TemporalClient) -> None:- """The same signature bundle validates in lenient mode (allow_signatures=True), and the- runnability verdict's input crosses the boundary (D10/F5): the unsatisfied header is- reported in pending_signatures, with a structures entry like any concrete pipe.- """- result = await self._execute(temporal_client, DryValidateArg(mthds_contents=[_SIGNATURE_MTHDS], allow_signatures=True))-- assert result.dry_run_outputs["dry_validate_sig.sig_caller"].status == DryRunStatus.SUCCESS- assert result.graph_spec is not None- assert result.pending_signatures == ["dry_validate_sig.unimplemented_sig"]- assert "dry_validate_sig.unimplemented_sig" in result.pipe_io_contracts- assert result.pipe_io_contracts["dry_validate_sig.unimplemented_sig"].output.concept_ref == "native.Text"-- async def test_validation_failure_does_not_retry_activity(self, temporal_client: TemporalClient, mocker: MockerFixture) -> None:- """D-C5 regression: a deterministic validation failure must run the activity exactly once.- ValidateBundleError is non-retryable at the activity tier via a string match on the- ApplicationError type name — if that match drifts (class rename, boundary change), the- whole sweep silently re-runs and only this assertion catches it.- """- sweep_spy = mocker.patch(- "pipelex.temporal.tprl_pipe.act_dry_validate.validate_bundle",- wraps=validate_bundle,- )- with pytest.raises(WorkflowFailureError):- await self._execute(temporal_client, DryValidateArg(mthds_contents=[_INVALID_MTHDS]))- assert sweep_spy.call_count == 1-- async def test_pipe_io_contracts_failure_does_not_retry_activity(self, temporal_client: TemporalClient, mocker: MockerFixture) -> None:- """PipeIOContractError is non-retryable at the workflow tier, like ValidateBundleError —- a deterministic schema-render failure must run the activity exactly once. Pins the- string match on the ApplicationError type name in wf_dry_validate's retry policy: if- the class is renamed without updating the policy, the sweep silently re-runs and only- this assertion catches it.- """- io_contracts_mock = mocker.patch(- "pipelex.temporal.tprl_pipe.act_dry_validate.build_pipe_io_contracts",- side_effect=PipeIOContractError(message="simulated schema render failure"),- )- with pytest.raises(WorkflowFailureError):- await self._execute(temporal_client, DryValidateArg(mthds_contents=[_ALPHA_MTHDS]))- assert io_contracts_mock.call_count == 1-- async def test_no_main_pipe_and_no_pipe_code_yields_no_graph(self, temporal_client: TemporalClient) -> None:- """Without a declared main_pipe and without an explicit pipe_code, the graph arm is- skipped: validation succeeds with graph_spec=None.- """- result = await self._execute(temporal_client, DryValidateArg(mthds_contents=[_NO_MAIN_MTHDS]))-- assert result.dry_run_outputs["dry_validate_no_main.lone_pipe"].status == DryRunStatus.SUCCESS- assert result.graph_spec is None-- async def test_no_main_pipe_with_explicit_pipe_code_produces_graph(self, temporal_client: TemporalClient) -> None:- """An explicit pipe_code graphs a no-main bundle in the Temporal validate activity."""- result = await self._execute(- temporal_client,- DryValidateArg(mthds_contents=[_NO_MAIN_MTHDS], pipe_code="dry_validate_no_main.lone_pipe"),- )-- assert result.dry_run_outputs["dry_validate_no_main.lone_pipe"].status == DryRunStatus.SUCCESS- assert result.graph_spec is not None- traced_pipe_codes = {node.pipe_code for node in result.graph_spec.nodes if node.pipe_code}- assert traced_pipe_codes == {"lone_pipe"}-- async def test_explicit_pipe_code_overrides_main_pipe(self, temporal_client: TemporalClient) -> None:- """The Temporal graph arm targets the explicit pipe_code rather than the bundle main_pipe."""- result = await self._execute(- temporal_client,- DryValidateArg(mthds_contents=[_ALPHA_MTHDS], pipe_code="dry_validate_alpha.alpha_second"),- )-- assert result.dry_run_outputs["dry_validate_alpha.alpha_sequence"].status == DryRunStatus.SUCCESS- assert result.graph_spec is not None- traced_pipe_codes = {node.pipe_code for node in result.graph_spec.nodes if node.pipe_code}- assert traced_pipe_codes == {"alpha_second"}-- async def test_unknown_explicit_pipe_code_degrades_to_no_graph(self, temporal_client: TemporalClient) -> None:- """Graph-arm parity: an unknown explicit pipe_code is a graph-arm domain failure- (PipeNotFoundError, a PipelexError) and degrades to graph_spec=None with validation- still successful — the same answer the direct route gives, never a failed request.- """- result = await self._execute(temporal_client, DryValidateArg(mthds_contents=[_ALPHA_MTHDS], pipe_code="dry_validate_alpha.does_not_exist"))-- assert result.graph_spec is None- assert result.dry_run_outputs["dry_validate_alpha.alpha_sequence"].status == DryRunStatus.SUCCESS-- async def test_direct_call_restores_outer_current_library(self) -> None:- """The activity's finally restores the caller's outer current-library (the- prev != validated arm) after tearing the validated library down. Exercised by calling- the activity directly — through a worker, the activity runs in its own copied context- and the test task could not observe the ContextVar restore.- """- outer_library_id = "outer_lib_dry_validate"- acquire_library(library_id=outer_library_id, mthds_contents=[_BETA_MTHDS])- try:- result = await act_dry_validate(DryValidateArg(mthds_contents=[_ALPHA_MTHDS]))-- assert result.graph_spec is not None- assert get_current_library_id_or_none() == outer_library_id- finally:- get_library_manager().teardown(library_id=outer_library_id)- clear_current_library()-- async def test_concurrent_invocations_do_not_cross_contaminate(self, temporal_client: TemporalClient, mocker: MockerFixture) -> None:- """Two concurrent dispatches return distinct GraphSpecs with no shared/merged trace events."""- self._forbid_event_log_factory(mocker)-- result_alpha, result_beta = await asyncio.gather(- self._execute(temporal_client, DryValidateArg(mthds_contents=[_ALPHA_MTHDS])),- self._execute(temporal_client, DryValidateArg(mthds_contents=[_BETA_MTHDS])),- )-- assert result_alpha.graph_spec is not None- assert result_beta.graph_spec is not None- assert result_alpha.graph_spec.graph_id != result_beta.graph_spec.graph_id- alpha_codes = {node.pipe_code for node in result_alpha.graph_spec.nodes if node.pipe_code}- beta_codes = {node.pipe_code for node in result_beta.graph_spec.nodes if node.pipe_code}- assert "alpha_sequence" in alpha_codes- assert "beta_single" not in alpha_codes- assert "beta_single" in beta_codes- assert "alpha_sequence" not in beta_codes- assert "dry_validate_beta.beta_single" not in result_alpha.dry_run_outputs- assert "dry_validate_alpha.alpha_sequence" not in result_beta.dry_run_outputsdiff --git a/tests/integration/pipelex/temporal/test_payload_codec_pipeline.py b/tests/integration/pipelex/temporal/test_payload_codec_pipeline.pydeleted file mode 100644index 6cbfbeab3..000000000--- a/tests/integration/pipelex/temporal/test_payload_codec_pipeline.py+++ /dev/null@@ -1,238 +0,0 @@-"""Integration tests for StoragePayloadCodec with real Pipelex pipe execution.--Validates that the codec is transparent to real pipe workflows — WorkingMemory-survives encode→storage→decode across workflow/activity hops when payloads are-offloaded to external storage. Tests single-step, multi-step, and dynamic concept-pipelines through the full WfPipeRouter path.-"""--from __future__ import annotations--import uuid-from typing import TYPE_CHECKING, AsyncGenerator, ClassVar--import pytest-import pytest_asyncio-from temporalio.testing import WorkflowEnvironment--from pipelex.pipelex import Pipelex-from pipelex.system.runtime import IntegrationMode, runtime_manager-from pipelex.temporal.codec.storage_payload_codec import StoragePayloadCodec-from pipelex.temporal.config_temporal import BUILTIN_SEARCH_ATTRIBUTES-from pipelex.temporal.tasks import Tasks-from pipelex.temporal.temporal_data_converter import make_data_converter-from pipelex.temporal.temporal_hub import get_task_manager, temporal_hub-from pipelex.temporal.temporal_task_manager import TemporalTaskManager-from pipelex.temporal.tprl.namespace_check import ensure_required_search_attributes_registered-from pipelex.tools.storage.local_storage_provider import LocalStorageProvider-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import assert_stuff_names, assert_text_stuff_names, execute_workflow-from tests.integration.pipelex.temporal.test_data import (- LargePayloadTestData,- PayloadCodecPipelineTestData,-)--if TYPE_CHECKING:- from collections.abc import Generator- from pathlib import Path-- from temporalio.client import Client as TemporalClient-- from pipelex.pipe_run.pipe_job import PipeJob--# ----------------------------------------------------------------------------# Override autouse fixtures from parent conftest — this test module manages-# its own codec-enabled WorkflowEnvironment while still using the full-# TemporalTaskManager infrastructure for real pipe execution.-# -----------------------------------------------------------------------------[email protected](scope="module", autouse=True)-def reset_pipelex_config_fixture() -> Generator[None, None, None]:- """Initialize Pipelex for logging/config."""- Pipelex.make(- integration_mode=IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST,- )- yield- Pipelex.teardown_if_needed()--[email protected](scope="module", autouse=True)-def boot_temporal(reset_pipelex_config_fixture: None) -> Generator[None, None, None]: # noqa: ARG001- """Boot the Temporal layer with pipe routers and content generator."""- manager = TemporalTaskManager()- temporal_hub.set_task_manager(manager)- manager.complement_catalog(- extra_catalog=Tasks.TASK_PACKS,- extra_workflows=[],- extra_activities=[],- )- manager.setup()-- from pipelex.hub import get_pipelex_hub # noqa: PLC0415- from pipelex.temporal.tprl_content_generation.content_generator_in_workflow_factory import ( # noqa: PLC0415- ContentGeneratorInWorkflowFactory,- )- from pipelex.temporal.tprl_pipe.temporal_pipe_router import make_temporal_pipe_router # noqa: PLC0415-- pipelex_hub = get_pipelex_hub()- pipelex_hub.set_pipe_router(make_temporal_pipe_router())-- pipelex_hub.set_content_generator(ContentGeneratorInWorkflowFactory.make_content_generator_in_workflow())-- yield- manager.teardown()- temporal_hub.reset()-- from pipelex.hub import get_inference_manager, get_plugin_manager # noqa: PLC0415-- get_inference_manager().teardown()- get_plugin_manager().plugin_sdk_registry.teardown()---# ----------------------------------------------------------------------------# Codec-enabled WorkflowEnvironment-# ------------------------------------------------------------------------------class CodecEnvData:- SIZE_THRESHOLD: ClassVar[int] = PayloadCodecPipelineTestData.SIZE_THRESHOLD- STORAGE_PREFIX: ClassVar[str] = PayloadCodecPipelineTestData.STORAGE_PREFIX---@pytest_asyncio.fixture(scope="class") # pyright: ignore[reportUntypedFunctionDecorator, reportUnknownMemberType]-async def codec_env(tmp_path_factory: pytest.TempPathFactory) -> AsyncGenerator[tuple[WorkflowEnvironment, Path], None]:- """Create a WorkflowEnvironment with StoragePayloadCodec enabled."""- storage_root: Path = tmp_path_factory.mktemp("payload-codec-pipeline")- storage_provider = LocalStorageProvider(root_path=storage_root)- codec = StoragePayloadCodec(- storage_provider=storage_provider,- size_threshold=CodecEnvData.SIZE_THRESHOLD,- storage_prefix=CodecEnvData.STORAGE_PREFIX,- )- converter = make_data_converter(payload_codec=codec)- async with await WorkflowEnvironment.start_local(data_converter=converter) as env: # pyright: ignore[reportUnknownMemberType]- # The in-process server rejects workflows that set custom search attributes- # until they are registered. Pipelex sets five on every workflow start; register- # them here so this module's class-scoped server accepts dispatches.- await ensure_required_search_attributes_registered(- temporal_client=env.client,- namespace=env.client.namespace,- configured_attributes=BUILTIN_SEARCH_ATTRIBUTES,- )- yield env, storage_root---# ----------------------------------------------------------------------------# Pipe job fixtures-# -----------------------------------------------------------------------------[email protected](scope="class")-def native_pipe_job() -> Generator[PipeJob, None, None]:- """PipeJob for the native text sequence (2-step, no dynamic concepts)."""- yield from pipe_job_from_bundle(- bundle_file=PayloadCodecPipelineTestData.NATIVE_BUNDLE_FILE,- pipe_code=PayloadCodecPipelineTestData.NATIVE_PIPE_CODE,- )--[email protected](scope="class")-def dynamic_pipe_job() -> Generator[PipeJob, None, None]:- """PipeJob for the dynamic concept sequence (Greeting concept created at runtime)."""- yield from pipe_job_from_bundle(- bundle_file=PayloadCodecPipelineTestData.DYNAMIC_BUNDLE_FILE,- pipe_code=PayloadCodecPipelineTestData.DYNAMIC_PIPE_CODE,- )--[email protected](scope="class")-def large_payload_pipe_job() -> Generator[PipeJob, None, None]:- """PipeJob for the large payload stress test (3-step verbose sequence)."""- yield from pipe_job_from_bundle(- bundle_file=LargePayloadTestData.BUNDLE_FILE,- pipe_code=LargePayloadTestData.PIPE_CODE,- )---# ----------------------------------------------------------------------------# Helpers-# ------------------------------------------------------------------------------def _count_stored_files(storage_root: Path) -> int:- """Count files in the storage root (excluding directories)."""- return len([path for path in storage_root.rglob("*") if path.is_file()])---# ----------------------------------------------------------------------------# Tests-# -----------------------------------------------------------------------------[email protected](loop_scope="class")-class TestPayloadCodecPipeline:- """Real pipe execution through Temporal with StoragePayloadCodec enabled."""-- async def test_native_sequence_with_codec(- self,- codec_env: tuple[WorkflowEnvironment, Path],- native_pipe_job: PipeJob,- ) -> None:- """A 2-step PipeSequence with native Text concepts round-trips through the codec."""- env, storage_root = codec_env- temporal_client: TemporalClient = env.client- task_queue = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output = await execute_workflow(native_pipe_job, temporal_client, task_queue)-- assert_text_stuff_names(pipe_output, PayloadCodecPipelineTestData.NATIVE_EXPECTED_STUFF_NAMES)- assert _count_stored_files(storage_root) > 0, "Codec should have offloaded payloads to storage"-- async def test_dynamic_concept_with_codec(- self,- codec_env: tuple[WorkflowEnvironment, Path],- dynamic_pipe_job: PipeJob,- ) -> None:- """Dynamic concept (Greeting) + codec: the most demanding combination."""- env, storage_root = codec_env- temporal_client: TemporalClient = env.client- task_queue = str(uuid.uuid4())- files_before = _count_stored_files(storage_root)-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output = await execute_workflow(dynamic_pipe_job, temporal_client, task_queue)-- assert_stuff_names(pipe_output, PayloadCodecPipelineTestData.DYNAMIC_EXPECTED_STUFF_NAMES)- assert _count_stored_files(storage_root) > files_before, "Codec should have stored additional payloads for dynamic concept pipeline"-- async def test_large_payload_multi_step_with_codec(- self,- codec_env: tuple[WorkflowEnvironment, Path],- large_payload_pipe_job: PipeJob,- ) -> None:- """3-step pipeline accumulating verbose output — codec stress test."""- env, storage_root = codec_env- temporal_client: TemporalClient = env.client- task_queue = str(uuid.uuid4())- files_before = _count_stored_files(storage_root)-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output = await execute_workflow(large_payload_pipe_job, temporal_client, task_queue)-- assert_text_stuff_names(pipe_output, LargePayloadTestData.EXPECTED_STUFF_NAMES)- assert _count_stored_files(storage_root) > files_before, "Codec should have stored payloads for large multi-step pipeline"diff --git a/tests/integration/pipelex/temporal/test_payload_codec_roundtrip.py b/tests/integration/pipelex/temporal/test_payload_codec_roundtrip.pydeleted file mode 100644index 4fd845840..000000000--- a/tests/integration/pipelex/temporal/test_payload_codec_roundtrip.py+++ /dev/null@@ -1,223 +0,0 @@-"""Integration test for StoragePayloadCodec round-trip through Temporal.--Verifies that large payloads are transparently offloaded to external storage-and reconstructed on the other side of a workflow → activity → return chain.-"""--from __future__ import annotations--import uuid-from datetime import timedelta-from typing import TYPE_CHECKING, ClassVar--import pytest-from pydantic import BaseModel-from temporalio import activity, workflow-from temporalio.testing import WorkflowEnvironment-from temporalio.worker import UnsandboxedWorkflowRunner, Worker--from pipelex.base_exceptions import ErrorDomain, ErrorReport-from pipelex.cogt.inference.error_classification import ProviderErrorMetadata, UserAction, UserActionKind-from pipelex.cogt.inference.provider_name import ProviderName-from pipelex.pipelex import Pipelex-from pipelex.system.runtime import IntegrationMode, runtime_manager-from pipelex.temporal.codec.storage_payload_codec import StoragePayloadCodec-from pipelex.temporal.temporal_data_converter import make_data_converter-from pipelex.tools.storage.local_storage_provider import LocalStorageProvider--if TYPE_CHECKING:- from collections.abc import Generator- from pathlib import Path-- from temporalio.client import Client as TemporalClient--# ----------------------------------------------------------------------------# Override autouse fixtures from parent conftest — this test needs Pipelex-# initialized (for logging/config) but not the full TemporalTaskManager.-# -----------------------------------------------------------------------------[email protected](scope="module", autouse=True)-def reset_pipelex_config_fixture() -> Generator[None, None, None]:- """Initialize Pipelex for logging/config without TemporalTaskManager."""- Pipelex.make(- integration_mode=IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST,- )- yield- Pipelex.teardown_if_needed()--[email protected](scope="module", autouse=True)-def boot_temporal(reset_pipelex_config_fixture: None) -> None:- """No-op override: this test manages its own Worker and client."""---# ----------------------------------------------------------------------------# Test model, activity, and workflow-# ------------------------------------------------------------------------------class LargePayloadModel(BaseModel):- """Simple model carrying a large string payload."""-- label: str- data: str--[email protected](name="act_echo_large_payload")-async def act_echo_large_payload(payload: LargePayloadModel) -> LargePayloadModel: # noqa: RUF029- """Activity that returns its input unchanged — exercises decode then encode."""- return payload--[email protected](name="wf_echo_large_payload")-class WfEchoLargePayload:- """Workflow that forwards a large payload through an activity and returns it."""-- @workflow.run- async def run(self, payload: LargePayloadModel) -> LargePayloadModel:- return await workflow.execute_activity(- act_echo_large_payload,- payload,- start_to_close_timeout=timedelta(seconds=30),- )--[email protected](name="act_echo_error_report")-async def act_echo_error_report(report: ErrorReport) -> ErrorReport: # noqa: RUF029- """Activity that returns its ErrorReport input unchanged — exercises the BaseModel round-trip."""- return report--[email protected](name="wf_echo_error_report")-class WfEchoErrorReport:- """Workflow that forwards an ErrorReport through an activity and returns it."""-- @workflow.run- async def run(self, report: ErrorReport) -> ErrorReport:- return await workflow.execute_activity(- act_echo_error_report,- report,- start_to_close_timeout=timedelta(seconds=30),- )---# ----------------------------------------------------------------------------# Test data-# ------------------------------------------------------------------------------class TestPayloadCodecRoundTripTestData:- # 1 KB threshold — low enough that our test payload triggers offloading- SIZE_THRESHOLD: ClassVar[int] = 1024- STORAGE_PREFIX: ClassVar[str] = "test-payloads/"- # Generate a payload well above the threshold- LARGE_DATA: ClassVar[str] = "X" * 10_000- LABEL: ClassVar[str] = "roundtrip-test"---# ----------------------------------------------------------------------------# Tests-# -----------------------------------------------------------------------------[email protected](loop_scope="class")-class TestPayloadCodecRoundTrip:- """End-to-end test: client encode → storage → worker decode → activity → worker encode → storage → client decode."""-- async def test_large_payload_survives_temporal_roundtrip(self, tmp_path: Path) -> None:- """A payload larger than the codec threshold round-trips through Temporal identically."""- # Arrange: build a codec-enabled environment- storage_root = tmp_path / "payload-store"- storage_root.mkdir()- storage_provider = LocalStorageProvider(root_path=storage_root)- codec = StoragePayloadCodec(- storage_provider=storage_provider,- size_threshold=TestPayloadCodecRoundTripTestData.SIZE_THRESHOLD,- storage_prefix=TestPayloadCodecRoundTripTestData.STORAGE_PREFIX,- )- converter = make_data_converter(payload_codec=codec)-- input_payload = LargePayloadModel(- label=TestPayloadCodecRoundTripTestData.LABEL,- data=TestPayloadCodecRoundTripTestData.LARGE_DATA,- )-- task_queue = str(uuid.uuid4())-- async with await WorkflowEnvironment.start_local(data_converter=converter) as env: # pyright: ignore[reportUnknownMemberType]- temporal_client: TemporalClient = env.client- async with Worker(- temporal_client,- task_queue=task_queue,- workflows=[WfEchoLargePayload],- activities=[act_echo_large_payload],- workflow_runner=UnsandboxedWorkflowRunner(),- ):- result: LargePayloadModel = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- WfEchoLargePayload.run,- input_payload,- id="wf-payload-codec-roundtrip",- task_queue=task_queue,- )-- # Assert: output matches input exactly- assert isinstance(result, LargePayloadModel)- assert result.label == input_payload.label- assert result.data == input_payload.data- assert result == input_payload-- # Assert: storage was actually used (files were written)- stored_files = list(storage_root.rglob("*"))- stored_files = [file_path for file_path in stored_files if file_path.is_file()]- assert len(stored_files) > 0, "Codec should have offloaded payloads to storage"-- async def test_error_report_round_trips_through_activity(self) -> None:- """``ErrorReport`` round-trips through a workflow→activity hop, including nested ``UserAction`` / ``ProviderErrorMetadata``."""- original = ErrorReport(- error_type="LLMCompletionError",- message="provider returned 429",- title="AI inference failed",- type_uri="https://docs.pipelex.com/latest/errors/llm-completion-error/",- error_category="transient",- error_domain=ErrorDomain.RUNTIME,- retryable=True,- user_action=UserAction(kind=UserActionKind.WAIT_AND_RETRY, detail="Wait a moment and retry"),- model="gpt-4o-mini",- provider="openai",- provider_metadata=ProviderErrorMetadata(- provider=ProviderName.OPENAI,- sdk_exception_type="RateLimitError",- message="429 Too Many Requests",- status_code=429,- request_id="req_round_trip_001",- retry_after_seconds=2.5,- ),- )-- task_queue = str(uuid.uuid4())- converter = make_data_converter()-- async with await WorkflowEnvironment.start_local(data_converter=converter) as env: # pyright: ignore[reportUnknownMemberType]- temporal_client: TemporalClient = env.client- async with Worker(- temporal_client,- task_queue=task_queue,- workflows=[WfEchoErrorReport],- activities=[act_echo_error_report],- workflow_runner=UnsandboxedWorkflowRunner(),- ):- result: ErrorReport = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- WfEchoErrorReport.run,- original,- id=f"wf-error-report-roundtrip-{uuid.uuid4().hex[:8]}",- task_queue=task_queue,- )-- assert isinstance(result, ErrorReport)- assert result == original- assert result.user_action is not None- assert result.user_action.kind == UserActionKind.WAIT_AND_RETRY- assert result.provider_metadata is not None- assert result.provider_metadata.status_code == 429- assert result.provider_metadata.retry_after_seconds == 2.5diff --git a/tests/integration/pipelex/temporal/test_tasks.py b/tests/integration/pipelex/temporal/test_tasks.pydeleted file mode 100644index f4656dd8a..000000000--- a/tests/integration/pipelex/temporal/test_tasks.py+++ /dev/null@@ -1,30 +0,0 @@-import pytest--from pipelex import log-from pipelex.config import get_config-from pipelex.temporal.temporal_hub import get_task_manager---class TestTasks:- def test_default_scope_is_known(self):- worker_scopes = get_config().temporal.worker_scopes- assert worker_scopes.default_scope in worker_scopes.scopes-- def test_full_scope_packs_present(self):- packs = get_task_manager().task_packs()- log.verbose(packs, title="Task packs")- full_scope = get_config().temporal.worker_scopes.scopes["full"]- for required_task_pack in full_scope.required_tasks_packs:- assert required_task_pack in packs, f"Required task pack '{required_task_pack}' not found in packs"-- def test_router_and_runner_names_resolve(self):- workflows, activities = get_task_manager().workflows_and_activities_str()- log.verbose(workflows, title="Workflows")- log.verbose(activities, title="Activities")- for scope_name in ("router", "runner"):- scope = get_config().temporal.worker_scopes.scopes[scope_name]- for required_workflow in scope.required_workflows:- assert required_workflow in workflows, f"Required workflow '{required_workflow}' not found in workflows"- for required_activity in scope.required_activities:- assert required_activity in activities, f"Required activity '{required_activity}' not found in activities"diff --git a/tests/integration/pipelex/temporal/test_utils.py b/tests/integration/pipelex/temporal/test_utils.pydeleted file mode 100644index 6782101e4..000000000--- a/tests/integration/pipelex/temporal/test_utils.py+++ /dev/null@@ -1,3 +0,0 @@-from rich import print as rprint--__all__ = ["rprint"]diff --git a/tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py b/tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.pydeleted file mode 100644index 5d71a2d75..000000000--- a/tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py+++ /dev/null@@ -1,69 +0,0 @@-"""Mode-1 Temporal guard: the BundleValidator dry-run sweep must NOT dispatch to Temporal.--Deployment-faithful companion to the in-process sentinel regression test in-``tests/integration/pipelex/pipeline/test_bundle_validator.py``-(``test_standalone_batch_sweep_scopes_in_process_router``). Where the sentinel installs a *raising-stand-in* as the hub default, this test resolves the **real** ``TemporalPipeRouter`` as the hub-default — wired by this suite's ``boot_temporal`` fixture exactly as a Temporal-enabled API process-wires it in ``pipelex.pipelex``. It therefore proves the config -> hub-default *wiring* the sentinel-deliberately bypasses, on top of the same contract.--The bug it guards: a standalone ``PipeBatch`` swept directly fans out over a mock list and-dispatches each branch through ``get_pipe_router()``. Without the sweep scoping its own in-process-router, those branches resolve the hub default — the ``TemporalPipeRouter`` — and fire concurrent-top-level workflow dispatches (``WorkflowExecutor.execute_workflow``), which produced the production-HTTP 422.--The contract asserted: the whole sweep stays in-process, so the submitter-side top-level dispatch-seam (``WorkflowExecutor.execute_workflow``) is never reached. Spying on that seam is the-deterministic analogue of the Mode-2 "worker received no dispatch" check. No live Temporal server is-needed precisely because GREEN means no dispatch ever happens; if the fix regressed, the spy fires-and the test goes RED.-"""--from pathlib import Path--import pytest-from pytest_mock import MockerFixture--from pipelex.hub import clear_current_library, get_library_manager, get_pipelex_hub, get_required_pipe-from pipelex.pipeline.bundle_validator import BundleValidator-from pipelex.pipeline.execution_seams import acquire_library-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor-from pipelex.temporal.tprl_pipe.temporal_pipe_router import TemporalPipeRouter-from tests.integration.pipelex.temporal.test_data import PipeBatchTemporalTestData--# The standalone ``type = "PipeBatch"`` pipe in temporal_batch.mthds — swept directly it fans out,-# which is the exact shape that turned the leak fatal (concurrent same-id top-level dispatches).-_STANDALONE_BATCH_PIPE_REF = f"{PipeBatchTemporalTestData.DOMAIN}.batch_temporal_describe_topics"--[email protected](loop_scope="class")-class TestValidateSweepStaysInProcess:- async def test_standalone_batch_sweep_never_dispatches_to_temporal(self, mocker: MockerFixture) -> None:- # Precondition (the wiring this test exists to prove): under boot_temporal the hub default- # router IS the real TemporalPipeRouter — the same router a Temporal-enabled API process runs.- hub = get_pipelex_hub()- assert isinstance(hub.get_required_pipe_router(), TemporalPipeRouter), (- "Temporal suite precondition: the hub default router must be the real TemporalPipeRouter"- )-- # Spy on the submitter-side top-level dispatch seam. A standalone-batch sweep that leaked to- # Temporal would reach this exact method (TemporalPipeRouter._run_pipe_job top-level branch ->- # WorkflowExecutorFactory.create_executor(...).execute_workflow(...)). With the fix the sweep- # stays in-process and never touches it.- execute_workflow_spy = mocker.spy(WorkflowExecutor, "execute_workflow")-- library_manager = get_library_manager()- library_id = "validate_sweep_temporal_leak_lib"- mthds_content = Path(PipeBatchTemporalTestData.BUNDLE_FILE).read_text(encoding="utf-8")- acquire_library(library_id=library_id, mthds_contents=[mthds_content])- try:- batch_pipe = get_required_pipe(pipe_code=_STANDALONE_BATCH_PIPE_REF)- results = await BundleValidator().validate_pipes([batch_pipe], library_id=library_id)- assert results[_STANDALONE_BATCH_PIPE_REF].status.is_success- execute_workflow_spy.assert_not_called()- finally:- library_manager.teardown(library_id=library_id)- clear_current_library()diff --git a/tests/integration/pipelex/temporal/test_wf_dataclass_roundtrip.py b/tests/integration/pipelex/temporal/test_wf_dataclass_roundtrip.pydeleted file mode 100644index 6fb2bd187..000000000--- a/tests/integration/pipelex/temporal/test_wf_dataclass_roundtrip.py+++ /dev/null@@ -1,136 +0,0 @@-"""Integration test: a pydantic dataclass round-trips through Temporal (workflow -> activity -> return).--Phase 0 widened ``BaseModelPayloadConverter`` to route pydantic dataclasses through kajson with-type preservation. This exercises that path end-to-end through the in-process test server: the-dataclass is encoded by the client, decoded by the worker for the activity, re-encoded on return,-and decoded again by the client — every hop driven by the widened converter.-"""--from __future__ import annotations--import uuid-from datetime import timedelta-from typing import TYPE_CHECKING--import pytest-from pydantic import BaseModel-from pydantic.dataclasses import dataclass as pydantic_dataclass-from pydantic.dataclasses import is_pydantic_dataclass-from temporalio import activity, workflow-from temporalio.testing import WorkflowEnvironment-from temporalio.worker import UnsandboxedWorkflowRunner, Worker--from pipelex.pipelex import Pipelex-from pipelex.system.runtime import IntegrationMode, runtime_manager-from pipelex.temporal.temporal_data_converter import make_data_converter--if TYPE_CHECKING:- from collections.abc import Generator-- from temporalio.client import Client as TemporalClient---# ----------------------------------------------------------------------------# Override autouse fixtures from parent conftest — this test manages its own-# Worker and client, and only needs Pipelex initialized for logging/config.-# -----------------------------------------------------------------------------[email protected](scope="module", autouse=True)-def reset_pipelex_config_fixture() -> Generator[None, None, None]:- """Initialize Pipelex for logging/config without TemporalTaskManager."""- Pipelex.make(- integration_mode=IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST,- )- yield- Pipelex.teardown_if_needed()--[email protected](scope="module", autouse=True)-def boot_temporal(reset_pipelex_config_fixture: None) -> None:- """No-op override: this test manages its own Worker and client."""---# ----------------------------------------------------------------------------# Test model, dataclass, activity, and workflow-# ------------------------------------------------------------------------------class DataclassInner(BaseModel):- """Nested BaseModel field, to prove type preservation survives inside a dataclass."""-- label: str- count: int---@pydantic_dataclass-class DataclassPayload:- """Pydantic dataclass crossing the Temporal wire — the type Phase 0 made wire-legal."""-- name: str- delay: timedelta- inner: DataclassInner- note: str | None = None--[email protected](name="act_echo_dataclass_payload")-async def act_echo_dataclass_payload(payload: DataclassPayload) -> DataclassPayload: # noqa: RUF029- """Return the dataclass unchanged — exercises decode (arg) then encode (return)."""- return payload--[email protected](name="wf_echo_dataclass_payload")-class WfEchoDataclassPayload:- """Forward a pydantic dataclass through an activity and return it."""-- @workflow.run- async def run(self, payload: DataclassPayload) -> DataclassPayload:- return await workflow.execute_activity(- act_echo_dataclass_payload,- payload,- start_to_close_timeout=timedelta(seconds=30),- )---# ----------------------------------------------------------------------------# Test-# -----------------------------------------------------------------------------[email protected](loop_scope="class")-class TestPydanticDataclassTemporalRoundTrip:- async def test_pydantic_dataclass_survives_temporal_roundtrip(self) -> None:- """A pydantic dataclass round-trips through workflow -> activity -> return with type preserved."""- original = DataclassPayload(- name="widget",- delay=timedelta(seconds=45),- inner=DataclassInner(label="L", count=3),- note="hello",- )-- task_queue = str(uuid.uuid4())- converter = make_data_converter()-- async with await WorkflowEnvironment.start_local(data_converter=converter) as env: # pyright: ignore[reportUnknownMemberType]- temporal_client: TemporalClient = env.client- async with Worker(- temporal_client,- task_queue=task_queue,- workflows=[WfEchoDataclassPayload],- activities=[act_echo_dataclass_payload],- workflow_runner=UnsandboxedWorkflowRunner(),- ):- result: DataclassPayload = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- WfEchoDataclassPayload.run,- original,- id=f"wf-dataclass-roundtrip-{uuid.uuid4().hex[:8]}",- task_queue=task_queue,- )-- assert is_pydantic_dataclass(type(result))- assert isinstance(result.inner, DataclassInner)- assert result.inner.label == "L"- assert result.inner.count == 3- assert result.delay == timedelta(seconds=45)- assert result.note == "hello"- assert result == originaldiff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_costs_only_flush_nondeterminism.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_costs_only_flush_nondeterminism.pydeleted file mode 100644index 3bbe1eb3c..000000000--- a/tests/integration/pipelex/temporal/test_wf_pipe_router_costs_only_flush_nondeterminism.py+++ /dev/null@@ -1,211 +0,0 @@-"""Regression guard for the [TMPRL1100] costs-only flush nondeterminism in ``WfPipeRouter``.--The bug (finding H1 in ``wip/distributed-execution/workflow-nondeterminism-audit.md``): in-costs-only tracing mode (``emit_usage_events=True, emit_graph_events=False``) the workflow's-``BufferingEventLog`` is populated exclusively by co-located *activity* completions — the-workflow registers the buffer in the process-global ``ReportingManager`` via ``set_event_log``,-and the activity-side ``report_inference_job`` writes into it from the activity thread. The-workflow's ``finally`` block then gates the ``act_flush_trace_events`` schedule on-``if buffered_events:``, a value that is NOT a pure function of the workflow payload + history.-Live execution drains a non-empty buffer (the co-located activity ran and emitted), so history-records the flush schedule. On replay, activity results come from history without executing,-the freshly recreated buffer stays empty, and no flush command is emitted — the command streams-diverge and Temporal raises ``[TMPRL1100] Nondeterminism error``. In production this fires on a-routine sticky-cache eviction on the SAME worker, no config drift or fleet heterogeneity-required: the silent-hang class.--How the reproduction works:--1. A leaf PipeLLM job runs in LIVE mode with a costs-only trace context. ``act_llm_gen_text``- is substituted with a stub that synthesizes token usage and reports it through- ``report_inference_job`` — co-located with the workflow, so the emission takes the- ReportingManager fast path into the workflow's registered buffer, exactly what a real- co-located worker does. ``act_flush_trace_events`` is substituted with a no-op so the test- does not depend on a tracing backend; its *schedule* is what matters.-2. The workflow runs to completion on a normal sticky-cache worker, so the live continuation- drains a non-empty buffer and schedules the flush. The test asserts from the fetched- history that the flush was actually scheduled, so it cannot pass vacuously.-3. The recorded history is replayed through ``temporalio.worker.Replayer`` — the exact code- path a worker takes after a sticky-cache eviction. With the bug, the replay's buffer is- empty, the flush command is never emitted, and the replay raises a nondeterminism error.- The test asserts the replay succeeds: RED while the bug is present (TDD).--The ``graph-and-usage`` control arm runs the identical setup with graph events enabled: inline-tracer emissions re-fire deterministically on replay, the buffer is non-empty on both sides,-and the replay succeeds even with the bug present — proving that the costs-only buffer gating,-not the replay harness, is what breaks the command stream.--Fix shape (as landed): activity-side usage events never route through the workflow's-in-sandbox buffer (they take the per-process runner fallback), and the flush schedule is a-pure function of the payload. The payload-pure gate goes further than "always schedule": in-costs-only LIVE mode the buffer is deterministically empty (no graph events, no inline usage-emissions), so the flush activity is skipped entirely — the costs-only arm therefore asserts-the flush is ABSENT from history and that the recorded history still replays cleanly. The-``graph-and-usage`` arm keeps the original vacuity guard (flush present in history).-"""--import uuid-from collections.abc import Generator-from datetime import timedelta--import pytest-from temporalio import activity-from temporalio.client import Client as TemporalClient-from temporalio.worker import Replayer, UnsandboxedWorkflowRunner--from pipelex.cogt.content_generation.assignment_models import LLMAssignment-from pipelex.config import get_config-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.graph.graph_tracer_manager import GraphTracerManager-from pipelex.hub import get_report_delegate-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.temporal_data_converter import data_converter-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text-from pipelex.temporal.tprl_pipe.act_flush_trace_events import act_flush_trace_events-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.tracing.helpers import (- act_flush_noop,- inject_trace_context,- make_synthetic_usage_llm_job,- route_activities_to,- scheduled_activity_names,-)-from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData--_LEAF_PIPE_CODE = "step_one"-_FLUSH_ACTIVITY_NAME = "act_flush_trace_events"-_FAKE_INFERENCE_MODEL_NAME = "costs_only_replay_fake"-_FAKE_INFERENCE_MODEL_ID = "costs_only_replay_fake_id"-_FAKE_RESPONSE_TEXT = "costs-only replay guard fake response"--[email protected](name="act_llm_gen_text")-async def _act_llm_gen_text_reports_usage(llm_assignment: LLMAssignment) -> str: # noqa: RUF029- """Substitute for ``act_llm_gen_text`` that emits usage from a CO-LOCATED activity.-- The point is that ``report_inference_job`` runs on the activity thread in the same- process as the workflow's registered ``BufferingEventLog`` — exactly the co-located- emission that pre-H1 took the ReportingManager fast path into the workflow's buffer- (the cross-thread population that replay cannot reproduce). Post-H1 it must take the- per-process activity event log instead.- """- synthetic_job = make_synthetic_usage_llm_job(- llm_assignment=llm_assignment,- inference_model_name=_FAKE_INFERENCE_MODEL_NAME,- inference_model_id=_FAKE_INFERENCE_MODEL_ID,- )- get_report_delegate().report_inference_job(inference_job=synthetic_job)- return _FAKE_RESPONSE_TEXT--[email protected](scope="class")-def live_leaf_tracing_job() -> Generator[PipeJob, None, None]:- """LIVE-mode PipeJob for a leaf PipeLLM.-- LIVE pins the arm under test: costs-only LIVE is the shape whose flush schedule- used to be gated on worker-local buffer content and hung on replay. (Since the- unified dry run, DRY also dispatches ``act_llm_gen_text`` and emits activity-side,- but the regression being guarded was recorded against LIVE.)- """- yield from pipe_job_from_bundle(- bundle_file=SequenceTracingTestData.BUNDLE_FILE,- pipe_code=_LEAF_PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- )--[email protected](loop_scope="class")-class TestWfPipeRouterCostsOnlyFlushNondeterminism:- @pytest.fixture(autouse=True)- def clear_graph_tracer_manager(self) -> Generator[None, None, None]:- """Reset the process-lifetime tracer singleton so arms cannot leak tracer keys."""- yield- GraphTracerManager.clear_instance()-- @pytest.fixture(scope="class", autouse=True)- def redirect_traces_dir(self, tmp_path_factory: pytest.TempPathFactory) -> Generator[None, None, None]:- """Point the NDJSON traces_dir at a temp directory for this class.-- The stub activity's ``report_inference_job`` takes the per-process activity- event log path, which lazily builds an NDJSON writer in the configured- traces_dir — without this redirect the test writes stray trace files into the- repo's default ``.pipelex/traces``.- """- ndjson_config = get_config().pipelex.tracing_config.ndjson- original_dir = ndjson_config.traces_dir if ndjson_config else ""- if ndjson_config:- ndjson_config.traces_dir = str(tmp_path_factory.mktemp("traces"))- yield- if ndjson_config:- ndjson_config.traces_dir = original_dir-- @pytest.mark.parametrize("emit_graph_events", [True, False], ids=["graph-and-usage", "costs-only"])- async def test_recorded_history_replays_deterministically(- self,- temporal_client: TemporalClient,- live_leaf_tracing_job: PipeJob,- emit_graph_events: bool,- ) -> None:- """A history recorded with co-located activity usage emission must replay cleanly.-- With the bug, the ``costs-only`` arm raises a nondeterminism error from- ``Replayer.replay_workflow``: the flush schedule recorded in history is not- re-emitted because the replay's buffer is empty. Flush scheduling must be a- pure function of the workflow payload + history.- """- execution_run_id = f"costs_only_replay_guard_{uuid.uuid4().hex[:12]}"- execution_job = inject_trace_context(- live_leaf_tracing_job,- execution_run_id,- emit_graph_events=emit_graph_events,- emit_usage_events=True,- )- task_queue = f"q_costs_only_replay_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_costs_only_replay_{uuid.uuid4().hex[:8]}"-- with route_activities_to(task_queue, [act_llm_gen_text.__name__]):- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- substitute_activities={- act_llm_gen_text: _act_llm_gen_text_reports_usage,- act_flush_trace_events: act_flush_noop,- },- ):- workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=execution_job,- id=workflow_id,- task_queue=task_queue,- # Safety net: a regression that turns the divergence into a- # workflow-task retry loop must fail the test, not hang it.- execution_timeout=timedelta(seconds=60),- )- pipe_output = await workflow_handle.result()- assert isinstance(pipe_output, PipeOutput)- history = await workflow_handle.fetch_history()-- scheduled_names = scheduled_activity_names(history)- if emit_graph_events:- # Vacuity guard for the replay assertion: the graph arm must record the flush- # schedule (inline graph events populate the buffer deterministically).- assert _FLUSH_ACTIVITY_NAME in scheduled_names, f"history must record the flush schedule, got: {scheduled_names}"- else:- # Costs-only LIVE: the buffer is deterministically empty (activity-side usage- # emissions take the per-process runner fallback, never the workflow buffer),- # so the payload-pure gate skips the guaranteed-empty flush round-trip.- assert _FLUSH_ACTIVITY_NAME not in scheduled_names, f"costs-only LIVE must skip the empty flush schedule, got: {scheduled_names}"-- # Replay the recorded history — the exact code path of a post-eviction worker.- # With the bug, the costs-only arm raises here with a nondeterminism error.- replayer = Replayer(- workflows=[WfPipeRouter],- workflow_runner=UnsandboxedWorkflowRunner(),- data_converter=data_converter,- )- await replayer.replay_workflow(history)diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_cleanup_ordering.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_cleanup_ordering.pydeleted file mode 100644index ddb70e179..000000000--- a/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_cleanup_ordering.py+++ /dev/null@@ -1,156 +0,0 @@-"""Pin the eviction-safe ``finally`` ordering invariant of ``WfPipeRouter`` (H2 fix).--The invariant: in the workflow's ``finally`` block, ALL synchronous worker-local cleanup-(tracer close, report-delegate event-log context clear, per-run library teardown) runs-BEFORE the awaited ``act_flush_trace_events`` activity. The flush await is a suspension-point where eviction raises a ``BaseException`` (``_WorkflowBeingEvictedError``) that-escapes the ``except Exception`` around the flush and aborts the rest of the ``finally``.-If any cleanup sat after the await, an eviction there would leak state keyed by the-deterministic ``wf_{run_id}`` into the worker-local singletons.--Why eviction and not cancellation: workflow cancellation delivered at an activity await-is converted by the SDK into an ``ActivityError`` (an ``Exception``), which the-best-effort ``except`` around the flush swallows — the workflow then completes normally,-running whatever code follows. Cancellation therefore cannot abort the ``finally`` and-cannot exercise the invariant. Eviction is the real BaseException-at-suspension-point,-and worker shutdown evicts every cached run deterministically.--The reproduction:--1. ``act_flush_trace_events`` is substituted with a stub that signals entry and then- blocks, parking the workflow exactly at the flush await — by which point the ordering- invariant says all worker-local cleanup must already have run.-2. The test shuts the worker down (exits the worker context). Shutdown evicts the cached- run: the workflow coroutine is resumed at the flush await with the eviction- ``BaseException``, aborting the rest of the ``finally`` — exactly the H2 interruption.-3. The test then asserts the worker-local singletons are clean: no per-run library in- ``LibraryManager``, no tracer under the run id, no report-delegate event-log context.--A regression that moves any cleanup after the flush await turns these assertions red.-"""--import asyncio-import uuid-from collections.abc import Generator-from datetime import timedelta-from typing import ClassVar--import pytest-from temporalio import activity-from temporalio.client import Client as TemporalClient--from pipelex.graph.graph_tracer_manager import GraphTracerManager-from pipelex.hub import get_library_manager, get_report_delegate-from pipelex.libraries.exceptions import LibraryError-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.reporting.reporting_manager import ReportingManager-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.act_flush_trace_events import FlushTraceEventsArg, act_flush_trace_events-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.tracing.helpers import inject_trace_context, scheduled_activity_names-from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData--_LEAF_PIPE_CODE = "step_one"---class _FlushBlockSignals:- """Events shared between the test coroutine and the blocking flush stub."""-- entered: ClassVar[asyncio.Event | None] = None- release: ClassVar[asyncio.Event | None] = None--[email protected](name="act_flush_trace_events")-async def _act_flush_blocking(arg: FlushTraceEventsArg) -> None: # noqa: ARG001- """Flush substitute that parks the workflow at its flush await."""- assert _FlushBlockSignals.entered is not None- assert _FlushBlockSignals.release is not None- _FlushBlockSignals.entered.set()- await _FlushBlockSignals.release.wait()--[email protected](scope="class")-def leaf_tracing_job() -> Generator[PipeJob, None, None]:- """DRY-mode PipeJob for a leaf PipeLLM: the trace flush is the only activity."""- yield from pipe_job_from_bundle(- bundle_file=SequenceTracingTestData.BUNDLE_FILE,- pipe_code=_LEAF_PIPE_CODE,- )--[email protected](loop_scope="class")-class TestWfPipeRouterEvictionCleanupOrdering:- @pytest.fixture(autouse=True)- def clear_graph_tracer_manager(self) -> Generator[None, None, None]:- """Reset the process-lifetime tracer singleton so runs cannot leak tracer keys."""- yield- GraphTracerManager.clear_instance()-- async def test_eviction_at_flush_await_leaks_no_worker_local_state(- self,- temporal_client: TemporalClient,- leaf_tracing_job: PipeJob,- ) -> None:- """Evicting the run while it sits at the flush await must leave the worker-local- singletons clean: all synchronous cleanup precedes the only await.- """- execution_job = inject_trace_context(leaf_tracing_job, f"eviction_ordering_guard_{uuid.uuid4().hex[:12]}")- assert execution_job.library_crate is not None, "the job must carry a library crate so the workflow opens a per-run library"-- task_queue = f"q_eviction_ordering_{uuid.uuid4().hex[:8]}"- _FlushBlockSignals.entered = asyncio.Event()- _FlushBlockSignals.release = asyncio.Event()-- library_manager = get_library_manager()- tracer_manager = GraphTracerManager.get_or_create_instance()- report_delegate = get_report_delegate()- assert isinstance(report_delegate, ReportingManager)-- try:- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- substitute_activities={act_flush_trace_events: _act_flush_blocking},- ):- workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=execution_job,- id=f"wf_eviction_ordering_{uuid.uuid4().hex[:8]}",- task_queue=task_queue,- execution_timeout=timedelta(seconds=60),- )- run_id = workflow_handle.first_execution_run_id- assert run_id is not None-- # The workflow now sits at the flush await — the only await in the- # finally. Per the ordering invariant, all worker-local cleanup has- # already run by the time the flush activity starts.- await asyncio.wait_for(_FlushBlockSignals.entered.wait(), timeout=30)-- # Vacuity guard while the worker is still up: the flush schedule must be- # in history, proving the workflow is parked at that await.- history = await workflow_handle.fetch_history()- assert "act_flush_trace_events" in scheduled_activity_names(history)-- # Exiting the worker context shuts the worker down, which evicts the- # cached run: the coroutine resumes at the flush await with the eviction- # BaseException and the rest of the finally is aborted — the H2 shape.-- # Post-eviction: no worker-local state may have leaked. With the bug- # (cleanup ordered after the flush await), the per-run library, tracer,- # and event-log context would all still be registered here.- with pytest.raises(LibraryError):- library_manager.get_library(library_id=f"wf_{run_id}")- assert tracer_manager.get_tracer(run_id) is None, "tracer must be closed before the flush await"- contexts = report_delegate._event_log_contexts # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- assert run_id not in contexts, "event-log context must be cleared before the flush await"-- # Tidy up the abandoned execution (no worker is listening anymore).- await workflow_handle.terminate(reason="eviction-ordering guard done")- finally:- _FlushBlockSignals.release.set()- _FlushBlockSignals.entered = None- _FlushBlockSignals.release = Nonediff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_library_leak.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_library_leak.pydeleted file mode 100644index e90fc65c8..000000000--- a/tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_library_leak.py+++ /dev/null@@ -1,163 +0,0 @@-"""Regression guard for the eviction-time library leak that poisons same-worker replay in ``WfPipeRouter``.--The bug (finding H2 in ``wip/distributed-execution/workflow-nondeterminism-audit.md``): the-worker-local library teardown sits in the workflow's ``finally`` block AFTER the awaited-``act_flush_trace_events`` activity. An eviction-time interruption raised at that await-(``_WorkflowBeingEvictedError`` is a ``BaseException``) bypasses the ``except Exception``-around the flush and aborts the rest of the ``finally``: the per-workflow library and its-crate fingerprint leak in the worker-local ``LibraryManager`` singleton, keyed by the-*deterministic, replay-stable* ``wf_{run_id}``. When the same worker then replays that-run, ``open_library`` returns the stale library, ``set_class_registry`` installs a-FRESH registry seeded only from the global one, and ``load_from_crate`` is-fingerprint-skipped — so the crate's dynamic classes are never registered into the new-registry and inline hydration fails where history recorded success: command-stream-divergence, and with dynamic-class activity results on a single-worker queue, a persistent-silent hang.--How the reproduction works:--Forcing a real eviction exactly during the flush await is racy, so the test installs the-*post-eviction leaked state* directly: it starts the workflow with no worker listening-(so the run id is known but nothing executes yet), performs the same setup steps the-interrupted predecessor execution would have performed — open the library under-``wf_{run_id}``, attach a scoped class registry, ``load_from_crate`` — and deliberately-skips teardown, leaving the library and the crate fingerprint behind in the-``LibraryManager``. Only then does it open the worker, so the workflow executes against-that leaked state with a payload whose-``working_memory_raw`` carries a dynamic-concept stuff (the deferred-hydration path).-With the bug, the fingerprint skip leaves the dynamic class unregistered in the fresh-per-workflow registry and the workflow fails inline at hydration. The test asserts the-workflow completes and hydrates correctly: RED while the bug is present (TDD).--The job is built with an isolated class registry on purpose: in shared mode the dynamic-classes also live in the global registry, the workflow's fresh per-workflow registry-inherits them on seeding, and the leak is masked.--Fix shape: make the per-workflow library setup self-healing against a leaked predecessor-under the same deterministic id — scope the fingerprint dedup to the registry instance,-or force-teardown any pre-existing library before opening — and additionally make the-``finally`` cleanup eviction-safe.-"""--import uuid-from collections.abc import Generator-from contextlib import suppress-from datetime import timedelta--import pytest-from kajson.class_registry import ClassRegistry-from kajson.kajson_manager import KajsonManager-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.hub import get_current_library, get_library_manager, set_current_library-from pipelex.libraries.exceptions import LibraryError-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import make_prepared_greeting_job, rehydrate_pipe_output-from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData--[email protected](scope="class")-def isolated_hydration_job() -> Generator[PipeJob, None, None]:- """PipeJob from the dynamic-concept bundle with a library-scoped class registry.-- Isolation is required for the reproduction (not parametrized on purpose): with the- shared registry the workflow's per-workflow registry inherits the dynamic classes- from the global one and the fingerprint-skip bug cannot manifest.- """- yield from pipe_job_from_bundle(- bundle_file=DeferredHydrationTestData.BUNDLE_FILE,- pipe_code=DeferredHydrationTestData.PIPE_CODE,- isolated_registry=True,- )--[email protected](loop_scope="class")-class TestWfPipeRouterEvictionLibraryLeak:- async def test_workflow_recovers_from_leaked_predecessor_library(- self,- temporal_client: TemporalClient,- isolated_hydration_job: PipeJob,- ) -> None:- """A leaked predecessor library under the same ``wf_{run_id}`` must not poison the run.-- With the bug, ``load_from_crate`` is fingerprint-skipped against the leaked- library while ``set_class_registry`` already swapped in a fresh registry, so- deferred hydration of the dynamic Greeting stuff fails inline and the workflow- fails terminally instead of completing.- """- # Build an input WorkingMemory carrying a dynamic-concept stuff, then dehydrate it- # for Temporal transit so the workflow must hydrate it inline after crate loading.- prepared_job = make_prepared_greeting_job(isolated_hydration_job, stuff_code="leak_input")- assert prepared_job.working_memory_raw is not None, "prepare_for_temporal must populate working_memory_raw"- library_crate = prepared_job.library_crate- assert library_crate is not None, "the dynamic-concept job must carry a library crate"-- workflow_id = f"wf_eviction_leak_{uuid.uuid4().hex[:8]}"- task_queue = f"q_eviction_leak_{uuid.uuid4().hex[:8]}"-- # Start the workflow with NO worker listening on the task queue: the server- # assigns the run id immediately but nothing executes until the worker opens- # below. The per-run library id is derived from that run id (run-scoped keying:- # workflow ids are reused across retry/reset/resubmission, run ids are not).- workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=prepared_job,- id=workflow_id,- task_queue=task_queue,- # With the bug this fails the workflow: the inline hydration failure is- # converted to a terminal TemporalError by the workflow's fail-safe floor.- # The execution timeout is a safety net in case a regression turns the- # inline failure into a workflow-task retry loop.- execution_timeout=timedelta(seconds=60),- )- run_id = workflow_handle.first_execution_run_id- assert run_id is not None- leaked_library_id = f"wf_{run_id}"-- # Install the post-eviction leaked state: replay the interrupted predecessor's- # setup steps (mirroring WfPipeRouter.run) and deliberately skip teardown, so the- # library AND the crate fingerprint stay behind in the worker-local LibraryManager.- library_manager = get_library_manager()- fixture_library_id = get_current_library()- global_registry = KajsonManager.get_class_registry()- predecessor_registry = ClassRegistry()- predecessor_registry.register_classes_dict(global_registry.get_classes_dict())- _opened_library_id, leaked_library = library_manager.open_library(library_id=leaked_library_id)- leaked_library.set_class_registry(predecessor_registry)- set_current_library(library_id=leaked_library_id)- try:- library_manager.load_from_crate(library_id=leaked_library_id, crate=library_crate)- finally:- set_current_library(library_id=fixture_library_id)-- try:- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await workflow_handle.result()- finally:- # The workflow's own finally tears the library down on most paths; suppress- # covers early failures so the leak never escapes into other tests.- with suppress(LibraryError):- library_manager.teardown(library_id=leaked_library_id)-- assert isinstance(pipe_output, PipeOutput)- rehydrate_pipe_output(pipe_output)- working_memory = pipe_output.working_memory- assert working_memory is not None-- # The hydrated dynamic-concept input must have survived the full round-trip.- assert working_memory.is_stuff_exists("greeting_result"), "greeting_result should be in output WM"- greeting_out = working_memory.get_stuff("greeting_result")- assert isinstance(greeting_out.content, StructuredContent), (- f"Expected StructuredContent after hydration, got {type(greeting_out.content).__name__}"- )- assert working_memory.is_stuff_exists("summary_result"), "summary_result should be in output WM"diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_request_id_logging.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_request_id_logging.pydeleted file mode 100644index dd3bb6600..000000000--- a/tests/integration/pipelex/temporal/test_wf_pipe_router_request_id_logging.py+++ /dev/null@@ -1,137 +0,0 @@-"""Integration test: ``WfPipeRouter`` binds ``request_id`` from the dispatched-``PipeJob`` onto every workflow log record.--The per-invocation binding at ``WfPipeRouter.run`` (``WorkflowLog(request_id=-workflow_arg.job_metadata.request_id)``) can break — reverting to a module-level-singleton, wrong attribute path — without any other test failing. This regression-net pins the wiring end-to-end against a real Temporal worker.--Mirrors ``test_wf_pipe_run_request_id_logging.py`` but exercises ``WfPipeRouter``-directly: the LLM activity is mocked to fail so the workflow short-circuits past-real inference while still emitting the opening ``Workflow start`` debug record-that carries ``request_id``.-"""--import logging-import uuid-from collections.abc import Generator--import pytest-from pytest_mock import MockerFixture-from temporalio.client import Client as TemporalClient-from temporalio.client import WorkflowFailureError-from temporalio.common import RetryPolicy-from typing_extensions import override--from pipelex.config import get_config-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.error_handling.test_data import ErrorReportParityTestData-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle--_REQUEST_ID = "r-wf-pipe-router-logging-9c2b"-_ACT_LLM_GEN_TEXT_TARGET = "pipelex.temporal.tprl_content_generation.act_llm_generate.llm_gen_text"--[email protected](scope="class")-def request_id_router_job() -> Generator[PipeJob, None, None]:- """A PipeJob whose ``job_metadata`` carries a known ``request_id``.-- LIVE mode is required so the workflow actually dispatches the LLM activity —- DRY mode would short-circuit before the activity (and the mock) fires.- """- for pipe_job in pipe_job_from_bundle(- bundle_file=ErrorReportParityTestData.BUNDLE_FILE,- pipe_code=ErrorReportParityTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- ):- yield pipe_job.model_copy(- update={"job_metadata": pipe_job.job_metadata.model_copy(update={"request_id": _REQUEST_ID})},- )---class _RecordingHandler(logging.Handler):- """Captures every emitted ``LogRecord`` for post-run inspection."""-- def __init__(self) -> None:- super().__init__(level=logging.DEBUG)- self.records: list[logging.LogRecord] = []-- @override- def emit(self, record: logging.LogRecord) -> None:- self.records.append(record)--[email protected](loop_scope="class")-class TestWfPipeRouterRequestIdLogging:- @pytest.fixture- def temporal_enabled(self) -> Generator[None, None, None]:- """Enable ``temporal.is_enabled`` for the test — the worker's LLM activity- path checks this flag. Mirrors ``test_workflow_error_report_full_chain.py``.- """- config = get_config()- previous = config.temporal.is_enabled- config.temporal = config.temporal.model_copy(update={"is_enabled": True})- yield- config.temporal = config.temporal.model_copy(update={"is_enabled": previous})-- async def test_dispatched_request_id_rides_on_router_workflow_log_records(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- request_id_router_job: PipeJob,- ) -> None:- # Make the LLM activity fail fast and non-retryably so WfPipeRouter- # short-circuits past real inference. The opening ``Workflow start``- # debug record (which we want to assert on) fires before any pipe work.- mocker.patch(- _ACT_LLM_GEN_TEXT_TARGET,- new=mocker.AsyncMock(side_effect=ErrorReportParityTestData.make_failing_llm_error()),- )-- task_queue = f"q_wfrouter_reqid_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_router_reqid_{uuid.uuid4().hex[:8]}"-- # Capture records straight off the Temporal workflow logger: the- # unsandboxed worker runs workflow code in-process, so its records reach- # this process's ``temporalio.workflow`` logger. DEBUG level so the- # opening ``Workflow start`` record (where request_id first binds) is- # captured alongside the rest.- temporal_workflow_logger = logging.getLogger("temporalio.workflow")- handler = _RecordingHandler()- original_level = temporal_workflow_logger.level- try:- temporal_workflow_logger.addHandler(handler)- temporal_workflow_logger.setLevel(logging.DEBUG)- # Use the production task manager's worker (registers WfPipeRouter- # and the full activity catalog including ``act_llm_gen_text``) so- # the mocked LLM call actually fires inside the activity. The- # unsandboxed runner keeps workflow code in-process so the- # workflow logger records reach this handler.- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- with pytest.raises(WorkflowFailureError):- await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=request_id_router_job,- id=workflow_id,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )- finally:- temporal_workflow_logger.removeHandler(handler)- temporal_workflow_logger.setLevel(original_level)-- records_with_request_id = [record for record in handler.records if getattr(record, "request_id", None) == _REQUEST_ID]- assert records_with_request_id, (- f"expected WfPipeRouter workflow log records carrying request_id={_REQUEST_ID!r}; "- f"captured {len(handler.records)} record(s), request_ids seen: "- f"{sorted({str(getattr(record, 'request_id', None)) for record in handler.records})}"- )diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_run_scoped_state_keying.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_run_scoped_state_keying.pydeleted file mode 100644index eaf16f363..000000000--- a/tests/integration/pipelex/temporal/test_wf_pipe_router_run_scoped_state_keying.py+++ /dev/null@@ -1,177 +0,0 @@-"""Regression guard for the cross-run worker-local state collision in ``WfPipeRouter``.--The hazard (priority follow-up of the H2 fix, see-``wip/distributed-execution/nondeterminism-fix-review-follow-ups.md`` item 0): the-per-workflow library, the tracer key, and the report-delegate event-log context are-worker-local singleton entries. If they are keyed by ``workflow.info().workflow_id``,-two *runs* of the same workflow id collide: workflow ids are reused across workflow-level-``retry_policy`` attempts, Temporal reset, and resubmission of the same-``pipeline_run_id`` (``make_workflow_id`` is deterministic). The H2 fix made eviction-reliably execute the ``finally``'s synchronous cleanup — which is destructive cross-run:-a closed predecessor run's late eviction would tear down a live successor run's library,-pop its tracer, and clear its event-log context, failing the successor inline.--The structural fix: key all per-run worker-local state by ``workflow.info().run_id`` —-replay-stable, but unique per run — so cross-run collisions are impossible by-construction. This test pins that keying.--How it works: ``act_llm_gen_text`` is substituted with a stub that blocks on an-``asyncio.Event``, holding the workflow mid-execution with its worker-local state live.-The test then probes the worker-local singletons directly (same process): the library,-tracer, and report-delegate context must exist under run-id-derived keys and must NOT-exist under workflow-id-derived keys. RED while the state is keyed by workflow_id (TDD).-"""--import asyncio-import uuid-from collections.abc import Generator-from datetime import timedelta-from typing import ClassVar--import pytest-from temporalio import activity-from temporalio.client import Client as TemporalClient--from pipelex.cogt.content_generation.assignment_models import LLMAssignment-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.graph.graph_tracer_manager import GraphTracerManager-from pipelex.hub import get_library_manager, get_report_delegate-from pipelex.libraries.exceptions import LibraryError-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.reporting.reporting_manager import ReportingManager-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text-from pipelex.temporal.tprl_pipe.act_flush_trace_events import act_flush_trace_events-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.tracing.helpers import act_flush_noop, inject_trace_context, route_activities_to-from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData--_LEAF_PIPE_CODE = "step_one"-_FAKE_RESPONSE_TEXT = "run-scoped keying guard fake response"---class _BlockSignals:- """Events shared between the test coroutine and the blocking activity stub.-- Created fresh by the test on the running loop; class-level so the module-level- activity stub can reach them.- """-- entered: ClassVar[asyncio.Event | None] = None- release: ClassVar[asyncio.Event | None] = None--[email protected](name="act_llm_gen_text")-async def _act_llm_gen_text_blocking(llm_assignment: LLMAssignment) -> str: # noqa: ARG001- """Substitute for ``act_llm_gen_text`` that parks the workflow mid-execution.-- While this activity is blocked, the workflow's worker-local state (library,- tracer, report-delegate context) is live and probe-able from the test.- """- assert _BlockSignals.entered is not None- assert _BlockSignals.release is not None- _BlockSignals.entered.set()- await _BlockSignals.release.wait()- return _FAKE_RESPONSE_TEXT--[email protected](scope="class")-def live_leaf_keying_job() -> Generator[PipeJob, None, None]:- """LIVE-mode PipeJob for a leaf PipeLLM, carrying a library crate.-- LIVE mode is required so the workflow dispatches ``act_llm_gen_text`` as an- activity the test can block on (DRY mode short-circuits it inline).- """- yield from pipe_job_from_bundle(- bundle_file=SequenceTracingTestData.BUNDLE_FILE,- pipe_code=_LEAF_PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- )--[email protected](loop_scope="class")-class TestWfPipeRouterRunScopedStateKeying:- @pytest.fixture(autouse=True)- def clear_graph_tracer_manager(self) -> Generator[None, None, None]:- """Reset the process-lifetime tracer singleton so runs cannot leak tracer keys."""- yield- GraphTracerManager.clear_instance()-- async def test_worker_local_state_is_keyed_by_run_id_not_workflow_id(- self,- temporal_client: TemporalClient,- live_leaf_keying_job: PipeJob,- ) -> None:- """While the workflow is in flight, its worker-local state must live under- run-id-derived keys, and nothing may be registered under workflow-id keys.- """- execution_job = inject_trace_context(- live_leaf_keying_job,- f"run_scoped_keying_guard_{uuid.uuid4().hex[:12]}",- emit_graph_events=False,- emit_usage_events=True,- )- assert execution_job.library_crate is not None, "the job must carry a library crate so the workflow opens a per-run library"-- task_queue = f"q_run_scoped_keying_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_run_scoped_keying_{uuid.uuid4().hex[:8]}"- _BlockSignals.entered = asyncio.Event()- _BlockSignals.release = asyncio.Event()-- library_manager = get_library_manager()- tracer_manager = GraphTracerManager.get_or_create_instance()- report_delegate = get_report_delegate()- assert isinstance(report_delegate, ReportingManager)-- try:- with route_activities_to(task_queue, [act_llm_gen_text.__name__]):- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- substitute_activities={- act_llm_gen_text: _act_llm_gen_text_blocking,- act_flush_trace_events: act_flush_noop,- },- ):- workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=execution_job,- id=workflow_id,- task_queue=task_queue,- execution_timeout=timedelta(seconds=60),- )- run_id = workflow_handle.first_execution_run_id- assert run_id is not None-- await asyncio.wait_for(_BlockSignals.entered.wait(), timeout=30)-- # Library: present under wf_{run_id}, absent under wf_{workflow_id}.- library_manager.get_library(library_id=f"wf_{run_id}")- with pytest.raises(LibraryError):- library_manager.get_library(library_id=f"wf_{workflow_id}")-- # Tracer: registered under run_id, not workflow_id.- assert tracer_manager.get_tracer(run_id) is not None, "tracer must be keyed by run_id"- assert tracer_manager.get_tracer(workflow_id) is None, "tracer must not be keyed by workflow_id"-- # Report-delegate event-log context: keyed by run_id, not workflow_id.- contexts = report_delegate._event_log_contexts # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- assert run_id in contexts, "event-log context must be keyed by run_id"- assert workflow_id not in contexts, "event-log context must not be keyed by workflow_id"-- _BlockSignals.release.set()- pipe_output = await workflow_handle.result()- assert isinstance(pipe_output, PipeOutput)-- # After completion, the run-scoped state must be fully cleaned up.- with pytest.raises(LibraryError):- library_manager.get_library(library_id=f"wf_{run_id}")- assert tracer_manager.get_tracer(run_id) is None- assert run_id not in contexts- finally:- _BlockSignals.entered = None- _BlockSignals.release = Nonediff --git a/tests/integration/pipelex/temporal/test_wf_pipe_router_tracing_config_nondeterminism.py b/tests/integration/pipelex/temporal/test_wf_pipe_router_tracing_config_nondeterminism.pydeleted file mode 100644index f2258da7f..000000000--- a/tests/integration/pipelex/temporal/test_wf_pipe_router_tracing_config_nondeterminism.py+++ /dev/null@@ -1,159 +0,0 @@-"""Regression guard for the [TMPRL1100] tracing-config nondeterminism in ``WfPipeRouter``.--The bug: ``wf_pipe_router.py`` reads ``get_config().pipelex.tracing_config`` inside the-workflow body to decide whether to set up tracing — and therefore whether the ``finally``-block schedules ``act_flush_trace_events``. That config is worker-local mutable state, so-activity scheduling is not a pure function of the workflow history. In production this-bites when a workflow recorded by a tracing-enabled worker is replayed by a worker whose-tracing config differs (worker restart, rolling deploy, scale-in): the replay generates a-different command stream and Temporal raises a nondeterminism error.--The ``config-flips`` arm reproduces that scenario in a single process against the-in-process Temporal server — no distributed setup, no frozen history fixture — and-asserts the workflow nevertheless completes. It is RED while the bug is present (TDD):-the workflow fails with ``[TMPRL1100] Nondeterminism error`` instead of completing.--How the reproduction works:--1. Tracing is enabled and a ``trace_context`` is injected on a DRY-mode leaf PipeLLM job,- so the workflow's first task buffers trace events and ends by scheduling- ``act_flush_trace_events`` (the only activity a dry leaf workflow schedules).-2. The worker is built with ``max_cached_workflows=0``: the workflow is evicted from the- cache after every workflow task, so each subsequent task replays the full history from- scratch — the in-process equivalent of the workflow landing on a *different* worker.-3. ``act_flush_trace_events`` is substituted with a stub that flips- ``tracing_config.is_enabled`` to ``False`` — the in-process equivalent of "the worker- that picks up the next workflow task has a different tracing config". Activities run- in the worker process, so the flip lands between the workflow task that scheduled the- flush and the one that follows its completion: a deterministic, race-free injection- point.-4. The post-flush workflow task replays from the start under tracing-off config. With the- bug, the workflow skips tracing setup, schedules no flush, and tries to complete — but- the history records an ``act_flush_trace_events`` schedule at that position. Command- streams diverge; the SDK raises ``workflow.NondeterminismError``.--The ``config-stable`` control arm runs the identical setup with a no-op flush stub that-leaves the config alone: it passes even with the bug present, proving the config flip —-not the forced replays — is what breaks the command stream. Both arms assert from the-fetched history that ``act_flush_trace_events`` was actually scheduled, so neither can-pass vacuously.--``NondeterminismError`` is added to ``workflow_failure_exception_types`` so a divergence-fails the workflow terminally and surfaces in the test. Without it, Temporal's default-turns nondeterminism into a workflow-task failure that retries forever — which is exactly-the production symptom (a silently hung workflow), but useless in a test.-"""--import uuid-from collections.abc import Generator-from datetime import timedelta--import pytest-from temporalio import activity, workflow-from temporalio.client import Client as TemporalClient-from temporalio.worker import UnsandboxedWorkflowRunner, Worker--from pipelex.config import get_config-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.graph.graph_tracer_manager import GraphTracerManager-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.act_flush_trace_events import FlushTraceEventsArg, act_flush_trace_events-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.tracing.helpers import act_flush_noop, inject_trace_context, scheduled_activity_names-from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData--_LEAF_PIPE_CODE = "step_one"-_FLUSH_ACTIVITY_NAME = "act_flush_trace_events"--[email protected](name=_FLUSH_ACTIVITY_NAME)-async def _act_flush_flips_worker_tracing_config(arg: FlushTraceEventsArg) -> None: # noqa: RUF029, ARG001- """Substitute flush activity simulating a worker whose tracing config differs.-- Flipping the live config here (the activity runs in the worker process) lands the- change exactly between the workflow task that scheduled this activity and the replay- triggered by its completion — deterministically reproducing what a rolling deploy or- config drift does across real workers.- """- get_config().pipelex.tracing_config.is_enabled = False---def enable_tracing_restored() -> Generator[None, None, None]:- """Enable tracing for the test; restore it even though the stub activity flips it.-- Function-scoped on purpose: the ``config-flips`` arm leaves the flag False, and the- other arm must start from True again.- """- tracing_config = get_config().pipelex.tracing_config- original_enabled = tracing_config.is_enabled- tracing_config.is_enabled = True- yield- tracing_config.is_enabled = original_enabled- GraphTracerManager.clear_instance()--[email protected](scope="class")-def leaf_tracing_job() -> Generator[PipeJob, None, None]:- """DRY-mode PipeJob for a leaf PipeLLM: no inference activity, so the trace flush is- the only activity the workflow schedules.- """- yield from pipe_job_from_bundle(- bundle_file=SequenceTracingTestData.BUNDLE_FILE,- pipe_code=_LEAF_PIPE_CODE,- )--[email protected](loop_scope="class")-class TestWfPipeRouterTracingConfigNondeterminism:- @pytest.mark.parametrize("worker_config_flips", [False, True], ids=["config-stable", "config-flips"])- async def test_replay_survives_worker_tracing_config_flip(- self,- temporal_client: TemporalClient,- enable_tracing_restored: None, # noqa: ARG002 - enables tracing so the workflow schedules the flush- leaf_tracing_job: PipeJob,- worker_config_flips: bool,- ) -> None:- execution_job = inject_trace_context(leaf_tracing_job, f"tracing_replay_guard_{uuid.uuid4().hex[:12]}")- flush_substitute = _act_flush_flips_worker_tracing_config if worker_config_flips else act_flush_noop-- # Build the worker directly (not via make_worker) for the two knobs the- # reproduction needs: cache eviction after every workflow task, and terminal- # workflow failure on nondeterminism.- workflows, activities = get_task_manager().workflows_and_activities(- substitute_activities={act_flush_trace_events: flush_substitute},- )- async with Worker(- temporal_client,- task_queue=f"q_tracing_replay_guard_{uuid.uuid4().hex[:8]}",- workflows=workflows,- activities=activities,- workflow_runner=UnsandboxedWorkflowRunner(),- max_cached_workflows=0,- workflow_failure_exception_types=[workflow.NondeterminismError],- ) as worker:- workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=execution_job,- id=f"wf_tracing_replay_guard_{uuid.uuid4().hex[:8]}",- task_queue=worker.task_queue,- # Safety net: if nondeterminism ever stops being terminal here, the- # workflow-task retry loop would hang the test forever without this.- execution_timeout=timedelta(seconds=60),- )- # With the bug, the config-flips arm raises WorkflowFailureError here, caused- # by "[TMPRL1100] Nondeterminism error". Flush scheduling must be a pure- # function of the workflow payload/history, so the worker-side config flip- # must not prevent completion.- pipe_output = await workflow_handle.result()- assert isinstance(pipe_output, PipeOutput)-- # Guard against a vacuous pass: the recorded history must contain the flush- # schedule, otherwise there is nothing for a config-divergent replay to- # diverge on (and the control arm would prove nothing).- history = await workflow_handle.fetch_history()- scheduled_names = scheduled_activity_names(history)- assert _FLUSH_ACTIVITY_NAME in scheduled_names, f"history must record the flush schedule, got: {scheduled_names}"diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py b/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.pydeleted file mode 100644index b1392972e..000000000--- a/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py+++ /dev/null@@ -1,133 +0,0 @@-"""Failure-path integration test for ``WfPipeRun``.--Pins the failure-path invariant: when the child ``WfPipeRouter`` raises,-``WfPipeRun`` must still fire ``act_deliver`` with-``status=DeliveryStatus.FAILED`` and an empty ``pipe_output``, then re-raise-the original execution error.--``WfPipeRun`` catches ``ChildWorkflowError`` from-``workflow.execute_child_workflow(...)`` and wraps it as-``WorkflowExecutionError`` in-place; the Worker's-``workflow_failure_exception_types=[WorkflowExecutionError]`` registration is-what makes the re-raise end the workflow terminally instead of triggering-indefinite task-failure retry. This test exercises that path end-to-end-through a real Temporal worker.-"""--import uuid-from collections.abc import Generator--import pytest-from temporalio import activity, workflow-from temporalio.client import Client as TemporalClient-from temporalio.client import WorkflowFailureError-from temporalio.common import RetryPolicy-from temporalio.exceptions import ApplicationError-from temporalio.worker import UnsandboxedWorkflowRunner, Worker--from pipelex.base_exceptions import PipelexError-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg-from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.test_data import LibraryCrateTestData--[email protected](name="wf_pipe_router")-class WfPipeRouterFailingStub:- """Stand-in for ``WfPipeRouter`` that always fails.-- Registered with the same ``@workflow.defn`` name so Temporal's dispatcher- routes ``WfPipeRouter.run`` invocations to this stub on the test worker.- The real ``WfPipeRouter`` is NOT registered on this worker — that would- collide on the duplicated Temporal name.- """-- @workflow.run- async def run(self, _pipe_job: PipeJob) -> PipeOutput:- # ``non_retryable=True`` avoids the default Temporal retry loop —- # otherwise the child workflow retries indefinitely and the test- # hangs instead of surfacing the failure to ``WfPipeRun.run``.- msg = "simulated router failure"- raise ApplicationError(msg, non_retryable=True)--[email protected](scope="class")-def failure_path_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """A PipeJob built from the simplest available bundle.-- The pipe itself is never invoked — the failing-router stub short-circuits- before reaching any pipe logic. The PipeJob is required only because- ``WfPipeRun`` reads ``pipe_job.pipe`` to derive static summary/details- before dispatching the child workflow.- """- yield from pipe_job_from_bundle(- bundle_file=LibraryCrateTestData.BUNDLE_FILE,- pipe_code=LibraryCrateTestData.PIPE_CODE,- isolated_registry=is_class_registry_isolated,- )--[email protected](loop_scope="class")-class TestWfPipeRunFailurePath:- async def test_router_failure_fires_delivery_with_failed_status_and_reraises(- self,- temporal_client: TemporalClient,- failure_path_job: PipeJob,- ) -> None:- task_queue = f"q_wfrun_fail_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_run_fail_{uuid.uuid4().hex[:8]}"-- # Capture the DeliveryActivityArg for post-run assertion.- captured_delivery_args: list[DeliveryActivityArg] = []-- @activity.defn(name="act_deliver")- async def stub_act_deliver(arg: DeliveryActivityArg) -> None: # noqa: RUF029- captured_delivery_args.append(arg)-- pipe_run_arg = PipeRunArg(- pipe_job=failure_path_job,- delivery_assignment=DeliveryAssignment(),- ).prepare_for_temporal()-- # Construct the Worker directly: registering ``WfPipeRouterFailingStub``- # alongside the real ``WfPipeRouter`` (which is registered globally in- # conftest's boot_temporal) would collide on the ``@workflow.defn``- # name. Bypassing the global task manager keeps this worker isolated.- async with Worker(- temporal_client,- task_queue=task_queue,- workflows=[WfPipeRun, WfPipeRouterFailingStub],- activities=[stub_act_deliver],- workflow_runner=UnsandboxedWorkflowRunner(),- # Mirror the production ``make_worker`` config: register- # ``WorkflowExecutionError`` (the wrapped child failure) and the- # ``PipelexError`` fail-safe floor as workflow failure types so a- # workflow re-raising either ends the execution terminally instead- # of triggering indefinite task-failure retry.- workflow_failure_exception_types=[WorkflowExecutionError, PipelexError],- ):- with pytest.raises(WorkflowFailureError):- # ``maximum_attempts=1`` disables the workflow-level retry so the- # WorkflowExecutionError propagates as a single terminal failure.- # Without it, the default config retry policy retries WfPipeRun- # indefinitely and the test hangs.- await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRun.run,- arg=pipe_run_arg,- id=workflow_id,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )-- assert len(captured_delivery_args) == 1, f"act_deliver should fire exactly once on the failure path, got {len(captured_delivery_args)}"- delivery_arg = captured_delivery_args[0]- assert delivery_arg.status == DeliveryStatus.FAILED, (- f"delivery status should be FAILED when the child router fails, got {delivery_arg.status}"- )- assert delivery_arg.pipe_output is None, "pipe_output should be unset when the child router fails (model_copy update only runs on success)"diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.py b/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.pydeleted file mode 100644index c4f77f286..000000000--- a/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.py+++ /dev/null@@ -1,139 +0,0 @@-"""Integration test: a pipeline run dispatched with a ``request_id`` produces-Temporal workflow log records carrying that id.--Phase 2 of the API-readiness follow-ups wires ``job_metadata.request_id`` into-the worker-side ``WorkflowLog``: ``WfPipeRun`` builds its logger from-``workflow_arg.pipe_job.job_metadata.request_id``, so every record it emits-carries ``request_id`` in its ``extra`` dict. This test pins that end-to-end-wiring against a real Temporal worker.--Modeled on ``test_wf_pipe_run_failure_path.py`` — a failing ``WfPipeRouter``-stub short-circuits before any real pipe execution, so the test exercises-``WfPipeRun``'s logging without needing inference.-"""--import logging-import uuid-from collections.abc import Generator--import pytest-from temporalio import activity, workflow-from temporalio.client import Client as TemporalClient-from temporalio.client import WorkflowFailureError-from temporalio.common import RetryPolicy-from temporalio.exceptions import ApplicationError-from temporalio.worker import UnsandboxedWorkflowRunner, Worker-from typing_extensions import override--from pipelex.base_exceptions import PipelexError-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg-from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.test_data import LibraryCrateTestData--_REQUEST_ID = "r-wf-pipe-run-logging-7f3a"--[email protected](name="wf_pipe_router")-class WfPipeRouterFailingStub:- """Stand-in for ``WfPipeRouter`` that always fails.-- Registered under the real ``WfPipeRouter``'s Temporal name so ``WfPipeRun``'s- child-workflow dispatch routes here. Failing immediately short-circuits- ``WfPipeRun`` past any real pipe execution — the test only needs its logs.- """-- @workflow.run- async def run(self, _pipe_job: PipeJob) -> PipeOutput:- msg = "simulated router failure"- raise ApplicationError(msg, non_retryable=True)--[email protected](scope="class")-def request_id_job() -> Generator[PipeJob, None, None]:- """A PipeJob whose ``job_metadata`` carries a known ``request_id``."""- for pipe_job in pipe_job_from_bundle(- bundle_file=LibraryCrateTestData.BUNDLE_FILE,- pipe_code=LibraryCrateTestData.PIPE_CODE,- isolated_registry=False,- ):- yield pipe_job.model_copy(- update={"job_metadata": pipe_job.job_metadata.model_copy(update={"request_id": _REQUEST_ID})},- )---class _RecordingHandler(logging.Handler):- """Captures every emitted ``LogRecord`` for post-run inspection."""-- def __init__(self) -> None:- super().__init__(level=logging.DEBUG)- self.records: list[logging.LogRecord] = []-- @override- def emit(self, record: logging.LogRecord) -> None:- self.records.append(record)--[email protected](loop_scope="class")-class TestWfPipeRunRequestIdLogging:- async def test_dispatched_request_id_rides_on_workflow_log_records(- self,- temporal_client: TemporalClient,- request_id_job: PipeJob,- ) -> None:- task_queue = f"q_wfrun_reqid_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_run_reqid_{uuid.uuid4().hex[:8]}"-- @activity.defn(name="act_deliver")- async def stub_act_deliver(_arg: DeliveryActivityArg) -> None:- """No-op delivery — the failure path still fires it; its result is irrelevant here."""-- pipe_run_arg = PipeRunArg(- pipe_job=request_id_job,- delivery_assignment=DeliveryAssignment(),- ).prepare_for_temporal()-- # Capture records straight off the Temporal workflow logger: the- # unsandboxed worker runs workflow code in-process, so its records reach- # this process's ``temporalio.workflow`` logger. DEBUG level so the- # opening ``WfPipeRun start`` record is captured alongside the rest.- temporal_workflow_logger = logging.getLogger("temporalio.workflow")- handler = _RecordingHandler()- original_level = temporal_workflow_logger.level- try:- temporal_workflow_logger.addHandler(handler)- temporal_workflow_logger.setLevel(logging.DEBUG)- async with Worker(- temporal_client,- task_queue=task_queue,- workflows=[WfPipeRun, WfPipeRouterFailingStub],- activities=[stub_act_deliver],- workflow_runner=UnsandboxedWorkflowRunner(),- workflow_failure_exception_types=[WorkflowExecutionError, PipelexError],- ):- with pytest.raises(WorkflowFailureError):- # ``maximum_attempts=1`` keeps the failure terminal and single — see- # test_wf_pipe_run_failure_path.py for why the default retry would hang.- await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRun.run,- arg=pipe_run_arg,- id=workflow_id,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )- finally:- temporal_workflow_logger.removeHandler(handler)- temporal_workflow_logger.setLevel(original_level)-- records_with_request_id = [record for record in handler.records if getattr(record, "request_id", None) == _REQUEST_ID]- assert records_with_request_id, (- f"expected WfPipeRun workflow log records carrying request_id={_REQUEST_ID!r}; "- f"captured {len(handler.records)} record(s), request_ids seen: "- f"{sorted({str(getattr(record, 'request_id', None)) for record in handler.records})}"- )diff --git a/tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py b/tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.pydeleted file mode 100644index ad3f22bf3..000000000--- a/tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py+++ /dev/null@@ -1,238 +0,0 @@-"""Temporal arm of the local / Temporal ``ErrorReport`` parity pair.--Runs the ``native_text_sequence`` pipe through a real ``WfPipeRouter`` workflow-on the in-process Temporal server, submitted via ``WorkflowExecutor.execute_workflow``,-with the activity-side LLM call mocked to fail. This exercises the genuine-activity → workflow → submitter serialization round-trip — the unit tests-(``tests/unit/pipelex/temporal/test_workflow_caller_error_recovery.py``) only-fake the ``WorkflowFailureError`` with a synthetic ``ApplicationError``.--It asserts the recovered ``ErrorReport`` carries the full classification-(``error_category`` / ``retryable`` / ``model`` / ``provider`` / ``user_action``)-and the real failure message — not the generic ``"Failed to execute workflow ..."``.--The local baseline is the separate module-``tests/integration/pipelex/error_handling/test_error_report_local_full_chain.py``;-both arms assert the same ``ErrorReportParityTestData`` constants, so local /-Temporal parity holds by construction.-"""--import uuid-from collections.abc import Generator-from typing import TYPE_CHECKING, Any--import pytest-from pytest_mock import MockerFixture-from temporalio.client import Client as TemporalClient-from temporalio.common import RetryPolicy--from pipelex.config import get_config-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus, WebhookTarget-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun-from tests.integration.pipelex.error_handling.test_data import ErrorReportParityTestData-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--ACT_LLM_GEN_TEXT_TARGET = "pipelex.temporal.tprl_content_generation.act_llm_generate.llm_gen_text"--[email protected](loop_scope="class")-class TestWorkflowErrorReportFullChain:- """A failing pipe's structured ``ErrorReport`` survives the Temporal workflow → submitter boundary."""-- @pytest.fixture- def temporal_enabled(self) -> Generator[None, None, None]:- """Enable ``temporal.is_enabled`` for the test — ``WorkflowExecutor.temporal_client()``- raises when it is off. Mirrors the pattern in- ``tests/integration/pipelex/temporal/content_generation/conftest.py``.- """- config = get_config()- previous = config.temporal.is_enabled- config.temporal = config.temporal.model_copy(update={"is_enabled": True})- yield- config.temporal = config.temporal.model_copy(update={"is_enabled": previous})-- @pytest.fixture- def failing_pipe_job(self) -> Generator[PipeJob, None, None]:- """A PipeJob for the failing pipe, in LIVE mode so the workflow dispatches the LLM activity.-- DRY mode short-circuits the ``act_llm_gen_text`` dispatch (the dry content generator- reports inline inside the workflow), so the activity — and the mock — never fire.- """- yield from pipe_job_from_bundle(- bundle_file=ErrorReportParityTestData.BUNDLE_FILE,- pipe_code=ErrorReportParityTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- )-- async def test_error_report_survives_temporal_boundary(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- failing_pipe_job: PipeJob,- ) -> None:- """A failing pipe run through Temporal surfaces a fully classified ``WorkflowExecutionError``."""- # Mock the LLM call inside the real activity (in-process unsandboxed worker, same- # process). @convert_pipelex_errors packs the ErrorReport into ApplicationError.details.- mocker.patch(- ACT_LLM_GEN_TEXT_TARGET,- new=mocker.AsyncMock(side_effect=ErrorReportParityTestData.make_failing_llm_error()),- )-- task_queue = f"q_err_fullchain_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_err_fullchain_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(- workflow_class=WfPipeRouter,- workflow_arg=failing_pipe_job,- workflow_id=workflow_id,- )-- error = exc_info.value- # The structured report was recovered across the workflow → submitter boundary.- assert error.error_report is not None- report = error.to_error_report()-- # The real failure message survived — not the generic "Failed to execute workflow ...".- assert ErrorReportParityTestData.FAILURE_MESSAGE in report.message- assert "Failed to execute workflow" not in report.message-- # The full classification survived the Temporal serialization round-trip.- assert report.error_category == ErrorReportParityTestData.FAILURE_CATEGORY- assert report.retryable == ErrorReportParityTestData.EXPECTED_RETRYABLE- assert report.model == ErrorReportParityTestData.FAILURE_MODEL- assert report.provider == ErrorReportParityTestData.FAILURE_PROVIDER- assert report.user_action is not None- assert report.user_action.kind == ErrorReportParityTestData.EXPECTED_USER_ACTION_KIND-- async def test_wf_pipe_run_failure_threads_error_report_to_webhook_and_submitter(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- failing_pipe_job: PipeJob,- ) -> None:- """The outer ``WfPipeRun`` wrap surfaces the inner classification on BOTH the webhook payload and the submitter-side error."""- mocker.patch(- ACT_LLM_GEN_TEXT_TARGET,- new=mocker.AsyncMock(side_effect=ErrorReportParityTestData.make_failing_llm_error()),- )-- captured_webhook_payloads: list[dict[str, Any]] = []- mock_httpx_client = mocker.AsyncMock()- mock_response = mocker.MagicMock()- mock_response.raise_for_status = mocker.Mock()- mock_httpx_client.__aenter__ = mocker.AsyncMock(return_value=mock_httpx_client)- mock_httpx_client.__aexit__ = mocker.AsyncMock(return_value=False)-- def capture_post(url: str, **kwargs: Any) -> Any: # noqa: ARG001- captured_webhook_payloads.append(kwargs["json"])- return mock_response-- mock_httpx_client.post = mocker.AsyncMock(side_effect=capture_post)- mocker.patch("pipelex.pipe_run.delivery_executor.httpx.AsyncClient", return_value=mock_httpx_client)-- delivery_assignment = DeliveryAssignment(webhooks=[WebhookTarget(url="https://test.example.com/hook")])- pipe_run_arg = PipeRunArg(pipe_job=failing_pipe_job, delivery_assignment=delivery_assignment).prepare_for_temporal()-- task_queue = f"q_wfrun_err_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_wfrun_err_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeRunArg, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(- workflow_class=WfPipeRun,- workflow_arg=pipe_run_arg,- workflow_id=workflow_id,- )-- assert len(captured_webhook_payloads) == 1, (- f"webhook should fire exactly once on the FAILED delivery path, got {len(captured_webhook_payloads)}"- )- payload = captured_webhook_payloads[0]- assert payload["status"] == DeliveryStatus.FAILED- assert "error" in payload, "FAILED delivery webhook must include the structured error report"- error_dict = payload["error"]- assert error_dict["error_type"] == "LLMCompletionError"- assert error_dict["error_category"] == ErrorReportParityTestData.FAILURE_CATEGORY- assert error_dict["retryable"] == ErrorReportParityTestData.EXPECTED_RETRYABLE- assert error_dict["model"] == ErrorReportParityTestData.FAILURE_MODEL- assert error_dict["provider"] == ErrorReportParityTestData.FAILURE_PROVIDER- # LLMCompletionError auto-derives its title (no _declared_title) — CogtError's curated title doesn't leak through.- assert error_dict["title"] == "LLM completion"- assert error_dict["type_uri"] == "https://docs.pipelex.com/latest/errors/llm-completion-error/"- assert error_dict["user_action"]["kind"] == ErrorReportParityTestData.EXPECTED_USER_ACTION_KIND-- submitter_error = exc_info.value- assert submitter_error.error_report is not None- submitter_report = submitter_error.to_error_report()- assert submitter_report.error_category == ErrorReportParityTestData.FAILURE_CATEGORY- assert submitter_report.retryable == ErrorReportParityTestData.EXPECTED_RETRYABLE- assert submitter_report.model == ErrorReportParityTestData.FAILURE_MODEL- assert submitter_report.provider == ErrorReportParityTestData.FAILURE_PROVIDER- assert submitter_report.user_action is not None- assert submitter_report.user_action.kind == ErrorReportParityTestData.EXPECTED_USER_ACTION_KIND-- async def test_wf_pipe_run_failure_without_delivery_assignment_surfaces_classification(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- failing_pipe_job: PipeJob,- ) -> None:- """Submitter-side path still surfaces classification when no ``delivery_assignment`` is configured."""- mocker.patch(- ACT_LLM_GEN_TEXT_TARGET,- new=mocker.AsyncMock(side_effect=ErrorReportParityTestData.make_failing_llm_error()),- )-- pipe_run_arg = PipeRunArg(pipe_job=failing_pipe_job, delivery_assignment=None).prepare_for_temporal()-- task_queue = f"q_wfrun_err_nd_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_wfrun_err_nd_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeRunArg, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(- workflow_class=WfPipeRun,- workflow_arg=pipe_run_arg,- workflow_id=workflow_id,- )-- submitter_error = exc_info.value- assert submitter_error.error_report is not None- submitter_report = submitter_error.to_error_report()- assert submitter_report.error_category == ErrorReportParityTestData.FAILURE_CATEGORY- assert submitter_report.retryable == ErrorReportParityTestData.EXPECTED_RETRYABLE- assert submitter_report.model == ErrorReportParityTestData.FAILURE_MODEL- assert submitter_report.provider == ErrorReportParityTestData.FAILURE_PROVIDER- assert submitter_report.user_action is not None- assert submitter_report.user_action.kind == ErrorReportParityTestData.EXPECTED_USER_ACTION_KINDdiff --git a/tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py b/tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.pydeleted file mode 100644index b0836fb0d..000000000--- a/tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py+++ /dev/null@@ -1,319 +0,0 @@-"""Regression tests for the Temporal workflow-level fail-safe floor.--The error-handling bridge converts every error that crosses an *activity*-boundary into a terminal, classified Temporal failure. But a pipelex domain-error raised **inline in workflow code** — never dispatched through an activity —-is neither an ``ActivityError`` nor an ``ApplicationError``. Without a backstop-Temporal treats it as a *workflow-task* failure and retries the task-indefinitely: a silent, resource-burning hang that only surfaces (as the wrong,-generic timeout error) after the workflow execution timeout.--These tests pin the floor that closes that hole:--- ``WfPipeRouter`` converts an escaping ``PipelexError`` to a terminal- ``TemporalError`` carrying the structured ``ErrorReport`` (rich path).-- ``WfPipeRun`` routes an inline ``PipelexError`` through its deferred-delivery- path so the FAILED webhook still fires, then re-raises terminally.-- The worker's ``workflow_failure_exception_types`` includes ``PipelexError`` so- even an *uncaught* domain error fails the workflow terminally (the floor) with- a synthesized report, rather than hanging.--Every case sets a short ``workflow_execution_timeout`` so a regression that-reopens the hole fails fast (and on the wrong, timeout-shaped classification)-instead of hanging the suite for the full default budget.-"""--import uuid-from collections.abc import Generator-from datetime import timedelta-from typing import Any--import pytest-from pytest_mock import MockerFixture-from temporalio import workflow-from temporalio.client import Client as TemporalClient-from temporalio.common import RetryPolicy--from pipelex.config import get_config-from pipelex.core.pipes.pipe_abstract import PipeAbstract-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus, WebhookTarget-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg-from pipelex.temporal.exceptions import WorkflowExecutionError, WorkflowInputError-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl.temporal_error import TemporalError-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun-from tests.integration.pipelex.error_handling.test_data import ErrorReportParityTestData-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle--# Bounds the regression hang: with the fail-safe in place every case fails-# near-instantly; only a regression that reopens the workflow-task-retry hole-# would consume this budget before surfacing a (wrong, timeout-shaped) error.-_FAILSAFE_EXECUTION_TIMEOUT = timedelta(seconds=30)--_RAW_INLINE_MESSAGE = "raw inline pipelex error, never caught"--[email protected](name="wf_raw_pipelex_error_stub")-class WfRawPipelexErrorStub:- """Workflow that raises a raw, uncaught ``PipelexError`` inline.-- Exercises the worker-level floor (``workflow_failure_exception_types``- contains ``PipelexError``) — no workflow-level catch-all is involved, so the- only thing that can make this terminal instead of a retrying task failure is- the registration on the production ``make_worker``.- """-- @workflow.run- async def run(self, workflow_arg: PipeJob) -> PipeOutput: # noqa: ARG002 - signature pinned by the WorkflowClass protocol; raises immediately- raise WorkflowInputError(_RAW_INLINE_MESSAGE)--[email protected](loop_scope="class")-class TestWorkflowInlineErrorFailsafe:- """A pipelex error escaping workflow code fails loud and classified, never hangs."""-- @pytest.fixture- def temporal_enabled(self) -> Generator[None, None, None]:- """Enable ``temporal.is_enabled`` so ``WorkflowExecutor.temporal_client()`` connects."""- config = get_config()- previous = config.temporal.is_enabled- config.temporal = config.temporal.model_copy(update={"is_enabled": True})- yield- config.temporal = config.temporal.model_copy(update={"is_enabled": previous})-- @pytest.fixture- def failing_pipe_job(self) -> Generator[PipeJob, None, None]:- """A PipeJob in LIVE mode (the workflow runs the pipe instead of short-circuiting in dry mode)."""- yield from pipe_job_from_bundle(- bundle_file=ErrorReportParityTestData.BUNDLE_FILE,- pipe_code=ErrorReportParityTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- )-- async def test_router_inline_pipelex_error_surfaces_classified_terminally(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- failing_pipe_job: PipeJob,- ) -> None:- """A ``PipelexError`` raised inline by the pipe (no activity) surfaces fully classified, not as a hang."""- # Make the pipe raise *inline* in the workflow event loop — the escape path the activity- # bridge never sees. ``run_pipe`` is @final on PipeAbstract, so patching the base covers- # whatever concrete pipe the bundle resolves to.- mocker.patch.object(- PipeAbstract,- "run_pipe",- new=mocker.AsyncMock(side_effect=ErrorReportParityTestData.make_failing_llm_error()),- )-- task_queue = f"q_inline_router_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_inline_router_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- workflow_execution_timeout=_FAILSAFE_EXECUTION_TIMEOUT,- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(- workflow_class=WfPipeRouter,- workflow_arg=failing_pipe_job,- workflow_id=workflow_id,- )-- error = exc_info.value- assert error.error_report is not None- report = error.to_error_report()- # The rich classification of the *inline* error survived — proving it went through the- # fail-safe conversion, not the regression timeout path (which would synthesize an- # UnrecoverableWorkflowFailureError instead).- assert report.error_type == "LLMCompletionError"- assert ErrorReportParityTestData.FAILURE_MESSAGE in report.message- assert report.error_category == ErrorReportParityTestData.FAILURE_CATEGORY- assert report.retryable == ErrorReportParityTestData.EXPECTED_RETRYABLE- assert report.model == ErrorReportParityTestData.FAILURE_MODEL- assert report.provider == ErrorReportParityTestData.FAILURE_PROVIDER- assert report.user_action is not None- assert report.user_action.kind == ErrorReportParityTestData.EXPECTED_USER_ACTION_KIND-- async def test_router_already_terminal_error_propagates_untouched_preserving_leaf_classification(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- failing_pipe_job: PipeJob,- ) -> None:- """An escaping error that ALREADY carries a Temporal failure propagates untouched.-- Pins the ``_carries_temporal_failure`` True branch — the reachable case where a controller- pipe's sub-pipe fails as a child workflow and ``TemporalPipeRouter`` wraps the- ``ChildWorkflowError`` as ``WorkflowExecutionError``, which then escapes the parent's- ``pipe.run_pipe``. The rich leaf report rides an ``ApplicationError`` (a Temporal- ``FailureError``) in the ``__cause__`` chain and is recoverable only by- ``recover_error_report`` at the submitter. A regression that negated or deleted the guard- would convert it via ``from_message_exception`` instead, flattening ``error_type`` to- ``WorkflowExecutionError`` and dropping model / provider / category — so these assertions- fail loudly on exactly that regression (the convert-branch sibling test above would still- pass, which is why this branch needs its own coverage).- """- # The structured leaf report the nested failure carries, built like the activity bridge:- # an LLMCompletionError -> ErrorReport -> dict, packed into ApplicationError.details.- leaf_report_dict = ErrorReportParityTestData.make_failing_llm_error().to_error_report().to_dict()- # The already-terminal carrier: a TemporalError IS an ApplicationError (a Temporal- # FailureError) holding the leaf report in details — what sits in the chain after a child- # WfPipeRouter fails. Wrapped via ``from`` as WorkflowExecutionError, mirroring- # TemporalPipeRouter's ``raise WorkflowExecutionError(...) from ChildWorkflowError``.- already_terminal = TemporalError(- message=ErrorReportParityTestData.FAILURE_MESSAGE,- error_type="LLMCompletionError",- non_retryable=True,- error_report=leaf_report_dict,- )- nested_failure = WorkflowExecutionError("nested sub-pipe failed")- nested_failure.__cause__ = already_terminal- mocker.patch.object(- PipeAbstract,- "run_pipe",- new=mocker.AsyncMock(side_effect=nested_failure),- )-- task_queue = f"q_inline_passthrough_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_inline_passthrough_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- workflow_execution_timeout=_FAILSAFE_EXECUTION_TIMEOUT,- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(- workflow_class=WfPipeRouter,- workflow_arg=failing_pipe_job,- workflow_id=workflow_id,- )-- report = exc_info.value.to_error_report()- # The LEAF classification survived end-to-end: error_type is the leaf's, NOT- # "WorkflowExecutionError" — proof the guard propagated the error untouched and let the- # submitter recover the original report, rather than re-wrapping it into a generic one.- assert report.error_type == "LLMCompletionError"- assert ErrorReportParityTestData.FAILURE_MESSAGE in report.message- assert report.error_category == ErrorReportParityTestData.FAILURE_CATEGORY- assert report.retryable == ErrorReportParityTestData.EXPECTED_RETRYABLE- assert report.model == ErrorReportParityTestData.FAILURE_MODEL- assert report.provider == ErrorReportParityTestData.FAILURE_PROVIDER-- async def test_wf_pipe_run_inline_error_fires_failed_webhook_and_surfaces_classification(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- failing_pipe_job: PipeJob,- ) -> None:- """An inline ``PipelexError`` in the parent workflow still fires the FAILED webhook and surfaces terminally."""- # Force an inline domain error inside WfPipeRun itself — build_search_attributes is- # evaluated as an argument to execute_child_workflow, before any child/activity exists.- inline_error = WorkflowInputError("simulated inline search-attribute failure")- mocker.patch(- "pipelex.temporal.tprl_pipe.wf_pipe_run.build_search_attributes",- side_effect=inline_error,- )-- captured_webhook_payloads: list[dict[str, Any]] = []- mock_httpx_client = mocker.AsyncMock()- mock_response = mocker.MagicMock()- mock_response.raise_for_status = mocker.Mock()- mock_httpx_client.__aenter__ = mocker.AsyncMock(return_value=mock_httpx_client)- mock_httpx_client.__aexit__ = mocker.AsyncMock(return_value=False)-- def capture_post(url: str, **kwargs: Any) -> Any: # noqa: ARG001 - url is positional, unused- captured_webhook_payloads.append(kwargs["json"])- return mock_response-- mock_httpx_client.post = mocker.AsyncMock(side_effect=capture_post)- mocker.patch("pipelex.pipe_run.delivery_executor.httpx.AsyncClient", return_value=mock_httpx_client)-- delivery_assignment = DeliveryAssignment(webhooks=[WebhookTarget(url="https://test.example.com/hook")])- pipe_run_arg = PipeRunArg(pipe_job=failing_pipe_job, delivery_assignment=delivery_assignment).prepare_for_temporal()-- task_queue = f"q_inline_run_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_inline_run_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeRunArg, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- workflow_execution_timeout=_FAILSAFE_EXECUTION_TIMEOUT,- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(- workflow_class=WfPipeRun,- workflow_arg=pipe_run_arg,- workflow_id=workflow_id,- )-- # The FAILED webhook fired even though the failure never reached pipe execution — a terminal- # failure must always notify the receiver, including on an inline workflow-setup error.- assert len(captured_webhook_payloads) == 1, f"expected exactly one FAILED webhook, got {len(captured_webhook_payloads)}"- payload = captured_webhook_payloads[0]- assert payload["status"] == DeliveryStatus.FAILED- assert "error" in payload, "FAILED delivery webhook must include the structured error report"- assert payload["error"]["error_type"] == "WorkflowInputError"- assert "simulated inline search-attribute failure" in payload["error"]["message"]-- # The submitter sees the same inline error, classified — not a generic timeout.- submitter_error = exc_info.value- assert submitter_error.error_report is not None- assert submitter_error.error_report.error_type == "WorkflowInputError"- assert "simulated inline search-attribute failure" in submitter_error.message-- async def test_uncaught_pipelex_error_fails_terminally_on_production_worker(- self,- temporal_client: TemporalClient,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- failing_pipe_job: PipeJob,- ) -> None:- """The worker-level floor: a raw, uncaught ``PipelexError`` fails terminally instead of retrying forever."""- task_queue = f"q_floor_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_floor_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- workflow_execution_timeout=_FAILSAFE_EXECUTION_TIMEOUT,- )-- # Use the production make_worker so the assertion is against the real- # workflow_failure_exception_types config, with the stub registered alongside it.- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- test_workflows=[WfRawPipelexErrorStub],- ):- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(- workflow_class=WfRawPipelexErrorStub,- workflow_arg=failing_pipe_job,- workflow_id=workflow_id,- )-- error = exc_info.value- assert error.error_report is not None- # No catch-all converted it, so the floor degrades to a synthesized report — but it is- # terminal and preserves the original message, proving the workflow failed loud rather- # than hanging (a hang would have surfaced a timeout message instead).- assert error.error_report.error_type == "UnrecoverableWorkflowFailureError"- assert _RAW_INLINE_MESSAGE in error.messagediff --git a/tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.py b/tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.pydeleted file mode 100644index 0c3bd0416..000000000--- a/tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.py+++ /dev/null@@ -1,114 +0,0 @@-"""Temporal arm of the local / Temporal ``ErrorReport`` parity pair for web search.--Runs the ``native_search`` pipe through a real ``WfPipeRouter`` workflow on the in-process Temporal-server, submitted via ``WorkflowExecutor.execute_workflow``, with the activity-side search call mocked to-fail. This is the regression test for the original bug: before ``PipeSearch`` routed its leaf through a-Temporal activity, a search failure raised a raw ``SearchJobFailureError`` inside the workflow, which was-neither an ``ActivityError`` nor a ``WorkflowExecutionError`` — so Temporal retried the workflow task-forever and the submitter hung. Now the failure crosses the activity → workflow → submitter boundary as a-terminal ``WorkflowExecutionError`` carrying the structured ``ErrorReport``.--The local baseline is the separate module-``tests/integration/pipelex/error_handling/test_search_error_report_local_full_chain.py``; both arms assert-the same ``SearchErrorReportParityTestData`` constants, so local / Temporal parity holds by construction.-"""--import uuid-from collections.abc import Generator-from typing import TYPE_CHECKING--import pytest-from pytest_mock import MockerFixture-from temporalio.client import Client as TemporalClient-from temporalio.common import RetryPolicy--from pipelex.config import get_config-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.error_handling.test_data import SearchErrorReportParityTestData-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--ACT_SEARCH_SOURCED_ANSWER_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_sourced_answer"--[email protected](loop_scope="class")-class TestWorkflowSearchErrorReportFullChain:- """A failing search pipe's structured ``ErrorReport`` survives the Temporal workflow → submitter boundary."""-- @pytest.fixture- def temporal_enabled(self) -> Generator[None, None, None]:- """Enable ``temporal.is_enabled`` for the test — ``WorkflowExecutor.temporal_client()`` raises when off."""- config = get_config()- previous = config.temporal.is_enabled- config.temporal = config.temporal.model_copy(update={"is_enabled": True})- yield- config.temporal = config.temporal.model_copy(update={"is_enabled": previous})-- @pytest.fixture- def failing_search_pipe_job(self) -> Generator[PipeJob, None, None]:- """A PipeJob for the search pipe, in LIVE mode so the workflow dispatches the search activity.-- DRY mode short-circuits the ``act_search_*`` dispatch (the dry content generator reports inline- inside the workflow), so the activity — and the mock — never fire.- """- yield from pipe_job_from_bundle(- bundle_file=SearchErrorReportParityTestData.BUNDLE_FILE,- pipe_code=SearchErrorReportParityTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- )-- async def test_search_error_report_survives_temporal_boundary(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- failing_search_pipe_job: PipeJob,- ) -> None:- """A failing search run through Temporal surfaces a fully classified ``WorkflowExecutionError`` (no hang)."""- # Mock the search core inside the real activity (in-process unsandboxed worker, same process).- # @convert_pipelex_errors packs the ErrorReport into ApplicationError.details.- mocker.patch(- ACT_SEARCH_SOURCED_ANSWER_TARGET,- new=mocker.AsyncMock(side_effect=SearchErrorReportParityTestData.make_failing_search_error()),- )-- task_queue = f"q_search_err_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_search_err_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(- workflow_class=WfPipeRouter,- workflow_arg=failing_search_pipe_job,- workflow_id=workflow_id,- )-- error = exc_info.value- # The structured report was recovered across the workflow → submitter boundary.- assert error.error_report is not None- report = error.to_error_report()-- # The real failure message survived — not the generic "Failed to execute workflow ...".- assert SearchErrorReportParityTestData.FAILURE_MESSAGE in report.message- assert "Failed to execute workflow" not in report.message-- # The full classification survived the Temporal serialization round-trip.- assert report.error_category == SearchErrorReportParityTestData.FAILURE_CATEGORY- assert report.retryable == SearchErrorReportParityTestData.EXPECTED_RETRYABLE- assert report.model == SearchErrorReportParityTestData.FAILURE_MODEL- assert report.provider == SearchErrorReportParityTestData.FAILURE_PROVIDER- assert report.user_action is not None- assert report.user_action.kind == SearchErrorReportParityTestData.EXPECTED_USER_ACTION_KINDdiff --git a/tests/integration/pipelex/temporal/test_workflow_search_structured_roundtrip.py b/tests/integration/pipelex/temporal/test_workflow_search_structured_roundtrip.pydeleted file mode 100644index 48590fc7b..000000000--- a/tests/integration/pipelex/temporal/test_workflow_search_structured_roundtrip.py+++ /dev/null@@ -1,114 +0,0 @@-"""Structured-output counterpart to the search success round-trip test.--Runs a ``PipeSearch`` step whose output is a *custom structured concept* (``TopicSummary``) through a real-``WfPipeRouter`` workflow on the in-process Temporal server, with the activity-side search call mocked to-*succeed*. This exercises the ``act_search_gen_structured`` path — the riskier of the two search activities:-the operator ships the output structure's JSON schema (not the live class) across the boundary via-``SearchObjectAssignment``, the activity reconstructs a throwaway class to drive the provider and returns the-**raw result dict**, and the submitter re-validates that dict against the original output class-(``output_structure_class.model_validate(result_dict)``) — a pure, deterministic step that keeps the dynamic-class on the submitter side. This test locks in that whole round-trip; the sourced-answer sibling-(``test_workflow_search_success_roundtrip.py``) covers the non-structured path. The mock firing also proves the-workflow dispatched ``act_search_gen_structured`` rather than running the leaf inline on the workflow loop.-"""--import uuid-from collections.abc import Generator-from typing import TYPE_CHECKING--import pytest-from pytest_mock import MockerFixture-from temporalio.client import Client as TemporalClient-from temporalio.common import RetryPolicy--from pipelex.config import get_config-from pipelex.core.stuffs.structured_content import StructuredContent-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--ACT_SEARCH_STRUCTURED_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_structured"--BUNDLE_FILE = "tests/integration/pipelex/temporal/library_crate/structured_search.mthds"-PIPE_CODE = "structured_search"--# Flat, schema-shaped dict — matches what the real worker returns (LinkupSearchWorker._search_structured-# returns response.model_dump()), which the submitter re-validates against the TopicSummary structure class.-EXPECTED_TITLE = "AI safety in 2026"-EXPECTED_SUMMARY = "Interpretability and scalable oversight lead the agenda."-EXPECTED_KEY_POINTS = ["Mechanistic interpretability", "Scalable oversight"]--[email protected](loop_scope="class")-class TestWorkflowSearchStructuredRoundtrip:- """A structured search result survives the Temporal activity → workflow → submitter round-trip."""-- @pytest.fixture- def temporal_enabled(self) -> Generator[None, None, None]:- config = get_config()- previous = config.temporal.is_enabled- config.temporal = config.temporal.model_copy(update={"is_enabled": True})- yield- config.temporal = config.temporal.model_copy(update={"is_enabled": previous})-- @pytest.fixture- def structured_search_pipe_job(self) -> Generator[PipeJob, None, None]:- """A PipeJob for the structured search pipe, in LIVE mode so the workflow dispatches the search activity."""- yield from pipe_job_from_bundle(- bundle_file=BUNDLE_FILE,- pipe_code=PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- )-- async def test_structured_search_result_survives_temporal_boundary(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- structured_search_pipe_job: PipeJob,- ) -> None:- """The activity's raw dict re-validates into the TopicSummary structure class on the submitter, intact."""- mock_result = {- "title": EXPECTED_TITLE,- "summary": EXPECTED_SUMMARY,- "key_points": EXPECTED_KEY_POINTS,- }- search_core_mock = mocker.AsyncMock(return_value=mock_result)- mocker.patch(ACT_SEARCH_STRUCTURED_TARGET, new=search_core_mock)-- task_queue = f"q_search_struct_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_search_struct_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- pipe_output = await executor.execute_workflow(- workflow_class=WfPipeRouter,- workflow_arg=structured_search_pipe_job,- workflow_id=workflow_id,- )-- # The structured activity dispatched (mock fired) — the leaf did not run inline in the workflow,- # and the SearchObjectAssignment (carrying output_class_schema) crossed the boundary.- search_core_mock.assert_awaited_once()-- # The raw dict re-validated against the original output class on the submitter side and round-tripped- # intact (rehydrate the deferred working memory exactly as a production submitter does).- rehydrate_pipe_output(pipe_output, structured_search_pipe_job)- content = pipe_output.main_stuff.content- assert isinstance(content, StructuredContent)- data = content.model_dump()- assert data["title"] == EXPECTED_TITLE- assert data["summary"] == EXPECTED_SUMMARY- assert data["key_points"] == EXPECTED_KEY_POINTSdiff --git a/tests/integration/pipelex/temporal/test_workflow_search_structured_validation_failure.py b/tests/integration/pipelex/temporal/test_workflow_search_structured_validation_failure.pydeleted file mode 100644index 48acf7b56..000000000--- a/tests/integration/pipelex/temporal/test_workflow_search_structured_validation_failure.py+++ /dev/null@@ -1,109 +0,0 @@-"""Regression test: a malformed structured-search result fails the Temporal workflow terminally, not a hang.--The structured-search activity (``act_search_gen_structured``) returns the provider's *raw* result dict — the-gateway worker returns ``json.loads(...)`` unvalidated against the schema — and the submitter re-validates it-against the original output structure class with ``output_structure_class.model_validate(result_dict)`` in-workflow code, *outside* the activity's ``convert_pipelex_errors`` boundary. Left to raise a bare-``pydantic.ValidationError`` there, it would be neither a ``WorkflowExecutionError`` nor a ``PipelexError`` — the-only two ``workflow_failure_exception_types`` — so Temporal would treat it as a *workflow-task* failure and retry-indefinitely, hanging the submitter (the exact failure mode the whole search seam exists to prevent).-``make_search_structured`` converts it to a terminal ``ContentGenerationError`` (a ``PipelexError``) instead; this-test locks that in by feeding a malformed dict and asserting the submitter gets a terminal-``WorkflowExecutionError`` rather than hanging. The success sibling is ``test_workflow_search_structured_roundtrip.py``.-"""--import uuid-from collections.abc import Generator-from typing import TYPE_CHECKING--import pytest-from pytest_mock import MockerFixture-from temporalio.client import Client as TemporalClient-from temporalio.common import RetryPolicy--from pipelex.config import get_config-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--ACT_SEARCH_STRUCTURED_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_structured"--BUNDLE_FILE = "tests/integration/pipelex/temporal/library_crate/structured_search.mthds"-PIPE_CODE = "structured_search"--# A dict the provider could plausibly return that does NOT satisfy the TopicSummary structure: the required-# ``title`` field is missing, so re-validating it against the original output class on the submitter must fail.-MALFORMED_RESULT = {- "summary": "Interpretability and scalable oversight lead the agenda.",- "key_points": ["Mechanistic interpretability", "Scalable oversight"],-}--[email protected](loop_scope="class")-class TestWorkflowSearchStructuredValidationFailure:- """A malformed structured-search result surfaces a terminal WorkflowExecutionError instead of hanging."""-- @pytest.fixture- def temporal_enabled(self) -> Generator[None, None, None]:- config = get_config()- previous = config.temporal.is_enabled- config.temporal = config.temporal.model_copy(update={"is_enabled": True})- yield- config.temporal = config.temporal.model_copy(update={"is_enabled": previous})-- @pytest.fixture- def structured_search_pipe_job(self) -> Generator[PipeJob, None, None]:- """A PipeJob for the structured search pipe, in LIVE mode so the workflow dispatches the search activity."""- yield from pipe_job_from_bundle(- bundle_file=BUNDLE_FILE,- pipe_code=PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- )-- @pytest.mark.timeout(60)- async def test_malformed_structured_result_fails_terminally(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- structured_search_pipe_job: PipeJob,- ) -> None:- """A structured-search dict that fails submitter-side re-validation terminates the workflow, not hangs."""- search_core_mock = mocker.AsyncMock(return_value=MALFORMED_RESULT)- mocker.patch(ACT_SEARCH_STRUCTURED_TARGET, new=search_core_mock)-- task_queue = f"q_search_struct_bad_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_search_struct_bad_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(- workflow_class=WfPipeRouter,- workflow_arg=structured_search_pipe_job,- workflow_id=workflow_id,- )-- # The structured activity dispatched (mock fired) — the malformed dict came back through the boundary,- # and the failure happened on the submitter side, not inside the (successful) activity.- search_core_mock.assert_awaited_once()-- # Terminal failure, not a hang: the submitter recovered a structured report whose message reflects the- # submitter-side structured-validation failure (not a generic "Failed to execute workflow ...").- error = exc_info.value- assert error.error_report is not None- report = error.to_error_report()- assert "validation" in report.message.lower()- assert "Failed to execute workflow" not in report.messagediff --git a/tests/integration/pipelex/temporal/test_workflow_search_success_roundtrip.py b/tests/integration/pipelex/temporal/test_workflow_search_success_roundtrip.pydeleted file mode 100644index bb85dcb07..000000000--- a/tests/integration/pipelex/temporal/test_workflow_search_success_roundtrip.py+++ /dev/null@@ -1,103 +0,0 @@-"""Success-path counterpart to the search error-report full-chain test.--Runs the ``native_search`` pipe through a real ``WfPipeRouter`` workflow on the in-process Temporal server-with the activity-side search call mocked to *succeed*, and asserts the ``SearchResultContent`` comes back-intact across the activity → workflow → submitter boundary. This locks in the other half of the fix: search-on Temporal now records its result via an activity (replay-safe) instead of running inline on the workflow-loop, and the native ``SearchResultContent`` serializes cleanly back to the submitter. The mock firing also-proves the workflow actually dispatched ``act_search_gen_sourced_answer`` rather than running the leaf inline.-"""--import uuid-from collections.abc import Generator-from typing import TYPE_CHECKING--import pytest-from pytest_mock import MockerFixture-from temporalio.client import Client as TemporalClient-from temporalio.common import RetryPolicy--from pipelex.config import get_config-from pipelex.core.stuffs.document_content import DocumentContent-from pipelex.core.stuffs.search_result_content import SearchResultContent-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from tests.integration.pipelex.error_handling.test_data import SearchErrorReportParityTestData-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--ACT_SEARCH_SOURCED_ANSWER_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_sourced_answer"--EXPECTED_ANSWER = "Recent AI safety work focuses on interpretability and scalable oversight."-EXPECTED_SOURCE_TITLE = "AI safety overview"--[email protected](loop_scope="class")-class TestWorkflowSearchSuccessRoundtrip:- """A successful search result survives the Temporal activity → workflow → submitter round-trip."""-- @pytest.fixture- def temporal_enabled(self) -> Generator[None, None, None]:- config = get_config()- previous = config.temporal.is_enabled- config.temporal = config.temporal.model_copy(update={"is_enabled": True})- yield- config.temporal = config.temporal.model_copy(update={"is_enabled": previous})-- @pytest.fixture- def search_pipe_job(self) -> Generator[PipeJob, None, None]:- """A PipeJob for the search pipe, in LIVE mode so the workflow dispatches the search activity."""- yield from pipe_job_from_bundle(- bundle_file=SearchErrorReportParityTestData.BUNDLE_FILE,- pipe_code=SearchErrorReportParityTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- )-- async def test_search_result_survives_temporal_boundary(- self,- temporal_client: TemporalClient,- mocker: MockerFixture,- temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration- search_pipe_job: PipeJob,- ) -> None:- """The activity's ``SearchResultContent`` comes back intact on the submitter's ``PipeOutput``."""- mock_result = SearchResultContent(- answer=EXPECTED_ANSWER,- sources=[DocumentContent(title=EXPECTED_SOURCE_TITLE, url="https://example.com/ai-safety", mime_type="text/html")],- )- search_core_mock = mocker.AsyncMock(return_value=mock_result)- mocker.patch(ACT_SEARCH_SOURCED_ANSWER_TARGET, new=search_core_mock)-- task_queue = f"q_search_ok_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_search_ok_{uuid.uuid4().hex[:8]}"- executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(- temporal_client=temporal_client,- task_queue=task_queue,- retry_policy=RetryPolicy(maximum_attempts=1),- )-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- pipe_output = await executor.execute_workflow(- workflow_class=WfPipeRouter,- workflow_arg=search_pipe_job,- workflow_id=workflow_id,- )-- # The activity dispatched (mock fired) — the leaf did not run inline in the workflow.- search_core_mock.assert_awaited_once()-- # The SearchResultContent round-tripped back to the submitter intact (rehydrate the deferred- # working memory exactly as a production submitter does).- rehydrate_pipe_output(pipe_output, search_pipe_job)- content = pipe_output.main_stuff.content- assert isinstance(content, SearchResultContent)- assert content.answer == EXPECTED_ANSWER- assert len(content.sources) == 1- assert content.sources[0].title == EXPECTED_SOURCE_TITLEdiff --git a/tests/integration/pipelex/temporal/tracing/conftest.py b/tests/integration/pipelex/temporal/tracing/conftest.pydeleted file mode 100644index 23804c8d8..000000000--- a/tests/integration/pipelex/temporal/tracing/conftest.py+++ /dev/null@@ -1,100 +0,0 @@-"""Fixtures for Temporal graph tracing integration tests.--Provides tracing-enabled PipeJob fixtures and a config fixture-that enables event tracing for the test class.-"""--from collections.abc import Generator-from pathlib import Path--import pytest--from pipelex.config import get_config-from pipelex.graph.graph_tracer_manager import GraphTracerManager-from pipelex.pipe_run.pipe_job import PipeJob-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.tracing.test_data import (- BatchTracingTestData,- ParallelTracingTestData,- SequenceTracingTestData,-)--_TRACING_DIR = Path(__file__).parent.resolve()---def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:- """Mark every test under this directory with gha_disabled.-- The hook receives all session-wide items (not just items below this- conftest), so filter by file path before adding the marker.-- TODO: Tests under tests/integration/pipelex/temporal/tracing/ hang- reliably in CI under pytest-xdist parallelism (worker timeouts at- 180s on py3.11+). Pass locally and serially. Root cause is concurrent- PipeBatch/PipeParallel/PipeSequence + WorkflowEnvironment.start_local- contention under load. Re-enable once the underlying flake is fixed.- """- skip_marker = pytest.mark.gha_disabled- for item in items:- item_path = Path(str(item.path)).resolve()- if _TRACING_DIR in item_path.parents:- item.add_marker(skip_marker)--[email protected](scope="class")-def tracing_tmp_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:- """Per-class temp directory for NDJSON trace files."""- return tmp_path_factory.mktemp("traces")--[email protected](scope="class", autouse=True)-def enable_tracing(tracing_tmp_dir: Path) -> Generator[None, None, None]:- """Enable tracing config and point traces_dir to the temp directory.-- Restores original config values and cleans up GraphTracerManager on teardown.- """- tracing_config = get_config().pipelex.tracing_config- original_enabled = tracing_config.is_enabled- ndjson_config = tracing_config.ndjson- original_dir = ndjson_config.traces_dir if ndjson_config else ""-- tracing_config.is_enabled = True- if ndjson_config:- ndjson_config.traces_dir = str(tracing_tmp_dir)-- yield-- tracing_config.is_enabled = original_enabled- if ndjson_config:- ndjson_config.traces_dir = original_dir- GraphTracerManager.clear_instance()--[email protected](scope="class")-def sequence_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob for native_text_sequence (trace_context injected per-execution by execute_and_assemble)."""- yield from pipe_job_from_bundle(- bundle_file=SequenceTracingTestData.BUNDLE_FILE,- pipe_code=SequenceTracingTestData.PIPE_CODE,- isolated_registry=is_class_registry_isolated,- )--[email protected](scope="class")-def parallel_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob for temporal_parallel (trace_context injected per-execution by execute_and_assemble)."""- yield from pipe_job_from_bundle(- bundle_file=ParallelTracingTestData.BUNDLE_FILE,- pipe_code=ParallelTracingTestData.PIPE_CODE,- isolated_registry=is_class_registry_isolated,- )--[email protected](scope="class")-def batch_tracing_job(is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """PipeJob for temporal_batch (trace_context injected per-execution by execute_and_assemble)."""- yield from pipe_job_from_bundle(- bundle_file=BatchTracingTestData.BUNDLE_FILE,- pipe_code=BatchTracingTestData.PIPE_CODE,- isolated_registry=is_class_registry_isolated,- )diff --git a/tests/integration/pipelex/temporal/tracing/helpers.py b/tests/integration/pipelex/temporal/tracing/helpers.pydeleted file mode 100644index 2774bac1e..000000000--- a/tests/integration/pipelex/temporal/tracing/helpers.py+++ /dev/null@@ -1,371 +0,0 @@-"""Shared test helpers for Temporal graph tracing integration tests."""--import uuid-from collections import Counter, defaultdict-from collections.abc import AsyncGenerator, Awaitable, Callable, Generator, Iterable-from contextlib import asynccontextmanager, contextmanager-from datetime import datetime-from pathlib import Path--from temporalio import activity-from temporalio.client import Client as TemporalClient-from temporalio.client import WorkflowHistory--from pipelex.cogt.content_generation.assignment_models import LLMAssignment-from pipelex.cogt.llm.llm_job import LLMJob-from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobReport-from pipelex.cogt.llm.llm_report import LLMTokensUsage-from pipelex.cogt.usage.cost_category import CostCategory-from pipelex.cogt.usage.token_category import TokenCategory-from pipelex.config import get_config-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.graph.graph_config import DataInclusionConfig-from pipelex.graph.graphspec import EdgeKind, EdgeSpec, GraphSpec, NodeStatus, PipelineRef-from pipelex.graph.trace_context import TraceContext-from pipelex.hub import get_report_delegate-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.temporal.config_temporal import ActivityRouteConfig-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text-from pipelex.temporal.tprl_pipe.act_flush_trace_events import FlushTraceEventsArg-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.tracing.graphspec_assembler import GraphSpecAssembler-from pipelex.tracing.ndjson_event_log import NdjsonEventLog-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output---@contextmanager-def route_activities_to(queue: str, activity_names: Iterable[str]) -> Generator[None, None, None]:- """Temporarily route the given activity names to ``queue`` via- ``worker_config.activity_queues``, restoring prior entries on exit.-- Tests that run an in-process worker on a UUID-based task queue and- substitute activities on that worker need this override: without it the- in-workflow dispatcher resolves to ``worker_config.default_task_queue``- (the production default) where no worker is listening, and the activity- hangs until pytest-timeout fires.- """- worker_config = get_config().temporal.worker_config- originals: dict[str, ActivityRouteConfig | None] = {}- for activity_name in activity_names:- originals[activity_name] = worker_config.activity_queues.get(activity_name)- worker_config.activity_queues[activity_name] = ActivityRouteConfig(default=queue, by_handle={})- try:- yield- finally:- for activity_name, original in originals.items():- if original is None:- worker_config.activity_queues.pop(activity_name, None)- else:- worker_config.activity_queues[activity_name] = original---def inject_trace_context(- pipe_job: PipeJob,- pipeline_run_id: str,- *,- emit_graph_events: bool = True,- emit_usage_events: bool = True,-) -> PipeJob:- """Deep-copy a PipeJob and inject a TraceContext onto its JobMetadata.-- Also overrides pipeline_run_id (must not be the dry-run sentinel,- since NdjsonEventLog uses it as a directory name).-- The emit flags default to True (the legacy "context present → emit both"- behavior); pass ``emit_graph_events=False`` to exercise costs-only mode.- """- trace_context = TraceContext(- graph_id=pipeline_run_id,- parent_node_id=None,- node_sequence=0,- data_inclusion=DataInclusionConfig(- pipe_and_concept_registry=True,- stuff_json_content=False,- stuff_text_content=False,- stuff_html_content=False,- error_stack_traces=False,- ),- emit_graph_events=emit_graph_events,- emit_usage_events=emit_usage_events,- )- new_metadata = pipe_job.job_metadata.model_copy(- update={- "trace_context": trace_context,- "pipeline_run_id": pipeline_run_id,- },- )- return pipe_job.model_copy(update={"job_metadata": new_metadata})---class TracingResult:- """Result of a traced workflow execution."""-- def __init__(self, pipe_output: PipeOutput, graph_spec: GraphSpec | None, pipeline_run_id: str) -> None:- self.pipe_output = pipe_output- self.graph_spec = graph_spec- self.pipeline_run_id = pipeline_run_id---async def execute_and_assemble(- pipe_job: PipeJob,- temporal_client: TemporalClient,- traces_dir: str,-) -> TracingResult:- """Execute a workflow on Temporal and assemble a GraphSpec from emitted events.-- Each call uses a unique pipeline_run_id to isolate NDJSON files across calls.-- Returns:- TracingResult with pipe_output, graph_spec, and pipeline_run_id.- """- # Give each execution its own pipeline_run_id to avoid event accumulation- execution_run_id = f"tracing_exec_{uuid.uuid4().hex[:12]}"- execution_job = inject_trace_context(pipe_job, execution_run_id)-- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=execution_job,- id=workflow_id,- task_queue=task_queue,- )-- rehydrate_pipe_output(pipe_output)-- # Read events and assemble GraphSpec- event_log = NdjsonEventLog(traces_dir=traces_dir)- try:- events = event_log.read_events(execution_run_id)- finally:- event_log.close()-- if not events:- return TracingResult(pipe_output=pipe_output, graph_spec=None, pipeline_run_id=execution_run_id)-- domain = pipe_job.pipe.domain_code- main_pipe = pipe_job.pipe.code- graph_spec = GraphSpecAssembler.assemble(- events=events,- graph_id=execution_run_id,- pipeline_ref=PipelineRef(domain=domain, main_pipe=main_pipe),- )- return TracingResult(pipe_output=pipe_output, graph_spec=graph_spec, pipeline_run_id=execution_run_id)---def assert_all_nodes_terminal(graph_spec: GraphSpec) -> None:- """Assert that no nodes are left in RUNNING or SCHEDULED state."""- terminal_statuses = {NodeStatus.SUCCEEDED, NodeStatus.FAILED, NodeStatus.CANCELED, NodeStatus.SKIPPED}- for node in graph_spec.nodes:- assert node.status in terminal_statuses, f"Node '{node.node_id}' (pipe_code={node.pipe_code}) has non-terminal status: {node.status}"---def edges_by_kind(graph_spec: GraphSpec) -> dict[EdgeKind, list[EdgeSpec]]:- """Group edges by their EdgeKind for targeted assertions."""- result: dict[EdgeKind, list[EdgeSpec]] = defaultdict(list)- for edge in graph_spec.edges:- result[edge.kind].append(edge)- return dict(result)---def ndjson_files_for_run(traces_dir: str, pipeline_run_id: str) -> list[Path]:- """List all NDJSON files generated for a pipeline run."""- run_dir = Path(traces_dir) / pipeline_run_id- if not run_dir.exists():- return []- return sorted(run_dir.glob("*.ndjson"))---def node_pipe_codes(graph_spec: GraphSpec) -> set[str | None]:- """Extract the set of pipe_codes from all nodes in a GraphSpec."""- return {node.pipe_code for node in graph_spec.nodes}---def scheduled_activity_names(history: WorkflowHistory) -> list[str]:- """Activity type names scheduled in a fetched workflow history, in event order."""- return [- event.activity_task_scheduled_event_attributes.activity_type.name- for event in history.events- if event.HasField("activity_task_scheduled_event_attributes")- ]--[email protected](name="act_flush_trace_events")-async def act_flush_noop(arg: FlushTraceEventsArg) -> None:- """No-op flush substitute: keeps a test independent of any tracing backend.-- Only the *schedule* of this activity matters to the command stream under test.- """---def make_synthetic_usage_llm_job(- llm_assignment: LLMAssignment,- inference_model_name: str,- inference_model_id: str,-) -> LLMJob:- """Synthesize an ``LLMJob`` with non-zero token usage from an activity's assignment.-- For activity substitutes that need ``report_inference_job`` to emit a real- ``UsageReportEvent`` without performing an actual LLM call.- """- now = datetime.now()- synthetic_metadata = llm_assignment.job_metadata.model_copy(- update={- "started_at": llm_assignment.job_metadata.started_at or now,- "completed_at": now,- },- )- tokens_usage = LLMTokensUsage(- job_metadata=synthetic_metadata,- inference_model_name=inference_model_name,- inference_model_id=inference_model_id,- unit_costs={CostCategory.INPUT: 0.0, CostCategory.OUTPUT: 0.0},- nb_tokens_by_category={TokenCategory.INPUT: 1, TokenCategory.OUTPUT: 1},- )- return LLMJob(- job_metadata=synthetic_metadata,- llm_prompt=llm_assignment.llm_prompt,- job_params=llm_assignment.llm_setting.make_llm_job_params(),- job_config=LLMJobConfig(schema_reask_max_attempts=1),- job_report=LLMJobReport(llm_tokens_usage=tokens_usage),- )---_RUNNER_FAKE_INFERENCE_MODEL_NAME = "split_runner_fake"-_RUNNER_FAKE_INFERENCE_MODEL_ID = "split_runner_fake_id"-_RUNNER_FAKE_RESPONSE_TEXT = "split-worker fake response"--[email protected](name="act_llm_gen_text")-async def _synthetic_usage_act_llm_gen_text(llm_assignment: LLMAssignment) -> str: # noqa: RUF029- """Substitute for `act_llm_gen_text` that emits synthetic usage from the activity.-- Synthesizes an `LLMJob` with non-zero usage and reports it via- `report_inference_job`, then returns a fixed string so the calling workflow- can finish without attempting a real LLM call. The emission unconditionally- takes the per-process activity event log: `_emit_usage_event` checks- `_is_in_temporal_activity()` before any context lookup, so an in-activity- emission never touches a workflow's registered buffer — co-located or split- deployment alike (audit finding H1).-- Since the unified dry run (leaf-level dry mock), DRY mode also dispatches- `act_llm_gen_text` (the leaf mocks inside the activity with zero-token,- suppressed usage), so this substitute is what guarantees a *reportable*- non-zero usage event lands on the runner regardless of the fixture's run- mode (we route `act_llm_gen_text` to `q_runner` via- `worker_config.activity_queues`) — keeping the cross-worker assertion- hermetic and deterministic.- """- synthetic_job = make_synthetic_usage_llm_job(- llm_assignment=llm_assignment,- inference_model_name=_RUNNER_FAKE_INFERENCE_MODEL_NAME,- inference_model_id=_RUNNER_FAKE_INFERENCE_MODEL_ID,- )- get_report_delegate().report_inference_job(inference_job=synthetic_job)- return _RUNNER_FAKE_RESPONSE_TEXT---@asynccontextmanager-async def make_split_workers(- temporal_client: TemporalClient,- q_router: str,- q_runner: str,- runner_act_llm_gen_text: Callable[[LLMAssignment], Awaitable[str]] = _synthetic_usage_act_llm_gen_text,-) -> AsyncGenerator[None, None]:- """Open two scoped workers on two task queues in the current process.-- - `q_router`: workflows + `act_flush_trace_events`. We don't use the bare- `router` scope (`disable_all_activities=True`) here because- `WfPipeRouter.run` dispatches `act_flush_trace_events` without a- `task_queue` argument — the activity lands on the workflow's own queue- and would never be picked up if the router registered no activities.- - `q_runner`: activity-only (runner scope, `disable_all_workflows=True`),- with `act_llm_gen_text` substituted by ``runner_act_llm_gen_text``. The- default substitute synthesizes usage instead of calling a real LLM; the- in-activity emission always lands in the per-process activity event log- (never the router's registered context — audit finding H1). Pass a- different substitute — e.g. one that runs real inference — to exercise- the same path with real provider token counts.-- Pair this with `worker_config.activity_queues[act_llm_gen_text.__name__] =- ActivityRouteConfig(default=q_runner, by_handle={})` so the workflow on- `q_router` actually dispatches `act_llm_gen_text` to `q_runner`.- """- worker_scopes = get_config().temporal.worker_scopes- base_router = worker_scopes.scopes["router"]- runner_scope = worker_scopes.scopes["runner"]-- # Override the bare router scope to also register `act_flush_trace_events`- # so the workflow's flush call does not deadlock waiting for an activity- # worker on q_router. We keep `disable_all_activities=True` and add the- # activity back via `required_activities` — but resolution clears- # `disable_all_activities` last, so we instead use an empty exclude set- # against the runner scope's activity surface.- router_scope = base_router.model_copy(- update={- "disable_all_activities": False,- "required_activities": ["act_flush_trace_events"],- "excluded_activities": [- "act_llm_gen_text",- "act_llm_gen_object",- "act_llm_gen_object_list",- "act_img_gen_images",- "act_jinja2_gen_text",- "act_extract_gen_extract_pages",- "act_render_page_views",- "act_assemble_tracing",- "act_deliver",- ],- },- )-- async with (- get_task_manager().make_worker(- temporal_client,- task_queue=q_router,- is_not_sandboxed=True,- scope=router_scope,- ),- get_task_manager().make_worker(- temporal_client,- task_queue=q_runner,- is_not_sandboxed=True,- scope=runner_scope,- substitute_activities={act_llm_gen_text: runner_act_llm_gen_text},- ),- ):- yield---async def collect_scheduled_activity_counts(temporal_client: TemporalClient, workflow_id: str) -> "Counter[str]":- """Walk a workflow tree (parent + child workflows, pinned by run_id) and count every scheduled activity.-- Counts ``ActivityTaskScheduled`` events by activity-type name across the parent history and every- child workflow it started — the leaf-dispatch evidence Tier 17 asserts on. Children are fetched by- ``(workflow_id, run_id)`` so a retried/continued run under the same id cannot substitute a history- the parent never observed.- """- scheduled_counts: Counter[str] = Counter()- pending_runs: list[tuple[str, str | None]] = [(workflow_id, None)]- visited_runs: set[tuple[str, str | None]] = set()- while pending_runs:- current_run = pending_runs.pop()- if current_run in visited_runs:- continue- visited_runs.add(current_run)- current_id, current_run_id = current_run- history = await temporal_client.get_workflow_handle(current_id, run_id=current_run_id).fetch_history()- for event in history.events:- if event.HasField("activity_task_scheduled_event_attributes"):- scheduled_counts[event.activity_task_scheduled_event_attributes.activity_type.name] += 1- elif event.HasField("child_workflow_execution_started_event_attributes"):- child_execution = event.child_workflow_execution_started_event_attributes.workflow_execution- pending_runs.append((child_execution.workflow_id, child_execution.run_id))- return scheduled_countsdiff --git a/tests/integration/pipelex/temporal/tracing/test_data.py b/tests/integration/pipelex/temporal/tracing/test_data.pydeleted file mode 100644index ad89a606b..000000000--- a/tests/integration/pipelex/temporal/tracing/test_data.py+++ /dev/null@@ -1,66 +0,0 @@-"""Test data constants for Temporal graph tracing integration tests.--Expected graph structures for each bundle used in tracing tests.-"""--from typing import ClassVar--_CRATE_DIR = "tests/integration/pipelex/temporal/library_crate"---class SequenceTracingTestData:- """Expected graph structure for native_text_sequence.mthds.-- Bundle: PipeSequence with 2 PipeLLM steps (step_one → step_two).- Nodes: sequence controller + step_one + step_two.- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/native_text_sequence.mthds"- PIPE_CODE: ClassVar[str] = "native_text_sequence"- DOMAIN: ClassVar[str] = "native_text_test"-- EXPECTED_PIPE_CODES: ClassVar[set[str | None]] = {- "native_text_sequence",- "step_one",- "step_two",- }- EXPECTED_NODE_COUNT: ClassVar[int] = 3- EXPECTED_CONTAINS_EDGE_COUNT: ClassVar[int] = 2- MIN_DATA_EDGES: ClassVar[int] = 1---class ParallelTracingTestData:- """Expected graph structure for temporal_parallel.mthds.-- Bundle: PipeSequence wrapping PipeParallel (2 branches) + summarize step.- Nodes: outer sequence + analyze_in_parallel + branch_tone + branch_length + summarize_results.- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_parallel.mthds"- PIPE_CODE: ClassVar[str] = "temporal_parallel_sequence"- DOMAIN: ClassVar[str] = "temporal_parallel_test"-- EXPECTED_PIPE_CODES: ClassVar[set[str | None]] = {- "temporal_parallel_sequence",- "analyze_in_parallel",- "branch_tone",- "branch_length",- "summarize_results",- }- MIN_NODE_COUNT: ClassVar[int] = 5---class BatchTracingTestData:- """Expected graph structure for temporal_batch.mthds.-- Bundle: PipeSequence wrapping PipeBatch (fan-out to per-item child workflows).- Nodes: outer sequence + generate_topics + batch controller + per-item processors.- """-- BUNDLE_FILE: ClassVar[str] = f"{_CRATE_DIR}/temporal_batch.mthds"- PIPE_CODE: ClassVar[str] = "temporal_batch_sequence"- DOMAIN: ClassVar[str] = "temporal_batch_test"-- MIN_NODE_COUNT: ClassVar[int] = 4- MIN_BATCH_ITEM_EDGES: ClassVar[int] = 1- MIN_BATCH_AGGREGATE_EDGES: ClassVar[int] = 1diff --git a/tests/integration/pipelex/temporal/tracing/test_dry_run_dispatches_and_mocks.py b/tests/integration/pipelex/temporal/tracing/test_dry_run_dispatches_and_mocks.pydeleted file mode 100644index f33313a0c..000000000--- a/tests/integration/pipelex/temporal/tracing/test_dry_run_dispatches_and_mocks.py+++ /dev/null@@ -1,103 +0,0 @@-"""Mode-1 companion of Tier 17: DRY honors the Temporal backend (Part B, req 1).--The default-dry analogue of ``test_mock_usage_temporal.py``. Pre-Part-B, a Temporal dry run never-dispatched ``act_llm_gen_*`` (operators swapped in a workflow-side dry generator); after Part B,-``run_mode=DRY`` rides ``CogtRunParams`` across the serialization boundary, the workflow dispatches-the REAL ``act_llm_gen_text`` activity, and the cogt leaf mocks INSIDE the activity — zero-token-synthetic usage (cost report suppressed by design), no provider call.--The dispatch assertion reads the workflow history (``ActivityTaskScheduled`` across the parent and-its child workflows), which is the same signal the Mode-2 / Tier 17 e2e arm checks against a real-Temporal server. Lives under ``tracing/`` so the directory conftest applies both the CI quarantine-(in-process WorkflowEnvironment + child workflows hang under CI xdist) and the tmp-dir trace-redirect (usage assembly must not write to the configured traces backend).-"""--import uuid-from collections.abc import Generator-from typing import TYPE_CHECKING--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.cogt.content_generation.dry_mock import DRY_RUN_INFERENCE_MODEL_NAME-from pipelex.cogt.usage.cost_registry import CostRegistry-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text-from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.tracing.helpers import collect_scheduled_activity_counts, inject_trace_context-from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--# native_text_sequence is a two-step PipeSequence of PipeLLMs: each step must dispatch its own-# act_llm_gen_text and report its own (zero-token) usage — a single dispatch would mean one step-# silently bypassed the distribution machinery.-EXPECTED_LLM_DISPATCHES = 2--[email protected](loop_scope="class")-class TestTemporalDryRunDispatchesAndMocks:- @pytest.fixture- def dry_sequence_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """A DRY native_text_sequence job — same bundle as the mock-usage arm, default dry reporting."""- yield from pipe_job_from_bundle(- bundle_file=SequenceTracingTestData.BUNDLE_FILE,- pipe_code=SequenceTracingTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.DRY,- isolated_registry=is_class_registry_isolated,- )-- async def test_dry_run_dispatches_activities_and_mocks_inside(- self,- dry_sequence_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """DRY dispatches the real act_llm_gen_* and the leaf mocks inside, zero-token suppressed."""- execution_run_id = f"dry_dispatch_{uuid.uuid4().hex[:12]}"- execution_job = inject_trace_context(- dry_sequence_job,- execution_run_id,- emit_graph_events=False,- emit_usage_events=True,- )- assert execution_job.pipe_run_params.run_mode.is_dry, "inject_trace_context must preserve run_mode"-- pipe_run_arg = PipeRunArg(pipe_job=execution_job).prepare_for_temporal()- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- workflow_handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRun.run,- arg=pipe_run_arg,- id=workflow_id,- task_queue=task_queue,- )- pipe_output: PipeOutput = await workflow_handle.result()- # The LLM activity is scheduled by the controller's CHILD workflows (one per step), so- # count ActivityTaskScheduled events across the whole workflow tree.- scheduled_counts = await collect_scheduled_activity_counts(temporal_client, workflow_id)-- # THE req-1 flip: the dry run DID dispatch the LLM activity, once per sequence step- # (pre-Part-B it dispatched nothing inference-related).- assert scheduled_counts[act_llm_gen_text.__name__] == EXPECTED_LLM_DISPATCHES-- rehydrate_pipe_output(pipe_output, pipe_job=dry_sequence_job)-- # The leaf mocked INSIDE the activity: the output is the DRY mock, not provider text.- assert "DRY RUN:" in pipe_output.main_stuff_as_text.text-- # One zero-token usage per step under the dry_run sentinel — assembled back cross-process,- # but the cost report stays suppressed (the inverse of is_mock_usage's reportable payload).- assert pipe_output.tokens_usages is not None- assert len(pipe_output.tokens_usages) == EXPECTED_LLM_DISPATCHES- assert all(usage.inference_model_name == DRY_RUN_INFERENCE_MODEL_NAME for usage in pipe_output.tokens_usages)- assert not CostRegistry.aggregate_costs(pipe_output.tokens_usages).has_reportable_usagediff --git a/tests/integration/pipelex/temporal/tracing/test_mock_usage_temporal.py b/tests/integration/pipelex/temporal/tracing/test_mock_usage_temporal.pydeleted file mode 100644index 91f3abe24..000000000--- a/tests/integration/pipelex/temporal/tracing/test_mock_usage_temporal.py+++ /dev/null@@ -1,95 +0,0 @@-"""Temporal (cross-process) coverage for the ``is_mock_usage`` dry sub-flag.--The DIRECT counterpart lives in-``tests/integration/pipelex/pipeline/test_mock_usage_direct.py``. This arm proves the part DIRECT-cannot: that ``CogtRunParams.is_mock_usage`` survives the Temporal serialization boundary and the-**real** ``act_llm_gen_text`` activity honors it — the dry leaf inside the activity body selects the-non-zero reporting payload (no provider call), with the synthetic, reportable usage assembling back-onto ``PipeOutput.tokens_usages`` via Step 2's ``act_assemble_tracing``.--Run DRY (the flag is a sub-flag of DRY): the workflow still dispatches the real activity tree-(run mode is orthogonal to backend — Tier 17), and ``is_mock_usage`` flips the leaf reporting from-the suppressed zero-token payload to the reportable non-zero one.--Like its sibling tracing tests this is ``gha_disabled`` by the directory conftest (the in-process-WorkflowEnvironment + concurrent pipe execution hangs under CI xdist); verify locally/serially.-"""--import uuid-from collections.abc import Generator-from typing import TYPE_CHECKING--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.cogt.content_generation.dry_mock import MOCK_USAGE_MODEL_NAME-from pipelex.cogt.usage.cost_registry import CostRegistry-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.tracing.helpers import inject_trace_context-from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--[email protected](loop_scope="class")-class TestTemporalMockUsage:- @pytest.fixture- def mock_usage_sequence_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """A DRY native_text_sequence job carrying is_mock_usage=True.-- DRY still dispatches the real ``act_llm_gen_text`` (run mode orthogonal to backend); the- flag makes that activity's leaf report non-zero synthetic usage instead of the suppressed- zero-token payload.- """- for pipe_job in pipe_job_from_bundle(- bundle_file=SequenceTracingTestData.BUNDLE_FILE,- pipe_code=SequenceTracingTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.DRY,- isolated_registry=is_class_registry_isolated,- ):- mocked_run_params = pipe_job.pipe_run_params.model_copy(update={"is_mock_usage": True})- yield pipe_job.model_copy(update={"pipe_run_params": mocked_run_params})-- async def test_mock_usage_assembles_reportable_usage_cross_process(- self,- mock_usage_sequence_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """The flag crosses the boundary, the real activity honors it, and reportable usage rides back."""- execution_run_id = f"mock_usage_{uuid.uuid4().hex[:12]}"- # costs-only (no graph) mirrors the realistic --no-graph --costs default the skill validates.- execution_job = inject_trace_context(- mock_usage_sequence_job,- execution_run_id,- emit_graph_events=False,- emit_usage_events=True,- )- assert execution_job.pipe_run_params.cogt_run_params.is_mock_usage, "inject_trace_context must preserve is_mock_usage"-- pipe_run_arg = PipeRunArg(pipe_job=execution_job).prepare_for_temporal()- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRun.run,- arg=pipe_run_arg,- id=workflow_id,- task_queue=task_queue,- )- rehydrate_pipe_output(pipe_output, pipe_job=mock_usage_sequence_job)-- assert pipe_output.tokens_usages is not None- assert len(pipe_output.tokens_usages) >= 1- # All usage came from the mocked leaf (the sentinel model) — proof no real provider ran.- assert all(usage.inference_model_name == MOCK_USAGE_MODEL_NAME for usage in pipe_output.tokens_usages)- # Non-zero synthetic usage -> a cost report would render (not suppressed like a default dry run).- assert CostRegistry.aggregate_costs(pipe_output.tokens_usages).has_reportable_usagediff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.pydeleted file mode 100644index 81588e923..000000000--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py+++ /dev/null@@ -1,148 +0,0 @@-"""Cross-child cost aggregation in split-worker mode.--The fan-out join is by ``pipeline_run_id`` on a shared partition: each child-workflow gets a distinct ``workflow_id`` but the same ``pipeline_run_id``, and-only the top-level run reads the whole partition. No automated test asserts that-usage emitted from *separate child workflows* sums into one parent total — the-existing fan-out tracing tests are graph-only/dry-run, and the sequence-split-worker tests have no child workflows.--This runs the PipeParallel bundle (two branches dispatched as separate child-workflows) in LIVE mode with the runner-side fallback, then asserts the usage-events from multiple workflows aggregate into a single priced total.-"""--import uuid-from collections.abc import Generator-from pathlib import Path-from typing import TYPE_CHECKING--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.cogt.usage.cost_registry import CostRegistry-from pipelex.cogt.usage.token_category import TokenCategory-from pipelex.config import get_config-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.config_temporal import ActivityRouteConfig-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.tracing.ndjson_event_log import NdjsonEventLog-from pipelex.tracing.trace_events import UsageReportEvent-from pipelex.tracing.usage_aggregator import UsageAggregator-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.tracing.helpers import (- inject_trace_context,- make_split_workers,- ndjson_files_for_run,-)-from tests.integration.pipelex.temporal.tracing.test_data import ParallelTracingTestData--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--[email protected](loop_scope="class")-class TestSplitWorkerCrossChildUsage:- """Usage from fan-out child workflows aggregates into one parent cost total."""-- @pytest.fixture- def split_queues(self) -> tuple[str, str]:- return (f"q_router_{uuid.uuid4().hex[:8]}", f"q_runner_{uuid.uuid4().hex[:8]}")-- @pytest.fixture- def live_parallel_tracing_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """The PipeParallel bundle in LIVE mode so branches dispatch ``act_llm_gen_text``.-- LIVE forces the in-workflow activity hop (the runner-side fallback under test); the fake- ``act_llm_gen_text`` installed by ``make_split_workers`` returns a stub string and- synthesizes usage, so no real LLM call is made.- """- yield from pipe_job_from_bundle(- bundle_file=ParallelTracingTestData.BUNDLE_FILE,- pipe_code=ParallelTracingTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- isolated_registry=is_class_registry_isolated,- )-- @pytest.fixture- def route_llm_text_to_runner_queue(self, split_queues: tuple[str, str]) -> Generator[None, None, None]:- _q_router, q_runner = split_queues- worker_config = get_config().temporal.worker_config- activity_name = act_llm_gen_text.__name__- original_entry = worker_config.activity_queues.get(activity_name)- worker_config.activity_queues[activity_name] = ActivityRouteConfig(default=q_runner, by_handle={})- yield- if original_entry is None:- worker_config.activity_queues.pop(activity_name, None)- else:- worker_config.activity_queues[activity_name] = original_entry-- async def _execute_split(- self,- live_parallel_tracing_job: PipeJob,- temporal_client: TemporalClient,- split_queues: tuple[str, str],- ) -> str:- q_router, q_runner = split_queues- execution_run_id = f"split_child_{uuid.uuid4().hex[:12]}"- execution_job = inject_trace_context(live_parallel_tracing_job, execution_run_id)- workflow_id = f"wf_{uuid.uuid4().hex[:12]}"-- async with make_split_workers(temporal_client, q_router=q_router, q_runner=q_runner):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=execution_job,- id=workflow_id,- task_queue=q_router,- )- rehydrate_pipe_output(pipe_output)- return execution_run_id-- async def test_cross_child_usage_aggregates_to_single_parent_total(- self,- live_parallel_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- split_queues: tuple[str, str],- route_llm_text_to_runner_queue: None, # noqa: ARG002- ) -> None:- """Two branch child workflows + the parent's summarize step each emit usage via the runner- fallback into the shared ``pipeline_run_id`` partition. Reading the partition once aggregates- usage across all of them into a single priced total — proving the cross-child fan-out join.- """- run_id = await self._execute_split(live_parallel_tracing_job, temporal_client, split_queues)-- # Multiple NDJSON files confirm the branches ran as separate child workflows (parent + children).- ndjson_files = ndjson_files_for_run(str(tracing_tmp_dir), run_id)- assert len(ndjson_files) >= 2, f"Expected parent + child-workflow NDJSON files, got {[path.name for path in ndjson_files]}"-- reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))- try:- events = reader.read_events(run_id)- finally:- reader.close()-- usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]- assert usage_events, "Expected usage events from the fan-out run"-- # The branches are distinct nodes living in distinct child workflows; their usage must land in- # the shared partition under more than one workflow_id.- distinct_workflow_ids = {evt.workflow_id for evt in usage_events}- distinct_node_ids = {evt.node_id for evt in usage_events}- assert len(distinct_workflow_ids) >= 2, f"Expected usage across >=2 workflows (cross-child), got {distinct_workflow_ids}"- assert len(distinct_node_ids) >= 2, f"Expected usage from >=2 distinct nodes, got {distinct_node_ids}"-- tokens_usages = UsageAggregator.aggregate(events)- expected_total = sum(- tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT, 0) + tokens_usage.nb_tokens_by_category.get(TokenCategory.OUTPUT, 0)- for tokens_usage in tokens_usages- )-- aggregated = CostRegistry.aggregate_costs(tokens_usages=tokens_usages)- assert aggregated.total_nb_tokens == expected_total- assert aggregated.total_nb_tokens >= 2 * len(usage_events), "Each runner emit carries at least one input + one output token"- assert aggregated.has_reportable_usage is Truediff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_extract_pages.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_extract_pages.pydeleted file mode 100644index 560b3774f..000000000--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_extract_pages.py+++ /dev/null@@ -1,195 +0,0 @@-"""Cross-process regression guard for ``ContentGeneratorInWorkflow.make_extract_pages``.--The two-activity branch of ``make_extract_pages`` is the only-``ContentGeneratorProtocol`` method that dispatches more than one activity in-a single call (``act_extract_gen_extract_pages`` then conditionally-``act_render_page_views``), and therefore the only one whose ``activity_id``-uniqueness mitigation (``f"{base_id}-pages"`` and-``f"{base_id}-render-page-views"``) materially affects production. This suite-mirrors ``content_generation/test_tprl_content_generator_pdf_page_views.py``-but pins the activity_id contract via ``WorkflowHandle.fetch_history()``.--Substitute activities return canonical fixtures so the test does not require-Azure Document Intelligence credentials or the real PDF renderer — the goal-is to validate the cross-process activity dispatch and the activity_id-contract, not the OCR backend itself (already covered by the in-process test).--Note on "split worker": single-worker setup is sufficient for the activity-dispatch contract — Temporal still serializes/deserializes activity payloads-across the activity boundary even when the same Python process hosts both-the workflow and the activities. A true cross-process upgrade is now-possible via ``worker_config.activity_queues`` (per-activity routing shipped-in v1; see ``wip/temporal-primitives/per-activity-queue-routing-v1.md``-§"Tests to upgrade when v1 lands"): route the extract activities to a-runner queue and register the substitutes only on that queue. Tracked as a-follow-up — this file is the primary caller of that upgrade.-"""--import uuid-from datetime import timedelta--import pytest-from temporalio import activity-from temporalio.client import Client as TemporalClient--from pipelex.cogt.content_generation.assignment_models import ExtractAssignment, RenderPageViewsAssignment-from pipelex.config import get_config-from pipelex.core.stuffs.image_content import ImageContent-from pipelex.core.stuffs.page_content import PageContent-from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent-from pipelex.temporal.config_temporal import QueueOptions-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.test_extras.temporal_test_tasks import TEMPORAL_TEST_WORKFLOWS-from pipelex.temporal.test_extras.wf_test_content_generator_pdf_page_views import WfTestContentGeneratorPdfPageViews-from pipelex.temporal.tprl_content_generation.act_extract_generate import act_extract_gen_extract_pages-from pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views-from tests.integration.pipelex.temporal.tracing.helpers import route_activities_to--_PAGE_VIEW_URLS = ("test://page-view-0.png", "test://page-view-1.png")---def _make_canonical_page_contents() -> list[PageContent]:- return [PageContent(text_and_images=TextAndImagesContent(text=None, images=None)) for _ in _PAGE_VIEW_URLS]---def _make_canonical_page_views() -> list[ImageContent]:- return [ImageContent(url=url) for url in _PAGE_VIEW_URLS]--[email protected](name="act_extract_gen_extract_pages")-async def _stub_act_extract_gen_extract_pages(_extract_assignment: ExtractAssignment) -> list[PageContent]: # noqa: RUF029- """Substitute returning two canonical pages so the in-workflow attachment- loop has a deterministic length to pair with ``act_render_page_views``.- """- return _make_canonical_page_contents()--[email protected](name="act_render_page_views")-async def _stub_act_render_page_views(_render_assignment: RenderPageViewsAssignment) -> list[ImageContent]: # noqa: RUF029- return _make_canonical_page_views()--[email protected](loop_scope="class")-class TestSplitWorkerExtractPages:- """Submit ``WfTestContentGeneratorPdfPageViews`` and pin the activity_id- contract via workflow history.-- The fixture workflow exercises the two-activity branch- (``document_uri`` + ``should_include_page_views=True``). Substitutes return- canonical 2-page outputs so the in-workflow attachment loop succeeds; the- test then inspects history to confirm Temporal scheduled both activities- (filtered by activity type) with distinct SDK-assigned activity_ids.- """-- @pytest.mark.timeout(60)- async def test_two_activity_branch_dispatches_with_distinct_activity_ids(- self,- temporal_client: TemporalClient,- ) -> None:- """Both ``act_extract_gen_extract_pages`` and ``act_render_page_views``- must appear in workflow history with distinct activity_ids. Pipelex- does not customize ``activity_id``; the Temporal SDK assigns- deterministic sequential integers per workflow run, which is sufficient- to guarantee per-``(workflow_id, run_id)`` uniqueness.- """- task_queue = f"q_extract_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_extract_{uuid.uuid4().hex[:8]}"-- # Route the substituted extract activities to this test's UUID queue- # so the in-workflow dispatcher (which now passes ``task_queue=resolved``- # for every activity) lands them on the worker registered here.- with route_activities_to(task_queue, [act_extract_gen_extract_pages.__name__, act_render_page_views.__name__]):- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- test_workflows=TEMPORAL_TEST_WORKFLOWS,- substitute_activities={- act_extract_gen_extract_pages: _stub_act_extract_gen_extract_pages,- act_render_page_views: _stub_act_render_page_views,- },- ):- handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfTestContentGeneratorPdfPageViews.run,- arg=False,- id=workflow_id,- task_queue=task_queue,- )- await handle.result()- history = await handle.fetch_history()-- # Filter by activity_type and confirm both activities scheduled with- # distinct activity_ids. The exact id strings are SDK-assigned integers- # (`"1"`, `"2"`, ...) determined by history position — we assert- # uniqueness, not the specific values, so the test stays robust if the- # SDK ever reformats them.- extract_activity_names = {"act_extract_gen_extract_pages", "act_render_page_views"}- scheduled = [- (event.activity_task_scheduled_event_attributes.activity_type.name, event.activity_task_scheduled_event_attributes.activity_id)- for event in history.events- if event.activity_task_scheduled_event_attributes.activity_type.name in extract_activity_names- ]- scheduled_types = [activity_type for activity_type, _ in scheduled]- scheduled_ids = [activity_id for _, activity_id in scheduled]- assert scheduled_types == ["act_extract_gen_extract_pages", "act_render_page_views"], (- f"Expected the extract + render activities to be scheduled in order; got {scheduled!r}"- )- assert len(set(scheduled_ids)) == 2, f"Expected distinct activity_ids for the two activities; got {scheduled!r}"-- @pytest.mark.timeout(60)- async def test_queue_options_start_to_close_timeout_flows_to_dispatch(- self,- temporal_client: TemporalClient,- ) -> None:- """Setting ``queue_options[X].start_to_close_timeout`` must make every- ``workflow.execute_activity`` call on queue ``X`` use that timeout- instead of the worker_config baseline.-- Submits the same two-activity workflow, but registers a per-queue- timeout (``0:07:00``) for the test's UUID queue and verifies the- scheduled-event ``start_to_close_timeout`` in workflow history matches- — proving the submitter-side resolver overlays correctly end-to-end.- """- task_queue = f"q_extract_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_extract_{uuid.uuid4().hex[:8]}"- expected_timeout = timedelta(minutes=7)-- queue_options = get_config().temporal.queue_options- assert task_queue not in queue_options, "Test task_queue collision — UUID should be unique."- queue_options[task_queue] = QueueOptions(start_to_close_timeout=expected_timeout)- try:- with route_activities_to(task_queue, [act_extract_gen_extract_pages.__name__, act_render_page_views.__name__]):- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- test_workflows=TEMPORAL_TEST_WORKFLOWS,- substitute_activities={- act_extract_gen_extract_pages: _stub_act_extract_gen_extract_pages,- act_render_page_views: _stub_act_render_page_views,- },- ):- handle = await temporal_client.start_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfTestContentGeneratorPdfPageViews.run,- arg=False,- id=workflow_id,- task_queue=task_queue,- )- await handle.result()- history = await handle.fetch_history()- finally:- queue_options.pop(task_queue, None)-- extract_activity_names = {"act_extract_gen_extract_pages", "act_render_page_views"}- scheduled_timeouts = [- event.activity_task_scheduled_event_attributes.start_to_close_timeout.ToTimedelta()- for event in history.events- if event.activity_task_scheduled_event_attributes.activity_type.name in extract_activity_names- ]- assert scheduled_timeouts, "No extract activity scheduled events found in history."- for observed_timeout in scheduled_timeouts:- assert observed_timeout == expected_timeout, (- f"start_to_close_timeout from queue_options[{task_queue}] not observed in dispatch: "- f"expected {expected_timeout!r}, got {observed_timeout!r}"- )diff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_object_gen.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_object_gen.pydeleted file mode 100644index 7c366acc3..000000000--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_object_gen.py+++ /dev/null@@ -1,133 +0,0 @@-"""Cross-process round-trip tests for ``act_llm_gen_object`` and-``act_llm_gen_object_list``.--Every existing ``library_crate/*.mthds`` bundle outputs ``Text``, so the-structured-output activities never go through Temporal's data converter-across the activity boundary. This suite closes that gap: it submits the-``WfTestStructuredOutputCrossProcess`` fixture workflow and substitutes-``act_llm_gen_object*`` with activities returning canonical ``FixtureInvoice``-instances. The fixture workflow asserts that nested ``FixtureCustomer`` and-``FixtureLineItem`` fields survived ``model_dump(mode="json", serialize_as_any=True)``-→ data converter → ``model_validate(...)``.--Pre-existing asymmetry pinned by these tests: ``make_object`` calls-``model_dump(mode="json", ...)`` while the legacy ``ContentGeneratorChild.make_object_list``-historically omitted ``mode="json"``. The in-workflow generator uses-``mode="json"`` on both, and these tests guard that contract through the-real Temporal data converter (kajson encoder/decoder).--Note on "split worker": single-worker setup is sufficient for the data-converter round-trip — Temporal still serializes/deserializes the activity-return value across the activity boundary even when the same Python process-hosts both the workflow and the activity. A true cross-process upgrade is-now possible via ``worker_config.activity_queues`` (per-activity routing-shipped in v1; see ``wip/temporal-primitives/per-activity-queue-routing-v1.md``-§"Tests to upgrade when v1 lands"): route ``act_llm_gen_object*`` to a-runner queue and register the substitutes only on that queue. Tracked as a-follow-up — this file is the primary caller of that upgrade.-"""--import uuid--import pytest-from temporalio import activity-from temporalio.client import Client as TemporalClient--from pipelex.cogt.content_generation.assignment_models import ObjectAssignment-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.test_extras.temporal_registry_test_models import FixtureCustomer, FixtureInvoice, FixtureLineItem-from pipelex.temporal.test_extras.temporal_test_tasks import TEMPORAL_TEST_WORKFLOWS-from pipelex.temporal.test_extras.wf_test_structured_output_cross_process import WfTestStructuredOutputCrossProcess-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_object, act_llm_gen_object_list-from tests.integration.pipelex.temporal.tracing.helpers import route_activities_to---def _make_canonical_invoice() -> FixtureInvoice:- """Canonical fixture matched by ``_assert_invoice_round_trip`` in the workflow."""- return FixtureInvoice(- invoice_number="INV-RT-001",- customer=FixtureCustomer(name="Alice Chen", email="[email protected]"),- line_items=[- FixtureLineItem(product_name="Widget", quantity=3, unit_price=9.99),- FixtureLineItem(product_name="Gadget", quantity=1, unit_price=49.50),- ],- total_amount=79.47,- )--[email protected](name="act_llm_gen_object")-async def _stub_act_llm_gen_object(_object_assignment: ObjectAssignment) -> FixtureInvoice: # noqa: RUF029- """Substitute that returns a canonical ``FixtureInvoice``.-- Bypasses the real LLM call so the test focuses on the activity-boundary- serialization. Annotated as ``act_llm_gen_object`` so the worker uses- this implementation in place of the registered one.- """- return _make_canonical_invoice()--[email protected](name="act_llm_gen_object_list")-async def _stub_act_llm_gen_object_list(_object_assignment: ObjectAssignment) -> list[FixtureInvoice]: # noqa: RUF029- return [- FixtureInvoice(- invoice_number="INV-RT-A",- customer=FixtureCustomer(name="Alice", email="[email protected]"),- line_items=[FixtureLineItem(product_name="Alpha", quantity=1, unit_price=1.0)],- total_amount=1.0,- ),- FixtureInvoice(- invoice_number="INV-RT-B",- customer=FixtureCustomer(name="Bob", email="[email protected]"),- line_items=[FixtureLineItem(product_name="Beta", quantity=2, unit_price=2.0)],- total_amount=4.0,- ),- ]--[email protected](loop_scope="class")-class TestSplitWorkerObjectGen:- """Round-trip nested structured output across the Temporal data converter.-- Workflow completion = round-trip preserved nested fields (the fixture- workflow's assertion helpers validate ``FixtureCustomer`` and- ``FixtureLineItem`` value preservation). A failure surfaces as a- ``WorkflowFailureError`` from ``execute_workflow``.- """-- @pytest.mark.parametrize(- "is_list",- [- pytest.param(False, id="single"),- pytest.param(True, id="list"),- ],- )- @pytest.mark.timeout(60)- async def test_object_gen_round_trip_preserves_nested_fields(- self,- temporal_client: TemporalClient,- is_list: bool,- ) -> None:- task_queue = f"q_obj_{uuid.uuid4().hex[:8]}"- workflow_id = f"wf_obj_{'list' if is_list else 'single'}_{uuid.uuid4().hex[:8]}"-- # Route the substituted object-gen activities to this test's UUID queue- # so the in-workflow dispatcher (which now passes ``task_queue=resolved``- # for every activity) lands them on the worker registered here.- with route_activities_to(task_queue, [act_llm_gen_object.__name__, act_llm_gen_object_list.__name__]):- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- test_workflows=TEMPORAL_TEST_WORKFLOWS,- substitute_activities={- act_llm_gen_object: _stub_act_llm_gen_object,- act_llm_gen_object_list: _stub_act_llm_gen_object_list,- },- ):- await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfTestStructuredOutputCrossProcess.run,- arg=is_list,- id=workflow_id,- task_queue=task_queue,- )diff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.pydeleted file mode 100644index cfbb43489..000000000--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py+++ /dev/null@@ -1,142 +0,0 @@-"""Gated real-inference cost test for split-worker (distributed) mode.--Every other split-worker usage test substitutes ``act_llm_gen_text`` with a-helper that hand-builds usage, so none of them prove that a *real* provider-response is captured, emitted through the runner fallback, and aggregated into a-non-zero cost total. Real distributed cost is otherwise validated only by manual-eyeballing. This test closes that blind spot end-to-end.--It is marked ``inference``/``llm`` so the default (no-spend) lanes skip it; run-it explicitly when validating real-inference cost capture in distributed mode.-The runner activity runs the genuine ``llm_gen_text`` — the real LLM call happens-on the runner queue and its usage report takes the per-process activity event log-unconditionally (in-activity emissions never touch a workflow's registered-context), just as a physically separate runner process would.-"""--import uuid-from collections.abc import Generator-from pathlib import Path-from typing import TYPE_CHECKING--import pytest-from temporalio import activity-from temporalio.client import Client as TemporalClient--from pipelex.cogt.content_generation.assignment_models import LLMAssignment-from pipelex.cogt.content_generation.llm_generate import llm_gen_text-from pipelex.cogt.usage.cost_registry import CostRegistry-from pipelex.cogt.usage.token_category import TokenCategory-from pipelex.config import get_config-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.config_temporal import ActivityRouteConfig-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.tracing.ndjson_event_log import NdjsonEventLog-from pipelex.tracing.trace_events import UsageReportEvent-from pipelex.tracing.usage_aggregator import UsageAggregator-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.tracing.helpers import inject_trace_context, make_split_workers-from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--[email protected](name="act_llm_gen_text")-async def _real_runner_act_llm_gen_text(llm_assignment: LLMAssignment) -> str:- """Real-inference substitute: run the genuine LLM call from the activity.-- The synchronous usage report takes the per-process activity event log- unconditionally — ``_emit_usage_event`` checks ``_is_in_temporal_activity()``- before any context lookup, so the router's registered context (which lives in- the same process during a single-process test) is never touched; ``llm_gen_text``- performs the real provider call so the captured token counts are real.- """- return await llm_gen_text(llm_assignment=llm_assignment)--[email protected](loop_scope="class")-class TestSplitWorkerRealInferenceCost:- """Real provider usage is captured, emitted via the runner fallback, and aggregated."""-- @pytest.fixture- def split_queues(self) -> tuple[str, str]:- return (f"q_router_{uuid.uuid4().hex[:8]}", f"q_runner_{uuid.uuid4().hex[:8]}")-- @pytest.fixture- def live_sequence_tracing_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- yield from pipe_job_from_bundle(- bundle_file=SequenceTracingTestData.BUNDLE_FILE,- pipe_code=SequenceTracingTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- isolated_registry=is_class_registry_isolated,- )-- @pytest.fixture- def route_llm_text_to_runner_queue(self, split_queues: tuple[str, str]) -> Generator[None, None, None]:- _q_router, q_runner = split_queues- worker_config = get_config().temporal.worker_config- activity_name = act_llm_gen_text.__name__- original_entry = worker_config.activity_queues.get(activity_name)- worker_config.activity_queues[activity_name] = ActivityRouteConfig(default=q_runner, by_handle={})- yield- if original_entry is None:- worker_config.activity_queues.pop(activity_name, None)- else:- worker_config.activity_queues[activity_name] = original_entry-- async def test_real_provider_usage_aggregates_to_nonzero_cost(- self,- live_sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- split_queues: tuple[str, str],- route_llm_text_to_runner_queue: None, # noqa: ARG002- ) -> None:- q_router, q_runner = split_queues- execution_run_id = f"split_real_{uuid.uuid4().hex[:12]}"- execution_job = inject_trace_context(live_sequence_tracing_job, execution_run_id)- workflow_id = f"wf_{uuid.uuid4().hex[:12]}"-- async with make_split_workers(- temporal_client,- q_router=q_router,- q_runner=q_runner,- runner_act_llm_gen_text=_real_runner_act_llm_gen_text,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=execution_job,- id=workflow_id,- task_queue=q_router,- )- rehydrate_pipe_output(pipe_output)-- reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))- try:- events = reader.read_events(execution_run_id)- finally:- reader.close()-- usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]- # The sequence bundle makes one real LLM call per step (step_one → step_two).- assert len(usage_events) == 2, f"Expected one usage event per inference call, got {len(usage_events)}"- assert all(evt.writer_id.startswith("act_") for evt in usage_events), "Runner fallback must have stamped act_* writer ids"-- tokens_usages = UsageAggregator.aggregate(events)- assert all(- tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT, 0) > 0 and tokens_usage.nb_tokens_by_category.get(TokenCategory.OUTPUT, 0) > 0- for tokens_usage in tokens_usages- ), "Real provider responses must carry non-zero input and output token counts"- model_names = {tokens_usage.inference_model_name for tokens_usage in tokens_usages}- assert all(model_names), "Every usage record must name its real model handle"-- aggregated = CostRegistry.aggregate_costs(tokens_usages=tokens_usages)- assert aggregated.total_nb_tokens > 0- assert aggregated.has_reportable_usage is Truediff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.pydeleted file mode 100644index 534e91b73..000000000--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py+++ /dev/null@@ -1,259 +0,0 @@-"""Integration tests for cross-worker UsageReportEvent emission via two task queues.--Validates the Phase 2 runner-side fallback: when `WfPipeRouter` dispatches-`act_llm_gen_text` to a separate task queue (the production deployment topology-where router and runner are physically separated), the runner-side worker —-with no `_event_log_contexts` entry registered for the workflow — emits its-`UsageReportEvent` into the same `traces_dir` partition via the per-process-activity event log, stamped with a stable `act_*` `writer_id`.--`ContentGeneratorInWorkflow.make_llm_text` calls-`workflow.execute_activity(act_llm_gen_text, ..., task_queue=worker_config.resolve_queue(...))`-directly from inside the workflow — there is no child workflow layer-between the router and the activity. The split-worker topology is configured-by overriding `worker_config.activity_queues[act_llm_gen_text].default` to-point at the runner queue.-"""--import uuid-from collections import defaultdict-from collections.abc import Generator-from pathlib import Path-from typing import TYPE_CHECKING--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.cogt.usage.cost_registry import CostRegistry-from pipelex.cogt.usage.token_category import TokenCategory-from pipelex.config import get_config-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.config_temporal import ActivityRouteConfig-from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text-from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter-from pipelex.tracing.ndjson_event_log import NdjsonEventLog-from pipelex.tracing.trace_events import PipeStartEvent, UsageReportEvent-from pipelex.tracing.usage_aggregator import UsageAggregator-from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.tracing.helpers import (- inject_trace_context,- make_split_workers,- ndjson_files_for_run,-)-from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_output import PipeOutput--[email protected](loop_scope="class")-class TestSplitWorkerUsageEmission:- """Cross-process emission: router on q_router, runner on q_runner.-- Each test uses its own pair of task queues (UUID-named) so concurrent- executions don't share queue state. The per-process activity event log- cache is reset around every test by the temporal conftest's autouse- ``reset_activity_event_log_cache`` fixture.- """-- @pytest.fixture- def split_queues(self) -> tuple[str, str]:- """A fresh (q_router, q_runner) pair per test."""- return (f"q_router_{uuid.uuid4().hex[:8]}", f"q_runner_{uuid.uuid4().hex[:8]}")-- @pytest.fixture- def live_sequence_tracing_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:- """Like the class-scoped `sequence_tracing_job` but in LIVE mode.-- Since Part B, DRY also dispatches `act_llm_gen_text` (the leaf mocks- inside the activity) — but its synthetic usage is zero-token and- suppressed by design, so the *reportable*-usage assertions here need- LIVE mode, where the fake activity substitute synthesizes non-zero- usage on the runner side.- The fake `act_llm_gen_text` substitute installed by- `make_split_workers` returns a stub string and synthesizes the- usage report, so no real LLM call is made.- """- yield from pipe_job_from_bundle(- bundle_file=SequenceTracingTestData.BUNDLE_FILE,- pipe_code=SequenceTracingTestData.PIPE_CODE,- pipe_run_mode=PipeRunMode.LIVE,- isolated_registry=is_class_registry_isolated,- )-- @pytest.fixture- def route_llm_text_to_runner_queue(self, split_queues: tuple[str, str]) -> Generator[None, None, None]:- """Route `act_llm_gen_text` to `q_runner` for the duration of the test.-- Without this override, `ContentGeneratorInWorkflow.make_llm_text` would- dispatch the activity to whichever task queue the workflow is running on- (the router queue), defeating the split-worker topology this suite is- supposed to exercise.- """- _q_router, q_runner = split_queues- worker_config = get_config().temporal.worker_config- activity_name = act_llm_gen_text.__name__- original_entry = worker_config.activity_queues.get(activity_name)- worker_config.activity_queues[activity_name] = ActivityRouteConfig(default=q_runner, by_handle={})- yield- if original_entry is None:- worker_config.activity_queues.pop(activity_name, None)- else:- worker_config.activity_queues[activity_name] = original_entry-- async def _execute_split(- self,- live_sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- split_queues: tuple[str, str],- ) -> str:- """Run the workflow on q_router; activities dispatch to q_runner."""- q_router, q_runner = split_queues- execution_run_id = f"split_exec_{uuid.uuid4().hex[:12]}"- execution_job = inject_trace_context(live_sequence_tracing_job, execution_run_id)- workflow_id = f"wf_{uuid.uuid4().hex[:12]}"-- async with make_split_workers(temporal_client, q_router=q_router, q_runner=q_runner):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRouter.run,- arg=execution_job,- id=workflow_id,- task_queue=q_router,- )- rehydrate_pipe_output(pipe_output)- return execution_run_id-- async def test_runner_usage_event_lands_in_same_ndjson_dir(- self,- live_sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- split_queues: tuple[str, str],- route_llm_text_to_runner_queue: None, # noqa: ARG002- ) -> None:- """Router on q_router dispatches inference activity to q_runner.-- The runner — with no `_event_log_contexts` entry — emits its- `UsageReportEvent` into the same `traces_dir` partition via the- Phase-2 fallback. Both the router-written `wf_{wf_id}.ndjson`- (from `act_flush_trace_events`) and at least one runner-written- `wf_{wf_id}__w_act_*.ndjson` are present.- """- run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)-- ndjson_files = ndjson_files_for_run(str(tracing_tmp_dir), run_id)- primary_files = [path for path in ndjson_files if "__w_" not in path.name]- runner_files = [path for path in ndjson_files if "__w_act_" in path.name]- assert primary_files, f"Expected a primary wf_*.ndjson router file in {[path.name for path in ndjson_files]}"- assert runner_files, f"Expected at least one wf_*__w_act_*.ndjson runner file in {[path.name for path in ndjson_files]}"-- reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))- try:- events = reader.read_events(run_id)- finally:- reader.close()-- primary_pipe_starts = [evt for evt in events if isinstance(evt, PipeStartEvent) and evt.writer_id == "primary"]- runner_usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent) and evt.writer_id.startswith("act_")]- assert primary_pipe_starts, "Expected at least one PipeStartEvent with writer_id='primary' from the router"- assert runner_usage_events, "Expected at least one UsageReportEvent with writer_id='act_*' from the runner"-- async def test_no_double_emit_in_split_worker_pool(- self,- live_sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- split_queues: tuple[str, str],- route_llm_text_to_runner_queue: None, # noqa: ARG002- ) -> None:- """Exactly-once invariant: `_emit_usage_event` takes either the fast- path OR the fallback path, never both. Submit one workflow; assert- exactly one `UsageReportEvent` per `(node_id, workflow_id)` across- all NDJSON files.- """- run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)-- reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))- try:- events = reader.read_events(run_id)- finally:- reader.close()-- usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]- assert usage_events, "Expected at least one UsageReportEvent across the run"-- counts: dict[tuple[str, str], int] = defaultdict(int)- for usage_event in usage_events:- counts[usage_event.node_id, usage_event.workflow_id] += 1-- for key, count in counts.items():- assert count == 1, f"Duplicate UsageReportEvent for (node_id, workflow_id)={key}: count={count}"-- async def test_runner_usage_aggregates_to_tokens_usages(- self,- live_sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- split_queues: tuple[str, str],- route_llm_text_to_runner_queue: None, # noqa: ARG002- ) -> None:- """The cross-worker usage events feed the production ``UsageAggregator`` (the same path that- rides ``tokens_usages`` back on PipeOutput): aggregating the read-back stream yields one usage- record per emitted ``UsageReportEvent``, none dropped.- """- run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)-- reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))- try:- events = reader.read_events(run_id)- finally:- reader.close()-- usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]- tokens_usages = UsageAggregator.aggregate(events)- assert tokens_usages, "Expected the aggregator to surface at least one token-usage record"- assert len(tokens_usages) == len(usage_events)- assert tokens_usages == [evt.tokens_usage for evt in usage_events]-- async def test_runner_usage_sums_to_expected_cost_total(- self,- live_sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- split_queues: tuple[str, str],- route_llm_text_to_runner_queue: None, # noqa: ARG002- ) -> None:- """The cross-worker usage events feed the production ``CostRegistry`` aggregation and price- into a single run total. This pins the *summed total* in split-worker mode — the previous- tests assert only landing / per-node de-dup / 1:1 passthrough, never that the numbers add up.-- ``total_nb_tokens`` is computed independently from each runner-emitted event's raw token- counts and must equal what ``aggregate_costs`` reports. The fake runner model is priced at- zero, so ``total_cost`` is 0 while the run still counts as reportable (tokens > 0) — the same- free-model path a real zero-cost model would take.- """- run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)-- reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))- try:- events = reader.read_events(run_id)- finally:- reader.close()-- usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]- assert usage_events, "Expected at least one UsageReportEvent across the run"-- tokens_usages = UsageAggregator.aggregate(events)- expected_total = sum(- tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT, 0) + tokens_usage.nb_tokens_by_category.get(TokenCategory.OUTPUT, 0)- for tokens_usage in tokens_usages- )-- aggregated = CostRegistry.aggregate_costs(tokens_usages=tokens_usages)- assert aggregated.total_nb_tokens == expected_total- assert expected_total >= len(usage_events), "Each runner emit carries at least one input + one output token"- assert aggregated.total_cost == 0.0- assert aggregated.has_reportable_usage is Truediff --git a/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_batch.py b/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_batch.pydeleted file mode 100644index c5d2dc0aa..000000000--- a/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_batch.py+++ /dev/null@@ -1,91 +0,0 @@-"""Integration tests for PipeBatch graph tracing through Temporal workflows.--Validates that PipeBatch controllers running as fan-out child workflows-produce BATCH_ITEM and BATCH_AGGREGATE edges in the assembled GraphSpec.-"""--from pathlib import Path--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.graph.graphspec import EdgeKind, GraphSpec, NodeStatus-from pipelex.pipe_run.pipe_job import PipeJob-from tests.integration.pipelex.temporal.tracing.helpers import (- TracingResult,- assert_all_nodes_terminal,- edges_by_kind,- execute_and_assemble,-)-from tests.integration.pipelex.temporal.tracing.test_data import BatchTracingTestData--[email protected](loop_scope="class")-class TestWfGraphTracingBatch:- """PipeBatch produces BATCH_ITEM and BATCH_AGGREGATE edges in GraphSpec via Temporal tracing."""-- async def _get_result(- self,- batch_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ) -> tuple[TracingResult, GraphSpec]:- """Execute workflow and return result with assembled GraphSpec, asserting it's not None."""- result = await execute_and_assemble(- pipe_job=batch_tracing_job,- temporal_client=temporal_client,- traces_dir=str(tracing_tmp_dir),- )- assert result.graph_spec is not None, "GraphSpec should be assembled from trace events"- return result, result.graph_spec-- async def test_batch_all_nodes_succeeded(- self,- batch_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """All nodes in the batch graph are SUCCEEDED."""- _result, graph_spec = await self._get_result(batch_tracing_job, temporal_client, tracing_tmp_dir)- for node in graph_spec.nodes:- assert node.status == NodeStatus.SUCCEEDED, f"Node '{node.pipe_code}' should be SUCCEEDED, got {node.status}"- assert_all_nodes_terminal(graph_spec)-- async def test_batch_node_count(- self,- batch_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """Graph has at least the minimum expected number of nodes."""- _result, graph_spec = await self._get_result(batch_tracing_job, temporal_client, tracing_tmp_dir)- assert len(graph_spec.nodes) >= BatchTracingTestData.MIN_NODE_COUNT-- async def test_batch_item_edges(- self,- batch_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """BATCH_ITEM edges present for fan-out from list to item processors."""- _result, graph_spec = await self._get_result(batch_tracing_job, temporal_client, tracing_tmp_dir)- edge_groups = edges_by_kind(graph_spec)- item_edges = edge_groups.get(EdgeKind.BATCH_ITEM, [])- assert len(item_edges) >= BatchTracingTestData.MIN_BATCH_ITEM_EDGES, (- f"Expected at least {BatchTracingTestData.MIN_BATCH_ITEM_EDGES} BATCH_ITEM edges, got {len(item_edges)}"- )-- async def test_batch_aggregate_edges(- self,- batch_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """BATCH_AGGREGATE edges present for fan-in from item processors back to controller."""- _result, graph_spec = await self._get_result(batch_tracing_job, temporal_client, tracing_tmp_dir)- edge_groups = edges_by_kind(graph_spec)- aggregate_edges = edge_groups.get(EdgeKind.BATCH_AGGREGATE, [])- assert len(aggregate_edges) >= BatchTracingTestData.MIN_BATCH_AGGREGATE_EDGES, (- f"Expected at least {BatchTracingTestData.MIN_BATCH_AGGREGATE_EDGES} BATCH_AGGREGATE edges, got {len(aggregate_edges)}"- )diff --git a/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_parallel.py b/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_parallel.pydeleted file mode 100644index 0f2a770a8..000000000--- a/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_parallel.py+++ /dev/null@@ -1,118 +0,0 @@-"""Integration tests for PipeParallel graph tracing through Temporal workflows.--Validates that PipeParallel controllers running as concurrent child workflows-produce correct graph structure in the assembled GraphSpec.-"""--from pathlib import Path--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.graph.graphspec import EdgeKind, GraphSpec, NodeStatus-from pipelex.pipe_run.pipe_job import PipeJob-from tests.integration.pipelex.temporal.tracing.helpers import (- TracingResult,- assert_all_nodes_terminal,- edges_by_kind,- execute_and_assemble,- ndjson_files_for_run,- node_pipe_codes,-)-from tests.integration.pipelex.temporal.tracing.test_data import ParallelTracingTestData--[email protected](loop_scope="class")-class TestWfGraphTracingParallel:- """PipeParallel produces correct graph structure via Temporal tracing."""-- async def _get_result(- self,- parallel_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ) -> tuple[TracingResult, GraphSpec]:- """Execute workflow and return result with assembled GraphSpec, asserting it's not None."""- result = await execute_and_assemble(- pipe_job=parallel_tracing_job,- temporal_client=temporal_client,- traces_dir=str(tracing_tmp_dir),- )- assert result.graph_spec is not None, "GraphSpec should be assembled from trace events"- return result, result.graph_spec-- async def test_parallel_produces_multiple_ndjson_files(- self,- parallel_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """Parallel branches spawn child workflows, producing multiple NDJSON files."""- result = await execute_and_assemble(- pipe_job=parallel_tracing_job,- temporal_client=temporal_client,- traces_dir=str(tracing_tmp_dir),- )- ndjson_files = ndjson_files_for_run(str(tracing_tmp_dir), result.pipeline_run_id)- # Parent workflow + child workflows (at least 2 for the 2 branches)- assert len(ndjson_files) >= 2, f"Expected multiple NDJSON files (parent + child workflows), got {len(ndjson_files)}"-- async def test_parallel_all_nodes_succeeded(- self,- parallel_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """All nodes in the parallel graph are SUCCEEDED."""- _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)- for node in graph_spec.nodes:- assert node.status == NodeStatus.SUCCEEDED, f"Node '{node.pipe_code}' should be SUCCEEDED, got {node.status}"- assert_all_nodes_terminal(graph_spec)-- async def test_parallel_pipe_codes(- self,- parallel_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """All expected pipe_codes are present in the assembled graph."""- _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)- actual_codes = node_pipe_codes(graph_spec)- assert ParallelTracingTestData.EXPECTED_PIPE_CODES.issubset(actual_codes), (- f"Missing pipe_codes: {ParallelTracingTestData.EXPECTED_PIPE_CODES - actual_codes}"- )-- async def test_parallel_node_count(- self,- parallel_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """Graph has at least the minimum expected number of nodes."""- _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)- assert len(graph_spec.nodes) >= ParallelTracingTestData.MIN_NODE_COUNT-- async def test_parallel_contains_edges(- self,- parallel_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """CONTAINS edges from controllers to their children are present."""- _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)- edge_groups = edges_by_kind(graph_spec)- contains_edges = edge_groups.get(EdgeKind.CONTAINS, [])- assert len(contains_edges) >= 2, "At least 2 CONTAINS edges expected (branches inside parallel)"-- async def test_parallel_data_edges(- self,- parallel_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """DATA edges connect branch outputs to downstream consumers."""- _result, graph_spec = await self._get_result(parallel_tracing_job, temporal_client, tracing_tmp_dir)- edge_groups = edges_by_kind(graph_spec)- data_edges = edge_groups.get(EdgeKind.DATA, [])- assert len(data_edges) >= 1, "At least 1 DATA edge expected (branch output → summarize step)"diff --git a/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_sequence.py b/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_sequence.pydeleted file mode 100644index 7225b2ec0..000000000--- a/tests/integration/pipelex/temporal/tracing/test_wf_graph_tracing_sequence.py+++ /dev/null@@ -1,117 +0,0 @@-"""Integration tests for PipeSequence graph tracing through Temporal workflows.--Validates that a PipeSequence (2 PipeLLM steps) running on a Temporal worker-produces correct NDJSON trace events that assemble into a valid GraphSpec-with CONTAINS and DATA edges.-"""--from pathlib import Path--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.graph.graphspec import EdgeKind, GraphSpec, NodeStatus-from pipelex.pipe_run.pipe_job import PipeJob-from tests.integration.pipelex.temporal.tracing.helpers import (- TracingResult,- assert_all_nodes_terminal,- edges_by_kind,- execute_and_assemble,- ndjson_files_for_run,- node_pipe_codes,-)-from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData--[email protected](loop_scope="class")-class TestWfGraphTracingSequence:- """PipeSequence produces correct GraphSpec via NDJSON event tracing on Temporal."""-- async def _get_result(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ) -> tuple[TracingResult, GraphSpec]:- """Execute workflow and return result with assembled GraphSpec, asserting it's not None."""- result = await execute_and_assemble(- pipe_job=sequence_tracing_job,- temporal_client=temporal_client,- traces_dir=str(tracing_tmp_dir),- )- assert result.graph_spec is not None, "GraphSpec should be assembled from trace events"- return result, result.graph_spec-- async def test_sequence_produces_ndjson_files(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """Workflow execution writes NDJSON event files to the traces directory."""- result = await execute_and_assemble(- pipe_job=sequence_tracing_job,- temporal_client=temporal_client,- traces_dir=str(tracing_tmp_dir),- )- ndjson_files = ndjson_files_for_run(str(tracing_tmp_dir), result.pipeline_run_id)- assert len(ndjson_files) >= 1, "At least one NDJSON file should be written"-- async def test_sequence_assembles_valid_graph(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """Assembled GraphSpec has expected node count and matching graph_id."""- result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)- assert graph_spec.graph_id == result.pipeline_run_id- assert len(graph_spec.nodes) == SequenceTracingTestData.EXPECTED_NODE_COUNT-- async def test_sequence_all_nodes_succeeded(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """All nodes in the sequence graph have SUCCEEDED status."""- _result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)- for node in graph_spec.nodes:- assert node.status == NodeStatus.SUCCEEDED, f"Node '{node.pipe_code}' should be SUCCEEDED, got {node.status}"- assert_all_nodes_terminal(graph_spec)-- async def test_sequence_pipe_codes(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """All expected pipe_codes are present in the assembled graph nodes."""- _result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)- actual_codes = node_pipe_codes(graph_spec)- assert actual_codes == SequenceTracingTestData.EXPECTED_PIPE_CODES-- async def test_sequence_contains_edges(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """CONTAINS edges from sequence controller to each child step."""- _result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)- edge_groups = edges_by_kind(graph_spec)- contains_edges = edge_groups.get(EdgeKind.CONTAINS, [])- assert len(contains_edges) == SequenceTracingTestData.EXPECTED_CONTAINS_EDGE_COUNT-- async def test_sequence_data_edges(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- tracing_tmp_dir: Path,- ):- """DATA edges connect step outputs to step inputs."""- _result, graph_spec = await self._get_result(sequence_tracing_job, temporal_client, tracing_tmp_dir)- edge_groups = edges_by_kind(graph_spec)- data_edges = edge_groups.get(EdgeKind.DATA, [])- assert len(data_edges) >= SequenceTracingTestData.MIN_DATA_EDGESdiff --git a/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.py b/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.pydeleted file mode 100644index 50a966c09..000000000--- a/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.py+++ /dev/null@@ -1,124 +0,0 @@-"""F1 invariant for the Temporal ``WfPipeRun`` assembly path (Step 2).--Mirrors the DIRECT-mode coverage in-``tests/integration/pipelex/pipeline/test_direct_tracing_assembly.py`` for the-TEMPORAL arm. A full ``WfPipeRun`` run wraps the real ``WfPipeRouter`` as a child-and then runs Step 2's ``act_assemble_tracing``; the assembled artifacts must ride-back on ``pipe_output`` per the run's emit flags:--- **costs-only (``emit_graph_events=False, emit_usage_events=True``):** usage rides- back on ``tokens_usages`` but ``graph_spec`` stays None. This is the exact- DIRECT/TEMPORAL symmetry F1 guards — both the ``WfPipeRun`` Step-2 dispatch- (``assemble_graph=False`` → activity returns ``graph_spec=None``) and the- ``WfPipeRouter`` finally assignment must avoid producing a GraphSpec under- ``--no-graph``.-- **graph+costs:** both ``graph_spec`` and ``tokens_usages`` populated.-- **graph-only (``emit_usage_events=False``):** ``graph_spec`` populated,- ``tokens_usages`` stays None.--Runs fully dry (``PipeRunMode.DRY``, the default for ``sequence_tracing_job``) so no-inference happens; the dry content generator reports token usage inline, which is-what populates the usage stream that ``act_assemble_tracing`` aggregates.-"""--import uuid--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun-from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output-from tests.integration.pipelex.temporal.tracing.helpers import inject_trace_context--[email protected](loop_scope="class")-class TestWfPipeRunTracingAssembly:- async def _run(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- *,- emit_graph_events: bool,- emit_usage_events: bool,- ) -> PipeOutput:- """Run a full ``WfPipeRun`` (router child + Step-2 assembly) on a single worker."""- execution_run_id = f"wfrun_assembly_{uuid.uuid4().hex[:12]}"- execution_job = inject_trace_context(- sequence_tracing_job,- execution_run_id,- emit_graph_events=emit_graph_events,- emit_usage_events=emit_usage_events,- )- pipe_run_arg = PipeRunArg(pipe_job=execution_job).prepare_for_temporal()- task_queue = str(uuid.uuid4())- workflow_id = str(uuid.uuid4())-- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- ):- pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfPipeRun.run,- arg=pipe_run_arg,- id=workflow_id,- task_queue=task_queue,- )-- rehydrate_pipe_output(pipe_output, pipe_job=sequence_tracing_job)- return pipe_output-- async def test_costs_only_leaves_graph_spec_none(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """F1 (TEMPORAL): a costs-only run rides usage back but must leave graph_spec None."""- pipe_output = await self._run(- sequence_tracing_job,- temporal_client,- emit_graph_events=False,- emit_usage_events=True,- )-- assert pipe_output.graph_spec is None, "costs-only WfPipeRun must not produce a GraphSpec under --no-graph"- assert pipe_output.tokens_usages is not None- assert len(pipe_output.tokens_usages) >= 1-- async def test_graph_and_costs_populate_both(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """Positive control: with both concerns on, both artifacts ride back (so the costs-only assertion is not vacuous)."""- pipe_output = await self._run(- sequence_tracing_job,- temporal_client,- emit_graph_events=True,- emit_usage_events=True,- )-- assert pipe_output.graph_spec is not None- assert pipe_output.tokens_usages is not None- assert len(pipe_output.tokens_usages) >= 1-- async def test_graph_only_leaves_tokens_usages_none(- self,- sequence_tracing_job: PipeJob,- temporal_client: TemporalClient,- ) -> None:- """The mirror asymmetry: graph-only assembles a GraphSpec but never sets tokens_usages."""- pipe_output = await self._run(- sequence_tracing_job,- temporal_client,- emit_graph_events=True,- emit_usage_events=False,- )-- assert pipe_output.graph_spec is not None- assert pipe_output.tokens_usages is Nonediff --git a/tests/integration/pipelex/temporal/workflows/conftest.py b/tests/integration/pipelex/temporal/workflows/conftest.pydeleted file mode 100644index 88ce5103b..000000000--- a/tests/integration/pipelex/temporal/workflows/conftest.py+++ /dev/null@@ -1,9 +0,0 @@-import uuid--import pytest---def workflow_run_id() -> str:- """Provide a workflow run ID, mirroring the job_metadata fixture pattern."""- return str(uuid.uuid4())diff --git a/tests/integration/pipelex/temporal/workflows/test_wf_child_crafter.py b/tests/integration/pipelex/temporal/workflows/test_wf_child_crafter.pydeleted file mode 100644index 1a89e90e5..000000000--- a/tests/integration/pipelex/temporal/workflows/test_wf_child_crafter.py+++ /dev/null@@ -1,32 +0,0 @@-import uuid--import pytest-from temporalio.client import Client as TemporalClient--from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.temporal.temporal_hub import get_task_manager-from pipelex.temporal.test_extras.temporal_test_tasks import TEMPORAL_TEST_ACTIVITIES, TEMPORAL_TEST_WORKFLOWS-from pipelex.temporal.test_extras.wf_test_content_generator_child import WfTestContentGeneratorChild--[email protected]_runnable[email protected](loop_scope="class")-class TestWfChildCrafter:- async def test_wf_child_craft(self, temporal_client: TemporalClient, workflow_run_id: str, pipe_run_mode: PipeRunMode):- task_queue = str(uuid.uuid4())- async with get_task_manager().make_worker(- temporal_client,- task_queue=task_queue,- is_not_sandboxed=True,- test_workflows=TEMPORAL_TEST_WORKFLOWS,- test_activities=TEMPORAL_TEST_ACTIVITIES,- ):- await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]- workflow=WfTestContentGeneratorChild.run,- arg=pipe_run_mode.is_dry,- id=workflow_run_id,- task_queue=task_queue,- )diff --git a/tests/unit/pipelex/cli/test_cli_entrypoint_smoke.py b/tests/unit/pipelex/cli/test_cli_entrypoint_smoke.pyindex 65befddcc..85b955633 100644--- a/tests/unit/pipelex/cli/test_cli_entrypoint_smoke.py+++ b/tests/unit/pipelex/cli/test_cli_entrypoint_smoke.py@@ -4,7 +4,7 @@ Regression guard for the "no active click context" crash. Undertyper >= 0.26 / click >= 8.4 the group callback (`app_callback`) ran without anactive *global* click context, so it raised `RuntimeError: There is no activeclick context` and **every subcommand exited 1 before doing anything** — root-`--help` happened to survive, but `pipelex build --help`, `pipelex worker --help`,+`--help` happened to survive, but `pipelex build --help`, `pipelex validate --help`,etc. did not. A fresh `pip install pipelex` pulls those versions, so the shippedCLI was broken for end users while our `uv.lock` (older typer) masked it locally.@@ -59,7 +59,6 @@ class TestCliEntrypointStarts:["run", "--help"],["validate", "--help"],["init", "--help"],- ["worker", "--help"],],)def test_subcommand_starts_and_exits_zero(self, args: list[str]) -> None:diff --git a/tests/unit/pipelex/cli/test_setup_temporal_namespace_cmd.py b/tests/unit/pipelex/cli/test_setup_temporal_namespace_cmd.pydeleted file mode 100644index 879a3fcf2..000000000--- a/tests/unit/pipelex/cli/test_setup_temporal_namespace_cmd.py+++ /dev/null@@ -1,211 +0,0 @@-"""Unit tests for the ``pipelex setup-temporal-namespace`` CLI command.--The command wraps ``ensure_required_search_attributes_registered`` with a-copy-paste-ready dry-run output and a permission-denied fallback runbook so-operators whose worker API key lacks ``OperatorService.AddSearchAttributes``-permission still get the exact ``temporal`` / ``tcld`` invocation they need-the namespace admin to run on their behalf.-"""--from typing import Any--import pytest-import typer-from pytest_mock import MockerFixture--from pipelex.cli.commands.setup_temporal_namespace_cmd import setup_temporal_namespace_cmd-from pipelex.temporal.tprl.namespace_check import RegistrationFailure---class TestSetupTemporalNamespaceCmd:- @pytest.fixture- def patch_pipelex_lifecycle(self, mocker: MockerFixture) -> None:- """Skip the heavy ``make_pipelex_for_cli`` / ``teardown`` lifecycle.-- The CLI command's interaction with the Pipelex boot is not what this- test exercises — we care about the search-attribute registration logic- and the dry-run / permission-denied branches.- """- mocker.patch("pipelex.cli.commands.setup_temporal_namespace_cmd.make_pipelex_for_cli")- mocker.patch("pipelex.cli.commands.setup_temporal_namespace_cmd.Pipelex.teardown_if_needed")-- def _patch_config(- self,- mocker: MockerFixture,- *,- enabled: bool = True,- attributes: list[str] | None = None,- namespace: str = "default",- selected_server: str = "local",- ) -> Any:- """Build a fake config root and patch the module-level ``get_config``."""- config_root = mocker.MagicMock()- config_root.temporal.search_attributes.enabled = enabled- config_root.temporal.search_attributes.attributes = (- attributes if attributes is not None else ["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"]- )- config_root.temporal.temporal_config.selected_server = selected_server- server_config = mocker.MagicMock()- server_config.namespace = namespace- config_root.temporal.temporal_config.temporal_server_configs = {selected_server: server_config}- mocker.patch("pipelex.cli.commands.setup_temporal_namespace_cmd.get_config", return_value=config_root)- return config_root-- @pytest.mark.usefixtures("patch_pipelex_lifecycle")- def test_dry_run_prints_temporal_cli_command_without_connecting(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:- self._patch_config(mocker)- connect_mock = mocker.patch(- "pipelex.temporal.temporal_connect.connect_to_temporal_selected_server",- new=mocker.AsyncMock(),- )- register_mock = mocker.patch(- "pipelex.temporal.tprl.namespace_check.ensure_required_search_attributes_registered",- new=mocker.AsyncMock(),- )-- setup_temporal_namespace_cmd(dry_run=True, server=None)-- captured = capsys.readouterr().out- assert "temporal operator search-attribute create" in captured- assert "--namespace default" in captured- for name in ("PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"):- assert f"--name {name} --type Keyword" in captured- assert "tcld namespace search-attributes add" in captured- connect_mock.assert_not_called()- register_mock.assert_not_called()-- @pytest.mark.usefixtures("patch_pipelex_lifecycle")- def test_happy_path_connects_and_registers(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:- self._patch_config(mocker)- fake_client = mocker.MagicMock()- connect_mock = mocker.patch(- "pipelex.temporal.temporal_connect.connect_to_temporal_selected_server",- new=mocker.AsyncMock(return_value=fake_client),- )- # Helper now returns the tuple of newly-registered names; two of the five- # attributes were missing and have just been added.- register_mock = mocker.patch(- "pipelex.temporal.tprl.namespace_check.ensure_required_search_attributes_registered",- new=mocker.AsyncMock(return_value=("SessionId", "DomainCode")),- )-- setup_temporal_namespace_cmd(dry_run=False, server=None)-- connect_mock.assert_awaited_once()- register_mock.assert_awaited_once()- kwargs = register_mock.call_args.kwargs- assert kwargs["temporal_client"] is fake_client- assert kwargs["namespace"] == "default"- assert kwargs["configured_attributes"] == ["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"]- # Success message names the actual delta, not the configured set size.- captured_out = capsys.readouterr().out- assert "Registered 2 new" in captured_out- assert "SessionId" in captured_out- assert "DomainCode" in captured_out-- @pytest.mark.usefixtures("patch_pipelex_lifecycle")- def test_idempotent_no_op_reports_already_registered(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:- """When every configured attribute is already registered, the helper- returns an empty tuple. The CLI must report the idempotent outcome- clearly instead of falsely claiming it just registered the full set- (the misleading "Registered N" that operators saw before).- """- self._patch_config(mocker)- mocker.patch(- "pipelex.temporal.temporal_connect.connect_to_temporal_selected_server",- new=mocker.AsyncMock(return_value=mocker.MagicMock()),- )- mocker.patch(- "pipelex.temporal.tprl.namespace_check.ensure_required_search_attributes_registered",- new=mocker.AsyncMock(return_value=()),- )-- setup_temporal_namespace_cmd(dry_run=False, server=None)-- out = capsys.readouterr().out- assert "already registered" in out- assert "All 5" in out-- @pytest.mark.usefixtures("patch_pipelex_lifecycle")- def test_permission_denied_prints_fallback_runbook_and_exits(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:- self._patch_config(mocker)- failure = RegistrationFailure(- namespace="default",- missing=("SessionId", "UserId"),- rpc_error_message="permission denied",- )- mocker.patch(- "pipelex.temporal.temporal_connect.connect_to_temporal_selected_server",- new=mocker.AsyncMock(return_value=mocker.MagicMock()),- )- mocker.patch(- "pipelex.temporal.tprl.namespace_check.ensure_required_search_attributes_registered",- new=mocker.AsyncMock(return_value=failure),- )-- with pytest.raises(typer.Exit) as exc_info:- setup_temporal_namespace_cmd(dry_run=False, server=None)-- assert exc_info.value.exit_code == 1- err = capsys.readouterr().err- # The error stream carries the actionable runbook.- assert "Permission denied" in err- assert "--name SessionId --type Keyword" in err- assert "--name UserId --type Keyword" in err- assert "tcld namespace search-attributes add" in err- assert "permission denied" in err # the underlying rpc message is surfaced-- @pytest.mark.usefixtures("patch_pipelex_lifecycle")- def test_disabled_config_short_circuits_with_message(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:- self._patch_config(mocker, enabled=False, attributes=[])- connect_mock = mocker.patch(- "pipelex.temporal.temporal_connect.connect_to_temporal_selected_server",- new=mocker.AsyncMock(),- )-- setup_temporal_namespace_cmd(dry_run=False, server=None)-- connect_mock.assert_not_called()- assert "enabled = false" in capsys.readouterr().err-- @pytest.mark.usefixtures("patch_pipelex_lifecycle")- def test_rpc_error_other_than_permission_denied_is_framed_for_operator(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:- """When the namespace doesn't exist or the control plane is down, the- helper propagates a raw ``RPCError``. The CLI must catch it and frame- a friendly message instead of leaking a traceback.- """- from temporalio.service import RPCError, RPCStatusCode # noqa: PLC0415-- self._patch_config(mocker)- mocker.patch(- "pipelex.temporal.temporal_connect.connect_to_temporal_selected_server",- new=mocker.AsyncMock(return_value=mocker.MagicMock()),- )- mocker.patch(- "pipelex.temporal.tprl.namespace_check.ensure_required_search_attributes_registered",- new=mocker.AsyncMock(side_effect=RPCError("namespace not found", RPCStatusCode.NOT_FOUND, raw_grpc_status=b"")),- )-- with pytest.raises(typer.Exit) as exc_info:- setup_temporal_namespace_cmd(dry_run=False, server=None)-- assert exc_info.value.exit_code == 1- err = capsys.readouterr().err- assert "Could not reach Temporal namespace" in err- assert "namespace not found" in err-- @pytest.mark.usefixtures("patch_pipelex_lifecycle")- def test_unknown_server_profile_exits_with_helpful_message(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:- self._patch_config(mocker, selected_server="local")- mocker.patch(- "pipelex.temporal.temporal_connect.connect_to_temporal_selected_server",- new=mocker.AsyncMock(),- )-- with pytest.raises(typer.Exit) as exc_info:- setup_temporal_namespace_cmd(dry_run=False, server="does_not_exist")-- assert exc_info.value.exit_code == 1- err = capsys.readouterr().err- assert "does_not_exist" in err- assert "Known:" in errdiff --git a/tests/unit/pipelex/cli/test_validate_orchestrator_flag.py b/tests/unit/pipelex/cli/test_validate_orchestrator_flag.pynew file mode 100644index 000000000..ee830cf45--- /dev/null+++ b/tests/unit/pipelex/cli/test_validate_orchestrator_flag.py@@ -0,0 +1,31 @@+"""The ``--orchestrator`` flag surface on ``pipelex validate`` subcommands.++Locks the public CLI contract that ``bundle`` / ``pipe`` / ``method`` each expose+``--orchestrator`` (parity with ``pipelex run``). The flag boots this process under the+named orchestrator plugin; the validation sweep stays in-process regardless, so the+*behavioural* guarantee (validation does not dispatch to an orchestrator runtime) is+exercised by the distributed e2e scenario, not here.++Asserts on the resolved Click parameters rather than on rendered ``--help`` text: the+help renderer line-wraps option names at narrow terminal widths, which is a presentation+detail, not the contract. Introspecting the command's params is width-independent and+short-circuits before any Pipelex boot, so this stays a true unit test.+"""++from __future__ import annotations++import click+import pytest+from typer.main import get_command++from pipelex.cli.commands.validate.app import validate_app+++class TestValidateOrchestratorFlag:+ @pytest.mark.parametrize("subcommand", ["bundle", "pipe", "method"])+ def test_orchestrator_flag_exposed(self, subcommand: str) -> None:+ group = get_command(validate_app)+ assert isinstance(group, click.Group)+ command = group.commands[subcommand]+ flag_opts: list[str] = [opt for param in command.params for opt in (*param.opts, *param.secondary_opts)]+ assert "--orchestrator" in flag_optsdiff --git a/tests/unit/pipelex/cli/test_validate_temporal_flag.py b/tests/unit/pipelex/cli/test_validate_temporal_flag.pydeleted file mode 100644index 745aee8d1..000000000--- a/tests/unit/pipelex/cli/test_validate_temporal_flag.py+++ /dev/null@@ -1,33 +0,0 @@-"""The ``--temporal`` flag surface on ``pipelex validate`` subcommands.--Locks the public CLI contract that ``bundle`` / ``pipe`` / ``method`` each expose-``--temporal/--no-temporal`` (parity with ``pipelex run``). The flag overrides-``temporal.is_enabled`` for the boot; the validation sweep stays in-process regardless,-so the *behavioural* guarantee (validation does not dispatch to Temporal under a-Temporal-enabled hub) is exercised by the distributed e2e scenario, not here.--Asserts on the resolved Click parameters rather than on rendered ``--help`` text: the-help renderer line-wraps option names at narrow terminal widths (CI's non-tty width-splits ``--temporal`` across lines), which is a presentation detail, not the contract.-Introspecting the command's params is width-independent and short-circuits before any-Pipelex boot, so this stays a true unit test.-"""--from __future__ import annotations--import click-import pytest-from typer.main import get_command--from pipelex.cli.commands.validate.app import validate_app---class TestValidateTemporalFlag:- @pytest.mark.parametrize("subcommand", ["bundle", "pipe", "method"])- def test_temporal_flag_exposed(self, subcommand: str) -> None:- group = get_command(validate_app)- assert isinstance(group, click.Group)- command = group.commands[subcommand]- flag_opts: list[str] = [opt for param in command.params for opt in (*param.opts, *param.secondary_opts)]- assert "--temporal" in flag_opts- assert "--no-temporal" in flag_optsdiff --git a/tests/unit/pipelex/cogt/img_gen/test_img_gen_worker_factory.py b/tests/unit/pipelex/cogt/img_gen/test_img_gen_worker_factory.pyindex 62ae74d05..c7752da7e 100644--- a/tests/unit/pipelex/cogt/img_gen/test_img_gen_worker_factory.py+++ b/tests/unit/pipelex/cogt/img_gen/test_img_gen_worker_factory.py@@ -1,11 +1,12 @@-"""Routing tests for ImgGenWorkerFactory: each plugin SDK string must build the right image+"""Routing tests for ImgGenWorkerFactory: each SDK string must build the right imagegeneration worker with the right SDK client, completions factory, and reporting delegate,-while caching the SDK instance in the plugin SDK registry.+while caching the SDK instance in the SDK client registry."""from __future__ import annotations-from typing import TYPE_CHECKING+from types import SimpleNamespace+from typing import TYPE_CHECKING, castimport pytest@@ -17,14 +18,38 @@ from pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.usage.cost_category import CostCategoryfrom pipelex.exceptions import MissingDependencyErrorfrom pipelex.plugins.blackboxai.blackboxai_completions_factory import BlackboxaiCompletionsFactory+from pipelex.plugins.builtins import BUILTIN_PLUGINS+from pipelex.plugins.exceptions import InferenceBackendNotFoundErrorfrom pipelex.plugins.gateway.gateway_completions_factory import GatewayCompletionsFactory+from pipelex.plugins.inference_backend_registry import InferenceBackendRegistry+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsFactoryfrom pipelex.plugins.openrouter.openrouter_completions_factory import OpenRouterCompletionsFactory-from pipelex.plugins.plugin_sdk_registry import Plugin, PluginSdkRegistry+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistryif TYPE_CHECKING:from pytest_mock import MockerFixture+ from pipelex.system.configuration.configs import PipelexConfig+++def build_builtin_inference_backend_registry() -> InferenceBackendRegistry:+ """Register the built-in driver plugins into a fresh registrar and snapshot the backends.++ This is exactly what boot does, so the routing tests below exercise the real+ plugin closures through the real registry lookup.+ """+ # No builtin's register() reads config: Temporal — the only plugin that did, via+ # ``temporal.is_enabled`` — now ships as the external pipelex-temporal dist. A bare+ # stub config suffices.+ stub_config = cast("PipelexConfig", SimpleNamespace())+ registrar = PluginRegistrar(config=stub_config)+ for plugin in BUILTIN_PLUGINS:+ plugin.register(registrar)+ return InferenceBackendRegistry(registrar.inference_backends)++FACTORY_MODULE = "pipelex.cogt.img_gen.img_gen_worker_factory"GATEWAY_CLIENT = "pipelex.plugins.gateway.gateway_factory.GatewayFactory.make_portkey_client"@@ -72,21 +97,22 @@ def make_backend(name: str = "test_backend") -> InferenceBackend:def patch_hub_getters(mocker: MockerFixture,backend: InferenceBackend,-) -> PluginSdkRegistry:+) -> SdkClientRegistry:"""Patch hub getters at the worker-factory namespace; return the fresh SDK registry."""- registry = PluginSdkRegistry()+ registry = SdkClientRegistry()models_manager = mocker.MagicMock()models_manager.get_required_inference_backend.return_value = backendmocker.patch(f"{FACTORY_MODULE}.get_models_manager", return_value=models_manager)- plugin_manager = mocker.MagicMock()- plugin_manager.plugin_sdk_registry = registry- mocker.patch(f"{FACTORY_MODULE}.get_plugin_manager", return_value=plugin_manager)+ sdk_client_manager = mocker.MagicMock()+ sdk_client_manager.sdk_client_registry = registry+ mocker.patch(f"{FACTORY_MODULE}.get_sdk_client_manager", return_value=sdk_client_manager)+ mocker.patch(f"{FACTORY_MODULE}.get_inference_backend_registry", return_value=build_builtin_inference_backend_registry())return registryclass TestImgGenWorkerFactory:@pytest.mark.parametrize(- ("sdk", "client_target", "worker_target", "factory_cls", "http_flag", "passes_plugin"),+ ("sdk", "client_target", "worker_target", "factory_cls", "http_flag", "passes_model_handle"),[pytest.param("gateway_img_gen", GATEWAY_CLIENT, GATEWAY_WORKER, None, None, False, id="gateway_img_gen"),pytest.param("openai_img_gen", OPENAI_CLIENT, OPENAI_WORKER, None, None, True, id="openai_img_gen"),@@ -112,7 +138,7 @@ class TestImgGenWorkerFactory:worker_target: str,factory_cls: type | None,http_flag: bool | None,- passes_plugin: bool,+ passes_model_handle: bool,) -> None:"""Each SDK string routes to its image generation worker with the SDK client wired in."""backend = make_backend()@@ -128,8 +154,8 @@ class TestImgGenWorkerFactory:client_factory_mock.assert_called_once()client_kwargs = client_factory_mock.call_args.kwargsassert client_kwargs["backend"] is backend- if passes_plugin:- assert client_kwargs["plugin"] == Plugin(sdk=sdk, backend="test_backend", variant=None)+ if passes_model_handle:+ assert client_kwargs["model_handle"] == ModelHandle(sdk=sdk, backend="test_backend", variant=None)worker_cls_mock.assert_called_once()worker_kwargs = worker_cls_mock.call_args.kwargs@@ -205,7 +231,7 @@ class TestImgGenWorkerFactory:worker = ImgGenWorkerFactory.make_img_gen_worker(inference_model=inference_model, reporting_delegate=reporting_delegate)worker_cls_mock.assert_called_once_with(- plugin=Plugin(sdk="azure_rest_img_gen", backend="test_backend", variant=None),+ model_handle=ModelHandle(sdk="azure_rest_img_gen", backend="test_backend", variant=None),inference_model=inference_model,reporting_delegate=reporting_delegate,)@@ -217,9 +243,9 @@ class TestImgGenWorkerFactory:backend = make_backend()registry = patch_hub_getters(mocker, backend=backend)inference_model = make_img_gen_model_spec(sdk="gateway_img_gen")- plugin = Plugin.make_for_inference_model(inference_model=inference_model)+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)cached_client = mocker.MagicMock(name="cached_client")- registry.set_sdk_instance(plugin=plugin, sdk_instance=cached_client)+ registry.set(model_handle=model_handle, sdk_instance=cached_client)client_factory_mock = mocker.patch(GATEWAY_CLIENT)worker_cls_mock = mocker.patch(GATEWAY_WORKER)@@ -234,6 +260,7 @@ class TestImgGenWorkerFactory:[pytest.param("fal", "fal", id="fal"),pytest.param("google", "google", id="google"),+ pytest.param("huggingface_img_gen", "huggingface", id="huggingface_img_gen"),],)def test_missing_dependency_raises(self, mocker: MockerFixture, sdk: str, expected_extra: str) -> None:@@ -249,14 +276,15 @@ class TestImgGenWorkerFactory:assert exc_info.value.extra_name == expected_extraassert f"pipelex[{expected_extra}]" in str(exc_info.value)- def test_unknown_sdk_raises_not_implemented(self, mocker: MockerFixture) -> None:- """An unrecognized SDK string raises NotImplementedError naming the plugin."""+ def test_unknown_sdk_raises_backend_not_found(self, mocker: MockerFixture) -> None:+ """An unrecognized SDK string raises a structured InferenceBackendNotFoundError naming the SDK (a registry miss)."""backend = make_backend()patch_hub_getters(mocker, backend=backend)inference_model = make_img_gen_model_spec(sdk="definitely_not_an_sdk")- with pytest.raises(NotImplementedError) as exc_info:+ with pytest.raises(InferenceBackendNotFoundError) as exc_info:ImgGenWorkerFactory.make_img_gen_worker(inference_model=inference_model)+ assert exc_info.value.sdk == "definitely_not_an_sdk"assert "definitely_not_an_sdk" in str(exc_info.value)- assert "not supported for image generation" in str(exc_info.value)+ assert "Is its plugin installed and enabled?" in str(exc_info.value)diff --git a/tests/unit/pipelex/cogt/llm/test_llm_worker_factory.py b/tests/unit/pipelex/cogt/llm/test_llm_worker_factory.pyindex f1298f83b..0eb75add0 100644--- a/tests/unit/pipelex/cogt/llm/test_llm_worker_factory.py+++ b/tests/unit/pipelex/cogt/llm/test_llm_worker_factory.py@@ -1,11 +1,12 @@-"""Routing tests for LLMWorkerFactory: each plugin SDK string must build the right worker+"""Routing tests for LLMWorkerFactory: each SDK string must build the right workerwith the right SDK client, completions/responses factory, and reporting delegate, while-caching the SDK instance in the plugin SDK registry.+caching the SDK instance in the SDK client registry."""from __future__ import annotations-from typing import TYPE_CHECKING+from types import SimpleNamespace+from typing import TYPE_CHECKING, castimport pytest@@ -16,18 +17,42 @@ from pipelex.cogt.model_backends.model_spec import InferenceModelSpecfrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.usage.cost_category import CostCategoryfrom pipelex.exceptions import MissingDependencyError+from pipelex.plugins.builtins import BUILTIN_PLUGINS+from pipelex.plugins.exceptions import InferenceBackendNotFoundErrorfrom pipelex.plugins.gateway.gateway_completions_factory import GatewayCompletionsFactoryfrom pipelex.plugins.gateway.gateway_responses_factory import GatewayResponsesFactory+from pipelex.plugins.inference_backend_registry import InferenceBackendRegistryfrom pipelex.plugins.mistral.mistral_factory import MistralFactory+from pipelex.plugins.model_handle import ModelHandlefrom pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsFactoryfrom pipelex.plugins.openai.openai_responses_factory import OpenAIResponsesFactory-from pipelex.plugins.plugin_sdk_registry import Plugin, PluginSdkRegistryfrom pipelex.plugins.portkey.portkey_completions_factory import PortkeyCompletionsFactoryfrom pipelex.plugins.portkey.portkey_responses_factory import PortkeyResponsesFactory+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.plugins.sdk_client_registry import SdkClientRegistryif TYPE_CHECKING:from pytest_mock import MockerFixture+ from pipelex.system.configuration.configs import PipelexConfig+++def build_builtin_inference_backend_registry() -> InferenceBackendRegistry:+ """Register the built-in driver plugins into a fresh registrar and snapshot the backends.++ This is exactly what boot does, so the routing tests below exercise the real+ plugin closures through the real registry lookup.+ """+ # No builtin's register() reads config: Temporal — the only plugin that did, via+ # ``temporal.is_enabled`` — now ships as the external pipelex-temporal dist. A bare+ # stub config suffices.+ stub_config = cast("PipelexConfig", SimpleNamespace())+ registrar = PluginRegistrar(config=stub_config)+ for plugin in BUILTIN_PLUGINS:+ plugin.register(registrar)+ return InferenceBackendRegistry(registrar.inference_backends)++FACTORY_MODULE = "pipelex.cogt.llm.llm_worker_factory"GATEWAY_COMPLETIONS_CLIENT = (@@ -79,21 +104,22 @@ def make_backend(name: str = "test_backend") -> InferenceBackend:def patch_hub_getters(mocker: MockerFixture,backend: InferenceBackend,-) -> PluginSdkRegistry:+) -> SdkClientRegistry:"""Patch hub getters at the worker-factory namespace; return the fresh SDK registry."""- registry = PluginSdkRegistry()+ registry = SdkClientRegistry()models_manager = mocker.MagicMock()models_manager.get_required_inference_backend.return_value = backendmocker.patch(f"{FACTORY_MODULE}.get_models_manager", return_value=models_manager)- plugin_manager = mocker.MagicMock()- plugin_manager.plugin_sdk_registry = registry- mocker.patch(f"{FACTORY_MODULE}.get_plugin_manager", return_value=plugin_manager)+ sdk_client_manager = mocker.MagicMock()+ sdk_client_manager.sdk_client_registry = registry+ mocker.patch(f"{FACTORY_MODULE}.get_sdk_client_manager", return_value=sdk_client_manager)+ mocker.patch(f"{FACTORY_MODULE}.get_inference_backend_registry", return_value=build_builtin_inference_backend_registry())return registryclass TestLLMWorkerFactory:@pytest.mark.parametrize(- ("sdk", "client_target", "worker_target", "factory_field", "factory_cls", "http_flag", "passes_plugin", "expects_extra_config"),+ ("sdk", "client_target", "worker_target", "factory_field", "factory_cls", "http_flag", "passes_model_handle", "expects_extra_config"),[pytest.param("gateway_completions",@@ -200,7 +226,7 @@ class TestLLMWorkerFactory:factory_field: str | None,factory_cls: type | None,http_flag: bool | None,- passes_plugin: bool,+ passes_model_handle: bool,expects_extra_config: bool,) -> None:"""Each SDK string routes to its worker class with the SDK client and factory wired in."""@@ -217,8 +243,8 @@ class TestLLMWorkerFactory:client_factory_mock.assert_called_once()client_kwargs = client_factory_mock.call_args.kwargsassert client_kwargs["backend"] is backend- if passes_plugin:- assert client_kwargs["plugin"] == Plugin(sdk=sdk, backend="test_backend", variant=None)+ if passes_model_handle:+ assert client_kwargs["model_handle"] == ModelHandle(sdk=sdk, backend="test_backend", variant=None)worker_cls_mock.assert_called_once()worker_kwargs = worker_cls_mock.call_args.kwargs@@ -243,9 +269,9 @@ class TestLLMWorkerFactory:backend = make_backend()registry = patch_hub_getters(mocker, backend=backend)inference_model = make_llm_model_spec(sdk="openai")- plugin = Plugin.make_for_inference_model(inference_model=inference_model)+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)cached_client = mocker.MagicMock(name="cached_client")- registry.set_sdk_instance(plugin=plugin, sdk_instance=cached_client)+ registry.set(model_handle=model_handle, sdk_instance=cached_client)client_factory_mock = mocker.patch(OPENAI_CLIENT)worker_cls_mock = mocker.patch(COMPLETIONS_WORKER)@@ -260,7 +286,7 @@ class TestLLMWorkerFactory:backend = make_backend()registry = patch_hub_getters(mocker, backend=backend)inference_model = make_llm_model_spec(sdk="openai")- plugin = Plugin.make_for_inference_model(inference_model=inference_model)+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)sdk_client = mocker.MagicMock(name="sdk_client")client_factory_mock = mocker.patch(OPENAI_CLIENT, return_value=sdk_client)worker_cls_mock = mocker.patch(COMPLETIONS_WORKER)@@ -269,7 +295,7 @@ class TestLLMWorkerFactory:LLMWorkerFactory.make_llm_worker(inference_model=inference_model)client_factory_mock.assert_called_once()- assert registry.get_sdk_instance(plugin=plugin) is sdk_client+ assert registry.get(model_handle=model_handle) is sdk_clientfirst_kwargs, second_kwargs = (call.kwargs for call in worker_cls_mock.call_args_list)assert first_kwargs["sdk_instance"] is sdk_clientassert second_kwargs["sdk_instance"] is sdk_client@@ -298,14 +324,15 @@ class TestLLMWorkerFactory:assert exc_info.value.extra_name == expected_extraassert f"pipelex[{expected_extra}]" in str(exc_info.value)- def test_unknown_sdk_raises_not_implemented(self, mocker: MockerFixture) -> None:- """An unrecognized SDK string raises NotImplementedError naming the plugin."""+ def test_unknown_sdk_raises_backend_not_found(self, mocker: MockerFixture) -> None:+ """An unrecognized SDK string raises a structured InferenceBackendNotFoundError naming the SDK (a registry miss)."""backend = make_backend()patch_hub_getters(mocker, backend=backend)inference_model = make_llm_model_spec(sdk="definitely_not_an_sdk")- with pytest.raises(NotImplementedError) as exc_info:+ with pytest.raises(InferenceBackendNotFoundError) as exc_info:LLMWorkerFactory.make_llm_worker(inference_model=inference_model)+ assert exc_info.value.sdk == "definitely_not_an_sdk"assert "definitely_not_an_sdk" in str(exc_info.value)- assert "is not supported" in str(exc_info.value)+ assert "Is its plugin installed and enabled?" in str(exc_info.value)diff --git a/tests/unit/pipelex/cogt/model_backends/test_model_lister_dispatch.py b/tests/unit/pipelex/cogt/model_backends/test_model_lister_dispatch.pynew file mode 100644index 000000000..b61468de5--- /dev/null+++ b/tests/unit/pipelex/cogt/model_backends/test_model_lister_dispatch.py@@ -0,0 +1,99 @@+"""Behavioral dispatch through the model-lister seam: ``ModelLister.list_models`` resolves each+SDK's lister from the registry on the hub (no ``match`` over SDK strings), threads ``any_listed``+across SDKs, treats a registry miss and the soft ``ModelListingUnsupportedError`` signal alike+(reported as unsupported-for-listing), and wraps any other lister failure in a ``PipelexCLIError``.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.cli.exceptions import PipelexCLIError+from pipelex.cogt.exceptions import ModelListingUnsupportedError+from pipelex.cogt.model_backends.model_lists import ModelLister+from pipelex.plugins.model_lister_registry import ListModelsFn, ModelListerRegistry++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++ from pipelex.cogt.model_backends.backend import InferenceBackend++_MODULE = "pipelex.cogt.model_backends.model_lists"+++def _fake_backend(model_specs: dict[str, str]) -> InferenceBackend:+ # The loop only reads ``.model_specs`` and each spec's ``.sdk``.+ return cast(+ "InferenceBackend",+ SimpleNamespace(model_specs={name: SimpleNamespace(sdk=sdk) for name, sdk in model_specs.items()}),+ )+++def _patch_hub(mocker: MockerFixture, *, backend: InferenceBackend, listers: dict[str, ListModelsFn]) -> SimpleNamespace:+ console = mocker.Mock()+ models_manager = mocker.Mock()+ models_manager.get_required_inference_backend.return_value = backend+ mocker.patch(f"{_MODULE}.get_models_manager", return_value=models_manager)+ mocker.patch(f"{_MODULE}.get_console", return_value=console)+ mocker.patch(f"{_MODULE}.get_model_lister_registry", return_value=ModelListerRegistry(listers))+ return SimpleNamespace(console=console)+++def _printed(console: SimpleNamespace) -> str:+ return "\n".join(str(call.args[0]) for call in console.print.call_args_list if call.args)++[email protected](loop_scope="class")+class TestModelListerDispatch:+ async def test_registered_lister_is_invoked_with_expected_kwargs(self, mocker: MockerFixture) -> None:+ lister = mocker.AsyncMock()+ backend = _fake_backend({"model-a": "fake_sdk"})+ _patch_hub(mocker, backend=backend, listers={"fake_sdk": lister})++ await ModelLister.list_models("my_backend", flat=False)++ lister.assert_awaited_once_with(sdk="fake_sdk", backend_name="my_backend", backend=backend, flat=False, any_listed=False)++ async def test_any_listed_progresses_across_sdks(self, mocker: MockerFixture) -> None:+ lister = mocker.AsyncMock()+ # Two SDKs, both served by the same recorded lister: the first call sees any_listed=False,+ # the second sees any_listed=True (so its CSV header is suppressed).+ backend = _fake_backend({"model-a": "sdk_a", "model-b": "sdk_b"})+ _patch_hub(mocker, backend=backend, listers={"sdk_a": lister, "sdk_b": lister})++ await ModelLister.list_models("my_backend", flat=True)++ assert lister.await_count == 2+ assert lister.await_args_list[0].kwargs["any_listed"] is False+ assert lister.await_args_list[1].kwargs["any_listed"] is True++ async def test_unknown_sdk_is_reported_unsupported(self, mocker: MockerFixture) -> None:+ backend = _fake_backend({"model-a": "no_lister_sdk"})+ ctx = _patch_hub(mocker, backend=backend, listers={})++ await ModelLister.list_models("my_backend", flat=False)++ assert "we don't support for remote listing" in _printed(ctx.console)+ assert "no_lister_sdk" in _printed(ctx.console)++ async def test_lister_unsupported_signal_is_reported_unsupported(self, mocker: MockerFixture) -> None:+ # The translate path: a lister raises the core soft signal (as the Anthropic lister does for+ # a bedrock-backed client) → same outcome as a missing lister, never a hard failure.+ lister = mocker.AsyncMock(side_effect=ModelListingUnsupportedError(sdk="fake_sdk"))+ backend = _fake_backend({"model-a": "fake_sdk"})+ ctx = _patch_hub(mocker, backend=backend, listers={"fake_sdk": lister})++ await ModelLister.list_models("my_backend", flat=False)++ assert "we don't support for remote listing" in _printed(ctx.console)++ async def test_lister_failure_is_wrapped_in_cli_error(self, mocker: MockerFixture) -> None:+ lister = mocker.AsyncMock(side_effect=RuntimeError("provider API exploded"))+ backend = _fake_backend({"model-a": "fake_sdk"})+ _patch_hub(mocker, backend=backend, listers={"fake_sdk": lister})++ with pytest.raises(PipelexCLIError, match="Error listing models for SDK 'fake_sdk' in backend 'my_backend'"):+ await ModelLister.list_models("my_backend", flat=False)diff --git a/tests/unit/pipelex/core/pipes/test_pipe_output_prepare_for_temporal.py b/tests/unit/pipelex/core/pipes/test_pipe_output_prepare_for_temporal.pydeleted file mode 100644index 9d0a6c2db..000000000--- a/tests/unit/pipelex/core/pipes/test_pipe_output_prepare_for_temporal.py+++ /dev/null@@ -1,63 +0,0 @@-from pytest_mock import MockerFixture--from pipelex.core.concepts.concept import Concept-from pipelex.core.domains.domain import SpecialDomain-from pipelex.core.memory.working_memory import WorkingMemory-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.stuff import Stuff-from pipelex.core.stuffs.text_content import TextContent---def _make_populated_pipe_output() -> PipeOutput:- working_memory = WorkingMemory()- working_memory.root["greeting"] = Stuff(- stuff_code="test",- stuff_name="greeting",- concept=Concept(- code="Text",- domain_code=SpecialDomain.NATIVE,- description="Plain text",- structure_class_name="TextContent",- ),- content=TextContent(text="Hello!"),- )- return PipeOutput(working_memory=working_memory)---class TestPipeOutputPrepareForTemporal:- def test_no_crate_returns_self_unchanged(self) -> None:- pipe_output = _make_populated_pipe_output()-- result = pipe_output.prepare_for_temporal(library_crate=None)-- assert result is pipe_output- assert result.working_memory_raw is None- assert "greeting" in result.working_memory.root-- def test_with_crate_dehydrates(self, mocker: MockerFixture) -> None:- from pipelex.libraries.library_crate import LibraryCrate # noqa: PLC0415-- fake_crate = mocker.MagicMock(spec=LibraryCrate)- pipe_output = _make_populated_pipe_output()-- result = pipe_output.prepare_for_temporal(library_crate=fake_crate)-- assert result is not pipe_output- assert result.working_memory_raw is not None- assert "greeting" in result.working_memory_raw.get("root", {})- # Original is unchanged.- assert pipe_output.working_memory_raw is None- assert "greeting" in pipe_output.working_memory.root- # The dehydrated copy has an empty WorkingMemory.- assert not result.working_memory.root-- def test_with_crate_empty_wm_returns_self(self, mocker: MockerFixture) -> None:- from pipelex.libraries.library_crate import LibraryCrate # noqa: PLC0415-- fake_crate = mocker.MagicMock(spec=LibraryCrate)- pipe_output = PipeOutput(working_memory=WorkingMemory())-- result = pipe_output.prepare_for_temporal(library_crate=fake_crate)-- assert result is pipe_output- assert result.working_memory_raw is Nonediff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.pyindex 4f6129032..5a4fa18bc 100644--- a/tests/unit/pipelex/pipe_run/test_delivery_executor.py+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py@@ -262,7 +262,7 @@ class TestDeliveryExecutor:async def test_generate_result_files_with_pydantic_instances_in_raw(self, mocker: MockerFixture) -> None:"""working_memory_raw can contain hydrated Pydantic instances after Temporal transit.- When `dump_for_temporal()` runs in a child workflow, it embeds `__class__` metadata+ When `dump_for_transport()` runs in a child workflow, it embeds `__class__` metadataon ListContent items so the parent can reconstruct them. Kajson's Temporal dataconverter then eagerly rehydrates those dicts back into BaseModel instances on theactivity worker that runs delivery — even though the typed slot is `dict[str, Any]`.diff --git a/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.pyindex 49d211a96..3b54d593f 100644--- a/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py+++ b/tests/unit/pipelex/pipe_run/test_pipe_job_hydration.py@@ -29,57 +29,7 @@ def _make_pipe_job()-def _make_dummy_crate() -> LibraryCrate:- """Build a minimal LibraryCrate for testing."""- return LibraryCrate(concepts={}, pipes={}, domains={}, source_map={})--class TestPipeJobHydration:- def test_prepare_for_temporal_moves_wm_to_raw(self) -> None:- """When library_crate is present, prepare_for_temporal() returns a copy with WM serialized to raw."""- working_memory = WorkingMemory()- crate = _make_dummy_crate()- pipe_job = _make_pipe_job(working_memory=working_memory, library_crate=crate)-- result = pipe_job.prepare_for_temporal()-- assert result.working_memory is None- assert result.working_memory_raw is not None- assert isinstance(result.working_memory_raw, dict)-- def test_prepare_for_temporal_noop_without_crate(self) -> None:- """Without library_crate, prepare_for_temporal() returns self unchanged."""- working_memory = WorkingMemory()- pipe_job = _make_pipe_job(working_memory=working_memory, library_crate=None)-- result = pipe_job.prepare_for_temporal()-- assert result is pipe_job- assert result.working_memory is working_memory- assert result.working_memory_raw is None-- def test_prepare_for_temporal_empty_wm(self) -> None:- """Empty WorkingMemory serializes to a dict with empty root."""- working_memory = WorkingMemory()- crate = _make_dummy_crate()- pipe_job = _make_pipe_job(working_memory=working_memory, library_crate=crate)-- result = pipe_job.prepare_for_temporal()-- assert result.working_memory_raw is not None- assert result.working_memory_raw["root"] == {}-- def test_prepare_for_temporal_does_not_mutate_original(self) -> None:- """prepare_for_temporal() leaves the original PipeJob unchanged."""- working_memory = WorkingMemory()- crate = _make_dummy_crate()- pipe_job = _make_pipe_job(working_memory=working_memory, library_crate=crate)-- _ = pipe_job.prepare_for_temporal()-- assert pipe_job.working_memory is working_memory- assert pipe_job.working_memory_raw is None-def test_get_working_memory_from_typed(self) -> None:"""get_working_memory() returns the typed WorkingMemory when set."""working_memory = WorkingMemory()@@ -103,14 +53,3 @@ class TestPipeJobHydration:result = pipe_job.get_working_memory()assert isinstance(result, WorkingMemory)assert len(result.root) == 0-- def test_prepare_for_temporal_crate_without_wm(self) -> None:- """library_crate present but working_memory is None returns self unchanged."""- crate = _make_dummy_crate()- pipe_job = _make_pipe_job(working_memory=None, library_crate=crate)-- result = pipe_job.prepare_for_temporal()-- assert result is pipe_job- assert result.working_memory is None- assert result.working_memory_raw is Nonediff --git a/tests/unit/pipelex/pipeline/test_direct_bundle_validator.py b/tests/unit/pipelex/pipeline/test_direct_bundle_validator.pynew file mode 100644index 000000000..5c6d3c77b--- /dev/null+++ b/tests/unit/pipelex/pipeline/test_direct_bundle_validator.py@@ -0,0 +1,83 @@+"""The core in-process bundle validator wraps ``validate_bundles_in_process`` as verdict-as-value.++Pins the wrapper's two branches with the in-process sweep mocked out (its own report shaping is+covered by test_validation_report.py / the integration test_protocol_validate.py): a successful+sweep is returned verbatim as the valid arm; a ``ValidateBundleError`` is converted to its+structured ``ErrorReport`` (the invalid arm) rather than propagated. Also pins argument+passthrough, including the ``library_dirs`` host context and the ``log_context`` label.+"""++from __future__ import annotations++from pathlib import Path+from typing import TYPE_CHECKING++import pytest++from pipelex.base_exceptions import ErrorReport+from pipelex.pipeline.direct_bundle_validator import DirectBundleValidator+from pipelex.pipeline.exceptions import ValidateBundleError++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++[email protected](loop_scope="class")+class TestDirectBundleValidator:+ async def test_valid_sweep_is_returned_as_the_valid_arm(self, mocker: MockerFixture) -> None:+ """A successful in-process sweep's report is returned verbatim, with arguments forwarded."""+ report = mocker.MagicMock(name="validation_report")+ in_process_mock = mocker.patch(+ "pipelex.pipeline.direct_bundle_validator.validate_bundles_in_process",+ new=mocker.AsyncMock(return_value=report),+ )++ verdict = await DirectBundleValidator().validate_bundles(+ mthds_contents=["bundle-content"],+ mthds_sources=["domain.mthds"],+ allow_signatures=True,+ library_dirs=[Path("lib_dir")],+ )++ assert verdict is report+ in_process_mock.assert_awaited_once_with(+ mthds_contents=["bundle-content"],+ mthds_sources=["domain.mthds"],+ library_dirs=[Path("lib_dir")],+ allow_signatures=True,+ log_context="API validate",+ )++ async def test_validate_bundle_error_becomes_the_invalid_arm(self, mocker: MockerFixture) -> None:+ """An invalid bundle's ValidateBundleError is converted to its ErrorReport, not propagated."""+ error = ValidateBundleError("bundle is invalid")+ mocker.patch(+ "pipelex.pipeline.direct_bundle_validator.validate_bundles_in_process",+ new=mocker.AsyncMock(side_effect=error),+ )++ verdict = await DirectBundleValidator().validate_bundles(+ mthds_contents=["bundle-content"],+ mthds_sources=None,+ allow_signatures=False,+ library_dirs=None,+ )++ assert isinstance(verdict, ErrorReport)+ assert verdict.error_type == ValidateBundleError.__name__+ assert verdict == error.to_error_report()++ async def test_other_errors_propagate_as_no_verdict_faults(self, mocker: MockerFixture) -> None:+ """A non-ValidateBundleError (a genuine infra fault) propagates — the host maps it to a 5xx."""+ mocker.patch(+ "pipelex.pipeline.direct_bundle_validator.validate_bundles_in_process",+ new=mocker.AsyncMock(side_effect=RuntimeError("library teardown blew up")),+ )++ with pytest.raises(RuntimeError, match="library teardown blew up"):+ await DirectBundleValidator().validate_bundles(+ mthds_contents=["bundle-content"],+ mthds_sources=None,+ allow_signatures=False,+ library_dirs=None,+ )diff --git a/tests/unit/pipelex/plugins/azure_rest/test_azure_img_gen_worker_error_handling.py b/tests/unit/pipelex/plugins/azure_rest/test_azure_img_gen_worker_error_handling.pyindex 3d3c1e981..2e6bf2a43 100644--- a/tests/unit/pipelex/plugins/azure_rest/test_azure_img_gen_worker_error_handling.py+++ b/tests/unit/pipelex/plugins/azure_rest/test_azure_img_gen_worker_error_handling.py@@ -38,7 +38,7 @@ def _make_worker(mocker: MockerFixture) -> AzureImgGenWorker:worker.api_key = "test-key"worker.endpoint = "https://test.azure.com"worker.api_version = "2024-02-01"- worker.plugin = mocker.MagicMock()+ worker.model_handle = mocker.MagicMock()return workerdiff --git a/tests/unit/pipelex/plugins/azure_rest/test_azure_worker_error_handling.py b/tests/unit/pipelex/plugins/azure_rest/test_azure_worker_error_handling.pyindex ee52e188c..40c94d54d 100644--- a/tests/unit/pipelex/plugins/azure_rest/test_azure_worker_error_handling.py+++ b/tests/unit/pipelex/plugins/azure_rest/test_azure_worker_error_handling.py@@ -35,7 +35,7 @@ def _make_azure_img_gen_worker(mocker: MockerFixture) -> AzureImgGenWorker:worker.api_key = "test-api-key"worker.endpoint = "https://test.azure.com"worker.api_version = "2024-02-01"- worker.plugin = mocker.MagicMock()+ worker.model_handle = mocker.MagicMock()return workerdiff --git a/tests/unit/pipelex/plugins/gateway/test_gateway_completions_client.py b/tests/unit/pipelex/plugins/gateway/test_gateway_completions_client.pyindex 527a5972d..699569fb4 100644--- a/tests/unit/pipelex/plugins/gateway/test_gateway_completions_client.py+++ b/tests/unit/pipelex/plugins/gateway/test_gateway_completions_client.py@@ -38,14 +38,14 @@ class TestGatewayCompletionsClient:"""Client construction validates the SDK variant and wires Portkey headers."""def test_wrong_sdk_variant_raises_factory_error(self, mocker: MockerFixture) -> None:- """A plugin whose sdk is not the completions variant is rejected before any client is built."""+ """A model handle whose sdk is not the completions variant is rejected before any client is built."""_patch_gateway_accessors(mocker, is_debug_enabled=False, endpoint="https://gateway.test", api_key="gw-test")mock_client = mocker.patch("openai.AsyncOpenAI")- plugin = mocker.MagicMock()- plugin.sdk = "gateway_responses"+ model_handle = mocker.MagicMock()+ model_handle.sdk = "gateway_responses"with pytest.raises(GatewayFactoryError, match="is not supported by 'GatewayCompletionsFactory'"):- GatewayCompletionsFactory.make_portkey_openai_client_for_completions(plugin=plugin, backend=mocker.MagicMock())+ GatewayCompletionsFactory.make_portkey_openai_client_for_completions(model_handle=model_handle, backend=mocker.MagicMock())mock_client.assert_not_called()@@ -67,10 +67,10 @@ class TestGatewayCompletionsClient:"""_patch_gateway_accessors(mocker, is_debug_enabled=is_debug_enabled, endpoint="https://gateway.example/v1", api_key="pk-secret")mock_client = mocker.patch("openai.AsyncOpenAI")- plugin = mocker.MagicMock()- plugin.sdk = "gateway_completions"+ model_handle = mocker.MagicMock()+ model_handle.sdk = "gateway_completions"- client = GatewayCompletionsFactory.make_portkey_openai_client_for_completions(plugin=plugin, backend=mocker.MagicMock())+ client = GatewayCompletionsFactory.make_portkey_openai_client_for_completions(model_handle=model_handle, backend=mocker.MagicMock())assert client is mock_client.return_valuemock_client.assert_called_once()diff --git a/tests/unit/pipelex/plugins/test_bundle_validator_registry.py b/tests/unit/pipelex/plugins/test_bundle_validator_registry.pynew file mode 100644index 000000000..8759fa84b--- /dev/null+++ b/tests/unit/pipelex/plugins/test_bundle_validator_registry.py@@ -0,0 +1,91 @@+"""The bundle-validator seam: the registrar collects per-mode validators into a registry.++Pins the seam independent of the real DIRECT/Temporal validators: a validator registered for+a mode is the one the registry hands back; a second registration for the same mode fails loud+naming both plugins; an empty registry reports the mode as absent.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistry+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.exceptions import DuplicateBundleValidatorError+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar+from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE++if TYPE_CHECKING:+ from collections.abc import Sequence+ from pathlib import Path++ from pipelex.base_exceptions import ErrorReport+ from pipelex.pipeline.validation_report import PipelexValidationReport+ from pipelex.system.configuration.configs import PipelexConfig+++class _FakeBundleValidator:+ """Stand-in validator: structurally a ``BundleValidatorProtocol`` (never invoked here)."""++ async def validate_bundles(+ self,+ *,+ mthds_contents: list[str],+ mthds_sources: list[str] | None,+ allow_signatures: bool,+ library_dirs: Sequence[Path] | None,+ ) -> PipelexValidationReport | ErrorReport:+ raise NotImplementedError+++def _make_registrar() -> PluginRegistrar:+ return PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))+++class TestBundleValidatorRegistry:+ def test_registered_validator_is_retrievable_by_mode(self) -> None:+ """A validator registered for a mode is the exact instance the built registry returns for it."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ validator = _FakeBundleValidator()++ registrar.add_bundle_validator(mode=DIRECT_ORCHESTRATION_MODE, validator=validator)++ registry = BundleValidatorRegistry(registrar.bundle_validators)+ assert registry.get_optional(mode=DIRECT_ORCHESTRATION_MODE) is validator+ assert registry.has(mode=DIRECT_ORCHESTRATION_MODE)+ assert registry.modes == [DIRECT_ORCHESTRATION_MODE]++ def test_contribution_recorded_on_the_active_plugin(self) -> None:+ """Registering a validator records a ``bundle validator <mode>`` contribution line on the plugin's discovery."""+ registrar = _make_registrar()+ discovery = registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)++ registrar.add_bundle_validator(mode="temporal", validator=_FakeBundleValidator())++ assert "bundle validator temporal" in discovery.contributions++ def test_duplicate_mode_fails_loud_naming_both_plugins(self) -> None:+ """Two plugins registering a validator for the same mode is a fail-loud conflict naming both."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_bundle_validator(mode=DIRECT_ORCHESTRATION_MODE, validator=_FakeBundleValidator())+ registrar.begin_plugin(name="beta", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)++ with pytest.raises(DuplicateBundleValidatorError) as exc_info:+ registrar.add_bundle_validator(mode=DIRECT_ORCHESTRATION_MODE, validator=_FakeBundleValidator())++ assert exc_info.value.first_plugin == "alpha"+ assert exc_info.value.second_plugin == "beta"+ assert exc_info.value.mode == DIRECT_ORCHESTRATION_MODE++ def test_empty_registry_reports_mode_absent(self) -> None:+ """A registry with no validators misses every mode (the API maps a miss to MissingBundleValidatorError)."""+ registry = BundleValidatorRegistry({})++ assert registry.get_optional(mode=DIRECT_ORCHESTRATION_MODE) is None+ assert not registry.has(mode="temporal")+ assert registry.modes == []diff --git a/tests/unit/pipelex/plugins/test_http_error_mapper_seam.py b/tests/unit/pipelex/plugins/test_http_error_mapper_seam.pynew file mode 100644index 000000000..ced3dc81d--- /dev/null+++ b/tests/unit/pipelex/plugins/test_http_error_mapper_seam.py@@ -0,0 +1,114 @@+"""Tests for the F3 HTTP-error-mapper seam: the registrar collects exc->ErrorReport mappers, resolving the exc type lazily (import-light register)."""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.base_exceptions import ErrorReport+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.exceptions import DuplicateHttpErrorMapperError+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar++if TYPE_CHECKING:+ from pipelex.system.configuration.configs import PipelexConfig+++class _FakeTransportError(Exception):+ """Stand-in for an orchestrator SDK's transport fault (e.g. temporalio.TemporalError)."""+++def _make_registrar() -> PluginRegistrar:+ return PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))+++def _to_report(exc: Exception) -> ErrorReport:+ return ErrorReport(+ error_type="FakeTransportError",+ message=str(exc),+ title="Fake transport error",+ type_uri="https://errors.pipelex.com/fake-transport-error",+ retryable=True,+ )+++class TestHttpErrorMapperSeam:+ """A plugin contributes an exc->ErrorReport mapper via a lazy exc-type provider; the registrar resolves it on read, fail-loud on dups."""++ def test_add_and_get_mapper(self) -> None:+ """A registered mapper is returned keyed by its resolved exc_type and produces the classified ErrorReport when invoked."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)++ registrar.add_http_error_mapper(exc_type_provider=lambda: _FakeTransportError, to_error_report=_to_report)++ mappers = registrar.get_http_error_mappers()+ assert set(mappers) == {_FakeTransportError}+ report = mappers[_FakeTransportError](_FakeTransportError("server unreachable"))+ assert report.retryable is True+ assert report.message == "server unreachable"+ assert report.type_uri == "https://errors.pipelex.com/fake-transport-error"++ def test_provider_resolved_lazily_not_at_registration(self) -> None:+ """The exc-type provider runs only on get_http_error_mappers (read time), never at register — the import-light invariant."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)++ resolutions: list[int] = []++ def _provider() -> type[Exception]:+ resolutions.append(1)+ return _FakeTransportError++ registrar.add_http_error_mapper(exc_type_provider=_provider, to_error_report=_to_report)+ assert resolutions == [] # registration must not have called the provider++ resolved = registrar.get_http_error_mappers()+ assert resolutions == [1] # resolution happens only on read+ assert set(resolved) == {_FakeTransportError}++ def test_contribution_is_recorded_without_resolving(self) -> None:+ """Adding a mapper records a contribution line on the active plugin's discovery — and does so without resolving the exc type."""+ registrar = _make_registrar()+ discovery = registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)++ def _exploding_provider() -> type[Exception]:+ pytest.fail("provider must not run at registration")++ registrar.add_http_error_mapper(exc_type_provider=_exploding_provider, to_error_report=_to_report)++ assert "http error mapper" in discovery.contributions++ def test_duplicate_exc_type_fails_loud_naming_both_plugins(self) -> None:+ """Two plugins resolving to the same exception type is a fail-loud conflict naming both — detected at resolution time."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_http_error_mapper(exc_type_provider=lambda: _FakeTransportError, to_error_report=_to_report)+ registrar.begin_plugin(name="beta", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_http_error_mapper(exc_type_provider=lambda: _FakeTransportError, to_error_report=_to_report)++ with pytest.raises(DuplicateHttpErrorMapperError) as exc_info:+ registrar.get_http_error_mappers()++ assert exc_info.value.first_plugin == "alpha"+ assert exc_info.value.second_plugin == "beta"+ assert "_FakeTransportError" in exc_info.value.exc_type++ def test_get_returns_a_fresh_dict(self) -> None:+ """The accessor hands back a freshly built dict, so a consumer cannot mutate the registrar's accumulated state."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_http_error_mapper(exc_type_provider=lambda: _FakeTransportError, to_error_report=_to_report)++ snapshot = registrar.get_http_error_mappers()+ snapshot.clear()++ assert registrar.get_http_error_mappers() == {_FakeTransportError: _to_report}++ def test_no_mappers_by_default(self) -> None:+ """A registrar with no contributions exposes an empty mapper set (host wraps nothing, resolves nothing)."""+ registrar = _make_registrar()++ assert registrar.get_http_error_mappers() == {}diff --git a/tests/unit/pipelex/plugins/test_import_light_boot.py b/tests/unit/pipelex/plugins/test_import_light_boot.pynew file mode 100644index 000000000..15f311834--- /dev/null+++ b/tests/unit/pipelex/plugins/test_import_light_boot.py@@ -0,0 +1,79 @@+"""Import-light guard (codex C12): discovering and registering the built-in plugins must+not import any backend SDK (whether an optional extra like anthropic/linkup or a heavy core+dep like openai/portkey_ai/pypdfium2 — each plugin must defer its import into make_worker) nor+``temporalio`` (which ships only with the external ``pipelex-temporal`` plugin — core discovery+must never pull the Temporal SDK). Enforced in a subprocess whose meta-path finder raises on those+SDKs, which is deterministic where an in-process sys.modules check is not.+"""++import subprocess # noqa: S404+import sys+import textwrap++_GUARD_SCRIPT = textwrap.dedent(+ """+ import sys+ import importlib.abc++ BLOCKED = (+ "anthropic",+ "mistralai",+ "google.genai",+ "boto3",+ "aioboto3",+ "fal_client",+ "huggingface_hub",+ "docling",+ "linkup",+ "openai",+ "portkey_ai",+ "pypdfium2",+ "temporalio",+ # No web framework either: the F3 HTTP-error-mapper seam is framework-agnostic+ # (it carries the core ``ErrorReport`` type). A host runtime owns FastAPI/Starlette;+ # discovery/registration in core must never pull one in.+ "fastapi",+ "starlette",+ )++ class _Blocker(importlib.abc.MetaPathFinder):+ def find_spec(self, fullname, path, target=None):+ for blocked in BLOCKED:+ if fullname == blocked or fullname.startswith(blocked + "."):+ raise ImportError(f"import-light guard blocked '{fullname}'")+ return None++ sys.meta_path.insert(0, _Blocker())++ from types import SimpleNamespace++ import pipelex.plugins.discovery as discovery++ # Isolate to the built-ins: ignore any external entry points installed in this venv,+ # whose load() could legitimately import an optional SDK.+ discovery._external_entry_points = lambda: []++ # Temporal ships externally (pipelex-temporal); core's built-ins claim no hub slots. temporalio+ # stays in the BLOCKED set above: core discovery must never pull the Temporal SDK, and+ # temporal.is_enabled=True must not change that (no core built-in reads it). The slot-claim+ # import-light guarantee for the Temporal thunks is pinned in the pipelex-temporal suite.+ config = SimpleNamespace(plugins=SimpleNamespace(disabled=[]), temporal=SimpleNamespace(is_enabled=True))+ registrar = discovery.build_registrar(config=config)+ assert registrar.inference_backends, "expected the built-in LLM backends to be registered"+ assert registrar.model_listers, "expected the built-in model listers to be registered import-light"+ assert registrar.orchestrators, "expected the built-in orchestrators to be registered"+ print("import-light OK")+ """+)+++class TestImportLightBoot:+ def test_building_registrar_imports_no_backend_sdk(self) -> None:+ result = subprocess.run( # noqa: S603+ [sys.executable, "-c", _GUARD_SCRIPT],+ capture_output=True,+ text=True,+ check=False,+ )+ assert result.returncode == 0, f"import-light guard failed:\nstdout={result.stdout}\nstderr={result.stderr}"+ assert "import-light OK" in result.stdoutdiff --git a/tests/unit/pipelex/plugins/test_inference_backend_coverage.py b/tests/unit/pipelex/plugins/test_inference_backend_coverage.pynew file mode 100644index 000000000..97062d5f1--- /dev/null+++ b/tests/unit/pipelex/plugins/test_inference_backend_coverage.py@@ -0,0 +1,103 @@+"""Registry round-trip coverage for the built-in inference backends: every (family, sdk)+that the old per-family worker-factory ``match`` statements handled must resolve through the+registry built from ``BUILTIN_PLUGINS``, and the cross-family vendors must register into every+family they serve from their single ``register`` call.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.plugins.builtins import BUILTIN_PLUGINS+from pipelex.plugins.inference_backend_registry import InferenceBackendRegistry, InferenceFamily+from pipelex.plugins.registrar import PluginRegistrar++if TYPE_CHECKING:+ from pipelex.system.configuration.configs import PipelexConfig+++def _build_registry() -> InferenceBackendRegistry:+ # No builtin's register() reads config: Temporal — the only plugin that did, via+ # ``temporal.is_enabled`` — now ships as the external pipelex-temporal dist. A bare+ # stub config suffices.+ stub_config = cast("PipelexConfig", SimpleNamespace())+ registrar = PluginRegistrar(config=stub_config)+ for plugin in BUILTIN_PLUGINS:+ plugin.register(registrar)+ return InferenceBackendRegistry(registrar.inference_backends)+++# Every (family, sdk) the migrated worker factories used to dispatch on via a `match`.+EXPECTED_BACKENDS: list[tuple[InferenceFamily, str]] = [+ # LLM+ (InferenceFamily.LLM, "openai"),+ (InferenceFamily.LLM, "azure_openai"),+ (InferenceFamily.LLM, "openai_responses"),+ (InferenceFamily.LLM, "azure_openai_responses"),+ (InferenceFamily.LLM, "gateway_completions"),+ (InferenceFamily.LLM, "gateway_responses"),+ (InferenceFamily.LLM, "portkey_completions"),+ (InferenceFamily.LLM, "portkey_responses"),+ (InferenceFamily.LLM, "anthropic"),+ (InferenceFamily.LLM, "bedrock_anthropic"),+ (InferenceFamily.LLM, "mistral"),+ (InferenceFamily.LLM, "bedrock_boto3"),+ (InferenceFamily.LLM, "bedrock_aioboto3"),+ (InferenceFamily.LLM, "google"),+ # IMG_GEN+ (InferenceFamily.IMG_GEN, "gateway_img_gen"),+ (InferenceFamily.IMG_GEN, "gateway_completions"),+ (InferenceFamily.IMG_GEN, "openai_img_gen"),+ (InferenceFamily.IMG_GEN, "blackboxai_img_gen"),+ (InferenceFamily.IMG_GEN, "openrouter_img_gen"),+ (InferenceFamily.IMG_GEN, "fal"),+ (InferenceFamily.IMG_GEN, "huggingface_img_gen"),+ (InferenceFamily.IMG_GEN, "azure_rest_img_gen"),+ (InferenceFamily.IMG_GEN, "google"),+ # EXTRACT+ (InferenceFamily.EXTRACT, "gateway_extract"),+ (InferenceFamily.EXTRACT, "mistral"),+ (InferenceFamily.EXTRACT, "pypdfium2"),+ (InferenceFamily.EXTRACT, "docling_sdk"),+ (InferenceFamily.EXTRACT, "linkup_fetch"),+ # SEARCH+ (InferenceFamily.SEARCH, "linkup"),+ (InferenceFamily.SEARCH, "gateway_search"),+]+++class TestInferenceBackendCoverage:+ @pytest.mark.parametrize(("family", "sdk"), EXPECTED_BACKENDS, ids=[f"{family}:{sdk}" for family, sdk in EXPECTED_BACKENDS])+ def test_backend_is_registered(self, family: InferenceFamily, sdk: str) -> None:+ """Each migrated (family, sdk) resolves to a make_worker via the built-in registry."""+ registry = _build_registry()+ assert registry.has(family=family, sdk=sdk), f"missing backend {family}:{sdk}"+ assert registry.lookup(family=family, sdk=sdk) is not None++ @pytest.mark.parametrize(+ ("vendor_families", "vendor"),+ [+ pytest.param([(InferenceFamily.LLM, "mistral"), (InferenceFamily.EXTRACT, "mistral")], "mistral", id="mistral"),+ pytest.param([(InferenceFamily.LLM, "google"), (InferenceFamily.IMG_GEN, "google")], "google", id="google"),+ pytest.param([(InferenceFamily.LLM, "openai"), (InferenceFamily.IMG_GEN, "openai_img_gen")], "openai", id="openai"),+ pytest.param([(InferenceFamily.EXTRACT, "linkup_fetch"), (InferenceFamily.SEARCH, "linkup")], "linkup", id="linkup"),+ pytest.param(+ [+ (InferenceFamily.LLM, "gateway_completions"),+ (InferenceFamily.IMG_GEN, "gateway_img_gen"),+ (InferenceFamily.EXTRACT, "gateway_extract"),+ (InferenceFamily.SEARCH, "gateway_search"),+ ],+ "gateway",+ id="gateway",+ ),+ ],+ )+ def test_cross_family_vendor_registers_into_all_its_families(self, vendor_families: list[tuple[InferenceFamily, str]], vendor: str) -> None:+ """A single vendor plugin's register() contributes backends across every family it serves."""+ registry = _build_registry()+ for family, sdk in vendor_families:+ assert registry.has(family=family, sdk=sdk), f"{vendor} did not register {family}:{sdk}"diff --git a/tests/unit/pipelex/plugins/test_linkup_plugin_guard.py b/tests/unit/pipelex/plugins/test_linkup_plugin_guard.pynew file mode 100644index 000000000..497cab106--- /dev/null+++ b/tests/unit/pipelex/plugins/test_linkup_plugin_guard.py@@ -0,0 +1,43 @@+from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.linkup.linkup_plugin import LinkupPlugin+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.system.exceptions import MissingDependencyError++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++ from pipelex.cogt.model_backends.backend import InferenceBackend+ from pipelex.cogt.model_backends.model_spec import InferenceModelSpec+ from pipelex.plugins.sdk_client_registry import SdkClientRegistry+ from pipelex.system.configuration.configs import PipelexConfig++REGISTRY_MODULE = "pipelex.plugins.inference_backend_registry"+++class TestLinkupSearchWorkerGuard:+ def test_missing_linkup_sdk_raises_missing_dependency(self, mocker: MockerFixture) -> None:+ """The registered Linkup *search* backend must guard the optional linkup SDK with require_sdk *before*+ importing its top-level-`linkup`-importing worker, so an absent extra surfaces as MissingDependencyError+ (with the pipelex[linkup] hint), not a raw ModuleNotFoundError — matching the extract backend's behavior.+ """+ mocker.patch(f"{REGISTRY_MODULE}.importlib.util.find_spec", return_value=None)++ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace()))+ LinkupPlugin().register(registrar)+ make_search_worker = registrar.inference_backends[InferenceFamily.SEARCH, "linkup"]++ with pytest.raises(MissingDependencyError) as exc_info:+ make_search_worker(+ inference_model=cast("InferenceModelSpec", None),+ backend=cast("InferenceBackend", None),+ sdk_clients=cast("SdkClientRegistry", None),+ reporting_delegate=None,+ )+ assert "pipelex[linkup]" in str(exc_info.value)diff --git a/tests/unit/pipelex/plugins/test_model_lister_coverage.py b/tests/unit/pipelex/plugins/test_model_lister_coverage.pynew file mode 100644index 000000000..043fc78d2--- /dev/null+++ b/tests/unit/pipelex/plugins/test_model_lister_coverage.py@@ -0,0 +1,72 @@+"""Registry round-trip coverage for the built-in model listers: every ``sdk`` that the old+``ModelLister.list_models`` ``match`` statement dispatched on must resolve to a lister through+the registry built from ``BUILTIN_PLUGINS``, and an SDK with no lister must yield a soft miss+(``get_optional`` returns ``None``) so the ``list-models`` loop reports it as unsupported.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.plugins.builtins import BUILTIN_PLUGINS+from pipelex.plugins.model_lister_registry import ModelListerRegistry+from pipelex.plugins.registrar import PluginRegistrar++if TYPE_CHECKING:+ from pipelex.system.configuration.configs import PipelexConfig+++def _build_registry() -> ModelListerRegistry:+ # No builtin's register() reads config: Temporal — the only plugin that did, via+ # ``temporal.is_enabled`` — now ships as the external pipelex-temporal dist. A bare+ # stub config suffices.+ stub_config = cast("PipelexConfig", SimpleNamespace())+ registrar = PluginRegistrar(config=stub_config)+ for plugin in BUILTIN_PLUGINS:+ plugin.register(registrar)+ return ModelListerRegistry(registrar.model_listers)+++# Every ``sdk`` the old ``model_lists.py`` ``match`` statement handled with a real lister arm.+EXPECTED_LISTER_SDKS: list[str] = [+ "openai",+ "azure_openai",+ "openai_responses",+ "azure_openai_responses",+ "anthropic",+ "mistral",+ "google",+ "bedrock",+ "bedrock_aioboto3",+]+++class TestModelListerCoverage:+ @pytest.mark.parametrize("sdk", EXPECTED_LISTER_SDKS)+ def test_registered_sdk_resolves_to_a_lister(self, sdk: str) -> None:+ registry = _build_registry()+ lister = registry.get_optional(sdk=sdk)+ assert lister is not None, f"expected a model lister registered for sdk '{sdk}'"+ assert callable(lister)+ assert registry.has(sdk=sdk)++ def test_registry_exposes_exactly_the_expected_sdks(self) -> None:+ registry = _build_registry()+ assert sorted(registry.sdks) == sorted(EXPECTED_LISTER_SDKS)++ @pytest.mark.parametrize(+ "sdk",+ [+ "gateway_completions", # an LLM/img-gen SDK that never supported remote listing+ "fal", # an image-gen-only SDK with no lister+ "linkup", # a search/extract SDK with no lister+ "totally_unknown_sdk",+ ],+ )+ def test_sdk_without_a_lister_is_a_soft_miss(self, sdk: str) -> None:+ registry = _build_registry()+ assert registry.get_optional(sdk=sdk) is None+ assert not registry.has(sdk=sdk)diff --git a/tests/unit/pipelex/plugins/test_plugin_discovery.py b/tests/unit/pipelex/plugins/test_plugin_discovery.pynew file mode 100644index 000000000..a6884e1a4--- /dev/null+++ b/tests/unit/pipelex/plugins/test_plugin_discovery.py@@ -0,0 +1,208 @@+"""Contract-conformance tests for plugin discovery: protocol satisfaction, fail-loud+conflict/version policy, the side-effect-free (idempotent) register invariant, and the+plugins.disabled denylist.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.plugins.contract import PLUGIN_API_VERSION, PipelexPlugin+from pipelex.plugins.discovery import build_registrar+from pipelex.plugins.exceptions import (+ BrokenPluginError,+ CoreUnconditionalPluginDisabledError,+ DuplicateInferenceBackendError,+ DuplicateOrchestratorError,+ HubSlotAlreadyClaimedError,+ PluginApiVersionMismatchError,+)+from pipelex.plugins.inference_backend_registry import InferenceFamily+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar, PluginStatus++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++ from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract+ from pipelex.plugins.orchestrator_registry import OrchestratorProtocol+ from pipelex.system.configuration.configs import PipelexConfig++DISCOVERY_MODULE = "pipelex.plugins.discovery"+++def _noop_make_worker(**_kwargs: object) -> InferenceWorkerAbstract:+ msg = "make_worker is never invoked by discovery tests"+ raise AssertionError(msg)+++def _fake_config(disabled: list[str]) -> PipelexConfig:+ # Discovery reads ``config.plugins.disabled`` to apply the denylist. No builtin's+ # register() reads config: Temporal — the only plugin that did, via ``temporal.is_enabled``+ # — now ships as the external pipelex-temporal dist.+ return cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=disabled)))+++class _InferencePlugin:+ def __init__(self, *, name: str, sdk: str, targets_api: int = PLUGIN_API_VERSION):+ self.name = name+ self.sdk = sdk+ self.targets_api = targets_api++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk=self.sdk, make_worker=_noop_make_worker)+++class _OrchestratorPlugin:+ def __init__(self, *, name: str):+ self.name = name+ self.targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_orchestrator(mode="temporal", orchestrator=cast("OrchestratorProtocol", object()))+++class _SlotPlugin:+ def __init__(self, *, name: str):+ self.name = name+ self.targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.claim_content_generator(object)+++class _BrokenPlugin:+ name = "broken"+ targets_api = PLUGIN_API_VERSION++ def register(self, _registrar: PluginRegistrar) -> None:+ msg = "kaboom in register"+ raise ValueError(msg)+++def _patch_builtins(mocker: MockerFixture, plugins: list[object]) -> None:+ mocker.patch(f"{DISCOVERY_MODULE}.BUILTIN_PLUGINS", plugins)+ mocker.patch(f"{DISCOVERY_MODULE}._external_entry_points", return_value=[])+++class TestPluginDiscovery:+ def test_builtin_plugins_satisfy_the_protocol(self) -> None:+ """Every shipped built-in is a structural PipelexPlugin; a bare object is not."""+ from pipelex.plugins.builtins import BUILTIN_PLUGINS # noqa: PLC0415++ for plugin in BUILTIN_PLUGINS:+ assert isinstance(plugin, PipelexPlugin)+ assert not isinstance(object(), PipelexPlugin)++ def test_registry_round_trip(self, mocker: MockerFixture) -> None:+ """A registered backend is retrievable by (family, sdk) and is the exact callable."""+ _patch_builtins(mocker, [_InferencePlugin(name="synthetic", sdk="synthetic_sdk")])+ registrar = build_registrar(config=_fake_config([]))++ from pipelex.plugins.inference_backend_registry import InferenceBackendRegistry # noqa: PLC0415++ registry = InferenceBackendRegistry(registrar.inference_backends)+ assert registry.lookup(family=InferenceFamily.LLM, sdk="synthetic_sdk") is _noop_make_worker++ def test_version_mismatch_fails_loud(self, mocker: MockerFixture) -> None:+ _patch_builtins(mocker, [_InferencePlugin(name="from_the_future", sdk="future_sdk", targets_api=PLUGIN_API_VERSION + 1)])+ with pytest.raises(PluginApiVersionMismatchError) as exc_info:+ build_registrar(config=_fake_config([]))+ assert "from_the_future" in str(exc_info.value)++ def test_duplicate_inference_backend_names_both(self, mocker: MockerFixture) -> None:+ _patch_builtins(mocker, [_InferencePlugin(name="alpha", sdk="dup"), _InferencePlugin(name="beta", sdk="dup")])+ with pytest.raises(DuplicateInferenceBackendError) as exc_info:+ build_registrar(config=_fake_config([]))+ message = str(exc_info.value)+ assert "alpha" in message+ assert "beta" in message++ def test_duplicate_orchestrator_names_both(self, mocker: MockerFixture) -> None:+ _patch_builtins(mocker, [_OrchestratorPlugin(name="orch_a"), _OrchestratorPlugin(name="orch_b")])+ with pytest.raises(DuplicateOrchestratorError) as exc_info:+ build_registrar(config=_fake_config([]))+ message = str(exc_info.value)+ assert "orch_a" in message+ assert "orch_b" in message++ def test_double_claimed_slot_names_both(self, mocker: MockerFixture) -> None:+ _patch_builtins(mocker, [_SlotPlugin(name="claimer_1"), _SlotPlugin(name="claimer_2")])+ with pytest.raises(HubSlotAlreadyClaimedError) as exc_info:+ build_registrar(config=_fake_config([]))+ message = str(exc_info.value)+ assert "claimer_1" in message+ assert "claimer_2" in message++ def test_broken_plugin_register_wrapped(self, mocker: MockerFixture) -> None:+ _patch_builtins(mocker, [_BrokenPlugin()])+ with pytest.raises(BrokenPluginError) as exc_info:+ build_registrar(config=_fake_config([]))+ assert "broken" in str(exc_info.value)+ assert "kaboom" in str(exc_info.value)++ def test_broken_external_entry_point_wrapped(self, mocker: MockerFixture) -> None:+ """An entry point that fails to load is wrapped in BrokenPluginError naming the entry point."""++ def _explode() -> object:+ msg = "no such module"+ raise ImportError(msg)++ bad_entry_point = SimpleNamespace(name="bad_ep", load=_explode)+ mocker.patch(f"{DISCOVERY_MODULE}.BUILTIN_PLUGINS", [])+ mocker.patch(f"{DISCOVERY_MODULE}._external_entry_points", return_value=[bad_entry_point])+ with pytest.raises(BrokenPluginError) as exc_info:+ build_registrar(config=_fake_config([]))+ assert "bad_ep" in str(exc_info.value)++ def test_disabled_broken_external_entry_point_is_skipped_before_load(self, mocker: MockerFixture) -> None:+ """A broken external entry point listed in plugins.disabled is skipped by its entry-point name+ *before* load(), so the denylist can recover from a bad installed plugin instead of raising BrokenPluginError.+ """++ def _explode() -> object:+ msg = "no such module"+ raise ImportError(msg)++ bad_entry_point = SimpleNamespace(name="bad_ep", load=_explode)+ mocker.patch(f"{DISCOVERY_MODULE}.BUILTIN_PLUGINS", [])+ mocker.patch(f"{DISCOVERY_MODULE}._external_entry_points", return_value=[bad_entry_point])++ registrar = build_registrar(config=_fake_config(["bad_ep"]))++ disabled_discovery = next(discovery for discovery in registrar.discoveries if discovery.name == "bad_ep")+ assert disabled_discovery.status == PluginStatus.DISABLED+ assert disabled_discovery.origin == PluginOrigin.EXTERNAL++ def test_build_registrar_is_idempotent(self, mocker: MockerFixture) -> None:+ """Two builds produce equivalent registrars — register is side-effect-free (D3)."""+ _patch_builtins(mocker, [_InferencePlugin(name="synthetic", sdk="synthetic_sdk")])+ first = build_registrar(config=_fake_config([]))+ second = build_registrar(config=_fake_config([]))+ assert set(first.inference_backends) == set(second.inference_backends)+ assert [discovery.name for discovery in first.discoveries] == [discovery.name for discovery in second.discoveries]++ def test_disabled_plugin_is_skipped(self, mocker: MockerFixture) -> None:+ _patch_builtins(mocker, [_InferencePlugin(name="optional", sdk="optional_sdk")])+ registrar = build_registrar(config=_fake_config(["optional"]))+ assert (InferenceFamily.LLM, "optional_sdk") not in registrar.inference_backends+ disabled_discovery = next(discovery for discovery in registrar.discoveries if discovery.name == "optional")+ assert disabled_discovery.status == PluginStatus.DISABLED++ def test_disabling_core_unconditional_plugin_raises(self) -> None:+ """Denylisting a plugin core requires unconditionally (e.g. openai) is a startup error."""+ with pytest.raises(CoreUnconditionalPluginDisabledError) as exc_info:+ build_registrar(config=_fake_config(["openai"]))+ assert "openai" in str(exc_info.value)++ def test_discoveries_describe_builtins(self) -> None:+ """build_registrar records each built-in with origin/status/contributions for `plugins list`."""+ registrar = build_registrar(config=_fake_config([]))+ by_name = {discovery.name: discovery for discovery in registrar.discoveries}+ assert "openai" in by_name+ openai_discovery = by_name["openai"]+ assert openai_discovery.origin == PluginOrigin.BUILTIN+ assert openai_discovery.status == PluginStatus.REGISTERED+ assert openai_discovery.contributions # registered at least one inference backenddiff --git a/tests/unit/pipelex/plugins/test_require_sdk.py b/tests/unit/pipelex/plugins/test_require_sdk.pynew file mode 100644index 000000000..914af0064--- /dev/null+++ b/tests/unit/pipelex/plugins/test_require_sdk.py@@ -0,0 +1,59 @@+from __future__ import annotations++from typing import TYPE_CHECKING++import pytest++from pipelex.plugins.inference_backend_registry import require_sdk+from pipelex.system.exceptions import MissingDependencyError++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++REGISTRY_MODULE = "pipelex.plugins.inference_backend_registry"+++class TestRequireSdk:+ def test_present_spec_does_not_raise(self, mocker: MockerFixture) -> None:+ """An importable spec is a no-op: find_spec returns a spec object."""+ mocker.patch(f"{REGISTRY_MODULE}.importlib.util.find_spec", return_value=object())+ require_sdk(spec="anything", extra="anything", msg="should not raise")++ def test_missing_spec_raises_missing_dependency(self, mocker: MockerFixture) -> None:+ """A spec that resolves to None raises MissingDependencyError with the install hint."""+ mocker.patch(f"{REGISTRY_MODULE}.importlib.util.find_spec", return_value=None)+ with pytest.raises(MissingDependencyError) as exc_info:+ require_sdk(spec="ghost", extra="ghost", msg="install it")+ message = str(exc_info.value)+ assert "ghost" in message+ assert "pipelex[ghost]" in message++ def test_dotted_spec_with_absent_parent_raises_missing_dependency(self, mocker: MockerFixture) -> None:+ """find_spec imports the dotted parent first; an absent parent raises ModuleNotFoundError,+ which must surface as MissingDependencyError (with the pipelex[<extra>] hint), not a raw import error.+ """+ mocker.patch(+ f"{REGISTRY_MODULE}.importlib.util.find_spec",+ side_effect=ModuleNotFoundError("No module named 'google'"),+ )+ with pytest.raises(MissingDependencyError) as exc_info:+ require_sdk(spec="google.genai", dependency_name="google-genai", extra="google", msg="install it")+ message = str(exc_info.value)+ assert "google-genai" in message+ assert "pipelex[google]" in message++ def test_multi_spec_partial_missing_names_only_absent(self, mocker: MockerFixture) -> None:+ """With several required specs and no explicit dependency_name, the error names ONLY the+ absent spec(s) — a user who already has one of them must not be told to (re)install it.+ """++ def fake_find_spec(name: str) -> object | None:+ return None if name == "aioboto3" else object()++ mocker.patch(f"{REGISTRY_MODULE}.importlib.util.find_spec", side_effect=fake_find_spec)+ with pytest.raises(MissingDependencyError) as exc_info:+ require_sdk(spec=["boto3", "aioboto3"], extra="bedrock", msg="install it")+ # boto3 is present, so only aioboto3 should be named (the trailing comma proves the old+ # joined "boto3,aioboto3" form is gone — "boto3" alone is a substring of "aioboto3").+ assert exc_info.value.dependency_name == "aioboto3"+ assert "boto3," not in str(exc_info.value)diff --git a/tests/unit/pipelex/plugins/test_transport_retry_wiring.py b/tests/unit/pipelex/plugins/test_transport_retry_wiring.pyindex e5fdedcda..2142fbea6 100644--- a/tests/unit/pipelex/plugins/test_transport_retry_wiring.py+++ b/tests/unit/pipelex/plugins/test_transport_retry_wiring.py@@ -40,10 +40,10 @@ class TestTransportRetryWiring:return_value=_config_with(mocker, 5),)mock_client = mocker.patch("pipelex.plugins.anthropic.anthropic_factory.AsyncAnthropic")- plugin = mocker.MagicMock()- plugin.sdk = "anthropic"+ model_handle = mocker.MagicMock()+ model_handle.sdk = "anthropic"- AnthropicFactory.make_anthropic_client(plugin=plugin, backend=mocker.MagicMock())+ AnthropicFactory.make_anthropic_client(model_handle=model_handle, backend=mocker.MagicMock())mock_client.assert_called_once()assert mock_client.call_args.kwargs["max_retries"] == 5@@ -54,10 +54,10 @@ class TestTransportRetryWiring:return_value=_config_with(mocker, 7),)mock_client = mocker.patch("openai.AsyncOpenAI")- plugin = mocker.MagicMock()- plugin.sdk = "openai"+ model_handle = mocker.MagicMock()+ model_handle.sdk = "openai"- OpenAIClientFactory.make_openai_client(plugin=plugin, backend=mocker.MagicMock())+ OpenAIClientFactory.make_openai_client(model_handle=model_handle, backend=mocker.MagicMock())mock_client.assert_called_once()assert mock_client.call_args.kwargs["max_retries"] == 7@@ -114,7 +114,7 @@ class TestTransportRetryWiring:mocker.patch("pipelex.plugins.portkey.portkey_completions_factory.PortkeyOpenAISdkVariant.is_completions", return_value=True)mock_client = mocker.patch("openai.AsyncOpenAI")- PortkeyCompletionsFactory.make_portkey_openai_client_for_completions(plugin=mocker.MagicMock(), backend=mocker.MagicMock())+ PortkeyCompletionsFactory.make_portkey_openai_client_for_completions(model_handle=mocker.MagicMock(), backend=mocker.MagicMock())mock_client.assert_called_once()assert mock_client.call_args.kwargs["max_retries"] == 6@@ -130,7 +130,7 @@ class TestTransportRetryWiring:mocker.patch("pipelex.plugins.portkey.portkey_responses_factory.PortkeyOpenAISdkVariant.is_responses", return_value=True)mock_client = mocker.patch("openai.AsyncOpenAI")- PortkeyResponsesFactory.make_portkey_openai_client_for_responses(plugin=mocker.MagicMock(), backend=mocker.MagicMock())+ PortkeyResponsesFactory.make_portkey_openai_client_for_responses(model_handle=mocker.MagicMock(), backend=mocker.MagicMock())mock_client.assert_called_once()assert mock_client.call_args.kwargs["max_retries"] == 8@@ -146,7 +146,7 @@ class TestTransportRetryWiring:mocker.patch("pipelex.plugins.gateway.gateway_completions_factory.GatewayOpenAISdkVariant.is_completions", return_value=True)mock_client = mocker.patch("openai.AsyncOpenAI")- GatewayCompletionsFactory.make_portkey_openai_client_for_completions(plugin=mocker.MagicMock(), backend=mocker.MagicMock())+ GatewayCompletionsFactory.make_portkey_openai_client_for_completions(model_handle=mocker.MagicMock(), backend=mocker.MagicMock())mock_client.assert_called_once()assert mock_client.call_args.kwargs["max_retries"] == 4@@ -162,7 +162,7 @@ class TestTransportRetryWiring:mocker.patch("pipelex.plugins.gateway.gateway_responses_factory.GatewayOpenAISdkVariant.is_responses", return_value=True)mock_client = mocker.patch("openai.AsyncOpenAI")- GatewayResponsesFactory.make_portkey_openai_client_for_responses(plugin=mocker.MagicMock(), backend=mocker.MagicMock())+ GatewayResponsesFactory.make_portkey_openai_client_for_responses(model_handle=mocker.MagicMock(), backend=mocker.MagicMock())mock_client.assert_called_once()assert mock_client.call_args.kwargs["max_retries"] == 9diff --git a/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.py b/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.pyindex f81dec116..13ad69322 100644--- a/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.py+++ b/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.py@@ -2,7 +2,7 @@``_is_in_temporal_activity`` must detect the activity context WITHOUT importing temporalio:a module-level import would put the entire temporalio extra (Rust bridge, protobuf, ~130ms)-on every pipelex boot's critical path wherever ``pipelex[temporal]`` is installed — including+on every pipelex boot's critical path wherever ``pipelex-temporal`` is installed — includingprocesses that never touch Temporal (CLI validate runs, direct-mode servers, the hostedrunner image). The gate sniffs ``sys.modules`` instead: a process that never imported``temporalio.activity`` cannot be inside a Temporal activity, because the activity context isdiff --git a/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py b/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.pyindex 4955cb3df..089f7c018 100644--- a/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py+++ b/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py@@ -52,7 +52,7 @@ class TestHydrateWorkingMemory:working_memory = WorkingMemory()working_memory.root["greeting"] = _make_text_stuff("greeting", "Hello, world!")- raw = working_memory.dump_for_temporal()+ raw = working_memory.dump_for_transport()hydrated = hydrate_working_memory(raw)assert "greeting" in hydrated.root@@ -64,7 +64,7 @@ class TestHydrateWorkingMemory:def test_hydrate_empty(self) -> None:"""An empty WorkingMemory raw dict hydrates to empty WorkingMemory."""working_memory = WorkingMemory()- raw = working_memory.dump_for_temporal()+ raw = working_memory.dump_for_transport()hydrated = hydrate_working_memory(raw)@@ -77,14 +77,14 @@ class TestHydrateWorkingMemory:working_memory.root["greeting"] = _make_text_stuff("greeting", "Hello!")working_memory.aliases["main_stuff"] = "greeting"- raw = working_memory.dump_for_temporal()+ raw = working_memory.dump_for_transport()hydrated = hydrate_working_memory(raw)assert hydrated.aliases == {"main_stuff": "greeting"}assert "greeting" in hydrated.rootdef test_hydrate_list_content_round_trip(self) -> None:- """ListContent survives dump_for_temporal/hydrate round-trip as a plain list."""+ """ListContent survives dump_for_transport/hydrate round-trip as a plain list."""working_memory = WorkingMemory()list_stuff = Stuff(stuff_code="test",@@ -100,7 +100,7 @@ class TestHydrateWorkingMemory:)working_memory.root["colors"] = list_stuff- raw = working_memory.dump_for_temporal()+ raw = working_memory.dump_for_transport()# Verify the Temporal format uses a plain list, not {"items": [...]}assert isinstance(raw["root"]["colors"]["content"], list)@@ -127,8 +127,8 @@ class TestHydrateWorkingMemory:assert list_content.items[1].text == "red"assert list_content.items[2].text == "green"- def test_dump_for_temporal_list_items_carry_pipelex_markers_only(self) -> None:- """dump_for_temporal must emit pipelex-private type markers, never kajson's __class__/__module__.+ def test_dump_for_transport_list_items_carry_pipelex_markers_only(self) -> None:+ """dump_for_transport must emit pipelex-private type markers, never kajson's __class__/__module__.This is the structural guard against the cross-process decode bug: if theseitems carried __class__ keys, kajson's universal decoder would try to bind@@ -144,7 +144,7 @@ class TestHydrateWorkingMemory:)working_memory.root["items"] = list_stuff- raw = working_memory.dump_for_temporal()+ raw = working_memory.dump_for_transport()serialized_items = cast("list[dict[str, Any]]", raw["root"]["items"]["content"])assert isinstance(serialized_items, list)diff --git a/tests/unit/pipelex/runtime_bridge/primitives/test_pipe_classification.py b/tests/unit/pipelex/runtime_bridge/primitives/test_pipe_classification.pydeleted file mode 100644index c391c0964..000000000--- a/tests/unit/pipelex/runtime_bridge/primitives/test_pipe_classification.py+++ /dev/null@@ -1,82 +0,0 @@-"""Unit tests for ``pipelex.runtime_bridge.primitives.pipe_classification``.--The classifier answers a single question: "should this pipe be dispatched-as a child workflow (controller) or as an activity (leaf)?". Both-``pipelex.temporal`` and ``pipelex_mistralai_workflows.primitives`` rely-on it to make that branching decision (see `wip/mistral-native-plan.md`-§1.2 row classification), so the test pins down all four controllers and-several leaf operators.--We use ``MagicMock(spec=Cls)`` because ``isinstance(mock, AnyBase)``-respects the spec's MRO — that's all the classifier checks. Constructing-the real Pydantic models would require a populated library and-domain/concept resolution that is irrelevant to this unit.-"""--from typing import TYPE_CHECKING-from unittest.mock import MagicMock--from pipelex.pipe_controllers.batch.pipe_batch import PipeBatch-from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition-from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel-from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence-from pipelex.pipe_operators.compose.pipe_compose import PipeCompose-from pipelex.pipe_operators.func.pipe_func import PipeFunc-from pipelex.pipe_operators.img_gen.pipe_img_gen import PipeImgGen-from pipelex.pipe_operators.llm.pipe_llm import PipeLLM-from pipelex.runtime_bridge.primitives.pipe_classification import (- is_controller_pipe,- is_leaf_pipe,-)--if TYPE_CHECKING:- from pipelex.core.pipes.pipe_abstract import PipeAbstract---def _spec_pipe(cls: "type[PipeAbstract]") -> "PipeAbstract":- """Return a MagicMock that ``isinstance``-passes as ``cls``."""- return MagicMock(spec=cls)---class TestIsControllerPipe:- def test_pipe_sequence_is_controller(self) -> None:- pipe = _spec_pipe(PipeSequence)- assert is_controller_pipe(pipe) is True- assert is_leaf_pipe(pipe) is False-- def test_pipe_batch_is_controller(self) -> None:- pipe = _spec_pipe(PipeBatch)- assert is_controller_pipe(pipe) is True- assert is_leaf_pipe(pipe) is False-- def test_pipe_condition_is_controller(self) -> None:- pipe = _spec_pipe(PipeCondition)- assert is_controller_pipe(pipe) is True- assert is_leaf_pipe(pipe) is False-- def test_pipe_parallel_is_controller(self) -> None:- pipe = _spec_pipe(PipeParallel)- assert is_controller_pipe(pipe) is True- assert is_leaf_pipe(pipe) is False---class TestIsLeafPipe:- def test_pipe_llm_is_leaf(self) -> None:- pipe = _spec_pipe(PipeLLM)- assert is_leaf_pipe(pipe) is True- assert is_controller_pipe(pipe) is False-- def test_pipe_compose_is_leaf(self) -> None:- pipe = _spec_pipe(PipeCompose)- assert is_leaf_pipe(pipe) is True- assert is_controller_pipe(pipe) is False-- def test_pipe_func_is_leaf(self) -> None:- pipe = _spec_pipe(PipeFunc)- assert is_leaf_pipe(pipe) is True- assert is_controller_pipe(pipe) is False-- def test_pipe_img_gen_is_leaf(self) -> None:- pipe = _spec_pipe(PipeImgGen)- assert is_leaf_pipe(pipe) is True- assert is_controller_pipe(pipe) is Falsediff --git a/tests/unit/pipelex/runtime_bridge/primitives/test_scoped_library_best_effort_teardown.py b/tests/unit/pipelex/runtime_bridge/primitives/test_scoped_library_best_effort_teardown.pydeleted file mode 100644index 4d5c5b6e4..000000000--- a/tests/unit/pipelex/runtime_bridge/primitives/test_scoped_library_best_effort_teardown.py+++ /dev/null@@ -1,58 +0,0 @@-"""Best-effort teardown contract of ``scoped_library_for_crate``.--The teardown in the ``finally`` must neither mask the body's in-flight exception nor-fail an otherwise-successful call: ``LibraryManager`` forgets the entry pop-first even-when the library's own teardown raises, so suppressing the teardown error is safe.-"""--import pytest-from pytest_mock import MockerFixture--from pipelex.hub import get_library_manager-from pipelex.libraries.library_crate import LibraryCrate-from pipelex.runtime_bridge.primitives.scoped_library import scoped_library_for_crate---def _run_scope_with_failing_body(fake_crate: LibraryCrate, captured_library_ids: list[str]) -> None:- with scoped_library_for_crate(fake_crate, library_id_prefix="test_scope") as library_id:- assert library_id is not None- captured_library_ids.append(library_id)- msg = "body boom"- raise ValueError(msg)---class TestScopedLibraryBestEffortTeardown:- def test_teardown_failure_does_not_fail_successful_body(self, mocker: MockerFixture) -> None:- fake_crate = mocker.MagicMock(spec=LibraryCrate)- library_manager = get_library_manager()- mocker.patch.object(library_manager, "load_from_crate", return_value=None)- teardown_mock = mocker.patch.object(library_manager, "teardown", side_effect=RuntimeError("teardown boom"))-- scoped_library_id: str | None = None- with scoped_library_for_crate(fake_crate, library_id_prefix="test_scope") as library_id:- assert library_id is not None- scoped_library_id = library_id-- assert teardown_mock.call_count == 1-- # The mocked teardown left the scoped entry registered on the singleton — clean it up.- mocker.stopall()- assert scoped_library_id is not None- library_manager.teardown(library_id=scoped_library_id)-- def test_teardown_failure_does_not_mask_body_exception(self, mocker: MockerFixture) -> None:- fake_crate = mocker.MagicMock(spec=LibraryCrate)- library_manager = get_library_manager()- mocker.patch.object(library_manager, "load_from_crate", return_value=None)- teardown_mock = mocker.patch.object(library_manager, "teardown", side_effect=RuntimeError("teardown boom"))-- captured_library_ids: list[str] = []- with pytest.raises(ValueError, match="body boom"):- _run_scope_with_failing_body(fake_crate, captured_library_ids)-- assert teardown_mock.call_count == 1-- # The mocked teardown left the scoped entry registered on the singleton — clean it up.- mocker.stopall()- assert len(captured_library_ids) == 1- library_manager.teardown(library_id=captured_library_ids[0])diff --git a/tests/unit/pipelex/runtime_bridge/primitives/test_submitter_hydration.py b/tests/unit/pipelex/runtime_bridge/primitives/test_submitter_hydration.pydeleted file mode 100644index c952ec481..000000000--- a/tests/unit/pipelex/runtime_bridge/primitives/test_submitter_hydration.py+++ /dev/null@@ -1,120 +0,0 @@-import pytest-from pytest_mock import MockerFixture--from pipelex.core.concepts.concept import Concept-from pipelex.core.domains.domain import SpecialDomain-from pipelex.core.memory.working_memory import WorkingMemory-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.stuff import Stuff-from pipelex.core.stuffs.text_content import TextContent-from pipelex.hub import get_class_registry, get_library_manager-from pipelex.pipe_run.exceptions import PipeJobError-from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate---def _text_concept() -> Concept:- return Concept(- code="Text",- domain_code=SpecialDomain.NATIVE,- description="Plain text",- structure_class_name="TextContent",- )---def _pipe_output_with_text(name: str, text: str) -> PipeOutput:- working_memory = WorkingMemory()- working_memory.root[name] = Stuff(- stuff_code="test",- stuff_name=name,- concept=_text_concept(),- content=TextContent(text=text),- )- raw = working_memory.dump_for_temporal()- return PipeOutput(working_memory=WorkingMemory(), working_memory_raw=raw)---class TestRehydratePipeOutputWithCrate:- @pytest.fixture(autouse=True)- def _register_text_content(self) -> None:- registry = get_class_registry()- if not registry.has_class(name="TextContent"):- registry.register_class(TextContent)-- def test_returns_same_instance_when_raw_is_none(self, mocker: MockerFixture) -> None:- pipe_output = PipeOutput(working_memory=WorkingMemory(), working_memory_raw=None)- open_spy = mocker.spy(get_library_manager(), "open_library")-- result = rehydrate_pipe_output_with_crate(pipe_output, library_crate=None)-- assert result is pipe_output- assert result.working_memory_raw is None- assert open_spy.call_count == 0-- def test_no_crate_uses_active_registry(self) -> None:- """With library_crate=None and only built-in concept, hydration uses the active registry."""- pipe_output = _pipe_output_with_text("greeting", "Hello, world!")-- result = rehydrate_pipe_output_with_crate(pipe_output, library_crate=None)-- assert result is pipe_output- assert pipe_output.working_memory_raw is None- stuff = pipe_output.working_memory.root["greeting"]- assert isinstance(stuff.content, TextContent)- assert stuff.content.text == "Hello, world!"-- def test_no_crate_propagates_pipejob_error_for_unknown_class(self) -> None:- """Without a crate, an unknown structure_class_name raises (existing failure mode)."""- raw = {- "root": {- "bad_stuff": {- "stuff_code": "test",- "stuff_name": "bad_stuff",- "concept": {- "code": "NonExistent",- "domain_code": "native",- "description": "missing",- "structure_class_name": "AbsolutelyNonExistentClass",- },- "content": {"text": "x"},- },- },- "aliases": {},- }- pipe_output = PipeOutput(working_memory=WorkingMemory(), working_memory_raw=raw)-- with pytest.raises(PipeJobError):- rehydrate_pipe_output_with_crate(pipe_output, library_crate=None)-- def test_teardown_runs_on_hydrate_failure_with_crate(self, mocker: MockerFixture) -> None:- """If hydration raises while a crate scope is open, the scoped library is still torn down."""- # Mock load_from_crate to be a no-op so we don't need a real crate.- # The raw dict will trigger a hydrate failure via unknown class.- from pipelex.libraries.library_crate import LibraryCrate # noqa: PLC0415-- fake_crate = mocker.MagicMock(spec=LibraryCrate)- library_manager = get_library_manager()- teardown_spy = mocker.spy(library_manager, "teardown")- mocker.patch.object(library_manager, "load_from_crate", return_value=None)-- raw = {- "root": {- "bad_stuff": {- "stuff_code": "test",- "stuff_name": "bad_stuff",- "concept": {- "code": "NonExistent",- "domain_code": "native",- "description": "missing",- "structure_class_name": "AbsolutelyNonExistentClass",- },- "content": {"text": "x"},- },- },- "aliases": {},- }- pipe_output = PipeOutput(working_memory=WorkingMemory(), working_memory_raw=raw)-- with pytest.raises(PipeJobError):- rehydrate_pipe_output_with_crate(pipe_output, library_crate=fake_crate)-- assert teardown_spy.call_count == 1, "Scoped rehydration library should be torn down on failure"diff --git a/tests/unit/pipelex/runtime_bridge/test_delivery_mode.py b/tests/unit/pipelex/runtime_bridge/test_delivery_mode.pynew file mode 100644index 000000000..73c5e8549--- /dev/null+++ b/tests/unit/pipelex/runtime_bridge/test_delivery_mode.py@@ -0,0 +1,12 @@+from pipelex.runtime_bridge.delivery_mode import DeliveryMode+++class TestDeliveryMode:+ def test_string_values_are_stable(self) -> None:+ """The closed delivery enum's wire strings are stable (it is a StrEnum, used as its own string)."""+ assert DeliveryMode.BLOCKING == "blocking"+ assert DeliveryMode.FIRE_AND_FORGET == "fire_and_forget"++ def test_members_are_exhaustive(self) -> None:+ """The axis is closed at exactly two members — adding a third (e.g. STREAMING) is core's call, never a plugin's."""+ assert set(DeliveryMode) == {DeliveryMode.BLOCKING, DeliveryMode.FIRE_AND_FORGET}diff --git a/tests/unit/pipelex/runtime_bridge/test_direct_router_scoping.py b/tests/unit/pipelex/runtime_bridge/test_direct_router_scoping.pydeleted file mode 100644index 3833d401c..000000000--- a/tests/unit/pipelex/runtime_bridge/test_direct_router_scoping.py+++ /dev/null@@ -1,67 +0,0 @@-import pytest-from pytest_mock import MockerFixture--from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.hub import get_pipe_router-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_router import PipeRouter-from pipelex.pipe_run.pipe_run import PipeRun-from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.runtime_bridge.bootstrap import ensure_pipelex_booted-from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge-from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode---class TestDirectRouterScoping:- async def test_direct_mode_scopes_router_so_nested_dispatch_skips_hub_default(self, mocker: MockerFixture) -> None:- """A DIRECT bridge run installs its in-process router as the active router for the whole call.-- Nested controller sub-pipes resolve their router via ``get_pipe_router()``.- Before the fix, ``_run_direct`` never set the contextvar override, so- nested dispatch fell back to the hub default — which is the Temporal- router in a Temporal-enabled worker, leaking nested pipes to Temporal.- After the fix, the run is wrapped in ``scoped_pipe_router`` so- ``get_pipe_router()`` returns the bridge's own in-process router for the- duration of the call, and the override is torn down on exit.- """- fake_pipe = mocker.MagicMock()- fake_pipe.code = "fake_pipe"- fake_pipe.domain_code = "fake_domain"- fake_job = PipeJob.model_construct(- pipe=fake_pipe,- working_memory=WorkingMemoryFactory.make_empty(),- pipe_run_params=PipeRunParamsFactory.make_run_params(),- job_metadata=JobMetadata(user_id="anonymous", pipeline_run_id="run-id"),- library_crate=None,- )- mocker.patch("pipelex.runtime_bridge.bridge.build_pipe_job_from_input", return_value=fake_job)-- ensure_pipelex_booted()- # No override is active here, so this is the hub default — exactly what a- # nested sub-pipe would resolve if the DIRECT run failed to scope a router.- router_before = get_pipe_router()-- captured: dict[str, object] = {}-- async def capture_active_router(_self: PipeRun, *_args: object, **_kwargs: object) -> PipeOutput: # noqa: RUF029 — replaces an awaited method, must be a coroutine- # Stand-in for what a nested controller does mid-run: resolve the- # active router. It must see the scoped in-process router, NOT the- # hub default.- captured["router_during_run"] = get_pipe_router()- return PipeOutput(working_memory=WorkingMemoryFactory.make_empty(), pipeline_run_id="run-id")-- mocker.patch.object(PipeRun, "run", new=capture_active_router)-- await run_pipe_via_bridge(- PipelexPipeRunInput(pipe_code="fake_pipe", execution_mode=PipelexExecutionMode.DIRECT),- )-- router_during_run = captured["router_during_run"]- assert isinstance(router_during_run, PipeRouter)- assert router_during_run is not router_before, "Nested dispatch must not resolve the hub default during a DIRECT run"- # The scope is torn down after the run: a later resolution falls back to- # the hub default again (override restored, not clobbered).- assert get_pipe_router() is router_beforediff --git a/tests/unit/pipelex/runtime_bridge/test_dispatch.py b/tests/unit/pipelex/runtime_bridge/test_dispatch.pydeleted file mode 100644index 0a57a3ec9..000000000--- a/tests/unit/pipelex/runtime_bridge/test_dispatch.py+++ /dev/null@@ -1,206 +0,0 @@-import builtins--import pytest-from pytest_mock import MockerFixture--from pipelex.base_exceptions import ErrorDomain, ErrorReport-from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run import PipeRun-from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge-from pipelex.runtime_bridge.exceptions import MissingMistralWorkflowsPluginError, PipelexBridgeDispatchError-from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode-from pipelex.temporal.exceptions import WorkflowExecutionError---def _make_fake_pipe_job(mocker: MockerFixture, pipe_code: str, pipeline_run_id: str) -> PipeJob:- """Build a PipeJob without triggering Pydantic's PipeAbstract validation.-- Tests at the dispatch layer don't care about the concrete pipe — only- that the bridge routes the right pipe_job to the right PipeRun. Using- ``model_construct`` lets us pass a MagicMock as ``pipe`` without- constructing a full PipeAbstract subclass.- """- fake_pipe = mocker.MagicMock()- fake_pipe.code = pipe_code- fake_pipe.domain_code = "fake_domain"- return PipeJob.model_construct(- pipe=fake_pipe,- working_memory=WorkingMemoryFactory.make_empty(),- pipe_run_params=PipeRunParamsFactory.make_run_params(),- job_metadata=JobMetadata(user_id="anonymous", pipeline_run_id=pipeline_run_id),- library_crate=None,- )---class TestDispatch:- async def test_direct_mode_calls_pipe_run_with_pipe_job(self, mocker: MockerFixture) -> None:- fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")- mocker.patch(- "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",- return_value=fake_job,- )-- fake_output = PipeOutput(- working_memory=WorkingMemoryFactory.make_empty(),- pipeline_run_id="injected-run-id",- )- mock_run = mocker.patch.object(PipeRun, "run", new_callable=mocker.AsyncMock, return_value=fake_output)-- result = await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code="fake_pipe",- execution_mode=PipelexExecutionMode.DIRECT,- pipeline_run_id="caller-run-id",- )- )-- assert mock_run.await_count == 1- await_args = mock_run.await_args- assert await_args is not None- call_kwargs: dict[str, object] = dict(await_args.kwargs)- assert call_kwargs["delivery_assignment"] is None- assert call_kwargs["pipe_job"] is fake_job-- assert result.is_completed is True- assert result.pipeline_run_id == "injected-run-id"- assert result.workflow_id is None- assert result.graph_spec_dump is None-- async def test_temporal_blocking_dispatches_to_temporal_pipe_run(self, mocker: MockerFixture) -> None:- fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")- mocker.patch(- "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",- return_value=fake_job,- )-- fake_output = PipeOutput(- working_memory=WorkingMemoryFactory.make_empty(),- pipeline_run_id="temporal-run-id",- )- fake_temporal_run = mocker.AsyncMock(return_value=fake_output)- fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")- fake_factory.return_value.run = fake_temporal_run- # The bridge reports the actual Temporal workflow id (make_workflow_id, which prefixes by run- # mode), not the bare pipeline_run_id. Stub it to a known value and assert the bridge surfaces it.- fake_factory.return_value.make_workflow_id.return_value = "ut-temporal-run-id"-- result = await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code="fake_pipe",- execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING,- )- )-- fake_factory.assert_called_once()- assert fake_temporal_run.await_count == 1- fake_factory.return_value.make_workflow_id.assert_called_once_with(pipeline_run_id="caller-run-id")- assert result.is_completed is True- assert result.workflow_id == "ut-temporal-run-id"-- async def test_temporal_blocking_failure_wraps_in_dispatch_error_preserving_report(self, mocker: MockerFixture) -> None:- """A Temporal-mode pipe failure surfaces as WorkflowExecutionError, which the bridge wraps into- the uniform PipelexBridgeDispatchError (same contract as DIRECT/mistral). The structured- ErrorReport is not lost: it stays reachable via __cause__, and PipelexBridgeDispatchError's- to_error_report() surfaces the underlying classification via cause-chain enrichment.- """- fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")- mocker.patch(- "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",- return_value=fake_job,- )-- report = ErrorReport(- error_type="CogtError",- message="rate limited on the worker",- title="AI inference failed",- type_uri="https://docs.pipelex.com/latest/errors/cogt-error/",- error_category="capacity",- error_domain=ErrorDomain.RUNTIME,- retryable=False,- model="gpt-5",- provider="openai",- )- workflow_failure = WorkflowExecutionError(report.message, error_report=report)- fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")- fake_factory.return_value.run = mocker.AsyncMock(side_effect=workflow_failure)-- with pytest.raises(PipelexBridgeDispatchError) as exc_info:- await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code="fake_pipe",- execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING,- )- )-- # The raw Temporal failure is preserved as the cause...- assert exc_info.value.__cause__ is workflow_failure- # ...and the structured classification is surfaced through the wrapper's report.- recovered = exc_info.value.to_error_report()- assert recovered.error_type == "PipelexBridgeDispatchError"- assert recovered.error_category == "capacity"- assert recovered.retryable is False- assert recovered.model == "gpt-5"- assert recovered.provider == "openai"-- async def test_temporal_fire_and_forget_returns_workflow_id_without_completion(self, mocker: MockerFixture) -> None:- fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")- mocker.patch(- "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",- return_value=fake_job,- )-- fake_handle = mocker.MagicMock()- fake_start = mocker.AsyncMock(return_value=("wf-id-42", fake_handle))- fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")- fake_factory.return_value.start = fake_start-- result = await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code="fake_pipe",- execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,- delivery_assignment_dump={"webhooks": [{"url": "https://example.test/hook"}], "storage": None},- pipeline_run_id="caller-run-id",- )- )-- fake_start.assert_awaited_once()- assert result.is_completed is False- assert result.workflow_id == "wf-id-42"- assert result.pipeline_run_id == "caller-run-id"- assert result.output_dict == {}-- async def test_mistral_native_without_plugin_raises_with_install_hint(self, mocker: MockerFixture) -> None:- fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")- mocker.patch(- "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",- return_value=fake_job,- )-- # Simulate the plugin not being installed by stubbing __import__ to fail on the- # plugin's primitives.pipe_run module. The bridge does the import lazily inside- # _run_mistral_native, so this only fires when MISTRAL_NATIVE is selected.- real_import = builtins.__import__-- def fake_import(name: str, *args: object, **kwargs: object) -> object:- if name.startswith("pipelex_mistralai_workflows.primitives.pipe_run"):- msg = f"No module named {name!r}"- raise ImportError(msg)- return real_import(name, *args, **kwargs) # type: ignore[arg-type]-- mocker.patch.object(builtins, "__import__", side_effect=fake_import)-- with pytest.raises(MissingMistralWorkflowsPluginError) as exc_info:- await run_pipe_via_bridge(- PipelexPipeRunInput(- pipe_code="fake_pipe",- execution_mode=PipelexExecutionMode.MISTRAL_NATIVE,- )- )-- assert "pipelex-mistralai-workflows" in str(exc_info.value)- assert "pip install" in str(exc_info.value)diff --git a/tests/unit/pipelex/runtime_bridge/test_exceptions_disclosure.py b/tests/unit/pipelex/runtime_bridge/test_exceptions_disclosure.pyindex 4e57e511c..8f3c84515 100644--- a/tests/unit/pipelex/runtime_bridge/test_exceptions_disclosure.py+++ b/tests/unit/pipelex/runtime_bridge/test_exceptions_disclosure.py@@ -1,33 +1,20 @@import pytestfrom pipelex.base_exceptions import INTERNAL_ERROR_PLACEHOLDER, DisclosureMode-from pipelex.runtime_bridge.exceptions import (- MissingMistralWorkflowsPluginError,- MissingPipelexTemporalExtraError,- PipelexRuntimeBridgeError,-)+from pipelex.runtime_bridge.exceptions import MissingOrchestratorErrorclass TestRuntimeBridgeExceptionDisclosure:- @pytest.mark.parametrize(- ("error_class", "hint_fragment"),- [- (MissingPipelexTemporalExtraError, "pip install 'pipelex[temporal]'"),- (MissingMistralWorkflowsPluginError, "pip install pipelex-mistralai-workflows"),- ],- )- def test_install_hint_survives_strict_disclosure(- self,- error_class: type[PipelexRuntimeBridgeError],- hint_fragment: str,- ) -> None:- """These errors carry an actionable pip-install hint in their message.+ @pytest.mark.parametrize("mode", ["temporal", "mistralai-workflows", "acme"])+ def test_generic_plugin_hint_survives_strict_disclosure(self, mode: str) -> None:+ """MissingOrchestratorError carries a caller-actionable, plugin-decoupled hint in its message.- ``_authors_caller_facing_message = True`` must keep that message intact- under STRICT disclosure; without the flag STRICT would replace it with- ``INTERNAL_ERROR_PLACEHOLDER`` and the hint would be lost.+ ``_authors_caller_facing_message = True`` must keep that message intact under+ STRICT disclosure; without the flag STRICT would replace it with+ ``INTERNAL_ERROR_PLACEHOLDER`` and the "is its plugin installed?" hint would be lost."""- message = f"mode unavailable. Install with: {hint_fragment}"- payload = error_class(message).to_error_report().to_dict(disclosure_mode=DisclosureMode.STRICT)- assert payload["message"] == message+ error = MissingOrchestratorError(mode=mode)+ payload = error.to_error_report().to_dict(disclosure_mode=DisclosureMode.STRICT)+ assert mode in payload["message"]+ assert "is its plugin installed?" in payload["message"]assert payload["message"] != INTERNAL_ERROR_PLACEHOLDERdiff --git a/tests/unit/pipelex/runtime_bridge/test_execution_mode.py b/tests/unit/pipelex/runtime_bridge/test_execution_mode.pydeleted file mode 100644index 4f0825c18..000000000--- a/tests/unit/pipelex/runtime_bridge/test_execution_mode.py+++ /dev/null@@ -1,27 +0,0 @@-from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode---class TestPipelexExecutionMode:- def test_string_values_are_stable(self):- assert PipelexExecutionMode.DIRECT == "direct"- assert PipelexExecutionMode.TEMPORAL_BLOCKING == "temporal_blocking"- assert PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET == "temporal_fire_and_forget"- assert PipelexExecutionMode.MISTRAL_NATIVE == "mistral_native"-- def test_requires_pipelex_temporal(self):- assert PipelexExecutionMode.DIRECT.requires_pipelex_temporal is False- assert PipelexExecutionMode.TEMPORAL_BLOCKING.requires_pipelex_temporal is True- assert PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET.requires_pipelex_temporal is True- assert PipelexExecutionMode.MISTRAL_NATIVE.requires_pipelex_temporal is False-- def test_requires_mistral_workflows_extra(self):- assert PipelexExecutionMode.DIRECT.requires_mistral_workflows_extra is False- assert PipelexExecutionMode.TEMPORAL_BLOCKING.requires_mistral_workflows_extra is False- assert PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET.requires_mistral_workflows_extra is False- assert PipelexExecutionMode.MISTRAL_NATIVE.requires_mistral_workflows_extra is True-- def test_is_fire_and_forget(self):- assert PipelexExecutionMode.DIRECT.is_fire_and_forget is False- assert PipelexExecutionMode.TEMPORAL_BLOCKING.is_fire_and_forget is False- assert PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET.is_fire_and_forget is True- assert PipelexExecutionMode.MISTRAL_NATIVE.is_fire_and_forget is Falsediff --git a/tests/unit/pipelex/runtime_bridge/test_input_models.py b/tests/unit/pipelex/runtime_bridge/test_input_models.pyindex b710ba8d7..7cda7ff7e 100644--- a/tests/unit/pipelex/runtime_bridge/test_input_models.py+++ b/tests/unit/pipelex/runtime_bridge/test_input_models.py@@ -1,8 +1,8 @@import pytestfrom pydantic import ValidationError-from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, PipelexPipeRunOutput-from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode+from pipelex.runtime_bridge.delivery_mode import DeliveryMode+from pipelex.runtime_bridge.payloads import PipelexPipeRunInput, PipelexPipeRunOutputclass TestInputOutputModels:@@ -14,7 +14,8 @@ class TestInputOutputModels:assert payload.pipeline_run_id is Noneassert payload.user_id is Noneassert payload.library_crate_dump is None- assert payload.execution_mode is PipelexExecutionMode.DIRECT+ assert payload.orchestration_mode == "direct"+ assert payload.delivery is DeliveryMode.BLOCKINGassert payload.delivery_assignment_dump is Nonedef test_input_forbids_extra_fields(self):@@ -34,7 +35,8 @@ class TestInputOutputModels:original = PipelexPipeRunInput(pipe_code="some_pipe",inputs={"foo": "bar"},- execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING,+ orchestration_mode="temporal",+ delivery=DeliveryMode.FIRE_AND_FORGET,pipeline_run_id="run-123",user_id="alice",)diff --git a/tests/unit/pipelex/runtime_bridge/test_missing_bundle_validator_error.py b/tests/unit/pipelex/runtime_bridge/test_missing_bundle_validator_error.pynew file mode 100644index 000000000..a986caf07--- /dev/null+++ b/tests/unit/pipelex/runtime_bridge/test_missing_bundle_validator_error.py@@ -0,0 +1,29 @@+"""MissingBundleValidatorError is generic and plugin-decoupled (the /validate counterpart of D-F).++The API raises this when a requested orchestration mode has no registered bundle validator. Like+MissingOrchestratorError, its message names the token but no orchestrator — *is its plugin installed?* —+so core never names temporal/mistral. The one special case is the core ``"direct"`` token: its validator+is always present, so a miss there is a boot/discovery fault.+"""++import pytest++from pipelex.runtime_bridge.exceptions import MissingBundleValidatorError+from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE+++class TestMissingBundleValidatorError:+ @pytest.mark.parametrize("mode", ["temporal", "mistralai-workflows", "acme"])+ def test_unregistered_mode_carries_generic_plugin_hint(self, mode: str) -> None:+ error = MissingBundleValidatorError(mode=mode)++ assert error.mode == mode+ assert mode in str(error)+ assert "is its plugin installed?" in str(error)++ def test_direct_mode_reports_a_boot_or_discovery_problem(self) -> None:+ """The core "direct" token has no install hint — its validator is always available, so a miss is a boot fault."""+ error = MissingBundleValidatorError(mode=DIRECT_ORCHESTRATION_MODE)++ assert error.mode == DIRECT_ORCHESTRATION_MODE+ assert "boot or plugin-discovery problem" in str(error)diff --git a/tests/unit/pipelex/runtime_bridge/test_orchestration_mode.py b/tests/unit/pipelex/runtime_bridge/test_orchestration_mode.pynew file mode 100644index 000000000..39a15606d--- /dev/null+++ b/tests/unit/pipelex/runtime_bridge/test_orchestration_mode.py@@ -0,0 +1,16 @@+from pipelex.runtime_bridge.orchestration_mode import DIRECT_ORCHESTRATION_MODE, OrchestrationMode+++class TestOrchestrationMode:+ def test_direct_token_is_stable(self) -> None:+ """Core's one built-in token is the literal ``"direct"`` — the registry key and the input default."""+ assert DIRECT_ORCHESTRATION_MODE == "direct"++ def test_orchestration_mode_is_a_plain_str_alias(self) -> None:+ """The open token is assignment-compatible with plain ``str`` so plugins register raw strings, cast-free."""+ # OrchestrationMode is a semantic alias for str: a bare token (here a synthetic plugin's)+ # is a valid OrchestrationMode with no construction/cast.+ token: OrchestrationMode = "acme"+ assert isinstance(token, str)+ assert token == "acme"+ assert OrchestrationMode is strdiff --git a/tests/unit/pipelex/runtime_bridge/test_output_serialization.py b/tests/unit/pipelex/runtime_bridge/test_output_serialization.pyindex 18ca38d9f..0f6b8cd3b 100644--- a/tests/unit/pipelex/runtime_bridge/test_output_serialization.py+++ b/tests/unit/pipelex/runtime_bridge/test_output_serialization.py@@ -1,8 +1,6 @@from pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.runtime_bridge.bridge import (- _serialize_completed_output, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]-)+from pipelex.runtime_bridge.serialization import serialize_completed_outputclass TestSerializeCompletedOutput:@@ -21,7 +19,7 @@ class TestSerializeCompletedOutput:graph_assembly_error="graph boom",)- dto = _serialize_completed_output(+ dto = serialize_completed_output(pipe_output=pipe_output,workflow_id=None,)@@ -39,7 +37,7 @@ class TestSerializeCompletedOutput:pipeline_run_id="run-without-usage",)- dto = _serialize_completed_output(+ dto = serialize_completed_output(pipe_output=pipe_output,workflow_id=None,)diff --git a/tests/unit/pipelex/runtime_bridge/test_temporal_blocking_workflow_id.py b/tests/unit/pipelex/runtime_bridge/test_temporal_blocking_workflow_id.pydeleted file mode 100644index 9180523b3..000000000--- a/tests/unit/pipelex/runtime_bridge/test_temporal_blocking_workflow_id.py+++ /dev/null@@ -1,58 +0,0 @@-"""TEMPORAL_BLOCKING reports the actual Temporal workflow id, not the bare run id.--``TemporalPipeRun.run()`` starts the workflow with ``make_workflow_id(pipeline_run_id)``,-which prefixes the id in non-NORMAL run modes (``ut-``/``ci-``/``cc-``/``cct-`` — pinned-per run mode by ``tests/unit/pipelex/temporal/test_workflow_id_construction.py``). The bridge-must surface that same id so a caller can query/cancel the real workflow — matching what-fire-and-forget already returns from ``start()``. This test pins the *wiring* (the bridge-reports ``make_workflow_id``'s output, called with the job's pipeline_run_id), independent of-the prefix table; the prefixing itself is covered by the temporal-manager test above.-"""--import pytest-from pytest_mock import MockerFixture--from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge-from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode---def _fake_pipe_job(mocker: MockerFixture) -> PipeJob:- fake_pipe = mocker.MagicMock()- fake_pipe.code = "fake_pipe"- fake_pipe.domain_code = "fake_domain"- return PipeJob.model_construct(- pipe=fake_pipe,- working_memory=WorkingMemoryFactory.make_empty(),- pipe_run_params=PipeRunParamsFactory.make_run_params(),- job_metadata=JobMetadata(user_id="anonymous", pipeline_run_id="bare-run-id"),- library_crate=None,- )---class TestTemporalBlockingWorkflowId:- async def test_blocking_reports_make_workflow_id_not_bare_run_id(self, mocker: MockerFixture) -> None:- fake_job = _fake_pipe_job(mocker)- mocker.patch("pipelex.runtime_bridge.bridge.build_pipe_job_from_input", return_value=fake_job)-- fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")- temporal_pipe_run = fake_factory.return_value- temporal_pipe_run.run = mocker.AsyncMock(- return_value=PipeOutput(working_memory=WorkingMemoryFactory.make_empty(), pipeline_run_id="bare-run-id"),- )- # Stand in for the real make_workflow_id, which prefixes by run mode. The bridge must report- # THIS value (the id Temporal actually started the workflow under), not pipe_output.pipeline_run_id.- temporal_pipe_run.make_workflow_id.return_value = "ut-bare-run-id"-- result = await run_pipe_via_bridge(- PipelexPipeRunInput(pipe_code="fake_pipe", execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING),- )-- temporal_pipe_run.make_workflow_id.assert_called_once_with(pipeline_run_id="bare-run-id")- assert result.workflow_id == "ut-bare-run-id"- assert result.workflow_id != result.pipeline_run_iddiff --git a/tests/unit/pipelex/runtime_bridge/test_trace_context_contract.py b/tests/unit/pipelex/runtime_bridge/test_trace_context_contract.pydeleted file mode 100644index 7a3655eb3..000000000--- a/tests/unit/pipelex/runtime_bridge/test_trace_context_contract.py+++ /dev/null@@ -1,89 +0,0 @@-import pytest-from pytest_mock import MockerFixture--from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory-from pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.graph.graph_config import DataInclusionConfig-from pipelex.graph.trace_context import TraceContext-from pipelex.pipe_run.pipe_job import PipeJob-from pipelex.pipe_run.pipe_run import PipeRun-from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge-from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode---def _make_trace_context() -> TraceContext:- data_inclusion = DataInclusionConfig(- stuff_json_content=False,- stuff_text_content=False,- stuff_html_content=False,- error_stack_traces=False,- pipe_and_concept_registry=False,- )- return TraceContext(graph_id="host-graph-id", data_inclusion=data_inclusion)---def _fake_pipe_job(mocker: MockerFixture) -> PipeJob:- fake_pipe = mocker.MagicMock()- fake_pipe.code = "fake_pipe"- fake_pipe.domain_code = "fake_domain"- return PipeJob.model_construct(- pipe=fake_pipe,- working_memory=WorkingMemoryFactory.make_empty(),- pipe_run_params=PipeRunParamsFactory.make_run_params(),- job_metadata=JobMetadata(user_id="anonymous", pipeline_run_id="run-id"),- library_crate=None,- )---class TestTraceContextContract:- async def test_direct_mode_forwards_host_trace_context(self, mocker: MockerFixture) -> None:- trace_context = _make_trace_context()- captured: dict[str, object] = {}- fake_job = _fake_pipe_job(mocker)-- def spy(**kwargs: object) -> PipeJob:- captured["trace_context"] = kwargs["trace_context"]- return fake_job-- mocker.patch("pipelex.runtime_bridge.bridge.build_pipe_job_from_input", side_effect=spy)- mocker.patch.object(- PipeRun,- "run",- new_callable=mocker.AsyncMock,- return_value=PipeOutput(working_memory=WorkingMemoryFactory.make_empty(), pipeline_run_id="run-id"),- )-- await run_pipe_via_bridge(- PipelexPipeRunInput(pipe_code="fake_pipe", execution_mode=PipelexExecutionMode.DIRECT),- trace_context=trace_context,- )-- assert captured["trace_context"] is trace_context-- async def test_temporal_mode_nulls_host_trace_context(self, mocker: MockerFixture) -> None:- trace_context = _make_trace_context()- captured: dict[str, object] = {}- fake_job = _fake_pipe_job(mocker)-- def spy(**kwargs: object) -> PipeJob:- captured["trace_context"] = kwargs["trace_context"]- return fake_job-- mocker.patch("pipelex.runtime_bridge.bridge.build_pipe_job_from_input", side_effect=spy)- fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")- fake_factory.return_value.run = mocker.AsyncMock(- return_value=PipeOutput(working_memory=WorkingMemoryFactory.make_empty(), pipeline_run_id="temporal-run-id"),- )- # The blocking branch reports make_workflow_id(...) as the workflow id; stub it to a string- # so PipelexPipeRunOutput validation passes (this test only asserts trace_context nulling).- fake_factory.return_value.make_workflow_id.return_value = "ut-temporal-run-id"-- await run_pipe_via_bridge(- PipelexPipeRunInput(pipe_code="fake_pipe", execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING),- trace_context=trace_context,- )-- assert captured["trace_context"] is Nonediff --git a/tests/unit/pipelex/runtime_bridge/test_validation.py b/tests/unit/pipelex/runtime_bridge/test_validation.pydeleted file mode 100644index e85ed8068..000000000--- a/tests/unit/pipelex/runtime_bridge/test_validation.py+++ /dev/null@@ -1,94 +0,0 @@-import pytest--from pipelex.libraries.library_crate import LibraryCrate-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment-from pipelex.runtime_bridge.bridge import (- PipelexPipeRunInput,- _decode_delivery_assignment, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]- _decode_library_crate, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]- _validate_input, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]- run_pipe_via_bridge,-)-from pipelex.runtime_bridge.exceptions import PipelexBridgeDispatchError-from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode---class TestBridgeValidationAndDecoding:- def test_validate_input_passes_for_direct_without_delivery(self):- payload = PipelexPipeRunInput(pipe_code="any", execution_mode=PipelexExecutionMode.DIRECT)- _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump)) # must not raise-- def test_validate_input_passes_for_temporal_blocking_without_delivery(self):- payload = PipelexPipeRunInput(pipe_code="any", execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING)- _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump)) # must not raise-- def test_validate_input_rejects_fire_and_forget_without_delivery(self):- payload = PipelexPipeRunInput(- pipe_code="any",- execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,- )- with pytest.raises(PipelexBridgeDispatchError, match="TEMPORAL_FIRE_AND_FORGET"):- _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump))-- def test_validate_input_rejects_fire_and_forget_with_empty_delivery(self):- # A DeliveryAssignment with no storage and no webhooks is a no-op: completion would- # be silently dropped, so it must be rejected just like a missing dump.- payload = PipelexPipeRunInput(- pipe_code="any",- execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,- delivery_assignment_dump={"webhooks": [], "storage": None},- )- with pytest.raises(PipelexBridgeDispatchError, match="delivery target"):- _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump))-- def test_validate_input_accepts_fire_and_forget_with_storage_target(self):- payload = PipelexPipeRunInput(- pipe_code="any",- execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,- delivery_assignment_dump={"webhooks": [], "storage": {"key_prefix": "runs/abc"}},- )- _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump)) # must not raise-- def test_validate_input_accepts_fire_and_forget_with_webhook_target(self):- payload = PipelexPipeRunInput(- pipe_code="any",- execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,- delivery_assignment_dump={"webhooks": [{"url": "https://example.test/hook"}], "storage": None},- )- _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump)) # must not raise-- def test_decode_library_crate_returns_none_for_none(self):- assert _decode_library_crate(None) is None-- def test_decode_library_crate_round_trips_empty(self):- empty = LibraryCrate()- decoded = _decode_library_crate(empty.model_dump(mode="json"))- assert decoded is not None- assert decoded.concepts == empty.concepts- assert decoded.pipes == empty.pipes-- def test_decode_delivery_assignment_returns_none_for_none(self):- assert _decode_delivery_assignment(None) is None-- def test_decode_delivery_assignment_round_trips(self):- assignment = DeliveryAssignment.model_validate(- {- "storage": {"key_prefix": "runs/abc"},- "webhooks": [{"url": "https://example.test/hook"}],- }- )- decoded = _decode_delivery_assignment(assignment.model_dump(mode="json"))- assert decoded is not None- assert decoded.storage is not None- assert decoded.storage.key_prefix == "runs/abc/" # storage validator appends trailing /- assert len(decoded.webhooks) == 1- assert decoded.webhooks[0].url == "https://example.test/hook"-- @pytest.mark.asyncio- async def test_run_pipe_via_bridge_rejects_fire_and_forget_without_delivery(self):- payload = PipelexPipeRunInput(- pipe_code="any",- execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,- )- with pytest.raises(PipelexBridgeDispatchError, match="TEMPORAL_FIRE_AND_FORGET"):- await run_pipe_via_bridge(payload)diff --git a/tests/unit/pipelex/system/test_plugin_config_loader.py b/tests/unit/pipelex/system/test_plugin_config_loader.pynew file mode 100644index 000000000..fad290ea3--- /dev/null+++ b/tests/unit/pipelex/system/test_plugin_config_loader.py@@ -0,0 +1,150 @@+"""Tests for the env-aware plugin-config loader (D2): packaged default -> {env} -> override, global -> project."""++from __future__ import annotations++from typing import TYPE_CHECKING++from pydantic import BaseModel, ConfigDict++from pipelex.system.configuration.config_loader import ConfigLoader+from pipelex.system.runtime import RunEnvironment, runtime_manager++if TYPE_CHECKING:+ from pathlib import Path++ from pytest_mock import MockerFixture+++class _ServerCfg(BaseModel):+ model_config = ConfigDict(extra="forbid")++ host: str+ port: int+++class _DemoPluginConfig(BaseModel):+ model_config = ConfigDict(extra="forbid")++ name: str+ retries: int+ server: _ServerCfg+++_PLUGIN_NAME = "demo"+_DEFAULT_TOML = 'name = "default"\nretries = 3\n\n[server]\nhost = "localhost"\nport = 7233\n'+++class TestPluginConfigLoader:+ """The env-aware plugin-config helper layers packaged default -> {env} -> override across global -> project."""++ @staticmethod+ def _build_loader(+ mocker: MockerFixture,+ *,+ global_dir: Path,+ project_dir: Path | None,+ environment: RunEnvironment,+ ) -> ConfigLoader:+ """Return a ConfigLoader whose config-dir resolution and env are pinned to the test fixtures."""+ mocker.patch.object(ConfigLoader, "global_config_dir", new_callable=mocker.PropertyMock, return_value=global_dir)+ mocker.patch.object(ConfigLoader, "project_config_dir", new_callable=mocker.PropertyMock, return_value=project_dir)+ mocker.patch.object(type(runtime_manager), "environment", new_callable=mocker.PropertyMock, return_value=environment)+ return ConfigLoader()++ def test_packaged_default_only(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """With no override files present, the packaged default alone is a valid, fully-resolved config."""+ package_dir = tmp_path / "pkg"+ package_dir.mkdir()+ (package_dir / f"{_PLUGIN_NAME}.toml").write_text(_DEFAULT_TOML)++ global_dir = tmp_path / "global"+ global_dir.mkdir()+ loader = self._build_loader(mocker, global_dir=global_dir, project_dir=None, environment=RunEnvironment.STAGING)++ config = loader.load_plugin_config(name=_PLUGIN_NAME, package_dir=package_dir, schema=_DemoPluginConfig)++ assert config == _DemoPluginConfig(name="default", retries=3, server=_ServerCfg(host="localhost", port=7233))++ def test_env_file_deep_merges_over_default_and_ignores_other_envs(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """{name}_{env}.toml deep-merges onto the default; a sibling env's file is not read."""+ package_dir = tmp_path / "pkg"+ package_dir.mkdir()+ (package_dir / f"{_PLUGIN_NAME}.toml").write_text(_DEFAULT_TOML)++ global_dir = tmp_path / "global"+ global_dir.mkdir()+ # Selected env: overrides retries and only the nested port; host + name must survive.+ (global_dir / f"{_PLUGIN_NAME}_staging.toml").write_text("retries = 5\n\n[server]\nport = 9999\n")+ # Wrong env: must be ignored entirely.+ (global_dir / f"{_PLUGIN_NAME}_dev.toml").write_text("retries = 99\n")+ loader = self._build_loader(mocker, global_dir=global_dir, project_dir=None, environment=RunEnvironment.STAGING)++ config = loader.load_plugin_config(name=_PLUGIN_NAME, package_dir=package_dir, schema=_DemoPluginConfig)++ assert config == _DemoPluginConfig(name="default", retries=5, server=_ServerCfg(host="localhost", port=9999))++ def test_override_file_wins_over_env_file(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """{name}_override.toml is layered after {name}_{env}.toml, so it wins on collision."""+ package_dir = tmp_path / "pkg"+ package_dir.mkdir()+ (package_dir / f"{_PLUGIN_NAME}.toml").write_text(_DEFAULT_TOML)++ global_dir = tmp_path / "global"+ global_dir.mkdir()+ (global_dir / f"{_PLUGIN_NAME}_staging.toml").write_text("retries = 5\n")+ (global_dir / f"{_PLUGIN_NAME}_override.toml").write_text("retries = 7\n")+ loader = self._build_loader(mocker, global_dir=global_dir, project_dir=None, environment=RunEnvironment.STAGING)++ config = loader.load_plugin_config(name=_PLUGIN_NAME, package_dir=package_dir, schema=_DemoPluginConfig)++ assert config.retries == 7++ def test_project_dir_wins_over_global_dir(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """Project .pipelex override files are layered after the global ones, so the project wins."""+ package_dir = tmp_path / "pkg"+ package_dir.mkdir()+ (package_dir / f"{_PLUGIN_NAME}.toml").write_text(_DEFAULT_TOML)++ global_dir = tmp_path / "global"+ global_dir.mkdir()+ (global_dir / f"{_PLUGIN_NAME}_staging.toml").write_text('name = "from-global"\n')+ project_dir = tmp_path / "project"+ project_dir.mkdir()+ (project_dir / f"{_PLUGIN_NAME}_staging.toml").write_text('name = "from-project"\n')+ loader = self._build_loader(mocker, global_dir=global_dir, project_dir=project_dir, environment=RunEnvironment.STAGING)++ config = loader.load_plugin_config(name=_PLUGIN_NAME, package_dir=package_dir, schema=_DemoPluginConfig)++ assert config.name == "from-project"++ def test_extra_overrides_win_last(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """Programmatic extra_overrides are the final layer, beating every file."""+ package_dir = tmp_path / "pkg"+ package_dir.mkdir()+ (package_dir / f"{_PLUGIN_NAME}.toml").write_text(_DEFAULT_TOML)++ global_dir = tmp_path / "global"+ global_dir.mkdir()+ (global_dir / f"{_PLUGIN_NAME}_override.toml").write_text("retries = 7\n")+ loader = self._build_loader(mocker, global_dir=global_dir, project_dir=None, environment=RunEnvironment.STAGING)++ config = loader.load_plugin_config(name=_PLUGIN_NAME, package_dir=package_dir, schema=_DemoPluginConfig, extra_overrides={"retries": 42})++ assert config.retries == 42++ def test_validation_runs_after_merge(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """A required field absent from the packaged default but supplied by an env file validates fine — merge precedes validation."""+ package_dir = tmp_path / "pkg"+ package_dir.mkdir()+ # Packaged default omits `retries` entirely.+ (package_dir / f"{_PLUGIN_NAME}.toml").write_text('name = "default"\n\n[server]\nhost = "localhost"\nport = 7233\n')++ global_dir = tmp_path / "global"+ global_dir.mkdir()+ (global_dir / f"{_PLUGIN_NAME}_staging.toml").write_text("retries = 11\n")+ loader = self._build_loader(mocker, global_dir=global_dir, project_dir=None, environment=RunEnvironment.STAGING)++ config = loader.load_plugin_config(name=_PLUGIN_NAME, package_dir=package_dir, schema=_DemoPluginConfig)++ assert isinstance(config, _DemoPluginConfig)+ assert config.retries == 11diff --git a/tests/unit/pipelex/temporal/test_act_assemble_tracing_error_propagation.py b/tests/unit/pipelex/temporal/test_act_assemble_tracing_error_propagation.pydeleted file mode 100644index dd32442ad..000000000--- a/tests/unit/pipelex/temporal/test_act_assemble_tracing_error_propagation.py+++ /dev/null@@ -1,44 +0,0 @@-"""Pins the error-propagation contract of ``act_assemble_tracing``.--Tracing assembly is best-effort observability: ``assemble_tracing`` catches the EXPECTED-read/assemble failures and returns them on the ``*_assembly_error`` fields (no raise). A-genuinely unexpected error (a programming bug in assembly) must instead PROPAGATE out of the-activity so that ``WfPipeRun``'s ``except ActivityError`` branch records it on the pipe output —-mirroring DIRECT mode, where ``assemble_tracing_on_output`` lets the same errors surface.-Swallowing it into an empty result would make the workflow see success and silently produce no-cost report and no diagnostic. These tests pin that the activity does not swallow unexpected-errors and otherwise passes the assembled result through unchanged.-"""--import pytest-from pytest_mock import MockerFixture--from pipelex.pipe_run.tracing_assembly import TracingAssembly-from pipelex.temporal.tprl_pipe.act_assemble_tracing import AssembleTracingArg, act_assemble_tracing--[email protected](loop_scope="class")-class TestActAssembleTracingErrorPropagation:- async def test_unexpected_error_propagates(self, mocker: MockerFixture) -> None:- """An unexpected error from assemble_tracing propagates instead of degrading to an empty result."""- mocker.patch(- "pipelex.temporal.tprl_pipe.act_assemble_tracing.assemble_tracing",- side_effect=KeyError("unexpected programming bug in assembly"),- )- arg = AssembleTracingArg(pipeline_run_id="plr-boom", assemble_graph=True, assemble_usage=True)-- with pytest.raises(KeyError):- await act_assemble_tracing(arg)-- async def test_assembled_result_passes_through(self, mocker: MockerFixture) -> None:- """On success the activity returns assemble_tracing's result unchanged (including *_error fields)."""- expected = TracingAssembly(usage_assembly_error="recorded read failure")- mocker.patch(- "pipelex.temporal.tprl_pipe.act_assemble_tracing.assemble_tracing",- return_value=expected,- )- arg = AssembleTracingArg(pipeline_run_id="plr-ok", assemble_graph=False, assemble_usage=True)-- result = await act_assemble_tracing(arg)-- assert result == expecteddiff --git a/tests/unit/pipelex/temporal/test_activity_error_boundary.py b/tests/unit/pipelex/temporal/test_activity_error_boundary.pydeleted file mode 100644index a499f9de5..000000000--- a/tests/unit/pipelex/temporal/test_activity_error_boundary.py+++ /dev/null@@ -1,74 +0,0 @@-"""Unit test for ``convert_pipelex_errors`` — the activity-side half of the Temporal error bridge.--Pins the invariants the decorator relies on:--- ``functools.wraps`` preserves ``__name__`` and ``__annotations__``. Both are- load-bearing: Temporal's ``@activity.defn`` reads the annotations for payload- typing, and ``content_generator_in_workflow.py`` reads ``__name__`` for dispatch- routing. A future refactor that drops ``functools.wraps`` would break silently- without this test.-- A ``PipelexError`` raised by the wrapped activity comes out as a ``TemporalError``.-- A non-``PipelexError`` propagates untouched, for Temporal's default converter.-"""--import pytest-from pytest_mock import MockerFixture--from pipelex.base_exceptions import PipelexError-from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors-from pipelex.temporal.tprl.temporal_error import TemporalError---async def sample_activity(value: int) -> int: # noqa: RUF029- """Stand-in activity body — exercises the success path and signature preservation."""- return value * 2---async def raises_pipelex_error() -> None: # noqa: RUF029- """Stand-in activity body that fails with a ``PipelexError``."""- msg = "simulated activity failure"- raise PipelexError(msg)---async def raises_value_error() -> None: # noqa: RUF029- """Stand-in activity body that fails with a non-``PipelexError`` exception."""- msg = "not a pipelex error"- raise ValueError(msg)--[email protected](loop_scope="class")-class TestConvertPipelexErrors:- @pytest.fixture(autouse=True)- def log_mocks(self, mocker: MockerFixture) -> None:- """Silence the bridge log helpers — they require a live Temporal context."""- mocker.patch.object(TemporalError, "_log_critical")- mocker.patch.object(TemporalError, "_log_error")-- async def test_preserves_signature_and_passes_success_through(self) -> None:- """``functools.wraps`` keeps ``__name__``/``__annotations__`` and the success value flows through."""- wrapped = convert_pipelex_errors(sample_activity)-- assert wrapped.__name__ == "sample_activity", "Temporal and workflow-side dispatch routing read __name__"- assert wrapped.__annotations__ == sample_activity.__annotations__, "Temporal reads __annotations__ for payload typing"- assert await wrapped(21) == 42-- async def test_pipelex_error_becomes_temporal_error(self) -> None:- """A ``PipelexError`` raised inside the activity is converted to a ``TemporalError``- that chains the original cause and packs the structured ``ErrorReport``.- """- wrapped = convert_pipelex_errors(raises_pipelex_error)-- with pytest.raises(TemporalError) as exc_info:- await wrapped()-- temporal_error = exc_info.value- assert isinstance(temporal_error.__cause__, PipelexError), "the original PipelexError must be chained as __cause__"- assert temporal_error.error_report is not None, "the decorator must pack the structured ErrorReport into the TemporalError"- assert temporal_error.error_report["message"] == "simulated activity failure"-- async def test_non_pipelex_error_propagates_untouched(self) -> None:- """A non-``PipelexError`` is left for Temporal's default converter — not wrapped."""- wrapped = convert_pipelex_errors(raises_value_error)-- with pytest.raises(ValueError, match="not a pipelex error"):- await wrapped()diff --git a/tests/unit/pipelex/temporal/test_all_non_retryable_error_types.py b/tests/unit/pipelex/temporal/test_all_non_retryable_error_types.pydeleted file mode 100644index ee8fcd513..000000000--- a/tests/unit/pipelex/temporal/test_all_non_retryable_error_types.py+++ /dev/null@@ -1,129 +0,0 @@-"""Unit tests for ``WorkerConfig.all_non_retryable_error_types``.--The helper unions every ``non_retryable_error_types`` declared anywhere in the-temporal config: baseline main list + every queue overlay's ``_extra`` +-every handle overlay's ``_extra``. ``TemporalError.from_app_error`` uses it-for ApplicationError severity classification — the error-handling site sees-only the error type string and can't know which overlay contributed the entry,-so the union must be precomputed.--Retry *behavior* is correct via the dispatch-time ``retry_policy`` (that path-is covered by ``test_resolve_dispatch.py::test_non_retryable_additive``).-These tests cover only the log-severity union.-"""--from datetime import timedelta--from pipelex.temporal.config_temporal import (- ActivityRouteConfig,- HandleOptions,- QueueOptions,- RetryPolicyConfig,- RetryPolicyConfigOverlay,- WorkerConfig,-)---def _make_baseline_retry(non_retryable: list[str] | None = None) -> RetryPolicyConfig:- return RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=3,- non_retryable_error_types=non_retryable or [],- )---def _make_overlay_retry(non_retryable_extra: list[str] | None = None) -> RetryPolicyConfigOverlay:- return RetryPolicyConfigOverlay(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=3,- non_retryable_error_types_extra=non_retryable_extra or [],- )---def _make_worker_config(- activity_queues: dict[str, ActivityRouteConfig] | None = None,- baseline_non_retryable: list[str] | None = None,-) -> WorkerConfig:- return WorkerConfig(- default_task_queue="default_q",- activity_queues=activity_queues or {},- workflow_execution_timeout=timedelta(hours=1),- default_activity_start_to_close_timeout=timedelta(minutes=10),- retry_policy_config=_make_baseline_retry(baseline_non_retryable),- )---class TestAllNonRetryableErrorTypes:- """Union across baseline + every queue overlay + every handle overlay."""-- def test_baseline_only(self) -> None:- """No overlays anywhere → result equals the baseline main list."""- worker_config = _make_worker_config(baseline_non_retryable=["ValidationError", "ModelNotFoundError"])- result = worker_config.all_non_retryable_error_types(queue_options_by_queue={})- assert result == {"ValidationError", "ModelNotFoundError"}-- def test_queue_overlay_extras_are_included(self) -> None:- """Every queue overlay's ``_extra`` contributes regardless of routing."""- worker_config = _make_worker_config(baseline_non_retryable=["ValidationError"])- queue_options = {- "anthropic_q": QueueOptions(retry_policy_config=_make_overlay_retry(["AnthropicBadRequest"])),- "openai_q": QueueOptions(retry_policy_config=_make_overlay_retry(["OpenAIBadRequest"])),- # Queue without retry overlay still mustn't break the union.- "vanilla_q": QueueOptions(start_to_close_timeout=timedelta(minutes=3)),- }- result = worker_config.all_non_retryable_error_types(queue_options_by_queue=queue_options)- assert result == {"ValidationError", "AnthropicBadRequest", "OpenAIBadRequest"}-- def test_handle_overlay_extras_are_included(self) -> None:- """Every handle overlay's ``_extra`` contributes regardless of routing."""- worker_config = _make_worker_config(- baseline_non_retryable=["ValidationError"],- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(- default="anthropic_q",- handle_options={- "claude-opus-4-7-1m": HandleOptions(- retry_policy_config=_make_overlay_retry(["HandleSpecificError"]),- ),- # Handle without retry overlay still mustn't break the union.- "claude-opus-4-7": HandleOptions(start_to_close_timeout=timedelta(minutes=15)),- },- ),- },- )- result = worker_config.all_non_retryable_error_types(queue_options_by_queue={})- assert result == {"ValidationError", "HandleSpecificError"}-- def test_dedupe_across_baseline_queue_handle(self) -> None:- """Entries repeating across layers collapse to a single set entry."""- worker_config = _make_worker_config(- baseline_non_retryable=["ValidationError"],- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(- default="anthropic_q",- handle_options={- "claude-opus-4-7-1m": HandleOptions(- retry_policy_config=_make_overlay_retry(["ValidationError", "HandleSpecificError"]),- ),- },- ),- },- )- queue_options = {- "anthropic_q": QueueOptions(- retry_policy_config=_make_overlay_retry(["ValidationError", "AnthropicBadRequest"]),- ),- }- result = worker_config.all_non_retryable_error_types(queue_options_by_queue=queue_options)- # ValidationError appears in baseline + queue + handle but only once in the union.- assert result == {"ValidationError", "AnthropicBadRequest", "HandleSpecificError"}-- def test_empty_everything(self) -> None:- """No baseline, no overlays → empty set, not None."""- worker_config = _make_worker_config(baseline_non_retryable=[])- result = worker_config.all_non_retryable_error_types(queue_options_by_queue={})- assert result == set()diff --git a/tests/unit/pipelex/temporal/test_async_execution_not_enabled.py b/tests/unit/pipelex/temporal/test_async_execution_not_enabled.pydeleted file mode 100644index cd985a5a3..000000000--- a/tests/unit/pipelex/temporal/test_async_execution_not_enabled.py+++ /dev/null@@ -1,170 +0,0 @@-"""Unit tests for the backend-neutral ``AsyncExecutionNotEnabledError`` guard.--Two boundaries must fail fast with a classifiable ``PipelexError`` when no-async execution backend is enabled on the deployment:--1. ``with_conditional_worker`` — the dispatch decorator on every- ``WorkflowExecutor`` entry point (``TemporalPipeRun.run`` /- ``TemporalPipeRun.start`` / ``TemporalPipeRouter._run_pipe_job``). The- guard fires inside the wrapper *before* the per-environment ``match``- block, so on the ``INTERNAL`` path it also precedes the worker bootstrap- (``self.temporal_client()`` + ``make_worker(...)``) and on the- ``EXTERNAL`` path it precedes the wrapped body. Without the guard the- disabled path either leaks an opaque ``RuntimeError`` from the- ``TemporalManager`` singleton accessor (``stamp_submitter_session_id``- reads ``get_temporal_manager().session_id`` on its first line) or — on- ``INTERNAL`` — falls through to ``temporal_client()``'s lower-level check,- silently inverting the facade docstring contract.-2. ``WorkflowExecutor.temporal_client()`` — kept on the path for callers that- reach the executor directly rather than through the pipe-run facade.--The error class is backend-neutral on purpose (it is shared with future async-backends like Mistral Workflows), so the assertions pin classification, not-backend-specific copy.-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture--from pipelex.base_exceptions import ErrorDomain-from pipelex.pipe_run.exceptions import AsyncExecutionNotEnabledError-from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor-from pipelex.temporal.tprl_pipe.temporal_pipe_run import TemporalPipeRun---def _disabled_config_root(mocker: MockerFixture) -> Any:- """A minimal ``get_config()`` stub whose ``temporal.is_enabled`` reads False."""- config_root = mocker.MagicMock()- config_root.temporal.is_enabled = False- return config_root--[email protected](loop_scope="class")-class TestAsyncExecutionNotEnabled:- async def test_error_classification(self) -> None:- """The error reports as ``CONFIG``-domain with a stable error_type and- a curated title — these three fields are the API and CLI contract.-- Declared ``async`` only to share the class-level- ``@pytest.mark.asyncio(loop_scope="class")`` mark with the dispatch- tests below (the project rule is one ``TestClass`` per module).- """- exc = AsyncExecutionNotEnabledError("anything")- report = exc.to_error_report()-- assert report.error_type == "AsyncExecutionNotEnabledError"- assert report.title == "Async execution not enabled"- assert report.error_domain == ErrorDomain.CONFIG-- async def test_temporal_pipe_run_start_external_raises_before_touching_manager(self, mocker: MockerFixture) -> None:- """``TemporalPipeRun.start()`` on the ``EXTERNAL`` path must raise the- error before reaching ``stamp_submitter_session_id`` — which reads the- ``TemporalManager`` singleton, only initialized when the async backend- is enabled. The patched stamper records calls; if the guard ever- regresses, the stamper will be called and this test will fail.- """- mocker.patch("pipelex.temporal.tprl.conditional_worker.get_config", return_value=_disabled_config_root(mocker))- stamper = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.stamp_submitter_session_id")- pipe_run = TemporalPipeRun(task_queue="ut-queue", worker_environment=TemporalWorkerEnvironment.EXTERNAL)-- with pytest.raises(AsyncExecutionNotEnabledError):- await pipe_run.start(pipe_job=mocker.MagicMock())-- stamper.assert_not_called()-- async def test_temporal_pipe_run_run_external_raises_before_touching_manager(self, mocker: MockerFixture) -> None:- """Same guarantee as ``start()`` for the blocking ``run()`` entry point on ``EXTERNAL``."""- mocker.patch("pipelex.temporal.tprl.conditional_worker.get_config", return_value=_disabled_config_root(mocker))- stamper = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.stamp_submitter_session_id")- pipe_run = TemporalPipeRun(task_queue="ut-queue", worker_environment=TemporalWorkerEnvironment.EXTERNAL)-- with pytest.raises(AsyncExecutionNotEnabledError):- await pipe_run.run(pipe_job=mocker.MagicMock())-- stamper.assert_not_called()-- async def test_temporal_pipe_run_start_internal_raises_before_worker_bootstrap(self, mocker: MockerFixture) -> None:- """On the ``INTERNAL`` path the decorator does ``await self.temporal_client()``- + ``make_worker(...)`` before calling the wrapped body. The guard must- fire ahead of that bootstrap so a disabled deployment never reaches- either — otherwise the facade-level guarantee degrades to- ``temporal_client()``'s lower-level check, silently inverting the- docstring contract. The ``temporal_client`` and ``make_worker`` patches- are tripwires: a call on the disabled path is the regression.- """- mocker.patch("pipelex.temporal.tprl.conditional_worker.get_config", return_value=_disabled_config_root(mocker))- temporal_client = mocker.patch.object(TemporalPipeRun, "temporal_client")- make_worker = mocker.patch("pipelex.temporal.tprl.conditional_worker.get_task_manager")- stamper = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.stamp_submitter_session_id")- pipe_run = TemporalPipeRun(task_queue="ut-queue", worker_environment=TemporalWorkerEnvironment.INTERNAL)-- with pytest.raises(AsyncExecutionNotEnabledError):- await pipe_run.start(pipe_job=mocker.MagicMock())-- temporal_client.assert_not_called()- make_worker.assert_not_called()- stamper.assert_not_called()-- async def test_temporal_pipe_run_run_internal_raises_before_worker_bootstrap(self, mocker: MockerFixture) -> None:- """Same INTERNAL-path guarantee as ``start()`` for the blocking ``run()`` entry point."""- mocker.patch("pipelex.temporal.tprl.conditional_worker.get_config", return_value=_disabled_config_root(mocker))- temporal_client = mocker.patch.object(TemporalPipeRun, "temporal_client")- make_worker = mocker.patch("pipelex.temporal.tprl.conditional_worker.get_task_manager")- stamper = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.stamp_submitter_session_id")- pipe_run = TemporalPipeRun(task_queue="ut-queue", worker_environment=TemporalWorkerEnvironment.INTERNAL)-- with pytest.raises(AsyncExecutionNotEnabledError):- await pipe_run.run(pipe_job=mocker.MagicMock())-- temporal_client.assert_not_called()- make_worker.assert_not_called()- stamper.assert_not_called()-- async def test_decorator_skips_guard_when_inside_temporal_workflow(self, mocker: MockerFixture) -> None:- """When ``with_conditional_worker`` is invoked from inside a Temporal- workflow context (the child-dispatch path through- ``TemporalPipeRouter._run_pipe_job``), the deployment-level guard must- be skipped: being inside a workflow proves the backend is already- running, and ``get_config()`` is unsafe to read from the workflow- sandbox (it returns a separate, unconfigured copy on which- ``temporal.is_enabled`` reads False — which would otherwise fail the- child dispatch with ``AsyncExecutionNotEnabledError``).-- Reproduces the integration-temporal regression that surfaced in- ``library_crate`` / ``tracing`` after the guard was first moved into- the decorator without the workflow-context bypass.- """- from pipelex.temporal.tprl.conditional_worker import with_conditional_worker # noqa: PLC0415-- mocker.patch("pipelex.temporal.tprl.conditional_worker.get_config", return_value=_disabled_config_root(mocker))- mocker.patch("pipelex.temporal.tprl.conditional_worker.is_in_temporal_workflow", return_value=True)- body_calls: list[WorkflowExecutor[Any, Any]] = []-- @with_conditional_worker- async def fake_execute_workflow(self_arg: WorkflowExecutor[Any, Any]) -> str: # noqa: RUF029- body_calls.append(self_arg)- return "ok"-- executor: WorkflowExecutor[Any, Any] = WorkflowExecutor(- task_queue="ut-queue",- worker_environment=TemporalWorkerEnvironment.EXTERNAL,- )- result = await fake_execute_workflow(executor)-- assert result == "ok"- assert body_calls == [executor]-- async def test_workflow_caller_temporal_client_raises_when_disabled(self, mocker: MockerFixture) -> None:- """``WorkflowExecutor.temporal_client()`` is the lower-level entry point- callers reach when they wire the executor directly. The same guard- applies there so a disabled-backend deployment never reaches the- ``temporalio`` client construction code.- """- mocker.patch("pipelex.temporal.tprl.workflow_caller.get_config", return_value=_disabled_config_root(mocker))- executor: WorkflowExecutor[Any, Any] = WorkflowExecutor(task_queue="ut-queue")-- with pytest.raises(AsyncExecutionNotEnabledError):- await executor.temporal_client()diff --git a/tests/unit/pipelex/temporal/test_codec_server.py b/tests/unit/pipelex/temporal/test_codec_server.pydeleted file mode 100644index 886eb4f3e..000000000--- a/tests/unit/pipelex/temporal/test_codec_server.py+++ /dev/null@@ -1,187 +0,0 @@-"""Unit tests for the Temporal codec HTTP server (encode/decode endpoints + CORS)."""--from __future__ import annotations--from typing import TYPE_CHECKING, cast--import pytest-from aiohttp.test_utils import TestClient, TestServer-from google.protobuf import json_format-from temporalio.api.common.v1 import Payload, Payloads--from pipelex.temporal.codec.codec_server import build_codec_server-from pipelex.temporal.codec.storage_payload_codec import STORAGE_REF_ENCODING, StoragePayloadCodec-from pipelex.tools.storage.exceptions import StorageFileNotFoundError-from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider--if TYPE_CHECKING:- from collections.abc import Sequence--ALLOWED_ORIGIN = "http://localhost:8233"-OTHER_ORIGIN = "http://evil.example.com"-SIZE_THRESHOLD = 64-STORAGE_PREFIX = "test-codec/"-JSON_HEADERS = {"Content-Type": "application/json"}---class _RaisingCodec:- """Stub codec whose encode/decode always raise the configured exception."""-- def __init__(self, exc: Exception) -> None:- self._exc = exc-- async def encode(self, _payloads: Sequence[Payload]) -> list[Payload]:- raise self._exc-- async def decode(self, _payloads: Sequence[Payload]) -> list[Payload]:- raise self._exc---def _make_real_codec(storage: InMemoryStorageProvider) -> StoragePayloadCodec:- return StoragePayloadCodec(- storage_provider=storage,- size_threshold=SIZE_THRESHOLD,- storage_prefix=STORAGE_PREFIX,- )---def _payloads_json(*payloads: Payload) -> bytes:- """Serialize payloads to the Payloads-proto JSON the Temporal Web UI sends."""- return json_format.MessageToJson(Payloads(payloads=list(payloads))).encode()---def _parse_payloads(body: bytes) -> Payloads:- return json_format.Parse(body, Payloads())---def _make_payload(size: int) -> Payload:- return Payload(metadata={"encoding": b"binary/plain"}, data=b"x" * size)--[email protected](loop_scope="class")-class TestCodecServer:- @pytest.mark.parametrize("endpoint", ["/encode", "/decode"])- async def test_cors_preflight_allowed_origin(self, endpoint: str) -> None:- """OPTIONS preflight from an allowed origin gets the full CORS header set."""- app = build_codec_server(codec=_make_real_codec(InMemoryStorageProvider()), cors_origins=[ALLOWED_ORIGIN])- async with TestClient(TestServer(app)) as client:- response = await client.options(endpoint, headers={"Origin": ALLOWED_ORIGIN})-- assert response.status == 200- assert response.headers["Access-Control-Allow-Origin"] == ALLOWED_ORIGIN- assert response.headers["Access-Control-Allow-Methods"] == "POST"- assert response.headers["Access-Control-Allow-Headers"] == "content-type,x-namespace"-- @pytest.mark.parametrize(- ("topic", "origin_headers"),- [- ("disallowed_origin", {"Origin": OTHER_ORIGIN}),- ("missing_origin", {}),- ],- )- async def test_cors_preflight_rejected_origin(self, topic: str, origin_headers: dict[str, str]) -> None:- """OPTIONS preflight from a disallowed or missing origin gets no CORS headers."""- app = build_codec_server(codec=_make_real_codec(InMemoryStorageProvider()), cors_origins=[ALLOWED_ORIGIN])- async with TestClient(TestServer(app)) as client:- response = await client.options("/encode", headers=origin_headers)-- assert response.status == 200- assert "Access-Control-Allow-Origin" not in response.headers, f"CORS leaked for {topic}"-- async def test_non_json_content_type_rejected_with_cors(self) -> None:- """A POST without application/json content type returns 415, still carrying CORS headers."""- app = build_codec_server(codec=_make_real_codec(InMemoryStorageProvider()), cors_origins=[ALLOWED_ORIGIN])- async with TestClient(TestServer(app)) as client:- response = await client.post(- "/encode",- data=b"whatever",- headers={"Content-Type": "text/plain", "Origin": ALLOWED_ORIGIN},- )-- assert response.status == 415- assert await response.text() == "Expected application/json"- assert response.headers["Access-Control-Allow-Origin"] == ALLOWED_ORIGIN-- async def test_malformed_payloads_json_rejected(self) -> None:- """A POST body that is not valid Payloads JSON returns 400."""- app = build_codec_server(codec=_make_real_codec(InMemoryStorageProvider()), cors_origins=[ALLOWED_ORIGIN])- async with TestClient(TestServer(app)) as client:- response = await client.post("/encode", data=b"this is not json", headers=JSON_HEADERS)-- assert response.status == 400- assert await response.text() == "Malformed Payloads JSON"-- async def test_encode_decode_round_trip_over_http(self) -> None:- """A large payload is offloaded by /encode and fully restored by /decode."""- storage = InMemoryStorageProvider()- app = build_codec_server(codec=_make_real_codec(storage), cors_origins=[ALLOWED_ORIGIN])- original = _make_payload(SIZE_THRESHOLD + 100)- original_bytes = original.SerializeToString()-- async with TestClient(TestServer(app)) as client:- encode_response = await client.post(- "/encode",- data=_payloads_json(original),- headers={**JSON_HEADERS, "Origin": ALLOWED_ORIGIN},- )-- assert encode_response.status == 200- assert encode_response.content_type == "application/json"- assert encode_response.headers["Access-Control-Allow-Origin"] == ALLOWED_ORIGIN-- encoded_body = await encode_response.read()- encoded = _parse_payloads(encoded_body)- assert len(encoded.payloads) == 1- assert encoded.payloads[0].metadata["encoding"] == STORAGE_REF_ENCODING- assert len(storage.root) == 1-- decode_response = await client.post("/decode", data=encoded_body, headers=JSON_HEADERS)-- assert decode_response.status == 200- decoded = _parse_payloads(await decode_response.read())- assert len(decoded.payloads) == 1- assert decoded.payloads[0].SerializeToString() == original_bytes-- async def test_encode_small_payload_passes_through(self) -> None:- """A payload below the threshold comes back from /encode unchanged."""- storage = InMemoryStorageProvider()- app = build_codec_server(codec=_make_real_codec(storage), cors_origins=[ALLOWED_ORIGIN])- original = _make_payload(10)-- async with TestClient(TestServer(app)) as client:- response = await client.post("/encode", data=_payloads_json(original), headers=JSON_HEADERS)-- assert response.status == 200- encoded = _parse_payloads(await response.read())- assert encoded.payloads[0].SerializeToString() == original.SerializeToString()- assert len(storage.root) == 0-- @pytest.mark.parametrize(- ("topic", "codec_exc", "expected_status", "expected_text"),- [- ("storage_object_missing", StorageFileNotFoundError("blob gone"), 404, "Storage object not found"),- ("storage_io_error", OSError("disk on fire"), 502, "Storage I/O error"),- ("file_not_found_is_io_error", FileNotFoundError("no such file"), 502, "Storage I/O error"),- ],- )- async def test_codec_failures_map_to_http_errors(- self,- topic: str,- codec_exc: Exception,- expected_status: int,- expected_text: str,- ) -> None:- """Storage lookup and I/O failures inside the codec map to diagnostic HTTP errors with CORS."""- raising_codec = cast("StoragePayloadCodec", _RaisingCodec(codec_exc))- app = build_codec_server(codec=raising_codec, cors_origins=[ALLOWED_ORIGIN])-- async with TestClient(TestServer(app)) as client:- response = await client.post(- "/decode",- data=_payloads_json(_make_payload(10)),- headers={**JSON_HEADERS, "Origin": ALLOWED_ORIGIN},- )-- assert response.status == expected_status, f"Wrong status for {topic}"- assert await response.text() == expected_text- assert response.headers["Access-Control-Allow-Origin"] == ALLOWED_ORIGINdiff --git a/tests/unit/pipelex/temporal/test_conditional_worker.py b/tests/unit/pipelex/temporal/test_conditional_worker.pydeleted file mode 100644index 08be136bb..000000000--- a/tests/unit/pipelex/temporal/test_conditional_worker.py+++ /dev/null@@ -1,98 +0,0 @@-from __future__ import annotations--from typing import TYPE_CHECKING, Any-from unittest.mock import AsyncMock--import pytest--if TYPE_CHECKING:- from pytest_mock import MockerFixture--from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment-from pipelex.temporal.tprl.conditional_worker import with_conditional_worker-from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor--[email protected](loop_scope="class")-class TestConditionalWorker:- """Verify that with_conditional_worker restores task_queue after execution."""-- @pytest.fixture(autouse=True)- def _enable_temporal(self, mocker: MockerFixture) -> None:- """The deployment-level async-enabled guard inside- ``with_conditional_worker`` short-circuits the wrapper when- ``temporal.is_enabled`` is False — disabled-path coverage lives in- ``test_async_execution_not_enabled.py``. These tests exercise the- worker-bootstrap branches, so the config must read enabled.- """- config_root = mocker.MagicMock()- config_root.temporal.is_enabled = True- mocker.patch("pipelex.temporal.tprl.conditional_worker.get_config", return_value=config_root)-- async def test_task_queue_not_mutated_after_multiple_calls(self, mocker: MockerFixture) -> None:- """Calling a decorated method multiple times must not grow the task_queue name."""- original_queue = "my-task-queue"-- # Build a concrete WorkflowExecutor with INTERNAL worker environment- executor: WorkflowExecutor[Any, Any] = WorkflowExecutor(- task_queue=original_queue,- worker_environment=TemporalWorkerEnvironment.INTERNAL,- )-- # Mock temporal_client() to return a fake client- mocker.patch.object(executor, "temporal_client", new_callable=AsyncMock)-- # Mock get_task_manager().make_worker() to return an async context manager- mock_task_manager = mocker.MagicMock()- mock_worker_ctx = AsyncMock()- mock_worker_ctx.__aenter__ = AsyncMock(return_value=None)- mock_worker_ctx.__aexit__ = AsyncMock(return_value=False)- mock_task_manager.make_worker.return_value = mock_worker_ctx- mocker.patch(- "pipelex.temporal.tprl.conditional_worker.get_task_manager",- return_value=mock_task_manager,- )-- # Define a decorated method- @with_conditional_worker- async def fake_execute_workflow(_self_arg: WorkflowExecutor[Any, Any]) -> str: # noqa: RUF029- return "done"-- # Call multiple times on the same instance- for _index in range(5):- result = await fake_execute_workflow(executor)- assert result == "done"- assert executor.task_queue == original_queue, f"task_queue mutated after call: expected {original_queue!r}, got {executor.task_queue!r}"-- async def test_task_queue_restored_after_exception(self, mocker: MockerFixture) -> None:- """task_queue must be restored even when the wrapped function raises."""- original_queue = "error-queue"-- executor: WorkflowExecutor[Any, Any] = WorkflowExecutor(- task_queue=original_queue,- worker_environment=TemporalWorkerEnvironment.INTERNAL,- )-- mocker.patch.object(executor, "temporal_client", new_callable=AsyncMock)-- mock_task_manager = mocker.MagicMock()- mock_worker_ctx = AsyncMock()- mock_worker_ctx.__aenter__ = AsyncMock(return_value=None)- mock_worker_ctx.__aexit__ = AsyncMock(return_value=False)- mock_task_manager.make_worker.return_value = mock_worker_ctx- mocker.patch(- "pipelex.temporal.tprl.conditional_worker.get_task_manager",- return_value=mock_task_manager,- )-- @with_conditional_worker- async def failing_execute_workflow(_self_arg: WorkflowExecutor[Any, Any]) -> str: # noqa: RUF029- msg = "workflow failed"- raise RuntimeError(msg)-- with pytest.raises(RuntimeError, match="workflow failed"):- await failing_execute_workflow(executor)-- assert executor.task_queue == original_queue, (- f"task_queue not restored after exception: expected {original_queue!r}, got {executor.task_queue!r}"- )diff --git a/tests/unit/pipelex/temporal/test_config_field_validators.py b/tests/unit/pipelex/temporal/test_config_field_validators.pydeleted file mode 100644index 608edd6bc..000000000--- a/tests/unit/pipelex/temporal/test_config_field_validators.py+++ /dev/null@@ -1,245 +0,0 @@-"""Unit tests for the Pydantic field validators on ``QueueOptions``,-``HandleOptions``, ``WorkerRuntimeProfile``, and ``RetryPolicyConfigBase``.--These guards reject negative/zero values that would pass Pydantic's basic-type coercion but cause runtime errors at the Temporal SDK boundary-(``Worker(...)``, ``workflow.execute_activity(...)``). Catching them at-config load gives a clear error pointing at the offending field instead of-a cryptic SDK error mid-workflow.-"""--from datetime import timedelta--import pytest-from pydantic import ValidationError--from pipelex.temporal.config_temporal import (- HandleOptions,- QueueOptions,- RetryPolicyConfig,- RetryPolicyConfigOverlay,- WorkerRuntimeProfile,- WorkerTuningMode,-)---def _valid_profile_kwargs() -> dict[str, object]:- """Build a kwargs dict for ``WorkerRuntimeProfile`` with all valid values.- Tests override one field at a time to verify each validator independently.- """- return {- "tuning_mode": WorkerTuningMode.EXPLICIT,- "max_cached_workflows": 10000,- "max_concurrent_workflow_tasks": 1000,- "max_concurrent_activities": 1000,- "max_concurrent_local_activities": 1000,- "max_concurrent_workflow_task_polls": 100,- "max_concurrent_activity_task_polls": 100,- "max_activities_per_second": 1000,- "sticky_queue_schedule_to_start_timeout": timedelta(minutes=30),- "max_heartbeat_throttle_interval": timedelta(minutes=60),- "default_heartbeat_throttle_interval": timedelta(minutes=60),- "graceful_shutdown_timeout": timedelta(minutes=30),- }---class TestConfigFieldValidators:- """Reject non-positive timedeltas + zero/negative ints/floats where invalid."""-- @pytest.mark.parametrize(- "field_name",- [- "start_to_close_timeout",- "schedule_to_close_timeout",- "schedule_to_start_timeout",- "heartbeat_timeout",- ],- )- def test_queue_options_rejects_non_positive_timedelta(self, field_name: str) -> None:- """Each ``QueueOptions`` timedelta field rejects zero and negative."""- with pytest.raises(ValidationError) as exc_info:- QueueOptions(**{field_name: timedelta(seconds=-5)}) # type: ignore[arg-type] # pyright: ignore[reportArgumentType]- errors = exc_info.value.errors()- assert any(err["loc"] == (field_name,) and err["type"] == "greater_than" for err in errors)- with pytest.raises(ValidationError):- QueueOptions(**{field_name: timedelta(0)}) # type: ignore[arg-type] # pyright: ignore[reportArgumentType]-- def test_queue_options_rejects_non_positive_rate_cap(self) -> None:- """``max_task_queue_activities_per_second`` must be > 0."""- with pytest.raises(ValidationError):- QueueOptions(max_task_queue_activities_per_second=0)- with pytest.raises(ValidationError):- QueueOptions(max_task_queue_activities_per_second=-1.5)-- def test_queue_options_accepts_none_and_positive_values(self) -> None:- """Sanity: all-None and explicitly positive values still construct."""- QueueOptions() # all defaults / None — must not raise- QueueOptions(- start_to_close_timeout=timedelta(minutes=5),- heartbeat_timeout=timedelta(seconds=30),- max_task_queue_activities_per_second=100,- )-- def test_handle_options_rejects_non_positive_start_to_close(self) -> None:- """``HandleOptions.start_to_close_timeout`` must be positive."""- with pytest.raises(ValidationError):- HandleOptions(start_to_close_timeout=timedelta(seconds=-1))- with pytest.raises(ValidationError):- HandleOptions(start_to_close_timeout=timedelta(0))- # Positive and None both fine.- HandleOptions()- HandleOptions(start_to_close_timeout=timedelta(minutes=10))-- @pytest.mark.parametrize(- ("field_name", "bad_value"),- [- ("max_concurrent_workflow_tasks", 0), # gt=0: 0 unsafe (worker can't progress)- ("max_concurrent_workflow_tasks", -1),- ("max_concurrent_workflow_task_polls", 0),- ("max_concurrent_workflow_task_polls", -1),- ("max_cached_workflows", -1), # ge=0- ("max_concurrent_activities", -1),- ("max_concurrent_local_activities", -1),- ("max_concurrent_activity_task_polls", -1),- ],- )- def test_worker_runtime_profile_rejects_invalid_int(self, field_name: str, bad_value: int) -> None:- kwargs = _valid_profile_kwargs()- kwargs[field_name] = bad_value- with pytest.raises(ValidationError):- WorkerRuntimeProfile(**kwargs) # type: ignore[arg-type]-- def test_worker_runtime_profile_accepts_zero_concurrent_activities(self) -> None:- """Workflow-only worker pattern: ``max_concurrent_activities = 0``- is valid (router profile example uses this).- """- kwargs = _valid_profile_kwargs()- kwargs["max_concurrent_activities"] = 0- kwargs["max_concurrent_activity_task_polls"] = 0- WorkerRuntimeProfile(**kwargs) # type: ignore[arg-type] # must not raise-- @pytest.mark.parametrize("bad_value", [0, -10.0])- def test_worker_runtime_profile_rejects_non_positive_rate(self, bad_value: float) -> None:- """``max_activities_per_second`` must be > 0."""- kwargs = _valid_profile_kwargs()- kwargs["max_activities_per_second"] = bad_value- with pytest.raises(ValidationError):- WorkerRuntimeProfile(**kwargs) # type: ignore[arg-type]-- @pytest.mark.parametrize(- "field_name",- [- "sticky_queue_schedule_to_start_timeout",- "max_heartbeat_throttle_interval",- "default_heartbeat_throttle_interval",- "graceful_shutdown_timeout",- ],- )- @pytest.mark.parametrize("bad_value", [timedelta(seconds=-1), timedelta(0)])- def test_worker_runtime_profile_rejects_non_positive_timedelta(self, field_name: str, bad_value: timedelta) -> None:- """Both negative and zero must be rejected — ``gt=timedelta(0)`` is the right- bound (a heartbeat throttle of 0 would mean "throttle immediately" and break- the worker).- """- kwargs = _valid_profile_kwargs()- kwargs[field_name] = bad_value- with pytest.raises(ValidationError):- WorkerRuntimeProfile(**kwargs) # type: ignore[arg-type]-- @pytest.mark.parametrize("bad_value", [timedelta(seconds=-1), timedelta(0)])- def test_retry_policy_config_rejects_non_positive_initial_interval(self, bad_value: timedelta) -> None:- """``initial_interval`` is the seed delay between the first attempt and the- first retry. ``gt=timedelta(0)`` rejects both negative and zero.- """- with pytest.raises(ValidationError):- RetryPolicyConfig(- initial_interval=bad_value,- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=3,- non_retryable_error_types=[],- )-- @pytest.mark.parametrize("bad_value", [timedelta(seconds=-1), timedelta(0)])- def test_retry_policy_config_rejects_non_positive_maximum_interval(self, bad_value: timedelta) -> None:- """Either a positive timedelta or the literal 'unlimited' — both negative- and zero are rejected by the ``Annotated[timedelta, Field(gt=...)]`` branch- of the Union.- """- with pytest.raises(ValidationError):- RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval=bad_value,- maximum_attempts=3,- non_retryable_error_types=[],- )-- @pytest.mark.parametrize("bad_value", [0, -1, -100])- def test_retry_policy_config_rejects_non_positive_maximum_attempts(self, bad_value: int) -> None:- """Either a positive int or the literal 'unlimited' — zero and negatives- are rejected by the ``Annotated[int, Field(gt=0)]`` branch of the Union.- """- with pytest.raises(ValidationError):- RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=bad_value,- non_retryable_error_types=[],- )-- def test_retry_policy_overlay_inherits_validators(self) -> None:- """The overlay subclass inherits the validators from the base — both- layers must reject the same invalid values, so a per-queue override- can't silently smuggle in a bad scalar.- """- with pytest.raises(ValidationError):- RetryPolicyConfigOverlay(- initial_interval=timedelta(0),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=3,- non_retryable_error_types_extra=[],- )-- def test_retry_policy_accepts_unlimited_literals(self) -> None:- """The Literal['unlimited'] escape hatches must not be rejected by the- positive-timedelta / positive-int validators.- """- RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts="unlimited",- non_retryable_error_types=[],- )-- def test_retry_policy_accepts_concrete_positive_union_branches(self) -> None:- """Happy path for the ``Annotated[..., Field(gt=...)]`` branches of the two- Unions: a positive timedelta for ``maximum_interval`` and a positive int- for ``maximum_attempts``. Complements ``test_retry_policy_accepts_unlimited_literals``- which exercises the ``Literal['unlimited']`` branch.- """- config = RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval=timedelta(minutes=5),- maximum_attempts=10,- non_retryable_error_types=[],- )- assert config.maximum_interval == timedelta(minutes=5)- assert config.maximum_attempts == 10-- def test_retry_policy_rejects_backoff_coefficient_below_one(self) -> None:- """``backoff_coefficient < 1`` would shrink retry intervals over time- — almost certainly a typo. Pydantic ge=1.0 catches it.- """- with pytest.raises(ValidationError):- RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=0.5,- maximum_interval="unlimited",- maximum_attempts=3,- non_retryable_error_types=[],- )diff --git a/tests/unit/pipelex/temporal/test_config_temporal_optional_dep.py b/tests/unit/pipelex/temporal/test_config_temporal_optional_dep.pydeleted file mode 100644index 45f6e02f2..000000000--- a/tests/unit/pipelex/temporal/test_config_temporal_optional_dep.py+++ /dev/null@@ -1,76 +0,0 @@-"""Regression: ``pipelex.temporal.config_temporal`` MUST NOT import ``temporalio``-at module level. ``temporalio`` is in ``[project.optional-dependencies].temporal``;-``pipelex.system.configuration.configs`` imports this module unconditionally,-so a top-level temporalio import breaks every install that didn't opt into the-``temporal`` extra.-"""--import ast-from pathlib import Path---def _is_type_checking_guard(test_node: ast.expr) -> bool:- """Recognize ``if TYPE_CHECKING:`` and ``if typing.TYPE_CHECKING:`` guards.-- Their body is only evaluated by type checkers, never at runtime, so imports- inside them are safe under the optional-dependency contract.- """- if isinstance(test_node, ast.Name) and test_node.id == "TYPE_CHECKING":- return True- return isinstance(test_node, ast.Attribute) and test_node.attr == "TYPE_CHECKING"---def _scan_for_runtime_temporalio_imports(nodes: list[ast.stmt], forbidden: list[str]) -> None:- """Recursively walk module-level statements that execute at import time and- record any ``temporalio`` import found in them.-- Handles guarded top-level imports the AI bot flagged on PR #880: anything- inside ``try`` / ``except`` / ``if`` (other than ``if TYPE_CHECKING:``)- still runs when the module is imported, so it counts as a runtime import.- Function and class bodies are skipped — their imports only fire when the- function is called, which is the intended lazy-import pattern.- """- for node in nodes:- if isinstance(node, ast.ImportFrom) and node.module and node.module.startswith("temporalio"):- forbidden.append(f"line {node.lineno}: from {node.module} import ...")- elif isinstance(node, ast.Import):- for alias in node.names:- if alias.name.startswith("temporalio"):- forbidden.append(f"line {node.lineno}: import {alias.name}")- elif isinstance(node, ast.If):- # The body of ``if TYPE_CHECKING:`` is invisible to the runtime — skip it.- # The else branch DOES run at runtime, so scan it regardless.- if not _is_type_checking_guard(node.test):- _scan_for_runtime_temporalio_imports(node.body, forbidden)- _scan_for_runtime_temporalio_imports(node.orelse, forbidden)- elif isinstance(node, ast.Try):- _scan_for_runtime_temporalio_imports(node.body, forbidden)- for handler in node.handlers:- _scan_for_runtime_temporalio_imports(handler.body, forbidden)- _scan_for_runtime_temporalio_imports(node.orelse, forbidden)- _scan_for_runtime_temporalio_imports(node.finalbody, forbidden)- # ast.FunctionDef / ast.AsyncFunctionDef / ast.ClassDef are intentionally- # skipped — bodies don't execute at module import time, so a lazy- # ``from temporalio.common import RetryPolicy`` inside a function body- # is the intended pattern.---class TestConfigTemporalOptionalDep:- def test_no_module_level_temporalio_import(self) -> None:- """No ``import temporalio`` or ``from temporalio... import ...`` reachable- at module import time (including imports nested in ``try`` / ``except``- / ``if`` branches other than ``if TYPE_CHECKING:``). ``TYPE_CHECKING``- blocks and function-local imports are fine.- """- source_path = Path(__file__).resolve().parents[4] / "pipelex" / "temporal" / "config_temporal.py"- assert source_path.is_file(), f"expected config_temporal.py at {source_path}"- tree = ast.parse(source_path.read_text())-- forbidden: list[str] = []- _scan_for_runtime_temporalio_imports(tree.body, forbidden)-- assert not forbidden, (- "config_temporal.py imports temporalio at module level — "- "this breaks installs without the 'temporal' extra. "- f"Move under TYPE_CHECKING or into function bodies. Found: {forbidden}"- )diff --git a/tests/unit/pipelex/temporal/test_content_generator_in_workflow.py b/tests/unit/pipelex/temporal/test_content_generator_in_workflow.pydeleted file mode 100644index f50e9ea45..000000000--- a/tests/unit/pipelex/temporal/test_content_generator_in_workflow.py+++ /dev/null@@ -1,353 +0,0 @@-"""Unit tests for ContentGeneratorInWorkflow.--Validates that each protocol method dispatches the correct activity with the-expected kwargs, including the hybrid ``task_queue`` routing via-``WorkerConfig.resolve_dispatch`` (empty ``activity_queues`` → omit-``task_queue`` so Temporal routes to the workflow's queue), and that-Pipelex never customizes ``activity_id`` (the SDK assigns deterministic-integers per workflow run).--Per-call meaning is carried in the ``summary=`` kwarg, formatted via-``pipelex.temporal.tprl.observability.build_activity_summary``.-"""--from typing import Any--import pytest-from pydantic import BaseModel-from pytest_mock import MockerFixture-from temporalio.exceptions import ActivityError, ApplicationError--from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams-from pipelex.cogt.extract.extract_input import ExtractInput-from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams-from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt-from pipelex.cogt.llm.llm_prompt import LLMPrompt-from pipelex.cogt.llm.llm_setting import LLMSetting-from pipelex.core.stuffs.image_content import ImageContent-from pipelex.core.stuffs.page_content import PageContent-from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.tprl_content_generation.content_generator_in_workflow import ContentGeneratorInWorkflow---class _Person(BaseModel):- name: str- age: int---def _make_job_metadata() -> JobMetadata:- return JobMetadata(user_id="test-user", pipeline_run_id="test-run", pipe_code="my_pipe")---def _make_generator() -> ContentGeneratorInWorkflow:- return ContentGeneratorInWorkflow()---def _make_llm_setting() -> LLMSetting:- return LLMSetting(model="test-llm", temperature=0.5)---def _make_page_content() -> PageContent:- return PageContent(text_and_images=TextAndImagesContent(text=None, images=None))---def _make_activity_error(message: str = "activity failed") -> ActivityError:- """Construct an ``ActivityError`` with stub identifiers for tests."""- return ActivityError(- message,- scheduled_event_id=0,- started_event_id=0,- identity="test-identity",- activity_type="test-activity",- activity_id="test-activity-id",- retry_state=None,- )---def patch_workflow_runtime(mocker: MockerFixture) -> None:- """Patch ``workflow.unsafe.is_replaying_history_events`` so the error-path- tests can exercise the catch block outside a real workflow context.-- The non-error tests rely solely on ``temporalio.workflow.execute_activity``- being patched, which the individual tests do themselves.- """- mocker.patch("temporalio.workflow.unsafe.is_replaying_history_events", return_value=False)--[email protected](loop_scope="class")[email protected]("patch_workflow_runtime")-class TestContentGeneratorInWorkflow:- """Per-method assertions on the activity-dispatch kwargs."""-- async def test_make_llm_text_omits_activity_id_and_sets_summary(self, mocker: MockerFixture) -> None:- """``make_llm_text`` must never customize ``activity_id`` (the SDK assigns it)- and must carry the per-call meaning in ``summary=``.- """- mock_execute = mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock)- mock_execute.return_value = "stub text"- generator = _make_generator()-- await generator.make_llm_text(- job_metadata=_make_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- llm_setting_main=_make_llm_setting(),- llm_prompt_for_text=LLMPrompt(user_text="hello"),- )-- kwargs = mock_execute.call_args.kwargs- assert kwargs.get("activity_id") is None- assert kwargs.get("summary") == "LLM text · pipe=my_pipe · model=test-llm"- # Empty activity_queues → hybrid fallback omits task_queue.- assert "task_queue" not in kwargs-- @pytest.mark.parametrize(- ("caller", "expected_summary_prefix"),- [- ("make_object", "LLM object · pipe=my_pipe · class=_Person"),- ("make_object_list", "LLM object list · pipe=my_pipe · class=_Person"),- ("make_single_image", "Img gen 1× · pipe=my_pipe · model=test-handle"),- ("make_image_list", "Img gen N× · pipe=my_pipe · model=test-handle · n=2"),- ("make_templated_text", "Templated text · pipe=my_pipe"),- ("make_render_page_views", "Render page views · pipe=my_pipe"),- ],- )- async def test_non_llm_text_methods_omit_activity_id_and_set_summary(- self,- mocker: MockerFixture,- caller: str,- expected_summary_prefix: str,- ) -> None:- """Every non-llm-text method routes through ``resolve_dispatch`` and must- never customize ``activity_id``. The per-call meaning lives in ``summary=``.- """- mock_execute = mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock)- mock_execute.return_value = self._stub_for(caller)- generator = _make_generator()-- await self._invoke(generator, caller)-- kwargs = mock_execute.call_args.kwargs- assert kwargs.get("activity_id") is None, f"{caller} must not customize activity_id"- assert kwargs.get("summary") == expected_summary_prefix- assert "task_queue" not in kwargs, f"{caller} should omit task_queue with empty activity_queues (got {kwargs.get('task_queue')!r})"-- async def test_make_extract_pages_dispatches_extract_only_when_no_page_views(self, mocker: MockerFixture) -> None:- """When ``should_include_page_views`` is false, only the extract activity is called."""- mock_execute = mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock)- mock_execute.return_value = [_make_page_content()]- generator = _make_generator()-- await generator.make_extract_pages(- job_metadata=_make_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- extract_input=ExtractInput(image_uri="test://image.png"),- extract_handle="test-handle",- extract_job_params=ExtractJobParams.make_default_extract_job_params(),- extract_job_config=ExtractJobConfig(),- )-- assert mock_execute.call_count == 1- kwargs = mock_execute.call_args.kwargs- assert "task_queue" not in kwargs- assert kwargs.get("activity_id") is None- assert kwargs.get("summary") == "Extract pages · pipe=my_pipe · handle=test-handle"-- async def test_make_extract_pages_image_uri_with_page_views_skips_render_activity(self, mocker: MockerFixture) -> None:- """When ``image_uri`` is set and ``should_include_page_views`` is true, the- single-image branch builds page_view inline without calling act_render_page_views.- """- mock_execute = mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock)- mock_execute.return_value = [_make_page_content()]- generator = _make_generator()-- params = ExtractJobParams.make_default_extract_job_params()- params.should_include_page_views = True- await generator.make_extract_pages(- job_metadata=_make_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- extract_input=ExtractInput(image_uri="test://image.png"),- extract_handle="test-handle",- extract_job_params=params,- extract_job_config=ExtractJobConfig(),- )-- # Only the extract activity should have been dispatched; the page-view branch- # synthesizes ImageContent inline from image_uri.- assert mock_execute.call_count == 1- assert mock_execute.call_args.kwargs.get("activity_id") is None- assert mock_execute.call_args.kwargs.get("summary") == "Extract pages · pipe=my_pipe · handle=test-handle"-- async def test_make_extract_pages_document_uri_with_page_views_dispatches_two_activities(self, mocker: MockerFixture) -> None:- """When ``document_uri`` is set and ``should_include_page_views`` is true, both- the extract and render activities are dispatched. ``activity_id`` is never- customized on either; the SDK assigns sequential integers.- """- page = _make_page_content()- page_view = ImageContent(url="test://view.png")- return_queue: list[Any] = [[page], [page_view]]-- def _side_effect(*_args: Any, **_kwargs: Any) -> Any:- return return_queue.pop(0)-- mock_execute = mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock)- mock_execute.side_effect = _side_effect- generator = _make_generator()-- params = ExtractJobParams.make_default_extract_job_params()- params.should_include_page_views = True- result = await generator.make_extract_pages(- job_metadata=_make_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- extract_input=ExtractInput(document_uri="test://doc.pdf"),- extract_handle="test-handle",- extract_job_params=params,- extract_job_config=ExtractJobConfig(),- )-- assert mock_execute.call_count == 2- for call in mock_execute.call_args_list:- assert call.kwargs.get("activity_id") is None- assert "task_queue" not in call.kwargs- summaries = [call.kwargs.get("summary") for call in mock_execute.call_args_list]- assert summaries == [- "Extract pages · pipe=my_pipe · handle=test-handle",- "Render page views (extract) · pipe=my_pipe",- ]- assert result[0].page_view is page_view-- async def test_make_templated_text_updates_content_generation_job_id(self, mocker: MockerFixture) -> None:- """``make_templated_text`` must set ``job_metadata.content_generation_job_id``- consistently with every other activity-dispatching method.- """- mock_execute = mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock)- mock_execute.return_value = "rendered"- generator = _make_generator()- job_metadata = _make_job_metadata()-- await generator.make_templated_text(- job_metadata=job_metadata,- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- context={"key": "value"},- template="{{ key }}",- )-- assert job_metadata.content_generation_job_id == "make_templated_text"-- async def test_activity_error_with_application_error_translates_to_temporal_error(self, mocker: MockerFixture) -> None:- """``ActivityError(cause=ApplicationError)`` must be translated to ``TemporalError``- with the original error_type and message preserved, chained via ``__cause__``.- """- from pipelex.temporal.tprl.temporal_error import TemporalError # noqa: PLC0415-- app_error = ApplicationError("upstream failure", type="MyDomainError")- activity_error = _make_activity_error()- activity_error.__cause__ = app_error- mock_execute = mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock)- mock_execute.side_effect = activity_error- generator = _make_generator()-- with pytest.raises(TemporalError) as exc_info:- await generator.make_llm_text(- job_metadata=_make_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- llm_setting_main=_make_llm_setting(),- llm_prompt_for_text=LLMPrompt(user_text="hello"),- )-- assert exc_info.value.message == "upstream failure"- assert exc_info.value.type == "MyDomainError"- assert exc_info.value.__cause__ is activity_error-- async def test_activity_error_with_other_cause_re_raises_unchanged(self, mocker: MockerFixture) -> None:- """An ``ActivityError`` whose cause is NOT ``ApplicationError`` must propagate- unchanged — we must not silently swallow or convert non-Application failures.- """- non_app_cause = RuntimeError("unexpected runtime error")- activity_error = _make_activity_error()- activity_error.__cause__ = non_app_cause- mock_execute = mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock)- mock_execute.side_effect = activity_error- generator = _make_generator()-- with pytest.raises(ActivityError) as exc_info:- await generator.make_llm_text(- job_metadata=_make_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- llm_setting_main=_make_llm_setting(),- llm_prompt_for_text=LLMPrompt(user_text="hello"),- )-- assert exc_info.value is activity_error-- @staticmethod- def _stub_for(caller: str) -> Any:- """Return a stub return value matching the activity's contract for each method."""- if caller == "make_object":- return _Person(name="John", age=30)- if caller == "make_object_list":- return [_Person(name="John", age=30)]- if caller == "make_single_image":- return [ImageContent(url="test://img.png")]- if caller == "make_image_list":- return [ImageContent(url="test://img1.png"), ImageContent(url="test://img2.png")]- if caller == "make_templated_text":- return "rendered"- if caller == "make_render_page_views":- return [ImageContent(url="test://view.png")]- msg = f"Unknown caller: {caller}"- raise AssertionError(msg)-- @staticmethod- async def _invoke(generator: ContentGeneratorInWorkflow, caller: str) -> Any:- """Invoke the named method on the generator with reasonable default args."""- job_metadata = _make_job_metadata()- if caller == "make_object":- return await generator.make_object(- job_metadata=job_metadata,- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- object_class=_Person,- llm_setting_for_object=_make_llm_setting(),- llm_prompt_for_object=LLMPrompt(user_text="hello"),- )- if caller == "make_object_list":- return await generator.make_object_list(- job_metadata=job_metadata,- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- object_class=_Person,- llm_setting_for_object_list=_make_llm_setting(),- llm_prompt_for_object_list=LLMPrompt(user_text="hello"),- )- if caller == "make_single_image":- return await generator.make_single_image(- job_metadata=job_metadata,- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- img_gen_handle="test-handle",- img_gen_prompt=ImgGenPrompt(positive_text="prompt"),- )- if caller == "make_image_list":- return await generator.make_image_list(- job_metadata=job_metadata,- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- img_gen_handle="test-handle",- img_gen_prompt=ImgGenPrompt(positive_text="prompt"),- nb_images=2,- )- if caller == "make_templated_text":- return await generator.make_templated_text(- job_metadata=job_metadata,- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- context={"key": "value"},- template="{{ key }}",- )- if caller == "make_render_page_views":- return await generator.make_render_page_views(- job_metadata=job_metadata,- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- extract_input=ExtractInput(document_uri="test://doc.pdf"),- extract_handle="test-handle",- )- msg = f"Unknown caller: {caller}"- raise AssertionError(msg)diff --git a/tests/unit/pipelex/temporal/test_content_generator_in_workflow_object_fidelity.py b/tests/unit/pipelex/temporal/test_content_generator_in_workflow_object_fidelity.pydeleted file mode 100644index 39a006c74..000000000--- a/tests/unit/pipelex/temporal/test_content_generator_in_workflow_object_fidelity.py+++ /dev/null@@ -1,98 +0,0 @@-"""Unit tests for the dry-run object-mock fidelity guard on the Temporal arm (review F2).--The activity (``act_llm_gen_object*``) builds the mock from the schema-reconstructed class, then-``ContentGeneratorInWorkflow.make_object`` re-validates it against the original class. When the original-class enforces an invariant the JSON-schema round-trip cannot capture (here a custom ``@field_validator``),-that re-validation fails — and must surface as a clear typed ``DryRunObjectFidelityError`` rather than-an opaque ``pydantic.ValidationError`` mid-workflow. The guard is armed by ``run_mode=DRY`` only: a LIVE-provider's invalid output keeps its existing ``ValidationError``. ``execute_activity`` is mocked to return-the synthetic object, so no Temporal server (or real provider) is involved.-"""--import pytest-from pydantic import BaseModel, ValidationError, field_validator-from pytest_mock import MockerFixture--from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams-from pipelex.cogt.content_generation.exceptions import DryRunObjectFidelityError-from pipelex.cogt.llm.llm_prompt import LLMPrompt-from pipelex.cogt.llm.llm_setting import LLMSetting-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.tprl_content_generation.content_generator_in_workflow import ContentGeneratorInWorkflow---class ConstrainedName(BaseModel):- """Original output class with an invariant the JSON-schema round-trip cannot capture."""-- name: str-- @field_validator("name")- @classmethod- def _require_prefix(cls, value: str) -> str:- if not value.startswith("PFX_"):- msg = "name must start with 'PFX_'"- raise ValueError(msg)- return value---class RawName(BaseModel):- """Stand-in for what the (mocked) activity returns: a permissive reconstructed-class instance."""-- name: str---def _make_generator() -> ContentGeneratorInWorkflow:- return ContentGeneratorInWorkflow()---def _job_metadata() -> JobMetadata:- return JobMetadata(user_id="u", pipeline_run_id="run", pipe_code="my_pipe")--[email protected](loop_scope="class")[email protected]("patch_workflow_runtime")-class TestContentGeneratorInWorkflowObjectFidelity:- @pytest.fixture- def patch_workflow_runtime(self, mocker: MockerFixture) -> None:- mocker.patch("temporalio.workflow.unsafe.is_replaying_history_events", return_value=False)-- async def test_make_object_raises_typed_error_on_fidelity_gap(self, mocker: MockerFixture) -> None:- """A dry-built object whose re-validation fails surfaces DryRunObjectFidelityError."""- mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock, return_value=RawName(name="bad"))-- with pytest.raises(DryRunObjectFidelityError) as exc_info:- await _make_generator().make_object(- job_metadata=_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),- object_class=ConstrainedName,- llm_setting_for_object=LLMSetting(model="test-llm", temperature=0.5),- llm_prompt_for_object=LLMPrompt(user_text="hello"),- )- assert ConstrainedName.__name__ in str(exc_info.value)-- async def test_make_object_list_raises_typed_error_on_fidelity_gap(self, mocker: MockerFixture) -> None:- """The list path applies the same per-item guard."""- mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock, return_value=[RawName(name="bad")])-- with pytest.raises(DryRunObjectFidelityError):- await _make_generator().make_object_list(- job_metadata=_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.DRY),- object_class=ConstrainedName,- llm_setting_for_object_list=LLMSetting(model="test-llm", temperature=0.5),- llm_prompt_for_object_list=LLMPrompt(user_text="hello"),- )-- async def test_live_run_keeps_raw_validation_error(self, mocker: MockerFixture) -> None:- """On a LIVE run the guard is inert: a re-validation failure stays a raw ValidationError."""- mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock, return_value=RawName(name="bad"))-- with pytest.raises(ValidationError):- await _make_generator().make_object(- job_metadata=_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- object_class=ConstrainedName,- llm_setting_for_object=LLMSetting(model="test-llm", temperature=0.5),- llm_prompt_for_object=LLMPrompt(user_text="hello"),- )diff --git a/tests/unit/pipelex/temporal/test_default_activity_id_collision_bug.py b/tests/unit/pipelex/temporal/test_default_activity_id_collision_bug.pydeleted file mode 100644index 0c32834f0..000000000--- a/tests/unit/pipelex/temporal/test_default_activity_id_collision_bug.py+++ /dev/null@@ -1,75 +0,0 @@-"""Canonical regression gate for the activity-id collision bug.--Background: ``ContentGeneratorInWorkflow`` used to default ``activity_id`` to a-constant per-method label (e.g. ``"craft-text"``). Temporal requires-``activity_id`` to be unique within a single workflow execution, so any-workflow that called the same method twice in a row crashed on the second-call.--After the activity-id naming redesign (see-``wip/temporal-primitives/id-and-naming-design.md``), Pipelex never passes-``activity_id=``: the Temporal Python SDK auto-assigns deterministic-sequential integers per workflow run. This test pins the invariant —-``activity_id`` is omitted from the dispatch kwargs — and trusts the SDK to-produce unique-per-run ids by construction.-"""--import pytest-from pytest_mock import MockerFixture--from pipelex.cogt.content_generation.cogt_run_params import CogtRunParams-from pipelex.cogt.llm.llm_prompt import LLMPrompt-from pipelex.cogt.llm.llm_setting import LLMSetting-from pipelex.pipe_run.pipe_run_mode import PipeRunMode-from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.tprl_content_generation.content_generator_in_workflow import ContentGeneratorInWorkflow---def _make_job_metadata() -> JobMetadata:- return JobMetadata(user_id="test-user", pipeline_run_id="test-run")---def _make_generator() -> ContentGeneratorInWorkflow:- return ContentGeneratorInWorkflow()---def _make_llm_setting() -> LLMSetting:- return LLMSetting(model="test-llm", temperature=0.5)--[email protected](loop_scope="class")-class TestDefaultActivityIdCollisionBug:- async def test_two_default_activity_calls_in_one_workflow_produce_distinct_activity_ids(- self,- mocker: MockerFixture,- ) -> None:- """Two back-to-back ``make_llm_text`` calls in the same workflow must- dispatch without colliding on ``activity_id``.-- The redesign satisfies this by never passing ``activity_id=`` — the- Temporal SDK assigns deterministic sequential integers per workflow- run, which guarantees per-``(workflow_id, run_id)`` uniqueness by- construction and is replay-safe (assigned by history position).- """- mock_execute = mocker.patch("temporalio.workflow.execute_activity", new_callable=mocker.AsyncMock)- mock_execute.return_value = "stub text"- generator = _make_generator()-- await generator.make_llm_text(- job_metadata=_make_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- llm_setting_main=_make_llm_setting(),- llm_prompt_for_text=LLMPrompt(user_text="hello"),- )- await generator.make_llm_text(- job_metadata=_make_job_metadata(),- cogt_run_params=CogtRunParams(run_mode=PipeRunMode.LIVE),- llm_setting_main=_make_llm_setting(),- llm_prompt_for_text=LLMPrompt(user_text="hello again"),- )-- assert mock_execute.call_count == 2- for call in mock_execute.call_args_list:- assert call.kwargs.get("activity_id") is None, (- "Pipelex must not customize activity_id — the SDK assigns deterministic integers per workflow run."- )diff --git a/tests/unit/pipelex/temporal/test_delivery_activity_arg.py b/tests/unit/pipelex/temporal/test_delivery_activity_arg.pydeleted file mode 100644index e5b6423cb..000000000--- a/tests/unit/pipelex/temporal/test_delivery_activity_arg.py+++ /dev/null@@ -1,59 +0,0 @@-"""Unit round-trip test for ``DeliveryActivityArg``.--Pins nested-model serialization through ``model_dump_json`` / ``model_validate_json``-so a regression in the nested ``ErrorReport`` round-trip is caught without standing-up a real Temporal worker.-"""--from pipelex.base_exceptions import ErrorDomain, ErrorReport-from pipelex.cogt.inference.error_classification import UserAction, UserActionKind-from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus, WebhookTarget-from pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg---class TestDeliveryActivityArg:- def test_json_roundtrip_with_populated_error_report(self) -> None:- """A ``DeliveryActivityArg`` carrying a fully populated nested ``ErrorReport``- (including a nested ``UserAction``) round-trips through JSON serialization.- """- error_report = ErrorReport(- error_type="CogtError",- message="rate limited on the worker",- title="AI inference failed",- type_uri="https://docs.pipelex.com/latest/errors/cogt-error/",- error_category="capacity",- error_domain=ErrorDomain.RUNTIME,- retryable=False,- user_action=UserAction(kind=UserActionKind.CHECK_BILLING, detail="check your billing page"),- model="gpt-5",- provider="openai",- )- arg = DeliveryActivityArg(- user_id="user-123",- pipeline_run_id="plr-456",- delivery_assignment=DeliveryAssignment(webhooks=[WebhookTarget(url="https://example.com/callback")]),- status=DeliveryStatus.FAILED,- error_report=error_report,- request_id="req-abc",- )-- restored = DeliveryActivityArg.model_validate_json(arg.model_dump_json())-- assert restored == arg- assert restored.error_report == error_report- assert restored.error_report is not None- assert restored.error_report.user_action == error_report.user_action- assert restored.request_id == "req-abc"-- def test_request_id_defaults_to_none(self) -> None:- """A run dispatched without an inbound API request id (e.g. cron, internal job) carries ``request_id=None``."""- arg = DeliveryActivityArg(- user_id="user-123",- pipeline_run_id="plr-no-req",- delivery_assignment=DeliveryAssignment(),- status=DeliveryStatus.COMPLETED,- )-- assert arg.request_id is None- restored = DeliveryActivityArg.model_validate_json(arg.model_dump_json())- assert restored.request_id is Nonediff --git a/tests/unit/pipelex/temporal/test_dispatch_options_no_temporalio.py b/tests/unit/pipelex/temporal/test_dispatch_options_no_temporalio.pydeleted file mode 100644index 95ce3927b..000000000--- a/tests/unit/pipelex/temporal/test_dispatch_options_no_temporalio.py+++ /dev/null@@ -1,85 +0,0 @@-"""Regression: ``DispatchOptions`` (a frozen Pydantic dataclass) MUST be-importable AND constructible without ``temporalio`` installed.--``temporalio`` is in ``[project.optional-dependencies].temporal``;-``pipelex.system.configuration.configs`` imports ``config_temporal`` unconditionally.-The ``retry_policy`` field is annotated ``RetryPolicy`` (a ``temporalio`` type),-which is bound to ``Any`` at runtime via the ``else`` branch of the-``TYPE_CHECKING`` guard. Without that runtime binding the dataclass carries an-unresolved forward ref: it would still import (Pydantic defers the schema) but-raise ``PydanticUserError`` the moment a ``DispatchOptions`` is constructed on an-install lacking the ``temporal`` extra.--The sibling ``test_config_temporal_optional_dep.py`` AST-scans for module-level-``temporalio`` imports; it does NOT catch a regression to a bare forward ref-(which adds no import statement). This runtime check is the complement: it-exercises the actual class-definition + construction path with ``temporalio``-made unavailable, in a fresh subprocess so the block precedes any import.-"""--import subprocess # noqa: S404-import sys-import textwrap--_NO_TEMPORALIO_SCRIPT = textwrap.dedent(- """- import sys- import importlib.abc--- class _Blocker(importlib.abc.MetaPathFinder):- def find_spec(self, name, path, target=None):- if name == "temporalio" or name.startswith("temporalio."):- raise ModuleNotFoundError(f"No module named {name!r} (blocked by test)")- return None--- sys.meta_path.insert(0, _Blocker())-- # Sanity: the block is actually in force.- try:- import temporalio # noqa: F401-- raise AssertionError("temporalio import was NOT blocked")- except ModuleNotFoundError:- pass-- from datetime import timedelta-- from pipelex.temporal.config_temporal import DispatchOptions-- # Construct + read back through to_execute_kwargs. retry_policy is Any at- # runtime, so an arbitrary sentinel stands in for the temporalio RetryPolicy.- dispatch = DispatchOptions(- task_queue=None,- start_to_close_timeout=timedelta(seconds=5),- retry_policy=object(),- heartbeat_timeout=timedelta(seconds=2),- )- kwargs = dispatch.to_execute_kwargs()- assert "task_queue" not in kwargs, kwargs- assert kwargs["start_to_close_timeout"] == timedelta(seconds=5), kwargs- assert kwargs["heartbeat_timeout"] == timedelta(seconds=2), kwargs- assert "retry_policy" in kwargs, kwargs-- print("NO_TEMPORALIO_OK")- """-)---class TestDispatchOptionsNoTemporalio:- def test_import_and_construct_without_temporalio(self) -> None:- """Fresh interpreter with ``temporalio`` blocked: importing- ``config_temporal`` and constructing a ``DispatchOptions`` must both- succeed (the load-bearing optional-dependency invariant).- """- result = subprocess.run( # noqa: S603- [sys.executable, "-c", _NO_TEMPORALIO_SCRIPT],- capture_output=True,- text=True,- check=False,- )- assert result.returncode == 0, (- f"DispatchOptions could not be imported/constructed without temporalio.\nstdout:\n{result.stdout}\nstderr:\n{result.stderr}"- )- assert "NO_TEMPORALIO_OK" in result.stdout, result.stdoutdiff --git a/tests/unit/pipelex/temporal/test_dispatch_resolution_tracing.py b/tests/unit/pipelex/temporal/test_dispatch_resolution_tracing.pydeleted file mode 100644index 9812f8673..000000000--- a/tests/unit/pipelex/temporal/test_dispatch_resolution_tracing.py+++ /dev/null@@ -1,150 +0,0 @@-"""Unit tests for the ``is_traced`` argument to ``WorkerConfig.resolve_dispatch``.--Tracing emits one log line per dispatch with the resolved queue +-start_to_close_timeout + retry attempts and the layer each scalar came from-(``baseline`` / ``queue_options`` / ``handle_options``). Off by default.-"""--import logging-from datetime import timedelta--import pytest--from pipelex.temporal.config_temporal import (- ActivityRouteConfig,- HandleOptions,- QueueOptions,- RetryPolicyConfig,- WorkerConfig,-)---def _make_worker_config(activity_queues: dict[str, ActivityRouteConfig] | None = None) -> WorkerConfig:- return WorkerConfig(- default_task_queue="default_q",- activity_queues=activity_queues or {},- workflow_execution_timeout=timedelta(hours=1),- default_activity_start_to_close_timeout=timedelta(minutes=10),- retry_policy_config=RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=3,- non_retryable_error_types=[],- ),- )---class TestDispatchResolutionTracing:- """``is_traced=True`` emits a structured log line with from= layer info."""-- def test_off_emits_no_log(self, caplog: pytest.LogCaptureFixture) -> None:- """With ``is_traced=False`` (default), no trace log line is emitted."""- worker_config = _make_worker_config()- with caplog.at_level(logging.INFO):- worker_config.resolve_dispatch(activity_name="act_llm_gen_text", routing_key="claude-opus-4-7")- trace_records = [record for record in caplog.records if "temporal.dispatch" in record.message]- assert not trace_records, f"trace log fired when is_traced=False: {[r.message for r in trace_records]!r}"-- def test_baseline_source_when_no_overrides(self, caplog: pytest.LogCaptureFixture) -> None:- """No queue_options / no handle_options → ``from=baseline`` on every scalar.-- With an empty ``activity_queues`` (no routing configured), the hybrid- fallback resolves the queue to ``None`` — the trace surfaces that- explicitly as ``queue=None`` so operators can see when dispatch will- ride on the workflow's own queue.- """- worker_config = _make_worker_config()- with caplog.at_level(logging.INFO):- worker_config.resolve_dispatch(activity_name="act_llm_gen_text", routing_key="claude-opus-4-7", is_traced=True)- trace_records = [record for record in caplog.records if "temporal.dispatch" in record.message]- assert len(trace_records) == 1, f"expected exactly one trace line, got {len(trace_records)}"- line = trace_records[0].message- assert "act=act_llm_gen_text" in line- assert "handle=claude-opus-4-7" in line- # Empty activity_queues → hybrid fallback returns None (workflow-local dispatch).- assert "queue=None" in line- # Baseline 10-minute timeout = 600s.- assert "timeout=600.0s" in line- assert "(from=baseline)" in line-- def test_queue_options_source(self, caplog: pytest.LogCaptureFixture) -> None:- """When ``queue_options[X].start_to_close_timeout`` overrides baseline,- the trace line reports ``from=queue_options`` for the timeout column.- """- worker_config = _make_worker_config(- activity_queues={"act_llm_gen_text": ActivityRouteConfig(default="anthropic_q", by_handle={})},- )- queue_options = {"anthropic_q": QueueOptions(start_to_close_timeout=timedelta(minutes=5))}- with caplog.at_level(logging.INFO):- worker_config.resolve_dispatch(- activity_name="act_llm_gen_text",- routing_key="some-handle",- queue_options_by_queue=queue_options,- is_traced=True,- )- trace_records = [record for record in caplog.records if "temporal.dispatch" in record.message]- assert len(trace_records) == 1- line = trace_records[0].message- assert "queue=anthropic_q" in line- assert "timeout=300.0s (from=queue_options)" in line-- def test_heartbeat_omitted_from_trace_when_unset(self, caplog: pytest.LogCaptureFixture) -> None:- """When no heartbeat is configured anywhere (baseline None, no overlay),- the trace line must not include a ``heartbeat=...`` segment.- """- worker_config = _make_worker_config()- with caplog.at_level(logging.INFO):- worker_config.resolve_dispatch(- activity_name="act_llm_gen_text",- routing_key="claude-opus-4-7",- is_traced=True,- )- trace_records = [record for record in caplog.records if "temporal.dispatch" in record.message]- assert len(trace_records) == 1- assert "heartbeat" not in trace_records[0].message-- def test_heartbeat_source_in_trace_from_queue_options(self, caplog: pytest.LogCaptureFixture) -> None:- """When ``queue_options[X].heartbeat_timeout`` is set, the trace reports- ``heartbeat=<s>s (from=queue_options)``.- """- worker_config = _make_worker_config(- activity_queues={"act_llm_gen_text": ActivityRouteConfig(default="anthropic_q", by_handle={})},- )- queue_options = {"anthropic_q": QueueOptions(heartbeat_timeout=timedelta(seconds=30))}- with caplog.at_level(logging.INFO):- worker_config.resolve_dispatch(- activity_name="act_llm_gen_text",- routing_key="some-handle",- queue_options_by_queue=queue_options,- is_traced=True,- )- trace_records = [record for record in caplog.records if "temporal.dispatch" in record.message]- assert len(trace_records) == 1- assert "heartbeat=30.0s (from=queue_options)" in trace_records[0].message-- def test_handle_options_source_wins(self, caplog: pytest.LogCaptureFixture) -> None:- """When ``handle_options.start_to_close_timeout`` is set, the trace- reports ``from=handle_options`` even if ``queue_options`` also had one.- """- worker_config = _make_worker_config(- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(- default="anthropic_q",- by_handle={},- handle_options={"claude-opus-4-7-1m": HandleOptions(start_to_close_timeout=timedelta(minutes=25))},- ),- },- )- queue_options = {"anthropic_q": QueueOptions(start_to_close_timeout=timedelta(minutes=5))}- with caplog.at_level(logging.INFO):- worker_config.resolve_dispatch(- activity_name="act_llm_gen_text",- routing_key="claude-opus-4-7-1m",- queue_options_by_queue=queue_options,- is_traced=True,- )- trace_records = [record for record in caplog.records if "temporal.dispatch" in record.message]- assert len(trace_records) == 1- line = trace_records[0].message- assert "timeout=1500.0s (from=handle_options)" in linediff --git a/tests/unit/pipelex/temporal/test_dry_validate_dispatch.py b/tests/unit/pipelex/temporal/test_dry_validate_dispatch.pydeleted file mode 100644index 75f94adaf..000000000--- a/tests/unit/pipelex/temporal/test_dry_validate_dispatch.py+++ /dev/null@@ -1,42 +0,0 @@-"""Unit tests for ``dispatch_dry_validate`` — the submitter-side round-trip the API route calls.--The helper owns two behavior decisions nothing else pins:--- D-C5 at the workflow tier: ``RetryPolicy(maximum_attempts=1)`` so a deterministically-failed- validation workflow is never re-run by the server (the config default policy would).-- The interactive execution bound: an explicit ``workflow_execution_timeout`` matching the- activity retry budget instead of the batch-tuned 1-hour config default, so a worker outage- fails the HTTP caller fast instead of queueing stale validation work.-"""--from datetime import timedelta--import pytest-from pytest_mock import MockerFixture--from pipelex.temporal.tprl.workflow_caller import WorkflowExecutorFactory-from pipelex.temporal.tprl_pipe.act_dry_validate import DryValidateArg, DryValidateResult-from pipelex.temporal.tprl_pipe.dry_validate_dispatch import dispatch_dry_validate-from pipelex.temporal.tprl_pipe.wf_dry_validate import WfDryValidate---class TestDryValidateDispatch:- @pytest.mark.asyncio- async def test_dispatch_pins_no_workflow_retry_and_interactive_timeout(self, mocker: MockerFixture) -> None:- expected_result = DryValidateResult(dry_run_outputs={}, pending_signatures=[], pipe_io_contracts={})- fake_executor = mocker.Mock()- fake_executor.make_workflow_id.return_value = "wf_dry_validate_test"- fake_executor.execute_workflow = mocker.AsyncMock(return_value=expected_result)- create_executor = mocker.patch.object(WorkflowExecutorFactory, "create_executor", return_value=fake_executor)-- arg = DryValidateArg(mthds_contents=['domain = "unit_dispatch"'])- result = await dispatch_dry_validate(arg)-- assert result is expected_result- create_kwargs = create_executor.call_args.kwargs- assert create_kwargs["retry_policy"].maximum_attempts == 1- assert create_kwargs["workflow_execution_timeout"] == timedelta(minutes=12)- execute_kwargs = fake_executor.execute_workflow.call_args.kwargs- assert execute_kwargs["workflow_class"] is WfDryValidate- assert execute_kwargs["workflow_arg"] is arg- assert execute_kwargs["workflow_id"] == "wf_dry_validate_test"diff --git a/tests/unit/pipelex/temporal/test_ensure_search_attributes_registered.py b/tests/unit/pipelex/temporal/test_ensure_search_attributes_registered.pydeleted file mode 100644index a57df27a0..000000000--- a/tests/unit/pipelex/temporal/test_ensure_search_attributes_registered.py+++ /dev/null@@ -1,161 +0,0 @@-"""Unit tests for ``ensure_required_search_attributes_registered``.--The helper is used both by the test conftest (in-process server bootstrap) and-by the ``pipelex setup-temporal-namespace`` CLI command. It MUST:--- Return the tuple of newly-registered attribute names on success.-- Return an empty tuple on the idempotent "everything already registered" no-op- so callers can report the actual delta instead of the configured-set size.-- Return ``RegistrationFailure`` on ``RPCError(PERMISSION_DENIED)`` from EITHER- RPC (``ListSearchAttributes`` or ``AddSearchAttributes``) so the CLI can- print the fallback runbook without leaking a "could not reach" message that- misdirects the operator.-- Propagate any other ``RPCError`` so the CLI can frame transient cluster- errors distinctly.-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture-from temporalio.service import RPCError, RPCStatusCode--from pipelex.temporal.tprl.namespace_check import RegistrationFailure, ensure_required_search_attributes_registered---def _make_client_stub(- mocker: MockerFixture,- *,- list_response: dict[str, Any] | Exception,- add_side_effect: Exception | None = None,-) -> Any:- """Build a fake Temporal client whose ``operator_service`` calls behave as configured."""- client = mocker.MagicMock()- if isinstance(list_response, Exception):- client.operator_service.list_search_attributes = mocker.AsyncMock(side_effect=list_response)- else:- response = mocker.MagicMock()- response.custom_attributes = list_response- client.operator_service.list_search_attributes = mocker.AsyncMock(return_value=response)- if add_side_effect is not None:- client.operator_service.add_search_attributes = mocker.AsyncMock(side_effect=add_side_effect)- else:- client.operator_service.add_search_attributes = mocker.AsyncMock()- return client--[email protected](loop_scope="class")-class TestEnsureSearchAttributesRegistered:- async def test_returns_newly_registered_tuple_when_some_missing(self, mocker: MockerFixture) -> None:- # Two of the five are already there; three need to be added.- client = _make_client_stub(- mocker,- list_response={"PipeCode": object(), "PipelineRunId": object()},- )-- outcome = await ensure_required_search_attributes_registered(- temporal_client=client,- namespace="default",- configured_attributes=["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"],- )-- assert outcome == ("SessionId", "UserId", "DomainCode")- client.operator_service.add_search_attributes.assert_awaited_once()-- async def test_returns_empty_tuple_when_everything_already_present(self, mocker: MockerFixture) -> None:- """Idempotent no-op: the CLI must be able to tell this apart from a- real registration so operators don't see a misleading "Registered 5".- """- client = _make_client_stub(- mocker,- list_response={name: object() for name in ("PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode")},- )-- outcome = await ensure_required_search_attributes_registered(- temporal_client=client,- namespace="default",- configured_attributes=["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"],- )-- assert outcome == ()- client.operator_service.add_search_attributes.assert_not_called()-- async def test_returns_empty_tuple_when_configured_set_is_empty(self, mocker: MockerFixture) -> None:- """When ``[temporal.search_attributes].enabled = false`` is wired- through, callers pass an empty configured subset. The helper must- short-circuit before touching the operator service.- """- client = _make_client_stub(mocker, list_response={})-- outcome = await ensure_required_search_attributes_registered(- temporal_client=client,- namespace="default",- configured_attributes=[],- )-- assert outcome == ()- client.operator_service.list_search_attributes.assert_not_called()- client.operator_service.add_search_attributes.assert_not_called()-- async def test_permission_denied_on_list_returns_registration_failure(self, mocker: MockerFixture) -> None:- """If the API key cannot ``ListSearchAttributes`` the helper must NOT- leak a raw ``RPCError`` (which the CLI frames as "could not reach"- — wrong direction); it must return a ``RegistrationFailure`` with- the full configured set so the operator's fallback runbook contains- the right list. Regression for the bug where only the ADD call had- PERMISSION_DENIED handling.- """- list_exc = RPCError("permission denied", RPCStatusCode.PERMISSION_DENIED, raw_grpc_status=b"")- client = _make_client_stub(mocker, list_response=list_exc)-- outcome = await ensure_required_search_attributes_registered(- temporal_client=client,- namespace="ns1",- configured_attributes=["PipeCode", "PipelineRunId", "SessionId"],- )-- assert isinstance(outcome, RegistrationFailure)- assert outcome.namespace == "ns1"- # Without read access we cannot compute the actual missing set; surface- # the whole configured list so the operator's runbook contains everything.- assert outcome.missing == ("PipeCode", "PipelineRunId", "SessionId")- assert "permission denied" in outcome.rpc_error_message- # The ADD RPC must never be attempted in this branch.- client.operator_service.add_search_attributes.assert_not_called()-- async def test_permission_denied_on_add_returns_registration_failure(self, mocker: MockerFixture) -> None:- add_exc = RPCError("permission denied", RPCStatusCode.PERMISSION_DENIED, raw_grpc_status=b"")- client = _make_client_stub(- mocker,- list_response={"PipeCode": object()},- add_side_effect=add_exc,- )-- outcome = await ensure_required_search_attributes_registered(- temporal_client=client,- namespace="ns1",- configured_attributes=["PipeCode", "PipelineRunId", "SessionId"],- )-- assert isinstance(outcome, RegistrationFailure)- assert outcome.namespace == "ns1"- # Only the actually-missing names are surfaced — read access succeeded.- assert outcome.missing == ("PipelineRunId", "SessionId")- assert "permission denied" in outcome.rpc_error_message-- async def test_non_permission_denied_rpc_error_on_list_propagates(self, mocker: MockerFixture) -> None:- """``UNAVAILABLE`` / ``NOT_FOUND`` etc. must propagate so the CLI can- frame them as connectivity / configuration issues rather than as- permission failures.- """- list_exc = RPCError("namespace not reachable", RPCStatusCode.UNAVAILABLE, raw_grpc_status=b"")- client = _make_client_stub(mocker, list_response=list_exc)-- with pytest.raises(RPCError) as exc_info:- await ensure_required_search_attributes_registered(- temporal_client=client,- namespace="ns1",- configured_attributes=["PipeCode"],- )-- assert exc_info.value.status == RPCStatusCode.UNAVAILABLEdiff --git a/tests/unit/pipelex/temporal/test_full_catalog_collisions.py b/tests/unit/pipelex/temporal/test_full_catalog_collisions.pydeleted file mode 100644index ad55c7273..000000000--- a/tests/unit/pipelex/temporal/test_full_catalog_collisions.py+++ /dev/null@@ -1,128 +0,0 @@-from typing import Any--import pytest--from pipelex.temporal.exceptions import TemporalConfigError-from pipelex.temporal.temporal_tasks import TaskPack, TemporalTasks-from pipelex.temporal.temporal_types import ActivityType, WorkflowType---def _make_workflow(name: str) -> WorkflowType:- return type(name, (), {})---def _make_activity(name: str) -> ActivityType:- def _impl(payload: Any) -> Any:- return payload-- _impl.__name__ = name- return _impl---class TestFullCatalogCollisions:- def test_same_workflow_callable_in_multiple_packs_dedups(self):- workflow = _make_workflow("WfShared")- activity = _make_activity("act_shared")- tasks = TemporalTasks()- tasks.complement_catalog(- extra_catalog={- "pack_one": TaskPack(workflow_list=[workflow], activity_list=[activity]),- "pack_two": TaskPack(workflow_list=[workflow], activity_list=[activity]),- },- extra_workflows=[],- extra_activities=[],- )-- workflows, activities = tasks.workflows_and_activities()-- assert workflows == [workflow]- assert activities == [activity]-- def test_same_callable_in_pack_and_extras_dedups(self):- workflow = _make_workflow("WfShared")- activity = _make_activity("act_shared")- tasks = TemporalTasks()- tasks.complement_catalog(- extra_catalog={- "pack_one": TaskPack(workflow_list=[workflow], activity_list=[activity]),- },- extra_workflows=[workflow],- extra_activities=[activity],- )-- workflows, activities = tasks.workflows_and_activities()-- assert workflows == [workflow]- assert activities == [activity]-- def test_distinct_workflows_with_same_name_across_packs_raises(self):- workflow_one = _make_workflow("WfClash")- workflow_two = _make_workflow("WfClash")- assert workflow_one is not workflow_two-- tasks = TemporalTasks()- tasks.complement_catalog(- extra_catalog={- "pack_one": TaskPack(workflow_list=[workflow_one], activity_list=[]),- "pack_two": TaskPack(workflow_list=[workflow_two], activity_list=[]),- },- extra_workflows=[],- extra_activities=[],- )-- with pytest.raises(TemporalConfigError, match="Workflow name collision: 'WfClash'") as exc_info:- tasks.workflows_and_activities()- assert "task pack 'pack_two'" in str(exc_info.value)-- def test_distinct_workflows_with_same_name_pack_and_extras_raises(self):- workflow_in_pack = _make_workflow("WfClash")- workflow_in_extras = _make_workflow("WfClash")-- tasks = TemporalTasks()- tasks.complement_catalog(- extra_catalog={- "pack_one": TaskPack(workflow_list=[workflow_in_pack], activity_list=[]),- },- extra_workflows=[workflow_in_extras],- extra_activities=[],- )-- with pytest.raises(TemporalConfigError, match="Workflow name collision: 'WfClash'") as exc_info:- tasks.workflows_and_activities()- assert "extra_workflows" in str(exc_info.value)-- def test_distinct_activities_with_same_name_across_packs_raises(self):- activity_one = _make_activity("act_clash")- activity_two = _make_activity("act_clash")- assert activity_one is not activity_two-- tasks = TemporalTasks()- tasks.complement_catalog(- extra_catalog={- "pack_one": TaskPack(workflow_list=[], activity_list=[activity_one]),- "pack_two": TaskPack(workflow_list=[], activity_list=[activity_two]),- },- extra_workflows=[],- extra_activities=[],- )-- with pytest.raises(TemporalConfigError, match="Activity name collision: 'act_clash'") as exc_info:- tasks.workflows_and_activities()- assert "task pack 'pack_two'" in str(exc_info.value)-- def test_distinct_activities_with_same_name_pack_and_extras_raises(self):- activity_in_pack = _make_activity("act_clash")- activity_in_extras = _make_activity("act_clash")-- tasks = TemporalTasks()- tasks.complement_catalog(- extra_catalog={- "pack_one": TaskPack(workflow_list=[], activity_list=[activity_in_pack]),- },- extra_workflows=[],- extra_activities=[activity_in_extras],- )-- with pytest.raises(TemporalConfigError, match="Activity name collision: 'act_clash'") as exc_info:- tasks.workflows_and_activities()- assert "extra_activities" in str(exc_info.value)diff --git a/tests/unit/pipelex/temporal/test_log_temporal_request_id.py b/tests/unit/pipelex/temporal/test_log_temporal_request_id.pydeleted file mode 100644index 3fd7a6bde..000000000--- a/tests/unit/pipelex/temporal/test_log_temporal_request_id.py+++ /dev/null@@ -1,60 +0,0 @@-"""Unit coverage for the bound ``request_id`` on the Temporal log helpers.--``WorkflowLog`` / ``ActivityLog`` are built once per workflow / activity-invocation, bound to that invocation's ``job_metadata.request_id``. The bound-id is packed into every log record's ``extra`` dict so downstream shippers read-``record.request_id``; an unbound helper emits no ``extra``.-"""--import logging-from typing import Any--import pytest-from pytest_mock import MockerFixture-from temporalio import activity, workflow--from pipelex.temporal.log_temporal import ActivityLog, WorkflowLog-from pipelex.tools.log.log_levels import LOGGING_LEVEL_DEV, LOGGING_LEVEL_VERBOSE--_LOG_CASES = [- pytest.param(WorkflowLog, workflow.logger, id="workflow"),- pytest.param(ActivityLog, activity.logger, id="activity"),-]--# Every severity method paired with the log level it must emit at.-_METHOD_CASES = [- pytest.param("verbose", LOGGING_LEVEL_VERBOSE, id="verbose"),- pytest.param("debug", logging.DEBUG, id="debug"),- pytest.param("dev", LOGGING_LEVEL_DEV, id="dev"),- pytest.param("info", logging.INFO, id="info"),- pytest.param("warning", logging.WARNING, id="warning"),- pytest.param("error", logging.ERROR, id="error"),- pytest.param("critical", logging.CRITICAL, id="critical"),-]---class TestLogTemporalRequestId:- @pytest.mark.parametrize(("log_class", "temporal_logger"), _LOG_CASES)- @pytest.mark.parametrize(("method_name", "expected_level"), _METHOD_CASES)- @pytest.mark.parametrize(- ("request_id", "expected_extra"),- [- pytest.param("r-abc-123", {"request_id": "r-abc-123"}, id="bound"),- pytest.param(None, None, id="unbound"),- ],- )- def test_request_id_binding_controls_log_extra(- self,- mocker: MockerFixture,- log_class: type[WorkflowLog | ActivityLog],- temporal_logger: Any,- method_name: str,- expected_level: int,- request_id: str | None,- expected_extra: dict[str, str] | None,- ) -> None:- """Every severity method packs the bound request_id into ``extra`` at its own level; an unbound helper emits none."""- mock_log = mocker.patch.object(temporal_logger, "log")- log_method = getattr(log_class(request_id=request_id), method_name)- log_method("hello")- mock_log.assert_called_once_with(level=expected_level, msg="hello", extra=expected_extra)diff --git a/tests/unit/pipelex/temporal/test_make_worker_uses_profile.py b/tests/unit/pipelex/temporal/test_make_worker_uses_profile.pydeleted file mode 100644index 1a39a4736..000000000--- a/tests/unit/pipelex/temporal/test_make_worker_uses_profile.py+++ /dev/null@@ -1,110 +0,0 @@-"""Unit test: ``TemporalTaskManager.make_worker`` plumbs every-``WorkerRuntimeProfile`` field into the underlying ``Worker(...)`` constructor,-and reads ``max_task_queue_activities_per_second`` from-``queue_options[task_queue]`` when present.--Mocks the ``Worker`` class so the test runs without a Temporal server and-focuses on the kwargs contract.-"""--from datetime import timedelta--from pytest_mock import MockerFixture--from pipelex.config import get_config-from pipelex.temporal.config_temporal import QueueOptions, WorkerRuntimeProfile, WorkerTuningMode-from pipelex.temporal.temporal_task_manager import TemporalTaskManager---def _make_profile() -> WorkerRuntimeProfile:- """Build a distinct-valued profile so every field can be uniquely asserted."""- return WorkerRuntimeProfile(- tuning_mode=WorkerTuningMode.EXPLICIT,- max_cached_workflows=42,- max_concurrent_workflow_tasks=43,- max_concurrent_activities=44,- max_concurrent_local_activities=45,- max_concurrent_workflow_task_polls=46,- max_concurrent_activity_task_polls=47,- max_activities_per_second=48,- sticky_queue_schedule_to_start_timeout=timedelta(minutes=10),- max_heartbeat_throttle_interval=timedelta(minutes=20),- default_heartbeat_throttle_interval=timedelta(minutes=30),- graceful_shutdown_timeout=timedelta(minutes=40),- )---class TestMakeWorkerUsesProfile:- """Profile fields and queue rate cap flow into ``Worker(...)`` kwargs."""-- def test_every_profile_field_flows_to_worker_kwarg(self, mocker: MockerFixture) -> None:- """Each ``WorkerRuntimeProfile`` field maps to the matching ``Worker`` kwarg."""- worker_mock = mocker.patch("pipelex.temporal.temporal_task_manager.Worker")- profile = _make_profile()-- TemporalTaskManager().make_worker(- temporal_client=mocker.MagicMock(),- task_queue="test_q",- is_not_sandboxed=True,- runtime_profile=profile,- )-- worker_mock.assert_called_once()- kwargs = worker_mock.call_args.kwargs- assert kwargs["task_queue"] == "test_q"- assert kwargs["max_cached_workflows"] == 42- assert kwargs["max_concurrent_workflow_tasks"] == 43- assert kwargs["max_concurrent_activities"] == 44- assert kwargs["max_concurrent_local_activities"] == 45- assert kwargs["max_concurrent_workflow_task_polls"] == 46- assert kwargs["max_concurrent_activity_task_polls"] == 47- assert kwargs["max_activities_per_second"] == 48- assert kwargs["sticky_queue_schedule_to_start_timeout"] == timedelta(minutes=10)- assert kwargs["max_heartbeat_throttle_interval"] == timedelta(minutes=20)- assert kwargs["default_heartbeat_throttle_interval"] == timedelta(minutes=30)- assert kwargs["graceful_shutdown_timeout"] == timedelta(minutes=40)-- def test_max_task_queue_rate_cap_from_queue_options(self, mocker: MockerFixture) -> None:- """``max_task_queue_activities_per_second`` comes from- ``queue_options[task_queue]`` when present, ``None`` otherwise.-- Mutates ``get_config().temporal.queue_options`` for the duration of the- test; the ``finally`` restores prior state.- """- worker_mock = mocker.patch("pipelex.temporal.temporal_task_manager.Worker")- profile = _make_profile()- queue_options = get_config().temporal.queue_options- test_q = "test_rate_capped_q"- assert test_q not in queue_options, "test queue name should be unique"- queue_options[test_q] = QueueOptions(max_task_queue_activities_per_second=17.5)- try:- TemporalTaskManager().make_worker(- temporal_client=mocker.MagicMock(),- task_queue=test_q,- is_not_sandboxed=True,- runtime_profile=profile,- )- finally:- queue_options.pop(test_q, None)-- worker_mock.assert_called_once()- assert worker_mock.call_args.kwargs["max_task_queue_activities_per_second"] == 17.5-- def test_max_task_queue_rate_cap_omitted_when_no_queue_options(self, mocker: MockerFixture) -> None:- """No ``queue_options`` entry → ``max_task_queue_activities_per_second=None``- flows to ``Worker(...)`` (the Temporal SDK treats this as 'no cap').- """- worker_mock = mocker.patch("pipelex.temporal.temporal_task_manager.Worker")- profile = _make_profile()- unconfigured_q = "test_unconfigured_q"- assert unconfigured_q not in get_config().temporal.queue_options-- TemporalTaskManager().make_worker(- temporal_client=mocker.MagicMock(),- task_queue=unconfigured_q,- is_not_sandboxed=True,- runtime_profile=profile,- )-- worker_mock.assert_called_once()- assert worker_mock.call_args.kwargs["max_task_queue_activities_per_second"] is Nonediff --git a/tests/unit/pipelex/temporal/test_message_from_exc.py b/tests/unit/pipelex/temporal/test_message_from_exc.pydeleted file mode 100644index 40c513f17..000000000--- a/tests/unit/pipelex/temporal/test_message_from_exc.py+++ /dev/null@@ -1,27 +0,0 @@-"""Unit tests for ``_message_from_exc`` — deepest-message extraction from a Temporal failure chain.--Covers the two robustness paths that have no other coverage: the ``id()``-based-cycle guard on a self-referential ``__cause__`` chain, and the ``repr(exc)``-fallback when every message in the chain is empty.-"""--from pipelex.temporal.tprl.temporal_error import _message_from_exc # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]---class TestMessageFromExc:- def test_self_referential_cause_chain_terminates(self) -> None:- """A cyclic ``__cause__`` chain must not loop forever — the ``id()`` guard stops the walk- once a node is revisited, after surfacing the deepest message reached.- """- first = RuntimeError("first failure")- second = RuntimeError("second failure")- first.__cause__ = second- second.__cause__ = first- assert _message_from_exc(first) == "second failure"-- def test_repr_fallback_when_every_message_empty(self) -> None:- """When every exception in the chain has an empty message, fall back to ``repr(exc)``."""- outer = RuntimeError("")- inner = ValueError("")- outer.__cause__ = inner- assert _message_from_exc(outer) == repr(outer)diff --git a/tests/unit/pipelex/temporal/test_non_retryable_baseline_pins.py b/tests/unit/pipelex/temporal/test_non_retryable_baseline_pins.pydeleted file mode 100644index f9e39c7ea..000000000--- a/tests/unit/pipelex/temporal/test_non_retryable_baseline_pins.py+++ /dev/null@@ -1,24 +0,0 @@-"""Pin the deterministic dry-run error classes into the baseline ``non_retryable_error_types``.--The terminality contract of the dry-run mock errors rests on a *string* linkage: the class name-literal in ``pipelex/pipelex.toml`` must match ``cls.__name__``. A rename on either side silently-makes a deterministic failure retryable again — the retry-forever / wasted-retry family this-project has already been bitten by — with the whole suite staying green. This test makes that-drift loud.-"""--from pathlib import Path--import tomli--from pipelex.cogt.content_generation.exceptions import DryRunMockBuildError, DryRunObjectFidelityError---class TestNonRetryableBaselinePins:- def test_deterministic_dry_errors_are_pinned_non_retryable(self) -> None:- toml_path = Path(__file__).parents[4] / "pipelex" / "pipelex.toml"- config = tomli.loads(toml_path.read_text(encoding="utf-8"))- baseline = config["temporal"]["worker_config"]["retry_policy_config"]["non_retryable_error_types"]-- assert DryRunMockBuildError.__name__ in baseline- assert DryRunObjectFidelityError.__name__ in baselinediff --git a/tests/unit/pipelex/temporal/test_observability_helpers.py b/tests/unit/pipelex/temporal/test_observability_helpers.pydeleted file mode 100644index bf93c39df..000000000--- a/tests/unit/pipelex/temporal/test_observability_helpers.py+++ /dev/null@@ -1,335 +0,0 @@-"""Unit tests for the observability helpers in ``pipelex.temporal.tprl.observability``.--The helpers build the Temporal observability surface (search attributes,-static summary/details, per-activity summary) from Pipelex identity. The-formatting policy is documented in-``wip/temporal-primitives/id-and-naming-design.md`` §"The four layers".-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture--from pipelex.pipeline.job_metadata import JobMetadata-from pipelex.temporal.tprl.observability import (- DOMAIN_CODE_KEY,- PIPE_CODE_KEY,- PIPELINE_RUN_ID_KEY,- SESSION_ID_KEY,- USER_ID_KEY,- build_activity_summary,- build_search_attributes,- build_static_details,- build_static_summary,- stamp_submitter_session_id,-)--_STUB_SESSION_ID = "EdgdJ7Yk4Q3HF2pXyZv9w8"---def _make_pipe_job_stub(- mocker: MockerFixture,- *,- pipe_code: str = "translate_doc",- domain_code: str = "documents",- description: str = "Translate a document from English to French",- input_keys: list[str] | None = None,- user_id: str = "acme-corp",- pipeline_run_id: str = "3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c",- session_id: str | None = _STUB_SESSION_ID,- library_crate_fingerprint: str | None = None,-) -> Any:- """Build a stub PipeJob with only the attributes the helpers read.-- ``session_id`` defaults to the canonical stub value because the helpers- now read it directly off ``pipe_job.job_metadata`` instead of touching- the worker-local ``TemporalManager``. Pass ``None`` to exercise the- "session_id not stamped" fallback path.- """- pipe_job = mocker.MagicMock()- pipe_job.pipe.code = pipe_code- pipe_job.pipe.domain_code = domain_code- pipe_job.pipe.description = description- pipe_job.pipe.inputs.root = {key: object() for key in (input_keys or [])}- pipe_job.job_metadata.user_id = user_id- pipe_job.job_metadata.pipeline_run_id = pipeline_run_id- pipe_job.job_metadata.session_id = session_id- if library_crate_fingerprint is None:- pipe_job.library_crate = None- else:- pipe_job.library_crate.fingerprint = library_crate_fingerprint- return pipe_job---def patch_temporal_manager(mocker: MockerFixture) -> None:- """Patch ``get_temporal_manager`` so the ``stamp_submitter_session_id``- helper can capture a value without needing a real ``TemporalManager``- singleton. The observability helpers themselves no longer read it.- """- manager = mocker.MagicMock()- manager.session_id = _STUB_SESSION_ID- mocker.patch("pipelex.temporal.tprl.observability.get_temporal_manager", return_value=manager)---def patch_search_attributes_config_all_enabled(mocker: MockerFixture) -> None:- """Patch ``get_config`` so ``build_search_attributes`` sees the default- "all five enabled" surface without booting Pipelex.- """- config_root = mocker.MagicMock()- config_root.temporal.search_attributes.enabled = True- config_root.temporal.search_attributes.attributes = ["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"]- mocker.patch("pipelex.temporal.tprl.observability.get_config", return_value=config_root)--[email protected]("patch_temporal_manager", "patch_search_attributes_config_all_enabled")-class TestObservabilityHelpers:- def test_build_search_attributes_returns_five_typed_keys(self, mocker: MockerFixture) -> None:- pipe_job = _make_pipe_job_stub(mocker)-- attrs = build_search_attributes(pipe_job)-- assert attrs[PIPE_CODE_KEY] == "translate_doc"- assert attrs[PIPELINE_RUN_ID_KEY] == "3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c"- assert attrs[SESSION_ID_KEY] == "EdgdJ7Yk4Q3HF2pXyZv9w8"- assert attrs[USER_ID_KEY] == "acme-corp"- assert attrs[DOMAIN_CODE_KEY] == "documents"- assert len(attrs) == 5-- def test_build_search_attributes_returns_empty_when_disabled(self, mocker: MockerFixture) -> None:- """``enabled = false`` short-circuits to an empty ``TypedSearchAttributes``- so the dashboard view degrades cleanly to WorkflowType / WorkflowId /- StartTime without rejecting workflow starts.- """- config_root = mocker.MagicMock()- config_root.temporal.search_attributes.enabled = False- config_root.temporal.search_attributes.attributes = []- mocker.patch("pipelex.temporal.tprl.observability.get_config", return_value=config_root)- pipe_job = _make_pipe_job_stub(mocker)-- attrs = build_search_attributes(pipe_job)-- assert len(attrs) == 0-- def test_build_search_attributes_reads_session_id_from_pipe_job_not_manager(self, mocker: MockerFixture) -> None:- """Determinism regression: ``build_search_attributes`` must be a pure- function of ``pipe_job``. Reading ``TemporalManager.session_id`` from- inside workflow code produced non-deterministic ``StartChildWorkflowExecution``- commands on replay across worker restarts. Two ``pipe_job`` stubs with- the same stamped ``job_metadata.session_id`` but different fake- managers must yield byte-equal ``SessionId`` pairs.- """- manager_alpha = mocker.MagicMock()- manager_alpha.session_id = "this-must-not-leak-into-attrs-A"- mocker.patch("pipelex.temporal.tprl.observability.get_temporal_manager", return_value=manager_alpha)- pipe_job = _make_pipe_job_stub(mocker, session_id="stamped-at-submitter")-- attrs_first = build_search_attributes(pipe_job)-- manager_beta = mocker.MagicMock()- manager_beta.session_id = "this-must-not-leak-into-attrs-B"- mocker.patch("pipelex.temporal.tprl.observability.get_temporal_manager", return_value=manager_beta)-- attrs_second = build_search_attributes(pipe_job)-- assert attrs_first[SESSION_ID_KEY] == "stamped-at-submitter"- assert attrs_second[SESSION_ID_KEY] == "stamped-at-submitter"-- def test_build_search_attributes_filters_to_configured_subset(self, mocker: MockerFixture) -> None:- """When ``attributes`` lists only a subset of the five built-ins, only- those pairs end up on the workflow start.- """- config_root = mocker.MagicMock()- config_root.temporal.search_attributes.enabled = True- config_root.temporal.search_attributes.attributes = ["PipeCode", "DomainCode"]- mocker.patch("pipelex.temporal.tprl.observability.get_config", return_value=config_root)- pipe_job = _make_pipe_job_stub(mocker)-- attrs = build_search_attributes(pipe_job)-- assert len(attrs) == 2- assert attrs[PIPE_CODE_KEY] == "translate_doc"- assert attrs[DOMAIN_CODE_KEY] == "documents"-- def test_build_static_summary_with_description(self, mocker: MockerFixture) -> None:- pipe_job = _make_pipe_job_stub(mocker)-- summary = build_static_summary(pipe_job.pipe)-- assert summary == "translate_doc — Translate a document from English to French"-- def test_build_static_summary_with_empty_description_omits_dash_tail(self, mocker: MockerFixture) -> None:- pipe_job = _make_pipe_job_stub(mocker, description="")-- summary = build_static_summary(pipe_job.pipe)-- assert summary == "translate_doc"-- def test_build_static_summary_truncates_at_200_bytes_with_ellipsis(self, mocker: MockerFixture) -> None:- long_description = "x" * 300- pipe_job = _make_pipe_job_stub(mocker, description=long_description)-- summary = build_static_summary(pipe_job.pipe)-- assert len(summary.encode("utf-8")) <= 200- assert summary.endswith("…")-- def test_build_static_summary_truncates_utf8_safely_at_multibyte_boundary(self, mocker: MockerFixture) -> None:- # Use 3-byte UTF-8 characters (CJK) to force a partial sequence near the cut.- long_description = "漢" * 100- pipe_job = _make_pipe_job_stub(mocker, description=long_description)-- summary = build_static_summary(pipe_job.pipe)-- # Decoding the result must not raise UnicodeDecodeError — i.e. the- # truncation correctly drops any partial multi-byte sequence at the cut.- summary.encode("utf-8").decode("utf-8")- assert summary.endswith("…")- assert len(summary.encode("utf-8")) <= 200-- def test_build_static_details_includes_all_required_rows(self, mocker: MockerFixture) -> None:- pipe_job = _make_pipe_job_stub(- mocker,- input_keys=["source_text", "target_language"],- library_crate_fingerprint="abcdef0123456789deadbeefcafebabe1234567890",- )-- details = build_static_details(pipe_job)-- assert "| Pipe | `translate_doc` |" in details- assert "| Domain | `documents` |" in details- assert "| Pipeline run | `3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c` |" in details- assert "| User | `acme-corp` |" in details- assert "| Session | `EdgdJ7Yk4Q3HF2pXyZv9w8` |" in details- # Library crate fingerprint is truncated to 12 chars.- assert "| Library crate | `abcdef012345` |" in details- assert "| Input | `source_text`, `target_language` |" in details-- def test_build_static_details_omits_optional_rows_when_unavailable(self, mocker: MockerFixture) -> None:- pipe_job = _make_pipe_job_stub(mocker, input_keys=[])-- details = build_static_details(pipe_job)-- assert "Library crate" not in details- assert "Input" not in details- # Core identity rows still present.- assert "| Pipe | `translate_doc` |" in details- assert "| Pipeline run |" in details-- def test_build_static_details_omits_library_crate_when_fingerprint_is_empty(self, mocker: MockerFixture) -> None:- """A LibraryCrate whose fingerprint has not been computed must not produce a row."""- pipe_job = _make_pipe_job_stub(mocker, library_crate_fingerprint="")-- details = build_static_details(pipe_job)-- assert "Library crate" not in details-- def test_stamp_submitter_session_id_sets_when_missing(self, mocker: MockerFixture) -> None:- """At top-level dispatch, the helper must read ``TemporalManager.session_id``- once and write it onto ``pipe_job.job_metadata`` so the value flows into- child workflows via the workflow input.- """- manager = mocker.MagicMock()- manager.session_id = "submitter-session-xyz"- mocker.patch("pipelex.temporal.tprl.observability.get_temporal_manager", return_value=manager)-- pipe_job = _make_pipe_job_stub(mocker, session_id=None)- # Configure the model_copy chain so the assertion sees the stamped value.- updated_metadata = mocker.MagicMock()- updated_metadata.session_id = "submitter-session-xyz"- pipe_job.job_metadata.model_copy.return_value = updated_metadata- stamped_pipe_job = mocker.MagicMock()- stamped_pipe_job.job_metadata = updated_metadata- pipe_job.model_copy.return_value = stamped_pipe_job-- stamped = stamp_submitter_session_id(pipe_job)-- assert stamped.job_metadata.session_id == "submitter-session-xyz"- # The helper must call model_copy with the right update payload.- pipe_job.job_metadata.model_copy.assert_called_once_with(update={"session_id": "submitter-session-xyz"})- pipe_job.model_copy.assert_called_once_with(update={"job_metadata": updated_metadata})-- def test_stamp_submitter_session_id_is_idempotent(self, mocker: MockerFixture) -> None:- """When ``session_id`` is already set on the incoming ``pipe_job``,- the helper must return the same object untouched — child workflows- inherit the parent's session_id and must not have it overwritten by- a worker-local fallback.- """- manager = mocker.MagicMock()- manager.session_id = "worker-local-value-should-be-ignored"- mocker.patch("pipelex.temporal.tprl.observability.get_temporal_manager", return_value=manager)- pipe_job = _make_pipe_job_stub(mocker, session_id="parent-session-preserved")-- stamped = stamp_submitter_session_id(pipe_job)-- assert stamped is pipe_job- assert stamped.job_metadata.session_id == "parent-session-preserved"- pipe_job.job_metadata.model_copy.assert_not_called()- pipe_job.model_copy.assert_not_called()-- def test_build_search_attributes_falls_back_to_empty_string_when_session_unset(self, mocker: MockerFixture) -> None:- """Defensive fallback: if a caller bypasses ``stamp_submitter_session_id``- somehow, ``build_search_attributes`` must emit an empty string for- ``SessionId`` rather than ``None`` (Keyword attributes reject ``None``).- """- pipe_job = _make_pipe_job_stub(mocker, session_id=None)-- attrs = build_search_attributes(pipe_job)-- assert attrs[SESSION_ID_KEY] == ""-- def test_build_static_details_truncates_at_20kb(self, mocker: MockerFixture) -> None:- """Module docstring claims ``static_details`` are capped at 20 KB.- Push the input list past the cap and assert the renderer truncates- with an ellipsis instead of returning an oversize payload.- """- long_input_names = [f"input_field_name_{index:04d}" for index in range(2000)]- pipe_job = _make_pipe_job_stub(mocker, input_keys=long_input_names)-- details = build_static_details(pipe_job)-- assert len(details.encode("utf-8")) <= 20 * 1024- assert details.endswith("…")-- def test_build_activity_summary_includes_method_pipe_and_extras(self) -> None:- job_metadata = JobMetadata(user_id="u", pipeline_run_id="r", pipe_code="translate_doc")-- summary = build_activity_summary("LLM text", job_metadata=job_metadata, extras={"model": "gpt-4o"})-- assert summary == "LLM text · pipe=translate_doc · model=gpt-4o"-- def test_build_activity_summary_accepts_reserved_word_keys(self) -> None:- """``class`` is a Python reserved word — the design table specifies- ``class={class_name}`` verbatim, and the ``extras`` mapping (vs ``**kwargs``)- lets callers use it as a key without contortions.- """- job_metadata = JobMetadata(user_id="u", pipeline_run_id="r", pipe_code="extract_text")-- summary = build_activity_summary("LLM object", job_metadata=job_metadata, extras={"class": "Section"})-- assert summary == "LLM object · pipe=extract_text · class=Section"-- def test_build_activity_summary_omits_pipe_segment_when_pipe_code_is_unset(self) -> None:- job_metadata = JobMetadata(user_id="u", pipeline_run_id="r")-- summary = build_activity_summary("LLM text", job_metadata=job_metadata, extras={"model": "gpt-4o"})-- assert summary == "LLM text · model=gpt-4o"-- def test_build_activity_summary_omits_extras_section_when_none(self) -> None:- job_metadata = JobMetadata(user_id="u", pipeline_run_id="r", pipe_code="my_pipe")-- summary = build_activity_summary("Templated text", job_metadata=job_metadata)-- assert summary == "Templated text · pipe=my_pipe"-- def test_build_activity_summary_truncates_at_200_bytes(self) -> None:- job_metadata = JobMetadata(user_id="u", pipeline_run_id="r", pipe_code="p")- long_value = "x" * 500-- summary = build_activity_summary("LLM text", job_metadata=job_metadata, extras={"extra": long_value})-- assert len(summary.encode("utf-8")) <= 200- assert summary.endswith("…")diff --git a/tests/unit/pipelex/temporal/test_recover_error_report.py b/tests/unit/pipelex/temporal/test_recover_error_report.pydeleted file mode 100644index 8a272522d..000000000--- a/tests/unit/pipelex/temporal/test_recover_error_report.py+++ /dev/null@@ -1,137 +0,0 @@-"""Unit tests for ``recover_error_report`` — submitter-side report recovery.--When a Temporal workflow fails, the structured ``ErrorReport`` rides on the-deserialized ``ApplicationError.details``. ``recover_error_report`` walks the-``__cause__`` chain, finds that ``ApplicationError``, and rebuilds the report.-"""--from typing import Any--import pytest-from temporalio.client import WorkflowFailureError-from temporalio.exceptions import ApplicationError--from pipelex.base_exceptions import ErrorDomain, ErrorReport-from pipelex.cogt.inference.error_classification import UserAction, UserActionKind-from pipelex.temporal.tprl.temporal_error import (- _ERROR_REPORT_VALIDATION_FAILED_MARKER, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]- recover_error_report,-)--_FULL_REPORT = ErrorReport(- error_type="CogtError",- message="rate limited on the worker",- title="AI inference failed",- type_uri="https://docs.pipelex.com/latest/errors/cogt-error/",- error_category="capacity",- error_domain=ErrorDomain.RUNTIME,- retryable=False,- user_action=UserAction(kind=UserActionKind.CHECK_BILLING, detail="check your billing page"),- model="gpt-5",- provider="openai",-)---def _app_error(*details: Any) -> ApplicationError:- """Build an ``ApplicationError`` as the activity bridge would — report dict in ``details``."""- return ApplicationError("rate limited on the worker", *details, type="CogtError")---def _workflow_failure(cause: BaseException) -> WorkflowFailureError:- """Wrap ``cause`` in a ``WorkflowFailureError`` — what ``client.execute_workflow`` raises."""- return WorkflowFailureError(cause=cause)---class TestRecoverErrorReport:- def test_recovers_full_report_from_application_error_in_chain(self) -> None:- """The packed report is rebuilt with every classification field intact."""- failure = _workflow_failure(_app_error(_FULL_REPORT.to_dict()))- assert recover_error_report(failure) == _FULL_REPORT-- def test_g4_application_error_without_report_details_synthesizes_unrecoverable(self) -> None:- """G4 — an ``ApplicationError`` carrying no report payload synthesizes the unrecoverable report."""- failure = _workflow_failure(_app_error())- report = recover_error_report(failure)- assert report.error_type == "UnrecoverableWorkflowFailureError"- assert report.error_domain == ErrorDomain.RUNTIME- assert "rate limited on the worker" in report.message-- def test_no_application_error_in_chain_synthesizes_unrecoverable(self) -> None:- """A failure with no ``ApplicationError`` in its ``__cause__`` chain synthesizes the unrecoverable report."""- failure = _workflow_failure(RuntimeError("plain non-Temporal failure"))- report = recover_error_report(failure)- assert report.error_type == "UnrecoverableWorkflowFailureError"- assert report.error_domain == ErrorDomain.RUNTIME- assert "plain non-Temporal failure" in report.message-- def test_found_but_invalid_report_dict_synthesizes_unrecoverable(self) -> None:- """A report dict found in the chain but failing ``ErrorReport`` validation is an- internal contract bug. Rather than raise — which would abort the caller before it- can deliver the failure webhook, leaving the receiver with no notification at all —- it synthesizes the ``UnrecoverableWorkflowFailureError`` fallback, carrying the- recovered ``message`` and a marker flagging the validation failure. The payload- carries every required key (``error_type``, ``message``, ``title``, ``type_uri``)- so ``_find_error_report_dict`` picks it up, but its extra key violates- ``ErrorReport``'s ``extra="forbid"`` and fails ``from_dict`` validation.- """- invalid_report_dict: dict[str, Any] = {- "error_type": "CogtError",- "message": "rate limited",- "title": "AI inference failed",- "type_uri": "https://docs.pipelex.com/latest/errors/cogt-error/",- "future_field_we_do_not_know_about": "unexpected",- }- failure = _workflow_failure(_app_error(invalid_report_dict))- report = recover_error_report(failure)- assert report.error_type == "UnrecoverableWorkflowFailureError"- assert report.error_domain == ErrorDomain.RUNTIME- assert "rate limited" in report.message- # Pin the exact marker (imported constant), not a substring — the marker- # is a stable contract documented in docs/under-the-hood/error-model.md.- assert _ERROR_REPORT_VALIDATION_FAILED_MARKER in report.message-- @pytest.mark.parametrize(- "empty_like_message",- [- pytest.param("", id="empty"),- pytest.param(" ", id="single-space"),- pytest.param(" ", id="multiple-spaces"),- pytest.param("\n", id="newline"),- pytest.param("\t", id="tab"),- pytest.param(" \n\t ", id="mixed-whitespace"),- ],- )- def test_invalid_report_with_empty_or_whitespace_message_falls_back_to_exc_chain(self, empty_like_message: str) -> None:- """An invalid report dict with an empty or whitespace-only ``message`` falls back to the exception chain.-- Without ``str.strip()``, a whitespace-only ``message`` (a space, a- newline, a tab) is truthy in Python, so the fallback never fires and the- synthesized preamble renders as a visually broken `` [error report- failed schema validation]``. The strip guard treats every- whitespace-only value the same as the empty string.- """- invalid_report_dict: dict[str, Any] = {- "error_type": "CogtError",- "message": empty_like_message,- "title": "AI inference failed",- "type_uri": "https://docs.pipelex.com/latest/errors/cogt-error/",- "future_field_we_do_not_know_about": "unexpected",- }- failure = _workflow_failure(_app_error(invalid_report_dict))- report = recover_error_report(failure)- assert report.error_type == "UnrecoverableWorkflowFailureError"- # The fallback message must include both the ApplicationError text- # (recovered via the exception-chain walk) and the schema-validation marker.- assert "rate limited on the worker" in report.message- assert "schema validation" in report.message-- def test_recovers_report_past_report_less_wrapper_application_error(self) -> None:- """A report-less wrapper ``ApplicationError`` (e.g. a ``WorkflowExecutionError`` raised- when a workflow wraps a failed child workflow) does not hide the report-carrying- ``ApplicationError`` deeper in the ``__cause__`` chain.- """- inner = _app_error(_FULL_REPORT.to_dict())- wrapper = ApplicationError("workflow wrapping a failed child workflow", type="WorkflowExecutionError")- wrapper.__cause__ = inner- failure = _workflow_failure(wrapper)- assert recover_error_report(failure) == _FULL_REPORTdiff --git a/tests/unit/pipelex/temporal/test_resolve_dispatch.py b/tests/unit/pipelex/temporal/test_resolve_dispatch.pydeleted file mode 100644index 4230fcbcc..000000000--- a/tests/unit/pipelex/temporal/test_resolve_dispatch.py+++ /dev/null@@ -1,342 +0,0 @@-"""Unit tests for ``WorkerConfig.resolve_dispatch`` — the submitter-side-resolution chain that returns a ``DispatchOptions`` bundle for every-``workflow.execute_activity(...)`` call.--Covers the three layering layers (baseline → queue_options → handle_options)-for scalars (last-wins), the additive composition of-``non_retryable_error_types``, the hybrid empty-routing fallback that omits-``task_queue`` so Temporal uses the workflow's own queue, and the layer-aware-split between baseline ``RetryPolicyConfig`` and overlay-``RetryPolicyConfigOverlay`` enforced by Pydantic's ``extra="forbid"``.-"""--from datetime import timedelta--import pytest-from pydantic import ValidationError--from pipelex.temporal.config_temporal import (- ActivityRouteConfig,- HandleOptions,- QueueOptions,- RetryPolicyConfig,- RetryPolicyConfigOverlay,- WorkerConfig,-)---def _make_baseline_retry(non_retryable: list[str] | None = None) -> RetryPolicyConfig:- return RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=3,- non_retryable_error_types=non_retryable or ["ValidationError"],- )---def _make_overlay_retry(- non_retryable_extra: list[str] | None = None,- maximum_attempts: int = 3,- initial_interval_seconds: int = 3,-) -> RetryPolicyConfigOverlay:- return RetryPolicyConfigOverlay(- initial_interval=timedelta(seconds=initial_interval_seconds),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=maximum_attempts,- non_retryable_error_types_extra=non_retryable_extra or [],- )---def _make_worker_config(- activity_queues: dict[str, ActivityRouteConfig] | None = None,- baseline_start_to_close: timedelta | None = None,- baseline_heartbeat: timedelta | None = None,- baseline_non_retryable: list[str] | None = None,-) -> WorkerConfig:- return WorkerConfig(- default_task_queue="default_q",- activity_queues=activity_queues or {},- workflow_execution_timeout=timedelta(hours=1),- default_activity_start_to_close_timeout=baseline_start_to_close or timedelta(minutes=10),- default_activity_heartbeat_timeout=baseline_heartbeat,- retry_policy_config=_make_baseline_retry(baseline_non_retryable),- )---class TestResolveDispatch:- """Three-layer last-wins for scalars + additive composition for non_retryable."""-- def test_empty_activity_queues_omits_task_queue_for_workflow_local_dispatch(self) -> None:- """Hybrid fallback: with ``activity_queues`` empty (default config),- ``resolve_dispatch`` returns ``task_queue=None`` and ``to_execute_kwargs``- omits the key so Temporal routes to the workflow's own queue. Required- by the ``with_conditional_worker`` test isolation pattern.- """- worker_config = _make_worker_config()-- dispatch = worker_config.resolve_dispatch(activity_name="act_llm_gen_text", routing_key="claude-opus-4-7")-- assert dispatch.task_queue is None- kwargs = dispatch.to_execute_kwargs()- assert "task_queue" not in kwargs- assert "start_to_close_timeout" in kwargs- assert "retry_policy" in kwargs-- def test_non_empty_activity_queues_unmapped_falls_back_to_default_task_queue(self) -> None:- """When ``activity_queues`` is non-empty (operator opted into routing- topology) but the queried activity isn't mapped, dispatch falls back- explicitly to ``default_task_queue`` — operator clearly wants explicit- routing, so we don't second-guess them.- """- worker_config = _make_worker_config(- activity_queues={"act_img_gen_images": ActivityRouteConfig(default="img_q")},- )-- dispatch = worker_config.resolve_dispatch(activity_name="act_llm_gen_text")-- assert dispatch.task_queue == "default_q"- assert dispatch.to_execute_kwargs()["task_queue"] == "default_q"-- def test_mapped_activity_uses_baseline_when_no_overlay(self) -> None:- """With routing pinning ``act_llm_gen_text`` to a specific queue but no- ``queue_options`` overlay, dispatch produces that queue + baseline- scalars verbatim, and ``to_execute_kwargs`` includes ``task_queue``.- """- worker_config = _make_worker_config(- activity_queues={"act_llm_gen_text": ActivityRouteConfig(default="anthropic_q")},- )-- dispatch = worker_config.resolve_dispatch(activity_name="act_llm_gen_text", routing_key="claude-opus-4-7")-- assert dispatch.task_queue == "anthropic_q"- assert dispatch.start_to_close_timeout == timedelta(minutes=10)- assert dispatch.heartbeat_timeout is None- kwargs = dispatch.to_execute_kwargs()- assert kwargs["task_queue"] == "anthropic_q"- # Optional timeouts are omitted when None — Temporal SDK rejects None values.- assert "schedule_to_close_timeout" not in kwargs- assert "heartbeat_timeout" not in kwargs-- def test_queue_options_override_baseline(self) -> None:- """When ``queue_options[resolved_queue]`` provides a scalar, it overrides- the worker_config baseline.- """- worker_config = _make_worker_config(- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(default="anthropic_q"),- },- )- queue_options = {- "anthropic_q": QueueOptions(- start_to_close_timeout=timedelta(minutes=5),- heartbeat_timeout=timedelta(seconds=30),- ),- }-- dispatch = worker_config.resolve_dispatch(- activity_name="act_llm_gen_text",- routing_key="claude-opus-4-7",- queue_options_by_queue=queue_options,- )-- assert dispatch.task_queue == "anthropic_q"- assert dispatch.start_to_close_timeout == timedelta(minutes=5)- assert dispatch.heartbeat_timeout == timedelta(seconds=30)-- def test_handle_options_override_queue(self) -> None:- """A per-handle ``handle_options.start_to_close_timeout`` wins over the- queue-level value (deepest layer wins for scalars).- """- worker_config = _make_worker_config(- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(- default="anthropic_q",- handle_options={- "claude-opus-4-7-1m": HandleOptions(start_to_close_timeout=timedelta(minutes=25)),- },- ),- },- )- queue_options = {- "anthropic_q": QueueOptions(start_to_close_timeout=timedelta(minutes=5)),- }-- dispatch = worker_config.resolve_dispatch(- activity_name="act_llm_gen_text",- routing_key="claude-opus-4-7-1m",- queue_options_by_queue=queue_options,- )-- assert dispatch.start_to_close_timeout == timedelta(minutes=25)-- def test_non_retryable_additive(self) -> None:- """``non_retryable_error_types`` composes additively across all layers- (baseline + queue extra + handle extra). Deduplication preserves the- result as a deterministic ordered list with no duplicates.- """- worker_config = _make_worker_config(- baseline_non_retryable=["ValidationError"],- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(- default="anthropic_q",- handle_options={- "claude-opus-4-7-1m": HandleOptions(- retry_policy_config=_make_overlay_retry(- non_retryable_extra=["HandleSpecificError"],- ),- ),- },- ),- },- )- queue_options = {- "anthropic_q": QueueOptions(- retry_policy_config=_make_overlay_retry(- non_retryable_extra=["AnthropicBadRequestError", "ValidationError"],- maximum_attempts=6,- initial_interval_seconds=5,- ),- ),- }-- dispatch = worker_config.resolve_dispatch(- activity_name="act_llm_gen_text",- routing_key="claude-opus-4-7-1m",- queue_options_by_queue=queue_options,- )-- non_retryable = dispatch.retry_policy.non_retryable_error_types- assert non_retryable is not None- # Set equality — same contents regardless of order — plus length check to- # verify deduplication actually fires (queue layer repeats ValidationError).- assert set(non_retryable) == {- "ValidationError",- "AnthropicBadRequestError",- "HandleSpecificError",- }- assert len(non_retryable) == 3-- def test_queue_options_partial_overlay(self) -> None:- """Unset fields in ``QueueOptions`` (None) MUST NOT clobber the baseline- — only explicit values overlay.- """- worker_config = _make_worker_config(- baseline_heartbeat=timedelta(minutes=1),- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(default="anthropic_q"),- },- )- queue_options = {- # Only start_to_close_timeout is set; heartbeat_timeout must fall through.- "anthropic_q": QueueOptions(start_to_close_timeout=timedelta(minutes=5)),- }-- dispatch = worker_config.resolve_dispatch(- activity_name="act_llm_gen_text",- routing_key="claude-opus-4-7",- queue_options_by_queue=queue_options,- )-- assert dispatch.start_to_close_timeout == timedelta(minutes=5)- assert dispatch.heartbeat_timeout == timedelta(minutes=1)-- def test_split_classes_lock_in_extra_forbid(self) -> None:- """The split-class invariant is load-bearing on ``ConfigModel``'s- ``extra="forbid"`` setting: if ``extra="allow"`` were ever set on- either subclass, the layer asymmetry would silently lift and the- original silent-drop bug for baseline ``_extra`` would come back.- Assert the setting directly so a future contributor flipping it gets- a unit-test failure rather than a runtime regression.- """- assert RetryPolicyConfig.model_config.get("extra") == "forbid"- assert RetryPolicyConfigOverlay.model_config.get("extra") == "forbid"-- def test_baseline_class_rejects_extra_field(self) -> None:- """Issue #880 #5 regression: ``RetryPolicyConfig`` (baseline) has no- ``non_retryable_error_types_extra`` field. ``ConfigModel``'s- ``extra="forbid"`` raises if a user sets it on the baseline — preventing- the silent-drop bug where baseline ``_extra`` entries never reach the- merged non-retryable list at dispatch time.- """- with pytest.raises(ValidationError) as exc_info:- RetryPolicyConfig.model_validate(- {- "initial_interval": "0:00:03",- "backoff_coefficient": 2.0,- "maximum_interval": "unlimited",- "maximum_attempts": 3,- "non_retryable_error_types": [],- "non_retryable_error_types_extra": ["should_be_rejected"],- },- )- assert "non_retryable_error_types_extra" in str(exc_info.value)-- def test_overlay_class_rejects_main_list_field(self) -> None:- """Issue #880 #5 regression: ``RetryPolicyConfigOverlay`` has no- ``non_retryable_error_types`` field. ``ConfigModel``'s ``extra="forbid"``- raises if a user sets it on an overlay — overlays MUST contribute via- ``_extra`` so the additive composition rule stays unambiguous from the- config alone.- """- with pytest.raises(ValidationError) as exc_info:- RetryPolicyConfigOverlay.model_validate(- {- "initial_interval": "0:00:03",- "backoff_coefficient": 2.0,- "maximum_interval": "unlimited",- "maximum_attempts": 3,- "non_retryable_error_types": ["should_be_rejected"],- },- )- assert "non_retryable_error_types" in str(exc_info.value)-- def test_queue_options_applies_via_default_task_queue_when_routing_empty(self) -> None:- """Empty-routing hybrid fallback asymmetry: dispatch still emits- ``task_queue=None`` (activities ride the workflow's own queue) but- ``queue_options[default_task_queue]`` still applies so single-queue- deployments can tune timeouts/retry/rate without opting into the- ``activity_queues`` routing topology.- """- worker_config = _make_worker_config() # empty activity_queues, default_task_queue="default_q"- queue_options = {"default_q": QueueOptions(start_to_close_timeout=timedelta(minutes=5))}-- dispatch = worker_config.resolve_dispatch(- activity_name="act_llm_gen_text",- routing_key="claude-opus-4-7",- queue_options_by_queue=queue_options,- )-- # Dispatch still omits task_queue (workflow-local).- assert dispatch.task_queue is None- assert "task_queue" not in dispatch.to_execute_kwargs()- # But the queue_options[default_task_queue] timeout overlay applies.- assert dispatch.start_to_close_timeout == timedelta(minutes=5)-- def test_resolve_queue_still_works(self) -> None:- """``resolve_queue`` must remain a thin delegate consistent with- ``resolve_dispatch``'s queue resolution. Regression guard for callers- that only need the queue name. Empty-routing returns None; non-empty- routing uses the v1 three-layer resolution.- """- worker_config = _make_worker_config(- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(- default="inference_q",- by_handle={"claude-opus-4-7": "anthropic_q"},- ),- },- )-- # Per-handle wins.- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="claude-opus-4-7") == "anthropic_q"- # Activity default for unmapped handle.- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="gpt-5") == "inference_q"- # Worker default for unmapped activity (routing IS configured, so non-empty hybrid path).- assert worker_config.resolve_queue("act_jinja2_gen_text") == "default_q"-- # resolve_dispatch agrees with resolve_queue on the queue name.- assert worker_config.resolve_dispatch("act_llm_gen_text", routing_key="claude-opus-4-7").task_queue == "anthropic_q"- assert worker_config.resolve_dispatch("act_llm_gen_text", routing_key="gpt-5").task_queue == "inference_q"- assert worker_config.resolve_dispatch("act_jinja2_gen_text").task_queue == "default_q"diff --git a/tests/unit/pipelex/temporal/test_search_attribute_bootstrap_check.py b/tests/unit/pipelex/temporal/test_search_attribute_bootstrap_check.pydeleted file mode 100644index 4c7209d3d..000000000--- a/tests/unit/pipelex/temporal/test_search_attribute_bootstrap_check.py+++ /dev/null@@ -1,147 +0,0 @@-"""Unit tests for the worker-startup hard-fail check of configured custom search attributes.--Phase 6 contract:--- All configured attributes present → no warning, no exception.-- Some configured attributes missing on a reachable namespace → raise- ``SearchAttributeRegistrationError`` with both the ``pipelex- setup-temporal-namespace`` invocation and the equivalent raw ``temporal- operator search-attribute create`` command in the message.-- ``RPCError`` from the operator service → worker boot continues, soft-fail- warning logged.-- Any other exception → propagates and crashes the worker (real bug).-- Empty ``configured_attributes`` (``[temporal.search_attributes].enabled =- false`` short-circuits to this) → check is skipped entirely; the operator- service is never called.-- Configured subset → only the subset is checked; missing built-ins outside- the subset don't trigger the error.-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture-from temporalio.service import RPCError, RPCStatusCode--from pipelex.temporal.config_temporal import BUILTIN_SEARCH_ATTRIBUTES-from pipelex.temporal.exceptions import SearchAttributeRegistrationError-from pipelex.temporal.tprl.namespace_check import check_required_search_attributes---def _make_temporal_client_stub(mocker: MockerFixture, custom_attributes: dict[str, Any] | Exception) -> Any:- """Stub a temporal client whose ``operator_service.list_search_attributes``- returns the given attributes (or raises the given exception).- """- client = mocker.MagicMock()- response = mocker.MagicMock()- response.custom_attributes = custom_attributes if not isinstance(custom_attributes, Exception) else {}- if isinstance(custom_attributes, Exception):- client.operator_service.list_search_attributes = mocker.AsyncMock(side_effect=custom_attributes)- else:- client.operator_service.list_search_attributes = mocker.AsyncMock(return_value=response)- return client--[email protected](loop_scope="class")-class TestSearchAttributeBootstrapCheck:- async def test_all_configured_attributes_present_is_silent(self, mocker: MockerFixture) -> None:- client = _make_temporal_client_stub(- mocker,- custom_attributes={name: object() for name in BUILTIN_SEARCH_ATTRIBUTES},- )- warning_mock = mocker.patch("pipelex.temporal.tprl.namespace_check.log.warning")-- await check_required_search_attributes(- temporal_client=client,- namespace="default",- configured_attributes=BUILTIN_SEARCH_ATTRIBUTES,- )-- warning_mock.assert_not_called()-- async def test_some_configured_attributes_missing_raises_with_both_commands(self, mocker: MockerFixture) -> None:- # Only two of the five present; three are missing.- client = _make_temporal_client_stub(- mocker,- custom_attributes={"PipeCode": object(), "PipelineRunId": object()},- )-- with pytest.raises(SearchAttributeRegistrationError) as exc_info:- await check_required_search_attributes(- temporal_client=client,- namespace="default",- configured_attributes=BUILTIN_SEARCH_ATTRIBUTES,- )-- message = str(exc_info.value)- # The missing names appear in the message.- assert "SessionId" in message- assert "UserId" in message- assert "DomainCode" in message- # The Pipelex CLI invocation is part of the actionable hint.- assert "pipelex setup-temporal-namespace" in message- # The equivalent raw Temporal CLI command is also embedded verbatim.- assert "temporal operator search-attribute create" in message- assert "--namespace default" in message- assert "--name SessionId --type Keyword" in message- assert "--name UserId --type Keyword" in message- assert "--name DomainCode --type Keyword" in message-- async def test_rpc_error_soft_fails_and_logs_warning(self, mocker: MockerFixture) -> None:- rpc_error = RPCError("namespace not reachable", RPCStatusCode.UNAVAILABLE, raw_grpc_status=b"")- client = _make_temporal_client_stub(mocker, custom_attributes=rpc_error)- warning_mock = mocker.patch("pipelex.temporal.tprl.namespace_check.log.warning")-- # Must NOT raise — soft-fail when the cluster control plane is unreachable.- await check_required_search_attributes(- temporal_client=client,- namespace="default",- configured_attributes=BUILTIN_SEARCH_ATTRIBUTES,- )-- warning_mock.assert_called_once()- assert "RPCError" in warning_mock.call_args.args[0]-- async def test_non_rpc_error_propagates_and_crashes(self, mocker: MockerFixture) -> None:- """Anything other than ``RPCError`` is a real bug — must propagate."""- client = _make_temporal_client_stub(mocker, custom_attributes=RuntimeError("unexpected bug"))-- with pytest.raises(RuntimeError, match="unexpected bug"):- await check_required_search_attributes(- temporal_client=client,- namespace="default",- configured_attributes=BUILTIN_SEARCH_ATTRIBUTES,- )-- async def test_configured_subset_ignores_missing_attributes_outside_the_subset(self, mocker: MockerFixture) -> None:- """When the configured subset is ``["PipeCode", "DomainCode"]`` and both- of those are registered, the check must pass even though the three other- built-ins are missing — they're not in the subset.- """- client = _make_temporal_client_stub(- mocker,- custom_attributes={"PipeCode": object(), "DomainCode": object()},- )-- await check_required_search_attributes(- temporal_client=client,- namespace="default",- configured_attributes=["PipeCode", "DomainCode"],- )-- async def test_empty_configured_attributes_skips_the_check_entirely(self, mocker: MockerFixture) -> None:- """When ``[temporal.search_attributes].enabled = false`` is wired- through, callers pass an empty configured subset. The operator service- must not be called at all.- """- client = _make_temporal_client_stub(mocker, custom_attributes={})- warning_mock = mocker.patch("pipelex.temporal.tprl.namespace_check.log.warning")-- await check_required_search_attributes(- temporal_client=client,- namespace="default",- configured_attributes=[],- )-- client.operator_service.list_search_attributes.assert_not_called()- warning_mock.assert_not_called()diff --git a/tests/unit/pipelex/temporal/test_search_attribute_dict_construction.py b/tests/unit/pipelex/temporal/test_search_attribute_dict_construction.pydeleted file mode 100644index 36fadda9e..000000000--- a/tests/unit/pipelex/temporal/test_search_attribute_dict_construction.py+++ /dev/null@@ -1,118 +0,0 @@-"""Unit tests for the five-keyed typed search attributes.--Pins the schema documented in-``wip/temporal-primitives/id-and-naming-design.md`` §"Layer 4 — Search & Filter":--| Attribute | Source |-|----------------|-------------------------------------------------|-| PipeCode | pipe_job.pipe.code |-| PipelineRunId | pipe_job.job_metadata.pipeline_run_id |-| SessionId | pipe_job.job_metadata.session_id (stamped |-| | from TemporalManager at submitter dispatch) |-| UserId | pipe_job.job_metadata.user_id |-| DomainCode | pipe_job.pipe.domain_code |--Every value flows off ``pipe_job`` — the helper is a pure function of the-workflow input. ``SessionId`` is read directly from ``TemporalManager`` only-once, by ``stamp_submitter_session_id`` at the top-level dispatch boundary;-inside workflow code the value comes through ``job_metadata`` unchanged so-``StartChildWorkflowExecution`` commands stay byte-equal across replays even-when the worker process restarts.--Child workflows reuse the same builder against the child's own ``pipe_job``:-``PipelineRunId`` / ``UserId`` / ``SessionId`` are inherited from the parent-via ``pipe_job.job_metadata`` (propagated as workflow input), and ``PipeCode`` /-``DomainCode`` naturally reflect the child's pipe.-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture--from pipelex.temporal.tprl.observability import (- DOMAIN_CODE_KEY,- PIPE_CODE_KEY,- PIPELINE_RUN_ID_KEY,- SESSION_ID_KEY,- USER_ID_KEY,- build_search_attributes,-)---def _make_pipe_job_stub(- mocker: MockerFixture,- *,- pipe_code: str,- domain_code: str,- user_id: str = "acme-corp",- pipeline_run_id: str = "3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c",- session_id: str = "EdgdJ7Yk4Q3HF2pXyZv9w8",-) -> Any:- pipe_job = mocker.MagicMock()- pipe_job.pipe.code = pipe_code- pipe_job.pipe.domain_code = domain_code- pipe_job.job_metadata.user_id = user_id- pipe_job.job_metadata.pipeline_run_id = pipeline_run_id- pipe_job.job_metadata.session_id = session_id- return pipe_job---def patch_search_attributes_config_all_enabled(mocker: MockerFixture) -> None:- """Patch ``get_config`` so ``build_search_attributes`` sees the default- "all five enabled" surface without booting Pipelex.- """- config_root = mocker.MagicMock()- config_root.temporal.search_attributes.enabled = True- config_root.temporal.search_attributes.attributes = ["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"]- mocker.patch("pipelex.temporal.tprl.observability.get_config", return_value=config_root)--[email protected]("patch_search_attributes_config_all_enabled")-class TestSearchAttributeDictConstruction:- def test_top_level_attrs_have_five_keys_with_correct_value_sources(self, mocker: MockerFixture) -> None:- pipe_job = _make_pipe_job_stub(- mocker,- pipe_code="translate_doc",- domain_code="documents",- user_id="acme-corp",- pipeline_run_id="3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c",- )-- attrs = build_search_attributes(pipe_job)-- assert len(attrs) == 5- assert attrs[PIPE_CODE_KEY] == "translate_doc"- assert attrs[PIPELINE_RUN_ID_KEY] == "3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c"- assert attrs[SESSION_ID_KEY] == "EdgdJ7Yk4Q3HF2pXyZv9w8"- assert attrs[USER_ID_KEY] == "acme-corp"- assert attrs[DOMAIN_CODE_KEY] == "documents"-- def test_child_pipe_job_carries_inherited_identity_and_own_pipe_code(self, mocker: MockerFixture) -> None:- """Child workflows reuse ``build_search_attributes`` on the child's own- ``pipe_job``. ``PipeCode`` / ``DomainCode`` reflect the child's pipe;- ``PipelineRunId`` / ``UserId`` / ``SessionId`` are inherited from the- parent because the child's ``job_metadata`` is propagated as workflow- input from the parent — including the ``SessionId`` stamped at the- submitter dispatch boundary, which keeps the start-child-workflow- command deterministic across replays.- """- child_pipe_job = _make_pipe_job_stub(- mocker,- pipe_code="extract_text",- domain_code="extraction",- pipeline_run_id="parent-run-id",- user_id="parent-user",- session_id="parent-session",- )-- child_attrs = build_search_attributes(child_pipe_job)-- # Child's own:- assert child_attrs[PIPE_CODE_KEY] == "extract_text"- assert child_attrs[DOMAIN_CODE_KEY] == "extraction"- # Inherited via propagated job_metadata:- assert child_attrs[PIPELINE_RUN_ID_KEY] == "parent-run-id"- assert child_attrs[USER_ID_KEY] == "parent-user"- assert child_attrs[SESSION_ID_KEY] == "parent-session"diff --git a/tests/unit/pipelex/temporal/test_search_attributes_config.py b/tests/unit/pipelex/temporal/test_search_attributes_config.pydeleted file mode 100644index 9f7ec6468..000000000--- a/tests/unit/pipelex/temporal/test_search_attributes_config.py+++ /dev/null@@ -1,57 +0,0 @@-"""Unit tests for ``SearchAttributesConfig`` validation.--Phase 6 introduces a config block that controls the custom Temporal search-attribute surface. The pydantic model rejects unknown attribute names so a-typo like ``"PipelineRunID"`` (wrong casing) fails at config-load instead of-silently producing no attribute at workflow-start time.-"""--import pytest-from pydantic import ValidationError--from pipelex.temporal.config_temporal import BUILTIN_SEARCH_ATTRIBUTES, SearchAttributesConfig---class TestSearchAttributesConfig:- def test_default_full_subset_accepts_all_built_in_names(self) -> None:- config = SearchAttributesConfig(- enabled=True,- attributes=list(BUILTIN_SEARCH_ATTRIBUTES),- )-- assert config.enabled is True- assert config.attributes == list(BUILTIN_SEARCH_ATTRIBUTES)-- def test_partial_subset_is_accepted(self) -> None:- config = SearchAttributesConfig(- enabled=True,- attributes=["PipeCode", "DomainCode"],- )-- assert config.attributes == ["PipeCode", "DomainCode"]-- def test_disabled_with_empty_attributes_is_accepted(self) -> None:- config = SearchAttributesConfig(enabled=False, attributes=[])-- assert config.enabled is False- assert config.attributes == []-- def test_unknown_attribute_name_raises_validation_error(self) -> None:- with pytest.raises(ValidationError) as exc_info:- SearchAttributesConfig(- enabled=True,- # Note the wrong casing on the last segment — this is the kind- # of typo the validator is designed to catch.- attributes=["PipeCode", "PipelineRunID"],- )-- message = str(exc_info.value)- assert "PipelineRunID" in message- # The error message lists the known good names so the operator can- # fix the typo without grepping the source.- for name in BUILTIN_SEARCH_ATTRIBUTES:- assert name in message-- def test_arbitrary_custom_name_is_rejected(self) -> None:- with pytest.raises(ValidationError):- SearchAttributesConfig(enabled=True, attributes=["MyCustomAttribute"])diff --git a/tests/unit/pipelex/temporal/test_shipping_default_queue_options_empty.py b/tests/unit/pipelex/temporal/test_shipping_default_queue_options_empty.pydeleted file mode 100644index 3cfa52098..000000000--- a/tests/unit/pipelex/temporal/test_shipping_default_queue_options_empty.py+++ /dev/null@@ -1,35 +0,0 @@-"""Regression guard: the package-default ``pipelex/pipelex.toml`` ships-``[temporal.queue_options]`` as an empty table — no ``[temporal.queue_options.<queue>]``-sub-tables.--Shipping an overlay for the baseline's ``default_task_queue`` name (e.g.-``temporal_task_queue``) would create an orphan overlay the moment a-downstream deployment renames its queue. Config merging cannot delete-baseline keys, so that orphan would be unrecoverable from the override.-The downstream `.pipelex/pipelex.toml` and `pipelex/kit/configs/pipelex.toml`-template were aligned to match this rationale.--If a future change re-adds a `[temporal.queue_options.<queue>]` block to-the package default, this test fails loudly.-"""--from pathlib import Path--import tomli--import pipelex--_PIPELEX_DEFAULT_CONFIG_PATH = Path(pipelex.__file__).parent / "pipelex.toml"---class TestShippingDefaultQueueOptionsEmpty:- def test_no_queue_specific_overlay_shipped(self) -> None:- with _PIPELEX_DEFAULT_CONFIG_PATH.open("rb") as handle:- config = tomli.load(handle)-- queue_options = config["temporal"]["queue_options"]- sub_tables = [key for key, value in queue_options.items() if isinstance(value, dict)]- assert sub_tables == [], (- f"Package-default `[temporal.queue_options]` must ship empty to avoid orphan-overlay drift "- f"when a deployment renames its default_task_queue. Found unexpected sub-tables: {sub_tables}."- )diff --git a/tests/unit/pipelex/temporal/test_storage_payload_codec.py b/tests/unit/pipelex/temporal/test_storage_payload_codec.pydeleted file mode 100644index f96189b10..000000000--- a/tests/unit/pipelex/temporal/test_storage_payload_codec.py+++ /dev/null@@ -1,310 +0,0 @@-from __future__ import annotations--import pytest-from temporalio.api.common.v1 import Payload--from pipelex.temporal.codec.storage_payload_codec import STORAGE_REF_ENCODING, StoragePayloadCodec-from pipelex.tools.storage.exceptions import StorageInvalidUriError-from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider--TEST_THRESHOLD = 1024-TEST_PREFIX = "test-payloads/"---def _make_payload(size: int) -> Payload:- """Build a Payload with the given data size."""- return Payload(- metadata={"encoding": b"binary/plain"},- data=b"x" * size,- )--[email protected](loop_scope="class")-class TestStoragePayloadCodec:- """Unit tests for StoragePayloadCodec encode/decode behavior."""-- @pytest.fixture- def storage(self) -> InMemoryStorageProvider:- return InMemoryStorageProvider()-- @pytest.fixture- def codec(self, storage: InMemoryStorageProvider) -> StoragePayloadCodec:- return StoragePayloadCodec(- storage_provider=storage,- size_threshold=TEST_THRESHOLD,- storage_prefix=TEST_PREFIX,- )-- async def test_below_threshold_passes_through(- self,- codec: StoragePayloadCodec,- ) -> None:- """Payloads smaller than the threshold pass through encode/decode unchanged."""- small_payload = _make_payload(TEST_THRESHOLD - 100)- original_bytes = small_payload.SerializeToString()-- encoded = await codec.encode([small_payload])-- assert len(encoded) == 1- assert encoded[0].SerializeToString() == original_bytes-- decoded = await codec.decode(encoded)-- assert len(decoded) == 1- assert decoded[0].SerializeToString() == original_bytes-- async def test_above_threshold_offloads_to_storage(- self,- codec: StoragePayloadCodec,- storage: InMemoryStorageProvider,- ) -> None:- """Payloads larger than the threshold are offloaded to storage and reconstructed on decode."""- large_payload = _make_payload(TEST_THRESHOLD + 100)- original_bytes = large_payload.SerializeToString()-- encoded = await codec.encode([large_payload])-- assert len(encoded) == 1- ref_payload = encoded[0]-- # The reference payload must carry the storage-ref encoding marker- assert ref_payload.metadata.get("encoding") == STORAGE_REF_ENCODING-- # The reference payload data is a URI pointing to the stored blob- uri = ref_payload.data.decode()- assert uri.startswith("pipelex-storage://")-- # The storage provider must contain the original serialized payload- assert len(storage.root) == 1-- # Decode must reconstruct the original payload exactly- decoded = await codec.decode(encoded)-- assert len(decoded) == 1- assert decoded[0].SerializeToString() == original_bytes-- async def test_content_addressed_deduplication(- self,- codec: StoragePayloadCodec,- storage: InMemoryStorageProvider,- ) -> None:- """Encoding the same large payload twice produces a single storage entry."""- large_payload = _make_payload(TEST_THRESHOLD + 50)-- await codec.encode([large_payload])- await codec.encode([large_payload])-- assert len(storage.root) == 1-- async def test_mixed_payloads(- self,- codec: StoragePayloadCodec,- storage: InMemoryStorageProvider,- ) -> None:- """In a batch, small payloads pass through and large payloads are offloaded."""- small = _make_payload(10)- large = _make_payload(TEST_THRESHOLD + 200)- small_bytes = small.SerializeToString()- large_bytes = large.SerializeToString()-- encoded = await codec.encode([small, large])-- assert len(encoded) == 2-- # First payload (small) passed through unchanged- assert encoded[0].SerializeToString() == small_bytes-- # Second payload (large) is a storage reference- assert encoded[1].metadata.get("encoding") == STORAGE_REF_ENCODING-- # Only the large payload was stored- assert len(storage.root) == 1-- # Full round-trip preserves both payloads- decoded = await codec.decode(encoded)-- assert len(decoded) == 2- assert decoded[0].SerializeToString() == small_bytes- assert decoded[1].SerializeToString() == large_bytes-- async def test_exactly_at_threshold_is_offloaded(- self,- storage: InMemoryStorageProvider,- ) -> None:- """A payload whose serialized size equals the threshold is offloaded, not passed through."""- payload = _make_payload(500)- serialized_size = len(payload.SerializeToString())-- codec = StoragePayloadCodec(- storage_provider=storage,- size_threshold=serialized_size,- storage_prefix=TEST_PREFIX,- )-- encoded = await codec.encode([payload])-- assert len(encoded) == 1- assert encoded[0].metadata.get("encoding") == STORAGE_REF_ENCODING- assert len(storage.root) == 1-- @pytest.mark.parametrize(- ("topic", "size"),- [- ("small", 10),- ("at_threshold", TEST_THRESHOLD),- ("above_threshold", TEST_THRESHOLD + 500),- ],- )- async def test_round_trip_fidelity(- self,- codec: StoragePayloadCodec,- topic: str,- size: int,- ) -> None:- """decode(encode(payloads)) returns identical payloads for various sizes."""- payload = _make_payload(size)- original_bytes = payload.SerializeToString()-- encoded = await codec.encode([payload])- decoded = await codec.decode(encoded)-- assert len(decoded) == 1- assert decoded[0].SerializeToString() == original_bytes, f"Round-trip failed for {topic} (size={size})"-- # -------------------------------------------------------------------- # _extract_job_routing tests- # --------------------------------------------------------------------- async def test_storage_key_includes_routing_when_available(- self,- codec: StoragePayloadCodec,- ) -> None:- """When job_metadata is present, storage key includes user_id and pipeline_run_id."""- routed_payload = Payload(- metadata={"encoding": b"json/plain"},- data=b'{"job_metadata": {"user_id": "user42", "pipeline_run_id": "run99"}, "big": "' + b"x" * (TEST_THRESHOLD + 500) + b'"}',- )-- encoded = await codec.encode([routed_payload])- assert len(encoded) == 1- uri = encoded[0].data.decode()- assert "user42" in uri- assert "run99" in uri-- @pytest.mark.parametrize(- ("topic", "payload_data"),- [- (- "non_json_encoding_no_routing",- b"x" * (TEST_THRESHOLD + 500),- ),- (- "json_missing_job_metadata_no_routing",- b'{"other_key": "value", "pad": "' + b"x" * (TEST_THRESHOLD + 500) + b'"}',- ),- (- "json_non_dict_job_metadata_no_routing",- b'{"job_metadata": "not a dict", "pad": "' + b"x" * (TEST_THRESHOLD + 500) + b'"}',- ),- ],- )- async def test_storage_key_falls_back_to_flat_when_no_routing(- self,- storage: InMemoryStorageProvider,- topic: str,- payload_data: bytes,- ) -> None:- """Payloads without valid job_metadata use a flat storage key (prefix + hash only)."""- codec = StoragePayloadCodec(- storage_provider=storage,- size_threshold=TEST_THRESHOLD,- storage_prefix=TEST_PREFIX,- )- payload = Payload(- metadata={"encoding": b"json/plain" if topic.startswith("json_") else b"binary/plain"},- data=payload_data,- )-- encoded = await codec.encode([payload])- assert len(encoded) == 1- uri = encoded[0].data.decode()- # Flat keys have no slashes after the prefix (just prefix + hash)- key_after_scheme = uri.split("://", 1)[-1]- segments = key_after_scheme.split("/")- # prefix is "test-payloads/" so we expect: test-payloads/{hash} = 2 segments- assert len(segments) == 2, f"Expected flat key for {topic}, got: {key_after_scheme}"-- # -------------------------------------------------------------------- # _sanitize_path_segment tests- # --------------------------------------------------------------------- @pytest.mark.parametrize(- ("topic", "user_id", "pipeline_run_id", "expected_user", "expected_run"),- [- ("path_traversal_dotdot", "../../../etc", "run1", "_________etc", "run1"),- ("path_traversal_in_run", "user1", "../../secret", "user1", "______secret"),- ("slashes_stripped", "user/sub/path", "run/id", "user_sub_path", "run_id"),- ("special_chars", "[email protected]", "run id+test", "user_evil_com", "run_id_test"),- ("clean_ids_unchanged", "user42", "run-99_abc", "user42", "run-99_abc"),- ],- )- async def test_storage_key_sanitizes_routing_segments(- self,- codec: StoragePayloadCodec,- topic: str,- user_id: str,- pipeline_run_id: str,- expected_user: str,- expected_run: str,- ) -> None:- """Routing segments are sanitized to prevent path traversal and special char issues."""- metadata_json = (- f'{{"job_metadata": {{"user_id": "{user_id}", "pipeline_run_id": "{pipeline_run_id}"}}, "big": "' + "x" * (TEST_THRESHOLD + 500) + '"}'- )- routed_payload = Payload(- metadata={"encoding": b"json/plain"},- data=metadata_json.encode(),- )-- encoded = await codec.encode([routed_payload])- assert len(encoded) == 1- uri = encoded[0].data.decode()- key_after_scheme = uri.split("://", 1)[-1]- # Expect: test-payloads/{safe_user}/{safe_run}/{hash} = 4 segments- segments = key_after_scheme.split("/")- assert len(segments) == 4, f"Expected routed key for {topic}, got: {key_after_scheme}"- assert segments[1] == expected_user, f"User segment mismatch for {topic}"- assert segments[2] == expected_run, f"Run segment mismatch for {topic}"-- # -------------------------------------------------------------------- # decode() URI prefix guard tests- # --------------------------------------------------------------------- @pytest.mark.parametrize(- ("topic", "tampered_uri"),- [- ("foreign_prefix", "pipelex-storage://other-tenant/abc123"),- ("missing_scheme", "test-payloads/abc123"),- ("foreign_scheme", "file:///etc/passwd"),- ("absolute_path_no_scheme", "/etc/passwd"),- ("empty_uri", ""),- ],- )- async def test_decode_rejects_uri_outside_configured_prefix(- self,- codec: StoragePayloadCodec,- storage: InMemoryStorageProvider,- topic: str,- tampered_uri: str,- ) -> None:- """decode() refuses storage references whose URI does not start with the configured prefix."""- tampered_payload = Payload(- metadata={"encoding": STORAGE_REF_ENCODING},- data=tampered_uri.encode(),- )-- with pytest.raises(StorageInvalidUriError, match="does not start with expected prefix"):- await codec.decode([tampered_payload])-- # Storage must not be touched when the URI is rejected by the prefix guard- assert len(storage.root) == 0, f"Storage should not be accessed for tampered URI ({topic})"diff --git a/tests/unit/pipelex/temporal/test_temporal_config_orphan_queues.py b/tests/unit/pipelex/temporal/test_temporal_config_orphan_queues.pydeleted file mode 100644index f5ffa42b7..000000000--- a/tests/unit/pipelex/temporal/test_temporal_config_orphan_queues.py+++ /dev/null@@ -1,236 +0,0 @@-"""Unit tests for ``Temporal.validate_no_orphan_queue_references``.--When ``activity_queues.*.default`` or ``activity_queues.*.by_handle.*`` names-a queue with no ``queue_options`` entry (and which isn't the worker's-``default_task_queue``), the config-load validator must raise-``TemporalConfigError`` so typos surface at boot rather than at runtime as-silent misroutes.--Pydantic wraps ``TemporalConfigError`` (a ``ValueError`` subclass) in-``pydantic.ValidationError`` when raised from a ``model_validator``, so tests-match on ``ValidationError`` and inspect the message for the offending queue.-"""--from datetime import timedelta--import pytest-from pydantic import ValidationError--from pipelex.temporal.config_temporal import (- BUILTIN_SEARCH_ATTRIBUTES,- ActivityRouteConfig,- PayloadCodecConfig,- QueueOptions,- RetryPolicyConfig,- SearchAttributesConfig,- SecretMethod,- Temporal,- TemporalConfig,- TemporalLogConfig,- TemporalServerConfig,- WorkerConfig,- WorkerRuntimeProfile,- WorkerRuntimeProfilesConfig,- WorkerScope,- WorkerScopesConfig,- WorkerTuningMode,-)-from pipelex.tools.storage.storage_config import StorageProviderConfig---def _make_storage_provider_config() -> StorageProviderConfig:- return StorageProviderConfig.model_validate({"method": "in_memory", "in_memory": {"uri_format": "{hash}"}})---def _make_temporal_config(activity_queues: dict[str, ActivityRouteConfig], queue_options: dict[str, QueueOptions]) -> Temporal:- """Build a minimal valid Temporal config for orphan-queue validation tests.-- All non-test fields are filled with sensible defaults; only- ``activity_queues`` and ``queue_options`` matter for the validation logic- under test.- """- return Temporal(- is_enabled=True,- temporal_config=TemporalConfig(- temporal_server_configs={- "local": TemporalServerConfig(- description="local",- target_host="localhost:7233",- namespace="default",- api_key_method=SecretMethod.NONE,- api_key_id="",- ),- },- selected_server="local",- temporal_log_config=TemporalLogConfig(- is_workflow_info_on_message=False,- is_workflow_info_on_extra=True,- is_full_workflow_info_on_extra=False,- is_activity_info_on_message=False,- is_activity_info_on_extra=True,- is_full_activity_info_on_extra=False,- is_formatter_enabled=True,- is_prefix_enabled=True,- managed_loggers=[],- is_dispatch_resolution_traced=False,- ),- ),- worker_config=WorkerConfig(- default_task_queue="default_q",- activity_queues=activity_queues,- workflow_execution_timeout=timedelta(hours=1),- default_activity_start_to_close_timeout=timedelta(minutes=10),- retry_policy_config=RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=3,- non_retryable_error_types=[],- ),- ),- queue_options=queue_options,- worker_runtime_profiles=WorkerRuntimeProfilesConfig(- default_profile="default",- profiles={- "default": WorkerRuntimeProfile(- tuning_mode=WorkerTuningMode.EXPLICIT,- max_cached_workflows=10000,- max_concurrent_workflow_tasks=1000,- max_concurrent_activities=1000,- max_concurrent_local_activities=1000,- max_concurrent_workflow_task_polls=100,- max_concurrent_activity_task_polls=100,- max_activities_per_second=1000,- sticky_queue_schedule_to_start_timeout=timedelta(minutes=30),- max_heartbeat_throttle_interval=timedelta(minutes=60),- default_heartbeat_throttle_interval=timedelta(minutes=60),- graceful_shutdown_timeout=timedelta(minutes=30),- ),- },- ),- worker_scopes=WorkerScopesConfig(- default_scope="full",- scopes={- "full": WorkerScope(- required_tasks_packs=[],- required_workflows=[],- required_activities=[],- excluded_workflows=[],- excluded_activities=[],- disable_all_workflows=False,- disable_all_activities=False,- ),- },- ),- payload_codec_config=PayloadCodecConfig(- is_enabled=False,- size_threshold=1_048_576,- storage_prefix="prefix/",- storage_provider_config=_make_storage_provider_config(),- ),- search_attributes=SearchAttributesConfig(- enabled=True,- attributes=list(BUILTIN_SEARCH_ATTRIBUTES),- ),- )---class TestTemporalConfigOrphanQueues:- """Config-load fails on routing entries that reference unknown queues."""-- def test_unknown_queue_in_activity_by_handle_raises(self) -> None:- """A ``by_handle`` entry naming a queue with no ``queue_options`` and- not equal to ``default_task_queue`` raises with the activity + handle- path and a fix suggestion.- """- with pytest.raises(ValidationError, match="anthrpic_q") as exc_info:- _make_temporal_config(- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(- default="default_q",- by_handle={"claude-opus-4-7": "anthrpic_q"}, # typo- ),- },- queue_options={},- )- message = str(exc_info.value)- assert "act_llm_gen_text" in message- assert "claude-opus-4-7" in message- assert "[temporal.queue_options.anthrpic_q]" in message-- def test_unknown_queue_in_activity_default_raises(self) -> None:- """An ``activity_queues.*.default`` entry naming a queue with no- ``queue_options`` and not equal to ``default_task_queue`` raises with- a fix suggestion that names the orphan queue.- """- with pytest.raises(ValidationError, match="missing_q") as exc_info:- _make_temporal_config(- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(- default="missing_q",- by_handle={},- ),- },- queue_options={},- )- message = str(exc_info.value)- assert "[temporal.queue_options.missing_q]" in message- assert "activity_queues.act_llm_gen_text.default" in message-- def test_no_error_when_queue_in_queue_options(self) -> None:- """A queue present in ``queue_options`` is known — config loads."""- _make_temporal_config(- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(- default="anthropic_q",- by_handle={},- ),- },- queue_options={"anthropic_q": QueueOptions(start_to_close_timeout=timedelta(minutes=5))},- )-- def test_empty_queue_options_stanza_is_sufficient(self) -> None:- """An empty ``[temporal.queue_options.<q>]`` stanza is the explicit- "use worker_config defaults" declaration and satisfies the validator.- """- _make_temporal_config(- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(default="my_q", by_handle={}),- },- queue_options={"my_q": QueueOptions()},- )-- def test_default_task_queue_is_known_without_queue_options(self) -> None:- """``default_task_queue`` is implicitly known — config loads even- when no ``queue_options`` entry exists for it.- """- _make_temporal_config(- activity_queues={- "act_llm_gen_text": ActivityRouteConfig(default="default_q", by_handle={}),- },- queue_options={},- )-- def test_unreachable_queue_options_entry_raises(self) -> None:- """A ``queue_options`` entry naming a queue that no ``activity_queues``- route references AND that isn't ``default_task_queue`` raises — the- overlay would never apply.- """- with pytest.raises(ValidationError, match="orphan_q") as exc_info:- _make_temporal_config(- activity_queues={},- queue_options={"orphan_q": QueueOptions(start_to_close_timeout=timedelta(minutes=5))},- )- message = str(exc_info.value)- assert "overlay will never apply" in message- assert "[temporal.queue_options.orphan_q]" in message-- def test_default_task_queue_in_queue_options_does_not_raise(self) -> None:- """``queue_options[default_task_queue]`` is the supported single-queue- tuning path — it must not trigger the unreachable-overlay error even- with empty ``activity_queues``.- """- _make_temporal_config(- activity_queues={},- queue_options={"default_q": QueueOptions(start_to_close_timeout=timedelta(minutes=5))},- )diff --git a/tests/unit/pipelex/temporal/test_temporal_connect.py b/tests/unit/pipelex/temporal/test_temporal_connect.pydeleted file mode 100644index d136452a4..000000000--- a/tests/unit/pipelex/temporal/test_temporal_connect.py+++ /dev/null@@ -1,159 +0,0 @@-"""Unit tests for temporal_connect — API-key resolution, TLS wiring, codec wiring, server selection."""--from __future__ import annotations--from typing import TYPE_CHECKING, Any--import pytest--from pipelex.temporal import temporal_connect-from pipelex.temporal.config_temporal import SecretMethod, TemporalServerConfig-from pipelex.temporal.exceptions import TemporalConfigError, TemporalServerError--if TYPE_CHECKING:- from pytest_mock import MockerFixture--TEST_NAMESPACE = "test-ns"-TEST_TARGET_HOST = "temporal.example.com:7233"---def _make_server_config(api_key_method: SecretMethod, api_key_id: str = "") -> TemporalServerConfig:- return TemporalServerConfig(- description="Test server",- target_host=TEST_TARGET_HOST,- namespace=TEST_NAMESPACE,- api_key_method=api_key_method,- api_key_id=api_key_id,- )--[email protected](loop_scope="class")-class TestTemporalConnect:- @pytest.fixture- def connect_mocks(self, mocker: MockerFixture) -> dict[str, Any]:- """Stub the config, the Temporal SDK client, and the secret/env resolvers."""- config = mocker.MagicMock()- config.temporal.payload_codec_config.is_enabled = False- client_cls = mocker.patch.object(temporal_connect, "TemporalClient")- client_cls.connect = mocker.AsyncMock(return_value=mocker.sentinel.client)- return {- "config": config,- "get_config": mocker.patch.object(temporal_connect, "get_config", return_value=config),- "connect": client_cls.connect,- "get_required_env": mocker.patch.object(temporal_connect, "get_required_env", return_value="env-api-key"),- "get_secret": mocker.patch.object(temporal_connect, "get_secret", return_value="provider-api-key"),- }-- async def test_no_api_key_connects_without_tls(self, connect_mocks: dict[str, Any]) -> None:- """api_key_method=none → plain connection: no key, no TLS, no RPC metadata, default converter."""- server_config = _make_server_config(SecretMethod.NONE)-- client = await temporal_connect.connect_to_temporal_server(server_config=server_config)-- assert client is connect_mocks["connect"].return_value- connect_kwargs = connect_mocks["connect"].call_args.kwargs- assert connect_kwargs["target_host"] == TEST_TARGET_HOST- assert connect_kwargs["namespace"] == TEST_NAMESPACE- assert connect_kwargs["api_key"] is None- assert connect_kwargs["tls"] is False- assert connect_kwargs["rpc_metadata"] == {}- assert connect_kwargs["data_converter"] is temporal_connect.data_converter- connect_mocks["get_required_env"].assert_not_called()- connect_mocks["get_secret"].assert_not_called()-- async def test_env_var_api_key_enables_tls_and_namespace_metadata(self, connect_mocks: dict[str, Any]) -> None:- """api_key_method=env_var resolves the key from the env and switches on TLS + namespace metadata."""- server_config = _make_server_config(SecretMethod.ENV_VAR, api_key_id="TEMPORAL_API_KEY")-- await temporal_connect.connect_to_temporal_server(server_config=server_config)-- connect_mocks["get_required_env"].assert_called_once_with("TEMPORAL_API_KEY")- connect_kwargs = connect_mocks["connect"].call_args.kwargs- assert connect_kwargs["api_key"] == "env-api-key"- assert connect_kwargs["tls"] is True- assert connect_kwargs["rpc_metadata"] == {"temporal-namespace": TEST_NAMESPACE}-- async def test_secret_provider_api_key_resolved_via_get_secret(self, connect_mocks: dict[str, Any]) -> None:- """api_key_method=secret_provider resolves the key from the secret provider."""- server_config = _make_server_config(SecretMethod.SECRET_PROVIDER, api_key_id="temporal-key-id")-- await temporal_connect.connect_to_temporal_server(server_config=server_config)-- connect_mocks["get_secret"].assert_called_once_with(secret_id="temporal-key-id")- connect_kwargs = connect_mocks["connect"].call_args.kwargs- assert connect_kwargs["api_key"] == "provider-api-key"- assert connect_kwargs["tls"] is True-- @pytest.mark.parametrize("api_key_method", [SecretMethod.ENV_VAR, SecretMethod.SECRET_PROVIDER])- async def test_missing_api_key_id_raises_config_error(- self,- connect_mocks: dict[str, Any],- api_key_method: SecretMethod,- ) -> None:- """A key-bearing method without an api_key_id is a config error, raised before any connect attempt."""- server_config = _make_server_config(api_key_method, api_key_id="")-- with pytest.raises(TemporalConfigError, match="api_key_id is required"):- await temporal_connect.connect_to_temporal_server(server_config=server_config)-- connect_mocks["connect"].assert_not_called()-- async def test_payload_codec_enabled_builds_custom_converter(- self,- connect_mocks: dict[str, Any],- mocker: MockerFixture,- ) -> None:- """When the payload codec is enabled, the converter is built from the configured codec."""- connect_mocks["config"].temporal.payload_codec_config.is_enabled = True- make_codec = mocker.patch.object(temporal_connect, "make_codec_from_config", return_value=mocker.sentinel.codec)- make_converter = mocker.patch.object(temporal_connect, "make_data_converter", return_value=mocker.sentinel.converter)-- await temporal_connect.connect_to_temporal_server(server_config=_make_server_config(SecretMethod.NONE))-- make_codec.assert_called_once_with()- make_converter.assert_called_once_with(payload_codec=mocker.sentinel.codec)- connect_kwargs = connect_mocks["connect"].call_args.kwargs- assert connect_kwargs["data_converter"] is mocker.sentinel.converter-- async def test_sdk_runtime_error_wrapped_in_server_error(self, connect_mocks: dict[str, Any]) -> None:- """A RuntimeError from the SDK connect is wrapped in TemporalServerError with the server description."""- connect_mocks["connect"].side_effect = RuntimeError("connection refused")- server_config = _make_server_config(SecretMethod.NONE)-- with pytest.raises(TemporalServerError, match="Failed to connect to Temporal server") as exc_info:- await temporal_connect.connect_to_temporal_server(server_config=server_config)-- assert server_config.full_description in str(exc_info.value)- assert "connection refused" in str(exc_info.value)-- async def test_selected_server_connects_with_named_config(self, connect_mocks: dict[str, Any]) -> None:- """connect_to_temporal_selected_server looks up the named config and connects with it."""- server_config = _make_server_config(SecretMethod.NONE)- connect_mocks["config"].temporal.temporal_config.temporal_server_configs = {"local": server_config}-- client = await temporal_connect.connect_to_temporal_selected_server(selected_server_config="local")-- assert client is connect_mocks["connect"].return_value- connect_kwargs = connect_mocks["connect"].call_args.kwargs- assert connect_kwargs["target_host"] == TEST_TARGET_HOST-- async def test_selected_server_unknown_name_raises_config_error(self, connect_mocks: dict[str, Any]) -> None:- """An unknown server-config name is a config error, raised before any connect attempt."""- connect_mocks["config"].temporal.temporal_config.temporal_server_configs = {"local": _make_server_config(SecretMethod.NONE)}-- with pytest.raises(TemporalConfigError, match="Server config not found for selected server: 'staging'"):- await temporal_connect.connect_to_temporal_selected_server(selected_server_config="staging")-- connect_mocks["connect"].assert_not_called()-- async def test_connect_to_temporal_uses_configured_selected_server(self, connect_mocks: dict[str, Any]) -> None:- """connect_to_temporal picks the server named by temporal_config.selected_server."""- server_config = _make_server_config(SecretMethod.NONE)- connect_mocks["config"].temporal.temporal_config.selected_server = "chosen"- connect_mocks["config"].temporal.temporal_config.temporal_server_configs = {"chosen": server_config}-- client = await temporal_connect.connect_to_temporal()-- assert client is connect_mocks["connect"].return_value- connect_kwargs = connect_mocks["connect"].call_args.kwargs- assert connect_kwargs["namespace"] == TEST_NAMESPACEdiff --git a/tests/unit/pipelex/temporal/test_temporal_error_bridge.py b/tests/unit/pipelex/temporal/test_temporal_error_bridge.pydeleted file mode 100644index e80bde467..000000000--- a/tests/unit/pipelex/temporal/test_temporal_error_bridge.py+++ /dev/null@@ -1,305 +0,0 @@-"""Unit tests for the Temporal error bridge (``TemporalError``).--Covers the category-aware retry decision and the ``ErrorReport`` details-payload that survives the activity → workflow boundary:--- ``from_message_exception`` derives ``non_retryable`` from- ``InferenceErrorCategory.is_retryable`` for a ``CogtError`` carrying a category- — including one wrapped under non-``CogtError`` exceptions — and falls back to- the class-name list otherwise.-- ``to_error_report().to_dict()`` is packed into ``ApplicationError.details``- and round-trips through Temporal's failure serialization intact.-- Log severity (critical / error) matches the retry decision on both the- ``from_message_exception`` and ``from_app_error`` paths.-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture-from temporalio.api.failure.v1 import Failure-from temporalio.converter import default as default_converter-from temporalio.exceptions import ApplicationError--from pipelex.base_exceptions import PipelexError-from pipelex.cogt.exceptions import CogtError, InferenceErrorCategory-from pipelex.cogt.inference.error_classification import UserAction, UserActionKind-from pipelex.temporal.tprl.temporal_error import TemporalError---class ModelNotFoundError(PipelexError):- """Non-``CogtError`` ``PipelexError`` whose name is in the default non-retryable list."""---class UnlistedRuntimeError(PipelexError):- """Non-``CogtError`` ``PipelexError`` whose name is absent from the non-retryable list."""---class PipeWrapperError(PipelexError):- """Non-``CogtError`` ``PipelexError`` wrapper — stands in for the real pipe-execution- wrappers (``PipeRunError`` / ``PipeRouterError`` / ``PipelineExecutionError``), none of- which are ``CogtError`` subclasses, so a retry decision cannot read a category off them.- """---def _round_trip_application_error(exc: ApplicationError) -> ApplicationError:- """Serialize an ``ApplicationError`` to a Temporal ``Failure`` and back.-- Mirrors what Temporal does when an activity raises across the- activity → workflow boundary, so the test exercises real payload conversion- rather than an in-process shortcut.- """- converter = default_converter()- failure = Failure()- converter.failure_converter.to_failure(exc, converter.payload_converter, failure)- recovered = converter.failure_converter.from_failure(failure, converter.payload_converter)- assert isinstance(recovered, ApplicationError)- return recovered---def _chain(outermost: PipelexError, *causes: BaseException) -> PipelexError:- """Link exceptions into a ``__cause__`` chain, outermost first, and return ``outermost``.-- ``_chain(a, b, c)`` sets ``a.__cause__ = b`` and ``b.__cause__ = c`` — the chain that- ``raise a from b`` / ``raise b from c`` would build inside nested ``except`` blocks.- """- current: BaseException = outermost- for cause in causes:- current.__cause__ = cause- current = cause- return outermost---class TestTemporalErrorBridge:- @pytest.fixture- def log_mocks(self, mocker: MockerFixture) -> tuple[Any, Any]:- """Replace the whole ``_log_critical`` / ``_log_error`` helpers — they- require a live Temporal context.-- Not ``autouse``: ``test_log_helpers_route_to_the_active_temporal_context``- needs the real helpers. Every other test must opt in, since- ``from_message_exception`` / ``from_app_error`` log unconditionally.-- Returns the ``(critical, error)`` mocks so a test can assert which- severity the retry decision routed the log line to.- """- critical_mock = mocker.patch.object(TemporalError, "_log_critical")- error_mock = mocker.patch.object(TemporalError, "_log_error")- return critical_mock, error_mock-- @pytest.mark.parametrize(- ("error_category", "expected_non_retryable"),- [- pytest.param(InferenceErrorCategory.TRANSIENT, False, id="transient-retryable"),- pytest.param(InferenceErrorCategory.CONFIGURATION, True, id="configuration-non-retryable"),- pytest.param(InferenceErrorCategory.CONTENT, True, id="content-non-retryable"),- pytest.param(InferenceErrorCategory.CAPACITY, True, id="capacity-non-retryable"),- pytest.param(InferenceErrorCategory.AMBIGUOUS, True, id="ambiguous-non-retryable"),- pytest.param(InferenceErrorCategory.UNKNOWN, True, id="unknown-non-retryable"),- ],- )- def test_cogt_error_category_drives_retryability(- self,- log_mocks: tuple[Any, Any],- error_category: InferenceErrorCategory,- expected_non_retryable: bool,- ) -> None:- """A ``CogtError`` with a category sets ``non_retryable = not is_retryable``."""- critical_mock, error_mock = log_mocks-- exc = CogtError("boom", error_category=error_category)- temporal_error = TemporalError.from_message_exception(exc=exc)-- assert temporal_error.non_retryable is expected_non_retryable- assert temporal_error.error_report is not None- assert temporal_error.error_report["error_category"] == error_category- assert temporal_error.error_report["retryable"] is (not expected_non_retryable)-- if expected_non_retryable:- assert critical_mock.call_count == 1- assert error_mock.call_count == 0- else:- assert error_mock.call_count == 1- assert critical_mock.call_count == 0-- @pytest.mark.parametrize(- ("exc", "expected_non_retryable"),- [- pytest.param(ModelNotFoundError("missing model"), True, id="listed-name-non-retryable"),- pytest.param(UnlistedRuntimeError("hiccup"), False, id="unlisted-name-retryable"),- ],- )- def test_non_cogt_pipelex_error_uses_name_list_fallback(- self,- log_mocks: tuple[Any, Any],- exc: PipelexError,- expected_non_retryable: bool,- ) -> None:- """A non-``CogtError`` ``PipelexError`` decides retryability by class name."""- _ = log_mocks # silences the bridge log helpers; severity is asserted elsewhere- temporal_error = TemporalError.from_message_exception(exc=exc)-- assert temporal_error.non_retryable is expected_non_retryable- assert temporal_error.error_report is not None- assert temporal_error.error_report["error_type"] == type(exc).__name__-- def test_cogt_error_without_category_falls_back_to_name_list(self, log_mocks: tuple[Any, Any]) -> None:- """A ``CogtError`` raised without a category falls back to the name list — no crash."""- _ = log_mocks # silences the bridge log helpers; severity is asserted elsewhere- exc = CogtError("boom")- assert exc.error_category is None-- temporal_error = TemporalError.from_message_exception(exc=exc)-- # "CogtError" is not in the default non-retryable list → retryable.- assert temporal_error.non_retryable is False- assert temporal_error.error_report is not None- assert temporal_error.error_report["error_type"] == "CogtError"-- def test_force_non_retryable_is_deterministic_and_preserves_report_retryable(- self,- log_mocks: tuple[Any, Any],- mocker: MockerFixture,- ) -> None:- """The inline workflow-side conversion forces ``non_retryable=True``.-- Workflow-inline callers (``WfPipeRouter`` / ``WfPipeRun``) pass- ``force_non_retryable=True`` so an inline domain error fails the workflow- terminally instead of triggering a blunt whole-workflow retry that re-runs- already-completed inline work — retry belongs at the activity boundary. The- forced path must also be config-free: it short-circuits the- ``_is_non_retryable`` decision (the only ``get_config()`` read in the- conversion), so the conversion stays deterministic for Temporal replay. The- report keeps its own ``retryable`` field as informational metadata,- independent of the Temporal retry flag.- """- _ = log_mocks # silences the bridge log helpers; severity is asserted elsewhere- # A genuinely retryable error: a TRANSIENT CogtError reports retryable=True.- exc = CogtError("transient boom", error_category=InferenceErrorCategory.TRANSIENT)-- # Default (activity-side) path still decides retryability normally — retryable here.- default_side = TemporalError.from_message_exception(exc=exc)- assert default_side.non_retryable is False-- # Forced (workflow-inline) path skips the retry-decision logic entirely: no config read.- decision_spy = mocker.spy(TemporalError, "_is_non_retryable")- forced = TemporalError.from_message_exception(exc=exc, force_non_retryable=True)- decision_spy.assert_not_called()-- # The Temporal retry flag is terminal regardless of the error's own retryability ...- assert forced.non_retryable is True- # ... while the report preserves the informational retryable metadata (still True).- assert forced.error_report is not None- assert forced.error_report["retryable"] is True-- @pytest.mark.parametrize(- ("error_category", "expected_non_retryable"),- [- pytest.param(InferenceErrorCategory.TRANSIENT, False, id="transient-retryable"),- pytest.param(InferenceErrorCategory.CONFIGURATION, True, id="configuration-non-retryable"),- pytest.param(InferenceErrorCategory.CONTENT, True, id="content-non-retryable"),- pytest.param(InferenceErrorCategory.CAPACITY, True, id="capacity-non-retryable"),- pytest.param(InferenceErrorCategory.AMBIGUOUS, True, id="ambiguous-non-retryable"),- pytest.param(InferenceErrorCategory.UNKNOWN, True, id="unknown-non-retryable"),- ],- )- def test_wrapped_cogt_error_category_drives_retryability(- self,- log_mocks: tuple[Any, Any],- error_category: InferenceErrorCategory,- expected_non_retryable: bool,- ) -> None:- """A categorized ``CogtError`` buried under non-``CogtError`` wrappers still drives- the retry decision — the bridge walks the ``__cause__`` chain, not just the outer- exception.-- Mirrors the production chain ``PipelineExecutionError`` -> ``PipeRouterError`` ->- ``PipeRunError`` -> ``CogtError``: the outer exception is not a ``CogtError``, so a- decision based on the outer type alone would wrongly fall back to the name list and- leave a non-retryable inference failure (content, configuration, capacity) retryable.- """- _ = log_mocks # silences the bridge log helpers; severity is asserted elsewhere-- wrapped = _chain(- PipeWrapperError("pipeline run failed"),- PipeWrapperError("pipe failed"),- CogtError("inference boom", error_category=error_category),- )-- temporal_error = TemporalError.from_message_exception(exc=wrapped)-- assert temporal_error.non_retryable is expected_non_retryable- assert temporal_error.error_report is not None- # The outer wrapper keeps its own error_type; the category is recovered from the cause.- assert temporal_error.error_report["error_type"] == "PipeWrapperError"- assert temporal_error.error_report["error_category"] == error_category- # non_retryable must agree with the chain-enriched `retryable` shipped alongside it.- assert temporal_error.error_report["retryable"] is (not expected_non_retryable)-- def test_error_report_round_trips_through_temporal_serialization(- self,- log_mocks: tuple[Any, Any],- ) -> None:- """The ``ErrorReport`` details payload survives the activity → workflow boundary."""- critical_mock, error_mock = log_mocks-- exc = CogtError(- "rate limited",- error_category=InferenceErrorCategory.CAPACITY,- user_action=UserAction(kind=UserActionKind.CHECK_BILLING, detail="check your billing page"),- )- activity_side = TemporalError.from_message_exception(exc=exc)-- serialized: ApplicationError = _round_trip_application_error(exc=activity_side)- workflow_side = TemporalError.from_app_error(exc=serialized)-- assert workflow_side.non_retryable is True- report: dict[str, Any] | None = workflow_side.error_report- assert report is not None- assert report["error_type"] == "CogtError"- assert report["message"] == "rate limited"- assert report["error_category"] == InferenceErrorCategory.CAPACITY- assert report["retryable"] is False- assert report["user_action"]["kind"] == UserActionKind.CHECK_BILLING- assert report["user_action"]["detail"] == "check your billing page"-- # CAPACITY is non-retryable → both bridge hops log at critical severity.- assert critical_mock.call_count == 2- assert error_mock.call_count == 0-- @pytest.mark.parametrize(- "in_activity",- [- pytest.param(True, id="in-activity-uses-activity-log"),- pytest.param(False, id="not-in-activity-uses-workflow-log"),- ],- )- def test_log_helpers_route_to_the_active_temporal_context(- self,- mocker: MockerFixture,- in_activity: bool,- ) -> None:- """``_log_critical`` / ``_log_error`` route through ``activity_log`` inside an- activity and ``workflow_log`` otherwise.-- ``from_message_exception`` runs activity-side and ``from_app_error``- workflow-side; ``workflow.logger`` raises ``_NotInWorkflowEventLoopError``- outside a workflow event loop, so the logger must follow the context.- """- mocker.patch("pipelex.temporal.tprl.temporal_error.activity.in_activity", return_value=in_activity)- activity_log_mock = mocker.patch("pipelex.temporal.tprl.temporal_error.activity_log")- workflow_log_mock = mocker.patch("pipelex.temporal.tprl.temporal_error.workflow_log")-- TemporalError._log_critical("non-retryable message") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- TemporalError._log_error("retryable message") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]-- if in_activity:- activity_log_mock.critical.assert_called_once_with("non-retryable message")- activity_log_mock.error.assert_called_once_with("retryable message")- workflow_log_mock.critical.assert_not_called()- workflow_log_mock.error.assert_not_called()- else:- workflow_log_mock.critical.assert_called_once_with("non-retryable message")- workflow_log_mock.error.assert_called_once_with("retryable message")- activity_log_mock.critical.assert_not_called()- activity_log_mock.error.assert_not_called()diff --git a/tests/unit/pipelex/temporal/test_worker_cli.py b/tests/unit/pipelex/temporal/test_worker_cli.pydeleted file mode 100644index c4624d114..000000000--- a/tests/unit/pipelex/temporal/test_worker_cli.py+++ /dev/null@@ -1,223 +0,0 @@-"""Unit tests for the Temporal worker CLI entry point — project resolution, fast-fail queue check, library load, arg wiring."""--from __future__ import annotations--from typing import TYPE_CHECKING, Any--import pytest-from typer.testing import CliRunner--from pipelex.system.runtime import RunMode-from pipelex.temporal import worker_cli-from pipelex.temporal.exceptions import WorkerTaskQueueUnknownError--if TYPE_CHECKING:- from pytest_mock import MockerFixture--DEFAULT_TASK_QUEUE = "default_queue"-PYPROJECT_WITH_PROJECT_NAME = {"project": {"name": "my-project"}}---class TestWorkerCli:- @pytest.fixture- def boot_mocks(self, mocker: MockerFixture) -> dict[str, Any]:- """Stub everything configure() touches: pyproject load, Pipelex boot, config, hub library getters, and the worker loop."""- config = mocker.MagicMock()- config.temporal.worker_config.default_task_queue = DEFAULT_TASK_QUEUE- config.temporal.is_enabled = True- library_manager = mocker.MagicMock()- return {- "config": config,- "load_toml": mocker.patch.object(worker_cli, "load_toml_from_path", return_value=PYPROJECT_WITH_PROJECT_NAME),- "pipelex_make": mocker.patch.object(worker_cli.Pipelex, "make"),- "get_config": mocker.patch.object(worker_cli, "get_config", return_value=config),- "set_run_mode": mocker.patch.object(type(worker_cli.runtime_manager), "set_run_mode"),- "library_manager": library_manager,- "get_library_manager": mocker.patch("pipelex.hub.get_library_manager", return_value=library_manager),- "set_current_library": mocker.patch("pipelex.hub.set_current_library"),- "resolve_library_dirs": mocker.patch("pipelex.hub.resolve_library_dirs", return_value=([], "PIPELEXPATH")),- "run_worker": mocker.patch.object(worker_cli, "run_worker", new=mocker.AsyncMock(return_value=None)),- }-- @pytest.mark.parametrize("project", [None, "chosen-project"])- @pytest.mark.asyncio- async def test_run_worker_forwards_kwargs_to_task_manager(self, mocker: MockerFixture, project: str | None) -> None:- """run_worker hands all worker options to the task manager, for both implicit and explicit project."""- task_manager = mocker.MagicMock()- task_manager.run_worker = mocker.AsyncMock(return_value=None)- mocker.patch.object(worker_cli, "get_task_manager", return_value=task_manager)-- await worker_cli.run_worker(- project=project,- is_not_sandboxed=True,- is_unit_testing=False,- task_queue="some_queue",- scope_name="router",- profile_name="anthropic-tier4",- )-- task_manager.run_worker.assert_awaited_once_with(- is_not_sandboxed=True,- is_unit_testing=False,- task_queue="some_queue",- scope_name="router",- profile_name="anthropic-tier4",- )-- def test_explicit_project_skips_pyproject(self, boot_mocks: dict[str, Any]) -> None:- """An explicit project argument bypasses pyproject.toml entirely."""- worker_cli.configure(project="explicit-project")-- boot_mocks["load_toml"].assert_not_called()- boot_mocks["pipelex_make"].assert_called_once_with(temporal_enabled=True)- assert boot_mocks["run_worker"].call_args.kwargs["project"] == "explicit-project"-- def test_project_resolved_from_pyproject_project_name(self, boot_mocks: dict[str, Any]) -> None:- """With no argument, the project name comes from [project].name in pyproject.toml."""- worker_cli.configure()-- boot_mocks["load_toml"].assert_called_once_with(path="pyproject.toml")- assert boot_mocks["run_worker"].call_args.kwargs["project"] == "my-project"-- def test_project_falls_back_to_poetry_name(self, boot_mocks: dict[str, Any]) -> None:- """When [project].name is absent, [tool.poetry].name is used."""- boot_mocks["load_toml"].return_value = {"tool": {"poetry": {"name": "poetry-project"}}}-- worker_cli.configure()-- assert boot_mocks["run_worker"].call_args.kwargs["project"] == "poetry-project"-- def test_missing_project_name_raises(self, boot_mocks: dict[str, Any]) -> None:- """A pyproject.toml without any project name aborts before booting Pipelex."""- boot_mocks["load_toml"].return_value = {"tool": {"poetry": {}}}-- with pytest.raises(ValueError, match=r"Project name not found in pyproject\.toml"):- worker_cli.configure()-- boot_mocks["pipelex_make"].assert_not_called()-- def test_unit_testing_flag_sets_run_mode(self, boot_mocks: dict[str, Any]) -> None:- """--is-unit-testing switches the runtime manager to UNIT_TEST mode."""- worker_cli.configure(is_unit_testing=True)-- boot_mocks["set_run_mode"].assert_called_once_with(RunMode.UNIT_TEST)-- def test_no_unit_testing_flag_leaves_run_mode_alone(self, boot_mocks: dict[str, Any]) -> None:- """Without the flag, the run mode is never touched."""- worker_cli.configure()-- boot_mocks["set_run_mode"].assert_not_called()-- @pytest.mark.parametrize(- ("topic", "task_queue", "expected_validated_queue"),- [- ("explicit_queue", "custom_queue", "custom_queue"),- ("default_queue_fallback", None, DEFAULT_TASK_QUEUE),- ],- )- def test_task_queue_validated_before_library_load(- self,- boot_mocks: dict[str, Any],- mocker: MockerFixture,- topic: str,- task_queue: str | None,- expected_validated_queue: str,- ) -> None:- """The effective task queue (explicit or config default) is validated against known queues."""- worker_cli.configure(task_queue=task_queue)-- validate_mock = boot_mocks["config"].temporal.validate_task_queue_known- assert validate_mock.call_args_list == [mocker.call(expected_validated_queue)], f"Wrong validated queue for {topic}"-- def test_unknown_task_queue_fast_fails_before_library_load(self, boot_mocks: dict[str, Any]) -> None:- """A typo'd task queue aborts before any library loading or worker start."""- boot_mocks["config"].temporal.validate_task_queue_known.side_effect = WorkerTaskQueueUnknownError("unknown queue 'tyop'")-- with pytest.raises(WorkerTaskQueueUnknownError, match="unknown queue 'tyop'"):- worker_cli.configure(task_queue="tyop")-- boot_mocks["get_library_manager"].assert_not_called()- boot_mocks["run_worker"].assert_not_called()-- def test_library_loaded_from_resolved_dirs(self, boot_mocks: dict[str, Any]) -> None:- """Resolved library dirs are loaded into the worker_base library."""- boot_mocks["resolve_library_dirs"].return_value = (["libs_a", "libs_b"], "PIPELEXPATH")-- worker_cli.configure()-- boot_mocks["library_manager"].open_library.assert_called_once_with(library_id="worker_base")- boot_mocks["set_current_library"].assert_called_once_with(library_id="worker_base")- boot_mocks["resolve_library_dirs"].assert_called_once_with(library_dirs=None)- boot_mocks["library_manager"].load_libraries.assert_called_once_with(library_id="worker_base", library_dirs=["libs_a", "libs_b"])-- def test_no_library_dirs_skips_load(self, boot_mocks: dict[str, Any]) -> None:- """With no resolved dirs (PIPELEXPATH unset), the library load is skipped but the library is still opened."""- worker_cli.configure()-- boot_mocks["library_manager"].open_library.assert_called_once_with(library_id="worker_base")- boot_mocks["library_manager"].load_libraries.assert_not_called()-- def test_temporal_disabled_in_config_is_forced_on(self, boot_mocks: dict[str, Any], mocker: MockerFixture) -> None:- """When temporal.is_enabled is false, worker mode forces it on via a config copy."""- temporal_mock = boot_mocks["config"].temporal- temporal_mock.is_enabled = False- temporal_mock.model_copy.return_value = mocker.sentinel.updated_temporal-- worker_cli.configure()-- temporal_mock.model_copy.assert_called_once_with(update={"is_enabled": True})- assert boot_mocks["config"].temporal is mocker.sentinel.updated_temporal-- def test_temporal_enabled_in_config_is_left_alone(self, boot_mocks: dict[str, Any]) -> None:- """When temporal.is_enabled is already true, the config is not copied or reassigned."""- temporal_mock = boot_mocks["config"].temporal-- worker_cli.configure()-- temporal_mock.model_copy.assert_not_called()- assert boot_mocks["config"].temporal is temporal_mock-- def test_cli_arg_wiring_reaches_run_worker(self, boot_mocks: dict[str, Any]) -> None:- """All CLI options travel through Typer parsing into run_worker, with the correct keyword wiring."""- runner = CliRunner()-- result = runner.invoke(- worker_cli.app,- [- "cli-project",- "--is-not-sandboxed",- "--is-unit-testing",- "--task-queue",- "queue_from_cli",- "--scope",- "router",- "--profile",- "anthropic-tier4",- ],- )-- assert result.exit_code == 0, result.output- boot_mocks["run_worker"].assert_called_once_with(- project="cli-project",- is_not_sandboxed=True,- is_unit_testing=True,- task_queue="queue_from_cli",- scope_name="router",- profile_name="anthropic-tier4",- )-- def test_cli_defaults_reach_run_worker(self, boot_mocks: dict[str, Any]) -> None:- """Invoking with no arguments resolves the project from pyproject and passes default options."""- runner = CliRunner()-- result = runner.invoke(worker_cli.app, [])-- assert result.exit_code == 0, result.output- boot_mocks["run_worker"].assert_called_once_with(- project="my-project",- is_not_sandboxed=False,- is_unit_testing=False,- task_queue=None,- scope_name=None,- profile_name=None,- )diff --git a/tests/unit/pipelex/temporal/test_worker_config_resolve_queue.py b/tests/unit/pipelex/temporal/test_worker_config_resolve_queue.pydeleted file mode 100644index 83aa8fadf..000000000--- a/tests/unit/pipelex/temporal/test_worker_config_resolve_queue.py+++ /dev/null@@ -1,148 +0,0 @@-"""Unit tests for ``WorkerConfig.resolve_queue`` — the per-activity, per-handle-routing resolver introduced in v1, with the hybrid empty-routing fallback-added in v2.--Covers the resolution layers:- 0. Empty ``activity_queues`` (no routing configured) → ``None``, signalling- the dispatch path to omit ``task_queue`` so Temporal uses the workflow's- own queue (supports ``with_conditional_worker`` test isolation).- 1. Non-empty ``activity_queues``, unmapped activity → worker-wide- ``default_task_queue``.- 2. Mapped activity, unmapped handle (or ``routing_key=None``) → activity ``default``.- 3. Mapped activity, mapped handle → per-handle queue.-"""--from datetime import timedelta--import pytest--from pipelex.temporal.config_temporal import ActivityRouteConfig, RetryPolicyConfig, WorkerConfig---def _make_worker_config(activity_queues: dict[str, ActivityRouteConfig] | None = None) -> WorkerConfig:- """Build a minimal ``WorkerConfig`` for resolver tests.-- Only the fields read by ``resolve_queue`` matter here; the rest are filled- with placeholder values that satisfy the schema.- """- return WorkerConfig(- default_task_queue="default_q",- activity_queues=activity_queues or {},- workflow_execution_timeout=timedelta(hours=1),- default_activity_start_to_close_timeout=timedelta(minutes=10),- retry_policy_config=RetryPolicyConfig(- initial_interval=timedelta(seconds=3),- backoff_coefficient=2.0,- maximum_interval="unlimited",- maximum_attempts=3,- non_retryable_error_types=[],- ),- )---class TestWorkerConfigResolveQueue:- """Hybrid empty-routing fallback + three-layer resolution when routing is configured."""-- def test_empty_activity_queues_returns_none_for_workflow_local_dispatch(self) -> None:- """When ``activity_queues`` is fully empty (default config, no routing- configured), ``resolve_queue`` returns ``None`` so the dispatch path- omits ``task_queue`` and Temporal routes the activity to the workflow's- own queue. Required by ``with_conditional_worker`` and by the pre-v1- default behavior.- """- worker_config = _make_worker_config()- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="claude-opus-4-7") is None- assert worker_config.resolve_queue("act_some_unknown_activity") is None-- def test_unmapped_activity_with_non_empty_routing_falls_back_to_default_task_queue(self) -> None:- """When the operator has configured routing for some activity, an- unmapped activity still falls back explicitly to ``default_task_queue``.- The empty-vs-non-empty distinction is the hybrid fallback hinge.- """- worker_config = _make_worker_config(- {- "act_img_gen_images": ActivityRouteConfig(default="img_q"),- },- )- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="claude-opus-4-7") == "default_q"- assert worker_config.resolve_queue("act_some_unknown_activity") == "default_q"-- def test_mapped_activity_unmapped_handle_falls_back_to_activity_default(self) -> None:- """When the activity is mapped but the handle does not match any- ``by_handle`` entry, the activity-level ``default`` queue applies.- """- worker_config = _make_worker_config(- {- "act_llm_gen_text": ActivityRouteConfig(- default="inference_q",- by_handle={"claude-opus-4-7": "anthropic_q"},- ),- },- )- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="gpt-5") == "inference_q"-- def test_mapped_activity_none_routing_key_uses_activity_default(self) -> None:- """When the caller passes ``routing_key=None`` (non-handle activities- like jinja2 or render-page-views), the activity-level ``default`` is- used — ``by_handle`` is never consulted.- """- worker_config = _make_worker_config(- {- "act_render_page_views": ActivityRouteConfig(- default="render_q",- by_handle={"some-handle": "should_not_be_used_q"},- ),- },- )- assert worker_config.resolve_queue("act_render_page_views", routing_key=None) == "render_q"-- def test_mapped_activity_and_handle_uses_per_handle_queue(self) -> None:- """When both the activity and the handle are mapped, the per-handle- queue wins over the activity default.- """- worker_config = _make_worker_config(- {- "act_llm_gen_text": ActivityRouteConfig(- default="inference_q",- by_handle={- "claude-opus-4-7": "anthropic_q",- "gpt-5": "openai_q",- },- ),- },- )- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="claude-opus-4-7") == "anthropic_q"- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="gpt-5") == "openai_q"-- @pytest.mark.parametrize(- ("activity_name", "routing_key", "expected_queue"),- [- # Image generation routed by handle (e.g. provider grouping).- ("act_img_gen_images", "flux-1.1-pro", "fal_q"),- ("act_img_gen_images", "dall-e-3", "openai_image_q"),- ("act_img_gen_images", "unknown-model", "image_gen_q"),- # Extract routed by backend handle.- ("act_extract_gen_extract_pages", "mistral-ocr", "mistral_extract_q"),- ("act_extract_gen_extract_pages", "azure-ocr", "extract_q"),- # Non-mapped activity stays on the default.- ("act_jinja2_gen_text", None, "default_q"),- ],- )- def test_realistic_routing_table_matrix(self, activity_name: str, routing_key: str | None, expected_queue: str) -> None:- """Exercise a representative routing table with multiple activities and handles."""- worker_config = _make_worker_config(- {- "act_img_gen_images": ActivityRouteConfig(- default="image_gen_q",- by_handle={- "flux-1.1-pro": "fal_q",- "dall-e-3": "openai_image_q",- },- ),- "act_extract_gen_extract_pages": ActivityRouteConfig(- default="extract_q",- by_handle={"mistral-ocr": "mistral_extract_q"},- ),- },- )- assert worker_config.resolve_queue(activity_name, routing_key=routing_key) == expected_queuediff --git a/tests/unit/pipelex/temporal/test_worker_config_toml_parsing.py b/tests/unit/pipelex/temporal/test_worker_config_toml_parsing.pydeleted file mode 100644index 145b0346c..000000000--- a/tests/unit/pipelex/temporal/test_worker_config_toml_parsing.py+++ /dev/null@@ -1,135 +0,0 @@-"""Unit test for ``activity_queues`` TOML parsing into ``WorkerConfig``.--The resolver tests in ``test_worker_config_resolve_queue.py`` construct-``WorkerConfig`` directly from Python kwargs — that does NOT exercise the-config-loading boundary (Pydantic parsing of a TOML-shaped dict). This test-closes that gap: it loads a TOML string with a representative-``activity_queues`` table and asserts that:-- - ``activity_queues`` materializes as ``dict[str, ActivityRouteConfig]``.- - Per-activity ``default`` and per-handle ``by_handle`` survive parsing.- - ``resolve_queue`` on the parsed config returns the expected queues.--This is the regression guard for the commented-out examples shipped in-``pipelex/pipelex.toml`` and ``pipelex/kit/configs/pipelex.toml``: if a-schema change ever breaks how operators are invited to override routing-in their projects, this test fails before the broken config reaches them.-"""--from datetime import timedelta--import tomli--from pipelex.temporal.config_temporal import ActivityRouteConfig, HandleOptions, RetryPolicyConfig, RetryPolicyConfigOverlay, WorkerConfig--_TOML_FRAGMENT = """-default_task_queue = "temporal_task_queue"-workflow_execution_timeout = "1:00:00"-run_timeout = "1:00:00"-task_timeout = "0:00:10"-start_delay = "0:00:00"-rpc_timeout = "1:00:00"-default_activity_start_to_close_timeout = "0:10:00"--[activity_queues.act_llm_gen_text]-default = "inference_q"-by_handle = { "claude-opus-4-7" = "anthropic_q", "gpt-5" = "openai_q" }--[activity_queues.act_llm_gen_text.handle_options."claude-opus-4-7-1m"]-start_to_close_timeout = "0:25:00"--[activity_queues.act_img_gen_images]-default = "image_gen_q"-by_handle = { "flux-1.1-pro" = "fal_q" }--[activity_queues.act_extract_gen_extract_pages]-default = "extract_q"-by_handle = {}--[retry_policy_config]-initial_interval = "0:00:03"-backoff_coefficient = 2.0-maximum_interval = "unlimited"-maximum_attempts = 3-non_retryable_error_types = []-"""---class TestWorkerConfigTomlParsing:- """Parse a TOML fragment shaped like the project override and validate- that the new per-activity routing keys (including ``handle_options``) round-trip into ``WorkerConfig``.- """-- def test_activity_queues_parse_into_typed_models(self) -> None:- worker_config = WorkerConfig.model_validate(tomli.loads(_TOML_FRAGMENT))-- assert worker_config.default_task_queue == "temporal_task_queue"- assert set(worker_config.activity_queues.keys()) == {- "act_llm_gen_text",- "act_img_gen_images",- "act_extract_gen_extract_pages",- }- for route in worker_config.activity_queues.values():- assert isinstance(route, ActivityRouteConfig)-- llm_route = worker_config.activity_queues["act_llm_gen_text"]- assert llm_route.default == "inference_q"- assert llm_route.by_handle == {"claude-opus-4-7": "anthropic_q", "gpt-5": "openai_q"}- # handle_options round-trip into HandleOptions models.- assert set(llm_route.handle_options.keys()) == {"claude-opus-4-7-1m"}- opus_1m = llm_route.handle_options["claude-opus-4-7-1m"]- assert isinstance(opus_1m, HandleOptions)- assert opus_1m.start_to_close_timeout == timedelta(minutes=25)-- img_route = worker_config.activity_queues["act_img_gen_images"]- assert img_route.default == "image_gen_q"- assert img_route.by_handle == {"flux-1.1-pro": "fal_q"}- # handle_options defaults to an empty dict when omitted from TOML.- assert img_route.handle_options == {}-- extract_route = worker_config.activity_queues["act_extract_gen_extract_pages"]- assert extract_route.default == "extract_q"- assert extract_route.by_handle == {}-- def test_resolve_queue_works_on_parsed_config(self) -> None:- """End-to-end: parse the TOML, then dispatch through ``resolve_queue``."""- worker_config = WorkerConfig.model_validate(tomli.loads(_TOML_FRAGMENT))-- # Layer 3: per-handle override wins.- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="claude-opus-4-7") == "anthropic_q"- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="gpt-5") == "openai_q"- # Layer 2: unmapped handle falls back to the activity default.- assert worker_config.resolve_queue("act_llm_gen_text", routing_key="some-other-model") == "inference_q"- # Layer 2 again: routing_key=None on a mapped activity uses its default- # without ever consulting by_handle.- assert worker_config.resolve_queue("act_extract_gen_extract_pages", routing_key=None) == "extract_q"- # Layer 1: completely unmapped activity falls back to the worker default.- assert worker_config.resolve_queue("act_jinja2_gen_text") == "temporal_task_queue"-- def test_retry_policy_finite_maximum_interval_from_toml(self) -> None:- """Finite TOML duration on ``maximum_interval`` must round-trip under- ``ConfigModel``'s ``strict=True``. Regression for the missing ``strict=False``- on the Annotated arm of the Union — without it, operator overlays like- ``maximum_interval = "0:02:00"`` blocked config load.- """- toml_fragment = """-initial_interval = "0:00:03"-backoff_coefficient = 2.0-maximum_interval = "0:02:00"-maximum_attempts = 3-non_retryable_error_types = []-"""- baseline = RetryPolicyConfig.model_validate(tomli.loads(toml_fragment))- assert baseline.maximum_interval == timedelta(minutes=2)-- # Same on the overlay class — both inherit RetryPolicyConfigBase, so the- # fix at the base must cover both layers.- overlay_fragment = """-initial_interval = "0:00:03"-backoff_coefficient = 2.0-maximum_interval = "0:02:00"-maximum_attempts = 3-non_retryable_error_types_extra = []-"""- overlay = RetryPolicyConfigOverlay.model_validate(tomli.loads(overlay_fragment))- assert overlay.maximum_interval == timedelta(minutes=2)diff --git a/tests/unit/pipelex/temporal/test_worker_runtime_profile.py b/tests/unit/pipelex/temporal/test_worker_runtime_profile.pydeleted file mode 100644index e1402a2da..000000000--- a/tests/unit/pipelex/temporal/test_worker_runtime_profile.py+++ /dev/null@@ -1,72 +0,0 @@-"""Unit tests for ``TemporalTaskManager._resolve_runtime_profile_by_name`` and-the ``WorkerTuningMode`` validator on ``WorkerRuntimeProfile``.--Mirrors the structure of ``test_worker_scope_resolution.py`` — exercises the-named-profile lookup including the unknown-profile error and the explicit-rejection of ``tuning_mode='resource_based'`` until that path ships.-"""--from datetime import timedelta--import pytest-from pydantic import ValidationError--from pipelex.temporal.config_temporal import WorkerRuntimeProfile, WorkerTuningMode-from pipelex.temporal.exceptions import WorkerProfileConfigError-from pipelex.temporal.temporal_task_manager import TemporalTaskManager---class TestWorkerRuntimeProfile:- """Resolution of named runtime profiles + rejection of unimplemented tuning_mode."""-- def test_default_profile_resolves_when_none(self) -> None:- """When ``profile_name=None``, resolution returns the configured ``default_profile``."""- profile = TemporalTaskManager._resolve_runtime_profile_by_name(profile_name=None) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- # Out-of-the-box default ships the values lifted from the pre-v2 Worker(...)- # construction. Test against the most distinctive field rather than the- # exact instance (which would couple to TOML loading).- assert profile.max_concurrent_activities == 1000-- def test_named_profile_resolves(self) -> None:- """Passing an explicit name resolves to that profile from config."""- # `default` is always present in shipping pipelex.toml.- profile = TemporalTaskManager._resolve_runtime_profile_by_name(profile_name="default") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- assert profile.max_concurrent_activities == 1000-- def test_unknown_profile_raises_with_known_names(self) -> None:- """An unknown profile name raises ``WorkerProfileConfigError`` with the- full list of known profiles in the message.- """- with pytest.raises(WorkerProfileConfigError) as exc_info:- TemporalTaskManager._resolve_runtime_profile_by_name(profile_name="nonexistent-profile") # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- message = str(exc_info.value)- assert "nonexistent-profile" in message- # Default profile is always known and must appear in the listing.- assert "default" in message-- def test_resource_based_tuning_mode_rejected(self) -> None:- """``WorkerTuningMode.RESOURCE_BASED`` is reserved for a future iteration;- constructing a profile with that mode must raise immediately with a- clear "not implemented" message.-- Pydantic v2 wraps validator-raised exceptions in ``ValidationError``;- the assertion checks the wrapped message rather than the exception class.- """- with pytest.raises(ValidationError) as exc_info:- WorkerRuntimeProfile(- tuning_mode=WorkerTuningMode.RESOURCE_BASED,- max_cached_workflows=10000,- max_concurrent_workflow_tasks=1000,- max_concurrent_activities=1000,- max_concurrent_local_activities=1000,- max_concurrent_workflow_task_polls=100,- max_concurrent_activity_task_polls=100,- max_activities_per_second=1000,- sticky_queue_schedule_to_start_timeout=timedelta(minutes=30),- max_heartbeat_throttle_interval=timedelta(minutes=60),- default_heartbeat_throttle_interval=timedelta(minutes=60),- graceful_shutdown_timeout=timedelta(minutes=30),- )- message = str(exc_info.value)- assert "resource_based" in message- assert "explicit" in messagediff --git a/tests/unit/pipelex/temporal/test_worker_scope_resolution.py b/tests/unit/pipelex/temporal/test_worker_scope_resolution.pydeleted file mode 100644index 60a596f45..000000000--- a/tests/unit/pipelex/temporal/test_worker_scope_resolution.py+++ /dev/null@@ -1,151 +0,0 @@-from typing import Any--import pytest--from pipelex.temporal.config_temporal import WorkerScope-from pipelex.temporal.exceptions import WorkerScopeConfigError-from pipelex.temporal.temporal_tasks import TaskPack, TemporalTasks---class _WfAlpha:- pass---class _WfBeta:- pass---class _WfGamma:- pass---def _act_one(_payload: Any) -> Any:- return _payload---def _act_two(_payload: Any) -> Any:- return _payload---def _act_three(_payload: Any) -> Any:- return _payload---def _make_tasks() -> TemporalTasks:- tasks = TemporalTasks()- tasks.complement_catalog(- extra_catalog={- "pack_one": TaskPack(workflow_list=[_WfAlpha, _WfBeta], activity_list=[_act_one, _act_two]),- "pack_two": TaskPack(workflow_list=[_WfGamma], activity_list=[_act_three]),- },- extra_workflows=[],- extra_activities=[],- )- return tasks---def _scope(- required_tasks_packs: list[str] | None = None,- required_workflows: list[str] | None = None,- required_activities: list[str] | None = None,- excluded_workflows: list[str] | None = None,- excluded_activities: list[str] | None = None,- disable_all_workflows: bool = False,- disable_all_activities: bool = False,-) -> WorkerScope:- return WorkerScope(- required_tasks_packs=required_tasks_packs or [],- required_workflows=required_workflows or [],- required_activities=required_activities or [],- excluded_workflows=excluded_workflows or [],- excluded_activities=excluded_activities or [],- disable_all_workflows=disable_all_workflows,- disable_all_activities=disable_all_activities,- )---class TestWorkerScopeResolution:- def test_no_scope_returns_full_catalog(self):- tasks = _make_tasks()- workflows, activities = tasks.workflows_and_activities()- assert set(workflows) == {_WfAlpha, _WfBeta, _WfGamma}- assert set(activities) == {_act_one, _act_two, _act_three}-- def test_pack_only_scope(self):- tasks = _make_tasks()- workflows, activities = tasks.workflows_and_activities(scope=_scope(required_tasks_packs=["pack_one"]))- assert set(workflows) == {_WfAlpha, _WfBeta}- assert set(activities) == {_act_one, _act_two}-- def test_required_workflows_and_activities_only(self):- tasks = _make_tasks()- workflows, activities = tasks.workflows_and_activities(- scope=_scope(required_workflows=["_WfGamma"], required_activities=["_act_three"]),- )- assert set(workflows) == {_WfGamma}- assert set(activities) == {_act_three}-- def test_pack_plus_required(self):- tasks = _make_tasks()- workflows, activities = tasks.workflows_and_activities(- scope=_scope(required_tasks_packs=["pack_one"], required_workflows=["_WfGamma"]),- )- assert set(workflows) == {_WfAlpha, _WfBeta, _WfGamma}- assert set(activities) == {_act_one, _act_two}-- def test_exclusions_apply_after_inclusions(self):- tasks = _make_tasks()- workflows, activities = tasks.workflows_and_activities(- scope=_scope(- required_tasks_packs=["pack_one", "pack_two"],- excluded_workflows=["_WfBeta"],- excluded_activities=["_act_one"],- ),- )- assert set(workflows) == {_WfAlpha, _WfGamma}- assert set(activities) == {_act_two, _act_three}-- def test_unknown_pack_raises(self):- tasks = _make_tasks()- with pytest.raises(WorkerScopeConfigError, match="Unknown task pack"):- tasks.workflows_and_activities(scope=_scope(required_tasks_packs=["pack_missing"]))-- def test_unknown_required_workflow_raises(self):- tasks = _make_tasks()- with pytest.raises(WorkerScopeConfigError, match="required_workflows"):- tasks.workflows_and_activities(scope=_scope(required_workflows=["WfUnknown"]))-- def test_unknown_required_activity_raises(self):- tasks = _make_tasks()- with pytest.raises(WorkerScopeConfigError, match="required_activities"):- tasks.workflows_and_activities(scope=_scope(required_activities=["act_unknown"]))-- def test_unknown_excluded_workflow_raises(self):- tasks = _make_tasks()- with pytest.raises(WorkerScopeConfigError, match="excluded_workflows"):- tasks.workflows_and_activities(- scope=_scope(required_tasks_packs=["pack_one"], excluded_workflows=["WfUnknown"]),- )-- def test_unknown_excluded_activity_raises(self):- tasks = _make_tasks()- with pytest.raises(WorkerScopeConfigError, match="excluded_activities"):- tasks.workflows_and_activities(- scope=_scope(required_tasks_packs=["pack_one"], excluded_activities=["act_unknown"]),- )-- def test_empty_resolved_scope_raises(self):- tasks = _make_tasks()- with pytest.raises(WorkerScopeConfigError, match="empty set"):- tasks.workflows_and_activities(scope=_scope())-- def test_full_excluded_to_empty_raises(self):- tasks = _make_tasks()- with pytest.raises(WorkerScopeConfigError, match="empty set"):- tasks.workflows_and_activities(- scope=_scope(- required_tasks_packs=["pack_one", "pack_two"],- excluded_workflows=["_WfAlpha", "_WfBeta", "_WfGamma"],- excluded_activities=["_act_one", "_act_two", "_act_three"],- ),- )diff --git a/tests/unit/pipelex/temporal/test_worker_scopes_specialized.py b/tests/unit/pipelex/temporal/test_worker_scopes_specialized.pydeleted file mode 100644index 193956e4d..000000000--- a/tests/unit/pipelex/temporal/test_worker_scopes_specialized.py+++ /dev/null@@ -1,81 +0,0 @@-"""Unit tests for the specialized worker scopes (runner-llm, runner-img-gen,-runner-extract, runner-jinja2, runner-search) shipped in `pipelex/pipelex.toml`.--These scopes let deployment manifests spin up one worker pool per backend-class. The tests verify (a) each scope registers exactly the activities its-name implies and (b) the union of the specialized scopes covers the-full ``runner`` registration with no orphan activities.-"""--import pytest--from pipelex.config import get_config-from pipelex.temporal.tasks import PackName, Tasks-from pipelex.temporal.temporal_tasks import TemporalTasks---def _make_tasks() -> TemporalTasks:- """Build a TemporalTasks populated from the production ``Tasks.TASK_PACKS``.-- Mirrors what ``TemporalTaskManager.setup`` registers at runtime — that- catalog determines which activities ``required_activities=[...]`` can- reference in a scope.- """- tasks = TemporalTasks()- tasks.complement_catalog(extra_catalog=Tasks.TASK_PACKS, extra_workflows=[], extra_activities=[])- return tasks---class TestSpecializedWorkerScopes:- """Specialized scopes register exactly their intended activity set and- collectively cover the full ``runner`` registration.- """-- @pytest.mark.parametrize(- ("scope_name", "expected_activity_names"),- [- ("runner-llm", {"act_llm_gen_text", "act_llm_gen_object", "act_llm_gen_object_list"}),- # img-gen + extract intentionally both register act_render_page_views.- ("runner-img-gen", {"act_img_gen_images", "act_render_page_views"}),- ("runner-extract", {"act_extract_gen_extract_pages", "act_render_page_views"}),- ("runner-jinja2", {"act_jinja2_gen_text"}),- ("runner-search", {"act_search_gen_sourced_answer", "act_search_gen_structured"}),- ],- )- def test_scope_registers_expected_activities(self, scope_name: str, expected_activity_names: set[str]) -> None:- """Each specialized scope registers exactly the activities its name implies."""- worker_scopes = get_config().temporal.worker_scopes- assert scope_name in worker_scopes.scopes, f"scope '{scope_name}' missing from worker_scopes.scopes"- scope = worker_scopes.scopes[scope_name]- _, activities = _make_tasks().workflows_and_activities(scope=scope)- registered_names = {activity.__name__ for activity in activities}- assert registered_names == expected_activity_names-- def test_specialized_scopes_cover_runner_activities(self) -> None:- """Union of specialized scope activity sets equals the ``runner`` scope- activity set — no orphan activities that would land on no specialized- worker in a manifest using only the specialized scopes.- """- worker_scopes = get_config().temporal.worker_scopes- tasks = _make_tasks()- _, runner_activities = tasks.workflows_and_activities(scope=worker_scopes.scopes["runner"])- runner_names = {activity.__name__ for activity in runner_activities}-- specialized_union: set[str] = set()- for scope_name in ("runner-llm", "runner-img-gen", "runner-extract", "runner-jinja2", "runner-search"):- scope = worker_scopes.scopes[scope_name]- _, scope_activities = tasks.workflows_and_activities(scope=scope)- specialized_union.update(activity.__name__ for activity in scope_activities)-- # Derive the content-generation set from the CRAFTING task pack — the- # single source of truth. The PIPE pack contributes control-plane- # activities (assemble_graph, deliver, flush_trace_events) that are- # wf_pipe_router / wf_pipe_run plumbing, not user content. Specialized- # runner scopes are only meant to cover the user-activity surface, so- # we compare on that subset. Adding a new activity to CRAFTING is now- # automatically tracked here without editing this test.- content_gen_runner_names = {activity.__name__ for activity in Tasks.TASK_PACKS[PackName.CRAFTING].activity_list}- # Every user-activity registered by `runner` must appear in at least- # one specialized scope.- orphans = content_gen_runner_names & runner_names - specialized_union- assert not orphans, f"specialized scopes leave content-gen activities orphan: {orphans!r}"diff --git a/tests/unit/pipelex/temporal/test_worker_task_queue_validation.py b/tests/unit/pipelex/temporal/test_worker_task_queue_validation.pydeleted file mode 100644index faa273d87..000000000--- a/tests/unit/pipelex/temporal/test_worker_task_queue_validation.py+++ /dev/null@@ -1,57 +0,0 @@-"""Unit tests for the strict ``--task-queue`` validation, now exposed as-``Temporal.validate_task_queue_known``.--A worker polling a queue that nothing routes to is almost always a typo, so-the worker CLI must fail fast at startup with a "did you mean?" suggestion-when a close match exists. The check also runs inside-``TemporalTaskManager.run_worker`` so programmatic callers (tests, library-code) benefit too.-"""--import pytest--from pipelex.config import get_config-from pipelex.temporal.exceptions import WorkerTaskQueueUnknownError---class TestWorkerTaskQueueValidation:- """Strict validation: known queues pass, unknown queues raise."""-- def test_default_task_queue_passes(self) -> None:- """The worker's ``default_task_queue`` is always a known queue and must- not raise even when no routing/options entries reference it.- """- # Shipping pipelex.toml uses "temporal_task_queue" as default.- get_config().temporal.validate_task_queue_known("temporal_task_queue")-- def test_unknown_queue_raises_with_known_queues_listed(self) -> None:- """Unknown queue raises ``WorkerTaskQueueUnknownError`` with the full- known-queue list in the message.- """- with pytest.raises(WorkerTaskQueueUnknownError) as exc_info:- get_config().temporal.validate_task_queue_known("totally_unknown_queue")- message = str(exc_info.value)- assert "totally_unknown_queue" in message- assert "Known queues" in message- assert "temporal_task_queue" in message-- def test_typo_close_to_known_queue_suggests_correction(self) -> None:- """A typo within Levenshtein-2 of a known queue includes a "Did you- mean?" suggestion in the error message.- """- # "temporal_task_queu" is one char away from "temporal_task_queue".- with pytest.raises(WorkerTaskQueueUnknownError) as exc_info:- get_config().temporal.validate_task_queue_known("temporal_task_queu")- message = str(exc_info.value)- assert "Did you mean 'temporal_task_queue'?" in message-- def test_queue_options_key_treated_as_known(self) -> None:- """A queue declared only in ``queue_options`` (no routing entry) still- counts as a known queue — workers can legitimately poll a queue that- operators tune via ``queue_options`` even before any ``activity_queues``- entry routes to it.- """- # The shipping default config has queue_options.temporal_task_queue —- # that's already covered by test_default_task_queue_passes. Use it as- # the regression for the queue_options.keys() branch.- get_config().temporal.validate_task_queue_known("temporal_task_queue")diff --git a/tests/unit/pipelex/temporal/test_workflow_caller_child_error_recovery.py b/tests/unit/pipelex/temporal/test_workflow_caller_child_error_recovery.pydeleted file mode 100644index 47a98c88f..000000000--- a/tests/unit/pipelex/temporal/test_workflow_caller_child_error_recovery.py+++ /dev/null@@ -1,138 +0,0 @@-"""Unit tests for the child-workflow boundary of ``WorkflowExecutor``.--When a child workflow fails, ``ChildWorkflowError`` exposes the deserialized-failure via ``.cause``. ``execute_child_workflow`` / ``start_child_workflow``-recover the structured ``ErrorReport`` via the total ``recover_error_report``-and carry it on the ``WorkflowExecutionError`` — mirroring the-``execute_workflow`` path — so the classification survives the child-workflow-hop. A cause carrying a report payload recovers it verbatim; an-``ApplicationError`` without a report, a non-``ApplicationError`` cause, or a-missing cause all synthesize an ``UnrecoverableWorkflowFailureError`` report.-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture-from temporalio import workflow-from temporalio.exceptions import ApplicationError, ChildWorkflowError-from typing_extensions import override--from pipelex.base_exceptions import ErrorDomain, ErrorReport-from pipelex.cogt.inference.error_classification import UserAction, UserActionKind-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.tprl.workflow_caller import WorkflowClass, WorkflowExecutor--_FULL_REPORT = ErrorReport(- error_type="CogtError",- message="rate limited on the worker",- title="AI inference failed",- type_uri="https://docs.pipelex.com/latest/errors/cogt-error/",- error_category="capacity",- error_domain=ErrorDomain.RUNTIME,- retryable=False,- user_action=UserAction(kind=UserActionKind.CHECK_BILLING, detail="check your billing page"),- model="gpt-5",- provider="openai",-)---class _StubWorkflow(WorkflowClass[Any, Any]):- @override- async def run(self, workflow_arg: Any) -> Any: # pragma: no cover - never actually invoked- return workflow_arg---def _child_workflow_error(cause: BaseException) -> ChildWorkflowError:- """Build a ``ChildWorkflowError`` whose ``.cause`` is the deserialized failure."""- error = ChildWorkflowError(- "child workflow execution error",- namespace="default",- workflow_id="ut-child",- run_id="ut-run",- workflow_type="WfPipeRun",- initiated_event_id=1,- started_event_id=2,- retry_state=None,- )- error.__cause__ = cause- return error--[email protected](loop_scope="class")-class TestWorkflowCallerChildErrorRecovery:- @pytest.mark.parametrize(- "method_name",- [- pytest.param("execute_child_workflow", id="execute"),- pytest.param("start_child_workflow", id="start"),- ],- )- async def test_child_application_error_with_report_recovers_classification(self, mocker: MockerFixture, method_name: str) -> None:- """A categorized child-workflow failure surfaces as a fully classified ``WorkflowExecutionError``."""- app_error = ApplicationError("rate limited on the worker", _FULL_REPORT.to_dict(), type="CogtError")- mocker.patch.object(workflow, method_name, new=mocker.AsyncMock(side_effect=_child_workflow_error(app_error)))- executor = WorkflowExecutor[Any, Any](task_queue="test-queue")-- with pytest.raises(WorkflowExecutionError) as exc_info:- await getattr(executor, method_name)(workflow_class=_StubWorkflow, workflow_arg={}, workflow_id="ut-run")-- error = exc_info.value- # The real failure message replaces the generic "Failed to ... child workflow ...".- assert error.message == "rate limited on the worker"- assert error.error_report is not None- report = error.to_error_report()- assert report.message == "rate limited on the worker"- assert report.error_category == "capacity"- assert report.retryable is False- assert report.model == "gpt-5"- assert report.provider == "openai"- assert report.user_action is not None- assert report.user_action.kind == UserActionKind.CHECK_BILLING-- @pytest.mark.parametrize(- "method_name",- [- pytest.param("execute_child_workflow", id="execute"),- pytest.param("start_child_workflow", id="start"),- ],- )- async def test_g4_application_error_without_report_details_synthesizes_unrecoverable(self, mocker: MockerFixture, method_name: str) -> None:- """G4 — a child ``ApplicationError`` carrying no report payload synthesizes the unrecoverable report."""- app_error = ApplicationError("worker failure", type="RuntimeError")- mocker.patch.object(workflow, method_name, new=mocker.AsyncMock(side_effect=_child_workflow_error(app_error)))- executor = WorkflowExecutor[Any, Any](task_queue="test-queue")-- with pytest.raises(WorkflowExecutionError) as exc_info:- await getattr(executor, method_name)(workflow_class=_StubWorkflow, workflow_arg={}, workflow_id="ut-run")-- error = exc_info.value- assert error.error_report is not None- assert error.error_report.error_type == "UnrecoverableWorkflowFailureError"- assert "worker failure" in error.message-- @pytest.mark.parametrize(- "method_name",- [- pytest.param("execute_child_workflow", id="execute"),- pytest.param("start_child_workflow", id="start"),- ],- )- async def test_non_application_error_cause_synthesizes_unrecoverable(self, mocker: MockerFixture, method_name: str) -> None:- """A child failure whose ``.cause`` is not an ``ApplicationError`` still synthesizes the unrecoverable report.-- ``recover_error_report`` is total, so a plain worker exception is recovered just- like the ``ApplicationError``-without-report case — keeping the child path- consistent with ``execute_workflow`` and the production ``wf_pipe_run`` child path.- """- cause = RuntimeError("worker crashed hard")- mocker.patch.object(workflow, method_name, new=mocker.AsyncMock(side_effect=_child_workflow_error(cause)))- executor = WorkflowExecutor[Any, Any](task_queue="test-queue")-- with pytest.raises(WorkflowExecutionError) as exc_info:- await getattr(executor, method_name)(workflow_class=_StubWorkflow, workflow_arg={}, workflow_id="ut-run")-- error = exc_info.value- assert error.error_report is not None- assert error.error_report.error_type == "UnrecoverableWorkflowFailureError"- assert "worker crashed hard" in error.messagediff --git a/tests/unit/pipelex/temporal/test_workflow_caller_error_recovery.py b/tests/unit/pipelex/temporal/test_workflow_caller_error_recovery.pydeleted file mode 100644index 96cfcbe80..000000000--- a/tests/unit/pipelex/temporal/test_workflow_caller_error_recovery.py+++ /dev/null@@ -1,122 +0,0 @@-"""Unit tests for ``WorkflowExecutor.execute_workflow``'s error path.--When ``client.execute_workflow`` raises a ``WorkflowFailureError``, the executor-recovers the structured ``ErrorReport`` packed by the activity bridge and-carries it on the ``WorkflowExecutionError`` — so the classification survives-the workflow → submitter hop instead of flooring to a generic ``RUNTIME`` error.-``WorkflowAlreadyStartedError`` / ``RPCError`` go through the sibling clause and-stay generic.-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture-from temporalio.client import WorkflowFailureError-from temporalio.exceptions import ApplicationError, WorkflowAlreadyStartedError-from temporalio.service import RPCError, RPCStatusCode-from typing_extensions import override--from pipelex.base_exceptions import ErrorDomain, ErrorReport-from pipelex.cogt.inference.error_classification import UserAction, UserActionKind-from pipelex.temporal.exceptions import WorkflowExecutionError-from pipelex.temporal.tprl.workflow_caller import WorkflowClass, WorkflowExecutor--_FULL_REPORT = ErrorReport(- error_type="CogtError",- message="rate limited on the worker",- title="AI inference failed",- type_uri="https://docs.pipelex.com/latest/errors/cogt-error/",- error_category="capacity",- error_domain=ErrorDomain.RUNTIME,- retryable=False,- user_action=UserAction(kind=UserActionKind.CHECK_BILLING, detail="check your billing page"),- model="gpt-5",- provider="openai",-)---class _StubWorkflow(WorkflowClass[Any, Any]):- @override- async def run(self, workflow_arg: Any) -> Any: # pragma: no cover - never actually invoked- return workflow_arg---def _make_executor_with_stub_client(mocker: MockerFixture) -> tuple[WorkflowExecutor[Any, Any], Any]:- """Build a ``WorkflowExecutor`` whose ``temporal_client()`` returns a stub client."""- stub_client = mocker.MagicMock()- mocker.patch.object(WorkflowExecutor, "temporal_client", new=mocker.AsyncMock(return_value=stub_client))- return WorkflowExecutor[Any, Any](task_queue="test-queue"), stub_client---def _workflow_failure_with_report() -> WorkflowFailureError:- """A ``WorkflowFailureError`` carrying the activity-bridge report payload."""- app_error = ApplicationError("rate limited on the worker", _FULL_REPORT.to_dict(), type="CogtError")- return WorkflowFailureError(cause=app_error)--[email protected](loop_scope="class")-class TestWorkflowCallerErrorRecovery:- async def test_workflow_failure_with_report_recovers_classification_and_real_message(self, mocker: MockerFixture) -> None:- """A categorized worker failure surfaces as a fully classified ``WorkflowExecutionError``."""- executor, client = _make_executor_with_stub_client(mocker)- client.execute_workflow = mocker.AsyncMock(side_effect=_workflow_failure_with_report())-- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(workflow_class=_StubWorkflow, workflow_arg={}, workflow_id="ut-run")-- error = exc_info.value- # The real failure message replaces the generic "Failed to execute workflow ...".- assert error.message == "rate limited on the worker"- assert error.error_report is not None- report = error.to_error_report()- assert report.message == "rate limited on the worker"- assert report.error_category == "capacity"- assert report.retryable is False- assert report.model == "gpt-5"- assert report.provider == "openai"- assert report.user_action is not None- assert report.user_action.kind == UserActionKind.CHECK_BILLING-- async def test_workflow_failure_without_report_synthesizes_unrecoverable(self, mocker: MockerFixture) -> None:- """A non-Pipelex workflow failure (no packed report) surfaces as a synthesized unrecoverable report.-- ``recover_error_report`` is total — there is no Pipelex-framed- ``"Failed to execute workflow X"`` fallback. The ``WorkflowExecutionError``- carries a stable-identity ``UnrecoverableWorkflowFailureError`` report- whose message preserves the underlying exception text.- """- executor, client = _make_executor_with_stub_client(mocker)- failure = WorkflowFailureError(cause=RuntimeError("worker crashed hard"))- client.execute_workflow = mocker.AsyncMock(side_effect=failure)-- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(workflow_class=_StubWorkflow, workflow_arg={}, workflow_id="ut-run")-- error = exc_info.value- assert error.error_report is not None- assert error.error_report.error_type == "UnrecoverableWorkflowFailureError"- assert error.error_report.error_domain == ErrorDomain.RUNTIME- # Synthesized message preserves the underlying exception text, NOT the- # legacy "Failed to execute workflow _StubWorkflow" framing.- assert "worker crashed hard" in error.message- assert error.message != "Failed to execute workflow _StubWorkflow"-- @pytest.mark.parametrize(- "side_effect",- [- pytest.param(WorkflowAlreadyStartedError(workflow_id="ut-run", workflow_type="WfPipeRun"), id="already-started"),- pytest.param(RPCError("temporal server unavailable", RPCStatusCode.UNAVAILABLE, b""), id="rpc-error"),- ],- )- async def test_g6_already_started_and_rpc_error_stay_generic(self, mocker: MockerFixture, side_effect: BaseException) -> None:- """G6 — the ``WorkflowAlreadyStartedError`` / ``RPCError`` clause stays generic, ``error_report=None``."""- executor, client = _make_executor_with_stub_client(mocker)- client.execute_workflow = mocker.AsyncMock(side_effect=side_effect)-- with pytest.raises(WorkflowExecutionError) as exc_info:- await executor.execute_workflow(workflow_class=_StubWorkflow, workflow_arg={}, workflow_id="ut-run")-- error = exc_info.value- assert error.error_report is None- assert error.message == "Failed to execute workflow _StubWorkflow"diff --git a/tests/unit/pipelex/temporal/test_workflow_caller_passthrough.py b/tests/unit/pipelex/temporal/test_workflow_caller_passthrough.pydeleted file mode 100644index bce952630..000000000--- a/tests/unit/pipelex/temporal/test_workflow_caller_passthrough.py+++ /dev/null@@ -1,131 +0,0 @@-"""Unit tests for the additive ``search_attributes`` / ``static_summary`` /-``static_details`` / ``memo`` pass-through on every ``WorkflowExecutor`` entry-point.--Phase 1 of the Temporal IDs and Naming redesign extends the executor's surface-so the workflow-layer (Phase 3) can set the observability attributes on every-workflow start, both top-level and child. These tests pin the wiring: the four-kwargs given to the executor reach the underlying SDK call unchanged.-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture-from temporalio.common import SearchAttributeKey, SearchAttributePair, TypedSearchAttributes-from typing_extensions import override--from pipelex.temporal.tprl.workflow_caller import WorkflowClass, WorkflowExecutor---class _StubWorkflow(WorkflowClass[Any, Any]):- @override- async def run(self, workflow_arg: Any) -> Any: # pragma: no cover - never actually invoked- return workflow_arg---_SEARCH_ATTRS = TypedSearchAttributes(- [- SearchAttributePair(SearchAttributeKey.for_keyword("PipeCode"), "translate_doc"),- SearchAttributePair(SearchAttributeKey.for_keyword("DomainCode"), "documents"),- ],-)-_SUMMARY = "translate_doc — Translate EN→FR"-_DETAILS = "| Field | Value |\n|---|---|\n| Pipe | `translate_doc` |"-_MEMO = {"pipelex": {"library_crate": "[email protected]"}}---def _make_executor_with_stub_client(mocker: MockerFixture) -> tuple[WorkflowExecutor[Any, Any], Any]:- """Build a ``WorkflowExecutor`` whose ``temporal_client()`` returns a stub client.-- Returns the executor and the stub client so tests can read ``call_args`` from- the methods the executor invokes on the client.- """- stub_client = mocker.MagicMock()- mocker.patch.object(WorkflowExecutor, "temporal_client", new=mocker.AsyncMock(return_value=stub_client))- return WorkflowExecutor[Any, Any](task_queue="test-queue"), stub_client--[email protected](loop_scope="class")-class TestWorkflowCallerPassthrough:- async def test_execute_workflow_passes_through_observability_kwargs(self, mocker: MockerFixture) -> None:- executor, client = _make_executor_with_stub_client(mocker)- client.execute_workflow = mocker.AsyncMock(return_value="result")-- await executor.execute_workflow(- workflow_class=_StubWorkflow,- workflow_arg={"foo": "bar"},- workflow_id="ut-test-run",- search_attributes=_SEARCH_ATTRS,- static_summary=_SUMMARY,- static_details=_DETAILS,- memo=_MEMO,- )-- kwargs = client.execute_workflow.call_args.kwargs- assert kwargs.get("search_attributes") == _SEARCH_ATTRS- assert kwargs.get("static_summary") == _SUMMARY- assert kwargs.get("static_details") == _DETAILS- assert kwargs.get("memo") == _MEMO-- async def test_start_workflow_passes_through_observability_kwargs(self, mocker: MockerFixture) -> None:- executor, client = _make_executor_with_stub_client(mocker)- client.start_workflow = mocker.AsyncMock(return_value=mocker.MagicMock())-- await executor.start_workflow(- workflow_class=_StubWorkflow,- workflow_arg={"foo": "bar"},- workflow_id="ut-test-run",- search_attributes=_SEARCH_ATTRS,- static_summary=_SUMMARY,- static_details=_DETAILS,- memo=_MEMO,- )-- kwargs = client.start_workflow.call_args.kwargs- assert kwargs.get("search_attributes") == _SEARCH_ATTRS- assert kwargs.get("static_summary") == _SUMMARY- assert kwargs.get("static_details") == _DETAILS- assert kwargs.get("memo") == _MEMO-- async def test_execute_child_workflow_passes_through_observability_kwargs(self, mocker: MockerFixture) -> None:- executor, _ = _make_executor_with_stub_client(mocker)- mock_execute_child = mocker.patch("temporalio.workflow.execute_child_workflow", new_callable=mocker.AsyncMock)- mock_execute_child.return_value = "child-result"-- await executor.execute_child_workflow(- workflow_class=_StubWorkflow,- workflow_arg={"foo": "bar"},- workflow_id="ut-test-run_pipe-router",- search_attributes=_SEARCH_ATTRS,- static_summary=_SUMMARY,- static_details=_DETAILS,- memo=_MEMO,- )-- kwargs = mock_execute_child.call_args.kwargs- assert kwargs.get("search_attributes") == _SEARCH_ATTRS- assert kwargs.get("static_summary") == _SUMMARY- assert kwargs.get("static_details") == _DETAILS- assert kwargs.get("memo") == _MEMO-- async def test_start_child_workflow_passes_through_observability_kwargs(self, mocker: MockerFixture) -> None:- executor, _ = _make_executor_with_stub_client(mocker)- mock_start_child = mocker.patch("temporalio.workflow.start_child_workflow", new_callable=mocker.AsyncMock)- mock_start_child.return_value = mocker.MagicMock()-- await executor.start_child_workflow(- workflow_class=_StubWorkflow,- workflow_arg={"foo": "bar"},- workflow_id="ut-test-run_pipe-router",- search_attributes=_SEARCH_ATTRS,- static_summary=_SUMMARY,- static_details=_DETAILS,- memo=_MEMO,- )-- kwargs = mock_start_child.call_args.kwargs- assert kwargs.get("search_attributes") == _SEARCH_ATTRS- assert kwargs.get("static_summary") == _SUMMARY- assert kwargs.get("static_details") == _DETAILS- assert kwargs.get("memo") == _MEMOdiff --git a/tests/unit/pipelex/temporal/test_workflow_execution_error.py b/tests/unit/pipelex/temporal/test_workflow_execution_error.pydeleted file mode 100644index e1992c367..000000000--- a/tests/unit/pipelex/temporal/test_workflow_execution_error.py+++ /dev/null@@ -1,53 +0,0 @@-"""Unit tests for ``WorkflowExecutionError.to_error_report()``.--When the submitter recovered a structured report, ``WorkflowExecutionError``-carries it and ``to_error_report()`` returns it verbatim. With no recovered-report it falls through to the base ``__cause__``-chain enrichment.-"""--from pipelex.base_exceptions import ErrorDomain, ErrorReport, PipelexConfigError-from pipelex.cogt.inference.error_classification import UserAction, UserActionKind-from pipelex.temporal.exceptions import WorkflowExecutionError--_FULL_REPORT = ErrorReport(- error_type="CogtError",- message="rate limited on the worker",- title="AI inference failed",- type_uri="https://docs.pipelex.com/latest/errors/cogt-error/",- error_category="capacity",- error_domain=ErrorDomain.RUNTIME,- retryable=False,- user_action=UserAction(kind=UserActionKind.CHECK_BILLING, detail="check your billing page"),- model="gpt-5",- provider="openai",-)---class TestWorkflowExecutionError:- def test_to_error_report_returns_recovered_report_verbatim(self) -> None:- """With a recovered report present, ``to_error_report`` returns it unchanged."""- error = WorkflowExecutionError(_FULL_REPORT.message, error_report=_FULL_REPORT)- assert error.to_error_report() == _FULL_REPORT-- def test_g7_to_error_report_falls_through_to_cause_enrichment_when_no_report(self) -> None:- """G7 — with no recovered report, ``to_error_report`` uses base ``__cause__`` enrichment."""- error = WorkflowExecutionError("Failed to execute workflow WfPipeRun")- error.__cause__ = PipelexConfigError("bad config")-- report = error.to_error_report()-- assert report.error_type == "WorkflowExecutionError"- assert report.message == "Failed to execute workflow WfPipeRun"- # error_domain is inherited from the PipelexError cause.- assert report.error_domain == ErrorDomain.CONFIG-- def test_to_error_report_is_generic_without_report_or_pipelex_cause(self) -> None:- """No recovered report and no ``PipelexError`` cause → an unclassified generic report."""- error = WorkflowExecutionError("Failed to execute workflow WfPipeRun")-- report = error.to_error_report()-- assert report.error_type == "WorkflowExecutionError"- assert report.error_category is None- assert report.error_domain is None- assert report.retryable is Nonediff --git a/tests/unit/pipelex/temporal/test_workflow_id_construction.py b/tests/unit/pipelex/temporal/test_workflow_id_construction.pydeleted file mode 100644index f15858164..000000000--- a/tests/unit/pipelex/temporal/test_workflow_id_construction.py+++ /dev/null@@ -1,131 +0,0 @@-"""Unit tests for Pipelex's workflow-id construction model.--Pins the shape documented in-``wip/temporal-primitives/id-and-naming-design.md`` §"Layer 1 — Identity":--- Top-level: ``{env_prefix}{pipeline_run_id}``.-- Fixed-role child: ``{parent}_pipe-router``.-- Dynamic child (sub-pipe spawned by a router): ``{parent}_{pipe_code}-{8-hex}``.--The dynamic-child id uses ``workflow.uuid4()`` for replay-safety — Temporal's-``workflow.uuid4()`` is deterministic, unlike stdlib ``uuid.uuid4()`` (which-the workflow sandbox forbids).-"""--from collections.abc import Iterator--import pytest-from pytest_mock import MockerFixture--from pipelex.system.runtime import RunMode, runtime_manager-from pipelex.temporal.temporal_manager import TemporalManager---def temporal_manager() -> Iterator[TemporalManager]:- """Ensure ``TemporalManager`` is set up with a known session_id for the test.-- Restores any previous singleton after the test by tearing down and re-seeding.- """- prior = TemporalManager._shared_instance # noqa: SLF001 # pyright: ignore[reportPrivateUsage]- TemporalManager.teardown()- TemporalManager.setup(session_id="EdgdJ7Yk4Q3HF2pXyZv9w8")- try:- yield TemporalManager.get_instance()- finally:- TemporalManager.teardown()- if prior is not None:- TemporalManager._shared_instance = prior # noqa: SLF001 # pyright: ignore[reportPrivateUsage]---class TestTopLevelWorkflowIdConstruction:- @pytest.mark.parametrize(- ("run_mode", "expected_prefix"),- [- (RunMode.UNIT_TEST, "ut-"),- (RunMode.NORMAL, ""),- (RunMode.CI_TEST, "ci-"),- (RunMode.CODEX_CLOUD, "cc-"),- (RunMode.CODEX_CLOUD_TEST, "cct-"),- ],- )- def test_make_top_workflow_id_per_run_mode(- self,- temporal_manager: TemporalManager,- run_mode: RunMode,- expected_prefix: str,- ) -> None:- prior_mode = runtime_manager.run_mode- runtime_manager.set_run_mode(run_mode)- try:- pipeline_run_id = "3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c"- workflow_id = temporal_manager.make_top_workflow_id(pipeline_run_id=pipeline_run_id)- assert workflow_id == f"{expected_prefix}{pipeline_run_id}"- finally:- runtime_manager.set_run_mode(prior_mode)-- def test_make_top_workflow_id_has_no_session_or_random_suffix(- self,- temporal_manager: TemporalManager,- ) -> None:- """The redesigned id is the bare ``{env_prefix}{pipeline_run_id}`` —- no truncated session id, no shortuuid suffix, no class name.- """- prior_mode = runtime_manager.run_mode- runtime_manager.set_run_mode(RunMode.UNIT_TEST)- try:- pipeline_run_id = "3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c"- workflow_id = temporal_manager.make_top_workflow_id(pipeline_run_id=pipeline_run_id)- # No 5-char session truncation followed by a dash.- assert "EdgdJ" not in workflow_id- # No "TemporalPipeRun" / "TemporalPipeRouter" tail.- assert "TemporalPipe" not in workflow_id- finally:- runtime_manager.set_run_mode(prior_mode)---class TestChildWorkflowIdConstruction:- """Build child workflow ids the way Phase 3's call sites do, and assert the- documented shape.-- The construction lives in ``temporal_pipe_router.py`` and ``wf_pipe_run.py``.- These tests reproduce the formula locally rather than driving the real call- sites — the formula itself is the contract.- """-- def test_fixed_role_child_uses_underscore_pipe_router_suffix(self) -> None:- parent_workflow_id = "ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c"-- # Mirrors the construction in wf_pipe_run.py:- child_id = f"{parent_workflow_id}_pipe-router"-- assert child_id == "ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c_pipe-router"- # The separator is `_`, never `/`: workflow ids must stay free of path- # separators so they can be reused verbatim as S3 keys / file names.- assert "/" not in child_id- assert child_id.endswith("_pipe-router")-- def test_dynamic_child_uses_underscore_pipe_code_and_8_hex_disambiguator(self, mocker: MockerFixture) -> None:- """The dynamic-child id must be fully determined by- (parent_workflow_id, pipe_code, workflow.uuid4() output) — i.e. the- implementation must use ``workflow.uuid4()`` (replay-safe), not- stdlib ``uuid.uuid4()`` which Temporal's workflow sandbox forbids.- """- # ``str(workflow.uuid4())`` returns a UUID string — mock with a stub- # whose ``__str__`` returns the deterministic 36-char form so the- # ``[:8]`` slice picks up the same prefix on every replay.- mocker.patch("temporalio.workflow.uuid4", return_value="7c1e2f8a-deadbeef-cafebabe-12345678")- from temporalio import workflow # noqa: PLC0415-- parent_workflow_id = "ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c"- pipe_code = "translate_doc"-- # Mirrors the construction in temporal_pipe_router.py (child branch):- child_id = f"{parent_workflow_id}_{pipe_code}-{str(workflow.uuid4())[:8]}"-- assert child_id == "ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c_translate_doc-7c1e2f8a"- assert "/" not in child_id- # Replay-determinism: re-running the same construction with the same- # mock yields the same id.- child_id_again = f"{parent_workflow_id}_{pipe_code}-{str(workflow.uuid4())[:8]}"- assert child_id_again == child_iddiff --git a/tests/unit/pipelex/temporal/tprl_pipe/test_marker_isolation.py b/tests/unit/pipelex/temporal/tprl_pipe/test_marker_isolation.pydeleted file mode 100644index 3dd6be18f..000000000--- a/tests/unit/pipelex/temporal/tprl_pipe/test_marker_isolation.py+++ /dev/null@@ -1,77 +0,0 @@-from typing import Any--import pytest-from kajson import kajson-from kajson.exceptions import KajsonDecoderError-from pydantic import BaseModel---class _Envelope(BaseModel):- """Mimics a PipeOutput-shaped payload: a BaseModel carrying a plain-dict field.-- The dehydrated working_memory_raw is exactly this shape: nested dicts that- travel through kajson.dumps / kajson.loads at the Temporal data-converter- boundary. The envelope itself is a BaseModel (so kajson encodes/decodes it),- but the payload field is a free-form dict whose nested keys are pipelex's- responsibility — kajson must leave it alone.- """-- payload: dict[str, Any]---class TestKajsonMarkerIsolation:- """Wire-format contract tests: kajson's universal decoder gates on __class__.-- Pipelex's deferred-binding wire format uses __pipelex_class__/__pipelex_module__- deliberately so kajson's decoder ignores it. These tests pin that contract:- if anyone ever re-collides the namespace, both tests catch it.- """-- def test_pipelex_markers_pass_through_unchanged(self) -> None:- """Nested dicts with __pipelex_class__ / __pipelex_module__ must round-trip untouched.-- The class name points at a name that is NOT registered anywhere — if kajson- were to attempt resolution, it would raise. The fact that round-trip succeeds- proves kajson does not touch the pipelex namespace.- """- envelope = _Envelope(- payload={- "items": [- {- "text": "hello",- "__pipelex_class__": "DefinitelyNotRegistered__xyz",- "__pipelex_module__": "builtins",- }- ]- }- )-- encoded = kajson.dumps(envelope)- decoded = kajson.loads(encoded)-- assert isinstance(decoded, _Envelope)- assert decoded.payload == envelope.payload-- def test_kajson_markers_raise_for_unknown_class(self) -> None:- """Inverse oracle: __class__ / __module__ for an unknown class MUST raise.-- This is the original failure mode that triggered the rename. We keep this- test as a sanity check that kajson's decoder is still strict on its own- namespace — so any accidental re-collision shows up immediately.- """- envelope = _Envelope(- payload={- "items": [- {- "text": "hello",- "__class__": "DefinitelyNotRegistered__xyz",- "__module__": "builtins",- }- ]- }- )-- encoded = kajson.dumps(envelope)-- with pytest.raises(KajsonDecoderError):- kajson.loads(encoded)diff --git a/tests/unit/pipelex/temporal/tprl_pipe/test_temporal_pipe_router_child_observability.py b/tests/unit/pipelex/temporal/tprl_pipe/test_temporal_pipe_router_child_observability.pydeleted file mode 100644index 385c899b3..000000000--- a/tests/unit/pipelex/temporal/tprl_pipe/test_temporal_pipe_router_child_observability.py+++ /dev/null@@ -1,118 +0,0 @@-"""Unit test pinning that ``TemporalPipeRouter`` passes the same observability-kwargs on child workflow dispatch as on top-level dispatch.--Regression for PR #891 review feedback: the child path was passing-``static_summary`` but not ``static_details``, so child workflows showed no-details pane in the Temporal UI. The top-level path at-``temporal_pipe_router.py:108-115`` already passes both — the child path at-``temporal_pipe_router.py:81-88`` must mirror that surface.-"""--from typing import Any--import pytest-from pytest_mock import MockerFixture--from pipelex.temporal.tprl.observability import build_static_details, build_static_summary-from pipelex.temporal.tprl_pipe.temporal_pipe_router import TemporalPipeRouter--_STUB_SESSION_ID = "EdgdJ7Yk4Q3HF2pXyZv9w8"---def _make_pipe_job_stub(mocker: MockerFixture) -> Any:- """Stub PipeJob shaped for the helpers the router calls during child dispatch.-- Covers ``prepare_for_temporal()`` (returns self), ``pipe.code``,- ``pipe.domain_code``, ``pipe.description``, ``pipe.inputs.root``,- ``job_metadata.*``, and ``library_crate`` (``None`` so rehydrate skips- the crate path).- """- pipe_job = mocker.MagicMock()- pipe_job.prepare_for_temporal.return_value = pipe_job- pipe_job.pipe.code = "translate_doc"- pipe_job.pipe.domain_code = "documents"- pipe_job.pipe.description = "Translate a document from English to French"- pipe_job.pipe.inputs.root = {}- pipe_job.job_metadata.user_id = "acme-corp"- pipe_job.job_metadata.pipeline_run_id = "3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c"- pipe_job.job_metadata.session_id = _STUB_SESSION_ID- pipe_job.library_crate = None- return pipe_job--[email protected](loop_scope="class")-class TestTemporalPipeRouterChildObservability:- @pytest.fixture(autouse=True)- def _enable_temporal(self, mocker: MockerFixture) -> None:- """The async-enabled guard inside ``with_conditional_worker`` runs- before ``_run_pipe_job``'s body — patch the decorator's ``get_config``- site so the guard reads enabled (disabled-path coverage lives in- ``test_async_execution_not_enabled.py``).- """- config_root = mocker.MagicMock()- config_root.temporal.is_enabled = True- mocker.patch("pipelex.temporal.tprl.conditional_worker.get_config", return_value=config_root)-- async def test_child_workflow_dispatch_passes_static_details(self, mocker: MockerFixture) -> None:- """Child workflow dispatch must pass ``static_details`` to- ``workflow.execute_child_workflow``, matching the top-level path.-- Without this kwarg the Temporal UI shows no details pane for child- workflows even though the parent renders the full identity-field- markdown table.- """- pipe_job = _make_pipe_job_stub(mocker)-- config_root = mocker.MagicMock()- config_root.temporal.search_attributes.enabled = True- config_root.temporal.search_attributes.attributes = ["PipeCode", "PipelineRunId", "SessionId", "UserId", "DomainCode"]- mocker.patch("pipelex.temporal.tprl.observability.get_config", return_value=config_root)-- manager = mocker.MagicMock()- manager.session_id = _STUB_SESSION_ID- mocker.patch("pipelex.temporal.tprl.observability.get_temporal_manager", return_value=manager)-- mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_router.is_in_temporal_workflow", return_value=True)- info_stub = mocker.MagicMock()- info_stub.workflow_id = "ut-parent-wf-id"- mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_router.workflow.info", return_value=info_stub)- mocker.patch(- "pipelex.temporal.tprl_pipe.temporal_pipe_router.workflow.uuid4",- return_value="deadbeef-aaaa-bbbb-cccc-ddddeeeeffff",- )-- mock_execute_child = mocker.patch(- "pipelex.temporal.tprl_pipe.temporal_pipe_router.workflow.execute_child_workflow",- new_callable=mocker.AsyncMock,- )- sentinel_output = mocker.MagicMock()- mock_execute_child.return_value = sentinel_output-- # Rehydrate is exercised by other tests; here we just want the value- # returned by ``execute_child_workflow`` to flow through untouched.- def _passthrough_rehydrate(pipe_output: Any, library_crate: Any) -> Any:- del library_crate- return pipe_output-- mocker.patch(- "pipelex.temporal.tprl_pipe.temporal_pipe_router.rehydrate_pipe_output_with_crate",- side_effect=_passthrough_rehydrate,- )-- router = TemporalPipeRouter(task_queue="ut-test-queue")- await router._run_pipe_job(pipe_job) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]-- kwargs = mock_execute_child.call_args.kwargs-- expected_details = build_static_details(pipe_job)- expected_summary = build_static_summary(pipe_job.pipe)-- assert kwargs.get("static_details") == expected_details- # Regression guard: don't allow static_details to become an empty- # string or be silently dropped.- assert kwargs.get("static_details")- # Existing kwargs must still be present on the child dispatch.- assert kwargs.get("static_summary") == expected_summary- assert kwargs.get("search_attributes") is not None- # Child workflow id is built as `{parent}_{pipe_code}-{8 hex}`.- assert kwargs.get("id") == "ut-parent-wf-id_translate_doc-deadbeef"diff --git a/tests/unit/pipelex/test_error_type_uri.py b/tests/unit/pipelex/test_error_type_uri.pydeleted file mode 100644index d52264cc8..000000000--- a/tests/unit/pipelex/test_error_type_uri.py+++ /dev/null@@ -1,14 +0,0 @@-from pipelex.temporal.exceptions import UnrecoverableWorkflowFailureError---class TestErrorTypeUri:- def test_type_uri_is_pure_constant_derived(self):- """``type_uri()`` derives the RFC 7807 ``type`` URI from a hardcoded constant.-- No Pipelex bootstrap, no mutable process config. Pins the determinism- fix: a synthesized ``UnrecoverableWorkflowFailureError`` on the Temporal- workflow-recovery path must produce a stable ``type_uri`` so the report- baked into workflow history survives replay unchanged.- """- report = UnrecoverableWorkflowFailureError("msg").to_error_report()- assert report.type_uri == "https://docs.pipelex.com/latest/errors/unrecoverable-workflow-failure-error/"diff --git a/tests/unit/pipelex/test_pipelex_error_message_init.py b/tests/unit/pipelex/test_pipelex_error_message_init.pynew file mode 100644index 000000000..51c43c6fa--- /dev/null+++ b/tests/unit/pipelex/test_pipelex_error_message_init.py@@ -0,0 +1,45 @@+"""Guard + regression for the message-setting ``__init__`` of ``PipelexError`` subclasses.++A subclass that mixes in ``ValueError`` / ``TypeError`` (so Pydantic validators wrap+its raises into a ``ValidationError``) must list the ``PipelexError``-derived base+*first*. Otherwise ``BaseException.__init__`` — inherited through ``ValueError`` /+``TypeError`` — shadows ``PipelexError.__init__`` in the MRO, ``self.message`` is never+set, and ``to_error_report()`` raises ``AttributeError`` the moment an error-report+rendering path touches the instance.+"""++import pytest++from pipelex.base_exceptions import PipelexError+from pipelex.cogt.exceptions import RoutingProfileBlueprintValueError+from pipelex.errors.error_pages_generator import iter_pipelex_error_subclasses+from pipelex.pipe_operators.compose.exceptions import ConstructFieldBlueprintValueError+from pipelex.system.exceptions import ConfigModelError+++class TestPipelexErrorMessageInit:+ def test_no_subclass_shadows_message_setting_init(self) -> None:+ """No loaded ``PipelexError`` subclass lets ``ValueError`` / ``TypeError`` precede+ ``PipelexError`` in its MRO — that ordering shadows the ``.message`` setter.+ """+ offenders: list[str] = []+ for error_cls in iter_pipelex_error_subclasses():+ mro = error_cls.__mro__+ pipelex_index = mro.index(PipelexError)+ for shadow_cls in (ValueError, TypeError):+ if shadow_cls in mro and mro.index(shadow_cls) < pipelex_index:+ offenders.append(f"{error_cls.__module__}.{error_cls.__name__} (lists {shadow_cls.__name__} before PipelexError)")+ assert not offenders, "Exception classes shadow PipelexError.__init__ — `.message` will be unset:\n" + "\n".join(offenders)++ @pytest.mark.parametrize("error_cls", [ConstructFieldBlueprintValueError, RoutingProfileBlueprintValueError, ConfigModelError])+ def test_message_set_and_error_report_succeeds(self, error_cls: type[PipelexError]) -> None:+ """A reordered ``(PipelexError, ValueError)`` mixin sets ``.message``, keeps its+ ``ValueError`` identity (so Pydantic still wraps validator raises), and renders a report.+ """+ message = "the configuration section is invalid"+ exc = error_cls(message)+ assert exc.message == message+ assert isinstance(exc, ValueError)+ report = exc.to_error_report()+ assert report.message == message+ assert report.error_type == error_cls.__name__diff --git a/uv.lock b/uv.lockindex 2ba0f1423..db7c1e249 100644--- a/uv.lock+++ b/uv.lock@@ -2933,18 +2933,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" },]-[[package]]-name = "nexus-rpc"-version = "1.4.0"-source = { registry = "https://pypi.org/simple" }-dependencies = [- { name = "typing-extensions" },-]-sdist = { url = "https://files.pythonhosted.org/packages/35/d5/cd1ffb202b76ebc1b33c1332a3416e55a39929006982adc2b1eb069aaa9b/nexus_rpc-1.4.0.tar.gz", hash = "sha256:3b8b373d4865671789cc43623e3dc0bcbf192562e40e13727e17f1c149050fba", size = 82367, upload-time = "2026-02-25T22:01:34.053Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/11/52/6327a5f4fda01207205038a106a99848a41c83e933cd23ea2cab3d2ebc6c/nexus_rpc-1.4.0-py3-none-any.whl", hash = "sha256:14c953d3519113f8ccec533a9efdb6b10c28afef75d11cdd6d422640c40b3a49", size = 29645, upload-time = "2026-02-25T22:01:33.122Z" },-]-[[package]]name = "nodeenv"version = "1.10.0"@@ -3812,17 +3800,12 @@ s3 = [{ name = "aioboto3" },{ name = "boto3" },]-temporal = [- { name = "aiohttp" },- { name = "temporalio" },-][package.metadata]requires-dist = [{ name = "aioboto3", marker = "extra == 'bedrock'", specifier = ">=13.4.0" },{ name = "aioboto3", marker = "extra == 's3'", specifier = ">=13.4.0" },{ name = "aiofiles", specifier = ">=23.2.1" },- { name = "aiohttp", marker = "extra == 'temporal'", specifier = ">=3.14.0" },{ name = "anthropic", marker = "extra == 'anthropic'", specifier = ">=0.78.0" },{ name = "backports-strenum", marker = "python_full_version < '3.11'", specifier = ">=1.3.0" },{ name = "boto3", marker = "extra == 'bedrock'", specifier = ">=1.34.131" },@@ -3896,7 +3879,6 @@ requires-dist = [{ name = "ruff", marker = "extra == 'dev'", specifier = "==0.14.13" },{ name = "semantic-version", specifier = ">=2.10.0" },{ name = "shortuuid", specifier = ">=1.0.13" },- { name = "temporalio", marker = "extra == 'temporal'", specifier = "==1.24.0" },{ name = "tomli", specifier = ">=2.3.0" },{ name = "tomlkit", specifier = ">=0.13.2" },{ name = "typer", specifier = ">=0.16.0,<0.27" },@@ -3909,7 +3891,7 @@ requires-dist = [{ name = "typing-extensions", specifier = ">=4.13.2" },{ name = "urllib3", specifier = ">=2.7.0" },]-provides-extras = ["anthropic", "bedrock", "docling", "fal", "gcp-storage", "google", "google-genai", "huggingface", "linkup", "mistralai", "dynamodb", "s3", "temporal", "docs", "dev"]+provides-extras = ["anthropic", "bedrock", "docling", "fal", "gcp-storage", "google", "google-genai", "huggingface", "linkup", "mistralai", "dynamodb", "s3", "docs", "dev"][[package]]name = "pipelex-tools"@@ -5580,26 +5562,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl", hash = "sha256:f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3", size = 39814, upload-time = "2026-03-04T18:55:31.284Z" },]-[[package]]-name = "temporalio"-version = "1.24.0"-source = { registry = "https://pypi.org/simple" }-dependencies = [- { name = "nexus-rpc" },- { name = "protobuf" },- { name = "python-dateutil", marker = "python_full_version < '3.11'" },- { name = "types-protobuf" },- { name = "typing-extensions" },-]-sdist = { url = "https://files.pythonhosted.org/packages/a7/b1/7d9b3104ab7994e7d49e765b92495aaff44810b1e066c874c284a93ebd55/temporalio-1.24.0.tar.gz", hash = "sha256:e534e2e71b4a721193ec4ff3dae521146d093554bd47a64f5605d4ca33e56718", size = 2040485, upload-time = "2026-03-23T15:33:33.638Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/84/a9/30517c21d6155bce1c3dc0e420db48da0231230dbc683f40ab6d5fe22b37/temporalio-1.24.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:7f11e7b4f4d09bafba499b43188353e23dc128b1fe3f3160014476e3dce70760", size = 12223918, upload-time = "2026-03-23T15:33:05.045Z" },- { url = "https://files.pythonhosted.org/packages/73/d0/11aa103bde794524008c1850a84e06cde98698395ca1f8b12e1bd2390aa8/temporalio-1.24.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:5cff75a0ca922575b808a7fca1b0de38f6eea061f49e026664b8be9d5bb06ab8", size = 11708887, upload-time = "2026-03-23T15:33:11.67Z" },- { url = "https://files.pythonhosted.org/packages/1d/f4/774b56100e6bb94e3757ec96fb5c2bc62d42defc7d6de0ee35a12273827a/temporalio-1.24.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ee7c13b6724dd0c304aa846aecf6da72a8550f4ade40a0a7f6dcc1c92ef35710", size = 12028303, upload-time = "2026-03-23T15:33:18.022Z" },- { url = "https://files.pythonhosted.org/packages/e5/91/c05d0e9c2432fe8b1ea0d6fae321866ee49a320ad5e494e6ec9424ca5c28/temporalio-1.24.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aa71b9bfa42f951dd04ade97ce7f92ecedee8903047b4b41b122bb8cbd87a337", size = 12375155, upload-time = "2026-03-23T15:33:24.234Z" },- { url = "https://files.pythonhosted.org/packages/c4/97/5c939e4609c164c8690a3b5a135eb828d531de8ef63ff447a2a439c0b0fb/temporalio-1.24.0-cp310-abi3-win_amd64.whl", hash = "sha256:52f6833647eceddbebcc376e2ea663a9f73b2b3a42675f503aeb27c98fd4daeb", size = 12720174, upload-time = "2026-03-23T15:33:30.826Z" },-]-[[package]]name = "tenacity"version = "9.1.4"@@ -6104,15 +6066,6 @@ wheels = [{ url = "https://files.pythonhosted.org/packages/60/1d/870a15630dc37a85aa0364bf623fad6ef9ef7cad14be62a0bab7541839f5/types_networkx-3.6.1.20260408-py3-none-any.whl", hash = "sha256:dc957f12bc9ffe7cab6704ccb719722fc5e508a491273c49b45c8786162a6f08", size = 162537, upload-time = "2026-04-08T04:34:37.385Z" },]-[[package]]-name = "types-protobuf"-version = "7.34.1.20260503"-source = { registry = "https://pypi.org/simple" }-sdist = { url = "https://files.pythonhosted.org/packages/a0/31/87969cb3e62287bde7598b78b3c098d2873d54f5fb5a7cfbcaa73b8c965e/types_protobuf-7.34.1.20260503.tar.gz", hash = "sha256:effbc819aa17e02448dde99f089c6794662d66f4b2797e922f185ffe0b24e766", size = 68830, upload-time = "2026-05-03T05:19:50.739Z" }-wheels = [- { url = "https://files.pythonhosted.org/packages/f9/67/a33fb18090a927794a5ee4b1a30730b528ace0dad6b18932540d21258184/types_protobuf-7.34.1.20260503-py3-none-any.whl", hash = "sha256:75fd66121d56785c91828b8bf7b511f39ba847f11e682573e41847f01e9cd1de", size = 86019, upload-time = "2026-05-03T05:19:49.486Z" },-]-[[package]]name = "types-pyyaml"version = "6.0.12.20260408"diff --git a/wip/archive/plugin-system-implementation-tracker.md b/wip/archive/plugin-system-implementation-tracker.mdnew file mode 100644index 000000000..080247d5e--- /dev/null+++ b/wip/archive/plugin-system-implementation-tracker.md@@ -0,0 +1,596 @@+> **ARCHIVED 2026-06-21.** This tracked the in-`pipelex` plugin-system work (Phases 0–5: invert every core↔integration seam, then externalize Temporal to the private `pipelex-temporal` plugin). Phases 0–4 are complete and committed; Phase 5's local cut-over is done, with only the publish/release-gated downstream pin-flips outstanding. The active tracker is now [`../../TODOS.md`](../../TODOS.md) — the **consumer-side** migration (orchestrator-agnostic `pipelex-api` base + deployment flavors). Kept here for the as-built history and the decision log (D1–D9).++# Pipelex plugin system — implementation TODOS++**Branch:** `refactor/Plugins-2` (worktree `_plugins`) · **Status:** Phases 0–3 complete + **Option A landed** (post-Phase-3). Phase 3 (orchestrators through the seam) = `5f61db323`. **Option A** — drop the CLI-command harvest entirely; Temporal `worker`/`setup-namespace` → standalone `pipelex-temporal` console script — = commit **`989c9beed`**; all gates green (`make agent-check`, `make tb`, `make agent-test` "All tests passed", Temporal suite **156 passed / 4 xpassed**, both `--help` smokes). The former **D3 is superseded** (see D1–D7). Full as-built + open follow-ups: [`wip/plugins/option-a-drop-cli-command-seam.md`](wip/plugins/option-a-drop-cli-command-seam.md) and the **Option A — as-built** note in the log below. **Phase 4 done** (model_lists, the 5th seam, per D6 — full record in `### Phase 4 — as-built` below; all gates green, committing at this checkpoint). With it, **core names no integration across every enumerated seam**; three pre-existing *unenumerated* core→vendor couplings remain, dispositioned in [`wip/plugins/phase-4-residual-core-vendor-couplings.md`](wip/plugins/phase-4-residual-core-vendor-couplings.md). **Phase 5 Step 0b (C6) DONE** — the Temporal config *schema* relocated out of `pipelex.temporal` into core (`pipelex/system/configuration/config_temporal.py` + its two exceptions in `…/exceptions.py`); `configs.py` no longer imports `pipelex.temporal` (**the last hard core→temporal import is gone**). In-`pipelex`, behavior-neutral (byte-equivalent exception semantics), zero cross-repo blast radius; all gates green (`make tb` · `make agent-check` · `make agent-test` "All tests passed"). Record in `### Phase 5 — Step 0b (C6) as-built` below. **Phase 5 LOCAL cut-over DONE = 2026-06-20** (user go/no-go for the reversible local work only): `pipelex-temporal` scaffolded + populated (top-level `pipelex_temporal`, D8) + **green** (`agent-check` 0/0, `agent-test` "All tests passed"); **core green without temporal** (`agent-check` clean, `tb` green, full `agent-test` 7678 passed); Option-A config-loader dead code removed; docs hygiene (install instructions → `pipelex-temporal`). Downstream pin flips **DEFERRED — publish/release-gated** (cut-list in the `### Phase 5 — cross-repo cut-over as-built` note). Local commits both repos, **NOT pushed**. **Next = downstream wiring** — **CORRECTION 2026-06-21: `pipelex-temporal` is PRIVATE / not-open-source → NOT PyPI.** It's distributed by a pinned `git+ssh` ref (precedent: `pipelex-shared @ git+ssh://…/infra-python-tools.git@<ref>` in `pipelex-platform`). Forced split: public MIT consumers (`pipelex-api`, `pipelex-mistralai-workflows`) **DROP** the `temporal` extra; only **private** infra pins `pipelex-temporal` via git+ssh: `pipelex-worker` **and `pipelex-api-hosted`** (confirmed 2026-06-21 — the hosted runner enqueues Temporal, so its child image needs the client too). Core `pipelex` stays public → normal release. Gated on the **hosted-Temporal rollout**; **user chose to DEFER all wiring (docs-only)** 2026-06-21. **Finalize-local pass (session 2, 2026-06-20):** core cleaned of the moved Temporal dev-tooling (both `temporal-e2e-validate` *and* the not-previously-listed `temporal-test-crate` skills relocated to `pipelex-temporal`; dead Makefile temporal targets removed), `pipelex-temporal` additions-side `/code-review` → **SHIP/clean**, review NITs triaged (stray `.pipelex/traces` untracked+gitignored; `test_extras→tests` back-import deferred), all gates green both repos; **local, uncommitted, still paused before publish.** Record: [`wip/plugins/phase-5-cutover-review-followups.md`](wip/plugins/phase-5-cutover-review-followups.md) → *Finalize-local pass*.++> **Delivery model for this branch.** Per the driving goal, all phases land on `refactor/Plugins` as **one commit per checkpoint** (not one PR per phase), and a **single PR to `dev`** opens once the tracker is done. Each phase's "PR: …" box below is satisfied by its checkpoint commit; the standalone-PR-per-phase wording in the original plan is superseded.++This is the **execution tracker**. The *why* and *how* live in [`wip/plugins/implementation-plan.md`](wip/plugins/implementation-plan.md) (the reviewed, decision-locked plan), built from [`wip/plugins/design.md`](wip/plugins/design.md) (the single decided design). **Read the implementation-plan before starting any phase** — this file tracks progress against it, it does not replace it. Background assessment: [`wip/plugins/README.md`](wip/plugins/README.md), plus the per-area notes (`inference-backends-as-plugins.md`, `orchestrators-as-plugins.md`, `temporal-as-plugin.md`).++> **How to use this file.** Tick boxes as you land work. **Do not skip a `🛑 CHECKPOINT`** — each one is a hard stop with three mandatory actions (verify · capture cold-start context · fan-out `/code-review`). The plan is one PR per phase; each phase lands green on its own.++---++## Cold-start primer (read this first if you're new to the session)++**Goal of the whole effort:** invert the *last* coupling between core and its optional integrations so **core names no integration — not by import, not by string — anywhere**. Two families ride one shared seam: inference *driver* plugins (the SDK `match` statements) and orchestrator *strategy* plugins (Temporal, Mistral).++**The seams to invert** (verified present; line numbers drift, locations hold):++| Seam | File | Today |+|---|---|---|+| Inference dispatch (×4) | `cogt/{llm,img_gen,extract,search}/*_worker_factory.py` | `match plugin.sdk:` → lazy import + `find_spec` guard + client cache + build worker |+| ~~Model listing (5th seam)~~ | `cogt/model_backends/model_lists.py` | **INVERTED — Phase 4.** `match sdk` → `get_model_lister_registry().get_optional(sdk=…)`; each backend plugin registers an optional `ListModelsFn` via `add_model_lister`. File names no integration. |+| Orchestrator dispatch | `runtime_bridge/bridge.py` (`match` ~`:147`, `_run_mistral_native` `:408`, `_run_direct` `:270`, `_run_temporal_*` `:297/:330`) | `match execution_mode:` → hard Mistral import + lazy Temporal + per-mode install messages |+| Boot/teardown hub swap | `pipelex.py` (~`:377-479`) | four `if get_config().temporal.is_enabled:` blocks + inlined teardown |+| ~~Config~~ | `system/configuration/configs.py:14` | **RELOCATED — Phase 5 Step 0b (C6).** Schema → `pipelex/system/configuration/config_temporal.py`; its two exceptions (`TemporalConfigError`, `WorkerTaskQueueUnknownError`) → `pipelex/system/configuration/exceptions.py` (re-based on `PipelexError`, byte-equivalent). `configs.py` imports from core — the one hard `pipelex.temporal` import in core config is gone. |+| ~~CLI~~ | `pipelex/temporal/temporal_cli.py` | **REMOVED — Option A.** No CLI-command seam/harvest at all; Temporal's `worker` + `setup-namespace` ship as the standalone `pipelex-temporal` console script ([doc](wip/plugins/option-a-drop-cli-command-seam.md)). |+| Naming overload | `plugins/{plugin,plugin_manager,plugin_sdk_registry,plugin_factory_abstract}.py`, `hub.py` `"PluginManager2 is not initialized"` | three-way "plugin" overload to kill |++**Locked decisions (D1–D7)** — do not re-litigate, see plan for full text:++- **D1** — inference factory = a typed callable `MakeWorkerFn`, not a one-method Protocol object (kills ~30 classes).+- **D2** — Phase 1 PR = seam machinery **merged** with the LLM family (contract meets a real backend before locking).+- **D3** — ~~plugin CLI commands harvested by running the pure `build_registrar` at CLI-build~~ **SUPERSEDED → Option A** ([`wip/plugins/option-a-drop-cli-command-seam.md`](wip/plugins/option-a-drop-cli-command-seam.md), landed post-Phase-3): the CLI-command *contribution path* was **removed**, not hardened. Temporal's `worker` + `setup-namespace` are now a standalone `pipelex-temporal` console script (`[project.scripts]`), so `pipelex --help` no longer loads config or scans entry points and a broken/colliding plugin can't brick the host CLI. The inference-backend / orchestrator / boot-slot seams and `pipelex plugins list` are **unchanged**; `add_cli_command` / `CliCommand` are gone from the registrar. Downstream `pipelex-worker` Dockerfile/Makefile flip is release-gated (see the plan's cross-repo section).+- **D4** — Phase 0 keeps `SdkClientManager` a **pure rename**; collapsing it into the registry is a **P3 follow-up commit**, not bundled.+- **D5** — hub slot claims are **deferred thunks** (`claim_*(factory: Callable[[], Impl])`), invoked only at the setup apply-point → `register` stays import-light (never imports `temporalio`).+- **D6** — the 5th seam (`model_lists.py`) is **inverted too**, as Phase 4, via a model-listing capability on the inference contract.+- **D7** — control surface: discovery is the source of truth for *presence*; add `pipelex plugins list` (observability) + a narrow `plugins.disabled` denylist; **no** redundant allowlist.++**Enablement model** (so you don't add a flag that shouldn't exist): *presence* = in `BUILTIN_PLUGINS` or an installed dist with the `pipelex.plugins` entry point (zero config). *Activation* is decomposed by registry shape — inference backends activate by **data** (a model names the `sdk`); orchestrators select per-call by `execution_mode`; boot-global orchestrators are claimed only when the plugin's own config slice says so (`temporal.is_enabled` = "boot *this process* as a Temporal runtime", a per-process gate, **not** a generic plugin on/off).++**Invariants that must survive every phase:**++- `register(registrar)` is **side-effect-free** — may only call registrar menu methods (no hub touch, no I/O, no client/SDK construction). This is what makes `build_registrar` safely repeatable (D3 runs it at CLI-build *and* boot).+- **Import-light boot** — booting with built-ins registered must not import any optional SDK (`anthropic`, `mistralai`, `google.genai`, `boto3`, `fal_client`, …) or `temporalio`. Enforced by the subprocess import-blocker guard (lands Phase 1).+- **Fail loud** — duplicate `(family, sdk)` / duplicate mode / double-claimed slot / version mismatch / broken entry point each raise a named, contextual error. A missing backend says "install X", never derefs `None`.++**Commands** (from `_plugins/CLAUDE.md`): `make agent-check` (lint/types — always after changes) · `make agent-test` (full suite, silent on success — at end of every phase) · `make tb` (boot/config sanity — fast, run after config or seam edits) · `make agent-test-debug` / `make atd` (when the suite hangs). Always use the venv: `.venv/bin/...`.++**PR boundaries:** Phase 0 · Phase 1 (merged seam+LLM) · Phase 2 (one sub-PR per family) · Phase 3 · Phase 4 — all `pipelex`-only, each lands green. Phase 5 is cross-repo, gated on Checkpoint 3.++---++## Phase 0 — Rename (free the word "plugin")++**Goal:** kill the three-way "plugin" overload. Pure rename, **no behavior change**. Own PR.++- [x] **Grep the blast radius first** (codex C11 — it's bigger than the four factories): `grep -rn 'import Plugin\b' pipelex/` and `grep -rn 'plugin_sdk_registry\|plugin_manager\|PluginManager\|PluginSdkRegistry\|PluginFactoryAbstract' pipelex/`. Note every adapter module (`openai`, `bedrock`, `anthropic`, `gateway`, `portkey` list+factory) and `search_worker_factory.py` (uses the *correct* `plugins.plugin` path while llm/img_gen/extract use the *wrong* `plugins.plugin_sdk_registry` path).+- [x] `Plugin` (`plugins/plugin.py`) → `ModelHandle` (`plugins/model_handle.py`). Keep `make_for_inference_model`. It's a backend *selector* (`sdk`+`backend`+`variant`+`sdk_handle`), not a plugin.+- [x] `PluginManager` (`plugins/plugin_manager.py`) → `SdkClientManager` (`plugins/sdk_client_manager.py`).+- [x] `PluginSdkRegistry` (`plugins/plugin_sdk_registry.py`) → `SdkClientRegistry` (`plugins/sdk_client_registry.py`). Rename `get_sdk_instance`/`set_sdk_instance` → `get`/`set` (the DRY `get_or_create` lands in Phase 1, **not here**).+- [x] `PluginFactoryAbstract` (`plugins/plugin_factory_abstract.py`, the `make_extras` helper — unrelated to the new system) → `BackendExtrasFactory` (`plugins/backend_extras_factory.py`). Name-hygiene; it collides with the new vocabulary.+- [x] `hub.py`: `get_plugin_manager`/`set_plugin_manager` → `get_sdk_client_manager`/`set_sdk_client_manager`; **fix the `"PluginManager2 is not initialized"` message** → `"SdkClientManager is not initialized"`.+- [x] `pipelex.py`: update setup/teardown call sites.+- [x] Fix the wrong-path imports (llm/img_gen/extract → corrected registry path) — free win during the rename.+- [x] **D4 guard:** `SdkClientManager` stays a thin wrapper holding one `SdkClientRegistry`. Do **not** collapse it here — that's a tracked P3 follow-up (see "Deferred / out of scope").++**Tests:** no new tests. Existing suite + `make tb` stay green.++### 🛑 CHECKPOINT 0 — hard stop++- [x] **Verify:** `make agent-check` clean · `make tb` green · `make agent-test` green. Confirm `PluginManager2` artifact gone (`grep -rn PluginManager2 pipelex/` → empty) and wrong-path `Plugin` imports fixed.+- [x] **Capture cold-start context:** see `### Phase 0 — as-built` in the As-built log.+- [x] **Fan-out `/code-review`:** sub-agent ran `/code-review --fix` on the Phase 0 diff (pure-rename scope). Findings triaged in the as-built note.+- [x] **PR:** superseded — committed on `refactor/Plugins` (single PR to `dev` opens after the tracker is done).++---++## Phase 1 — Seam machinery + LLM family (merged, per D2)++**Goal:** build discovery + contract + registry machinery **and** migrate the LLM family through it in one PR, so the contract is proven against a real, messy family before it locks. No behavior change. Own PR.++> ✅ **DONE — committed on `refactor/Plugins`.** Full record in **`### Phase 1 — as-built`** (As-built log). The unchecked boxes below are the original plan items; all are delivered (or noted as deferred to Phase 2/3) in the as-built.++**New modules (all import-light — must not import any backend SDK or `temporalio`):**++- [ ] `plugins/contract.py` — `PipelexPlugin` (`@runtime_checkable Protocol`: `name: str`, `targets_api: int`, `register(self, registrar) -> None`) + `PLUGIN_API_VERSION: int = 1`. Document the **side-effect-free `register`** invariant here.+- [ ] `plugins/registrar.py` — `PluginRegistrar` accumulator: inference registry, orchestrator registry, **deferred** hub-slot-claim thunks, CLI-command list, teardown-callback list, read-only `config: ConfigRoot`. Menu methods:+ - [ ] `add_inference_backend(*, family, sdk, make_worker: MakeWorkerFn)` (D1 — callable).+ - [ ] `add_orchestrator(*, mode, orchestrator)`.+ - [ ] `claim_content_generator | pipe_router | pipe_run | task_manager(factory: Callable[[], Impl])` (D5 — **thunk, not instance**).+ - [ ] `add_cli_command(*, name, help, command)`, `add_teardown(callback)`.+- [ ] `plugins/inference_backend_registry.py` — `InferenceFamily` (StrEnum: `LLM`/`IMG_GEN`/`EXTRACT`/`SEARCH`), `MakeWorkerFn` TypeAlias, `InferenceBackendRegistry` keyed by `(family, sdk)`. Signature: `make_worker(*, inference_model, backend, sdk_clients, reporting_delegate)`.+- [ ] `plugins/orchestrator_registry.py` — `OrchestratorProtocol` (`async def run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput`), `OrchestratorRegistry` keyed by `PipelexExecutionMode`.+- [ ] `plugins/discovery.py` — `build_registrar(*, config) -> PluginRegistrar`: **pure function** iterating `BUILTIN_PLUGINS` + `importlib.metadata.entry_points(group="pipelex.plugins")`, version-checking each, calling `plugin.register(registrar)`, skipping any `name` in `config.plugins.disabled` (logged; raise if a core-unconditional plugin is denylisted), fail-loud. Pure ⇒ safe at CLI-build and boot.+- [ ] `plugins/builtins.py` — `BUILTIN_PLUGINS`. Phase 1 fills it with the LLM drivers.+- [ ] `plugins/exceptions.py` — `PluginApiVersionMismatchError`, `DuplicateInferenceBackendError`, `DuplicateOrchestratorError`, `HubSlotAlreadyClaimedError`, `BrokenPluginError`. (`MissingOrchestratorError` lives in `runtime_bridge/exceptions.py` — added in Phase 3.)++**DRY the two repeated blocks (design §6.2):**++- [ ] `SdkClientRegistry.get_or_create(*, handle, build)` replaces the `get_sdk_instance(...) or set_sdk_instance(...)` dance in every arm.+- [ ] `require_sdk(*, spec, extra, msg)` (raises `MissingDependencyError` if `find_spec(spec) is None`) replaces the repeated guard — moved **into** `make_worker` so a missing extra fails at use, not boot.++**LLM family migration:**++- [ ] Collapse `llm_worker_factory.py`'s `match` to `get_inference_backend_registry().lookup(family=LLM, sdk=model_handle.sdk)(...)`. A registry miss → distinct "backend `<sdk>` not registered; is its plugin installed?" error.+- [ ] Each LLM driver plugin's `register` calls `add_inference_backend(family=LLM, sdk="anthropic", make_worker=<closure>)`. Add them to `BUILTIN_PLUGINS`.++**Boot wiring:**++- [ ] In `Pipelex.setup()`, after config is fully resolved (incl. the `temporal_enabled` override at `pipelex.py:193-196`) and before the hub setup points, call `build_registrar(config=get_config())`. Store the two keyed registries on the hub (`set_inference_backend_registry` / `set_orchestrator_registry`). Hold slot-claim thunks, CLI commands, teardown callbacks for their apply-points.+- [ ] **Conflict & version policy (fail loud):** duplicate `(family, sdk)` / duplicate mode / double-claimed slot each raise naming **both** contributors; `targets_api != PLUGIN_API_VERSION` → `PluginApiVersionMismatchError` with the remedy; a broken entry point → `BrokenPluginError` (the one sanctioned `except Exception` site — annotate it).++**Control surface (D7):**++- [ ] Add a core-owned `plugins: PluginsConfig` section (`disabled: list[str]`) to `configs.py`; default `[]` in `pipelex/pipelex.toml` **and** `.pipelex/pipelex.toml` (never a class-def default — repo rule). Apply the denylist in `build_registrar` (skip + log; error on a core-unconditional name).+- [ ] Add the core `pipelex plugins list` command — lists discovered plugins (built-in vs external), what each registered, denylist state.++**Tests (red-green):**++- [ ] Contract conformance (synthetic plugins): protocol satisfaction; `targets_api` mismatch fails loud; duplicate `(family, sdk)` / mode / slot each raise naming both; broken entry point → `BrokenPluginError`; **`build_registrar` idempotent** (two calls, no side effect — pins the side-effect-free invariant).+- [ ] Denylist (D7): a discovered non-core plugin in `plugins.disabled` is skipped + logged; denylisting a core-unconditional plugin raises at startup; `pipelex plugins list` shows built-ins + synthetic externals with contributions + denylist state.+- [ ] **Import-light boot** (codex C12): subprocess with a `sys.meta_path` finder that *raises* on `anthropic`/`mistralai`/`google.genai`/`boto3`/`fal_client`/… → boot with built-ins → assert no raise.+- [ ] Registry round-trip (LLM): register → `lookup(LLM, sdk)` → right `LLMWorkerAbstract`.+- [ ] Error parity: an Anthropic model with the `anthropic` extra absent raises the *same* `MissingDependencyError` text (lib + extra + alternative).++### 🛑 CHECKPOINT 1 — hard stop (contract locks here)++- [x] **Verify:** `make agent-check` clean · `make tb` green · `make agent-test` green. Import-light subprocess guard green; the LLM `match` is gone (replaced by a registry lookup).+- [x] **Capture cold-start context:** see `### Phase 1 — as-built` in the As-built log.+- [x] **Fan-out `/code-review`:** sub-agent ran `/code-review --fix` on the Phase 1 diff (see as-built for triage).+- [x] **PR:** superseded — committed on `refactor/Plugins` (single PR to `dev` opens after the tracker is done).++---++## Phase 2 — Remaining inference families (ImgGen / Extract / Search)++**Goal:** migrate the other three families onto the proven contract. **One family per sub-PR** (lanes A/B/C are independent — can run in parallel worktrees).++> **Pre-work analysis (done — read this before starting; the three factories were already audited so you don't have to re-read them).** Each closure follows the Phase 1 LLM shape: `def _make_x(*, inference_model, backend, sdk_clients, reporting_delegate) -> InferenceWorkerAbstract`, derives `model_handle` from `inference_model`, lazy-imports its factory/worker (`# noqa: PLC0415`), guards optional SDKs with `require_sdk(...)` inside the closure, and caches via `sdk_clients.get_or_create(handle=model_handle, build=lambda: ...)`. Then the family factory (`ImgGen/Extract/Search WorkerFactory`) collapses its `match` to `get_inference_backend_registry().lookup(family=<FAM>, sdk=model_handle.sdk)(...)` exactly like `llm_worker_factory.py` already does, passing `sdk_clients=get_sdk_client_manager().sdk_client_registry` and `cast`ing the result to the family worker abstract. Registry-miss `NotImplementedError` message comes from `InferenceBackendRegistry.lookup` (the per-family "is not supported" / "for image generation" / "for search" wording in the old `case _:` arms goes away — update those families' `test_*_worker_factory.py` unknown-sdk asserts to the new "Is its plugin installed?" wording, as was done for LLM).+>+> **SDK → (vendor plugin · worker · client build) map:**+>+> *IMG_GEN* (`img_gen_worker_factory.py`):+>+> - `gateway_img_gen` → **gateway** plugin · `GatewayImgGenWorker` · `GatewayFactory.make_portkey_client(backend=backend)`+> - `gateway_completions` → **gateway** · `OpenAICompletionsImgGenWorker(openai_completions_factory=GatewayCompletionsFactory(is_http_url_enabled=False))` · `GatewayCompletionsFactory.make_portkey_openai_client_for_completions(model_handle, backend)`+> - `openai_img_gen` → **openai** · `OpenAIImgGenWorker` · `OpenAIClientFactory.make_openai_client(model_handle, backend)`+> - `google` → **google** · `GoogleImgGenWorker` · `GoogleFactory.make_google_client(backend)` · `require_sdk(spec="google.genai", dependency_name="google-genai", extra="google", msg="The google-genai SDK is required in order to use Google Gemini Image models.")`+> - `fal` → **NEW `fal` plugin** · `FalImgGenWorker` · `fal_client.AsyncClient(key=backend.api_key)` (import `from fal_client import AsyncClient`) · `require_sdk(spec="fal_client", dependency_name="fal-client", extra="fal", msg="The fal-client SDK is required in order to use FAL models (generation of images).")`+> - `huggingface_img_gen` → **NEW `huggingface` plugin** · `HuggingFaceImgGenWorker` · `AsyncInferenceClient(provider=provider_literal, token=backend.api_key)` where `provider_literal = HuggingFaceFactory.make_huggingface_inference_provider(provider_str=model_handle.variant)` if `model_handle.variant` else `"auto"` · **C10 BUG FIX: add `require_sdk(spec="huggingface_hub", extra="huggingface", msg="The huggingface_hub SDK is required in order to use HuggingFace image generation models.")`** (the old arm had no guard) — and add the huggingface error-parity test.+> - `blackboxai_img_gen` → **NEW `blackboxai` plugin** · `OpenAICompletionsImgGenWorker(openai_completions_factory=BlackboxaiCompletionsFactory(is_http_url_enabled=True))` · `OpenAIClientFactory.make_openai_client(model_handle, backend)`+> - `openrouter_img_gen` → **NEW `openrouter` plugin** · `OpenAICompletionsImgGenWorker(openai_completions_factory=OpenRouterCompletionsFactory(is_http_url_enabled=True))` · `OpenAIClientFactory.make_openai_client(model_handle, backend)`+> - `azure_rest_img_gen` → **NEW `azure_rest` plugin** · `AzureImgGenWorker(model_handle=model_handle, inference_model=..., reporting_delegate=...)` — **DIRECT construction: no `sdk_clients`/`get_or_create`** (its `test_img_gen_worker_factory.py` asserts `registry.root == {}`).+>+> *EXTRACT* (`extract_worker_factory.py`) — workers take `extra_config=backend.extra_config`:+>+> - `gateway_extract` → **gateway** · `GatewayExtractWorker` · `GatewayFactory.make_portkey_client(backend)`+> - `mistral` → **mistral** · `MistralExtractWorker` · `MistralFactory.make_mistral_client(backend)` · `require_sdk(spec="mistralai", extra="mistral", msg="The mistralai SDK is required in order to use Mistral OCR models through the mistralai client.")`+> - `docling_sdk` → **NEW `docling` plugin** · `DoclingExtractWorker` · `DoclingFactory.make_docling_sdk()` · `require_sdk(spec="docling", extra="docling", msg="The docling library is required in order to use Docling for PDF and image text extraction.")`+> - `pypdfium2` → **NEW `pypdfium2` plugin** · `Pypdfium2Worker(extra_config=..., inference_model=..., reporting_delegate=...)` — **STATELESS: no `get_or_create`, no `require_sdk`** (built directly).+> - `linkup_fetch` → **`linkup` plugin** · `LinkupExtractWorker(extra_config=...)` — **STATELESS (no `get_or_create`)** · `require_sdk(spec="linkup", extra="linkup", msg="The linkup SDK is required in order to use Linkup Fetch extraction models.")`+>+> *SEARCH* (`search_worker_factory.py`) — **C9 normalization: `make_search_worker` currently takes NO `reporting_delegate` and pulls `get_report_delegate()` from the hub twice. Change its signature to accept `reporting_delegate` and pass it through (drop the hub coupling); update its callers.** Closures then receive `reporting_delegate` like every family.+>+> - `linkup` → **`linkup` plugin** · `LinkupSearchWorker(inference_model=..., reporting_delegate=...)` — STATELESS (no client/`get_or_create`).+> - `gateway_search` → **gateway** · `GatewaySearchWorker` · `GatewayFactory.make_portkey_client(backend)`+>+> **NEW vendor plugin modules to create:** `fal`, `huggingface`, `blackboxai`, `openrouter`, `azure_rest`, `docling`, `pypdfium2`, `linkup` (each `pipelex/plugins/<vendor>/<vendor>_plugin.py`, added to `BUILTIN_PLUGINS`). **EXTEND existing plugins** (one `register` adds across families — the cross-family-vendor coordination point): `openai` (+IMG_GEN), `gateway` (+IMG_GEN ×2, +EXTRACT, +SEARCH), `google` (+IMG_GEN), `mistral` (+EXTRACT), `linkup` (EXTRACT + SEARCH — one new plugin serving two families). None of these new SDKs are core-unconditional (only `openai` is). **Build-time check:** `azure_rest`/`pypdfium2`/`linkup`/blackboxai/openrouter closures must stay import-light at module load (lazy factory/worker imports), and the import-light subprocess guard's BLOCKED set should grow to cover `fal_client`, `huggingface_hub`, `docling`, `linkup` so the guard actually proves it.++- [x] **ImgGen (Lane A).** Collapse the `match`. **Fix the pre-existing huggingface bug (codex C10):** `img_gen_worker_factory.py` imports `huggingface_hub` with **no `find_spec` guard` → a missing extra raises raw `ImportError` instead of `MissingDependencyError`. Add `require_sdk(spec="huggingface_hub", extra="huggingface", …)` during migration. Provider-literal handling (`model_handle.variant` → provider) goes inside the closure. Substrate-reuse workers (`blackboxai`/`gateway_completions`/`openrouter` → `OpenAICompletionsImgGenWorker`) become closures capturing the per-vendor completions factory.+- [x] **Extract (Lane B).** Collapse the `match`. Stateless arms (`pypdfium2`, `linkup_fetch`) skip `get_or_create`.+- [x] **Search (Lane C) — normalize the call surface (codex C9).** `search_worker_factory.py` takes **no `reporting_delegate` param** and pulls `get_report_delegate()` from the hub; the uniform `MakeWorkerFn` passes it explicitly. Normalize `make_search_worker` + callers to accept `reporting_delegate` (removes search's hidden hub coupling — a strict improvement), so search fits the uniform signature.+- [x] Add each migrated driver to `BUILTIN_PLUGINS`. OpenAI is an always-on built-in driver, no privileged arm. The OpenAI-compat *substrate* stays an in-tree library the gateway/portkey/openrouter/blackboxai closures import (substrate extraction into a named module is deferred).++**Tests:**++- [x] Registry round-trip per family.+- [x] Import-light subprocess guard now covers all optional SDKs.+- [x] **huggingface error-parity** (missing `huggingface` extra now raises `MissingDependencyError` — proves the bug fix).+- [x] **Cross-family vendor** (e.g. `mistral` registers into both LLM and Extract from one `register`).++**Cross-cutting deliverable (ships with this phase):**++- [x] **Inference SPI reference** (design §9.1) + **plugin-authoring guide** + **minimal example backend plugin** — delivered as one comprehensive page `docs/under-the-hood/inference-backend-plugins.md` (seam walkthrough + SPI module/symbol table + authoring guide with a copy-pasteable `acme` example, the entry-point declaration, the denylist, and the fail-loud table), added to both mkdocs nav blocks under "Under the Hood". A standalone runnable example *package* was judged disproportionate for this checkpoint (the embedded example is complete and copy-pasteable) — flag as a follow-up if a separate example repo is wanted.++### 🛑 CHECKPOINT 2 — hard stop (natural session handoff — inference done)++- [x] **Verify:** `make agent-check` clean · `make agent-test` green (exit 0). All four worker-factory `match` statements gone, every backend goes via the registry, the huggingface guard bug is fixed (require_sdk added), and search's hub coupling is removed (factory takes `reporting_delegate`, caller supplies `get_report_delegate()`).+- [x] **Capture cold-start context:** `### Phase 2 — as-built` note below — vendor plugins created/extended, cross-family coordination points, the search C9 normalization, test + docs locations.+- [x] **Fan-out `/code-review`:** `pr-review-toolkit:code-reviewer` sub-agent reviewed the full Phase 2 working-tree diff, comparing every new closure against the pre-Phase-2 `match` arm via `git show HEAD:…`. **Verdict: clean — no blockers, no should-fix, no nits.** It confirmed byte-equivalent parity (worker classes, ctor kwargs, `is_http_url_enabled` flags), the C10 require_sdk fix + TYPE_CHECKING-only `PROVIDER_OR_POLICY_T`, import-light (no SDK at any plugin module top), every `# noqa: ARG001` justified by the old arm, C9 production-safety (real caller still supplies the delegate; the two integration tests assert on results not reporting so `None` is harmless), and cross-family non-collision (distinct `(family, sdk)` keys + distinct closure names).+- [x] **Commit:** single checkpoint commit on `refactor/Plugins` (per the branch's one-commit-per-checkpoint delivery model). **Good point to end the session** before opening the orchestrator phase.++---++## Phase 3 — Orchestrators through the seam (in-tree) — HIGHEST RISK++**Goal:** collapse the bridge `match` + the four `temporal.is_enabled` boot blocks into registry/slot lookups; move Temporal's modes/CLI/boot-swap/teardown behind its (still-in-repo) plugin; flip Mistral's hard import to discovery; publish the orchestrator SPI. No behavior change. Own PR.++> **Spike the CLI-timing + thunk integration (D3+D5) FIRST** — it's the one genuinely novel integration point. Prove `build_registrar` at CLI-build harvests `worker` into `--help` without constructing a Temporal impl, before doing the rest.++**Extract orchestrators (verbatim bodies):**++- [x] `DirectOrchestrator.run` from `_run_direct` → `runtime_bridge/direct_orchestrator.py`. **Landmine kept verbatim:** `with scoped_pipe_router(PipeRouter())`. Registered in core by `plugins/direct/direct_plugin.py` (`name="direct"`, in `CORE_UNCONDITIONAL_PLUGIN_NAMES`).+- [x] `TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` from `_run_temporal_*` → `temporal/temporal_orchestrators.py`, keeping the `WorkflowExecutionError` catch + `make_workflow_id` recompute. Each `run` first calls `_require_temporal_extra(mode=...)` (find_spec → `MissingOrchestratorError` if the extra is absent).+- [x] (`MistralWorkflowsOrchestrator` authored in `pipelex-mistralai-workflows` — Phase 5.)++**Bridge + Mistral:**++- [x] Collapsed the bridge `match` → `get_orchestrator_registry().get_optional(mode=...)` → `orchestrator.run`, else `MissingOrchestratorError(mode)`. `bridge.py` names no integration (only docstrings/comments mention temporal/mistral). Serialize helpers extracted to `runtime_bridge/serialization.py` (breaks the `bridge→bootstrap→pipelex` cycle that the orchestrators would otherwise re-form); `serialize_pipe_output` re-exported from bridge for back-compat.+- [x] **Per-mode error quality (C7):** `MissingOrchestratorError(*, mode)` maps mode→exact message via match/case (temporal modes → `pip install 'pipelex[temporal]'`; mistral → `pip install pipelex-mistralai-workflows`; direct → boot-problem msg). `_authors_caller_facing_message=True`.+- [x] **Mistral → discovery:** the `_run_mistral_native` hard import is gone. MISTRAL_NATIVE has no in-core orchestrator → registry miss → `MissingOrchestratorError(MISTRAL_NATIVE)`. Suite proves dispatch with a fake orchestrator (`test_orchestrator_dispatch.py`).++**The Temporal plugin (`temporal/temporal_plugin.py`), its `register`:**++- [x] **always** `add_orchestrator(TEMPORAL_BLOCKING|TEMPORAL_FIRE_AND_FORGET, …)` + `add_cli_command(worker|setup-temporal-namespace, import_path=…)`.+- [x] **if `config.temporal.is_enabled`:** `claim_content_generator|task_manager|pipe_router|pipe_run(thunk)` + `add_teardown(thunk)` — each a module-level zero-arg thunk that lazy-imports `temporalio`; `register` imports no temporalio (proven by the import-light guard with `is_enabled=True`).++**Boot/teardown collapse (C8 — explicit injection precedence):**++- [x] The four `if temporal.is_enabled:` boot blocks → `_resolve_hub_slot(slot, default)` (generic over `_HubSlotImplT`) at each ordered point. Precedence: explicit `setup()` param > slot-claim thunk > core default. TASK_MANAGER runs its thunk (full temporal-hub wiring); no core default/param. Pinned by `test_hub_slot_injection_precedence.py`.+- [x] The inlined teardown → registered teardown callbacks run **LIFO** (`for cb in reversed(self._plugin_registrar.teardown_callbacks)`). `pipelex.py` teardown names no integration. `temporal_hub.get_optional_task_manager()` added so the temporal teardown thunk re-fetches the tm.++**CLI collapse (C4 + D3):**++- [x] The two hard temporal-command imports + static registrations + hardcoded `list_commands` entries are gone. `_cli.py` harvests plugin commands at module load via `build_registrar` (`_register_discovered_cli_commands`), importing each `import_path` lazily; `list_commands` returns `_CORE_COMMAND_ORDER + _PLUGIN_COMMAND_NAMES`.+- [x] CLI-build config load is side-effect-free (`config_manager.load_config(ensure_global_if_missing=False)`, new param) with a bulletproof `load_base_config_dict()` fallback so a broken/absent user config can't brick `--help`/`init`. D5 thunks ⇒ harvest constructs no temporal impl (proven: `worker` harvested with temporalio import-blocked).++**SPI:**++- [x] Orchestrator SPI published: `docs/under-the-hood/orchestrator-plugins.md` (seam, contract, boot-orchestrator slot claims, injection precedence, CLI-by-import-path, the SPI module/symbol table from design §9.2, the Temporal worked example, out-of-tree authoring) — added to both mkdocs nav blocks. `MissingOrchestratorError` added to `runtime_bridge/exceptions.py`; `MissingPipelexTemporalExtraError` + `MissingMistralWorkflowsPluginError` removed (error pages regenerated).++**Tests:**++- [x] Bridge dispatch by mode (fake orchestrator) — `test_orchestrator_dispatch.py`.+- [x] **Per-mode error parity** — `test_orchestrator_dispatch.py` (per-mode miss) + `test_exceptions_disclosure.py` (per-mode hint survives STRICT).+- [x] DIRECT router scoping (`test_direct_router_scoping.py`, unchanged + green) + DIRECT parity (`test_dispatch.py`, `test_bridge_direct.py`).+- [x] **Injection-precedence** (explicit param wins over slot claim — C8) — `test_hub_slot_injection_precedence.py`.+- [x] Boot-via-slots (temporal-enabled → content-generator slot resolves to `ContentGeneratorInWorkflow`) — existing `test_keyless_boot_forced_dry.py::test_keyless_temporal_boot_*` (green through the new seam) + slot-claim arm in the precedence test; teardown LIFO pinned there.+- [x] **CLI-command contribution** (`worker`/`setup-temporal-namespace` harvested, in order) — `test_plugin_cli_command_harvest.py`. Import-light guard extended with `temporalio` + `is_enabled=True` slot-claim arm.+- [x] **Full Temporal suite green in-tree** (§14.5 — the extraction gate) — `.venv/bin/pytest tests/integration/pipelex/temporal/` → **156 passed, 4 xpassed** (the xpassed are pre-existing xdist class-registration flakiness markers, unrelated to this phase); also green inside the full `make agent-test`.++### 🛑 CHECKPOINT 3 — hard stop (THE gate before externalization)++- [x] **Verify:** `make agent-check` clean · `make agent-test` green ("All tests passed", exit 0) · **full Temporal suite green through the seam** (`.venv/bin/pytest tests/integration/pipelex/temporal/` → 156 passed, 4 xpassed). bridge + boot/teardown + CLI name **no** integration (only docstrings/comments mention temporal/mistral; dispatch/boot/teardown go via the registry + slot thunks + config).+- [x] **Capture cold-start context:** `### Phase 3 — as-built` note above — dispatch seam, verbatim extractions, the cycle-break (serialization.py + `import_path` CLI), injection-precedence ordering + where slot thunks apply, the CLI harvest mechanism, the orchestrator SPI doc, and the §14.5 green evidence.+- [x] **Fan-out `/code-review`:** `pr-review-toolkit:code-reviewer` reviewed the full Phase 3 diff against `HEAD` — **clean, no BLOCKERs**; byte-equivalence + all landmines verified. S1/N1/N2 applied, N3 deliberate (see the CHECKPOINT-3 `/code-review` note in the as-built).+- [x] **Commit:** checkpoint commit `19e6ca66b` on `refactor/Plugins-2` (one-commit-per-checkpoint delivery model). **Do NOT begin Phase 5** until this Temporal-green gate is recorded (it is, above). Phase 4 (model_lists) may start next.++---++## Phase 4 — Invert model listing (the 5th seam, per D6)++**Goal:** close the goal. `model_lists.py`'s `match sdk:` (the `list-models` CLI path) still hardcodes `pipelex.plugins.{openai,anthropic,mistral,google,bedrock}.*_list` imports. Invert it so core names no integration *anywhere*. Own PR.++- [x] Add a **model-listing capability** to the inference contract: `add_model_lister(*, sdk, lister: ListModelsFn)` (callable mirroring `MakeWorkerFn` — import-light, lazy). A backend plugin that lists models registers its lister alongside its worker factory.+- [x] Collapse `ModelLister.list_models` (`model_lists.py`) to a registry lookup keyed by `sdk`; a miss → the same friendly "is its plugin installed?" guidance. Move the `find_spec` guards (already in this file) into each lister.+- [x] The contract grows by one **optional** method — backends without listing simply don't register a lister (progressive disclosure preserved).++**Tests:**++- [x] Round-trip (a registered lister is invoked for its `sdk`).+- [x] A backend with no lister → the friendly miss.+- [x] Import-light subprocess guard still green (listers are lazy).++### 🛑 CHECKPOINT 4 — hard stop ("core names no integration" holds without an asterisk)++- [x] **Verify:** `make agent-check` clean (pyright 0 over 2243 files / mypy 0 / keyword-only pass) · `make agent-test` green ("All tests passed.") · targeted plugins+cogt suite 715 passed. `model_lists.py` names no integration; the final sweep returns only **pre-existing, out-of-scope** vendor refs (`config_cogt.py` typed config = by-design D7; `img_gen_args_factory.py` + `backend_factory.py` = genuine but unenumerated couplings) — captured in [`wip/plugins/phase-4-residual-core-vendor-couplings.md`](wip/plugins/phase-4-residual-core-vendor-couplings.md). No dispatch-path leak remains.+- [x] **Capture cold-start context:** `### Phase 4 — as-built` note below.+- [x] **Fan-out `/code-review`:** `pr-review-toolkit:code-reviewer` reviewed the Phase 4 working-tree diff against HEAD. **Verdict: byte-equivalent and clean — no BLOCKER / no SHOULD-FIX.** It walked all five correctness questions against `git show HEAD:…model_lists.py` and confirmed: each lister's kwargs match its old `match` arm (mistral's no-`backend` + sync `# noqa: RUF029`, bedrock sync, the rest async); the deleted `find_spec` guards survive **byte-for-byte** inside each `list_*_models` fn (same lib/extra/message; openai has none in old or new); the anthropic `AnthropicSDKUnsupportedError`→`ModelListingUnsupportedError` translation is correct and core imports no anthropic symbol; `any_listed` threads identically (not set on the unsupported `continue`); the registered sdk keys are exactly the old arms' set. Plus import-light, `get_optional`-vs-`lookup`, keyword-only, and the Case-2 boundary (with `except PipelexCLIError: raise` ordering preserved so internal auth `PipelexCLIError`s aren't re-wrapped). One **NIT** (doc recommended `require_sdk` while the built-in listers reuse their `list_*_models` inline `find_spec` guard) — **applied**: the "Listing models" doc section now tells one story (recommends `require_sdk`, notes the built-ins reuse the existing inline guard). The wip residual-couplings doc was confirmed accurate. A follow-up **xhigh workflow `/code-review`** (10 finder angles → adversarial verify) re-confirmed byte-equivalence and surfaced reuse/altitude follow-ups: the two cheap ones were **applied** (a shared `PluginRegistrar._add` helper mirroring `_claim`; the hardcoded item-counts removed from this as-built), the four design-tradeoffs **deferred** — all recorded in [`wip/plugins/phase-4-review-followups.md`](wip/plugins/phase-4-review-followups.md).+- [x] **PR:** checkpoint commit on `refactor/Plugins-2` (one-commit-per-checkpoint). Phases 0–4 complete — the in-`pipelex` work is done.++---++## Phase 5 — Externalize Temporal → `pipelex-temporal` (cross-repo, GATED on Checkpoint 3)++**Goal:** lift `pipelex/temporal/` into a separate dist. A *packaging* move — but only after a prerequisite the design missed. **Cross-repo; the only phase that can break downstream deploys. Separate effort.**++> **Destination repo exists.** The user created an **empty** `pipelex-temporal` repo (sibling dir at `../pipelex-temporal`, remote `[email protected]:Pipelex/pipelex-temporal.git`, branch `main`, **no commits / no scaffolding** yet) ahead of this phase, so the move has a target from the first commit. **Do this in a fresh session** (cross-repo; not a normal in-`pipelex` checkpoint).+>+> **Progress:** the in-`pipelex` prerequisite **Step 0b (C6) is DONE** (committed; behavior-neutral; gates green). What remains is the **cross-repo cut-over** — Step 0a (scaffold the empty repo), the breaking move of `pipelex/temporal/` out, the downstream pin flips, and publishing. Those are outward-facing/deploy-breaking and were intentionally **paused for user go/no-go** after Step 0b.++> **Cut-over in progress (local-only, paused before publish — user go/no-go given 2026-06-20 for the *reversible* local work; publish/merge/deploy still gated).** Two foundational decisions locked this session:+> - **D8 — package layout = top-level `pipelex_temporal`** (NOT a `pipelex.temporal` namespace stitch). Rationale: `pipelex` is a *regular* package (owns `pipelex/__init__.py`, hatchling `packages=["pipelex"]`), so two wheels can't cleanly co-own `pipelex/`; the family convention is top-level (`pipelex-api`→`api`, the existing plugin→`pipelex_mistralai_workflows`); and two *editable* installs both mapping into `pipelex/` is fragile, which would break the local cross-pin testing this scope needs. Cost: mechanical `pipelex.temporal.*` → `pipelex_temporal.*` rewrite across the moving code + its tests (which travel anyway) + ~9 import lines in `pipelex-mistralai-workflows` tests + the entry point. Supersedes the design's "keep the import path / lowest-churn" lean (written before the regular-package packaging reality was confronted).+> - **D9 — scaffold tool-configs are sourced from `pipelex`'s own `pyproject.toml`** (ruff/pyright/mypy/pytest/coverage — the moved code passes these verbatim today), while repo *shape* (Makefiles/, metadata, `.gitignore`, LICENSE, family feel) mirrors `pipelex-api`. Avoids lint churn on the moved code.+>+> **Recon facts (three agents, 2026-06-20):** core→temporal coupling is a *single* hard import (`pipelex/plugins/builtins.py:18,40` → `TemporalPlugin`); everything else core-side is config (`temporal: Temporal` field, `[temporal]` toml, `--temporal/--no-temporal` flags) that correctly stays. temporal→core spans ~46 modules (`cogt.*`, `core.*`, `runtime_bridge.*`, `pipe_run.*`, `pipeline.*`, `hub`, `tools.*`) ⇒ `pipelex-temporal` depends on base `pipelex`. Downstream: `pipelex-worker` = pin + `pipelex worker` CLI (no temporal imports); `pipelex-mistralai-workflows` = `temporal` extra + ~9 `pipelex.temporal.*` test imports; `pipelex-api-hosted` = config-only wrapper, **no Python pin** (temporal reaches it through the publish-gated `pipelex-api` image). `pipelex-api` itself pins `pipelex[…,temporal]==X` — a publish-gated flip, out of this local scope.+>+> **Local cross-pin strategy:** `pipelex-temporal` depends on editable `pipelex` via `[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }` (mirrors how `pipelex-mistralai-workflows` pins `../_workflows`); downstream repos pin editable `../pipelex-temporal`. The published `==Y` pins are the publish-time flip (deferred). Order constraint: drop the `pipelex-temporal` console script from `_plugins/pyproject.toml` **before** any `uv sync` in the new repo (else two `pipelex-temporal` scripts collide).++- [x] **Step 0a — scaffold `pipelex-temporal` to match the sibling Python repos' conventions. ✅ DONE** (D9; see cut-over as-built). Don't invent a new toolchain — mirror the habits already used across the workspace's Python repos (`pipelex-platform` / `pipelex-admin-api` / `pipelex-api` are the references): same Makefile targets (`agent-check`, `agent-test`, `format`, `lint`), ruff (format + lint) + pyright + mypy strict, pytest layout, `uv` for deps, hatchling build backend, the standard `pyproject.toml` shape + CHANGELOG. The scaffold should feel like it belongs in the family before any Temporal code lands in it.+- [x] **Step 0b — relocate the Temporal config schema to a core-owned module (codex C6 — the prerequisite). ✅ DONE** (in-`pipelex`, behavior-neutral, committed at this checkpoint). Schema → `pipelex/system/configuration/config_temporal.py`; the two exceptions it needs (`TemporalConfigError`, `WorkerTaskQueueUnknownError`) → `pipelex/system/configuration/exceptions.py`, re-based on `PipelexError` (byte-equivalent: same `error_domain=None`, `ValueError` mixin intact, subclasses preserved). All importers repointed; `configs.py:14` imports from core; the `if TYPE_CHECKING: … RetryPolicy = Any` placeholder kept. Gates green (`make tb` · `make agent-check` pyright 0/mypy clean · `make agent-test` "All tests passed"); zero external importers of the moved symbols. Full record + the now-fixed `(ValueError, PipelexError)` `.message` latent bug (reordered to `(PipelexError, ValueError)` in an `xhigh`-review follow-up): `### Phase 5 — Step 0b (C6) as-built` below. *(Was: low-risk, could be pulled into Phase 3.)*+- [x] Move `pipelex/temporal/` (impl) + `temporal_plugin.py` + tests + the `temporal` marker + the `--temporal-server` conftest option into `pipelex-temporal`; declare `[project.entry-points."pipelex.plugins"]`. **✅ DONE** (top-level `pipelex_temporal`, D8).+- [x] Keep protocol-level conformance tests in core; Temporal's behavioral suite travels. **✅ DONE** (incl. the 5 mixed-test splits + 2 config tests kept in core — see cut-over as-built).+- [ ] **Flip downstream pins** (cross-repo blast radius): `pipelex-worker`, `pipelex-api-hosted` from `pipelex[temporal]==X` → `pipelex-temporal==Y`. **⏸ DEFERRED — publish/release-gated** (precise cut-list in the cut-over as-built; per Option-A the `pipelex-worker` flip lands in the same commit as the release pin bump).+- [ ] **Repoint `pipelex-mistralai-workflows`** (design §15): its `temporal` extra + `pipelex.temporal.*` imports → `pipelex-temporal`. **⏸ DEFERRED — publish-gated + coordinate with that repo's in-flight mistralai-2.x work.**++**Tests:**++- [ ] Re-run the relocated Temporal suite against published `pipelex`.+- [ ] Run downstream consumers' suites against the new pins **before** publishing.++### 🛑 CHECKPOINT 5 — hard stop (final)++- [ ] **Verify:** Temporal is an external plugin; core has no `temporal` extra; downstream pins flipped; consumer suites green.+- [ ] **Capture cold-start context:** `### Phase 5 — as-built` note — the published `pipelex-temporal` version, the pins flipped in each downstream repo, and the consumer-suite green evidence.+- [ ] **Fan-out `/code-review`:** core *deletion* side reviewed (session 1, on `a8382d1a1`); `pipelex-temporal` *additions* side reviewed (session 2) → **SHIP/clean** (no BLOCKER/SHOULD-FIX; wheel built offline, pin-no-leak + import-light proven at runtime). 2 NITs triaged (traces fixed; `test_extras→tests` back-import deferred). Remaining: re-review downstream pin-flip diffs at publish. Triage: [`wip/plugins/phase-5-cutover-review-followups.md`](wip/plugins/phase-5-cutover-review-followups.md).+- [ ] **Ship:** coordinate the cross-repo release. Effort complete.++---++## Deferred / out of scope (considered, parked — do NOT do these in 0–5)++- [ ] **P3 follow-up commit:** collapse `SdkClientManager` into `SdkClientRegistry` (hub holds the registry directly) — D4 keeps it out of the Phase 0 rename. Land as its own commit after Phase 0.+- Third-party-defined execution modes (open string-keyed space) — enum stays closed (design D5).+- Generic per-plugin typed-config namespace — Temporal config stays typed in core (design D7).+- Deliberately *overriding* a built-in (third party replacing core's OpenAI driver) — design §5.3.+- Per-SPI versions / semver-range matching — single coarse `int` (design §5.4).+- Vendor inference dists + OpenAI-substrate extraction into a named module — design §6.4/§6.6/§11.+- Distribution/CI for `pipelex-temporal` — part of Phase 5's cross-repo effort.++---++## Sequencing++Phase 0 (rename) → Phase 1 (seam + LLM, merged) → Phase 2 (ImgGen/Extract/Search, parallel sub-PRs) → Phase 3 (orchestrators) → Phase 4 (model_lists) → **[gate: Checkpoint 3 §14.5 Temporal-green]** → Phase 5 (externalize Temporal, cross-repo). Phases 0–4 are independent PRs in `pipelex`; each lands green. Inference SPI + authoring guide ship with Phase 2; orchestrator SPI with Phase 3. Phases 0 and 1 are strictly sequential; after Phase 1 lands, lanes A (ImgGen) / B (Extract) / C (Search) / D (Phase 3) / E (Phase 4) can run in parallel worktrees — **conflict flag:** lanes A/B/C and E both touch `plugins/<vendor>/` for multi-family vendors; coordinate the per-vendor plugin object.++---++## As-built log (append per phase at each checkpoint — keep this current for cold starts)++> Each checkpoint appends an `### Phase N — as-built` subsection here with: final names/signatures, divergences from plan, test evidence, and anything a cold resume needs.++### Phase 0 — as-built++**Status:** done, committed on `refactor/Plugins`. `make agent-check` clean (ruff/plxt/pyright 0/mypy 0/keyword-only pass) · `make tb` green · `make agent-test` green.++**Final module → class map** (`pipelex/plugins/`):++| Old module / symbol | New module / symbol |+|---|---|+| `plugin.py` · `Plugin` | `model_handle.py` · `ModelHandle` |+| `plugin_sdk_registry.py` · `PluginSdkRegistry` (`PluginSdkRegistryRoot`) | `sdk_client_registry.py` · `SdkClientRegistry` (`SdkClientRegistryRoot`) |+| `plugin_manager.py` · `PluginManager` | `sdk_client_manager.py` · `SdkClientManager` |+| `plugin_factory_abstract.py` · `PluginFactoryAbstract` | `backend_extras_factory.py` · `BackendExtrasFactory` |++- `SdkClientRegistry.get_sdk_instance`/`set_sdk_instance` → **`get(self, model_handle)`** / **`set(self, *, model_handle, sdk_instance)`** (subject `model_handle` positional; the DRY `get_or_create` is still Phase 1, not added here).+- `SdkClientManager.plugin_sdk_registry` attribute → **`sdk_client_registry`** (the D4 thin-wrapper shape preserved; collapse into the registry remains the deferred P3 commit).+- `hub.py`: `get_plugin_manager`/`set_plugin_manager` → `get_sdk_client_manager`/`set_sdk_client_manager` (both the `PipelexHub` methods and the module-level functions); field `_plugin_manager` → `_sdk_client_manager`; the `"PluginManager2 is not initialized"` message is now `"SdkClientManager is not initialized"`.+- `pipelex.py`: `self.plugin_manager` → `self.sdk_client_manager` at construction/setup/teardown call sites.++**Scope beyond the literal rename (kept it a clean "free the word plugin"):** the `Plugin` *type* rename was carried through to the variables/params that held it — `plugin` → `model_handle` — across the four worker factories and every in-tree adapter (`openai`, `bedrock`, `anthropic`, `gateway`, `portkey`, `azure_rest`). The four wrong-path imports (`llm`/`img_gen`/`extract` imported `Plugin` from `plugin_sdk_registry`; `search` used the correct path) are all now `from pipelex.plugins.model_handle import ModelHandle`. Adapter error messages `f"Plugin '{...}'…"` → `f"ModelHandle '{...}'…"`. Prose/comment uses of "plugin" that mean the *future plugin concept* (e.g. `pipe_llm.py`, `provider_name.py`, `model_deck.py`, `cogt/exceptions.py`, `error_pages_generator.py`) were left untouched (one capital-P comment in `pipe_llm.py` was lowercased to read as the concept, not the type).++**Tests touched:** symbol/kwarg renames in the worker-factory routing tests, the gateway/transport/azure error-handling tests, and the temporal teardown call sites. The `plugin=`→`model_handle=` kwarg flip and `client_kwargs["plugin"]`→`["model_handle"]` assertions track the renamed param. Fixtures `plugin_for_openai`/`plugin_for_anthropic` → `model_handle_for_openai`/`model_handle_for_anthropic`, and `tests/integration/pipelex/fixtures/plugin_fixtures.py` → `model_handle_fixtures.py` (conftest import + `__all__` updated). The `tests/integration/pipelex/plugins/` dir name was kept — it reads correctly as backend tests under the new vocabulary.++**No behavior change.** Pure rename; the dispatch `match` statements, caching dance, and error surfaces are byte-equivalent modulo identifiers.++**Deferred (still open):** P3 follow-up — collapse `SdkClientManager` into `SdkClientRegistry` (hub holds the registry directly), own commit after Phase 0. Not done here by D4.++### Phase 1 — as-built++**Status:** done, committed on `refactor/Plugins`. `make agent-check` clean (pyright 0 / mypy 0 over 2219 files / keyword-only pass) · `make tb` green · `make agent-test` fully green. `/code-review --fix` ran clean. This is the locked contract Phases 2–4 build against.++**New seam modules** (all in `pipelex/plugins/`, all import-light — importing them imports no backend SDK or `temporalio`):++- `contract.py` — `PipelexPlugin` (`@runtime_checkable Protocol`: `name: str`, `targets_api: int`, `register(self, registrar) -> None`) + `PLUGIN_API_VERSION: int = 1`. Docstring states the **side-effect-free `register`** invariant.+- `inference_backend_registry.py` — `InferenceFamily(StrEnum)` = LLM/IMG_GEN/EXTRACT/SEARCH; `MakeWorkerFn: TypeAlias = Callable[..., InferenceWorkerAbstract]`; `InferenceBackendRegistry` keyed by `(family, sdk)` with `.lookup(*, family, sdk)` (miss → `NotImplementedError` "No inference backend registered for sdk '<sdk>' in the <family> family. Is its plugin installed?"), `.has(...)`, `.keys`. Also hosts **`require_sdk(*, spec, extra, msg, dependency_name=None)`** — `spec` accepts a `str` or a `Sequence[str]` (bedrock needs both `boto3`+`aioboto3`); `dependency_name` overrides the displayed package when it differs from the import name (google: spec `google.genai` / dep `google-genai`); raises `MissingDependencyError`.+- `orchestrator_registry.py` — `OrchestratorProtocol` (`async def run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput`) + `OrchestratorRegistry` keyed by `PipelexExecutionMode` (`.get_optional`, `.has`, `.modes`). **Skeleton only — not consumed until Phase 3.**+- `registrar.py` — `PluginRegistrar` accumulator + `PluginDiscovery` (mutable BaseModel: name/origin/status/targets_api/contributions/detail), `HubSlot`/`PluginOrigin`/`PluginStatus` StrEnums, `CliCommand` NamedTuple. Menu surface: `add_inference_backend(*, family, sdk, make_worker)`, `add_orchestrator(*, mode, orchestrator)`, `claim_content_generator|pipe_router|pipe_run|task_manager(factory)` (D5 thunks), `add_cli_command(*, name, help, command)`, `add_teardown(callback)`. `begin_plugin(...)` (driven by `build_registrar`) sets the "active" discovery so contributions are attributed and **duplicate errors name both contributors**. Duplicate `(family,sdk)`/mode/slot each raise their fail-loud error.+- `discovery.py` — **`build_registrar(*, config: PipelexConfig) -> PluginRegistrar`** (pure, idempotent, D3-safe). Iterates `BUILTIN_PLUGINS` then `importlib.metadata.entry_points(group="pipelex.plugins")` (external entry points resolve to a plugin instance or a zero-arg factory); version-checks each (`targets_api != PLUGIN_API_VERSION` → `PluginApiVersionMismatchError`); skips + logs names in `config.plugins.disabled` (raises `CoreUnconditionalPluginDisabledError` if the name is in `CORE_UNCONDITIONAL_PLUGIN_NAMES`); wraps any other failure in `BrokenPluginError` (the two sanctioned `except Exception` sites, annotated "Case 2"). The `_external_entry_points()` helper is the monkeypatch seam for tests.+- `builtins.py` — `BUILTIN_PLUGINS` = the seven LLM driver plugin instances (OpenAI, Gateway, Portkey, Anthropic, Mistral, Bedrock, Google); `CORE_UNCONDITIONAL_PLUGIN_NAMES = frozenset({"openai"})`.+- `exceptions.py` — `PluginError` base (CONFIG domain) + `PluginApiVersionMismatchError`, `DuplicateInferenceBackendError`, `DuplicateOrchestratorError`, `HubSlotAlreadyClaimedError`, `CoreUnconditionalPluginDisabledError`, `BrokenPluginError`. (`MissingOrchestratorError` is Phase 3, in `runtime_bridge/exceptions.py`.)++**Driver plugins** (one per vendor, in `pipelex/plugins/<vendor>/<vendor>_plugin.py`): each is a tiny class (`name`, `targets_api`, `register`) whose `register` calls `add_inference_backend(family=LLM, sdk=..., make_worker=<module-level closure>)`. The closures do the lazy SDK imports (`# noqa: PLC0415`), the `require_sdk` guard (anthropic/mistral/bedrock/google), and `sdk_clients.get_or_create(handle=model_handle, build=lambda: ...)`. They are byte-equivalent to the old `match` arms. **Phase 2 extends these same vendor plugins** to register IMG_GEN/EXTRACT/SEARCH backends (the cross-family-vendor coordination point).++**DRY helpers (design §6.2):** `SdkClientRegistry.get_or_create(*, handle, build)` (uses `is not None`, not truthiness) replaced the `get(...) or set(...)` dance; `require_sdk(...)` replaced the per-arm `find_spec` guard, moved **into** the closures so a missing extra fails at use.++**`MakeWorkerFn` call shape (the locked contract):** `make_worker(*, inference_model, backend, sdk_clients, reporting_delegate)` → `InferenceWorkerAbstract`. The closure derives its own `model_handle` from `inference_model`. The dispatcher (`LLMWorkerFactory.make_llm_worker`) builds `model_handle`, resolves `backend` via the models manager, `lookup`s the registry, calls the closure with `sdk_clients=get_sdk_client_manager().sdk_client_registry`, and `cast`s the result to `LLMWorkerInternalAbstract`.++**Boot wiring (`pipelex.py:setup()`):** `build_registrar(config=get_config())` is called **after** the gateway/model-setup checks (so `Pipelex.__new__(...).setup()` tests that expect `InferenceSetupRequiredError`/`GatewayTermsNotAcceptedError` still raise first) and **before** the content-generator/router/run hub setup points (right before `set_dry_run_forced`). The two registries are stored on the hub via `set_inference_backend_registry` / `set_orchestrator_registry`; the registrar is held on `self._plugin_registrar` for Phase 3's slot-thunk/CLI/teardown apply-points (empty in Phase 1).++**Hub:** added `_inference_backend_registry` / `_orchestrator_registry` fields, `set_/get_inference_backend_registry`, `set_/get_orchestrator_registry` (raise if unset), and module-level `get_inference_backend_registry()` / `get_orchestrator_registry()`. Registry types imported under `TYPE_CHECKING` (hub is imported everywhere — avoids cycles).++**Import-cycle break (not in the original plan, required):** `PipelexPipeRunInput` / `PipelexPipeRunOutput` were extracted from `runtime_bridge/bridge.py` into a new import-light **`runtime_bridge/payloads.py`**; `bridge.py` re-imports them (so `from ...bridge import PipelexPipeRunInput` still works for the existing tests). Without this, `orchestrator_registry → bridge → bootstrap → pipelex` was a `reportImportCycles` error.++**Control surface (D7):** core-owned `PluginsConfig` (`disabled: list[str]`) added to `configs.py` as `PipelexConfig.plugins`; `[plugins] disabled = []` added to **both** `pipelex/pipelex.toml` and `.pipelex/pipelex.toml` (real default, not commented). New `pipelex plugins list` command (`cli/commands/plugins_cmd.py`, registered in `_cli.py` + `list_commands`) renders `build_registrar(...).discoveries` as a Rich table.++**Tests:** `tests/unit/pipelex/plugins/test_plugin_discovery.py` (protocol satisfaction, version mismatch, duplicate backend/mode/slot naming both, broken plugin + broken entry point, **idempotent `build_registrar`**, denylist skip + core-unconditional raise, discoveries describe built-ins) and `tests/unit/pipelex/plugins/test_import_light_boot.py` (subprocess `sys.meta_path` import-blocker). The existing `test_llm_worker_factory.py` was updated to build the real registry from `BUILTIN_PLUGINS` and patch `get_inference_backend_registry` — so it exercises the real closures through the real lookup (registry round-trip + caching + huggingface-style missing-dependency parity + the registry-miss `NotImplementedError`). New error classes → `pipelex-dev generate-error-pages` regenerated `docs/errors/`.++**Behavior unchanged** in LLM dispatch except the deliberate registry-miss error wording (was `"ModelHandle '<...>' is not supported"`, now `"No inference backend registered for sdk '<sdk>' ... Is its plugin installed?"`).++**Deferred / notes for later phases:** docs (Inference SPI reference + plugin-authoring guide + example backend plugin) ship with **Phase 2** per plan — not written in Phase 1. The orchestrator registry/protocol is a skeleton; Phase 3 wires the bridge `match` collapse, slot-thunk apply-points (with injection-precedence), CLI-command harvesting, and adds `MissingOrchestratorError`.++### Phase 2 — as-built++**Status:** done, uncommitted on `refactor/Plugins`. `make agent-check` clean (ruff/plxt, pyright **0 errors**, mypy 0 over 2228 files, keyword-only pass) · `make tb` green · targeted plugins+cogt unit suite green (977 passed) · **full `make agent-test` green (exit 0, "All tests passed.")**. All four worker-factory `match` statements are gone — every inference backend now resolves through `get_inference_backend_registry().lookup(family, sdk)`.++**Registry shape after Phase 2** — `BUILTIN_PLUGINS` now has 15 plugins contributing **30** `(family, sdk)` backends: LLM 14, IMG_GEN 9, EXTRACT 5, SEARCH 2.++**New vendor plugin modules** (`pipelex/plugins/<vendor>/<vendor>_plugin.py`, each added to `BUILTIN_PLUGINS`):++- `fal` → IMG_GEN `fal` (require_sdk `fal_client`/extra `fal`, dep name `fal-client`).+- `huggingface` → IMG_GEN `huggingface_img_gen`. **C10 bug fixed:** added `require_sdk(spec="huggingface_hub", extra="huggingface")` (the old arm imported `huggingface_hub` with no guard). The `PROVIDER_OR_POLICY_T` annotation is a module-level `if TYPE_CHECKING` import (import-light at runtime); the `model_handle.variant → provider` resolution lives in the closure.+- `blackboxai` → IMG_GEN `blackboxai_img_gen` (OpenAI-completions substrate, `BlackboxaiCompletionsFactory(is_http_url_enabled=True)`; no require_sdk — reuses OpenAI client).+- `openrouter` → IMG_GEN `openrouter_img_gen` (same substrate shape, `OpenRouterCompletionsFactory(is_http_url_enabled=True)`).+- `azure_rest` → IMG_GEN `azure_rest_img_gen`. **DIRECT construction** — `AzureImgGenWorker(model_handle=…)`, no `sdk_clients`/`get_or_create`; `backend` + `sdk_clients` params carry `# noqa: ARG001`.+- `docling` → EXTRACT `docling_sdk` (require_sdk `docling`; `get_or_create(build=DoclingFactory.make_docling_sdk)`).+- `pypdfium2` → EXTRACT `pypdfium2`. **STATELESS** — built directly, no `get_or_create`, no require_sdk; `sdk_clients` param `# noqa: ARG001`.+- `linkup` → EXTRACT `linkup_fetch` (require_sdk `linkup`, stateless) **and** SEARCH `linkup` (stateless, **no** require_sdk — matches the original search arm exactly; raw `ImportError` if linkup absent, a deliberate no-behavior-change choice). One plugin, two families.++**Extended existing vendor plugins** (one `register` now spans families — the cross-family coordination point):++- `openai` += IMG_GEN `openai_img_gen`.+- `gateway` += IMG_GEN `gateway_img_gen` + `gateway_completions`, EXTRACT `gateway_extract`, SEARCH `gateway_search` (now serves **all four** families). Closures named `_make_gateway_{img_gen,completions_img_gen,extract,search}_worker` (distinct from the existing LLM `_make_gateway_{completions,responses}_worker`).+- `google` += IMG_GEN `google` (own missing-msg constant `_GOOGLE_IMG_GEN_MISSING_MSG`, distinct from the LLM one).+- `mistral` += EXTRACT `mistral` (own `_MISTRAL_EXTRACT_MISSING_MSG`).++Note: `gateway_completions` and `google` are SDK strings registered in **two** families each — distinct `(family, sdk)` keys, no conflict. The OpenAI-compat substrate (`OpenAICompletionsImgGenWorker`, the per-vendor `*CompletionsFactory`) stays an in-tree library the substrate-reuse closures import; extraction into a named module remains deferred.++**Factory collapses** (`cogt/{img_gen,extract,search}/*_worker_factory.py`): each is now the LLM shape — derive `model_handle`, resolve `backend` via the models manager, `lookup(family=<FAM>, sdk=model_handle.sdk)`, call with `sdk_clients=get_sdk_client_manager().sdk_client_registry`, `cast` to the family worker abstract. Kept `@classmethod` (call sites unchanged). The per-family `case _:` wording ("is not supported …") is gone — registry miss now raises the uniform `NotImplementedError("… Is its plugin installed?")` from `InferenceBackendRegistry.lookup`.++**Search C9 normalization:** `make_search_worker` now takes `*, reporting_delegate: ReportingProtocol | None = None` (default `None`, so existing test callers that omit it still work) and passes it through — the factory no longer pulls `get_report_delegate()` from the hub. The hub coupling moved to the caller: `cogt/content_generation/search_generate.py::_make_search_worker` now passes `reporting_delegate=get_report_delegate()` (import added). No production behavior change (the real caller still supplies the delegate); integration tests that call the factory directly get `None`, which the workers already accept.++**Tests:** `tests/unit/pipelex/cogt/img_gen/test_img_gen_worker_factory.py` updated to build the real registry from `BUILTIN_PLUGINS` (new `build_builtin_inference_backend_registry` + patches `get_inference_backend_registry` in `patch_hub_getters`), the unknown-sdk assert switched to "Is its plugin installed?", and `huggingface_img_gen` added to the missing-dependency parametrize (proves the C10 fix). New `tests/unit/pipelex/plugins/test_inference_backend_coverage.py` pins the full 30-key `(family, sdk)` round-trip + the cross-family vendors (mistral LLM+EXTRACT, google/openai LLM+IMG_GEN, linkup EXTRACT+SEARCH, gateway all four). Import-light guard BLOCKED set grew `docling` + `linkup` (fal_client/huggingface_hub were already there) — proves the new vendor plugin modules import no optional SDK at boot.++**Docs:** `docs/under-the-hood/inference-backend-plugins.md` (added to both mkdocs nav blocks) — the seam walkthrough, the `PipelexPlugin` contract, `MakeWorkerFn` + import-light/fail-at-use invariants, a complete copy-pasteable `acme` example (built-in + `[project.entry-points."pipelex.plugins"]` form), the `plugins.disabled` denylist, the SPI module/symbol table, and the fail-loud table.++**No factory test modules for extract/search** existed before and none added — those families are covered by the new registry-coverage test + the integration suites (`test_search.py`, extract integration). Only img_gen had a routing test to update.++#### Post-checkpoint xhigh `/code-review` (2026-06-20, on PR #997 = the whole Phases 0–2 diff)++Beyond the per-phase CP reviews above, an **xhigh** whole-PR pass ran (10 finder angles → 35 candidates → independent verify → 13 kept). **No live correctness bug.** Three CONFIRMED pre-merge gaps fixed + committed as `94fa908df` (`make agent-check` green, 78 affected tests pass). These **supersede specifics in the CP2 record above**:++- **Lookup-miss error (supersedes the line about `NotImplementedError("… Is its plugin installed?")`):** `InferenceBackendRegistry.lookup` now raises a structured **`InferenceBackendNotFoundError(PluginError)`** (new in `plugins/exceptions.py`, carries `family`/`sdk`, message ends "Is its plugin installed **and enabled?**"). Both factory miss-tests assert the structured type + `.sdk`.+- **Import-light blocklist (supersedes "BLOCKED grew docling+linkup"):** added `openai`, `portkey_ai`, `pypdfium2` — the migrated plugins' deferred SDKs the guard wasn't covering. (The review said "openrouter," but that closure rides the `openai` SDK; verified by probe that none of the three are imported at registration time, so the invariant holds.) The guard docstring now says "any backend SDK (optional extra **or** heavy core dep)."+- **Multi-spec `require_sdk`:** on a partial miss it named ALL specs; now collects + names only the absent one(s) (so a user with `boto3` but not `aioboto3` isn't told to reinstall `boto3`). New `test_require_sdk` multi-spec case.++**Deferred follow-ups → `wip/plugins/phase-2-review-deferred.md`** (latent footguns reachable only via external entry-point plugins / future callers + quality cleanups). **Two bear on Phase 3 — read that doc before starting:**++- `build_registrar` accumulates `cli_commands` / `slot_claims` / `teardown_callbacks` but boot only consumes `inference_backends` + `orchestrators` — i.e. those three menus are **wired for the first time in Phase 3** (CLI collapse / Boot-teardown collapse). The doc frames the current state as a teardown-leak footgun for *external* plugins; closing it is literally Phase 3 work. Until then, an external plugin contributing those is silently inert.+- search `reporting_delegate` defaults to `None` (the C9 normalization above) — fine for the one real caller, but a future Temporal search-activity path that omits it gets silent zero-reporting. Relevant when Phase 3/5 routes search through a worker.++(The `ModelHandle` double-construction cleanup is cross-referenced there to the existing `phase-1-review-deferred.md`; it dissolves if Phase 3 threads `model_handle` through `MakeWorkerFn`.)++### Phase 3 — as-built++**Status:** done, checkpoint commit `19e6ca66b` on `refactor/Plugins-2`. `make agent-check` clean (ruff/plxt, **pyright 0**, mypy 0 over 2239 files, keyword-only pass) · `make tb` green · **full `make agent-test` green** ("All tests passed", exit 0) · **Temporal integration suite green** (`tests/integration/pipelex/temporal/` → 156 passed) · `/code-review` clean (no BLOCKERs; S1/N1/N2 applied).++**The dispatch seam.** `bridge.py`'s `match execution_mode:` is gone. `run_pipe_via_bridge` now does `get_orchestrator_registry().get_optional(mode=…)` → `orchestrator.run(...)`, else `MissingOrchestratorError(mode=…)`. `bridge.py` names no integration (only docstrings/comments). `build_pipe_job_from_input`, `_validate_input`, `_decode_*` stay; `serialize_pipe_output` is re-exported (`# noqa: F401`/`TC001`) for host-runtime back-compat.++**Orchestrators (verbatim extraction):**++- `runtime_bridge/direct_orchestrator.py::DirectOrchestrator` — body of the old `_run_direct`, **`scoped_pipe_router(PipeRouter())` landmine kept verbatim**. Core, always-on.+- `temporal/temporal_orchestrators.py::TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` — bodies of `_run_temporal_*`, lazy `temporalio`/`make_temporal_pipe_run`/`WorkflowExecutionError` inside `run`; each `run` first calls `_require_temporal_extra(mode=…)` (`importlib.util.find_spec("temporalio")` → `MissingOrchestratorError` on absence — no relabelling of deeper import bugs).++**Cycle break (the one non-obvious structural move).** Two extractions were forced by `reportImportCycles` (pyright counts **function-level** imports too — verified empirically; lazy-import does NOT break a pyright cycle, only a runtime one):++1. `runtime_bridge/serialization.py` — `PIPE_DISPATCH_ERRORS`, `serialize_pipe_output`, `resolve_main_stuff_root_key`, `serialize_completed_output` (import-light: core memory + pipe/pipeline exceptions + payloads). Both DIRECT and Temporal orchestrators serialize through it, so they never import `bridge.py` (which would re-form `bridge→bootstrap→pipelex→discovery→builtins→orchestrator→bridge`).+2. **CLI commands declared by `import_path` string, not by importing the callable.** A builtin Temporal plugin contributing `worker`/`setup-temporal-namespace` (whose modules boot Pipelex) would cycle `builtins→temporal_plugin→worker_cmd→pipelex→discovery→builtins`. Fix: `CliCommand.command: Callable` → `CliCommand.import_path: str` (`"module:attr"`); `_cli.py` resolves it via `importlib.import_module` at harvest (dynamic ⇒ off pyright's static graph). The two command modules are unchanged from `main`. Temporal stays a **builtin** (not an entry point) — the plan's Phase-3 intent.++**Plugins:** `plugins/direct/direct_plugin.py::DirectOrchestratorPlugin` (`name="direct"`) + `temporal/temporal_plugin.py::TemporalPlugin` (`name="temporal"`) added to `BUILTIN_PLUGINS`; `CORE_UNCONDITIONAL_PLUGIN_NAMES = {"direct", "openai"}`. `TemporalPlugin.register`: always two `add_orchestrator` + two `add_cli_command(import_path=…)`; iff `registrar.config.temporal.is_enabled`, four `claim_*` thunks + `add_teardown` (all module-level zero-arg thunks lazy-importing temporalio).++**Boot/teardown (`pipelex.py`):** the four `temporal.is_enabled` blocks → injection-precedence at each ordered point via `_resolve_hub_slot(*, slot, default: Callable[[], _HubSlotImplT]) -> _HubSlotImplT` (explicit param > slot thunk > core default). TASK_MANAGER just runs its thunk (full temporal-hub wiring inside it). `teardown` runs `reversed(self._plugin_registrar.teardown_callbacks)` (LIFO) in place of the old temporal block. `_temporal_task_manager` attribute removed; `_plugin_registrar` now declared `PluginRegistrar | None` in `__init__` and assigned via a local in `setup` (narrowing). `temporal_hub.get_optional_task_manager()` added (+ `teardown()` promoted onto the `TaskManager` protocol with `@override` on the impl).++**CLI (`_cli.py`):** `_CORE_COMMAND_ORDER` (core block) + harvested `_PLUGIN_COMMAND_NAMES`; `list_commands` returns their concatenation. `_register_discovered_cli_commands()` runs `build_registrar(config=_config_for_cli_harvest())` at module load and `app.command(...)`-registers each resolved callable. `_config_for_cli_harvest` = `load_config(ensure_global_if_missing=False)` (new side-effect-free param) with `(TomlError, ValidationError)` → `load_base_config_dict()` fallback (new method; package-default only, always valid).++**Exceptions:** `MissingOrchestratorError(*, mode)` (mode→message match/case, `_authors_caller_facing_message=True`) replaces `MissingPipelexTemporalExtraError` + `MissingMistralWorkflowsPluginError` (both removed). `pipelex-dev generate-error-pages` regenerated `docs/errors/` (the two old pages removed, `missing-orchestrator-error.md` added; also picked up `inference-backend-not-found-error.md` from CP2).++**Tests:** new `test_orchestrator_dispatch.py` (fake-orchestrator dispatch + per-mode miss parity), `test_hub_slot_injection_precedence.py` (explicit>slot>default at content-generator/pipe-router + teardown LIFO, fake registrar via patched `build_registrar`), `test_plugin_cli_command_harvest.py` (worker/setup-temporal-namespace harvested, in order, after core). Updated: `test_dispatch.py` (mistral → `MissingOrchestratorError`), `test_exceptions_disclosure.py` (per-mode `MissingOrchestratorError`), `test_output_serialization.py` (import from `serialization`), `test_import_light_boot.py` (BLOCKED += `temporalio`; config stub gains `temporal.is_enabled=True` to exercise the slot-claim branch import-light), `test_inference_backend_coverage.py` + `test_plugin_discovery.py` (stub configs gain `temporal` so the now-builtin TemporalPlugin's `register` can read `config.temporal.is_enabled` — closes deferred-doc item 4 for these two tests). Boot-via-slots for the temporal content generator is already covered end-to-end by `test_keyless_boot_forced_dry.py` (green through the new seam).++**Deferred-doc items closed by Phase 3:** the three previously-unconsumed registrar menus are now wired — `cli_commands` (consumed by `_cli.py` harvest), `slot_claims` + `teardown_callbacks` (consumed by `pipelex.py` boot/teardown). So an external plugin contributing those is no longer silently inert.++**Notes for Phase 4/5:** Phase 4 (model_lists 5th seam) is independent. Phase 5 externalizes `pipelex/temporal/` → `pipelex-temporal`: the Temporal plugin + orchestrators + commands are already a self-contained unit behind the seam; the move is (a) relocate the Temporal config schema out of `pipelex.temporal` (codex C6 — `configs.py:14` still imports `pipelex.temporal.config_temporal`), then (b) ship `pipelex/temporal/` + declare the `pipelex.plugins` entry point in the new dist (today Temporal is an in-tree builtin, not yet an entry point). Gate: this CHECKPOINT-3 Temporal-green evidence.++**Post-checkpoint xhigh `/code-review` follow-ups (separate from the CHECKPOINT-3 reviewer pass above).** A later xhigh `/code-review` over the Phase 3 commit surfaced a dominant correctness cluster plus medium/low items, captured for triage:++- 🔴 **Critical — read before Phase 5 / before externalizing any plugin:** [`wip/plugins/phase-3-critical-cli-harvest-fragility.md`](wip/plugins/phase-3-critical-cli-harvest-fragility.md). The import-time CLI plugin-command harvest (D3) is an unguarded surface: a broken/incompatible/colliding **external** plugin or an unreadable user config can brick *every* `pipelex` command — including the `--help`/`doctor`/`plugins`/`init` recovery commands — and `add_cli_command` has no collision guard, so a plugin can silently shadow a core command (`run`, …). Latent for in-tree-only installs (suite green), live the moment Phase 5 ships an external dist. **Cold-start problem statement only — solution to be explored in a fresh session.**+- 🟡 **Medium/low deferred items → [`wip/plugins/phase-3-review-deferred.md`](wip/plugins/phase-3-review-deferred.md):** the misleading `MISTRAL_NATIVE` "install a package you already have" hint (dead mode until the Mistral plugin lands), the bridge dropping its orchestrator error-wrap (external-orchestrator contract), `teardown()` with no `try`/`finally`, `_teardown_temporal` re-fetching from the global hub, the wrong SPI doc about Temporal entry-point discovery, an enum identity comparison, double-discovery/import-time cost (coupled to the critical fix), a dead `serialize_pipe_output` re-export, and a hardcoded slot count.++#### CHECKPOINT-3 `/code-review` (pr-review-toolkit:code-reviewer, on the Phase 3 working-tree diff)++**Verdict: clean — no BLOCKERs.** The reviewer compared every extracted body against `git show HEAD:…bridge.py` and confirmed **byte-equivalence** of all four orchestrator bodies (modulo the `_PIPE_DISPATCH_ERRORS`→`PIPE_DISPATCH_ERRORS` rename + the added `_require_temporal_extra` guard) and verified all named landmines: DIRECT `scoped_pipe_router` preserved, temporal-blocking `make_workflow_id` recompute + `WorkflowExecutionError` catch verbatim, per-mode error parity (exact deleted hint strings, `_authors_caller_facing_message=True`), injection precedence (explicit>slot>default at all four points), import-light (no module-top temporalio; register reads only `config.temporal.is_enabled`), CLI determinism, teardown LIFO + no-integration.++Applied: **S1** (the one should-fix) — added `test_harvest_config_falls_back_to_base_on_broken_user_config` pinning the malformed-user-config fallback in `_config_for_cli_harvest` (named Phase-3 risk #6, previously untested); **N1** — fixed the two stale `runtime_bridge.bridge._run_direct` doc references in `bundle_validator.py` → `runtime_bridge.direct_orchestrator.DirectOrchestrator.run`; **N2** — re-anchored the harvest-order test on "plugin commands are the last registrations" (`names[-2:]`) rather than the brittle `which` positional. **N3** (unused `OrchestratorRegistry.has`/`.modes`) left as a deliberate read-API symmetry surface mirroring `InferenceBackendRegistry`. The reviewer also confirmed the `if pipe_router:`-truthy vs `if content_generator is None:` asymmetry is **pre-existing** (identical in HEAD), not a Phase-3 regression.++### Option A — as-built (post-Phase-3, commit `989c9beed`)++**Status:** done. `make agent-check` clean (pyright 0 / mypy 0 over 2240 files / keyword-only) · `make tb` green · `make agent-test` "All tests passed." · `pytest tests/integration/pipelex/temporal/` → **156 passed, 4 xpassed** (the 4 xpassed = pre-existing xdist class-registration flakiness markers, matching the Phase-3 baseline) · both `--help` smokes pass. Full record + rationale: [`wip/plugins/option-a-drop-cli-command-seam.md`](wip/plugins/option-a-drop-cli-command-seam.md).++**Why.** The post-Phase-3 xhigh review found a critical fragility: the Phase-3 CLI-command harvest (the former D3) ran `build_registrar` at *import* of `pipelex/cli/_cli.py` on **every** `pipelex` invocation, so a broken/colliding external plugin or an unreadable user config could brick every command — including the `--help`/`doctor`/`plugins`/`init` recovery commands — and `add_cli_command` had no collision guard, so a plugin could silently shadow a core command. **Option A resolves it by removing the surface, not hardening it.** CLI commands are not a plugin contribution type any more.++**What changed:**++- **New:** `pipelex/temporal/temporal_cli.py` — a grouped `typer.Typer()` app registering `worker` + `setup-namespace`; `pipelex-temporal = "pipelex.temporal.temporal_cli:app"` in `pyproject.toml` `[project.scripts]`. Import-light at module top (callables pull `temporalio` lazily).+- **Moved (git renames):** `pipelex/cli/commands/worker_cmd.py` → `pipelex/temporal/worker_cmd.py`; `…/setup_temporal_namespace_cmd.py` → `pipelex/temporal/setup_namespace_cmd.py` (callable `setup_temporal_namespace_cmd` → `setup_namespace_cmd`). They now travel with Temporal into Phase 5.+- **Removed:** the harvest in `_cli.py` (`_config_for_cli_harvest`, `_register_discovered_cli_commands`, `_PLUGIN_COMMAND_NAMES`; `list_commands` now returns `list(_CORE_COMMAND_ORDER)`); `add_cli_command` + `CliCommand` + `cli_commands` in `plugins/registrar.py`; the two `add_cli_command` calls in `temporal/temporal_plugin.py` (docstrings rewritten; the hardcoded "four hub slots" count also dropped while there).+- **Strings/docs:** `PIPELEX_SETUP_CLI_COMMAND` + all docstrings/examples → `pipelex-temporal setup-namespace` / `pipelex-temporal worker`; 5 distributed-execution docs swapped; `docs/under-the-hood/orchestrator-plugins.md` CLI section rewritten ("Operational commands ship as console scripts") + SPI table + worked-example + the factually-wrong "self-entry-point" claim fixed; `docs/errors/search-attribute-registration-error.md` regenerated (not hand-edited).+- **Tests:** deleted `test_plugin_cli_command_harvest.py`; added `tests/unit/pipelex/temporal/test_temporal_cli.py` (console-script smoke); updated `test_setup_temporal_namespace_cmd.py` (import + use-site mock targets + callable); fixed `test_cli_entrypoint_smoke.py` (dropped the now-invalid `pipelex worker --help` case) and `test_search_attribute_bootstrap_check.py` (assert the new string).++**Untouched (deliberately out of scope — do NOT unwind in later phases):** the four inference-family registry lookups, the bridge → `OrchestratorRegistry` dispatch, the `temporal.is_enabled` boot-slot claims + LIFO teardown, `build_registrar` / `BUILTIN_PLUGINS` / the `pipelex.plugins` entry-point group for *runtime* contributions, `pipelex plugins list`, the denylist.++**Decision lineage:** the former **D3** (harvest at CLI-build) is **superseded** by Option A (see the D1–D7 list). [`phase-3-critical-cli-harvest-fragility.md`](wip/plugins/phase-3-critical-cli-harvest-fragility.md) → RESOLVED-BY-REMOVAL; two items in [`phase-3-review-deferred.md`](wip/plugins/phase-3-review-deferred.md) (double-discovery/import-time cost; factually-wrong SPI doc) → resolved.++**Open follow-ups (NOT done):**++1. **Config-loader cleanup (flagged, low-risk):** `_config_for_cli_harvest` was the *sole* caller of `config_manager.load_base_config_dict()` and `load_config(ensure_global_if_missing=False)`. Both are now dead (the method has zero callers; the `False` branch is unreachable) in `pipelex/system/configuration/config_loader.py`. Left in place to keep Option A surgical — a follow-up should delete `load_base_config_dict()` and simplify `load_config` to always-ensure-global.+2. **Cross-repo, RELEASE-GATED (do NOT do ahead of the pipelex release shipping `pipelex-temporal`):** flip `pipelex-worker/Dockerfile` (`CMD ["pipelex","worker","--no-sandbox"]` → `["pipelex-temporal","worker","--no-sandbox"]`) and `pipelex-worker/Makefile` (`pipelex worker …` → `pipelex-temporal worker …`) in the **same** commit that bumps `pipelex-worker`'s `pipelex` pin to that release. `pipelex-api-hosted` / `sandbox` / `pipelex-api` were checked — they do **not** invoke the worker. Until that bump, the old pinned pipelex still has `pipelex worker`, so nothing breaks. (Mirror this into the Phase-5 downstream-pin step.)+3. **Legacy `pipelex/temporal/worker_cli.py`** (the `python -m pipelex.temporal.worker_cli` / `configure` entrypoint) is a unification candidate with the new `temporal_cli.py` — noted, not addressed.++**Env gotchas for the next session:** (a) `uv sync --all-extras` materializes the `pipelex-temporal` console script into `.venv/bin/`; it also brings the venv up to the already-committed `uv.lock` (which had drifted on `cryptography`/`mthds`) — `uv.lock` itself unchanged. (b) `make cleanderived` deletes the gitignored `tests/integration/pipelex/fixtures/_generated_model_sets.py`; run `make regenerate-test-models-quiet` (alias `rtm`) before `make agent-check` or pyright fails on the missing import.++### Phase 4 — as-built (the 5th seam: model listing)++**Status:** done (working-tree, pre-commit at time of writing). `make agent-check` clean (ruff/plxt, **pyright 0 over 2243 files**, mypy 0, keyword-only pass) · `make tb` green · **full `make agent-test` green** ("All tests passed.") · targeted `tests/unit/pipelex/plugins/` + `tests/unit/pipelex/cogt/model_backends/` → 715 passed. `model_lists.py`'s `match sdk:` is gone — `pipelex show models <backend>` now dispatches through a registry on the hub.++**The seam.** `cogt/model_backends/model_lists.py` `ModelLister.list_models` no longer branches on `match sdk:` with per-arm hardcoded `from pipelex.plugins.{openai,anthropic,mistral,google,bedrock}.*_list import …`. It now does `get_model_lister_registry().get_optional(sdk=sdk)` → `await lister(...)`, else (miss) → `unsupported_sdks`. The file names **no integration** (imports only `ModelListingUnsupportedError`/`ModelManagerError` from `cogt.exceptions`, `PipelexCLIError`, and the three hub getters).++**New module** `plugins/model_lister_registry.py` (mirrors `inference_backend_registry.py` / `orchestrator_registry.py`, dependency-free):+- `ListModelsFn: TypeAlias = Callable[..., Awaitable[None]]` — the uniform lister callable. Always `async` (the loop awaits it); import-light to reference, lazy inside. Call shape: `await lister(*, sdk, backend_name, backend, flat, any_listed)`.+- `ModelListerRegistry` keyed by `sdk` alone (listing is per-SDK, not per-`(family, sdk)`). `get_optional(*, sdk) -> ListModelsFn | None` (a miss is a **soft** "unsupported-for-listing" outcome, mirroring `OrchestratorRegistry.get_optional` — **not** the inference registry's raising `lookup`), `has`, `sdks`.++**Registrar** (`plugins/registrar.py`): new `add_model_lister(*, sdk, lister)` menu method + `model_listers: dict[str, ListModelsFn]` accumulator + `_model_lister_sources`. Duplicate `sdk` → `DuplicateModelListerError` naming both plugins. Contributions line `f"model lister {sdk}"` (so `pipelex plugins list` shows listers automatically).++**Exceptions:** `DuplicateModelListerError(PluginError)` in `plugins/exceptions.py`; **`ModelListingUnsupportedError(CogtError)`** in `cogt/exceptions.py` — the core soft signal the loop catches (carries `sdk`). `pipelex-dev generate-error-pages` wrote new pages (`duplicate-model-lister-error.md`, `model-listing-unsupported-error.md`) + the `inference-and-providers.md` index; the rest unchanged.++**Hub + boot:** `_model_lister_registry` field + `set_/get_model_lister_registry` + module-level `get_model_lister_registry()`. `pipelex.py setup()` builds `ModelListerRegistry(plugin_registrar.model_listers)` and sets it on the hub, right after the inference-backend registry. The `show models` CLI path boots Pipelex via `make_pipelex_for_cli(needs_inference=True)`, so the registry is always set before `list_models` runs.++**Vendor listers** — each of the 5 plugins grew one `async` lister closure that lazy-imports its `list_*_models` fn (`# noqa: PLC0415`) and registers it:+- `openai` → `_list_openai_models` for sdks `openai`, `azure_openai`, `openai_responses`, `azure_openai_responses` (one closure across those keys; async underlying).+- `anthropic` → `_list_anthropic_models` for `anthropic`. **The one non-trivial case:** the closure catches the vendor `AnthropicSDKUnsupportedError` and re-raises core `ModelListingUnsupportedError(sdk=…)` — so core names no Anthropic-specific symbol while preserving the old "bedrock-backed Anthropic client can't list → unsupported_sdks" behavior.+- `mistral` → `_list_mistral_models` for `mistral`. Underlying fn is **sync and takes no `backend`** → closure is `async def` with `# noqa: RUF029` and `backend: … # noqa: ARG001`.+- `google` → `_list_google_models` for `google` (async underlying).+- `bedrock` → `_list_bedrock_models` for `bedrock`, `bedrock_aioboto3` (**sync** underlying → `# noqa: RUF029`).++**Behavior preserved byte-for-byte** (verified against `git show HEAD:…model_lists.py`): the per-sdk `find_spec` guards removed from `model_lists.py` were **already duplicated inside each `list_*_models` fn** (same lib/extra/message for anthropic/mistral/google/bedrock; openai has none — always installed), so a missing extra still raises `MissingDependencyError` inside the lister and is wrapped by the loop's Case-2 `except Exception → PipelexCLIError` exactly as before. The `list_*_models` fns themselves are **unchanged**. `any_listed` threads identically (set `True` only after a successful `await`; not reached when the lister raises `ModelListingUnsupportedError`, matching the old `continue`). The unsupported-SDK display path is untouched.++**Tests:** `tests/unit/pipelex/plugins/test_model_lister_coverage.py` (registry built from `BUILTIN_PLUGINS`: every expected sdk key resolves to a lister + exact-set assertion + soft-miss cases) · `tests/unit/pipelex/cogt/model_backends/test_model_lister_dispatch.py` (behavioral: lister invoked with expected kwargs; `any_listed` progresses False→True across two SDKs; unknown sdk → unsupported message + lister not invoked; `ModelListingUnsupportedError` translate path → unsupported; generic lister failure → wrapped `PipelexCLIError`). `test_import_light_boot.py` grew an `assert registrar.model_listers` (proves listers register import-light).++**Docs:** `docs/under-the-hood/inference-backend-plugins.md` gained a "Listing models — an optional capability" section (the `add_model_lister` call, the `ListModelsFn` shape, import-light/fail-at-use rules, the `ModelListingUnsupportedError` soft-signal); SPI table += `ListModelsFn` / `ModelListingUnsupportedError`; fail-loud table += `DuplicateModelListerError` (and fixed the adjacent stale `NotImplementedError` row → `InferenceBackendNotFoundError`).++**CHECKPOINT-4 sweep result — "core names no integration" now holds for every enumerated seam.** The broad grep surfaces three **pre-existing, unchanged** `pipelex.plugins.<vendor>` refs that were never in the Phase 0–4 seam list: `cogt/config_cogt.py` (vendor typed-config models — **by design**, design D7), `cogt/img_gen/img_gen_args_factory.py` (`OpenAIImgGenFactory`) and `cogt/model_backends/backend_factory.py` (`VertexAIFactory`, lazy, vertexai auth) — the latter two genuine but **unenumerated** couplings (each would need its own contract capability to invert). Recorded with disposition in [`wip/plugins/phase-4-residual-core-vendor-couplings.md`](wip/plugins/phase-4-residual-core-vendor-couplings.md). **No dispatch-path integration remains.**++**Notes for Phase 5:** Phase 4 is independent of the Temporal externalization. No new cross-repo surface. The model-lister registry is a pure additive capability — an external backend plugin can now contribute a lister via the same `register` it uses for `add_inference_backend`.++### Phase 5 — Step 0b (C6) as-built (the config-schema relocation prerequisite)++**Status:** done, committed at this checkpoint on `refactor/Plugins-2`. **In-`pipelex` only, behavior-neutral, zero cross-repo blast radius.** Gates: `make tb` green · `make agent-check` clean (ruff/plxt, **pyright 0/0**, mypy 0 over 2244 source files, keyword-only pass) · targeted `tests/unit/pipelex/{temporal,system,errors}/` → 463 passed · **full `make agent-test` "All tests passed."** This is the prerequisite that makes the rest of Phase 5 a packaging move: **core config no longer imports `pipelex.temporal`.**++**What moved.**++- `git mv pipelex/temporal/config_temporal.py → pipelex/system/configuration/config_temporal.py` (history preserved; the whole schema module — it was already import-light via the `if TYPE_CHECKING: from temporalio.common import RetryPolicy / else: RetryPolicy = Any` placeholder, kept verbatim). Its internal `from pipelex.temporal.exceptions import …` → `from pipelex.system.configuration.exceptions import …`.+- **New** `pipelex/system/configuration/exceptions.py` holds the two exceptions the schema raises: `TemporalConfigError(ValueError, PipelexError)` + `WorkerTaskQueueUnknownError(TemporalConfigError)` *(base order later corrected to `(PipelexError, ValueError)` by the `.message` fix — see the flagged-bug note below)*. **Re-based** from `TemporalFlowError` to `PipelexError` directly — deliberate: `TemporalFlowError` is the base of the *runtime/workflow* errors that stay with (and externalize with) `pipelex/temporal/`, so dragging it into core would be wrong for the end-state. `TemporalFlowError` is **never imported/caught anywhere** (only mentioned in comments — verified), so dropping it from these two classes' ancestry is invisible. `error_domain` stays `None` (both old `TemporalFlowError` and new `PipelexError` leave it unset — chose `PipelexError` over the domain-setting `PipelexConfigError` to preserve exact behavior).++**What stayed (in `pipelex/temporal/exceptions.py`, travels external in the cut-over).** `TemporalFlowError` + the runtime errors (`WorkflowExecutionError`, `UnrecoverableWorkflowFailureError`, `WorkflowInputError`, `ContentGenerationError`, `TemporalServerError`) + the worker-boot config subclasses (`WorkerScopeConfigError`, `WorkerProfileConfigError`, `SearchAttributeRegistrationError`), which now subclass the **core** `TemporalConfigError` (imported at the module top).++**Importers repointed** (mechanical `perl` rename `pipelex.temporal.config_temporal` → `pipelex.system.configuration.config_temporal` across `pipelex/` + `tests/`, then hand-split the lines that mixed a moved exception with a staying symbol): source = `configs.py:14`, `temporal_tasks.py`, `temporal_connect.py`, `temporal_manager.py`, `temporal_task_manager.py`, `task_manager.py` (TYPE_CHECKING), `codec/codec_server_cli.py`, and the `namespace_check.py` docstring; tests = the ~20 temporal config/exception test modules. The two regression tests got their hardcoded source path / import updated: `test_config_temporal_optional_dep.py` (now AST-scans `pipelex/system/configuration/config_temporal.py`) and `test_dispatch_options_no_temporalio.py`. The `temporal-e2e-validate` skill doc's `BUILTIN_SEARCH_ATTRIBUTES` reference was updated. `pipelex-dev generate-error-pages` rewrote 4 pages (the two moved errors' `Defined in`/`Parent` rows + the `execution-and-runtime`/`platform-and-tooling` macro listings, which correctly regroup the two from "Temporal execution" → "System & configuration").++**Behavior preserved byte-for-byte** (the relocation is a pure move): the schema's validators/methods (`resolve_dispatch`, `make_retry_policy`, `validate_task_queue_known`, the orphan-queue validators) are unchanged; the exception MRO/`error_domain`/`isinstance(_, ValueError)`/`str()`/`.args` are identical to pre-move (verified empirically). The error-class-location convention test passes (new module is a proper `exceptions.py` importing only `PipelexError`).++**Flagged at C6, FIXED in follow-up (`xhigh` review action):** the `(ValueError, PipelexError)` MRO meant `ValueError.__init__` won, so `.message` was never set and `to_error_report()` raised `AttributeError`. Fixed by reordering bases so the message-setting `PipelexError` base resolves first — `TemporalConfigError(PipelexError, ValueError)` and `ConfigModelError(FatalError, ValueError)` (the `Worker*` / `SearchAttribute*` subclasses inherit the fix transitively; the two `(PipeComposeError, ValueError/TypeError)` compose mixins already had the safe order). `isinstance(_, ValueError)` is unaffected by base order, so Pydantic still wraps validator raises; `str()`/`.args` unchanged. Behavior change at one site: `ConfigModelError` now runs the restored `TracebackMessageError.__init__`, so it logs at its single misuse raise site (`ConfigModel.transform_dict_str_to_enum`) — appropriate fatal-error behavior, matching its `ConfigValidationError` / `FatalError` siblings. Guard + regression locked in by `tests/unit/pipelex/test_pipelex_error_message_init.py` (MRO sweep over every loaded `PipelexError` subclass asserting `PipelexError` precedes `ValueError`/`TypeError`, plus `.message` / `to_error_report()` assertions on the reordered classes). `make agent-check` clean, full `make agent-test` green.++**Cross-repo readiness for the cut-over:** scanned `pipelex-worker` / `pipelex-mistralai-workflows` / `pipelex-api` / `pipelex-api-hosted` / `pipelex-platform` / `pipelex-admin-api` / `sandbox` / `cocode` — **none** import the moved config schema or the two moved exceptions (the only external `pipelex.temporal.*` import is `pipelex-api` pulling the *runtime* `WorkflowExecutionError`, which stays in `pipelex/temporal/` and is a normal cut-over pin concern, not a C6 concern). So Step 0b ships with no downstream coordination.++**Remaining Phase 5 (cross-repo, paused for go/no-go):** Step 0a scaffold `../pipelex-temporal` (empty repo, remote `Pipelex/pipelex-temporal`, mirror `pipelex-api`/`pipelex-platform` conventions) → move `pipelex/temporal/` + `temporal_plugin.py` + temporal tests + the `temporal` marker + `--temporal-server` conftest option out, declare the `pipelex.plugins` entry point → flip `pipelex-worker` / `pipelex-api-hosted` pins → repoint `pipelex-mistralai-workflows` → run consumer suites before publishing. Also fold in the Option-A follow-up (delete the now-dead `load_base_config_dict()` / `ensure_global_if_missing=False` in `config_loader.py`).++### Phase 5 — cross-repo cut-over as-built (LOCAL work, pre-publish; 2026-06-20)++**Status: the reversible local cut-over is DONE and green on both sides.** User gave go/no-go for the *reversible local work only* (scaffold + move + flip-to-editable + suites green), explicitly **stopping before any PyPI publish / merge / deploy**. Nothing pushed. Two decisions locked this session: **D8** (package layout = top-level `pipelex_temporal`, not a `pipelex.temporal` namespace stitch — see the header note) and **D9** (scaffold tool-configs sourced from `pipelex`, repo shape from `pipelex-api`).++**Green evidence.**+- Core (`_plugins`): **`make agent-check` clean** (pyright 0 / mypy 0 over 2077 src / keyword-only ✓), **`make tb` green**, **full `make agent-test` = 7678 passed** (after fixing 2 error-handling bundle couplings, below; 4 xfailed + 2 skipped are pre-existing/expected).+- `pipelex-temporal`: **`make agent-check` clean** (ruff/pyright **0** / mypy 0/208), **`make agent-test` "All tests passed"** (≈509 pass, 4 xpassed = the same pre-existing xdist class-registration flake markers as pipelex's baseline). Entry point, `pipelex-temporal` console script (`worker`/`setup-namespace`), and import smokes verified.++**The new dist (`/Users/lchoquel/repos/Pipelex/pipelex-temporal`, branch `main`, NO commits yet at time of writing — local only).** Top-level package `pipelex_temporal/` (64 modules, `pipelex.temporal.*`→`pipelex_temporal.*` rewrite). `pyproject.toml`: deps `pipelex>=0.35.0` + `temporalio==1.24.0` + `aiohttp>=3.14.0`; **`[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }`** (local-dev cross-pin → flips to a published `pipelex==Y` pin at release); `[project.scripts] pipelex-temporal = "pipelex_temporal.temporal_cli:app"`; `[project.entry-points."pipelex.plugins"] temporal = "pipelex_temporal.temporal_plugin:TemporalPlugin"`; tool-configs copied from pipelex; **`[tool.ruff.lint.isort] known-first-party = ["pipelex","pipelex_temporal"]`** so the workflow modules' deliberate `# noqa: TC001` on passed-through `pipelex.*` imports keep working (without it `pipelex` is third-party → TC002 → noqa stripped). `Makefile`+`Makefiles/Makefile_basics.mk` (trimmed from pipelex-api: env/install/lock/format/lint/pyright/mypy/agent-check/agent-test/clean*). README, CHANGELOG (`[Unreleased]`), LICENSE, `.gitignore`, `pipelex_temporal/py.typed`.++**Core (`_plugins`) cut.** `TemporalPlugin` removed from `BUILTIN_PLUGINS` (+ comment that temporal is now an external entry-point dist); `CORE_UNCONDITIONAL_PLUGIN_NAMES` unchanged (`{direct, openai}`). `pyproject.toml`: dropped the `temporal` extra, the `pipelex-temporal` console script (so the new repo's editable install doesn't collide), and the `temporal` pytest marker. `git rm`'d `pipelex/temporal/` + `tests/{unit,integration}/pipelex/temporal/` + 2 scattered moved tests. Core source names no integration (only doc-comments, repointed to `pipelex_temporal`). **`MissingOrchestratorError` install hint changed** `pip install 'pipelex[temporal]'` → `pip install pipelex-temporal` (+ docstring; + the 2 asserting tests `test_exceptions_disclosure.py`/`test_orchestrator_dispatch.py`). Stale `pipelex[temporal]` docstrings in `reporting_manager.py`/`execution_mode.py`/`test_temporal_activity_gate_lazy_import.py` → `pipelex-temporal`. **Option-A fold-in done:** deleted dead `load_base_config_dict()` + the `ensure_global_if_missing` param/branch in `config_loader.py`.++**Test reconciliation (the fiddly part).** Temporal behavioral suite (~110 files) + shared fixtures (`tests/integration/fixtures/` + `tests/integration/conftest.py`), root `tests/conftest.py`, and shared `error_handling/test_data.py` copied into `pipelex-temporal/tests/` (layout `tests/{unit,integration}/pipelex_temporal/`; the blanket rewrite maps both `pipelex.temporal`→`pipelex_temporal` and the test path `tests.integration.pipelex.temporal`→`tests.integration.pipelex_temporal`). **5 mixed core bridge/boot tests split** (core keeps the DIRECT/core-mode methods, the temporal-mode methods move to new pipelex-temporal files): `test_dispatch.py` (3 methods → `test_bridge_temporal_dispatch.py`), `test_temporal_blocking_workflow_id.py` (wholesale), `test_trace_context_contract.py` (→ `test_trace_context_temporal.py`), `test_keyless_boot_forced_dry.py` (→ `test_keyless_temporal_boot.py` under `tests/integration/system/` to escape the temporal conftest's autouse `boot_temporal`), `test_error_type_uri.py` (wholesale). **`test_cv_batch_screening.py` stayed** (DIRECT e2e) — decoupled from the temporal tree via a local `test_data.py` + its own `cv_batch_screening.mthds`. **`test_config_temporal_optional_dep` + `test_non_retryable_baseline_pins` relocated BACK to core** `tests/unit/pipelex/system/` (they validate the core-relocated `config_temporal.py` / `pipelex.toml`, were mis-swept). **2 shared bundles restored to core** `tests/integration/pipelex/error_handling/bundles/{native_text_sequence,native_search}.mthds` (the core local-parity error tests read them; they were mislocated under the temporal tree). `test_import_light_boot.py` reworded (temporalio stays BLOCKED; the temporal slot-claim arm moved to the pipelex-temporal suite — core guarantee "boot imports no temporalio even with is_enabled=True" preserved). pipelex-temporal needed its own `tests/__init__.py` rootdir anchor + a `.pipelex/` project config + a rootdir `conftest.py` (`pytest_plugins` + placeholder creds) for hermetic standalone boot.++**Docs hygiene done:** `tests/CLAUDE.md` temporal row removed; all `pipelex[temporal]` install instructions in `docs/` → `pipelex-temporal`; the stale `MissingPipelexTemporalExtraError`/`MissingMistralWorkflowsPluginError` doc refs → `MissingOrchestratorError`.++> **⚠ CORRECTION 2026-06-21 — read before using this cut-list.** `pipelex-temporal` is **PRIVATE / not-open-source → NOT PyPI.** The `pipelex-temporal==Y` pins written below are the wrong (public-PyPI) form. The correct form is a **`git+ssh` pin** (`pipelex-temporal @ git+ssh://[email protected]/Pipelex/pipelex-temporal.git@<tag>`), matching `pipelex-shared` in `pipelex-platform`. And only **private** repos may pin it: `pipelex-worker` and (newly) `pipelex-api-hosted`'s child image (confirmed 2026-06-21: the hosted runner enqueues Temporal) — the "`pipelex-api-hosted` = no-op" line below is now WRONG. The **public MIT** repos `pipelex-api` and `pipelex-mistralai-workflows` must **DROP** the `temporal` extra instead of repinning it. Reconcile the bullets below to git+ssh when the wiring is actually done.++**DEFERRED — publish/release-gated downstream cut-list (do NOT execute ahead of the pipelex release that ships `pipelex-temporal`):**+- **`pipelex-worker`** — `pyproject.toml:8` `pipelex[dynamodb,s3,temporal]==0.34.0` → `pipelex[dynamodb,s3]==X` + `pipelex-temporal==Y`; **Dockerfile:25** `CMD ["pipelex","worker","--no-sandbox"]` → `["pipelex-temporal","worker","--no-sandbox"]`; **Makefile:37** `$(PIPELEX) worker …` → `pipelex-temporal worker …`; **Makefile:52** editable install `-e "$(PIPELEX_REPO)[dynamodb,s3,temporal]"` → drop `temporal`, add `pipelex-temporal`. Per the Option-A note this MUST land in the same commit as the `pipelex` pin bump (until then the old pinned pipelex still has `pipelex worker`, so nothing breaks).+- **`pipelex-mistralai-workflows`** — `pyproject.toml:41` `temporal = ["pipelex[temporal]>=0.27.0"]` → `temporal = ["pipelex-temporal>=Y"]`; rewrite the ~9 `pipelex.temporal.*` imports in `tests/integration/test_bridge_temporal_{blocking,fire_and_forget}.py` → `pipelex_temporal.*`. Gated on published `pipelex-temporal` AND coordinate with that repo's in-flight mistralai-2.x work (don't disrupt it).+- **`pipelex-api`** pins `pipelex[…,temporal]==0.35.0` (PyPI) → drop `temporal`, add `pipelex-temporal` — publish-gated. **`pipelex-api-hosted`** = config-only wrapper, no Python pin (temporal reaches it through the `pipelex-api` image) — no-op.++**DEFERRED — other follow-ups:**+- `docs/errors/` temporal error pages (`unrecoverable-workflow-failure-error.md`, `workflow-execution-error.md`, `temporal-flow-error.md`, etc.) left in core ON PURPOSE: they are the `type_uri` dereference targets on `docs.pipelex.com`; deleting them before pipelex-temporal's docs exist would 404 those URLs. Moving them (and repointing/aggregating the type_uris) is a docs-deploy task for the publish phase. Do NOT `generate-error-pages` to prune them yet. (`temporal-config-error.md` stays — that error is core.)+- Deeper `docs/under-the-hood/` internal-path references (`temporal-integration.md`, `orchestrator-plugins.md`, `distributed-content-generation.md`, `pipe-routing-and-execution.md`, `error-model.md`) still say `pipelex.temporal` in prose — accurate-keeping pass deferred (the system behaves identically; coupled to the docs-deploy story). `orchestrator-plugins.md` should eventually present temporal as the worked **external** plugin example.+- Cross-repo `error_handling/test_data.py` parity constants are now duplicated in core + pipelex-temporal (can drift; inherent to externalization — no shared-file enforcement across repos).+- Local commits made on both sides (NOT pushed): `_plugins` = Phase 5 checkpoint commit on `refactor/Plugins-2`; `pipelex-temporal` = initial commit on `main`.+- **Post-local-cut-over `/code-review` triage (2026-06-20)** → [`wip/plugins/phase-5-cutover-review-followups.md`](wip/plugins/phase-5-cutover-review-followups.md). xhigh review of the core-side cut-over commit. **Fixed this session** (reachable/core-hygiene that stays in core): kit `commands.md` temporal-test section removed + `CLAUDE.md` regen; `uv.lock` temporal-extra drop; `bridge.py` + `test_config_temporal_optional_dep` docstrings; 5 plugin/cogt test stubs (dropped vestigial `temporal` stub field); **restored two core-coverage tests** — `test_trace_context_contract::test_non_direct_mode_nulls_host_trace_context` (core-side guard for the bridge's non-DIRECT trace_context nulling — distinct from the temporal-mode test that moved to pipelex-temporal) + `test_dispatch_options_no_temporalio.py` (DispatchOptions constructible without temporalio — the `RetryPolicy = Any` forward-ref contract the AST scan can't catch). **Still deferred** (in the wip doc): skill `temporal-e2e-validate` relocation, Makefile temporal targets (`tw`/`ttm`/…, CI-isolated), `debugging-hanging-pytest-runs.md`, boot eager-validation design call, `non_retryable_error_types` cross-repo class-name desync, dead `cv_batch_screening_inputs.json` mirror.+- **Finalize-local pass (session 2, 2026-06-20)** → same wip doc, *Finalize-local pass* section. Closed two of the prior bullet's "Still deferred" items: **both** Temporal skills relocated to `pipelex-temporal` (`temporal-e2e-validate` *and* `temporal-test-crate` — the latter was never on the list, also dead in core) + removed from core, and the **core** Makefile temporal targets deleted (help + `.PHONY` + `test-temporal`/`ttm` + `temporal-server`/`-stop`/`-worker`/`-run`; the `pipelex-temporal` Makefile *add* stays open — unverifiable without a running server). Plus the `pipelex-temporal` *additions*-side `/code-review` → **SHIP/clean**; NITs: stray `.pipelex/traces/*.ndjson` untracked+gitignored, shipped `test_extras→tests` back-import deferred (pre-existing/dormant). Gates: core `make tb` + full `make agent-test` green (HEAD `27da150a5`); pipelex-temporal `make agent-check` clean. **Local, uncommitted; still paused before publish.**diff --git a/wip/plugins/README.md b/wip/plugins/README.mdnew file mode 100644index 000000000..87d2a6205--- /dev/null+++ b/wip/plugins/README.md@@ -0,0 +1,56 @@+# Pipelex plugin architecture++**Status:** assessment / design notes (not started)++> **The decided design lives in [`design.md`](design.md)** — a single unified design covering both inference (driver) plugins and orchestration (strategy) plugins, with decisions and rationale. **The executable [`implementation-plan.md`](implementation-plan.md)** is built from it (phases 0–5, grounded `file:line` refs, per-phase tests, decisions D1–D6, reviewed via `/plan-eng-review` + codex). The four documents here ([`README.md`](README.md), [`inference-backends-as-plugins.md`](inference-backends-as-plugins.md), [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md), [`temporal-as-plugin.md`](temporal-as-plugin.md)) are the background assessment that fed the design. Start with `design.md`, then `implementation-plan.md`; read these for the deeper per-area analysis.++Pipelex has two families of optional integrations that should become real, discoverable **plugins** behind a single shared seam:++1. **[Orchestrators](orchestrators-as-plugins.md)** — *strategy* plugins. They decide *where/how* a pipe runs and replace core orchestration seams. Core ships only the in-process `DIRECT` orchestrator; the distributed ones are optional plugins. Two worked instances:+ - **[Temporal](temporal-as-plugin.md)** — `pipelex/temporal/` → `pipelex-temporal`. The heavy case: boot-global hub swap + per-call modes + CLI + config.+ - **Mistral Workflows** — the external `pipelex-mistralai-workflows` repo. Per-call only; also a plugin for *its* host (a two-host adapter). Already wired (by a hard-coded core import — the coupling to invert).+2. **[Inference backends](inference-backends-as-plugins.md)** — *driver* plugins. The SDK wrappers under `pipelex/plugins/` (OpenAI, Anthropic, Google, Mistral, Bedrock, Fal, Docling, Linkup, …). Homogeneous, many, selected at runtime by a model's `sdk` handle.++All are already ~80–90% decoupled (lazy imports, optional extras, abstract contracts, hub/registry seams). None is yet a *plugin* in the installable sense — orchestrators are still named by string from core (both Temporal *and* Mistral); inference backends are still dispatched from hardcoded `match` statements. The work is the same shape for all: **invert the last coupling through a registry, discover via entry points, optionally repackage.**++> **Naming caveat:** "plugin" is currently overloaded. `pipelex/plugins/` holds in-tree SDK adapters, `Plugin` is a backend selector, `PluginManager`/`PluginSdkRegistry` is an SDK-client cache — none is a real plugin system. The inference doc proposes renaming those before building the real thing. Read that first if the vocabulary trips you up.++---++## Synthesis: best practices for plugins on a codebase like this++Concise, ordered roughly by what to do first.++1. **Invert the dependency, always.** Core must never name a plugin module — not by import, not by string. Plugins depend on core's *published protocols*; core depends on nothing downstream. Every current coupling (Temporal's lazy imports, the inference `match` statements) becomes "ask a registry / call a hook." This is the whole game; the rest is mechanism.++2. **One discovery mechanism: entry points.** Use `importlib.metadata.entry_points(group="pipelex.plugins")`, loaded once at boot. Installing a dist makes it discoverable — zero config, no import-by-string, no scanning. Both strategy and driver plugins ride the same group.++3. **A thin `Plugin` protocol whose hooks are a *menu*, called at known lifecycle points.** Core invokes hooks (`register_hub_implementations`, `register_cli_commands`, `register_orchestrators`, `register_workers`, `setup`, `teardown`); each plugin implements only the subset it needs. Temporal uses nearly all of them (boot-global hub swap + per-call modes + CLI + config); Mistral uses only per-call mode registration. Designing the protocol against *both* keeps it from over-fitting one backend. No magic, no base-class inheritance from core internals — structural typing against a `Protocol`.++4. **Registries replace both `match` statements and hub hard-wiring.** Core owns the registry + the abstract contract (`InferenceWorkerAbstract`, `PipeRouterProtocol`, …); plugins own the entries. Three shapes, one idea: driver registries keyed by data (the `sdk` handle); the execution-mode → orchestrator registry keyed by `PipelexExecutionMode` (per-call dispatch, replacing the bridge `match`); singleton hub setters (boot-global strategy swap). Keep the *taxonomy* enums (`PipelexExecutionMode`) in core so the names exist without the implementations — that's what lets a missing backend produce "install pipelex-temporal" instead of a `None` deref.++5. **Dogfood the plugin API in-tree before externalizing.** Make the built-in backends and Temporal register *through the seam* while still living in the repo. This proves the seam with a green test suite (Phase 1 in both docs) and turns later extraction into a packaging move, not a refactor. Don't externalize on faith.++6. **Lazy SDK imports + a great `MissingDependencyError`.** Never touch a heavy SDK until its plugin is selected. Gate with `importlib.util.find_spec(...)`, and when missing, raise an error naming the lib *and* the extra *and* (where possible) an alternative — the inference factories already do this well; preserve that quality when the guard moves into registration.++7. **Optional extras are a stepping stone, not the destination.** `pipelex[temporal]`, `pipelex[anthropic]` are good today. A real plugin dist packages its own deps, so the extra collapses into the dist's dependency list. Split a dist out only when dependency weight or release cadence justifies it.++8. **Keep config schema importable without the SDK.** The `if TYPE_CHECKING: from sdk import X / else: X = Any` placeholder (see `config_temporal.py`) lets a typed config field survive on installs that skipped the extra. Decide per integration whether the schema stays in core (recommended — keeps static typing) or moves into the plugin (needs a pluggable-config hook).++9. **Detect capability without importing.** Cross-cutting code that must *not* drag an SDK onto the hot path should sniff `sys.modules.get("...")` rather than import (see `reporting_manager.py` detecting a Temporal activity context). Survives extraction unchanged.++10. **Lifecycle symmetry.** `setup()` / `teardown()` hooks, idempotent, with teardown in `finally`/registry cleanup (the `SdkClientRegistry.teardown()` and Temporal teardown already model this). Move the teardown currently inlined in `pipelex.py` behind the plugin so the boot file stops knowing about specific integrations.++11. **Publish the dependency surface as a designed SPI — don't let consumers reach into internals.** Once an out-of-tree plugin depends on you, what it imports *is* a contract. Orchestrators legitimately need more than the runtime bridge (library-crate access, `PipeJob`/`PipeOutput`, the router/run protocols, tracer hooks) — so define an orchestrator SPI rather than an honor-system "this package only" rule (which `pipelex-mistralai-workflows` already breaches). Same theme as the OpenAI *substrate* in the driver doc: a designed surface, not an accidental one. Anything imported outside the SPI is a design bug — promote it or remove the need.++12. **One repo can plug into several hosts.** An adapter is naturally multi-sided (`pipelex-mistralai-workflows` is a Pipelex orchestrator plugin *and* a Mistral Workflows activity library). One wheel, two contracts, two registrations — and they can use different mechanisms (Pipelex: entry-point discovery; Mistral: registration-by-import, since it has no registry). Don't split an adapter whose whole job is bridging two worlds.++13. **Version the core↔plugin contract.** A plugin should declare which core API it targets so an incompatible pairing fails loud at load time, not mysteriously at runtime. "No backward compatibility" is the repo policy — make the contract break *visible*. **Not yet achieved by the implementation** — the shipped `PLUGIN_API_VERSION` gate is documentary only (every plugin imports the constant, so its declared version always equals the installed core and the exact-match gate can never fire; `#1000` already shipped a breaking orchestrator-SPI change without a bump and nothing caught it). The task to build this properly (pin a literal built-against version; semver compat rule) is [`plugin-api-version-gating-semver.md`](plugin-api-version-gating-semver.md).++14. **Tests, markers, and CLI options travel with the plugin.** The `temporal` pytest marker and `--temporal-server` option move with `pipelex-temporal`; backend tests move with their dist. Keep protocol-level conformance tests in core so any plugin can be checked against the contract.++15. **Fail loud on a broken plugin.** If a discovered entry point fails to import or register, surface it with context — don't silently skip (a silently-missing backend looks like a config bug to the user). Distinguish "not installed" (fine, expected) from "installed but broken" (loud).++### The one-line version++> Define a single entry-point–discovered `Plugin` protocol; turn every place core currently *names* an integration (Temporal's lazy imports, the inference `match` statements) into a *registry lookup* or *hook call*; prove it in-tree with a green suite; repackage only what's worth repackaging.diff --git a/wip/plugins/design.md b/wip/plugins/design.mdnew file mode 100644index 000000000..9d74ba736--- /dev/null+++ b/wip/plugins/design.md@@ -0,0 +1,482 @@+# Pipelex plugin system — design++**Status:** design (proposed, not started)+**Supersedes for design intent:** the four assessment notes in this folder ([`README.md`](README.md), [`inference-backends-as-plugins.md`](inference-backends-as-plugins.md), [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md), [`temporal-as-plugin.md`](temporal-as-plugin.md)) are the research that fed this document. This is the single, decided design the implementation plan is built from.++This document defines one plugin system covering both families of optional integration Pipelex has today: **inference backends** (the SDK adapters selected by a model's `sdk` handle) and **orchestrators** (the strategies that decide where/how a whole pipe runs — Temporal, Mistral Workflows). They are different in shape but share one discovery mechanism, one plugin contract, and one set of design rules.++---++## 1. Motivation++Pipelex already has both integration families, and both are already ~80–90% decoupled — lazy imports, optional extras, abstract contracts, hub/registry seams. What's missing in both is the *last* inversion: core still **names** each integration.++- **Inference backends** are dispatched from centralized `match plugin.sdk:` statements inside four worker factories (`cogt/{llm,img_gen,extract,search}/*_worker_factory.py`). A third party cannot add a backend without editing core.+- **Orchestrators** are named by string in `runtime_bridge/bridge.py`'s `match input_payload.execution_mode:` — the `TEMPORAL_*` arms lazy-import `pipelex.temporal`, the `MISTRAL_NATIVE` arm **hard-imports** `pipelex_mistralai_workflows`. Temporal is further wired into core boot/teardown (`pipelex.py`), config (`system/configuration/configs.py`, the one hard import of the package), and the CLI (`cli/_cli.py`, two hard imports).++The job is the same for both: **invert the last coupling through a registry, discover via entry points, and (only where it pays off) repackage.** This is not a rewrite of the integration code — it is a seam in core plus rewiring a small, enumerated set of call sites.++### The naming overload (fix first)++"Plugin" is currently overloaded three ways in `pipelex/plugins/`, and the ambiguity has already leaked into the codebase (the hub raises `"PluginManager2 is not initialized"`). None of the three is a plugin in the installable sense:++- `pipelex/plugins/<backend>/` — directories holding **in-tree SDK adapters**.+- `Plugin` (`plugins/plugin.py`) — a value object (`sdk` + `backend` + `variant`, with an `sdk_handle` like `anthropic@bedrock/claude`). A **backend selector / handle**.+- `PluginManager` / `PluginSdkRegistry` — a **runtime cache of instantiated SDK clients**, keyed by that handle.++The word "plugin" is reserved, from here on, for the new concept: an installable, discoverable unit that registers extensions at boot. See §4 for the rename.++---++## 2. Goals and non-goals++### Goals++- **Core names no integration.** No import of an integration module, no string literal naming one, anywhere on the core code path.+- **One discovery mechanism** for both families: the `pipelex.plugins` entry-point group, plus a static built-in list for in-tree plugins.+- **One plugin contract** — a small, menu-style registration surface that a plugin implements only the subset of.+- **Dogfood in-tree first.** Built-in backends and Temporal register *through the seam* while still living in the repo, proven by a green suite, before anything is externalized. Externalization then becomes a packaging move, not a refactor.+- **Designed dependency surfaces (SPIs).** What an out-of-tree plugin imports is a contract; publish it deliberately instead of letting plugins reach into internals.+- **Preserve today's user-facing behavior and error quality** — same model selection by data, same friendly "install the X extra" messages, same execution modes.++### Non-goals++- **Not** splitting every vendor backend into its own dist. In-tree-but-registerable is the default; externalize only where dependency weight or release cadence justifies it (Temporal qualifies; vendor inference dists do not, yet).+- **Not** a third-party-defined execution-mode space. Plugins supply orchestrators for the *existing* `PipelexExecutionMode` values; inventing new modes is an open string-keyed space we defer until something needs it.+- **Not** a generic typed-config-contribution mechanism in v1. First-party plugin config (Temporal's) stays typed in core; a generic per-plugin config namespace is a documented future extension, built when a third party needs it.+- **Not** backward compatibility shims. Per repo policy, breaks are noted in changelogs and made loud, not smoothed over.++---++## 3. Concepts and vocabulary++| Term | Meaning |+|---|---|+| **Plugin** | An installable, discoverable unit that registers extensions into core at boot. May ship in-repo (built-in) or as a separate dist (external). Implements the plugin contract (§5). |+| **Driver plugin** | A plugin that *supplies an implementation* for a backend core asked for. Many, homogeneous, selected by data (a model's `sdk` handle). The inference backends. |+| **Orchestrator** | A strategy that decides *where/how a whole pipe job runs* (in-process, on a Temporal fleet, on Mistral Workflows). Distinct from a *pipe controller*, which sequences sub-pipes within one run. |+| **Orchestrator plugin** | A plugin that *replaces core orchestration behavior* for one or more execution modes. Few, heterogeneous. |+| **SPI** (Service Provider Interface) | The published, versioned surface of core that plugins are allowed to import. Two of them: an inference SPI and an orchestrator SPI. |+| **`ModelHandle`** | The renamed value object (was `Plugin`): `sdk` + `backend` + `variant`, with `sdk_handle`. Selects which driver runs a model. |+| **`SdkClientRegistry` / `SdkClientManager`** | The renamed runtime cache (was `PluginSdkRegistry` / `PluginManager`): instantiated SDK clients keyed by `sdk_handle`, with lifecycle teardown. |++"Plugin" and "ships as a separate dist" are **orthogonal axes**. Being a plugin means *registering through the seam* instead of being a hardcoded special case. Shipping separately is a packaging decision. A built-in plugin that lives in this repo is the default, not a contradiction.++| | In-repo | Separate dist |+|---|---|---|+| **Through the seam** | built-in plugins (the default for every backend) | external plugins (`pipelex-temporal`, `pipelex-mistralai-workflows`, future vendor dists) |+| **Hardcoded in core** | the wart we are deleting | n/a |++---++## 4. Architecture overview++One discovery pass produces a list of plugin objects. Each plugin is handed a **registrar** and fills whichever of three registry shapes it serves. Core then consults those registries at the points where it used to name an integration. Plugins import only the published **SPIs**.++```+boot+ └─ discover plugins ─── built-in static list (in-repo: openai, gateway, anthropic, …, temporal)+ └─── pipelex.plugins entry points (external dists)+ │+ ▼ for each plugin (after API-version check): plugin.register(registrar)+ registrar fans contributions into three shapes:+ ├─ keyed registries (lookup by data)+ │ ├─ InferenceBackendRegistry — keyed by (family, sdk handle) ← driver plugins+ │ └─ OrchestratorRegistry — keyed by PipelexExecutionMode ← orchestrator plugins (per-call)+ ├─ exclusive hub slots (boot-global swap, at most one claimant)+ │ └─ content_generator / pipe_router / pipe_run / task_manager ← orchestrator plugins (boot-global)+ └─ additive contributions+ ├─ CLI commands (Typer)+ └─ teardown callbacks (run LIFO at shutdown)+ │+ ▼ core consults the registries where it used to name an integration:+ worker factories → InferenceBackendRegistry.lookup(family, sdk)+ bridge dispatch → OrchestratorRegistry.get(execution_mode)+ hub defaults → whatever claimed each exclusive slot, else core default+```++The three registry shapes are deliberate — they match the three ways an integration plugs in:++- **Keyed registries** — looked up by runtime data. Inference backends keyed by `(family, sdk)`; orchestrators keyed by `PipelexExecutionMode`. Many entries, additive.+- **Exclusive hub slots** — a boot-global *swap* of a singleton (the default content generator / pipe router / pipe run / task manager). At most one plugin may claim each slot; two claimants is a loud conflict. Only Temporal uses these today.+- **Additive contributions** — CLI commands and teardown callbacks. Appended, never exclusive.++Two SPIs sit underneath:++- **Inference SPI** — the contracts a driver plugin compiles against (the `*WorkerAbstract` hierarchy, `ModelHandle`, `MissingDependencyError`, the `SdkClientRegistry`, the registration factory contract, and the OpenAI-compat substrate).+- **Orchestrator SPI** — the contracts an orchestrator plugin compiles against (the execution protocols, the boundary/core payload types, library-crate access, tracing hooks). Sized to what the existing orchestrators actually import (§9).++---++## 5. Discovery and the plugin contract++### 5.1 The registrar pattern++A plugin is an object exposing identity plus a single `register` method:++```python+@runtime_checkable+class PipelexPlugin(Protocol):+ name: str+ targets_api: int # the PLUGIN_API_VERSION it was built against+ def register(self, registrar: PluginRegistrar) -> None: ...+```++`register` is the *menu*: the plugin calls whichever registrar methods it needs and ignores the rest. This is preferred over a six-method "god protocol" (every plugin stubbing methods it doesn't use) or per-capability `isinstance` dispatch (reflection that ignores signatures): one method, one well-defined call point, no magic.++```python+class PluginRegistrar:+ # keyed registries+ def add_inference_backend(self, *, family: InferenceFamily, sdk: str, factory: InferenceBackendFactory) -> None: ...+ def add_orchestrator(self, *, mode: PipelexExecutionMode, orchestrator: OrchestratorProtocol) -> None: ...+ # exclusive boot-global slots (raise on a second claimant)+ def claim_content_generator(self, generator: ContentGeneratorProtocol) -> None: ...+ def claim_pipe_router(self, router: PipeRouterProtocol) -> None: ...+ def claim_pipe_run(self, pipe_run: PipeRunProtocol) -> None: ...+ def claim_task_manager(self, task_manager: TaskManagerProtocol) -> None: ...+ # additive+ def add_cli_command(self, *, name: str, help: str, command: Callable[..., object]) -> None: ...+ def add_teardown(self, callback: Callable[[], None]) -> None: ...+ # read access to the resolved config, so a plugin can gate its own contributions+ @property+ def config(self) -> ConfigRoot: ...+```++A plugin gates its own contributions on config. Temporal's `register` claims the exclusive hub slots **only when `config.temporal.is_enabled`**, but registers its per-call orchestrators and CLI **unconditionally** (so the bridge can dispatch a `TEMPORAL_*` run, and the worker CLI exists, regardless of whether *this* process boots as a Temporal-default runtime). This split is the heart of the two activation models — see §8.++### 5.2 Built-ins vs externals (one contract, two discovery paths)++- **Built-in plugins** are registered from a static list in core (`BUILTIN_PLUGINS = [OpenAIDriver(), GatewayDriver(), AnthropicDriver(), …, TemporalOrchestrator(), …]`). Explicit, fast, no self-scanning of dist metadata. One plugin per *externalizable unit* (per vendor for drivers; one for Temporal), so promoting a built-in to an external dist later is "move it out of the static list, add an entry point in its own dist."+- **External plugins** are discovered via `importlib.metadata.entry_points(group="pipelex.plugins")`, loaded once at boot. Installing a dist makes it discoverable — zero config.++Both paths produce `PipelexPlugin` objects fed through the identical `register(registrar)` pipeline. The seam is dogfooded by every built-in; entry-point discovery is exercised by every external (and by Mistral, which is already a separate repo).++Rationale for the hybrid over pure entry-point discovery for built-ins: declaring a dozen self-referential entry points in `pipelex`'s own `pyproject.toml` and having `pipelex` discover *itself* via `importlib.metadata` is needless ceremony. A static list is clearer and faster, and externalization stays a one-line move. (Alternative considered: pure entry points for built-ins too — rejected as over-engineered for the in-tree default.)++### 5.3 Ordering, timing, and safety++- Discovery and `register` run at one well-defined point in boot — after config is loaded (so plugins can gate on it), before the first worker or pipe run is built. Boot-global slot claims (§8) are *collected* at this point and *applied* at the existing ordered setup points in `Pipelex.setup()` (content generator, then task manager, then router, then run), each falling back to the core default when unclaimed.+- **Core defaults are unconditional.** The `DIRECT` orchestrator and every core-dependency inference backend (`openai`, `gateway`, `portkey`, `pypdfium2`, `azure_rest`) register from core's built-in list and can never be dropped by a misconfigured environment.+- **Registration is import-light** (the lazy-load contract — see §6.5). Building `BUILTIN_PLUGINS` and running every plugin's `register` happens at boot for *all* backends; it must not import any backend SDK. Heavy imports stay inside the factory's `make_worker` / the orchestrator's `run`, exactly where the `match` arms lazy-import today. A plugin object and its `register` therefore live in import-light modules; the SDK is touched only when the backend is actually selected.+- **Conflict policy: duplicate keys fail loud.** Registering a second factory for the same `(family, sdk)` or a second orchestrator for the same `execution_mode` is a startup error naming both contributors — never a silent shadow. Same for an exclusive hub slot claimed twice (§8). This is consistent with the repo's "make breaks visible" stance and guards the transition window where an externalized backend could collide with a still-present built-in. *Deliberately overriding* a built-in (a third party replacing core's OpenAI driver) is a separate, explicit feature — deferred until something needs it, never an accident.+- **Teardown** runs registered callbacks LIFO at shutdown, replacing the Temporal teardown currently inlined in `pipelex.py`.++### 5.4 Versioning the contract++Core exposes `PLUGIN_API_VERSION: int`. Each plugin declares `targets_api`. On load, an incompatible pairing fails loud at startup (naming the plugin, its target, and the core version) — never mysteriously at runtime. A single integer, bumped on any breaking change to the contract or either SPI; "no backward compatibility" means we make the break *visible*, not that we tolerate silent skew.++The single integer is deliberately coarse: a change to *either* SPI bumps it, forcing *every* plugin to re-declare even if its own surface was untouched. With the small set of external plugins we control (Temporal, Mistral), that friction is acceptable and the simplicity wins. Per-SPI versions or semver-range matching are deferred until there are external plugins on independent release trains.++### 5.5 Fail-loud, and "not installed" vs "broken"++- **Not installed** is expected and friendly: a requested backend/mode whose plugin isn't present yields a typed error naming the package *and* the extra *and*, where one exists, an alternative (e.g. "use Anthropic via Bedrock instead"). This is preserved verbatim from today's `MissingDependencyError` quality. Today's typed orchestrator errors (`MissingPipelexTemporalExtraError`, `MissingMistralWorkflowsPluginError`) map onto the registry-miss path — the bridge's `MissingOrchestratorError` carries the same "install `pipelex-temporal`" guidance instead of an `ImportError` caught inline.+- **Installed but broken** is loud: a discovered entry point that fails to import or whose `register` raises is surfaced with context — never silently skipped (a silently-missing backend reads to the user as a config bug).+- The **taxonomy enums stay in core** (`PipelexExecutionMode`, the `requires_*` properties). Core knows the *names* of strategies without their *implementations* — that is what lets a missing orchestrator produce "install pipelex-temporal" instead of a `None` dereference, and keeps the gates statically typed.++### 5.6 Developer experience for plugin authors++The plugin author *is* the user of this system, and the SPI + contract *is* the product they touch. Treat that surface with the same care as a public API.++- **Progressive disclosure — learn only your slice.** The registrar menu means a backend author calls only `add_inference_backend` and implements one `InferenceBackendFactory`; an orchestrator author calls only `add_orchestrator` (plus, for the boot-global case, the `claim_*` slots). Neither has to learn the other family, the hub, or the parts of the SPI they don't use. The minimal backend plugin is: a `PipelexPlugin` (name + `targets_api` + `register`), one factory with `make_worker`, and (if external) one entry-point line. That short path is the time-to-first-plugin we optimize for.+- **Author-facing error empathy.** Errors caused by a plugin's *own* mistake must name the offending plugin and the fix, never surface as an opaque core failure: a factory whose `make_worker` returns the wrong `*WorkerAbstract` for its family, a registration under the wrong `(family, sdk)`, a duplicate key (§5.3), or a missing required identity field. Validate at registration where it's cheap (identity present, `targets_api` set), and at first use where it isn't (worker type matches family) — each error names the plugin.+- **Actionable upgrades (fight upgrade fear).** A `targets_api` mismatch fails loud with the *remedy* in the message: "plugin `pipelex-foo` targets plugin API N, this `pipelex` is API M — upgrade `pipelex-foo`, or pin `pipelex<X`." Any `PLUGIN_API_VERSION` bump or SPI change ships with a migration note in the changelog so an external author knows what moved and why.+- **A getting-started guide and a minimal example are part of the deliverable** (§16), not a follow-up — an SPI without an authoring guide has no real time-to-first-plugin. The example is the canonical "hello world" plugin authors copy from.++---++## 6. Inference (driver) plugins++### 6.1 The seam today++Four worker factories, each a centralized `match plugin.sdk:` that, per arm, does the same three things: optionally `importlib.util.find_spec(...)` to raise a friendly `MissingDependencyError`, lazily import the backend's factory + worker, then build the worker (reusing a cached SDK client from `SdkClientRegistry`). The lazy import inside each arm is already "don't touch the SDK until selected" — it just needs to become "ask the registry for the selected backend's factory."++| Family | File | Selected by |+|---|---|---|+| LLM | `cogt/llm/llm_worker_factory.py` | `plugin.sdk` (gateway/portkey/openai/anthropic/mistral/bedrock/google variants) |+| ImgGen | `cogt/img_gen/img_gen_worker_factory.py` | `plugin.sdk` (gateway/fal/huggingface/openai/blackboxai/azure_rest/google/openrouter variants) |+| Extract | `cogt/extract/extract_worker_factory.py` | `plugin.sdk` (gateway/mistral/pypdfium2/docling/linkup variants) |+| Search | `cogt/search/search_worker_factory.py` | `plugin.sdk` (linkup/gateway variants) |++### 6.2 The registry and the factory contract++One `InferenceBackendRegistry`, keyed by `(family, sdk)`, mapping to an `InferenceBackendFactory`. Core owns the registry and the abstract contracts; driver plugins own the entries.++```python+class InferenceFamily(StrEnum):+ LLM = "llm"+ IMG_GEN = "img_gen"+ EXTRACT = "extract"+ SEARCH = "search"++class InferenceBackendFactory(Protocol):+ def make_worker(+ self,+ *,+ inference_model: InferenceModelSpec,+ backend: InferenceBackend,+ sdk_clients: SdkClientRegistry,+ reporting_delegate: ReportingProtocol | None,+ ) -> InferenceWorkerAbstract: ...+```++Each worker factory collapses to a lookup:++```python+factory = inference_backend_registry.lookup(family=InferenceFamily.LLM, sdk=model_handle.sdk)+return factory.make_worker(inference_model=…, backend=…, sdk_clients=…, reporting_delegate=…)+```++The `find_spec` guard and its friendly `MissingDependencyError` move **into the factory's `make_worker`**, preserving today's semantics: a model needing an absent extra fails *when used*, with the same helpful message — not at boot. A registry *miss* (backend not registered at all) is a distinct, equally-loud error ("backend `<sdk>` is not registered; is its plugin installed?").++Inconsistent factory shapes today (some have a factory class, some instantiate the worker inline) are normalized onto this one `InferenceBackendFactory` contract so every entry looks the same.++**The refactor is a DRY win, not just a relocation.** Every `match` arm today repeats the same `sdk_clients.get(...) or sdk_clients.set(...)` client-cache dance and the same `find_spec → MissingDependencyError(lib, extra, msg)` shape. Lift both into the shared surface: the client-memoization belongs in a small base (`make_worker` asks `sdk_clients.get_or_create(handle, build_client)`), and the extra-guard becomes one helper (`require_sdk(spec="anthropic", extra="anthropic", msg=…)`). Each backend factory then carries only its genuinely-unique parts: which client to build and which worker to wrap. This collapses the largest block of duplication in the current factories.++### 6.3 Cross-family vendors++`mistral` appears in LLM and Extract; `google` in LLM and ImgGen; `linkup` in Extract and Search; `gateway_*` in all four. A vendor's driver plugin simply calls `add_inference_backend` once per `(family, sdk)` it serves. With `(family, sdk)` as the key this is natural — no special handling, and a future per-vendor dist registers into multiple families from its one `register`.++### 6.4 OpenAI: substrate vs driver++The OpenAI SDK is two concerns with opposite fates:++- **Substrate** — the OpenAI-compatible client (`AsyncOpenAI(base_url=…)`), the shared `ChatCompletion*` types, and the completions/responses plumbing that `gateway`, `portkey`, `openrouter`, `blackboxai`, `azure_rest` reuse (and that `anthropic`/`mistral`/`google` import for shared types). This is **infrastructure other backends import**, not a backend. It belongs in **core as an always-on library** (`openai` is already a core dependency). If it were an optional dist, a dozen backends would diamond-depend on it.+- **Driver** — the `openai` / `azure_openai` / `openai_img_gen` registrations. An **always-on built-in plugin, through the seam** — no extra, registered unconditionally, but with no privileged `match` arm. The seam's whole value is that core names no backend; a surviving special case is a two-tier system the next "important" backend will want to join.++**Decision:** the OpenAI driver is an always-on built-in plugin in v1. Promoting the substrate into a deliberately-named core module (e.g. `pipelex.openai_compat`) with a stable signature is **deferred to the externalization phase** — it only becomes a *contract* problem when an out-of-tree dist imports it, and in-tree backends importing each other's modules is acceptable. Doing it earlier is churn with no in-tree payoff. (Captured as the first step of any future "externalize an OpenAI-compatible backend" work.)++### 6.5 Preserving lazy loading (import-light registration)++The current lazy-load behavior — never touch a backend's SDK until a model selects it — is load-bearing for boot time across the CLI, the API, the workers, and the test suite. The registry must preserve it exactly, and that imposes one hard constraint: **registration is import-light**.++- The built-in plugin objects and their `register` methods live in modules that import no backend SDK. `BUILTIN_PLUGINS = [AnthropicDriver(), …]` must be constructible without importing `anthropic`.+- `add_inference_backend(family, sdk, factory)` receives a factory whose construction is also SDK-free. The backend's worker/client modules are imported *inside* `make_worker`, mirroring today's per-arm lazy import.+- The `find_spec` guard runs inside `make_worker` too, so an absent extra fails at use with the friendly message — never at boot.++A boot-time test asserts the invariant (booting with the built-ins registered imports no optional SDK; see §14.3). This is the one place where a careless refactor would silently regress performance, so it gets an explicit guard.++### 6.6 Packaging granularity++In-tree-but-registerable is the v1 destination for all inference backends: third parties can add a backend in their own dist without a core PR, the `match` statements die, and there is no distribution split to maintain. Per-vendor or bundle dists (`pipelex-anthropic`, `pipelex-aws` = bedrock+s3+dynamodb, …) are **deferred** — split a backend out only when its dependency weight or release cadence actually justifies it. The seam makes that a packaging move.++---++## 7. Orchestration (strategy) plugins — the category++An orchestrator answers *"where/how does this pipe run?"*. Pipelex already models the choice as `PipelexExecutionMode`:++- **`DIRECT`** — in-process async. The trivial orchestrator. **Ships in core**, registered through the seam like everything else.+- **`TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`** — distributed via Temporal. Today in `pipelex/temporal/`; destined for `pipelex-temporal`.+- **`MISTRAL_NATIVE`** — distributed via Mistral Workflows. Already in the external `pipelex-mistralai-workflows` repo.++`DIRECT` is the only mode that belongs in core. The other two are **peers** — optional, discovered, neither privileged. Mistral is the proof that "orchestrator" is a category with more than one member, so the seam must be generic and N-ary, not "Temporal plus a special case." Core today fails this test: it names *both* external orchestrators by string.++---++## 8. Orchestration — the two activation models++The single most important subtlety: an orchestrator plugs into **two distinct execution entry points**, and a complete plugin serves both.++- **Per-call dispatch (the bridge).** Host runtimes call `run_pipe_via_bridge(...)` and pick an orchestrator by `execution_mode` for each run. This is the `OrchestratorRegistry`, keyed by mode. Both Temporal modes *and* Mistral use this. Registrar hook: `add_orchestrator(mode=…, orchestrator=…)`.+- **Boot-global default (the hub).** A normal (non-bridge) pipe run resolves `get_pipe_run()` / `get_pipe_router()` / `get_content_generator()` from the hub. Inside a Temporal worker these defaults must *be* the Temporal implementations, because the content generator has to dispatch each operation as an activity. This is the **exclusive hub-slot swap**, claimed only when the plugin's config slice is enabled. Registrar hooks: `claim_content_generator/pipe_router/pipe_run/task_manager`.++These are orthogonal and both necessary:++| | Per-call (bridge) | Boot-global (hub default) |+|---|---|---|+| Entry point | `run_pipe_via_bridge`, by `execution_mode` | `get_pipe_run()` / `get_pipe_router()` / `get_content_generator()` |+| Shape | keyed `OrchestratorRegistry` | exclusive hub slots |+| Used by | host runtimes selecting a mode per call | this process running as a Temporal-default runtime |+| Temporal | registers both modes, unconditionally | claims slots only when `temporal.is_enabled` |+| Mistral | registers `MISTRAL_NATIVE`, unconditionally | claims nothing — installs its router *inside* the child workflow at runtime |++Mistral is the counter-example that keeps the contract honest: it uses only `add_orchestrator` and nothing at boot. Temporal is the heaviest instance — it exercises per-call orchestrators, the boot-global swap, CLI commands, lifecycle teardown, and a config slice. Designing the registrar against *both* keeps it from over-fitting Temporal.++### 8.1 The bridge collapse++`bridge.py`'s `match input_payload.execution_mode:` — whose arms hard-code Mistral and lazy-import Temporal — collapses to a registry lookup:++```python+orchestrator = orchestrator_registry.get(input_payload.execution_mode)+if orchestrator is None:+ raise MissingOrchestratorError(+ mode=input_payload.execution_mode,+ hint="install pipelex-temporal / pipelex-mistralai-workflows",+ )+return await orchestrator.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)+```++After this, `bridge.py` names no integration. `DIRECT` registers from core and flows through the same lookup. The `requires_pipelex_temporal` / `requires_mistral_workflows_extra` properties on the enum stay as the typed source of truth for *which modes need what*.++**Correctness watch — the DIRECT orchestrator must keep its router scoping.** Today `_run_direct` wraps the run in `scoped_pipe_router(PipeRouter())` so that, inside a Temporal-enabled worker (whose hub-default router is the Temporal one), a `DIRECT` call's nested controller sub-pipes resolve the in-process router instead of leaking to Temporal. The extracted `DirectOrchestrator.run` must preserve this scope verbatim — dropping it is a silent correctness bug that only surfaces for DIRECT-inside-a-Temporal-worker (a real configuration). A regression test pins it (§14.3).++### 8.2 Boot/teardown collapse++The four `if get_config().temporal.is_enabled: from pipelex.temporal…` blocks in `Pipelex.setup()` (content generator, task manager, pipe router, pipe run) and the matching teardown become: core consults the exclusive hub slots at the same ordered points, using whatever a plugin claimed, else the core default. The Temporal plugin's `register` populates those slots (gated on its config). The teardown inlined in `pipelex.py` becomes a registered teardown callback. CLI registration in `_cli.py` (today two hard imports) becomes plugin-contributed commands.++---++## 9. The published SPIs++What an out-of-tree plugin imports *is* a contract. We publish two SPIs deliberately rather than letting plugins reach into internals, and we size them to what the orchestrators **actually import today** — measured, not guessed.++### 9.1 Inference SPI++The surface a driver plugin compiles against:++- the worker contracts: `InferenceWorkerAbstract` → `{LLM,ImgGen,Extract,Search}WorkerAbstract`;+- `InferenceModelSpec`, `ModelHandle`, `InferenceBackend` (the backend config record);+- `MissingDependencyError` and the inference error base;+- `SdkClientRegistry` (client memoization + teardown);+- the `InferenceBackendFactory` / `InferenceFamily` registration contract;+- *(deferred)* the `openai_compat` substrate, once extracted (§6.4).++### 9.2 Orchestrator SPI++A measurement of `pipelex-mistralai-workflows`'s actual imports shows the real surface an out-of-process orchestrator needs — far beyond the honor-system "`runtime_bridge.*` only" rule it already breaches:++- **`runtime_bridge.*`** — `bridge`, `bootstrap`, `execution_mode`, `exceptions`, **and `runtime_bridge.primitives.*`** (`delivery`, `hydration`, `pipe_classification`, `submitter_hydration`, `trace_flush`). The primitives live under `runtime_bridge` and are clearly the intended host-runtime helper surface; sanction them explicitly.+- **execution protocols** — `PipeRouterProtocol`, `PipeRunProtocol`, `ContentGeneratorProtocol`, and the task-manager protocol.+- **boundary/core payload types** — `PipeJob`, `PipeOutput`, `DeliveryAssignment`, `WorkingMemory` (+ factory), `JobMetadata`, the `LibraryCrate`.+- **library-crate access + hub scoping** — `set/get_current_library`, `scoped_pipe_router`, `get_class_registry` (per-call library hydration; Mistral already does this via `library_crate_dump`).+- **tracing/graph hooks** — `trace_events`, `graph_tracer_manager`, `tracing_assembly` — an orchestrator must integrate with these to emit per-step trace/usage events across the boundary.++Anything an orchestrator imports *outside* the SPI is a design bug to resolve — promote it into the SPI or remove the need. This is the same theme as the OpenAI substrate becoming a deliberate contract: once an out-of-tree consumer depends on you, the dependency surface must be designed, not accidental.++**Mistral boundary resolution.** `pipelex-mistralai-workflows` documents (its `wip/boundary-violation-mistral-native.md`) that the native tier imports `pipelex.hub`, `pipelex.pipe_run.*`, `pipelex.core.*`, `pipelex.graph.*`, `pipelex.tracing.*` — outside its declared surface. The resolution is **(A) widen the published surface to the SPI above**, then re-point the plugin at it — *not* keep an honor-system rule that's already broken. Until the SPI lands, the plugin's pinned-rev coupling to `pipelex` is the pragmatic interim (already in place).++### 9.3 SPI shape++The SPI is a **documented, versioned set of modules and symbols** — not an `__init__.py` re-export shim (the repo bans re-exports). It is gated by `PLUGIN_API_VERSION` (§5.4). The plan decides whether to additionally provide a thin curated import surface; the contract itself is the documented module/symbol list plus the version marker.++---++## 10. Configuration++Temporal is the only integration with a config slice today (`temporal: Temporal` on `PipelexConfig`, the one hard import of the package in `configs.py`). The `Temporal` schema is pure data with **no `temporalio` runtime dependency** (it uses the `if TYPE_CHECKING: from temporalio.common import RetryPolicy / else: RetryPolicy = Any` placeholder), so it is importable on installs that skipped the extra.++**Decision: keep first-party plugin config schemas typed in core.** The `Temporal` schema stays in `pipelex`; only the *implementation* moves to `pipelex-temporal`. This keeps the root config statically typed, keeps boot's config-load test (`make tb`) meaningful, and makes the config seam a near-no-op — the implementation move does not drag config plumbing with it.++A **generic per-plugin config namespace** for *third-party* plugins (which cannot add typed fields to core's model) is a documented future extension, not built in v1 — there are no third-party plugins yet, and building the mechanism now is speculative. When one needs config, add a `plugins: dict[str, dict[str, Any]]` namespace each plugin parses its own slice of. (Alternative considered: move Temporal config out to the plugin via a pluggable section now — rejected: loses static typing on that subtree and adds validation plumbing for no present benefit.)++---++## 11. Packaging and distribution++| Integration | v1 home | Externalization |+|---|---|---|+| Core (`DIRECT`, openai/gateway/portkey/pypdfium2/azure_rest drivers, openai-compat substrate) | in `pipelex`, always-on | never leaves core |+| Vendor inference drivers (anthropic, google, mistral, bedrock, fal, docling, linkup, huggingface) | in `pipelex`, through the seam | deferred — split only when dep weight / cadence justifies |+| Temporal | in `pipelex/temporal/`, through the seam (Phase: wire first) | **→ `pipelex-temporal`** (depends on `pipelex` + `temporalio`, declares the entry point). High value: independent cadence, heavy SDK, already ~90% separable, and downstream (`pipelex-worker`, `pipelex-api-hosted`) already pin temporal separately. |+| Mistral Workflows | already `pipelex-mistralai-workflows` (external repo) | flip the **hard import** to **entry-point discovery**; resolve its boundary against the SPI; flip its dev path-pin to a published `pipelex` once the SPI lands |++### 11.1 Tests, markers, and CLI travel with the plugin++When an integration is externalized, its tests move with it. The `temporal` pytest marker and the `--temporal-server` CLI option (in `conftest.py`) relocate to `pipelex-temporal`; backend tests move with their dist. **Protocol-level conformance tests stay in core** so any plugin — built-in or external — can be checked against the contract. Downstream pins flip from `pipelex[temporal]==X` to `pipelex-temporal==Y` (which itself pins a compatible `pipelex`).++### 11.2 One repo, two hosts (the adapter shape)++`pipelex-mistralai-workflows` depends on **both** `pipelex` and `mistralai-workflows`: a Pipelex orchestrator plugin on one side, a Mistral Workflows activity/workflow library on the other. The two directions use **different mechanisms** and must not be conflated:++- **→ Pipelex: entry-point discovery.** `[project.entry-points."pipelex.plugins"]` makes `MISTRAL_NATIVE` available; core stops hard-coding the import.+- **→ Mistral: registration-by-import.** Mistral has no plugin registry; the user (or the package's `register_pipelex_primitives(...)` helper) hands workflow/activity classes to their worker. No entry point. (Temporal is identical on *its* host — you always register workflows/activities with a worker explicitly.)++This is the natural shape for an adapter — do not split it.++---++## 12. Worked examples++**A driver plugin (built-in, e.g. Anthropic).** Implements `PipelexPlugin`; its `register` calls `add_inference_backend(family=LLM, sdk="anthropic", factory=AnthropicLLMBackendFactory())` (and `bedrock_anthropic` likewise). The factory's `make_worker` runs the `find_spec("anthropic")` guard → friendly error, builds/caches the client via `sdk_clients`, returns an `AnthropicLLMWorker`. Lives in core's `BUILTIN_PLUGINS`. Externalizing later = move to `pipelex-anthropic` + add an entry point.++**The Temporal orchestrator plugin (heaviest).** Its `register`: always `add_orchestrator(TEMPORAL_BLOCKING, …)` and `add_orchestrator(TEMPORAL_FIRE_AND_FORGET, …)`; always `add_cli_command("worker", …)` and `add_cli_command("setup-temporal-namespace", …)`; **if `config.temporal.is_enabled`**, `claim_content_generator(ContentGeneratorInWorkflow())`, `claim_pipe_router(make_temporal_pipe_router())`, `claim_pipe_run(make_temporal_pipe_run())`, `claim_task_manager(TemporalTaskManager())` and `add_teardown(...)` for the task manager. Config schema stays in core; implementation moves to `pipelex-temporal`.++**The Mistral orchestrator plugin (per-call only).** Its `register`: `add_orchestrator(MISTRAL_NATIVE, MistralWorkflowsOrchestrator())`. Nothing at boot. Installs its own router inside the child workflow at runtime. Discovered via its entry point.++**An out-of-tree third-party backend.** A dist declaring `[project.entry-points."pipelex.plugins"] my-backend = "my_pkg.plugin:MyBackendPlugin"`, depending on `pipelex>=X` and its own SDK. Installing it makes its `sdk` selectable from model TOML — no core change.++---++## 13. Key decisions++| # | Decision | Choice | Why / alternative |+|---|---|---|---|+| D1 | Naming | `Plugin`→`ModelHandle`; `PluginManager`/`PluginSdkRegistry`→`SdkClientManager`/`SdkClientRegistry`; "plugin" reserved for the new concept | Kills the three-way overload (and the `PluginManager2` artifact). Pure rename, lands first. Alt name `BackendSelector` noted. |+| D2 | Discovery | Hybrid: static built-in list + `pipelex.plugins` entry points for externals | Avoids self-referential entry points / self-metadata-scan; externalization stays a one-line move. Alt: pure entry points everywhere — over-engineered for the in-tree default. |+| D3 | Plugin contract | One `PipelexPlugin.register(registrar)` method; registrar is the menu | Idiomatic, no god-protocol, no `isinstance`-reflection. Alts: capability protocols / six optional hooks — more magic or more boilerplate. |+| D4 | Registry shapes | Keyed registries (inference, orchestrators) + exclusive hub slots + additive (CLI, teardown) | Matches the three real ways integrations plug in; exclusive slots fail loud on conflict. |+| D5 | Execution-mode space | Closed; enum stays in core; plugins supply orchestrators for existing modes | Enables "install X" errors and typed gates. Open string-keyed modes deferred until needed. |+| D6 | OpenAI | Substrate → core library (always-on); driver → always-on built-in plugin, no special case | Substrate is shared infra (diamond-dep otherwise). Substrate *extraction into a named module* deferred to externalization (no in-tree payoff). |+| D7 | Temporal config | Schema stays typed in core; only implementation moves | Keeps static typing + `make tb`; config seam becomes a near-no-op. Generic third-party config namespace deferred. |+| D8 | SPI | Two published, versioned SPIs sized to measured imports; sanction `runtime_bridge.primitives.*`; widen to cover Mistral's real needs | A designed surface beats an honor rule that's already breached. |+| D9 | Versioning | `PLUGIN_API_VERSION: int`, plugins declare `targets_api`, mismatch fails loud | Make breaks visible (repo policy). Semver ranges deferred. |+| D10 | Packaging | In-tree-but-registerable default; externalize Temporal; flip Mistral to discovery; vendor inference dists deferred | Most value from the seam alone; split only where cadence/weight justifies. |++---++## 14. Test strategy++Each rollout phase lands green; tests are written with the change (red-green), not after. Most of the suite already exists (the integrations work today) — the new tests target the *seam* and the *invariants the seam must not break*. Trivial pass-throughs (a factory that just wraps a client) are not worth bespoke tests; the contract and the invariants are.++### 14.1 Contract conformance (core, protocol-level)++Kept in core so any plugin — built-in or external — can be checked against the contract:++- every `BUILTIN_PLUGINS` entry satisfies the `PipelexPlugin` protocol and declares a compatible `targets_api`;+- a plugin declaring an incompatible `targets_api` fails loud at load (named);+- duplicate `(family, sdk)` / duplicate `execution_mode` / double-claimed hub slot each raise, naming both contributors (§5.3 conflict policy);+- a discovered entry point that fails to import or whose `register` raises is surfaced with context, not skipped (§5.5).++### 14.2 Registry round-trip and DIRECT parity++- each worker family: register → `lookup(family, sdk)` → `make_worker` returns the right `*WorkerAbstract`, for a representative built-in;+- the bridge dispatches by `execution_mode` to a fake registered orchestrator (proves the lookup replaced the `match` with no behavior change);+- `DIRECT` routed through the registry produces byte-identical output to the pre-refactor `_run_direct` for a sample pipe.++### 14.3 Correctness invariants (the silent-bug guards)++- **Import-light boot** (§6.5): booting with the built-ins registered imports no optional SDK — assert `anthropic`, `mistralai`, `google.genai`, `boto3`, … are absent from `sys.modules` after boot. This is the guard against a careless refactor regressing lazy-load.+- **DIRECT router scoping** (§8.1): a `DIRECT` run inside a Temporal-enabled hub keeps nested controller sub-pipes in-process (they resolve the scoped `PipeRouter`, not the Temporal default). Pins the one correctness landmine in the orchestrator extraction.+- **Core defaults unconditional** (§5.3): boot in a stripped environment still resolves `DIRECT` and every core-dep backend (extends the existing `make tb` boot test).++### 14.4 Error parity++A representative missing-extra path (e.g. an Anthropic model with the `anthropic` extra absent) raises the *same* friendly `MissingDependencyError` text — lib + extra + alternative — proving the message survived the move into `make_worker` (preserves the security-perimeter habit of asserting the user-facing message, not just the type).++### 14.5 Temporal: green in-tree before extraction++The full Temporal suite (unit + integration, with the `temporal` marker and `--temporal-server`) must pass with Temporal **wired through the seam but still in-repo** before the package is lifted. Extraction then re-runs the relocated suite against the published base. This is the checkpoint that proves the seam before the packaging move.++## 15. Risks and mitigations++- **Registration timing** — anything used before it's registered breaks. Mitigation: single ordered discovery point after config, before first use; core defaults unconditional; a boot test (`make tb`) that asserts `DIRECT` + every core-dep backend resolve.+- **Silently dropping a core backend** — a misconfigured env must not lose openai/gateway/etc. Mitigation: built-in static list (not entry-point-discovered), registered unconditionally; conformance test asserting presence.+- **Over-fitting the contract to Temporal** — Temporal exercises every hook; tempting to model the protocol on it. Mitigation: design the registrar against *both* Temporal (heaviest) and Mistral (per-call only); Mistral is the standing counter-example.+- **SPI under-sizing** — if the orchestrator SPI omits something Mistral genuinely needs, the boundary stays breached. Mitigation: size it to the *measured* import set (§9.2), not a guess; treat any out-of-SPI import as a tracked design bug.+- **Cross-plugin coupling on extraction** — `pipelex-mistralai-workflows`'s `temporal` extra imports `pipelex.temporal.*`; extracting Temporal to `pipelex-temporal` breaks those imports. Mitigation: in the Temporal-extraction phase, repoint Mistral's temporal extra/imports from `pipelex[temporal]` / `pipelex.temporal.*` to `pipelex-temporal`; this is the known downstream-consumer breakage already tracked.+- **Error-quality regression** — moving the `find_spec` guard into factories could dull today's helpful messages. Mitigation: move the messages verbatim; a test asserts the friendly text (lib + extra + alternative) for a representative missing extra.++---++## 16. Rollout shape (detail belongs to the plan)++A natural ordering, each step provable by a green suite before the next. The implementation plan expands these into phases with checkpoints.++1. **Rename** (D1) — `Plugin`/`PluginManager`/`PluginSdkRegistry` → new names. Pure rename, unblocks clear vocabulary.+2. **The seam** — `PipelexPlugin` protocol, `PluginRegistrar`, the three registry shapes, discovery (static + entry points), version check, fail-loud. No behavior change.+3. **Inference through the seam** (in-tree) — `InferenceBackendRegistry` + `InferenceBackendFactory`; every `match` arm (OpenAI driver included, no special case) becomes a registration + factory; built-in drivers in the static list. Suite green with all backends in-repo but selected via the registry.+4. **Orchestrators through the seam** (in-tree) — `OrchestratorRegistry` + exclusive hub slots; `DIRECT` registered in core; the bridge `match` collapses to a lookup; Temporal's boot/teardown/CLI/modes move behind its plugin (still in-repo); Mistral's hard import becomes entry-point discovery. Publish the orchestrator SPI; resolve Mistral's boundary against it.+5. **Externalize Temporal** — move `pipelex/temporal/` + tests + marker + CLI option to `pipelex-temporal`; entry point; flip downstream pins; repoint Mistral's temporal extra. Keep config schema in core.++Vendor inference dists and the OpenAI-substrate extraction are out of this rollout — they happen if/when a specific externalization is justified.++**Cross-cutting deliverables.** The published SPI reference (module/symbol list with import examples, §9) lands with the phase that publishes each SPI. A plugin-authoring guide and a minimal example plugin — the "hello world" for plugin authors (§5.6) — land once the seam exists (after the inference phase, since a backend plugin is the simplest example) and are kept current as orchestrators land. These are deliverables, not follow-ups.++---++## 17. Open questions for the plan++- **SPI delivery shape** (§9.3): documented module/symbol list only, or also a thin curated import surface? Resolve without violating the no-re-export rule.+- **Registrar config access timing**: confirm the single `register` call point sees a fully-resolved `ConfigRoot` for every plugin's gating needs (Temporal's `is_enabled`, and the `temporal_enabled` boot override in `Pipelex.make`).+- **`InferenceBackendFactory` granularity**: one factory object per `(family, sdk)`, or one per vendor that switches on family internally? (Leaning per-`(family, sdk)` for uniform entries — confirm against the messier arms like huggingface's provider-literal handling.)+- **Where the committed docs land** (§5.6, §16): the plugin-authoring guide + SPI reference are deliverables — the open question is placement (a dedicated user-facing `docs/.../plugins/` section vs `docs/under-the-hood/`), given the authoring guide is user-facing while the seam internals are under-the-hood.diff --git a/wip/plugins/execute-per-request-execution-mode.md b/wip/plugins/execute-per-request-execution-mode.mdnew file mode 100644index 000000000..5f6348be4--- /dev/null+++ b/wip/plugins/execute-per-request-execution-mode.md@@ -0,0 +1,128 @@+# Plan — `/execute` honors per-request `execution_mode` (extend F1)++> **⚠️ SUPERSEDED (2026-06-22) by [`orchestration-mode-and-delivery-split.md`](orchestration-mode-and-delivery-split.md).** This plan shipped a working surface on top of the **flat** `PipelexExecutionMode` enum. The orchestration-mode/delivery split (now DONE across all repos, incl. Phase 4 — the `pipelex-mistralai-workflows` migration) **removed that flatness**: `PipelexExecutionMode` is gone, replaced by an open `orchestration_mode` string token (the backend) + an endpoint-set `DeliveryMode` (the wait-semantics). The `/execute` change this plan made still stands conceptually (per-call orchestration via the `OrchestratorRegistry`), but its wire field is now `orchestration_mode` and the f&f-rejection workaround it relied on (`FIRE_AND_FORGET_NOT_SUPPORTED`) is deleted — delivery is no longer a requestable mode value. Read the split plan for the current surface; this doc is kept for history.++**Status:** **Phase E0 DONE — PR #27 MERGE-READY.** Implemented in `pipelex-api` on `feature/Execute-per-request-mode` (off the validate tip `72c0efc`); **PR [#27](https://github.com/Pipelex/pipelex-api/pull/27) → `feature/Orchestrator-dispatched-validate`**. CI green (Lint/Tests/doc-check across 3.11–3.13); review bots satisfied — **Greptile 5/5 "safe to merge"**, Codex's findings addressed (the OpenAPI `execution_mode`-advertisement gap fixed; the dangling-`$defs`-ref + error-wrapping items deferred as pre-existing/cross-endpoint), no open threads. gstack pre-landing `/review`: **merge-ready, no blockers.** All gates green: `make agent-check`, `make agent-test`/`make check` (openapi-check + pylint 10/10). PR head `191c900`. As-built in §"As-built (Phase E0)"; deferred/flagged items (incl. review findings #5–#9) in `execute-per-request-mode-deferred.md`. **Not merged** (targets a feature branch; user's call). The git-SHA `pipelex` pin (`c60be1f`) reverts to the PyPI `==<ver>` pin once core ships. **Plan 2 of 2** — its sibling `wip/plugins/orchestrator-dispatched-validate.md` is now **DONE** (V0–V3 implemented + committed across all three repos: core PR **#998** vs `refactor/Plugins-4` with CI green + Greptile happy; `pipelex-api` tip `72c0efc` and `pipelex-temporal` tip `459e04d`, both on `feature/Orchestrator-dispatched-validate`, pushed). This plan **extends locked decision F1** of the orchestrator-agnostic-runner effort (tracker: `_plugins/TODOS.md`; parent plan: `_plugins/wip/plugins/orchestrator-agnostic-runner-and-flavors.md`).++> **START HERE (cold start).** Read this whole doc. This is a **single-repo change in `pipelex-api`** — no core (`pipelex`) and no `pipelex-temporal` change. It converts `POST /execute` from the boot-global pipe-run hub slot to the **per-call `OrchestratorRegistry`** (the same seam `/start` already uses), so `/execute` honors a per-request `execution_mode` (policy-gated), exactly like `/start`. The change reuses the **existing** orchestrator seam — the validate work added a *parallel* `BundleValidatorRegistry` and never touched the `OrchestratorRegistry` — so it is logically independent of the validate plan. **But both edit `pipeline.py`**, so branch off the now-pushed validate tip (`72c0efc`, see §"Cross-repo state") rather than the older Phase-C base `a39841e`, to stack cleanly and avoid a `pipeline.py` merge. The execute change is still its own commit (the plan's "don't bundle" intent holds — stacking ≠ bundling). Begin on the branch named in §"Cross-repo state".++> **⚠️ Anchors below were refreshed against the post-validate `pipeline.py` (tip `72c0efc`).** If you instead branch off the clean `a39841e`, the validate work is absent there: `ApiRunner.validate` (the `@override`) still exists at the old line, `validate_verdict` does not, and the line numbers are the pre-validate ones — re-derive them. Prefer the stacked tip.++---++## 1. The ask + locked decisions++**The ask (user):** make `/execute` honor a per-request `execution_mode`, for consistency with `/start`. Today `/start` resolves the mode per request through the `OrchestratorRegistry`, but `/execute` runs through the **boot-global** `get_pipe_run()` hub slot and ignores any requested mode. Unify them.++**Locked decisions:**++- **Convert `/execute` to the per-call `OrchestratorRegistry`** (the `/start` family), so `execution_mode` (not `boot_orchestrator`) is the single source of truth for top-level dispatch backend. `boot_orchestrator` narrows to its real job — the *execution stack* used wherever the pipe actually runs (worker-side, and the in-process scoping inside `DirectOrchestrator`).+- **`/execute` honors a per-request override**, gated by the existing deployment policy (`allow_request_execution_mode_override`), **symmetric with `/start`**. The user chose this explicitly over keeping the asymmetry.+- **Reject fire-and-forget on `/execute`.** `/execute` is synchronous (it returns the full output); `temporal_fire_and_forget` is meaningless for it. Resolve the mode, then refuse a fire-and-forget resolution with a clear 4xx ("`/execute` is synchronous; use `/start` for fire-and-forget"). (Alternative — coerce f&f→blocking — is listed in §"Open sub-decisions"; default is reject.)+- **Keep it a separate change from `/validate`.** `/validate` adds a *new* seam; `/execute` *reuses* the existing one. They share only `resolve_execution_mode` (already shipped). Do not bundle them.++**Why this is the cleaner direction (context for the cold session):** the asymmetry is *incidental*, not principled. Phase C had to move `/start` to the registry (it needs per-request mode, the override policy, fire-and-forget, and a `workflow_id` return). `/execute` was *already* orchestrator-agnostic through the boot slot (`get_pipe_run()` returns whatever's installed; the base imports no `temporalio`), so Phase C had no forcing reason to touch it. This plan finishes that unification. The boot slots do **not** go away — they remain the execution stack on workers and for in-process runs; this only changes how the top-level `/execute` *entry* selects a backend.++---++## 2. Current state++All anchors in `pipelex-api/api/routes/pipelex/pipeline.py` unless noted.++- **`/execute` route** (`370-388`): `async def execute(request)` (`370`) calls `_parse_request(request)` and **discards the extras** — line `376`: `run_request, _extras = await _parse_request(request)` — then calls `runner.execute(...)` **without** any `requested_execution_mode`. So the wire **already parses** `PipelineApiExtras.execution_mode`; `/execute` simply throws it away.+- **`ApiRunner.execute` is NOT overridden.** `ApiRunner(PipelexMTHDSProtocol)` (line `67`) overrides `start` (`@override`, `82`) and defines `validate_verdict` (`185`, **not** an `@override`) but **not** `execute` — so `/execute` falls through to the base `PipelexMTHDSProtocol.execute` in core `_plugins/pipelex/pipeline/runner.py:112`. (Note: the validate work **dropped** the old `ApiRunner.validate` `@override` in favor of `validate_verdict`; `ApiRunner` now inherits the base `validate`. That does not affect this plan — `execute` is still un-overridden and still falls through to the boot slot.)+- **The base `execute`** (core `runner.py:112` ff.) does `pipeline_run_setup(...)` → `effective_pipe_run = self._pipe_run or get_pipe_run()` (`205`) → `pipe_output = await effective_pipe_run.run(pipe_job, delivery_assignment=…)` (`206`) → wraps `pipe_output` into `PipelexRunResultExecute`. `get_pipe_run()` is the **boot-global** hub slot (Temporal claims it only when `boot_orchestrator == "temporal"`; else in-process). No `resolve_execution_mode`, no registry.+- **`/start` (the model to mirror)** — `ApiRunner.start` (`82` ff.): `resolve_execution_mode(requested_execution_mode, config=get_api_config())` (`125`) → `pipeline_run_setup(...)` (rich `PipeJob`) → `orchestrator = get_orchestrator_registry().get_optional(mode=execution_mode)` (`173`) → `MissingOrchestratorError` if absent (`175`) → `run_output = await orchestrator.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)` (`176`) → returns `PipelexRunResultStart(..., workflow_id=run_output.workflow_id)`. The `/start` route passes `requested_execution_mode=extras.execution_mode` (`439`). The `/validate` route (now `validate.py`, post-validate work) is a **second** in-file example of threading `execution_mode` through `resolve_execution_mode` — but it uses the typed `ValidateRequest` model, whereas `/execute` mirrors `/start`'s raw-body `_parse_request` + `PipelineApiExtras.execution_mode` path.+- **`resolve_execution_mode`** — `pipelex-api/api/api_config.py:81-99`: `resolve_execution_mode(requested, *, config) -> PipelexExecutionMode`. Deployment default wins unless the caller supplied a *different* mode **and** `config.allow_request_execution_mode_override` is true; a forbidden override raises **403** (`EXECUTION_MODE_OVERRIDE_FORBIDDEN`). `ApiConfig` (`40-53`) + `api.toml` (base: `execution_mode = "direct"`, `allow_request_execution_mode_override = false`).+- **The orchestrators already return the full output.** `_plugins/pipelex/runtime_bridge/direct_orchestrator.py`: `DirectOrchestrator.run` returns `serialize_completed_output(pipe_output=…, workflow_id=None)`. `/start` only reads `.workflow_id`; `/execute` will read the **output** off the same `PipelexPipeRunOutput`. `PipelexExecutionMode`: `DIRECT`, `TEMPORAL_BLOCKING`, `TEMPORAL_FIRE_AND_FORGET`, `MISTRAL_NATIVE` (`_plugins/pipelex/runtime_bridge/execution_mode.py:4-27`).++For full orientation on the two dispatch seams (per-call registry vs boot-global hub slots), see §3 "Background" of the sibling plan `wip/plugins/orchestrator-dispatched-validate.md`.++---++## 3. The design (locked)++A **single-repo override in `pipelex-api`**, mirroring `ApiRunner.start`. No core or `pipelex-temporal` change — the `DirectOrchestrator` / `TemporalBlockingOrchestrator` / `MistralNativeOrchestrator` in the registry already do everything needed.++- **Override `ApiRunner.execute`** (`@override`, alongside `start` and `validate`) with a `requested_execution_mode: PipelexExecutionMode | None = None` parameter. Body mirrors `start`:+ - `execution_mode = resolve_execution_mode(requested_execution_mode, config=get_api_config())` — **first**, so a forbidden per-request override 403s before any library load / run setup (exactly like `start`).+ - **Reject fire-and-forget:** if `execution_mode` resolves to `TEMPORAL_FIRE_AND_FORGET` (use a `match`/`is_*` on the enum — do not `==`-compare per the StrEnum standard), refuse with a 4xx (`/execute` is synchronous). See §"Open sub-decisions" for the coerce-to-blocking alternative.+ - `pipe_job, resolved_pipeline_run_id, _ = await pipeline_run_setup(...)` — build the rich `PipeJob` as `start` does.+ - `orchestrator = get_orchestrator_registry().get_optional(mode=execution_mode)`; `MissingOrchestratorError(mode=...)` if `None`.+ - `run_output = await orchestrator.run(pipe_job=pipe_job, delivery_assignment=None)` — `/execute` is synchronous, no delivery webhooks.+ - **Map `run_output` (`PipelexPipeRunOutput`) → `PipelexRunResultExecute`** (the protocol's execute result wrapping the full pipe output). This mapping is the one integration detail to verify (the orchestrator returns a *serialized* `PipelexPipeRunOutput` via `serialize_completed_output`, whereas the base `execute` wraps the raw `pipe_output`); confirm `PipelexPipeRunOutput` carries everything `PipelexRunResultExecute` needs and write the conversion. **This is the "clean vs more-complex-than-expected" hinge** — check it early.+- **Route** (`execute`, `340-373`): stop discarding extras. Parse like `/start` does (`run_request, extras = await _parse_request(request)`), then pass `requested_execution_mode=extras.execution_mode` into `runner.execute(...)`. No new wire field — `/execute` simply starts honoring the `execution_mode` extra it already parses.+- **Base `execute` stays untouched** in core (`runner.py:112`) — the local/CLI runtime legitimately uses the boot slot (it is not mode-dispatched). Only `ApiRunner` overrides, mirroring how it overrides `start`/`validate`.++**Behavior shift to call out (changelog + docs):** `/execute`'s backend is now selected by `execution_mode` (resolved from `api.toml` + optional policy-gated per-request override), **not** by `boot_orchestrator`. For a correctly-configured deployment these already agree (a Temporal deployment that runs `/start` distributed sets `execution_mode = "temporal_blocking"`), so the real-world delta is ~nil — but it removes the two-knobs-can-silently-disagree footgun and makes `execution_mode` authoritative. The base `api.toml` keeps `allow_request_execution_mode_override = false`, so per-request override only takes effect where a deployment opts in.++---++## 4. Phase + checkpoint++Single contained phase (one repo, one commit).++### Phase E0 — `pipelex-api` (branch off the validate tip `72c0efc` on `feature/Orchestrator-dispatched-validate`)++- Add the `ApiRunner.execute` override + fire-and-forget rejection + the `PipelexPipeRunOutput → PipelexRunResultExecute` mapping; rewire the `/execute` route to thread `extras.execution_mode`.+- Tests (see §"Testing").+- Docs: changelog entry (`/execute` now honors `execution_mode`, rejects fire-and-forget); document the dual-knob model (`execution_mode` authoritative for top-level dispatch; `boot_orchestrator` = execution stack) where `/execute`/`execution_mode` is described — likely `pipelex-api/docs/` and, if a section exists, the relevant `docs/specs/` doc (see §"Spec/conformance").+- Gates: `make agent-check`, `make agent-test`.++> **Checkpoint E-A** (the gate for `/execute`): `/execute` resolves and dispatches by `execution_mode`, honors the policy-gated override symmetrically with `/start`, rejects fire-and-forget, all gates green, docs + changelog updated. Clean-context `/code-review` on the `pipelex-api` diff. Capture an as-built (final signature, the output-mapping decision, test evidence).++---++## As-built (Phase E0)++**Repo / branch:** `pipelex-api` @ `feature/Execute-per-request-mode` (off validate tip `72c0efc`). Single repo; no core / `pipelex-temporal` change.++**Final shape (`api/routes/pipelex/pipeline.py`):**++- `ApiRunner.execute` is now an `@override` taking `requested_execution_mode: PipelexExecutionMode | None = None` (trailing optional param, LSP-compatible with the base). Body: resolve `execution_mode` FIRST (403 on forbidden override), refuse fire-and-forget with a 400 (`is_fire_and_forget`, not `==`), look up the orchestrator (`MissingOrchestratorError` if absent), then **inject it as `self._pipe_run` and delegate to `super().execute(...)`** — so the base owns the entire run lifecycle (library setup/teardown, tracer close, pipeline-manager cleanup, telemetry, error mapping). The `ApiRunner` is per-request, so mutating `_pipe_run` is request-scoped.+- `_OrchestratorPipeRun(PipeRunProtocol)` — the injected adapter: its `run(pipe_job, *, delivery_assignment)` calls `orchestrator.run(...)` and rehydrates the result.+- `_pipe_output_from_run_output(run_output)` — **the output-mapping decision (the §3 hinge):** the orchestrator returns the JSON-safe `PipelexPipeRunOutput`; this rebuilds the rich `PipeOutput` via `hydrate_working_memory(output_dict)` (same routine the Temporal workers use; runs while the run library is still open) + `PipeOutput.model_validate({...}, strict=False)`. `strict=False` is **required** to reverse the orchestrator's `model_dump(mode="json")` of `graph_spec` (its `created_at: datetime` is `strict=True`; str→datetime is rejected under strict). Confirmed empirically. The base `execute` then wraps this into `PipelexRunResultExecute` exactly as before, so `created_at`/`finished_at`/`state`/telemetry are identical to the old path.+- Route `execute()` now threads `requested_execution_mode=extras.execution_mode` (stopped discarding the parsed extras).+- New `ErrorType.FIRE_AND_FORGET_NOT_SUPPORTED` (400, via `raise_bad_request`).++**Why inject-and-delegate (not a full re-implementation):** the base `execute`'s teardown is correct and battle-tested; duplicating it risks drift, and the orchestrator seam differs from the boot slot only in (a) which backend runs and (b) the serialized vs rich output. The adapter bridges exactly those two; everything else is inherited.++**Test evidence:** `tests/unit/test_execute_dispatch.py` (new, mirrors `test_validate_dispatch.py`): direct dispatch returns the rehydrated full output (real `serialize_completed_output` → `hydrate_working_memory` round-trip), per-request override honored when policy allows, forbidden override → 403, fire-and-forget → 400, missing orchestrator → `MissingOrchestratorError`. Existing `test_pipeline_routes.py` / `test_protocol_conformance.py` unaffected (they mock the runner). Full `make agent-test` green.++**Deferred / flagged:** see `execute-per-request-mode-deferred.md` — DIRECT round-trip cost (accepted), `strict=False` rationale, a pre-existing `/start` DIRECT-path resource leak (flagged, not fixed — out of scope), and the absorbed pre-existing `/validate` OpenAPI + config-doc drift.++---++## 5. Cross-repo state, pins, gates++- **Branch:** cut a new branch (e.g. `feature/Execute-per-request-mode`) in `pipelex-api` off `feature/Orchestrator-dispatched-validate` @ `72c0efc` (the now-pushed validate tip), **not** off the clean Phase-C base `a39841e`. Rationale: both changes edit `pipeline.py` (the validate work dropped the `validate` override + added `validate_verdict` + adjusted imports; this adds an `execute` override), so stacking avoids a `pipeline.py` merge. The execute change reuses the existing `OrchestratorRegistry` and depends on **none** of the validate seam's logic, so it stays an independent commit (own PR). This is the only repo touched. *(If you deliberately want full independence from validate, branch off `a39841e` instead and accept a small `pipeline.py` merge when both land — and re-derive the §2 anchors, which are written against the post-validate tip.)*+- **Pins:** `pipelex-api/pyproject.toml` pins `pipelex = { path = "../_plugins", editable = true }`, so the core orchestrators/registry are live without a republish. No `pipelex-temporal` dependency is added (the `TemporalBlockingOrchestrator` is exercised by its own repo's tests; the API change is testable with `DirectOrchestrator` + a stub).+- **Gates:** `make agent-check`, `make agent-test` (in `pipelex-api`). If a `docs/specs/` heading is touched, run `make check-spec-links` (in `conformance/`).++---++## 6. Spec / conformance++- **No spec pins `/execute`'s backend-selection today.** The `/validate` verdict spec (`docs/specs/pipelex-mthds-protocol.md`) doesn't cover `/execute`; `/execute` dispatch lives in `pipelex-platform-api.md` / `pipelex-hosted-config.md`, where the HTTP arm is Phase-3-deferred and per-request override is not yet detailed. So this is **low conformance risk** and adds **no new wire field** (the `execution_mode` extra already exists; `/execute` just stops ignoring it).+- **Add documentation** of the new behavior: `/execute` honors a policy-gated per-request `execution_mode` and rejects fire-and-forget; `execution_mode` is authoritative for top-level dispatch while `boot_orchestrator` selects the execution stack. Put it where `/execute` / `execution_mode` is described (a `pipelex-api/docs/` page, and the hosted-config spec if you extend it). If you add or edit a spec heading, wire a `> Verified by:` link to a test and a matching `pytest.mark.spec(...)`, then `make check-spec-links`.+- **Conformance:** there is **no** `/execute` conformance module yet (deferred with the HTTP arm). Don't invent the deferred HTTP conformance arm here; cover the behavior with `pipelex-api`'s own tests (§"Testing"). If you later add a conformance surface for `/execute`'s mode-selection, link it bidirectionally.++---++## 7. Testing++In `pipelex-api` (no Temporal needed):++- **Direct dispatch:** `/execute` with `execution_mode` unset (or `direct`) runs in-process via `DirectOrchestrator` and returns the full output — equivalent result to today's boot-slot path on the direct base.+- **Per-request override, policy on:** with `allow_request_execution_mode_override = true`, a requested mode is honored; with it `false`, a *different* requested mode 403s (`EXECUTION_MODE_OVERRIDE_FORBIDDEN`) — reuse/mirror the existing `/start` override-policy tests.+- **Fire-and-forget rejection:** a resolved `temporal_fire_and_forget` on `/execute` returns the chosen 4xx, not a hang or a silent coerce (unless the coerce alternative is chosen — then assert blocking semantics).+- **Missing orchestrator:** a mode with no registered orchestrator raises `MissingOrchestratorError` with the install hint.+- **Output mapping:** assert the `PipelexPipeRunOutput → PipelexRunResultExecute` mapping preserves the full output (the integration hinge from §3). Use `DirectOrchestrator` (or a stub orchestrator returning a known `PipelexPipeRunOutput`).++---++## 8. Open sub-decisions for the implementing session++- **Fire-and-forget on `/execute`: reject (default) vs coerce-to-blocking.** Reject is more honest (you asked for something incompatible with a synchronous endpoint). Coerce-to-blocking is more lenient (execute always waits anyway, so f&f could just mean `temporal_blocking` here). Plan defaults to **reject** with a clear 4xx; flip only with a reason.+- **Output mapping shape.** Confirm `PipelexPipeRunOutput` (from `serialize_completed_output`) carries everything `PipelexRunResultExecute` needs. If it doesn't, decide whether to enrich the orchestrator output or to keep a thin execute-specific adapter — and note it in the as-built. This determines whether the change stays "clean" or grows.+- **Deployment-config coherence.** Note for Phase D (hosted Temporal flavor, in the parent plan): the hosted runner's `api.toml execution_mode` must be set coherently (it already is for `/start`); after this change it also governs `/execute`. `boot_orchestrator` remains for the execution stack. No code change here — just a deployment note so the two knobs are set with intent. **Related runtime constraint** (surfaced by the validate work, see `validate-dispatch-requires-boot-active.md`): the shared `WorkflowExecutor` hard-requires `boot_orchestrator == "temporal"` (`is_temporal_boot_active()`) to open its client, so a resolved `temporal_blocking` `/execute` on a runner **not** booted under Temporal would fail the same way a temporal-mode `/start` or `/validate` does. That is pre-existing and out of scope here (the agnostic base tests use `DirectOrchestrator`), but it reinforces why `execution_mode = temporal_*` and `boot_orchestrator = "temporal"` must be set together on a Temporal flavor.diff --git a/wip/plugins/execute-per-request-mode-deferred.md b/wip/plugins/execute-per-request-mode-deferred.mdnew file mode 100644index 000000000..ce974ed2c--- /dev/null+++ b/wip/plugins/execute-per-request-mode-deferred.md@@ -0,0 +1,68 @@+# Deferred / flagged items — `/execute` honors per-request `execution_mode`++Companion to `execute-per-request-execution-mode.md` (the plan). Captures findings surfaced while implementing **Phase E0** that were deliberately **not** acted on, plus pre-existing drift that the implementation absorbed. None block the feature.++## 1. DIRECT dispatch does a serialize→rehydrate round-trip (accepted tradeoff, not a bug)++`/execute` now dispatches every mode — including `direct` — through the `OrchestratorRegistry`. The orchestrator answers with the JSON-safe `PipelexPipeRunOutput` (the same shape that crosses the Temporal worker boundary, via `serialize_completed_output`), so the synchronous `/execute` path rehydrates it back into a rich `PipeOutput` (`_pipe_output_from_run_output` in `api/routes/pipelex/pipeline.py`). For `direct` this means the working memory is `dump_for_temporal()`'d and then `hydrate_working_memory()`'d **in-process** — one redundant re-serialization the old boot-slot path didn't pay.++**Why accepted:** it keeps `/execute` on the *same* per-call dispatch seam as `/start` and `/validate` (no per-mode branch), which is the whole point of the locked decision. The cost is bounded (one extra round-trip of a working memory that `/execute` already serializes once for the HTTP response).++**If it ever matters:** the clean fix is at the *contract* layer, not here — e.g. let `OrchestratorProtocol.run` optionally hand back the rich `PipeOutput` for the in-process arm so the DIRECT path skips the round-trip. That's a core (`pipelex`) change to a shared SPI, out of scope for this single-repo plan. Don't special-case DIRECT inside `pipelex-api` (that reintroduces the per-mode branch the seam removed).++## 2. `graph_spec` / `tokens_usages` reconstruction needs `strict=False` (correct, worth knowing)++`serialize_completed_output` dumps `graph_spec` / `tokens_usages` with `model_dump(mode="json")`, and those models are `strict=True` (e.g. `GraphSpec.created_at: datetime`). A strict re-validation of the JSON dump fails (str→datetime is rejected in strict mode — confirmed empirically). `_pipe_output_from_run_output` therefore validates with `strict=False`, which is the correct tool for reversing our *own* trusted JSON dump (a round-trip, not untrusted ingest). Nothing else in the tree rehydrates these dump fields today, so there was no prior precedent to copy — flagged here so a future reader doesn't "tighten" it back to strict.++## 3. Pre-existing `/start` DIRECT-path resource leak (flagged, NOT fixed — out of scope)++While mirroring `start`, noticed that `ApiRunner.start` calls `pipeline_run_setup(...)` then the orchestrator, with **no** teardown on the success path. `pipeline_run_setup` only tears the run library / pipeline-manager entry down on its *error* path; `PipeRun.run` closes the tracer but not the library or the pipeline-manager registration. The base `PipelexMTHDSProtocol.execute` does that teardown in its `finally`. So on the **DIRECT** `/start` success path (reachable on the agnostic base, where `start` runs in-process and blocks), the run library and its pipeline-manager entry appear to leak per request.++`/execute` does **not** have this problem: this change delegates to the base `execute`, inheriting its full teardown. The finding is purely about `/start`.++**Not fixed here** because it touches `/start` (a different surface) and the plan is scoped to `/execute`. Worth a small follow-up: give `ApiRunner.start` the same lifecycle guarantees (or factor the base `execute`'s setup+teardown into a shared context manager both `execute` and `start` use). Verify first that it is a genuine leak and not cleaned up elsewhere before changing `/start`.++## 4. Absorbed pre-existing drift from the `/validate` work (informational)++The committed `docs/openapi/pipelex-api.openapi.yaml` and `docs/configuration.md` at the validate tip (`72c0efc`) were **stale** w.r.t. the `/validate`-by-`execution_mode` work: the OpenAPI artifact lacked the `PipelexExecutionMode` component + the validate request's `execution_mode` field, and `configuration.md` still said `/validate` "always runs in-process and ignores this setting." Both are regenerated/corrected in this change (the OpenAPI artifact is generated, so it must match the current app exactly; the config doc was rewritten holistically for `/execute` + `/start` + `/validate`). Mentioned so the diff's validate-flavored hunks aren't mistaken for execute-scope creep — they are pre-existing drift this change necessarily resolves.++## 5. Reverse helper location — altitude (independent review, deferred)++`_pipe_output_from_run_output` is the exact inverse of `serialize_completed_output` (which lives in `pipelex/runtime_bridge/serialization.py`), but the reverse currently lives as a private function in the `pipelex-api` route module. If `PipelexPipeRunOutput` gains/renames a field, forward (pipelex) and reverse (pipelex-api) could drift independently. **Deferred:** single consumer; moving it crosses a repo boundary; and the reverse must run *inside* the still-open run-library scope (a precondition naturally satisfied at this call site). Confirmed no existing reverse helper is being reinvented (`rehydrate_pipe_output_with_crate` operates on a `PipeOutput.working_memory_raw`, not a `PipelexPipeRunOutput`; the new helper correctly reuses `hydrate_working_memory`). Promote to a `deserialize_completed_output` in `serialization.py` only if `pipelex-temporal`'s blocking path ever needs the same reverse.++## 6. OpenAPI request schema doesn't advertise the `execution_mode` body extra++`/execute`, `/start`, and `/validate` all honor a per-request `execution_mode` override, but the override is parsed from the raw kajson body as a `PipelineApiExtras` extra, so the *generated* OpenAPI request schema only lists it where the route uses a typed body model.++**`/execute` half — RESOLVED (PR #27 review fix).** `/execute`'s `openapi_extra` now documents a new `PipelexApiExecuteRequest(RunRequest)` schema (mirroring the existing `PipelexApiStartRequest` pattern) that declares `execution_mode`, reusing the shared `_EXECUTION_MODE_DESCRIPTION` already on `PipelineApiExtras.execution_mode` (factored to a module constant so the two can't drift). The committed OpenAPI artifact was regenerated; the drift check (`scripts/export_openapi.py --check`) is green. Greptile (P1) / Codex (P2) flagged exactly this gap on `/execute` — the route reads `extras.execution_mode` and honors it, but the published schema (`RunRequest.model_json_schema()`) omitted the field.++**Dangling `#/$defs/` ref for the advertised enum — DEFERRED (pre-existing artifact-generation pattern).** A follow-up Codex pass (P2) noted that `/execute`'s newly-advertised `execution_mode` references `#/$defs/PipelexExecutionMode`, which a root-based OpenAPI resolver can't dereference (the document has no top-level `$defs`; the enum also exists at `#/components/schemas/PipelexExecutionMode`). This is a **pre-existing pattern**, not introduced here: both `/execute` and `/start` already emit dangling `#/$defs/StuffContentAbstract` and `#/$defs/WorkingMemoryAbstract_Any_` refs, because both parse the body through the raw `Request` and supply their schema via `openapi_extra: <Model>.model_json_schema()` — FastAPI hoists nested defs into `components/schemas` only for *typed-body* routes (like `/validate`), not for a manually-supplied schema dict. My `execution_mode` ref is consistent with the existing dangling refs on the same endpoint. Fixing only `execution_mode` would leave it inconsistent with its siblings and still not make the endpoint's schema resolvable; the clean fix is a cross-endpoint OpenAPI-generation improvement (hoist each `openapi_extra` schema's `$defs` into `components/schemas` and rewrite the refs, or move `/execute` + `/start` to typed bodies) — a separate change spanning both endpoints, out of scope for this PR. Tracked here.++**`/start` half — STILL DEFERRED (scope).** `/start`'s `PipelexApiStartRequest` schema *also* omits `execution_mode` (it extends `RunRequest` and only redeclares `callback_urls`; the field lives on `PipelineApiExtras`). That gap is **pre-existing** — `/start`'s `execution_mode` consumption was introduced by the F1 work, not by PR #27, whose diff touches only `/execute`. Not fixed here to honor scope discipline. The clean fix is the same one-liner pattern: have `PipelexApiStartRequest` redeclare `execution_mode = Field(default=None, description=_EXECUTION_MODE_DESCRIPTION)`, then regenerate the artifact. Note: the published `make openapi-export` target does not exist in `pipelex-api`'s Makefile; regenerate by running `.venv/bin/python scripts/export_openapi.py docs/openapi/pipelex-api.openapi.yaml` (write) and `--check` to verify. Never hand-edit the yaml.++## 7. `is_completed` invariant the synchronous path relies on (independent review, rejected-as-fix)++`_pipe_output_from_run_output` unconditionally builds a completed `PipeOutput` without checking `run_output.is_completed`. **Rejected as a fix** (not a bug): `/execute` refuses fire-and-forget upfront with a 400, and every non-f&f orchestrator today (`DirectOrchestrator`, `TemporalBlockingOrchestrator`) returns `is_completed=True`, so a `False` here is unreachable — guarding it is a defensive check for an impossible-today scenario. Recorded only so a future orchestrator author knows the invariant the synchronous path assumes.++## 8. Worker-dispatched (`temporal_blocking`) rehydration is unexercised here (pre-landing review, INVESTIGATE — out of scope)++The adversarial pre-landing review traced the `temporal_blocking` arm: a worker runs the pipe and returns a `PipelexPipeRunOutput` whose `output_dict` carries `__pipelex_class__` markers for any dynamic concept classes registered **on the worker**. `_pipe_output_from_run_output` then calls `hydrate_working_memory` against the **API-side** run library's class registry. This resolves correctly *because* `pipeline_run_setup` already loaded the same bundle API-side (the rich `PipeJob` is built locally before dispatch, registering the same dynamic concept classes) — the same property `/start` and `/validate` already rely on for their worker-dispatched arms. So `/execute` introduces **no new** boundary risk: it reuses the shared `PipelexPipeRunOutput` contract and the same `hydrate_working_memory` round-trip.++**INVESTIGATE, not fixed:** this path cannot be exercised on the orchestrator-agnostic base (no `pipelex-temporal` installed; `test_execute_dispatch.py` covers DIRECT only, with a stub orchestrator that round-trips through the real `serialize_completed_output` → `hydrate_working_memory`). End-to-end coverage of the `temporal_blocking` `/execute` rehydration belongs in `pipelex-temporal`'s own test suite (where the blocking orchestrator and a real worker exist), alongside the analogous `/start` / `/validate` worker-path tests. Out of scope for this single-repo `pipelex-api` PR. Verify there when the hosted-Temporal flavor lands; reinforced by the runtime constraint in `validate-dispatch-requires-boot-active.md` (a `temporal_*` `/execute` on a runner not booted under Temporal fails the same way a temporal-mode `/start`/`/validate` does).++## 9. Failing-pipe `/execute` now yields a double-wrapped error message + unwound `pipe_stack` (pre-landing review, DEFERRED — seam tradeoff, cross-endpoint)++The adversarial pre-landing review traced how a *failing* pipe surfaces differently after the dispatch change. Verified against source:++- **Old `/execute`** ran the pipe through the boot slot (`get_pipe_run()` → `PipeRun.run`), which observe-and-reraises the original `PipeRouterError` (`runner.py` core, `pipe_run.py: raise execution_error`). The base `execute`'s `except PipeRouterError` arm then built `PipelineExecutionError(message=exc.message, pipe_stack=exc.pipe_stack)` — the raw router message and the **failure-site** pipe-stack snapshot.+- **New `/execute`** runs through `DirectOrchestrator.run`, which catches `PIPE_DISPATCH_ERRORS` and re-raises `PipelexBridgeDispatchError("Pipe execution failed in DIRECT mode for pipe 'X': <original>")` (`direct_orchestrator.py`). That no longer matches `except PipeRouterError`, so it falls to the base's `except PipelexError` arm → `PipelineExecutionError(message=exc.message, pipe_stack=pipe_job.pipe_run_params.pipe_stack)`. Net: the operator-facing **message gains the "in DIRECT mode" wrapper** and the **`pipe_stack` is the unwound live stack**, not the failure-site snapshot.++**Not a wire-status / classification regression.** `error_domain` / `http_status` / `error_category` survive because `PipelexBridgeDispatchError.to_error_report()` enriches from the full `__cause__` chain, and `pipe_stack` is not an `ErrorReport` wire field. So a machine consumer sees the same `is_valid`/status/error-domain; only the human-readable `message` wording and log-stack fidelity change.++**DEFERRED, not fixed** — it is a property of the **orchestrator seam, not of `/execute`**: the seam deliberately funnels every mode's pipe failure through the single `PipelexBridgeDispatchError` type so a host catches one error regardless of backend. `/start` (also routed through `DirectOrchestrator`) and the dispatched `/validate` already exhibit the identical wrapping (the agent's symmetric finding: `/start` and `/execute` now render different `message`/`title` for the same underlying pipe failure than the legacy boot-slot path did). Harmonizing the wording — or restoring the failure-site `pipe_stack` for the DIRECT arm — is a change to the **shared seam spanning all three surfaces** (e.g. have `DirectOrchestrator` re-raise the original `PipeRouterError` so the base's precise arm still fires, or teach the base `except` to unwrap a `PipelexBridgeDispatchError` cause), not a `pipelex-api`-local `/execute` edit. Out of scope for this PR; no shipped Temporal flavor means no prod error-shape contract is at risk. Worth a small follow-up if product wants byte-identical error bodies across `/execute` / `/start` and parity with the legacy message.++### Minor (independent review, no action)++- `_validate_extras` reports a malformed `execution_mode` value under `error_type=INVALID_CALLBACK_URLS` (still a correct 422 with the real pydantic detail in the message) — pre-existing label imprecision surfaced only because `execution_mode` joined that model; cosmetic.+- The base `execute`'s `if extra:` rejection is unreachable via the API `execute` override (the override omits `extra`; the route never passes it) — dead-but-harmless, identical to how `start()` already works.+- `ErrorType.FIRE_AND_FORGET_NOT_SUPPORTED` sits just above the `# Request validation` section header in `api/error_types.py` rather than under it (it is a 400 request-shape refusal). Purely cosmetic enum ordering; not worth a diff.diff --git a/wip/plugins/implementation-plan.md b/wip/plugins/implementation-plan.mdnew file mode 100644index 000000000..08e022ba7--- /dev/null+++ b/wip/plugins/implementation-plan.md@@ -0,0 +1,295 @@+# Pipelex plugin system — implementation plan++**Status:** plan (proposed, not started) — reviewed via `/plan-eng-review` + codex outside voice; decisions D1–D6 locked (see below).+**Built from:** [`design.md`](design.md) — the single decided design. This plan expands design §16's rollout into executable phases with grounded `file:line` references, per-phase red-green tests, and checkpoints, and resolves the §17 open questions. Read `design.md` first; this is the *how*. Where this plan diverges from the design (deferred slot-claim thunks; a fifth seam; config relocation), the divergence is a review finding folded in, called out inline.++## 0. Orientation++**Why now (not speculative generality).** Both families are already ~80–90% decoupled; this inverts the *last* coupling. The justification is present pain, not hypothetical third-party authors: `pipelex-mistralai-workflows` is *already* an external repo whose native tier *already* breaches its declared import boundary (it reaches into `pipelex.hub`, `pipe_run`, `core`, `graph`, `tracing`), held together by a pinned-rev coupling — the orchestrator SPI is the real fix. And Temporal's heavy SDK + independent release cadence + the fact that downstream (`pipelex-worker`, `pipelex-api-hosted`) *already* pin it separately mean externalization pays for itself. The third-party-author story is a bonus the seam enables for free.++**The seams, located.** The design names four inference seams; the review found a fifth (`model_lists.py`). Core still names integrations at each:++| Seam | File | What it does today |+|---|---|---|+| Inference dispatch (×4) | `cogt/{llm,img_gen,extract,search}/*_worker_factory.py` | `match plugin.sdk:` → per-arm lazy import + `find_spec` guard + SDK-client cache + build worker |+| **Model listing (the 5th seam)** | `cogt/model_backends/model_lists.py:58` | `match sdk:` → hardcoded imports of `pipelex.plugins.{openai,anthropic,…}.*_list` (the `list-models` CLI path) — **missed by the design; codex finding** |+| Orchestrator dispatch | `runtime_bridge/bridge.py:147-158` | `match execution_mode:` → hard-imports Mistral (`:414`), lazy-imports Temporal; per-mode install messages at `:399/:417` |+| Boot/teardown hub swap | `pipelex.py:377-479` | four `if get_config().temporal.is_enabled:` blocks + inlined teardown; explicit `content_generator`/`pipe_router` params override at `:376/:450` |+| Config | `system/configuration/configs.py:14,250` | the one hard import of `pipelex.temporal.config_temporal` |+| CLI | `cli/_cli.py:15,20,234,238` | two hard imports registering `worker` + `setup-temporal-namespace`; `PipelexCLI.list_commands` hardcodes order |+| Naming overload | `plugins/{plugin,plugin_manager,plugin_sdk_registry}.py`, `hub.py:289-293` | `Plugin` value object, `PluginManager`/`PluginSdkRegistry` cache, the `PluginManager2 is not initialized` artifact |++| Phase | Scope | PR |+|---|---|---|+| 0 — Rename | `pipelex` only | own PR |+| 1 — Seam + LLM family | `pipelex` only | own PR (merged per D2) |+| 2 — Remaining inference families | `pipelex` only | own PR(s) per family |+| 3 — Orchestrators through the seam | `pipelex` only | own PR |+| 4 — Invert model listing (5th seam) | `pipelex` only | own PR (D6) |+| 5 — Externalize Temporal | cross-repo | separate effort, gated |++## Decisions locked in this review++| # | Decision | Choice |+|---|---|---|+| D1 | Inference backend factory shape | **Typed callable** (`MakeWorkerFn` TypeAlias), not a single-method `Protocol` object — kills ~30 one-method classes |+| D2 | Phase 1 PR boundary | **Merge** the seam machinery with the LLM family so the contract meets a real backend before it locks |+| D3 | Plugin CLI command harvesting | **Run the pure `build_registrar` at CLI-build** (after loading config); same function runs again at boot |+| D4 | `SdkClientManager` wrapper | **Keep as pure rename**; collapse into the registry deferred to a P3 follow-up commit |+| D5 | Hub slot claims | **Deferred thunks** — `claim_*(factory: Callable[[], Impl])`, invoked only at the setup apply-point, so `register` stays import-light (codex finding) |+| D6 | The 5th seam (`model_lists.py`) | **Invert it too**, as a follow-on phase, with a model-listing capability on the inference contract (codex finding) |+| D7 | Plugin control surface | Discovery is the source of truth for *presence*; add `pipelex plugins list` (observability) + a narrow `plugins.disabled` denylist escape hatch; **no** redundant allowlist (model config already gates inference) |++## Plugin enablement model++No central "plugins on/off" list exists, by design. Enablement is two separate things:++- **Presence (is the plugin discovered?) — not config.** A plugin is present iff it is in the static `BUILTIN_PLUGINS` list (core) or is an installed dist declaring the `pipelex.plugins` entry point. Installing enables presence; uninstalling removes it. Zero config. `build_registrar` walks both sources at boot and calls each plugin's `register(registrar)` — that call *is* the plug.+- **Activation (is its contribution used?) — decomposed by registry shape, never a generic flag:**+ - *Inference backends* activate by **data**: a model in the backend config names the `sdk`. The plugin registers its `(family, sdk)` entry unconditionally and import-light; it runs only when a model selects it. The model/backend config (`get_models_manager()`) is the de-facto central enable surface for inference — a per-plugin toggle would duplicate it and drift.+ - *Orchestrators, per-call* select by `execution_mode` per run.+ - *Orchestrators, boot-global* are claimed only when the plugin's own config slice says so. **`temporal.is_enabled` is not a generic "Temporal plugin on" flag** — it means "boot *this process* as a Temporal-default runtime" (swap the hub content-generator/router/run to the Temporal ones). A worker process sets it; a runner does not. The name misleads; treat it as a per-process runtime-mode gate, not plugin enablement. Renaming it is out of scope (touches the live config) but flagged.++**Control surface (D7) — discovery stays the source of truth for presence; on top of it:**++- `pipelex plugins list` — a core (not plugin-contributed) command: prints every discovered plugin (built-in vs external), what each registered (backends / modes / slots / CLI), and whether it's denylisted. The answer to "what's plugged in?"+- `plugins.disabled: list[str]` — a denylist escape hatch in a core-owned `plugins` config section. A discovered plugin whose `name` is listed is skipped in `build_registrar`, **with a log line (never silent)**, without uninstalling it — for testing and quarantining a misbehaving external. A flat list of *names*, **not** the deferred *generic per-plugin typed-config namespace* (design §10), which stays out of scope.+- **The denylist cannot disable a core-unconditional plugin** (the DIRECT orchestrator, or a core-dependency backend: openai/gateway/portkey/pypdfium2/azure_rest). Listing one is a startup error — you cannot boot without DIRECT — preserving design §5.3's "core defaults unconditional" invariant.++---++## Phase 0 — Rename (free the word "plugin")++**Goal.** Kill the three-way "plugin" overload so the new vocabulary is unambiguous. Pure rename, no behavior change.++**Renames** (class → file):++- `Plugin` (`plugins/plugin.py`) → `ModelHandle` (`plugins/model_handle.py`). A backend *selector* (`sdk` + `backend` + `variant` + `sdk_handle`), not a plugin. Keep `make_for_inference_model`.+- `PluginManager` (`plugins/plugin_manager.py`) → `SdkClientManager` (`plugins/sdk_client_manager.py`).+- `PluginSdkRegistry` (`plugins/plugin_sdk_registry.py`) → `SdkClientRegistry` (`plugins/sdk_client_registry.py`). Rename `get_sdk_instance`/`set_sdk_instance` → `get`/`set` (the DRY `get_or_create` lands in Phase 1, not here).+- `PluginFactoryAbstract` (`plugins/plugin_factory_abstract.py`, the `make_extras` helper — *unrelated* to the new system) → `BackendExtrasFactory` (`plugins/backend_extras_factory.py`). Name-hygiene; it collides with the new contract vocabulary.++**Blast radius is bigger than "the four factories" (codex C11).** `Plugin` / `plugin_sdk_registry` is imported across the in-tree adapter modules — `openai`, `bedrock`, `anthropic`, `gateway`, `portkey` list+factory modules, plus `search_worker_factory.py:4` (which uses the *correct* `plugins.plugin` path while `llm/img_gen/extract` import from the *wrong* `plugins.plugin_sdk_registry` path). Grep `import Plugin\b` and `plugin_sdk_registry` across `pipelex/` before starting; the rename touches more than the dispatch sites. The wrong-path imports get fixed for free here.++- `hub.py`: `get_plugin_manager`/`set_plugin_manager` → `get_sdk_client_manager`/`set_sdk_client_manager`; **fix the `"PluginManager2 is not initialized"` message** (`hub.py:291`) to `"SdkClientManager is not initialized"`.+- `pipelex.py`: setup/teardown call sites.++**D4 — keep this a pure rename.** The `SdkClientManager` post-rename is still a thin wrapper holding one `SdkClientRegistry`. *Collapsing* it (hub holds the registry directly) is a separate structural change captured as a **P3 TODO** (own commit), not bundled here — rename and restructure don't mix.++**Tests.** No new tests; existing suite + `make tb` (boot/config) stay green.++**Checkpoint 0.** Green suite, clean vocabulary, `PluginManager2` artifact gone, wrong-path `Plugin` imports fixed. Standalone PR.++---++## Phase 1 — Seam machinery + LLM family (merged, per D2)++**Goal.** Build the discovery + contract + registry machinery *and* migrate the LLM family through it in one PR, so the contract is proven against a real, messy family before it locks. No behavior change.++**New modules** (all import-light — must not import any backend SDK or `temporalio`):++- `plugins/contract.py` — `PipelexPlugin` (`@runtime_checkable Protocol`: `name: str`, `targets_api: int`, `register(self, registrar) -> None`) and `PLUGIN_API_VERSION: int = 1`. **Contract invariant — `register` is side-effect-free:** it may *only* call registrar methods (no hub touch, no I/O, no client construction). This is what makes `build_registrar` safely repeatable (D3 runs it at CLI-build *and* boot).+- `plugins/registrar.py` — `PluginRegistrar`, the accumulator. Holds the inference registry, orchestrator registry, **deferred** hub-slot-claim thunks, CLI-command list, teardown-callback list, and a read-only `config: ConfigRoot`. Menu methods:+ - `add_inference_backend(*, family, sdk, make_worker: MakeWorkerFn)` — D1: a callable, not a factory object.+ - `add_orchestrator(*, mode, orchestrator)`.+ - `claim_content_generator | pipe_router | pipe_run | task_manager(factory: Callable[[], Impl])` — **D5: a thunk, not an instance.** `register` never constructs the impl, so it never imports `temporalio`.+ - `add_cli_command(*, name, help, command)`, `add_teardown(callback)`.+- `plugins/inference_backend_registry.py` — `InferenceFamily` (`StrEnum`: `LLM`/`IMG_GEN`/`EXTRACT`/`SEARCH`), `MakeWorkerFn` (TypeAlias for the callable signature), `InferenceBackendRegistry` keyed by `(family, sdk)`.+- `plugins/orchestrator_registry.py` — `OrchestratorProtocol` (`async def run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput`), `OrchestratorRegistry` keyed by `PipelexExecutionMode`.+- `plugins/discovery.py` — `build_registrar(*, config) -> PluginRegistrar`: a **pure function** iterating `BUILTIN_PLUGINS` + `importlib.metadata.entry_points(group="pipelex.plugins")`, running the version check per plugin, calling `plugin.register(registrar)` (skipping any plugin whose `name` is in `config.plugins.disabled`, logged; raising if a core-unconditional plugin is denylisted — D7), applying fail-loud. Pure ⇒ safely called at CLI-build and at boot.+- `plugins/builtins.py` — `BUILTIN_PLUGINS`. Phase 1 fills it with the LLM drivers.+- `plugins/exceptions.py` — `PluginApiVersionMismatchError`, `DuplicateInferenceBackendError`, `DuplicateOrchestratorError`, `HubSlotAlreadyClaimedError`, `BrokenPluginError`. (`MissingOrchestratorError` lives in `runtime_bridge/exceptions.py` — near its domain — replacing the two old typed errors; see Phase 3.)++The `MakeWorkerFn` signature (D1 — callable, uniform across families):++```python+MakeWorkerFn = Callable[..., InferenceWorkerAbstract]+# called as: make_worker(*, inference_model, backend, sdk_clients, reporting_delegate)+```++**DRY the two repeated blocks** (design §6.2):++- **Client cache** — `SdkClientRegistry.get_or_create(*, handle, build)` replaces the `get_sdk_instance(...) or set_sdk_instance(...)` dance in every arm.+- **Extra guard** — `require_sdk(*, spec, extra, msg)` (raises `MissingDependencyError` if `find_spec(spec) is None`) replaces the repeated guard, moved **into** `make_worker` so a missing extra fails at use, not boot.++**LLM family migration.** `llm_worker_factory.py`'s `match` collapses to `get_inference_backend_registry().lookup(family=LLM, sdk=model_handle.sdk)(...)`. Each LLM driver plugin's `register` calls `add_inference_backend(family=LLM, sdk="anthropic", make_worker=<closure>)`. A registry *miss* raises a distinct "backend `<sdk>` not registered; is its plugin installed?" error.++**Boot wiring.** In `Pipelex.setup()`, after config is fully resolved (incl. the `temporal_enabled` override at `pipelex.py:193-196`) and before the hub setup points, call `build_registrar(config=get_config())`. Store the two keyed registries on the hub (`set_inference_backend_registry` / `set_orchestrator_registry`). Hold the slot-claim thunks, CLI commands, teardown callbacks for their apply-points.++**Conflict & version policy** (fail loud): duplicate `(family, sdk)` / duplicate `execution_mode` / double-claimed slot each raise naming both contributors; `targets_api != PLUGIN_API_VERSION` → `PluginApiVersionMismatchError` with the remedy; a broken entry point → `BrokenPluginError` with context (the one sanctioned `except Exception` site — wraps unbounded third-party plugin code; annotate it).++**Control-surface deliverables (D7).** Add a core-owned `plugins: PluginsConfig` section (`disabled: list[str]`, default `[]` in `pipelex.toml` and `.pipelex/pipelex.toml` — never a class-def default) to `configs.py`; apply the denylist in `build_registrar` (skip + log; error on a core-unconditional name). Add the core `pipelex plugins list` command (lists discovered plugins, their contributions, and denylist state). See the enablement-model section above.++**Tests:**++- Contract conformance (synthetic plugins): protocol satisfaction; `targets_api` mismatch fails loud; duplicate `(family, sdk)` / duplicate mode / double-claimed slot each raise naming both; broken entry point → `BrokenPluginError`; **`build_registrar` idempotent** (two calls, no side effect — pins the side-effect-free invariant).+- Denylist (D7): a discovered non-core plugin named in `plugins.disabled` is skipped and logged; denylisting a core-unconditional plugin (e.g. the OpenAI driver or DIRECT) raises at startup; `pipelex plugins list` shows discovered built-ins + (synthetic) externals with contributions and denylist state.+- **Import-light boot via subprocess + import-blocker (codex C12, replaces the weak `sys.modules` check):** spawn a subprocess with a `sys.meta_path` finder that *raises* on `anthropic`, `mistralai`, `google.genai`, `boto3`, `fal_client`, … then boot with built-ins registered and assert no raise. A `sys.modules`-absence assertion in-process is noisy (prior imports leak); the import-blocker is deterministic.+- Registry round-trip (LLM): register → `lookup(LLM, sdk)` → returns the right `LLMWorkerAbstract`.+- Error parity: an Anthropic model with the `anthropic` extra absent raises the *same* `MissingDependencyError` text (lib + extra + alternative).++**Checkpoint 1.** Seam exists, proven against the real LLM family; import-light enforced by the subprocess guard; conflict/version policy tested. The contract is now locked against a real consumer. Mergeable.++---++## Phase 2 — Remaining inference families (ImgGen, Extract, Search)++**Goal.** Migrate the other three families onto the proven contract. One family per sub-PR.++- **ImgGen.** Collapse the `match`. **Fix the pre-existing huggingface bug (codex C10):** `img_gen_worker_factory.py:64` imports `huggingface_hub` with **no `find_spec` guard** — unlike every other arm, a missing extra raises a raw `ImportError` instead of the friendly `MissingDependencyError`. Add the `require_sdk(spec="huggingface_hub", extra="huggingface", …)` guard during migration (flag-and-fix per repo policy). The provider-literal handling (`model_handle.variant` → provider) lives inside the closure. The substrate-reuse workers (`blackboxai`/`gateway_completions`/`openrouter` → `OpenAICompletionsImgGenWorker` with a per-vendor completions factory) become closures capturing that factory.+- **Extract.** Collapse the `match`. Stateless arms (`pypdfium2`, `linkup_fetch`) skip `get_or_create`.+- **Search — normalize the call surface (codex C9).** `search_worker_factory.py` takes **no `reporting_delegate` param** and pulls `get_report_delegate()` from the hub (`:31`, `:41`), and `make_search_worker` itself omits it. The uniform `MakeWorkerFn` passes `reporting_delegate` explicitly. Resolution: normalize `make_search_worker` and its callers to accept `reporting_delegate` (removing search's hidden hub coupling — a strict improvement), so search fits the uniform signature. This is a small call-surface change, not a special case.++Add each migrated driver to `BUILTIN_PLUGINS`. OpenAI is an always-on built-in driver, no privileged arm (design D6-design); the OpenAI-compat *substrate* stays an in-tree library the gateway/portkey/openrouter/blackboxai closures import — substrate *extraction into a named module* is deferred.++**Tests:** registry round-trip per family; the import-light subprocess guard now covers all optional SDKs; **huggingface error-parity** (missing `huggingface` extra now raises `MissingDependencyError`, proving the bug fix); a **cross-family vendor** test (e.g. `mistral` registers into both LLM and Extract from one `register`).++**Cross-cutting deliverable:** the **inference SPI reference** (design §9.1) + the **plugin-authoring guide and minimal example backend plugin** land here (a backend plugin is the simplest example). Placement: authoring guide + SPI reference in a user-facing `docs/.../plugins/` Guide section; seam internals in `docs/under-the-hood/`; the example as runnable in-repo code.++**Checkpoint 2.** All four `match` worker-factory statements gone; every backend via the registry while in-repo; the huggingface guard bug fixed; search's hub coupling removed; SPI + authoring guide shipped. **Natural session handoff** — inference side done, orchestrators open next.++---++## Phase 3 — Orchestrators through the seam (in-tree)++**Goal.** Collapse the bridge `match` and the four `temporal.is_enabled` boot blocks into registry/slot lookups; move Temporal's modes/CLI/boot-swap/teardown behind its (still-in-repo) plugin; flip Mistral's hard import to discovery; publish the orchestrator SPI. No behavior change.++**Extract the orchestrators** (verbatim bodies):++- `DirectOrchestrator.run` — from `_run_direct` (`bridge.py:270-294`). **Correctness landmine (design §8.1): keep `with scoped_pipe_router(PipeRouter())` verbatim** — dropping it leaks DIRECT-mode nested sub-pipes to Temporal inside a Temporal worker. Registered in core, always-on.+- `TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` — from `_run_temporal_*` (`:297-355`), keeping the `WorkflowExecutionError` catch and `make_workflow_id` recompute.+- `MistralWorkflowsOrchestrator` — **authored in the `pipelex-mistralai-workflows` repo**, not here.++**The bridge collapses** (design §8.1) to a registry lookup. After it, `bridge.py` names no integration.++**Preserve per-mode error quality (codex C7 — a generic hint is a regression).** Today `bridge.py:399/:417` carry exact, mode-specific install messages ("install `pipelex[temporal]`" vs "install `pipelex-mistralai-workflows`"). The registry-miss path must reproduce *those* messages, not a single generic hint. `MissingOrchestratorError` takes the `mode` and maps it to its exact message via the `PipelexExecutionMode.requires_*` properties (the typed source of truth for which mode needs what). One message per mode, preserved verbatim.++**The Temporal plugin** (`TemporalOrchestrator`). Its `register`:++- **always** `add_orchestrator(TEMPORAL_BLOCKING|TEMPORAL_FIRE_AND_FORGET, …)` and `add_cli_command("worker"|"setup-temporal-namespace", …)`;+- **if `config.temporal.is_enabled`**: `claim_content_generator(factory)`, `claim_pipe_router(factory)`, `claim_pipe_run(factory)`, `claim_task_manager(factory)`, `add_teardown(...)` — **D5: each `claim_*` is handed a thunk** (e.g. `claim_content_generator(lambda: ContentGeneratorInWorkflowFactory.make_content_generator_in_workflow())`), so `register` never imports `temporalio`. The thunk runs at the setup apply-point.++**Boot/teardown collapse with explicit injection precedence (codex C8).** The four `if get_config().temporal.is_enabled:` blocks (`pipelex.py:377-466`) become, at each existing ordered point (content generator → task manager → router → run): **explicit `setup()` param (test/user injection at `:376/:450`) > plugin slot-claim thunk > core default.** The slot claim must *not* silently override an explicit injection — pin this precedence in a test. The inlined teardown (`:471-479`) becomes a registered teardown callback run LIFO.++**CLI collapse (codex C4 + D3).** The two hard imports in `_cli.py:15,20,234,238` become plugin-contributed commands. `PipelexCLI.list_commands` (`:32`, which hardcodes command order) is reworked to merge core commands with `registrar.cli_commands` deterministically (stable order, clean `--help`, unknown-command behavior intact). Per D3, the CLI entry point loads config then runs the pure `build_registrar` once to harvest `registrar.cli_commands`; D5's thunks mean this harvest never constructs a Temporal impl even when `temporal.is_enabled`. **Spike this first in Phase 3** — it's the one genuinely novel integration point.++**Mistral → entry-point discovery** (this repo's side). Remove the `_run_mistral_native` hard import (`bridge.py:414`). Uninstalled (CI default) → `MISTRAL_NATIVE` absent → `MissingOrchestratorError` with the exact Mistral message. The real plugin (entry point + orchestrator re-pointed at the SPI) is a `pipelex-mistralai-workflows` change. pipelex's suite proves dispatch with a *fake* registered orchestrator.++**Publish the orchestrator SPI** (design §9.2, sized to Mistral's *measured* imports): `runtime_bridge.*` incl. `runtime_bridge.primitives.*`; the execution protocols; the boundary/core payload types; library-crate access + hub scoping; the tracing hooks. Documented module/symbol list. Mistral's boundary resolves against it.++**Tests:** bridge dispatch by mode (fake orchestrator); **per-mode error parity** (each missing mode → its exact message — C7); DIRECT router scoping (the landmine); DIRECT parity (byte-identical); **injection-precedence** (explicit param wins over a slot claim — C8); boot-via-slots (stripped env resolves DIRECT + core backends; temporal-enabled resolves the four slots to Temporal impls via thunks); teardown LIFO order; **CLI-command contribution** (`worker` appears in `--help` when the plugin is discoverable, absent otherwise — pins D3); **full Temporal suite green in-tree** (§14.5 — the extraction gate; Phase 5 does not start until this is green).++**Checkpoint 3.** Bridge + boot/teardown + CLI name no integration; Temporal fully behind its plugin in-repo (import-light preserved via thunks); per-mode errors preserved; injection precedence pinned; Mistral via discovery; orchestrator SPI published; Temporal suite green through the seam. **The** checkpoint before externalization.++---++## Phase 4 — Invert model listing (the 5th seam, per D6)++**Goal.** Close the goal: `model_lists.py:58`'s `match sdk:` (the `list-models` CLI path) still hardcodes `pipelex.plugins.{openai,anthropic,mistral,google,bedrock}.*_list` imports. Invert it onto the inference contract so core names no integration *anywhere*.++- Add a **model-listing capability** to the inference plugin contract: `add_model_lister(*, sdk, lister: ListModelsFn)` (a callable mirroring `MakeWorkerFn` — import-light, lazy). A backend plugin that lists models registers its lister alongside its worker factory.+- `ModelLister.list_models` (`model_lists.py`) collapses to a registry lookup keyed by `sdk`; a miss → the same friendly "is its plugin installed?" guidance. The `find_spec` guards (already present in this file's arms) move into each lister.+- The contract grows by one optional method — backends that don't support listing simply don't register a lister (progressive disclosure preserved).++**Tests:** round-trip (a registered lister is invoked for its `sdk`); a backend with no lister yields the friendly miss; import-light subprocess guard still green (listers are lazy).++**Checkpoint 4.** `model_lists.py` names no integration; "core names no integration by import or string" holds without an asterisk.++---++## Phase 5 — Externalize Temporal → `pipelex-temporal` (cross-repo, gated on Checkpoint 3)++**Goal.** Lift `pipelex/temporal/` into a separate dist. A *packaging* move — but only after a prerequisite the design missed.++**Step 0 — relocate the Temporal config schema to a core-owned module (codex C6 — the design's D7 is self-contradictory as written).** `config_temporal.py` currently *lives at* `pipelex.temporal.config_temporal` and imports `pipelex.temporal.exceptions` (`:10`: `TemporalConfigError`, `WorkerTaskQueueUnknownError`). "Keep config in core, move `pipelex/temporal/` out" is impossible while the schema sits under the `pipelex.temporal` namespace. Relocate the schema **and the specific exceptions it needs** to a core-owned module outside `pipelex.temporal` (e.g. `pipelex/system/configuration/config_temporal.py` + the two exceptions into a core `exceptions.py`). Update `configs.py:14`. The `if TYPE_CHECKING: … RetryPolicy = Any` placeholder keeps it importable without `temporalio`. This step is low-risk and could be pulled into Phase 3; it is the true prerequisite that makes the rest a packaging move. `make tb` stays meaningful.++**Then:**++- Move `pipelex/temporal/` (impl) + `temporal_plugin.py` + tests + the `temporal` marker + the `--temporal-server` conftest option into `pipelex-temporal`; declare `[project.entry-points."pipelex.plugins"]`.+- Protocol-level conformance tests stay in core; Temporal's behavioral suite travels.+- **Flip downstream pins** (cross-repo blast radius): `pipelex-worker`, `pipelex-api-hosted` from `pipelex[temporal]==X` → `pipelex-temporal==Y`.+- **Repoint `pipelex-mistralai-workflows`** (design §15): its `temporal` extra + `pipelex.temporal.*` imports → `pipelex-temporal`.++**Tests.** Re-run the relocated Temporal suite against the published `pipelex`; run downstream consumers' suites against the new pins before publishing.++**Checkpoint 5.** Temporal is an external plugin; core has no `temporal` extra; downstream pins flipped; consumer suites green.++> Vendor inference dists and OpenAI-substrate extraction are **out of this rollout** (design §6.6, §11) — one-line moves if/when justified.++---++## Resolved open questions (design §17)++- **SPI delivery shape →** documented module/symbol list only, no curated re-export surface (the repo bans `__init__.py` re-exports). The version marker `PLUGIN_API_VERSION` lives in `plugins/contract.py`.+- **Registrar config-access timing →** `build_registrar` runs after config resolution (incl. the `temporal_enabled` override) and before the hub-slot apply-points; every `register` sees a resolved `ConfigRoot`. The CLI-build second call is safe because D5's thunks make slot-claiming import-free and side-effect-free.+- **Factory granularity →** one **callable** per `(family, sdk)` (D1).+- **Where the committed docs land →** authoring guide + SPI reference in a user-facing `docs/.../plugins/` Guide section; seam internals in `docs/under-the-hood/`; minimal example as runnable in-repo code.++## NOT in scope (considered, deferred)++- Third-party-defined execution modes (open string-keyed space) — design D5; enum stays closed.+- Generic per-plugin typed-config namespace — design D7; Temporal config stays typed in core.+- Deliberately *overriding* a built-in (a third party replacing core's OpenAI driver) — design §5.3.+- Per-SPI versions / semver-range matching — design §5.4; single coarse `int`.+- Vendor inference dists + OpenAI-substrate extraction into a named module — design §6.4/§6.6/§11.+- **Collapsing `SdkClientManager` into `SdkClientRegistry`** — P3 follow-up commit after Phase 0 (D4), not bundled into the rename.+- Distribution/CI for `pipelex-temporal` — part of Phase 5's cross-repo effort, not Phases 0–4.++## What already exists (reused, not rebuilt)++The plan inverts the *last* coupling; it reuses, not rebuilds: the lazy-import pattern in every `match` arm; the optional extras (`pipelex[anthropic|temporal|…]`); the `*WorkerAbstract` hierarchy (`cogt/inference/inference_worker_abstract.py` + the four family abstracts); the `SdkClientRegistry` client cache (renamed); the `PipelexExecutionMode` enum + `requires_*` properties; the hub singletons + `scoped_pipe_router` ContextVar scoping; the `Temporal` config placeholder (`RetryPolicy = Any`). The seam + registries are the only genuinely new machinery; everything downstream of a lookup is existing code.++## Failure modes (new codepaths)++| Codepath | Realistic failure | Test? | Error handling? | User sees |+|---|---|---|---|---|+| `build_registrar` discovery | a broken external entry point | yes (conformance) | `BrokenPluginError` w/ context | clear, named |+| inference lookup | backend not registered (plugin absent) | yes (round-trip miss) | distinct "not registered" error | clear |+| `make_worker` closure | missing extra (incl. the fixed hf arm) | yes (error parity) | `MissingDependencyError` (lib+extra+alt) | clear, friendly |+| `register` at CLI-build | importing `temporalio` at CLI startup | yes (import-light subprocess + CLI-contribution) | D5 thunks prevent it | n/a (prevented) |+| DIRECT inside Temporal worker | nested sub-pipes leak to Temporal | yes (router-scoping) | `scoped_pipe_router` preserved | n/a (prevented) |+| slot claim vs explicit injection | slot silently overrides test/user inject | yes (injection-precedence) | explicit param wins | n/a (prevented) |+| orchestrator miss | wrong/absent mode | yes (per-mode parity) | per-mode `MissingOrchestratorError` | exact install message |++No critical gaps (no failure mode is simultaneously untested, unhandled, and silent).++## Worktree parallelization++| Step | Modules | Depends on |+|---|---|---|+| Phase 0 rename | `plugins/`, `hub.py`, `pipelex.py`, all adapter modules | — |+| Phase 1 seam+LLM | `plugins/`, `cogt/llm/` | Phase 0 |+| Phase 2 ImgGen / Extract / Search | `cogt/{img_gen,extract,search}/`, `plugins/<vendor>/` | Phase 1 (contract) |+| Phase 3 orchestrators | `runtime_bridge/`, `pipelex.py`, `cli/`, `temporal/` | Phase 1 |+| Phase 4 model_lists | `cogt/model_backends/`, `plugins/<vendor>/` | Phase 1 (contract) |++**Lanes after Phase 1 lands:** Phase 2's three families are independent of each other and of Phase 3/4 → `Lane A: ImgGen`, `Lane B: Extract`, `Lane C: Search`, `Lane D: Phase 3 orchestrators`, `Lane E: Phase 4 model_lists` can run in parallel worktrees. **Conflict flag:** Lanes A/B/C and E both touch `plugins/<vendor>/` modules (e.g. a vendor serving multiple families) — coordinate the per-vendor plugin object so two lanes don't both edit it. Phase 0 and Phase 1 are strictly sequential (everything depends on the rename then the contract).++## Execution risks++- **Phase 3 carries most risk** — bridge + boot + teardown + CLI at once, plus the DIRECT-router-scoping landmine and the per-mode-error and injection-precedence subtleties codex surfaced. Mitigation: verbatim orchestrator extraction; pin each subtlety with its own test; land the §14.5 Temporal-green checkpoint before externalizing.+- **CLI timing (D3) + thunks (D5)** are the novel integration point. Mitigation: spike first in Phase 3; the CLI-contribution test proves it.+- **Import-light regression** is invisible until profiled. Mitigation: the subprocess import-blocker guard (C12) lands with Phase 1.+- **Phase 5 is cross-repo** — the only phase that can break downstream deploys; gated, separate, with config relocation as its de-risking step-0.++## Sequencing summary++Phase 0 (rename) → Phase 1 (seam + LLM, merged) → Phase 2 (ImgGen/Extract/Search, parallel sub-PRs) → Phase 3 (orchestrators) → Phase 4 (model_lists) → **[gate: §14.5 green]** → Phase 5 (externalize Temporal, cross-repo). Phases 0–4 are independent PRs in `pipelex`; each lands green. Inference SPI + authoring guide ship with Phase 2; orchestrator SPI with Phase 3.++---++## GSTACK REVIEW REPORT++| Review | Trigger | Why | Runs | Status | Findings |+|--------|---------|-----|------|--------|----------|+| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | — | not run |+| Codex Review | outside voice | Independent 2nd opinion | 1 | issues_found | found 5th seam + import-light hole + Phase-4 config contradiction + hf bug — all folded |+| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 1 | clean | 18 findings, 0 critical gaps, 0 unresolved |+| Design Review | `/plan-design-review` | UI/UX gaps | 0 | — | n/a (backend-only) |+| DX Review | `/plan-devex-review` | Developer experience gaps | 0 | — | not run |++- **CODEX:** outside voice caught what the interactive review missed — the 5th seam (`model_lists.py`), the import-light-at-CLI-build hole (resolved via D5 deferred slot thunks), the Phase-5 config self-contradiction (resolved via core-owned config relocation), a pre-existing huggingface `find_spec` bug, and the per-mode-error / injection-precedence gaps. All folded into the plan.+- **CROSS-MODEL:** no standing tension — codex's findings were additive gaps, not disagreements with the review; all resolved by user decision (D5, D6) or clear fold.+- **VERDICT:** ENG CLEARED — ready to implement.++NO UNRESOLVED DECISIONSdiff --git a/wip/plugins/inference-backends-as-plugins.md b/wip/plugins/inference-backends-as-plugins.mdnew file mode 100644index 000000000..093784031--- /dev/null+++ b/wip/plugins/inference-backends-as-plugins.md@@ -0,0 +1,111 @@+# Inference backends as Pipelex plugins++**Status:** assessment / proposed plan (not started)+**Scope:** turn the inference-SDK wrappers under `pipelex/plugins/` (OpenAI, Anthropic, Google, Mistral, Bedrock, Fal, Docling, Linkup, …) into *registerable plugins* — selected out of a registry the backends populate, rather than dispatched from hardcoded `match` statements in core. Optionally, package the heavier ones as separate installable dists.++> This is the **driver** half of the plugin story. Inference backends are homogeneous, many, and *selected by data* (a model's `sdk` handle). The **strategy** half — orchestrators (Temporal, Mistral Workflows) that swap core orchestration seams — is in [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md). The cross-cutting best practices — and the shared discovery mechanism both halves use — are in [`README.md`](README.md).++## Verdict++Partially decoupled, and confusingly named. The wrappers already live in one place (`pipelex/plugins/<backend>/`), already share clean abstract base classes (`LLMWorkerAbstract`, `ImgGenWorkerAbstract`, `ExtractWorkerAbstract`, `SearchWorkerAbstract`, all under `InferenceWorkerAbstract`), already gate heavy SDKs behind optional extras + lazy `importlib.util.find_spec(...)` checks, and already cache instantiated SDK clients in a runtime registry (`PluginSdkRegistry`, keyed by an `sdk@backend/variant` handle). **What's missing is the inversion**: core still owns the backend→implementation mapping in centralized `match plugin.sdk:` statements inside the four worker factories. A third party cannot add a backend without editing core.++So the work here is the mirror of Temporal's: define the registry, have backends register into it (via the *same* entry-point discovery), and delete the `match` statements.++## First: resolve the "plugin" naming overload++The word is already overloaded three ways, and that has to be fixed before adding a real plugin system or every conversation will be ambiguous:++- `pipelex/plugins/<backend>/` — the directories. These are **in-tree SDK adapters**, not installable plugins.+- `Plugin` (`pipelex/plugins/plugin.py`) — a value object: `sdk` + `backend` + `variant`, with an `sdk_handle` like `anthropic@bedrock/claude`. This is a **backend selector / handle**, not a plugin.+- `PluginManager` / `PluginSdkRegistry` — a **runtime cache of instantiated SDK clients** keyed by that handle, with `teardown()`. Not a discovery mechanism.++None of these is a plugin in the installable-distribution sense. Proposed vocabulary:++- Rename the existing trio to what it is: `SdkClientRegistry` / `SdkClientManager`, `BackendSelector` (or keep `Plugin` as `ModelHandle`). Pure rename, no behaviour change.+- Reserve **"plugin"** for the new concept shared with Temporal: an installable unit, discovered via the `pipelex.plugins` entry-point group, that registers extensions at boot.++(This rename can land independently and ahead of everything else — it's low-risk and unblocks clear naming for the rest.)++## What kind of plugin these are++**Driver plugins.** Many of them, all the same shape, chosen at runtime by the model spec's `sdk` field. They don't replace core behavior; they *supply an implementation* for a backend the core asked for. The right seam is a **typed registry keyed by `sdk` handle**, one per worker family. A driver plugin's job is to call `registry.register(sdk_handle, factory)` at boot. Contrast Temporal, which swaps singleton seams on the hub — different hook, same discovery.++## The seams today++Four worker factories, each a centralized dispatch on `plugin.sdk`:++| Factory | File | Backends in its `match` |+|---|---|---|+| `LLMWorkerFactory` | `cogt/llm/llm_worker_factory.py` | `gateway_completions`, `portkey_completions`, `openai` / `azure_openai`, `anthropic` / `bedrock_anthropic`, `mistral`, `bedrock_boto3` / `bedrock_aioboto3`, `google` |+| `ImgGenWorkerFactory` | `cogt/img_gen/img_gen_worker_factory.py` | `gateway_img_gen`, `fal`, `huggingface_img_gen`, `openai_img_gen`, `blackboxai_img_gen`, `azure_rest_img_gen`, `google` |+| `ExtractWorkerFactory` | `cogt/extract/extract_worker_factory.py` | `gateway_extract`, `mistral`, `pypdfium2`, `docling_sdk`, `linkup_fetch` |+| `SearchWorkerFactory` | `cogt/search/search_worker_factory.py` | `linkup`, `gateway_search` |++Each `case` does the same three things: optionally `importlib.util.find_spec(...)` to raise a friendly `MissingDependencyError` (lib name + extra name), lazily import the backend's factory + worker, then build the worker (reusing a cached SDK client from `PluginSdkRegistry`). The lazy import inside each `case` is exactly the inversion point — it's already "don't touch the SDK until this backend is selected"; it just needs to become "ask the registry for the selected backend's factory."++Supporting structure already in place:++- **Abstract contracts.** `InferenceWorkerAbstract` → `{LLM,ImgGen,Extract,Search}WorkerAbstract`. A registered factory returns one of these. `PluginFactoryAbstract.make_extras(...)` is a partial start at a common factory contract.+- **Optional extras.** `pyproject.toml`: `anthropic`, `bedrock`, `docling`, `fal`, `google` / `google-genai`, `huggingface`, `linkup`, `mistralai` (plus `dynamodb` / `s3` infra extras). `openai`, `portkey`, `pypdfium2` are **core deps** (always present), so `gateway`/`portkey`/`openai`/`azure_rest` backends never need a guard. (OpenAI is a core dep for a deeper reason than "it's always installed" — see "OpenAI is two things" below.)+- **Runtime client cache.** `PluginSdkRegistry` keyed by `sdk_handle` — keep this as-is (rename aside); it's the right place to memoize clients and already has lifecycle teardown.++## "Plugin" and "in-repo" are orthogonal axes++Before deciding what ships where, kill one confusion: **being a plugin** (registering through the seam) and **shipping as a separate dist** are independent choices. A built-in plugin that lives in this repo is the *default*, not a contradiction.++| | **In-repo** | **Separate dist** |+|---|---|---|+| **Through the seam** (registry / entry point) | built-in plugins — the default for *every* backend (OpenAI driver, gateway, portkey, …) | external plugins (`pipelex-anthropic`, `pipelex-google`, …) |+| **Hardcoded in core** | the wart we're deleting | n/a |++"Plugin" = *registers through the seam* instead of a `match` arm. "In-repo vs separate dist" = a *packaging* decision (the granularity choices below). They don't constrain each other — so "keep it internal" never has to mean "keep it a hardcoded special case."++## OpenAI is two things: substrate vs driver++The OpenAI SDK is not just another backend — it is the lingua franca a large fraction of the others are built on. `gateway`, `portkey`, `openrouter`, `blackboxai` all construct `openai.AsyncOpenAI(base_url=...)` (OpenAI-compatible endpoints), and even `anthropic`, `mistral`, `google` import the SDK for the shared `ChatCompletion*` message types. So "the OpenAI SDK" hides two concerns with different fates:++- **Substrate** — the OpenAI-compatible client (`AsyncOpenAI(base_url=…)`), the shared request/response types, and the completions/responses plumbing that gateway/portkey/openrouter/blackboxai/azure_rest reuse. This is **infrastructure other backends import**, not a backend itself.+- **Driver** — the `openai` / `azure_openai` / `openai_img_gen` registrations: the actual "talk to api.openai.com" worker.++They live in the same `pipelex/plugins/openai/` directory today but want opposite treatment:++- **Substrate → core internal library, always a hard dependency.** Promote it out of `pipelex/plugins/openai/` into a deliberate module (e.g. `pipelex.openai_compat`) with a stable signature, importable by any backend — in-repo or external. The "OpenAI API is a de-facto standard" argument is precisely the argument for keeping the substrate in core: if it were instead an optional dist, gateway/portkey/openrouter/blackboxai/azure would *all* have to depend on it — a diamond where a dozen "plugins" pull `pipelex-openai`. Core is the right home. And once plugins import it, the substrate becomes a **contract** — shape it deliberately rather than letting plugins reach into `pipelex/plugins/openai/` internals (which `anthropic_factory.py` et al. effectively do today).+- **Driver → always-on built-in plugin, through the seam.** No extra, registered unconditionally — but registered like everyone else. Do **not** leave OpenAI as the one surviving hardcoded `match` arm: the seam's whole value is *core names no backend*, and a privileged special case is a two-tier system the next "important" backend will want to join, eroding the abstraction. It ships in-repo and is always present; it just goes through the same door.++Net: **substrate in core (a library, not a plugin); OpenAI driver is an always-on built-in plugin through the seam.** OpenAI never leaves the repo and is never gated — without keeping a privileged backend.++## Recommended approach: a worker-factory registry the backends populate++1. **One registry per worker family** (or one registry, partitioned by family), keyed by `sdk` handle, mapping to a factory callable returning the right `*WorkerAbstract`. Core owns the registry + the abstract contracts; backends own the entries.++2. **Each backend registers itself.** Replace the `match` arms with `registry.lookup(plugin.sdk)`. The friendly `MissingDependencyError` (lib + extra name) moves into the backend's registration/factory so the error stays as helpful as today. Built-in backends register at boot through the same `pipelex.plugins` entry-point mechanism Temporal uses — **dogfood the plugin API in-tree first**, before any backend is externalized.++3. **Keep selection data-driven.** Nothing changes for users: models still declare `sdk` / `backend_name` / `variant` in TOML; `Plugin.make_for_inference_model(...)` (→ `ModelHandle`) still builds the handle; the registry lookup replaces the `match`. The `MISSING extra` failure mode is preserved, just relocated.++4. **Decide packaging granularity** (this is the real product decision, deferrable):+ - **In-tree but registerable (lowest churn).** All backends stay in `pipelex`, but register through the seam. Wins: third parties can add a backend in their own dist without a core PR; the `match` statements die; no distribution split to maintain. This alone delivers most of the value.+ - **Per-vendor dists** (`pipelex-anthropic`, `pipelex-google`, `pipelex-bedrock`, …): each owns its SDK dep and its entry point; base `pipelex` ships only the always-on backends (openai/portkey/gateway). Wins: smallest base install, independent cadence, clean dependency surface. Costs: many small repos/dists, version-matrix testing, more release overhead.+ - **Bundles** (e.g. `pipelex-aws` = bedrock + s3 + dynamodb; `pipelex-google` = genai + vertex): a middle ground grouping by shared SDK/credentials.++ Recommendation: **adopt the registry seam first (in-tree), externalize selectively later** — only split out a backend when its dependency weight or release cadence actually justifies a separate dist. The seam makes that a packaging move, not a refactor.++## Watch-items++- **Granularity of the handle.** `sdk` vs `backend` vs `variant` already encode a lot (`anthropic@bedrock/claude`). The registry key should be the `sdk` handle the factories currently switch on — confirm that's stable and 1:1 with a factory.+- **Inconsistent factory shapes.** Some backends have a factory class (`AnthropicFactory`, `MistralFactory`), some instantiate the worker directly (Fal). Normalize on one registration contract (extend `PluginFactoryAbstract`) so every entry looks the same.+- **Core-dep backends.** openai/portkey/gateway/pypdfium2/azure_rest have no extra and must always be registered — make their registration unconditional so a misconfigured plugin environment can't drop them.+- **OpenAI substrate is a shared contract.** Once the OpenAI-compat substrate is promoted to a core module that other backends import (see "OpenAI is two things"), its signature is part of the surface plugins depend on — version and shape it deliberately, don't let it drift as an accidental internal.+- **Cross-family backends.** `mistral` appears in LLM *and* Extract; `google` in LLM *and* ImgGen; `linkup` in Extract *and* Search; `gateway_*` everywhere. A per-vendor dist must register into multiple family registries — the plugin's `register_*` hook needs access to all of them.+- **Error parity.** The `MissingDependencyError` messages today are genuinely helpful (they suggest alternatives, e.g. "use Anthropic via Bedrock instead"). Preserve that quality when moving the guard into registration.++## Effort & risk++- **Moderate, and mostly mechanical** — but with more surface than Temporal (four factories, ~15 backends, two extras families) and a naming refactor up front. No new hard algorithms; the lazy-import + extras + client-cache machinery already exists.+- The risk concentrates in the **registration timing** (everything must be registered before the first worker is built) and in **not silently dropping a core backend**. Both are covered by keeping built-in registration unconditional and tested.++## Suggested phasing++- **Phase 0 — rename.** `Plugin`/`PluginManager`/`PluginSdkRegistry` → `ModelHandle` (or `BackendSelector`) / `SdkClientManager` / `SdkClientRegistry`. Pure rename. _Checkpoint: suite green, naming unambiguous, "plugin" freed for the real concept._+- **Phase 0.5 — extract the OpenAI substrate.** Promote the OpenAI-compatible client + shared types + completions/responses plumbing out of `pipelex/plugins/openai/` into a deliberate core module (e.g. `pipelex.openai_compat`); repoint gateway/portkey/openrouter/blackboxai/azure_rest (and the type-only imports in anthropic/mistral/google) at it. Pure move + import rewrite, still in core. _Checkpoint: suite green; no backend reaches into `pipelex/plugins/openai/` internals — the substrate is the only OpenAI surface they touch._+- **Phase 1 — registry seam, in-tree.** Add the per-family worker-factory registry; convert each `match` arm (OpenAI driver included — no special case) to a registration + the factory body; built-in backends register through the shared `pipelex.plugins` entry point. No packaging change, no behaviour change. _Checkpoint: full suite green with all backends in-repo but selected via the registry — proves the seam (mirrors Temporal Phase 1)._+- **Phase 2 — externalize selectively (optional).** Split out only the backends whose dependency weight or cadence justifies it, into per-vendor or bundle dists, each declaring its entry point and owning its extra. The OpenAI driver and the `openai_compat` substrate stay in core. _Checkpoint: base `pipelex` ships only always-on backends + the substrate; each external backend's suite green against the published base._diff --git a/wip/plugins/option-a-drop-cli-command-seam.md b/wip/plugins/option-a-drop-cli-command-seam.mdnew file mode 100644index 000000000..a4dbb86fa--- /dev/null+++ b/wip/plugins/option-a-drop-cli-command-seam.md@@ -0,0 +1,115 @@+# Option A — drop the CLI-command seam; Temporal ops as the `pipelex-temporal` console script (as-built)++> ✅ **IMPLEMENTED.** Committed on `refactor/Plugins-2` as `989c9beed` ("Refactor CLI command handling and introduce standalone pipelex-temporal console script"). All gates green: `make agent-check` (pyright 0 / mypy 0 over 2240 files / keyword-only), `make tb`, `make agent-test` ("All tests passed."), `pytest tests/integration/pipelex/temporal/` (156 passed, 4 xpassed = the Phase-3 baseline xdist markers), plus both `--help` smokes. This is the faithful as-built record — the basis Phases 4–5 build on.+>+> **What it did:** removed the plugin CLI-command-contribution seam (the former decision **D3**) entirely. The two Temporal operational commands — `worker` and `setup-namespace` — now ship as a standalone `pipelex-temporal` console script (`[project.scripts]` → `pipelex.temporal.temporal_cli:app`), not harvested `pipelex` subcommands. This resolved the whole [critical finding](phase-3-critical-cli-harvest-fragility.md) by **removing the surface**, not hardening it: `pipelex --help` no longer loads config or scans entry points, and no plugin can shadow a core command or brick the recovery commands. The inference-backend / orchestrator / boot-slot seams are **untouched**.+>+> Read [TODOS.md cold-start primer](../../TODOS.md#cold-start-primer-read-this-first-if-youre-new-to-the-session) for the plugin-seam vocabulary (registrar, `build_registrar`, `BUILTIN_PLUGINS`, slot-claims, D1–D7).++## Why this, in one screen++The seam treats "CLI command" as just another contribution type alongside "inference backend" and "orchestrator." It isn't — and that mismatch is the whole bug. The other contributions are consumed **lazily, at point-of-use, inside a booted Pipelex** (a backend when a model needs a worker; an orchestrator at execution time; a slot-claim at boot). The seam's **fail-loud** invariant is correct there. CLI commands are consumed **eagerly, at module-import of `pipelex/cli/_cli.py`, before anything boots, on every `pipelex` invocation including `--help`**. On that path fail-loud is a liability: one broken/colliding installed plugin (or an unreadable config) bricks every command — including `doctor`/`plugins`/`init`, the very commands you'd use to disable the bad plugin. The four facets in the critical doc and the "double discovery / import-time cost" item in [`phase-3-review-deferred.md`](phase-3-review-deferred.md) are all symptoms of this one decision.++The consumer that justified the seam is tiny and operational: `add_cli_command` has exactly **one** caller (the Temporal plugin), contributing exactly **two** commands — `worker` (start a long-running worker daemon) and `setup-temporal-namespace` (one-time namespace bootstrap). Neither is how a pipeline *runs* on Temporal — distributed execution goes through the orchestrator registry via `execution_mode`, which is untouched by any of this. Operational daemons are the textbook use case for a **console script** (`[project.scripts]`), which pip materializes into its own executable — no host-CLI harvest, no import-time discovery, no fragility. Pipelex already ships three (`pipelex`, `pipelex-agent`, `pipelex-dev`). And it is exactly where Phase 5 wants to land: when Temporal externalizes to `pipelex-temporal`, that dist owns its own `pipelex-temporal` console script natively, with no custom machinery to move.++**Net effect of this change:** deletes the harvest, all four facets, the double-discovery, and the per-invocation config-load + entry-point-scan on `--help`. Strictly less code, strictly less fragility, strictly better Phase-5 alignment. The only real cost is one user-facing rename (`pipelex worker` → `pipelex-temporal worker`) and a one-line Dockerfile change in the `pipelex-worker` repo (gated on the release — see the cross-repo stage).++## The design (as built)++A single grouped Temporal CLI exposed as a console script:++- New console script in `pyproject.toml`: `pipelex-temporal = "pipelex.temporal.temporal_cli:app"`.+- New module `pipelex/temporal/temporal_cli.py`: a `typer.Typer()` app registering two subcommands — `worker` and `setup-namespace`.+- The two command callables **move** from `pipelex/cli/commands/` to `pipelex/temporal/` (they are Temporal-owned and must travel together in Phase 5; absolute imports keep working from the new location):+ - `pipelex/cli/commands/worker_cmd.py` → `pipelex/temporal/worker_cmd.py` (`worker_cmd`)+ - `pipelex/cli/commands/setup_temporal_namespace_cmd.py` → `pipelex/temporal/setup_namespace_cmd.py` (`setup_namespace_cmd`)+- Resulting invocations: `pipelex-temporal worker [--no-sandbox --task-queue … --scope … --profile …]` and `pipelex-temporal setup-namespace [--dry-run --server …]`.++Both command callables are already import-light at module top (temporalio is pulled lazily inside their bodies), so importing `temporal_cli.py` when you run `pipelex-temporal` is import-light, and — crucially — `pipelex/cli/_cli.py` no longer references them at all, so `pipelex --help` cannot touch Temporal. The import-light *boot* guard is unaffected and still applies to boot.++**Design choices (as built):**++- *Grouped script vs two flat scripts.* Grouped `pipelex-temporal` (chosen) over flat `pipelex-worker` + `pipelex-setup-temporal-namespace`. Grouped maps 1:1 to the Phase-5 dist and avoids colliding with the `pipelex-worker` *repo/dist* name.+- *Move the command modules vs leave them in `pipelex/cli/commands/`.* Move (chosen) — Phase-5-aligned, "solid over quick." Cost is mechanical: update import + mock paths in one test (`test_setup_temporal_namespace_cmd.py`).+- *Subcommand name `setup-namespace` vs `setup-temporal-namespace`.* `setup-namespace` (chosen) — "temporal" is redundant under a `pipelex-temporal` group, and every embedding string changes anyway.++## Out of scope — do NOT unwind these++The valuable coupling inversion stays exactly as shipped. **Touch none of it:**++- The four inference-family `match` → `InferenceBackendRegistry` lookups (Phases 1–2).+- The bridge `match` → `OrchestratorRegistry.get_optional(mode=…)` dispatch (Phase 3).+- The `temporal.is_enabled` boot-slot claims + LIFO teardown via `_resolve_hub_slot` (Phase 3).+- `build_registrar`, `BUILTIN_PLUGINS`, the `pipelex.plugins` entry-point group for **runtime** contributions, `pipelex plugins list`, the denylist.++This change removes only the **CLI-command** contribution path. Everything else the seam does is consumed at boot/use where fail-loud is correct.++## Implementation steps (all done — retained as the record of what changed)++### Step 1 — Stand up the new Temporal CLI surface++1. Create `pipelex/temporal/temporal_cli.py`: `app = typer.Typer()`; register `worker_cmd` as `worker` and `setup_namespace_cmd` as `setup-namespace` (use `app.command(name="worker")(worker_cmd)` / `app.command(name="setup-namespace")(setup_namespace_cmd)`, or `@app.command` wrappers — match the existing Typer idiom in the repo). Keep it import-light at module top.+2. Move `pipelex/cli/commands/worker_cmd.py` → `pipelex/temporal/worker_cmd.py` and `pipelex/cli/commands/setup_temporal_namespace_cmd.py` → `pipelex/temporal/setup_namespace_cmd.py` (rename the callable `setup_temporal_namespace_cmd` → `setup_namespace_cmd`). Their internal imports (`pipelex.cli.cli_factory`, `pipelex.cli.error_handlers`, `pipelex.config`, `pipelex.pipelex`) are absolute and unaffected.+3. Add to `pyproject.toml` `[project.scripts]`: `pipelex-temporal = "pipelex.temporal.temporal_cli:app"`.++### Step 2 — Remove the seam++4. `pipelex/cli/_cli.py`: delete `_config_for_cli_harvest()`, `_register_discovered_cli_commands()`, and the module-level `_PLUGIN_COMMAND_NAMES = …`. Change `PipelexCLI.list_commands` to `return list(_CORE_COMMAND_ORDER)`. Update the `_CORE_COMMAND_ORDER` comment block (drop the "plugin-contributed commands are appended after" paragraph). Let `make fix-unused-imports` + `make agent-check` strip the now-dead imports (expected: `importlib`, `ValidationError`, `build_registrar`, `config_manager`, `PipelexConfig`, `TomlError`, possibly `log`).+5. `pipelex/plugins/registrar.py`: remove `add_cli_command`, the `CliCommand` NamedTuple, the `self.cli_commands` field (and its mention in the class docstring), and the `"cli command {name}"` contribution line. `plugins_cmd.py` renders `discoveries[].contributions` generically, so no change needed there — those lines simply stop appearing.+6. `pipelex/temporal/temporal_plugin.py`: remove the two `registrar.add_cli_command(...)` calls and the comment above them about declaring commands by `import_path`. Update the module + class docstrings (the bullet that says the plugin "contributes the … `worker` / `setup-temporal-namespace` CLI commands"). The plugin now contributes only orchestrators (always) and slot-claims/teardown (when `is_enabled`).++### Step 3 — Update in-code user-facing strings++7. `pipelex/temporal/tprl/namespace_check.py`: `PIPELEX_SETUP_CLI_COMMAND` → `"pipelex-temporal setup-namespace"`; update the two docstring mentions (~`:19`, `:165`).+8. `pipelex/temporal/exceptions.py` (~`:90`): docstring mention of `pipelex setup-temporal-namespace` → new invocation.+9. The two moved command modules' own docstrings + `Examples:` blocks: `pipelex worker …` → `pipelex-temporal worker …`; `pipelex setup-temporal-namespace …` → `pipelex-temporal setup-namespace …`.++### Step 4 — Tests++10. **Delete** `tests/unit/pipelex/cli/test_plugin_cli_command_harvest.py` — its entire subject (the harvest + `_config_for_cli_harvest`) is gone. Optionally replace with a ~5-line smoke test asserting `pipelex.temporal.temporal_cli.app` exposes `worker` + `setup-namespace` (`[c.name for c in app.registered_commands]`).+11. **Update** `tests/unit/pipelex/cli/test_setup_temporal_namespace_cmd.py`: import path `from pipelex.temporal.setup_namespace_cmd import setup_namespace_cmd` and all `mocker.patch("pipelex.temporal.setup_namespace_cmd.…")` targets; update the callable name if renamed.+12. Confirm the import-light boot guard test (the subprocess that blocks `temporalio`) still passes — it pins boot, not the harvest, so it should be untouched. Grep showed `test_plugin_cli_command_harvest.py` is the *only* harvest-coupled test.++### Step 5 — Docs++13. Mechanical prefix swap in **current** user docs (`pipelex worker` → `pipelex-temporal worker`; `pipelex setup-temporal-namespace` → `pipelex-temporal setup-namespace`; flags `--scope/--profile/--task-queue/--dry-run/--server` unchanged): `docs/distributed-execution/workers.md`, `docs/distributed-execution/temporal/index.md`, `docs/distributed-execution/cluster-setup.md`, `docs/features/distributed-execution.md`, `docs/running-locally-on-temporal.md`. **Do NOT touch** `docs/history/**`, `wip/history/**`, `docs/plans/**` (archival).+14. **Rewrite** `docs/under-the-hood/orchestrator-plugins.md`'s "CLI-by-import-path" section: the orchestrator SPI no longer harvests CLI commands; a plugin that wants an operational command ships its own console script (`[project.scripts]`). **Fold in the deferred-doc fix** here too: the line (~`:116`) claiming the Temporal plugin is "discovered via an entry point pipelex declares on itself" is factually wrong — it is in `BUILTIN_PLUGINS`, and `pyproject.toml` declares no `pipelex.plugins` self-entry-point. State the builtin-list mechanism + the real Phase-5 delta. (`docs/under-the-hood/inference-backend-plugins.md` does **not** mention the CLI seam — leave it.)+15. Sanity-check (almost certainly clean): no `docs/specs/**` or `conformance/**` command-surface entry references these invocations (verified — the grep found none), so no `make check-spec-links` concern. Re-verify after editing.++### Step 6 — Tracker bookkeeping++16. `TODOS.md`: mark **D3 superseded** by this doc (one line in the locked-decisions list + the CLI seam row of the cold-start primer). Note the CLI-command contribution path was removed; runtime contributions (backends/orchestrators) and `plugins list` are unchanged.+17. `phase-3-critical-cli-harvest-fragility.md`: add a top banner — **RESOLVED by removal**, pointing here.+18. `phase-3-review-deferred.md`: mark the "Double discovery + import-time cost" Low item **resolved** (the import-time harvest is gone); mark the "SPI doc factually wrong" Medium item resolved (folded into step 14).++### ✅ CHECKPOINT — verify gate (PASSED)++- ✅ `make agent-check` clean (pyright 0 / mypy 0 over 2240 files / ruff+plxt / keyword-only).+- ✅ `make tb` green — 5 passed (registrar shape changed — boot/config sanity).+- ✅ `make agent-test` green ("All tests passed.", exit 0).+- ✅ `.venv/bin/pytest tests/integration/pipelex/temporal/` — **156 passed, 4 xpassed** (the §14.5 Temporal gate; the 4 xpassed are the pre-existing xdist class-registration flakiness markers, matching the Phase-3 baseline exactly).+- ✅ Manual smoke: `.venv/bin/pipelex --help` lists core commands only — no `worker`/`setup-temporal-namespace`, no config load / entry-point scan; `.venv/bin/pipelex-temporal --help` lists `worker` + `setup-namespace`.+- ✅ Committed on `refactor/Plugins-2` as `989c9beed` (one-commit-per-checkpoint delivery model).++> **Note on the env:** materializing the new console script needs `uv sync --all-extras` (it regenerates `[project.scripts]` wrappers into `.venv/bin/`). That sync also brings the venv up to the already-committed `uv.lock` (it had drifted behind on `cryptography`/`mthds`); `uv.lock` itself is unchanged. Separately, `make cleanderived` deletes the gitignored generated fixture `tests/integration/pipelex/fixtures/_generated_model_sets.py` — regenerate it with `make regenerate-test-models-quiet` (alias `rtm`) before `make agent-check`, or pyright fails on the missing import.++## Cross-repo follow-up — GATED on the release that ships the script++This is the one downstream cost, and it is **release-gated** — do **not** push it ahead of the pipelex release that carries `pipelex-temporal`. Blast radius is exactly one repo (`pipelex-api-hosted`, `sandbox`, `pipelex-api` were checked and do **not** invoke the worker):++- `pipelex-worker/Dockerfile` (~`:25`): `CMD ["pipelex", "worker", "--no-sandbox"]` → `CMD ["pipelex-temporal", "worker", "--no-sandbox"]`.+- `pipelex-worker/Makefile` (~`:125`): `pipelex worker --no-sandbox --task-queue test` → `pipelex-temporal worker …`.++**Rule:** whoever bumps `pipelex-worker`'s `pipelex` pin to the release containing this change must flip these two lines in the **same** commit. Until that bump, the old pinned pipelex still has `pipelex worker`, so nothing breaks. This session is **pipelex-only** — capture the above as a standing follow-up (this section is that record; mirror it into the Phase-5 downstream-pin step in `TODOS.md` so it isn't lost).++## Open cleanups (investigate, note — do NOT expand scope)++- **`ensure_global_if_missing` / `load_base_config_dict`.** ⚠️ **Investigated — confirmed sole caller, FLAGGED for a follow-up (not removed here).** `_config_for_cli_harvest` (now deleted) was the **only** caller of both `config_manager.load_config(ensure_global_if_missing=False)` and `config_manager.load_base_config_dict()` (verified by grep across `pipelex/` + `tests/`). After this change: `load_base_config_dict()` has **zero** callers (dead method), and every `load_config` caller uses the default `ensure_global_if_missing=True` so the `False` branch (`config_loader.py:~260`) is dead. Both are in `pipelex/system/configuration/config_loader.py` (`load_base_config_dict` ~`:195`, the param ~`:205`/`:242`/`:260`). Left in place deliberately — removing the param means reshaping `load_config`'s signature + collapsing the branch, a config-loader edit out of scope for "drop the CLI seam." Neither breaks a gate (ruff/pyright don't flag unused public methods). **Follow-up:** delete `load_base_config_dict()` and simplify `load_config` to always-ensure-global.+- **Legacy `pipelex/temporal/worker_cli.py`.** A second, heavier worker entrypoint (`python -m pipelex.temporal.worker_cli`, the `configure` command) that imports `temporalio` at module top and duplicates worker-start logic. Out of scope here; note it as a unification candidate (it could become a thin alias of, or be folded into, the new `temporal_cli.py`). Don't fix unless trivial.++## Pointers++- Seam removal: `pipelex/cli/_cli.py` (harvest), `pipelex/plugins/registrar.py` (`add_cli_command`/`CliCommand`), `pipelex/temporal/temporal_plugin.py` (the two calls).+- Command callables to move: `pipelex/cli/commands/worker_cmd.py`, `pipelex/cli/commands/setup_temporal_namespace_cmd.py`.+- Strings: `pipelex/temporal/tprl/namespace_check.py` (`PIPELEX_SETUP_CLI_COMMAND`), `pipelex/temporal/exceptions.py`.+- Decision lineage: supersedes **D3** in [TODOS.md](../../TODOS.md); resolves [phase-3-critical-cli-harvest-fragility.md](phase-3-critical-cli-harvest-fragility.md) and two items in [phase-3-review-deferred.md](phase-3-review-deferred.md).diff --git a/wip/plugins/orchestration-mode-and-delivery-split.md b/wip/plugins/orchestration-mode-and-delivery-split.mdnew file mode 100644index 000000000..b32c6c319--- /dev/null+++ b/wip/plugins/orchestration-mode-and-delivery-split.md@@ -0,0 +1,387 @@+# Plan — split `PipelexExecutionMode` into `orchestration_mode` (open token) + `DeliveryMode` (closed enum)++**Status:** **ALL PHASES DONE — Phases 1 (core) + 2 (Temporal) + 3 (`pipelex-api`, THE gate) + 4 (Mistral plugin migration + bookkeeping) complete.** Phase 1: `_plugins` commit `bd4e52827` (+ docs `9c7a7785d`, + this tracker `919c09348`) on `feature/orchestration-mode-delivery-split`. Phase 2: `pipelex-temporal` commit `01d7dc3` on `feature/orchestration-mode-delivery-split`. Phase 3: `pipelex-api` commit `dd84a9a` on `feature/orchestration-mode-delivery-split` (cut off `06cd5e6`). All three with all gates green + clean-context `/code-review` at their checkpoints (Phase 3 review: **no must-fix issues**). The §8 `/start`-honesty conformance risk is **resolved** (honest 400 is safe, landed). **Phase 4 DONE** — the external `pipelex-mistralai-workflows` plugin migrated to the new core surface (commit `0eb3878` on `feature/Mistral-native`): deleted-enum import + registration under the deleted member + own `execution_mode` wire field + old `run` signature → `orchestration_mode`/`delivery`/new protocol, `supports_fire_and_forget=False`. Gates green + clean-context `/code-review` (one must-fix — a stale README — applied). **DEVIATION:** the plugin pins core editable at `../_workflows` (`feature/mistralai-2x-bump`), NOT the orchestration-split worktree, so Phase 4 first merged `feature/orchestration-mode-delivery-split` → `feature/mistralai-2x-bump` (`_workflows` merge `a1448ad8f`, conflict-free, core gates green). `make check-spec-links` was NOT triggered (no `docs/specs/` heading or `conformance/` test asserts mode values). See **"### Phase 4 — as-built"** in §10 for the full record. The one carry-forward from Phase 3 (the `pipelex-api` editable-pin → pushed-git-SHA flip before any PR) is still open and now joined by the `_workflows`/`_plugins` doc-reconciliation note (see §10 Phase 4 as-built). Design **locked** in discussion (2026-06-22). This is **Direction A** — the real two-axis split across all three repos (`pipelex` core → `pipelex-temporal` → `pipelex-api`), not the API-edge-only reshape (Direction B, rejected: it relocates the smell to the boundary instead of removing it). It **supersedes** the dual-mode framing in the two follow-on plans `wip/plugins/execute-per-request-execution-mode.md` and `wip/plugins/orchestrator-dispatched-validate.md` — both shipped a working surface on top of the *flat* `PipelexExecutionMode`; this plan removes the flatness those plans worked around (`_async_start_mode`, the `/execute` fire-and-forget 400, the `ApiConfig` f&f field-validator, the `/validate` double-registration). Tracker: `_plugins/TODOS.md`. Parent effort: `wip/plugins/orchestrator-agnostic-runner-and-flavors.md`.++> **START HERE (cold start).** Read this whole doc. The change is conceptual, not big-mechanical-only: today one enum `PipelexExecutionMode {DIRECT, TEMPORAL_BLOCKING, TEMPORAL_FIRE_AND_FORGET, MISTRAL_NATIVE}` conflates **two orthogonal axes** — *which orchestrator runs the pipe* and *whether the caller waits*. We split them: an **open string token** `orchestration_mode` (core owns only `"direct"`; plugins own `"temporal"`, `"mistralai-workflows"`) and a **closed enum** `DeliveryMode {BLOCKING, FIRE_AND_FORGET}` that endpoints set themselves and never receive from a caller. Delivery becomes a parameter to `OrchestratorProtocol.run`, so the two Temporal orchestrators collapse into one and the `/validate` double-registration disappears. Lands core → temporal → api, one repo / branch / commit per phase, clean-context `/code-review` at each checkpoint — same discipline as the rest of the effort.++---++## 1. The ask + locked decisions++**The ask (user):** the API surface for choosing *how* a call to `/execute`, `/start`, `/validate` runs is wrong. `PipelexExecutionMode` carries two things at once — the backend and the wait-semantics — and that conflation forced three separate workarounds (`_async_start_mode`, the `/execute` f&f 400, the `/validate` collapse). Re-cut it into the right enum(s) passed as args.++**Why the conflation is the root cause.** The blocking-vs-fire-and-forget decision is **already expressed by which endpoint you call** — `/execute` and `/validate` are synchronous, `/start` is fire-and-forget. Encoding it a *second* time inside the mode enum is the duplication, and every smell is a workaround for it:++- `_async_start_mode` (`pipeline.py:109-134`) maps `temporal_blocking → temporal_fire_and_forget` because `/start` has to re-derive the delivery axis the enum shouldn't carry.+- `/execute` rejects `temporal_fire_and_forget` with a 400 (`ErrorType.FIRE_AND_FORGET_NOT_SUPPORTED`) — refusing a delivery value the caller should never have been able to express.+- `ApiConfig._reject_fire_and_forget_default` (`api_config.py:54-74`) rejects a *configured* f&f mode — same problem on the config side.+- `TemporalPlugin.register` registers the **same** bundle validator under **both** temporal modes (`temporal_plugin.py:136-138`, decision D of the validate plan) — collapsing the axis because validate doesn't have it either.++Three endpoints, three kludges, one root cause: a wait-semantics dimension the caller/deployment should never receive.++**Locked decisions (do not re-litigate; raise explicitly before deviating):**++- **D-A — Two axes, two types.** Replace `PipelexExecutionMode` with `orchestration_mode` (the backend) + `DeliveryMode` (the wait-semantics). They are orthogonal: orchestration is per-call/config-chosen; delivery is endpoint-intrinsic.+- **D-B — `orchestration_mode` is an OPEN string token, not a closed enum.** Core defines only `"direct"` (its single built-in orchestrator). Each plugin defines and registers its own token *where its orchestrator lives* (`pipelex-temporal` owns `"temporal"`; `pipelex-mistralai-workflows` owns `"mistralai-workflows"`). The registry is keyed by `str`; validation is *"is a mode registered?"* — a lookup miss raises `MissingOrchestratorError` — never *"is it in an enum?"*. **This is the deliberate, justified exception to the repo's StrEnum-everywhere standard:** that standard exists to make the linter scream when a *closed* value is added and a branch forgotten; a plugin-contributed set is genuinely open, exhaustive matching is impossible, and the dict-lookup-or-raise *is* the correct unknown-handling. It is also the only choice faithful to the effort's north star (*the public base names no orchestrator*) — a closed `{DIRECT, TEMPORAL, MISTRAL}` enum would re-introduce the exact core→plugin coupling Phase C removed.+- **D-C — `DeliveryMode {BLOCKING, FIRE_AND_FORGET}` is a closed core StrEnum, endpoint-intrinsic, never on the wire or in config.** Endpoints set it: `/execute` → `BLOCKING`, `/validate` → `BLOCKING`, `/start` → `FIRE_AND_FORGET`. It stays a real StrEnum with exhaustive `match`/`case` (the standard applies — this axis is genuinely closed; a future `STREAMING` would be core's call, never a plugin's).+- **D-D — Delivery is a parameter to dispatch, not a registry key.** `OrchestratorProtocol.run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)`. Each orchestrator honors `delivery` per its nature. This collapses the two Temporal orchestrators into one (`run` branches on `delivery`: `temporal_pipe_run.run(...)` for blocking vs `.start(...)` for f&f) registered **once** under `"temporal"`, and collapses the `/validate` double-registration to a single `"temporal"` validator.+- **D-E — `/start` is HONEST, not degrading.** An orchestrator advertises `supports_fire_and_forget: bool` (`direct`=False, `temporal`=True, `mistralai-workflows`=False). `/start` checks the *resolved* mode's orchestrator and, when it can't do genuine async, returns a 4xx — *"this deployment has no async-capable orchestration; use /execute"* — instead of silently running blocking and acking. (Replaces today's `_async_start_mode` identity-arm degrade.) **Consequence to verify (§8):** on the OSS base (`orchestration_mode = "direct"`) every `/start` now 4xxs — confirm the MTHDS-Protocol/conformance position on `/start` over a non-async backend before committing.+- **D-F — Generic `MissingOrchestratorError` / `MissingBundleValidatorError` message.** *"No orchestrator registered for mode '{mode}'; is its plugin installed?"* — fully decoupled. Drop the per-mode `match`-on-enum install hints (`requires_pipelex_temporal` etc.). Core no longer names `temporal`/`mistral` in its error text.+- **D-G — Rename the Mistral token `"mistral_native"` → `"mistralai-workflows"`** for consistency with the package name (`pipelex-mistralai-workflows`). Mistral is a deferred/secondary plugin; this is just the token string + any doc reference.+- **D-H — Rename the wire field + config key `execution_mode` → `orchestration_mode`** (and `allow_request_execution_mode_override` → `allow_request_orchestration_mode_override`). Breaking wire/config change — fine per the workspace "no backward compatibility" principle; note in changelogs.++---++## 2. Current state (anchors verified 2026-06-22)++**Core (`_plugins/pipelex/`):**++- `runtime_bridge/execution_mode.py` — `PipelexExecutionMode(StrEnum)`: the 4 members + three properties to delete: `requires_pipelex_temporal` (`30`), `requires_mistral_workflows_extra` (`38`), `is_fire_and_forget` (`46`).+- `plugins/orchestrator_registry.py` — `OrchestratorProtocol.run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput` (`21`); `OrchestratorRegistry(orchestrators: dict[PipelexExecutionMode, OrchestratorProtocol])` with `get_optional(*, mode)` / `has(*, mode)` / `modes` (`24-42`).+- `plugins/bundle_validator_registry.py` — the `/validate` twin (`BundleValidatorProtocol`, `BundleValidatorRegistry`), same `dict[PipelexExecutionMode, …]` keying (per the validate plan's V0 as-built).+- `plugins/registrar.py` — stores `orchestrators` / `bundle_validators` typed `dict[PipelexExecutionMode, …]` (`111-112`) + the `_*_sources` mirrors (`122-123`); `add_orchestrator(*, mode, …)` (`167`) and `add_bundle_validator(*, mode, …)` (`179`).+- `runtime_bridge/exceptions.py` — `MissingOrchestratorError._build_message` (`27-40`) and `MissingBundleValidatorError._build_message` (`61-74`) both `match` on enum members for per-mode install hints (→ generic, D-F).+- `runtime_bridge/direct_orchestrator.py` — `DirectOrchestrator.run(*, pipe_job, delivery_assignment)` (`29`), returns `serialize_completed_output(pipe_output, workflow_id=None)` (`46-49`). Add `supports_fire_and_forget = False` + the `delivery` param (ignored — in-process always blocks).+- `plugins/direct/direct_plugin.py` — `register` calls `add_orchestrator(mode=DIRECT, …)` + `add_bundle_validator(mode=DIRECT, …)` (`20-22`). `DIRECT` becomes the `"direct"` string constant.+- `runtime_bridge/bridge.py` — `run_pipe_via_bridge` (`51`): `is_direct = input_payload.execution_mode is PipelexExecutionMode.DIRECT` (`85`); dispatch `get_orchestrator_registry().get_optional(mode=input_payload.execution_mode)` → `orchestrator.run(pipe_job, delivery_assignment)` (`92-95`); `_validate_input` checks `execution_mode is TEMPORAL_FIRE_AND_FORGET` for the delivery-target requirement (`164`). This is the **Tier-3 bridge** for *other* host runtimes (Mistral Workflows, raw Temporal) — pipelex-api's routes do **not** go through it (they hit the registry directly), but it must split consistently and pass `delivery`.+- `runtime_bridge/payloads.py` — `PipelexPipeRunInput.execution_mode: PipelexExecutionMode = DIRECT` (`30`). Splits into `orchestration_mode: str = "direct"` + `delivery: DeliveryMode = BLOCKING`. `PipelexPipeRunOutput` (`34-52`) already carries `workflow_id` / `is_completed` and needs no change.++**Temporal (`pipelex-temporal/pipelex_temporal/`):**++- `temporal_orchestrators.py` — `TemporalBlockingOrchestrator.run` calls `temporal_pipe_run.run(...)` (await completion, `58`) and reports `make_workflow_id(...)` (`69`); `TemporalFireAndForgetOrchestrator.run` calls `temporal_pipe_run.start(...)` (return immediately, `91`) and builds an `is_completed=False` output (`96-103`). **The only difference is `run` vs `start`** → one `TemporalOrchestrator` whose `run` branches on `delivery`. Both already guard the `pipelex[temporal]` extra via `_require_temporal_extra` (`32-40`).+- `temporal_plugin.py` — `register` (`127`): two `add_orchestrator` calls under the two temporal modes (`128-129`), two `add_bundle_validator` calls under both temporal modes (`136-138`), the F3 mapper (`145`, unchanged), and the `boot_orchestrator == self.name` slot-claims (`152-157`, unchanged — `boot_orchestrator` is the *separate* hub-slot gate, orthogonal to the orchestration token even though both read `"temporal"`).+- `temporal_bundle_validator.py` — `TemporalBundleValidator` (validate is inherently blocking; register once under `"temporal"`).++**API (`pipelex-api/api/`):**++- `api_config.py` — `ApiConfig.execution_mode: PipelexExecutionMode` + `allow_request_execution_mode_override: bool` (`51-52`); the `_reject_fire_and_forget_default` field-validator to **delete** (`54-74`); `resolve_execution_mode(requested, *, config)` (`103-121`) → rename `resolve_orchestration_mode`, logic unchanged (string compare; 403 on forbidden override); `api.toml` ships `execution_mode = "direct"`.+- `schemas/models.py` — `PipelineApiExtras.execution_mode: PipelexExecutionMode | None` (`153`); `PipelexApiExecuteRequest.execution_mode` (`204`); `_EXECUTION_MODE_DESCRIPTION` (`110-115`, lists the four enum values → open-token wording). Rename field → `orchestration_mode`.+- `routes/pipelex/pipeline.py` — `_async_start_mode` (`109-134`, **delete**); `ApiRunner.execute` resolves + rejects f&f (`208-214`, drop the rejection) and injects `_OrchestratorPipeRun` (`222`); `ApiRunner.start` calls `_async_start_mode(resolve_…(…))` (`283`, → resolve + capability-check); `_OrchestratorPipeRun` / `_pipe_output_from_run_output` (`73-153`, thread `delivery`); `validate_verdict` (`343-390`, rename the mode param). Routes `execute` (`530`) / `start` (`572`) thread the renamed extra.+- `error_types.py` — `EXECUTION_MODE_OVERRIDE_FORBIDDEN` (`23`, rename) and `FIRE_AND_FORGET_NOT_SUPPORTED` (`32`, **delete**); add the honest-`/start` rejection type (D-E).+- `routes/pipelex/validate.py` — threads `execution_mode` into `validate_verdict` (rename to `orchestration_mode`).+- The committed OpenAPI artifact (`docs/openapi/pipelex-api.openapi.yaml`) re-exports via `make openapi-export`; drift-gated by `make openapi-check`.++> **Mechanical-usage inventory** (every stray `PipelexExecutionMode` / `execution_mode` / mode-literal site across the three repos) is being swept and is appended in §9 — run that grep before declaring a phase done.++---++## 3. The design (locked)++### 3.1 Core types (`_plugins/pipelex/runtime_bridge/`)++- **`orchestration_mode.py`** (replaces `execution_mode.py`):+ - `OrchestrationMode: TypeAlias = str` — a *semantic* alias documenting intent at registry/protocol signatures (`dict[OrchestrationMode, …]`), assignment-compatible with plain `str` so plugins pass raw strings with no casts. (Chose the alias over `NewType` deliberately: a `NewType` forces `OrchestrationMode("temporal")` casts at every plugin boundary for zero validation benefit — the registry is the validator. See §8.)+ - `DIRECT_ORCHESTRATION_MODE: Final[str] = "direct"` — core's one built-in token, referenced by core code instead of the literal.+- **`delivery_mode.py`** — `class DeliveryMode(StrEnum): BLOCKING = "blocking"; FIRE_AND_FORGET = "fire_and_forget"`. Closed; exhaustive `match`/`case`; no default arm.+- **`OrchestratorProtocol`** (`plugins/orchestrator_registry.py`):+ - `async def run(self, *, pipe_job, delivery_assignment, delivery: DeliveryMode) -> PipelexPipeRunOutput`+ - `supports_fire_and_forget: bool` — capability the runner reads *before* dispatch (D-E). A `@property` or a plain class attribute; the protocol declares it as a read-only attribute.+- **`OrchestratorRegistry`** / **`BundleValidatorRegistry`** — re-key `dict[OrchestrationMode, …]` (i.e. `dict[str, …]`); `get_optional(*, mode: OrchestrationMode)` / `has`. `BundleValidatorProtocol.validate_bundles` stays blocking — pass no `delivery` (or `DeliveryMode.BLOCKING` if uniformity reads better; validate has no async variant).+- **`PluginRegistrar`** — `orchestrators` / `bundle_validators` + `_*_sources` become `dict[str, …]`; `add_orchestrator(*, mode: OrchestrationMode, …)` / `add_bundle_validator(*, mode: OrchestrationMode, …)`. The `_add` dup-guard is unchanged (works on any hashable key).+- **`Missing*Error`** — drop the `match`; generic message keyed on the string (D-F). Optionally keep a one-line special case for `"direct"` (*"DIRECT is core; this indicates a boot/discovery problem"*) via a plain `if mode == DIRECT_ORCHESTRATION_MODE:` — now a string compare, not an enum, so it does not offend the no-`==`-on-enums rule.+- **`payloads.py`** — `PipelexPipeRunInput`: `execution_mode` → `orchestration_mode: str = DIRECT_ORCHESTRATION_MODE` + `delivery: DeliveryMode = DeliveryMode.BLOCKING`.+- **`bridge.py`** — `is_direct = input_payload.orchestration_mode == DIRECT_ORCHESTRATION_MODE`; pass `delivery=input_payload.delivery` into `orchestrator.run(...)`; `_validate_input` checks `input_payload.delivery is DeliveryMode.FIRE_AND_FORGET` for the delivery-target requirement.+- **`DirectOrchestrator`** — `supports_fire_and_forget = False`; `run(*, pipe_job, delivery_assignment, delivery)` accepts `delivery` for protocol uniformity and ignores it (in-process always blocks; `/start` never reaches it because the capability check rejects first — the bridge/Tier-3 path that *could* pass f&f to direct simply blocks, see §8).++### 3.2 Temporal plugin (`pipelex-temporal`)++- **One `TemporalOrchestrator`** (`temporal_orchestrators.py`) — `supports_fire_and_forget = True`; `run` branches:+ - `DeliveryMode.BLOCKING` → `temporal_pipe_run.run(...)` then `make_workflow_id(...)` (today's `TemporalBlockingOrchestrator` body).+ - `DeliveryMode.FIRE_AND_FORGET` → `temporal_pipe_run.start(...)` → `is_completed=False` output (today's `TemporalFireAndForgetOrchestrator` body).+ - Keep the `_require_temporal_extra` guard and the `WorkflowExecutionError` wrapping in both arms.+- **`TEMPORAL_ORCHESTRATION_MODE: Final = "temporal"`** defined in `pipelex-temporal` (the token lives with the orchestrator). Distinct constant from the plugin's `name` / `boot_orchestrator` value even though both equal `"temporal"` — they are different concepts (registry key vs hub-slot gate).+- **Register once** — `add_orchestrator(mode=TEMPORAL_ORCHESTRATION_MODE, orchestrator=TemporalOrchestrator())`; `add_bundle_validator(mode=TEMPORAL_ORCHESTRATION_MODE, validator=TemporalBundleValidator())`. The two-mode duplication is gone. F3 mapper + boot-slot claims unchanged.++### 3.3 API (`pipelex-api`)++- **`ApiConfig`** — `orchestration_mode: str` + `allow_request_orchestration_mode_override: bool`; **delete** `_reject_fire_and_forget_default` (no f&f token can be configured anymore). `resolve_orchestration_mode(requested: str | None, *, config) -> str` — same default/override/403 logic. `api.toml`: `orchestration_mode = "direct"`, `allow_request_orchestration_mode_override = false`.+- **Wire** (`schemas/models.py`) — `PipelineApiExtras.orchestration_mode: str | None`; `PipelexApiExecuteRequest.orchestration_mode`; description reworded for the open token (*"e.g. `direct`, `temporal`; plugin-provided modes also accepted; an unregistered mode is refused at dispatch"*; drop the `temporal_fire_and_forget` value — no such token exists).+- **`/execute`** — resolve `orchestration_mode`; **no f&f rejection** (delete the `is_fire_and_forget` branch + `FIRE_AND_FORGET_NOT_SUPPORTED`); dispatch via `_OrchestratorPipeRun(orchestrator, delivery=DeliveryMode.BLOCKING)`.+- **`/start`** — resolve `orchestration_mode`; get orchestrator; **capability check (D-E):** `if not orchestrator.supports_fire_and_forget: raise_<4xx>(…, error_type=START_REQUIRES_ASYNC_ORCHESTRATION)`; dispatch with `delivery=DeliveryMode.FIRE_AND_FORGET`. **Delete `_async_start_mode`.**+- **`/validate`** — `validate_verdict(*, …, requested_orchestration_mode)`; resolve + dispatch through the validator registry (blocking). Verdict-as-value unchanged.+- **`error_types.py`** — rename `EXECUTION_MODE_OVERRIDE_FORBIDDEN` → `ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN`; **delete** `FIRE_AND_FORGET_NOT_SUPPORTED`; **add** `START_REQUIRES_ASYNC_ORCHESTRATION` (D-E). Status code: §8.+- **OpenAPI** — `make openapi-export`; the request schemas now advertise `orchestration_mode`.++---++## 4. Phased implementation + checkpoints++Each phase: own repo, own branch, own commit, gates green, clean-context `/code-review` (hand the reviewer only the diff/SHA, never this plan). Ordering is load-bearing — core must land before its consumers.++### Phase 1 — core split (`pipelex`, `_plugins/`) ✅ DONE (`bd4e52827`)++Add `OrchestrationMode` (open `str` alias + `DIRECT_ORCHESTRATION_MODE`) and `DeliveryMode`; re-key both registries + the registrar stores on `str`; add `delivery` + `supports_fire_and_forget` to `OrchestratorProtocol`; update `DirectOrchestrator`; generic `Missing*Error`; split `PipelexPipeRunInput`; update `bridge.py` dispatch + `_validate_input`; delete `PipelexExecutionMode` and its three properties. Sweep §9. Gates: `make agent-check`, `make tb` (boot builds the re-keyed registries), `make agent-test`.++> **🛑 Checkpoint 1** — core green; registries keyed by string; boot still builds them; no orchestrator named in core error text. Capture an as-built (final type names/locations, the `OrchestrationMode` alias decision, the `supports_fire_and_forget` shape). Clean-context `/code-review` on the core diff.++### Phase 2 — Temporal collapse (`pipelex-temporal`) · **MAJOR** ✅ DONE (`01d7dc3`)++Collapse the two orchestrators into `TemporalOrchestrator` (delivery-branch + `supports_fire_and_forget=True`); define `TEMPORAL_ORCHESTRATION_MODE`; register orchestrator + validator **once** under `"temporal"`; F3 mapper + boot-slot claims unchanged. Update the in-memory integration tests to exercise both delivery values through the single orchestrator. Gates: `make agent-check`, `make agent-test` against the Phase 1 core.++> **🛑 Checkpoint 2** — the seam has a real consumer through the new shape: one `"temporal"` arm serving both deliveries; the base still has no idea Temporal exists. As-built + clean-context `/code-review` on both diffs.++### Phase 3 — API (`pipelex-api`) · **THE gate**++`ApiConfig` rename + drop the f&f validator + `resolve_orchestration_mode`; wire-model rename + open-token description; routes — delete `_async_start_mode`, `/execute` drop f&f-reject (delivery=BLOCKING), `/start` honest capability-reject (delivery=FIRE_AND_FORGET), `/validate` param rename; `error_types` changes; `make openapi-export`; docs + changelog. Sweep §9. Gates: `make agent-check`, `make agent-test`, `make openapi-check`.++> **🛑 Checkpoint 3** — `/execute`, `/start`, `/validate` all take `orchestration_mode`; delivery is endpoint-set; `/start` honestly rejects non-async modes; all the conflation workarounds are gone; OpenAPI in sync. **Verify the `/start`-honesty conformance question (§8) here.** As-built + clean-context `/code-review` on the API diff.++### Phase 4 — Mistral plugin migration (`pipelex-mistralai-workflows`) + bookkeeping ✅ DONE (`0eb3878`)++> **✅ DONE — see "### Phase 4 — as-built" in §10 for the full record** (incl. the load-bearing deviation: the plugin pins `../_workflows`, not the split worktree, so the split was first merged into `feature/mistralai-2x-bump`). The anchors below were re-derived against the live tip and were accurate; the migration matched this scope.++**⚠️ Scope correction (2026-06-22, from the Phases 1–2 `/code-review`): the Mistral work is NOT "token string + doc refs only" — that earlier estimate was wrong.** The external `pipelex-mistralai-workflows` plugin pins core editable, so it is already **red against Phase 1 core**: it carries live *code* that breaks under the `execution_mode → orchestration_mode` rename (`extra="forbid"`) + the `PipelexExecutionMode` deletion. This is the same Phase-2 shape the Temporal plugin took — a real code migration, not a string swap. **Anchors verified against tip `e508e73` (re-derive before starting — the tip will have moved):**++- `mistral_plugin.py:17` + `streaming.py:28` — `from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode` → **`ImportError`** (module deleted). Re-point to `orchestration_mode` / `delivery_mode`.+- `mistral_plugin.py:31` — `add_orchestrator(mode=PipelexExecutionMode.MISTRAL_NATIVE, …)` registers under the **deleted enum member** → migrate to `add_orchestrator(mode="mistralai-workflows", …)` (the D-G token; the `"mistral_native"` literal lives at `mistral_plugin.py:21`).+- `mistral_orchestrator.py` — `MistralNativeOrchestrator.run` must adopt the new `OrchestratorProtocol.run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)` signature and declare `supports_fire_and_forget = False` (Mistral-native is blocking-only).+- `streaming.py:77,117,121` — its own `execution_mode: str` wire field + `input_payload.execution_mode` read + `is PipelexExecutionMode.DIRECT` compare → rename to `orchestration_mode`, drop the enum compare (string-compare `== DIRECT_ORCHESTRATION_MODE` or branch on `delivery`).+- `primitives/__init__.py`, `primitives/pipe_run.py`, `primitives/scoped_library.py` — docstring/comment/`mistral_native_…` string refs (the genuinely doc-only part).++**Then the bookkeeping:** update `_plugins/TODOS.md` (Phase C as-built + follow-on section), mark the two superseded sibling plans, `pipelex-api` `CHANGELOG.md`, and `docs/specs/` / `conformance/` if any heading or test asserts mode values (run `make check-spec-links`).++---++## 5. Cross-repo state, pins, gates++- **Verify state FIRST (cold start).** Before cutting any branch, run the effort's orientation ritual — [`wip/plugins/resume-and-verify-state.md`](resume-and-verify-state.md) — and confirm each repo's working tree is clean and sits on the tip `_plugins/TODOS.md` "Current branches" names. This plan was written against those tips; if they have moved, re-derive the §2 / §9 anchors against the new tip rather than trusting the line numbers.+- **Branches:** cut fresh per phase off the verified current tips (`_plugins` core worktree; `pipelex-temporal`; `pipelex-api` off the `feature/Execute-per-request-mode` tip — this work stacks on the landed follow-ons it supersedes).+- **Pins:** `pipelex-api` and `pipelex-temporal` both pin `pipelex = { path = "../_plugins", editable = true }`, so a core edit is live in both immediately. `pipelex-api` does **not** depend on `pipelex-temporal` (Phase C dropped it) — test the API split with a stub orchestrator/validator registered for a non-`direct` test token; never import `temporalio` in `pipelex-api`. For a `pipelex-api` PR, the editable pin breaks CI — pin core to a git+https SHA + relock, flip back at release (the PR #27 `c60be1f` gotcha).+- **Gates:** core → `make agent-check` / `make tb` / `make agent-test`; temporal + api → `make agent-check` / `make agent-test`; api also `make openapi-check`; conformance → `make check-spec-links` if a spec heading is touched.++---++## 6. Spec / conformance++- **No verdict/result wire contract changes** — `/validate`'s 200-always + `is_valid` discriminant + `validation_errors[]`, and the `/execute` / `/start` result shapes, are all backend-independent and untouched. The only wire change is the **request** field rename `execution_mode → orchestration_mode` (a pipelex-api extension field, not protocol-core).+- **The one real conformance question is D-E's `/start` honesty** (§8): if the MTHDS-Protocol conformance suite asserts `/start` returns a `202` on the reference implementation's default (direct) config, honest-reject breaks it. Resolve before Phase 3 lands.+- If any `docs/specs/` heading documents the mode values or the per-endpoint delivery behavior, update it + its `> Verified by:` link and run `make check-spec-links`.++---++## 7. Testing++- **Core (Phase 1):** registry/registrar keyed by string (register + get + dup under arbitrary tokens incl. a synthetic `"acme"`); `MissingOrchestratorError` generic message for an unregistered token + the `"direct"` boot-bug special case; `DeliveryMode` exhaustive; `OrchestratorProtocol` round-trip with `delivery` + `supports_fire_and_forget`; `DirectOrchestrator.supports_fire_and_forget is False`; `PipelexPipeRunInput` defaults (`orchestration_mode="direct"`, `delivery=BLOCKING`); bridge `_validate_input` requires a delivery target iff `delivery is FIRE_AND_FORGET`. `make tb` green (boot builds the re-keyed registries).+- **Temporal (Phase 2):** in-memory-Temporal integration — the single `TemporalOrchestrator` returns a completed output for `delivery=BLOCKING` and an `is_completed=False` + `workflow_id` for `delivery=FIRE_AND_FORGET`; one registration serves both; validator registered once.+- **API (Phase 3, no temporal import):** `/execute` dispatches blocking with no f&f branch; `/start` honestly 4xxs when the resolved mode's stub orchestrator has `supports_fire_and_forget=False`, and acks when it's `True`; override policy on string tokens (default honored, different+allowed honored, different+forbidden → 403); an unregistered requested token → `MissingOrchestratorError` at dispatch; `/validate` unchanged verdict mapping; OpenAPI advertises `orchestration_mode`.++---++## 8. Open sub-decisions / risks for the implementing session++- **[RISK — ✅ RESOLVED 2026-06-22, before Phase 3] `/start` honesty vs MTHDS-Protocol conformance.** D-E makes the runner's `/start` 4xx on a direct-only deployment (the OSS default). **Investigated and cleared:** (1) the protocol spec `docs/specs/pipelex-mthds-protocol.md` (§"operations") explicitly **defers `execute`/`start` artifacts to pipelex-api's OpenAPI + run-lifecycle docs** — it details only the `validate` and `models` artifacts, so the protocol mandates **no** `/start` status code; (2) the `conformance/tests/pipelex_api/` suite exercises only `validate` / `models` / `version` — **no test POSTs to the runner `/v1/start`**; every conformance `202` (`conformance/tests/pipelex_platform/test_runs_results.py`, `docs/specs/pipelex-platform-api.md`) is the **platform's run-results polling** (a RUNNING run → 202), a different endpoint untouched by the runner's dispatch semantics; (3) the platform's `/v1/start` (wallet-gated front door) is distinct from the runner's `/start`, and the hosted platform runs the `temporal` orchestration mode (async-capable → no 4xx). **Net: honest 4xx on a direct-only base breaks no conformance test and no protocol clause.** Proceed with D-E; status code **400** (§ next bullet). The only consumer to update is pipelex-api's own `/start` tests + its OpenAPI/docs (owned by Phase 3).+- **Honest-`/start` status code.** `400` (symmetric with the deleted `FIRE_AND_FORGET_NOT_SUPPORTED`), `409 Conflict` (request conflicts with deployment capability), or `422`. Recommend `400` with a crisp message; flag `409` as the more semantically precise alternative.+- **`OrchestrationMode` shape.** Recommended: `TypeAlias = str` + a `Final` `"direct"` constant (open, cast-free for plugins). Alternative: a `NewType` (stronger signatures, but forces casts at every plugin boundary for no validation gain since the registry validates). Decide in Phase 1; record in the as-built.+- **`supports_fire_and_forget` location.** On the orchestrator (recommended — it owns its capability) vs registry-level metadata. Keep it on the protocol unless a second consumer needs it elsewhere.+- **Unknown requested token validation timing.** Let dispatch raise `MissingOrchestratorError` (recommended — single validation point, keeps `resolve_orchestration_mode` decoupled from the hub) vs an early `get_orchestrator_registry().has(...)` check in resolve (earlier 4xx but couples config to the hub). Default: dispatch-time.+- **[RESOLVED 2026-06-22 — bridge-level guard added] Tier-3 bridge + incompatible (mode, delivery).** A raw bridge caller could pass `delivery=FIRE_AND_FORGET` to `direct`. The original disposition was "let it block — `DirectOrchestrator` blocks regardless." **The combined Phases 1–2 `/code-review` found that understated the harm:** with a valid `delivery_assignment_dump` the request passes `_validate_input`'s delivery-target gate, then `DirectOrchestrator.run` ignores `delivery`, runs to completion, and returns `is_completed=True` — so the caller **blocks** for the full run AND its delivery target is **never invoked** while completion is **falsely acked**. **Decided: add the bridge-level capability guard** (the Tier-3 counterpart of `/start`'s D-E check), not "let it block." The bridge now resolves the orchestrator up front and `_validate_input` rejects `FIRE_AND_FORGET` on any orchestrator with `supports_fire_and_forget = False` (raises `PipelexBridgeDispatchError`), regardless of whether a delivery target was supplied. As-landed shape + tests: see the post-checkpoint hardening note in §10. (This is pure core/`_plugins` work; pipelex-api routes hit the registry directly, not the bridge, so Phase 3 is unaffected.)+- **`boot_orchestrator` vs the `"temporal"` token.** Both strings equal `"temporal"` but are distinct concepts (hub-slot boot gate vs orchestrator-registry key). Keep separate named constants; do not unify them just because the values coincide today.++---++## 9. Mechanical-usage inventory++> Swept across `_plugins/`, `pipelex-temporal/`, `pipelex-api/`. Actionable per-repo, keyed to the §3 decisions. **Re-run the grep at each phase boundary — do not trust a stale list:** `grep -rn 'PipelexExecutionMode\|execution_mode\|is_fire_and_forget\|requires_pipelex_temporal\|requires_mistral_workflows\|temporal_blocking\|temporal_fire_and_forget\|mistral_native' <repo>`.++### Core (`_plugins/pipelex/`)++**Source — split / rename / delete:**+- `runtime_bridge/execution_mode.py` → **replace** with `orchestration_mode.py` (`OrchestrationMode` alias + `DIRECT_ORCHESTRATION_MODE`) and `delivery_mode.py` (`DeliveryMode`). Delete the enum + its 3 properties.+- `runtime_bridge/payloads.py:16,30` — `PipelexPipeRunInput.execution_mode` → `orchestration_mode: str` + `delivery: DeliveryMode`.+- `runtime_bridge/bridge.py:36,85,92,94,164` — import; `is_direct` string-compare; registry dispatch; `MissingOrchestratorError`; `_validate_input` f&f-target check → `delivery is FIRE_AND_FORGET`; pass `delivery` to `run`.+- `runtime_bridge/exceptions.py:2,22-40,56-74` — `Missing{Orchestrator,BundleValidator}Error` — drop the per-mode `match`; generic message (D-F).+- `plugins/orchestrator_registry.py:3,21,31-42` — protocol `run(*, …, delivery)` + `supports_fire_and_forget`; registry `dict[str, …]`.+- `plugins/bundle_validator_registry.py:33,77-89` — registry `dict[str, …]` (validate stays blocking).+- `plugins/registrar.py:19,111-123,167-189` — `orchestrators`/`bundle_validators` + `_*_sources` → `dict[str, …]`; `add_orchestrator`/`add_bundle_validator` mode param typed `OrchestrationMode`.+- `runtime_bridge/direct_orchestrator.py:29,46-49` — `supports_fire_and_forget = False`; accept+ignore `delivery`.+- `plugins/direct/direct_plugin.py:5,20-22` — register under `DIRECT_ORCHESTRATION_MODE`.++**Tests — update/retarget:** `tests/unit/pipelex/runtime_bridge/test_execution_mode.py` (→ `test_orchestration_mode.py` + new `test_delivery_mode.py`; the property tests die with the properties), `test_input_models.py`, `test_dispatch.py`, `test_validation.py` (the f&f-target requirement now keys on `delivery`), `test_orchestrator_dispatch.py`, `test_direct_router_scoping.py`, `test_missing_bundle_validator_error.py` (parametrized over per-mode install hints → assert the generic message + the `"direct"` boot-bug case), `test_exceptions_disclosure.py`, `test_trace_context_contract.py`, `tests/unit/pipelex/plugins/test_bundle_validator_registry.py`, `test_plugin_discovery.py`, `tests/integration/pipelex/runtime_bridge/test_bridge_direct.py`.++**Docs:** the `execution_mode.py` module docstring (moves to the new modules); `TODOS.md` two-axis narrative (the `boot_orchestrator` vs `execution_mode` table, the f&f-derivation notes) — update at Phase 4 bookkeeping.++### Temporal (`pipelex-temporal/pipelex_temporal/`)++**Source:** `temporal_plugin.py:29,128-138` — collapse the two `add_orchestrator` + two `add_bundle_validator` calls to one each under `TEMPORAL_ORCHESTRATION_MODE`; `temporal_orchestrators.py` — merge `TemporalBlocking`/`TemporalFireAndForget` into one `TemporalOrchestrator` (`run` branches on `delivery`, `supports_fire_and_forget=True`); `temporal_bundle_validator.py` — unchanged impl, registered once; `temporal_cli.py` — docstring mention of `execution_mode`. `boot_orchestrator == self.name` gate (`:152`) is **unchanged** (separate concept — leave it).++**Tests:** `tests/unit/pipelex_temporal/test_bridge_temporal_dispatch.py` (both modes → one orchestrator + `delivery`), `test_trace_context_temporal.py`, `test_temporal_blocking_workflow_id.py`; the in-memory validator integration test (single registration).++### API (`pipelex-api/api/`)++**Source:** `api_config.py:27,51-74,103-122` — field rename + **delete** `_reject_fire_and_forget_default` + `.is_fire_and_forget` use; `resolve_execution_mode` → `resolve_orchestration_mode`. `schemas/models.py:19,110-115,153,204` — `PipelineApiExtras`/`PipelexApiExecuteRequest` field rename + open-token description. `routes/pipelex/pipeline.py:22,109-135,183,208,247,283,349,375` — **delete** `_async_start_mode`; `/execute` drop f&f-reject; `/start` capability-reject; `_OrchestratorPipeRun`/`_pipe_output_from_run_output` thread `delivery`; `validate_verdict` param rename. `routes/pipelex/validate.py:8,67-75,188` — `ValidateRequest.execution_mode` → `orchestration_mode` (confirm the model's home — route vs `models.py`). `error_types.py:13,23,32` — rename `EXECUTION_MODE_OVERRIDE_FORBIDDEN`; delete `FIRE_AND_FORGET_NOT_SUPPORTED`; add `START_REQUIRES_ASYNC_ORCHESTRATION`.++**Config:** `api/api.toml:15,19` — `execution_mode = "direct"` → `orchestration_mode = "direct"`; `allow_request_execution_mode_override` → `allow_request_orchestration_mode_override`; update the file comments.++**Tests:** `tests/unit/test_api_config.py` (drop the f&f-rejection test; rename), `test_execute_dispatch.py` (drop the 400-f&f case), `test_start_dispatch.py` (was "derives f&f variant" → now "honest-reject when not async-capable" + ack when capable), `test_validate_dispatch.py`, `test_protocol_conformance.py`.++**Docs + OpenAPI:** `api_config.py` / `api.toml` / `models.py` docstrings; `make openapi-export` regenerates `docs/openapi/pipelex-api.openapi.yaml`; `CHANGELOG.md`.++### Cross-cutting++- **`boot_orchestrator` is untouched** — it is the separate hub-slot boot gate (`registrar.config.plugins.boot_orchestrator`, the `temporal_plugin.py:152` claim guard), orthogonal to the orchestration token. Do not fold the two together even though both read `"temporal"`.+- **Fire-and-forget delivery-target validation** (`bridge.py:164-170`) moves from `execution_mode is TEMPORAL_FIRE_AND_FORGET` to `delivery is DeliveryMode.FIRE_AND_FORGET` — the only place that check should live post-split (the API config f&f-rejection and the `/execute` f&f-400 both delete).++---++## 10. Checkpoint log / as-built (append per phase — keep cold-start-ready)++### ⏸️ COLD-START RESUME — paused before Phase 3 (2026-06-22) — ✅ SUPERSEDED (Phase 3 DONE)++> **✅ This section is historical — Phase 3 is now landed (`pipelex-api` `dd84a9a`).** Read **"### Phase 3 — as-built (API split) — Checkpoint 3 cleared"** further down in §10 for the as-built + the one carry-forward item. The locked decisions below were applied as-is. The next cold start is **Phase 4** (Mistral plugin migration + bookkeeping); see the head **Status** line and the revised §4 Phase 4.++**Read this first on resume.** Phases 1 + 2 are landed, reviewed clean, and committed. Phase 3 (`pipelex-api`, THE gate) is **not started** — its branch is cut and the tree is clean. Everything Phase 3 needs is in **§3.3** (the locked design) and **§9 → API** (the mechanical inventory); the decisions below are already settled — do not re-litigate.++**Exact state — verify with `git -C <repo> log --oneline -1` before touching anything:**++- `_plugins` (core): branch `feature/orchestration-mode-delivery-split`. As of this tracker update the tip is **`5b70c2699`** (the cold-start checkpoint, successor of the Phase-1 tracker commit `919c09348`); **this commit lands the post-checkpoint bridge-guard change on top** (always verify the real tip with `git log` — it moves with each tracker update). (Phase 1 = `bd4e52827`; docs = `9c7a7785d`.) The bridge-guard change adds the Tier-3 fire-and-forget capability guard to `runtime_bridge/bridge.py` (+ tests + the `orchestrator-plugins.md` worked-example fix) — see the post-checkpoint hardening note just below.+- `pipelex-temporal`: branch `feature/orchestration-mode-delivery-split`, tip **`01d7dc3`**, clean. (Phase 2.)+- `pipelex-api`: branch `feature/orchestration-mode-delivery-split`, tip **`06cd5e6`** (= the cut point; **no Phase 3 code**), clean. Two partial edits (`error_types.py`, `api.toml`) were made then **reverted** — start Phase 3 from scratch against §3.3/§9.++**Cross-repo wiring (already true):** both `pipelex-temporal` and `pipelex-api` pin core **editable** (`../_plugins`), so the Phase 1 core is already live in both. `pipelex-api` does **not** depend on `pipelex-temporal` — test the API split with a **stub** orchestrator/validator registered under a non-`direct` test token (one with `supports_fire_and_forget=True` and one `False`); never `import temporalio` in `pipelex-api`.++**§8 RESOLVED — honest `/start` is safe.** The MTHDS-Protocol spec defers `execute`/`start` to pipelex-api's OpenAPI/docs; no conformance test POSTs the runner `/v1/start`; the hosted platform runs the async-capable `temporal` mode. So the runner's `/start` may honestly **400** on a direct-only base. (Details in §8.)++**Locked Phase 3 design decisions (apply as-is):**++- **Honest `/start` status code = `400`** via `raise_bad_request(message=…, error_type=ErrorType.START_REQUIRES_ASYNC_ORCHESTRATION)`. (409 was the flagged alternative; 400 chosen per §8.)+- **Restructure `ApiRunner.start`** so the capability check runs **before** `pipeline_run_setup` (no library load on a doomed request): resolve `orchestration_mode` → `get_orchestrator_registry().get_optional(mode=…)` (→ `MissingOrchestratorError` if `None`) → **`if not orchestrator.supports_fire_and_forget: raise_bad_request(START_REQUIRES_ASYNC_ORCHESTRATION)`** → `pipeline_run_setup(...)` → build `delivery_assignment` → `orchestrator.run(pipe_job=…, delivery_assignment=…, delivery=DeliveryMode.FIRE_AND_FORGET)`. **Delete `_async_start_mode`** entirely.+- **`_OrchestratorPipeRun`** gains a `delivery: DeliveryMode` ctor field; its `run` passes `delivery=self._delivery` to `orchestrator.run`. `/execute` constructs it with `delivery=DeliveryMode.BLOCKING`.+- **`/execute`**: delete the `execution_mode.is_fire_and_forget` → 400 block (and `FIRE_AND_FORGET_NOT_SUPPORTED`); dispatch `BLOCKING`.+- **`api_config.py`**: `ApiConfig.orchestration_mode: str` + `allow_request_orchestration_mode_override: bool`; **delete** the `_reject_fire_and_forget_default` field-validator (and the now-unused `field_validator` import + the `PipelexExecutionMode` import). `resolve_execution_mode` → `resolve_orchestration_mode(requested: str | None, *, config) -> str` (same default/override/403 logic; `error_type=ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN`). `load_api_config`/`get_api_config` unchanged in shape.+- **`error_types.py`**: rename `EXECUTION_MODE_OVERRIDE_FORBIDDEN → ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN`; delete `FIRE_AND_FORGET_NOT_SUPPORTED`; add `START_REQUIRES_ASYNC_ORCHESTRATION` (400). _(These exact edits were drafted then reverted — redo them.)_+- **`schemas/models.py`**: rename `_EXECUTION_MODE_DESCRIPTION → _ORCHESTRATION_MODE_DESCRIPTION`, reworded for the open token (drop the `direct`/`temporal_blocking`/`temporal_fire_and_forget`/`mistral_native` value list; say "e.g. `direct`, `temporal`; plugin-provided tokens accepted; an unregistered token is refused at dispatch; delivery is endpoint-set, not requestable"). `PipelineApiExtras.execution_mode → orchestration_mode: str | None`; `PipelexApiExecuteRequest.execution_mode → orchestration_mode`. Drop the `PipelexExecutionMode` import.+- **`routes/pipelex/pipeline.py`**: `_validate_extras` reads the `orchestration_mode` body key; rename every `requested_execution_mode → requested_orchestration_mode` (params on `execute`/`start`/`validate_verdict` + the two route handlers); drop the `PipelexExecutionMode` import, add `from pipelex.runtime_bridge.delivery_mode import DeliveryMode`; update docstrings (the `/start` one: no `_async_start_mode`; direct now honest-4xx, not blocking-202). **Pre-existing, do NOT widen scope:** `_validate_extras` labels all three extras with `error_type=INVALID_CALLBACK_URLS` on a malformed value — leave it (flagged in the execute follow-on's deferred list).+- **`routes/pipelex/validate.py`**: `ValidateRequest.execution_mode → orchestration_mode` (reworded description); route passes `requested_orchestration_mode=request_data.orchestration_mode`; docstring "a forbidden `execution_mode` override is a 403" → `orchestration_mode`.+- **`api.toml`**: `orchestration_mode = "direct"` + `allow_request_orchestration_mode_override = false`; reworded comments (open token; delivery is endpoint-set; flavor sets `orchestration_mode = "temporal"`). _(drafted then reverted — redo.)_++**Gates + after-work (Phase 3):** `make fui && make c` (this repo's lint/type — Python 3.11+), `make tp` (or `make t`) for tests, **`make openapi-export`** then **`make openapi-check`** (the request schemas now advertise `orchestration_mode`), update docs (`docs/configuration.md` exec-mode section → orchestration/delivery; `docs/pipe-run.md` `/start` "blocks in-process on direct" note → now honest-4xx), and `CHANGELOG.md`. **Tests to update (§9):** `test_api_config.py` (drop the configured-f&f-rejection test; rename to orchestration), `test_execute_dispatch.py` (drop the 400-f&f case), `test_start_dispatch.py` (was "derives f&f variant" → now "honest-400 when the resolved mode's stub orchestrator has `supports_fire_and_forget=False`" + "acks when True"), `test_validate_dispatch.py`, `test_protocol_conformance.py`. **CI gotcha:** the editable `../_plugins` pin breaks any `pipelex-api` PR CI — pin core to a git+https SHA + relock `uv.lock` (flip back at release). Not needed for local gates.++**Then Checkpoint 3** (§4): as-built here + clean-context `/code-review` on the `pipelex-api` diff. **Phase 4** (Mistral plugin migration + bookkeeping) is still pending — and its Mistral scope is **bigger than first estimated, not smaller**. Phase 1 moved the core-doc Mistral prose + `pipe_classification.py` + the Mistral `execution-modes.md` to `"mistralai-workflows"`, but the external `pipelex-mistralai-workflows` plugin still has live `execution_mode` / `PipelexExecutionMode` **code** — a deleted-enum import, a registration under the deleted enum member, its own `execution_mode` wire field + an `is PipelexExecutionMode.DIRECT` compare, and an orchestrator on the old `run` signature — all of which must migrate to `orchestration_mode` + `delivery` + the new protocol (same shape as the Temporal collapse). See the revised §4 Phase 4 for the verified anchors. The trackers/superseded-plan markers + `make check-spec-links` remain.++---++### Post-checkpoint hardening — Phases 1–2 combined `/code-review` (2026-06-22)++After Checkpoints 1 + 2 cleared, a clean-context `/code-review` over "what we've done so far" (Phases 1–2) surfaced a reachable behavioral gap in landed core code and a stale doc. Both fixed on the `_plugins` branch (uncommitted at time of writing — **commit before Phase 3**).++**Fix 1 — Tier-3 fire-and-forget capability guard (resolves the §8 Tier-3 open question).** The bridge forwarded `delivery=FIRE_AND_FORGET` to any resolved orchestrator without consulting `supports_fire_and_forget`, so `direct` + fire-and-forget + a valid `delivery_assignment_dump` passed `_validate_input`'s target gate, ran blocking to completion, returned `is_completed=True`, and **never invoked the delivery target** — block + false-ack + silent drop. As-landed shape:++- `run_pipe_via_bridge` now **resolves the orchestrator up front** (before opening the scoped library / building the pipe job), then calls `_validate_input`. A doomed request is rejected before any library/job work — symmetric with `/start`'s "capability check before `pipeline_run_setup`" (D-E). (Note: `MissingOrchestratorError` for an unregistered token now raises *before* the pipe-job build, not after — a behavior-equivalent reorder.)+- `_validate_input(input_payload, *, orchestrator, delivery_assignment)` gained the `orchestrator` param and a **capability gate checked first**: `if input_payload.delivery is FIRE_AND_FORGET and not orchestrator.supports_fire_and_forget → raise PipelexBridgeDispatchError` ("Orchestration mode '…' cannot honor fire-and-forget delivery…"), *before* the existing delivery-target gate. A valid target does not rescue a blocking-only orchestrator (the mode, not the target, is the problem). This is the first read site of `supports_fire_and_forget` in landed code (it was write-only after Phases 1–2).+- Tests: `test_validation.py` splits the FF tests into capability-gate vs target-gate (target-gate tests use an async-capable stand-in to reach that gate) + a real-path `run_pipe_via_bridge` rejection on `direct`+FF+target; `test_orchestrator_dispatch.py` adds a seam test proving a blocking-only orchestrator's `run` is **never awaited** for an FF request (`blocking_only.calls == []` — directly refutes the block/false-ack failure); `test_trace_context_contract.py`'s non-direct nulling test was updated to register a stand-in `temporal` orchestrator (it previously leaned on `MissingOrchestratorError` as a post-build stop-point, which the reorder removed).++**Fix 2 — stale worked-example doc.** `docs/under-the-hood/orchestrator-plugins.md` "Worked example" still described the deleted `TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` (the Phase 1 as-built had deferred this section to Phase 2, but Phase 2 — a different repo — missed the core-repo doc). Rewritten to the single collapsed `TemporalOrchestrator` (one `"temporal"` registration, delivery-branched `run`, `supports_fire_and_forget = True`).++**Deferred / folded elsewhere:** the review's other substantial cluster — `pipelex-api` + `pipelex-mistralai-workflows` unmigrated against Phase 1 core — is expected load-bearing-ordering breakage owned by Phases 3/4. The one correction it forced: **Phase 4's Mistral scope was underestimated** (real code, not a token rename) — folded into §4 Phase 4 + the cold-start resume note above. **Gates:** `make agent-check` green (pyright 0 / mypy 0 / ruff / plxt / keyword-only); targeted `runtime_bridge` + `plugins` suites green.++---++### Phase 1 — as-built (core split) — Checkpoint 1 cleared++**Branch / commit.** `feature/orchestration-mode-delivery-split` in the `_plugins` core worktree, cut from `feature/Orchestrator-dispatched-validate` tip `fa1b46637`. One commit: **`bd4e52827`**. Nothing pushed.++**Open sub-decisions (§8) — resolved with the recommended defaults:**++- **`OrchestrationMode` shape** → `TypeAlias = str` + `DIRECT_ORCHESTRATION_MODE: Final[str] = "direct"` in the new `pipelex/runtime_bridge/orchestration_mode.py`. No `NewType` (cast-free for plugins; the registry is the validator). A `test_orchestration_mode.py` pins `OrchestrationMode is str` + the `"direct"` literal.+- **`supports_fire_and_forget` location** → on the orchestrator. Declared as a protocol **data member** (`supports_fire_and_forget: bool` on `OrchestratorProtocol`, still `@runtime_checkable`); `DirectOrchestrator.supports_fire_and_forget = False` (plain class attr).+- **Generic `Missing*Error`** → both `MissingOrchestratorError` and `MissingBundleValidatorError` delegate to a shared module-level `_build_missing_message(*, noun, mode)`; the `"direct"` boot-bug case is a plain `if mode == DIRECT_ORCHESTRATION_MODE:` **string** compare (not an enum `match`, so it does not offend the no-`==`-on-enums rule). Message: *"No {orchestrator|bundle validator} is registered for orchestration mode '{mode}'; is its plugin installed?"* / for direct: *"...for the core 'direct' mode; this indicates a boot or plugin-discovery problem."* Names no plugin.+- **Delivery is a `run` parameter; validate stays blocking** → `OrchestratorProtocol.run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)`. `BundleValidatorProtocol.validate_bundles` is unchanged (no `delivery` — validation is inherently blocking).+- **`_validate_input`** keys on `input_payload.delivery is DeliveryMode.FIRE_AND_FORGET`; message reworded "Fire-and-forget delivery requires …".++**Files.** New: `runtime_bridge/orchestration_mode.py`, `runtime_bridge/delivery_mode.py`. Deleted: `runtime_bridge/execution_mode.py`. Re-keyed `dict[str, …]`: `orchestrator_registry.py`, `bundle_validator_registry.py`, `registrar.py` (stores + `_*_sources` + `add_orchestrator`/`add_bundle_validator` mode params typed `OrchestrationMode`). Split `payloads.py` (`orchestration_mode: str` + `delivery: DeliveryMode`). `direct_plugin.py` registers under `DIRECT_ORCHESTRATION_MODE`. `direct_orchestrator.py` + `bridge.py` per §3.1.++**Deviations / extras beyond the literal §9 list (reconciled here):**++- **`plugins/exceptions.py`** — reworded `Duplicate{Orchestrator,BundleValidator}Error` prose "execution mode" → "orchestration mode" (their `mode` field was already `str`; only prose changed). Not in §9 but consistent with D-H.+- **`runtime_bridge/primitives/pipe_classification.py`** — two **docstring** prose mentions of the dead `MISTRAL_NATIVE` enum updated to `"mistralai-workflows"` (prose only; no token *registration* exists in core). The real D-G Mistral-token rename stays a Phase 4 item (deferred plugin).+- **`docs/under-the-hood/orchestrator-plugins.md`** — updated the **core-seam** sections (intro two-axis model, the seam-in-one-view pseudocode, the `OrchestratorProtocol` signature + `supports_fire_and_forget`, the `add_orchestrator(mode="temporal", …)` example, the generic-error section, the SPI table "Mode + delivery + errors" row pointing at the two new modules). **Deferred to Phase 2:** the "Worked example: the in-tree Temporal plugin" section still describes the two-orchestrator (`TemporalBlocking`/`TemporalFireAndForget`) shape — accurate until Phase 2 collapses them.++**Tests.** Deleted `test_execution_mode.py`; added `test_orchestration_mode.py` + `test_delivery_mode.py`. Updated `test_input_models.py`, `test_dispatch.py` (mistral→generic-hint), `test_validation.py` (delivery-centric f&f), `test_orchestrator_dispatch.py` (`_FakeOrchestrator` gains `supports_fire_and_forget` + `delivery` param; miss test asserts the generic message over tokens incl. a synthetic `"acme"`), `test_direct_router_scoping.py`, `test_missing_bundle_validator_error.py` (generic + `"direct"` boot-bug), `test_exceptions_disclosure.py` (generic hint survives STRICT), `test_trace_context_contract.py`, `test_bundle_validator_registry.py`, `test_plugin_discovery.py`, `test_bridge_direct.py`.++**Gates.** `make agent-check` (ruff/plxt/pyright 0/mypy 0/keyword-only ✓), `make tb` (boot builds the re-keyed registries), `make agent-test` — all green.++**⚠️ Cold-start note for Phase 2.** `pipelex-temporal` pins core **editable** (`../_plugins`), so this commit is live in it immediately. Its `TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` do **not** yet implement the new `OrchestratorProtocol.run(*, …, delivery)` signature and lack `supports_fire_and_forget`, so `pipelex-temporal`'s suite is expected to fail against this core until Phase 2 adopts the new shape (collapse to one `TemporalOrchestrator`). This is the intended load-bearing ordering (core lands first), not a regression.++### Phase 2 — as-built (Temporal collapse) — Checkpoint 2++**Branch / commit.** `feature/orchestration-mode-delivery-split` in `pipelex-temporal`, cut from `feature/Orchestrator-dispatched-validate` tip `459e04d`. One commit: **`01d7dc3`**. Nothing pushed. (`pipelex-temporal` pins core editable, so it built against the Phase 1 core `bd4e52827` live.)++**The collapse.** `temporal_orchestrators.py` — the two classes (`TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator`) merged into one **`TemporalOrchestrator`** with `supports_fire_and_forget = True`. Its `run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)` runs the shared prelude (`_require_temporal_extra()`, lazy `WorkflowExecutionError` / `make_temporal_pipe_run` imports, `dispatch_errors` tuple, `temporal_pipe_run = make_temporal_pipe_run()`) then `match delivery:` — `BLOCKING` = the old blocking body (`temporal_pipe_run.run(...)` → `make_workflow_id(...)` → `serialize_completed_output`), `FIRE_AND_FORGET` = the old f&f body (`.start(...)` → `is_completed=False` output). Exhaustive match, no default arm; `WorkflowExecutionError` + `PIPE_DISPATCH_ERRORS` wrapping kept in both arms; extra-guard runs before any temporalio import. Dispatch-error messages reworded to name the *delivery* ("temporal blocking delivery" / "temporal fire-and-forget delivery").++**The token.** `TEMPORAL_ORCHESTRATION_MODE: Final[str] = "temporal"` defined in `temporal_orchestrators.py` (lives with the orchestrator). **Distinct constant** from `temporal_activation.TEMPORAL_PLUGIN_NAME` / `boot_orchestrator` (the hub-slot gate) — both equal `"temporal"` but are different concepts; left separate per §8. `temporal_bundle_validator.py` imports the token from `temporal_orchestrators` (import-light; no cycle — orchestrators never imports the validator) and its missing-extra guard now raises `MissingBundleValidatorError(mode=TEMPORAL_ORCHESTRATION_MODE)`.++**Register once.** `temporal_plugin.py`: one `add_orchestrator(mode=TEMPORAL_ORCHESTRATION_MODE, …)` + one `add_bundle_validator(mode=TEMPORAL_ORCHESTRATION_MODE, …)` (the two-mode duplication is gone). F3 mapper + `boot_orchestrator == self.name` slot-claims **unchanged**. Smoke-confirmed: `registrar.orchestrators` / `bundle_validators` keys == `['temporal']`.++**Import-light preserved.** `register` imports no `temporalio` (the import-light subprocess test still passes: blocks `temporalio`, runs `register` under both `boot_orchestrator` arms). The subprocess assertion changed `len(orchestrators) == 2 → == 1`.++**Tests.** `test_bridge_temporal_dispatch.py` (now `orchestration_mode="temporal"` + `delivery=FIRE_AND_FORGET` for the f&f case — exercises both delivery branches through the **single** real `TemporalOrchestrator.run`, with `make_temporal_pipe_run` mocked), `test_temporal_blocking_workflow_id.py`, `test_trace_context_temporal.py`, `test_temporal_plugin_http_error_mapper.py` (count 2→1). The validator in-memory integration test (`test_temporal_bundle_validator_in_memory.py`) constructs the validator directly — no mode reference, unchanged. Minor docstring fix in `temporal_cli.py` (`execution_mode` → `orchestration_mode`).++**Gates.** `pipelex-temporal` `make agent-check` (ruff/pyright 0/mypy 0) + `make agent-test` (full suite incl. `temporal`-marked in-memory integration; the markers exclude only inference/llm/img_gen/extract/search/pipelex_api, NOT temporal) — all green against the Phase 1 core. **Bonus:** this repo runs Python 3.11, so the run also validated the new core modules' 3.10+ compatibility (`TypeAlias`/`Final`/`StrEnum`-from-`pipelex.types`).++**Cold-start note for Phase 3.** `pipelex-api` is the next consumer (THE gate). It pins core editable too, so Phase 1 core is live in it; `pipelex-api` does NOT depend on `pipelex-temporal` (test the API split with a stub orchestrator/validator under a non-`direct` token; never import `temporalio`). Resolve the §8 `/start`-honesty conformance question before Phase 3 lands. CI gotcha: the editable `../_plugins` pin breaks `pipelex-api` CI — pin core to a git+https SHA + relock for any PR.++**Code-review triage (Checkpoint 2).** One clean-context `pr-review-toolkit:code-reviewer` (pointed only at `01d7dc3`). **Verdict: no material issues, correct + standards-compliant.** It verified against parent `459e04d` that the collapse is byte-faithful (BLOCKING still recomputes/reports `make_workflow_id(...)` not the bare run id; FIRE_AND_FORGET still returns `is_completed=False` + `.start()`'s workflow id; `WorkflowExecutionError` + `PIPE_DISPATCH_ERRORS` wrapping kept in both arms; extra-guard runs before any temporalio import), the `match delivery:` is exhaustive with no default arm and no `==`/`is` on enum members, and **empirically confirmed import-light** (a `sys.meta_path` blocker proves importing the plugin + `temporal_bundle_validator` importing `TEMPORAL_ORCHESTRATION_MODE` from `temporal_orchestrators` pulls no `temporalio`, no cycle). Two non-defect observations (cosmetic error-string rewording; the test's `expected_claims = 4` structural count is the legitimate test-assertion exception to the no-hardcoded-counts rule). **No applied or deferred items.**++---++**Code-review triage (Checkpoint 1).** One clean-context `pr-review-toolkit:code-reviewer` (no plan/rationale handed in; pointed only at `bd4e52827`). **Verdict: correct + standards-compliant, no bugs, tests not weakened.** It independently confirmed: the `is_direct` move to a `==` string compare and `_validate_input`'s move to `delivery is FIRE_AND_FORGET` are faithful (the latter a deliberate, intended broadening — *any* f&f run now needs a delivery target, not just the old Temporal-f&f member); defaults preserved (`direct` + `BLOCKING`, never f&f); the `"direct"` boot-bug `==` compare and generic messages are standards-clean; no `==`-on-enum-member introduced. **One actionable finding — APPLIED:** three user-facing Mistral-workflows doc pages still described the deleted `execution_mode` / `PipelexExecutionMode` model — `docs/distributed-execution/mistral-workflows/execution-modes.md` (rewritten to the two-axis `orchestration_mode` + `delivery` model, retitled "Orchestration & Delivery", generic error wording, `mistralai-workflows` token), `index.md`, `your-first-pipelex-workflow.md`, and `docs/reliability/durable-execution.md` (field/token refs + the now-stale link labels). My §9 grep had been `.py`-only — these `.md` pages slipped it; folded into this commit. **No deferred items.**++---++### Phase 3 — as-built (API split) — Checkpoint 3 cleared (2026-06-22)++**Branch / commit.** `feature/orchestration-mode-delivery-split` in `pipelex-api`, cut from `06cd5e6` (the cut point named in the cold-start note). One commit: **`dd84a9a`**. Nothing pushed. THE gate — the third and last code repo of Direction A — is landed; all the conflation workarounds (`_async_start_mode`, the `/execute` f&f-400, the `ApiConfig` f&f field-validator) are gone.++**Applied exactly per §3.3 / §9 / §10 locked decisions:**++- **`error_types.py`** — `EXECUTION_MODE_OVERRIDE_FORBIDDEN → ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN` (wire value `OrchestrationModeOverrideForbidden`); **deleted** `FIRE_AND_FORGET_NOT_SUPPORTED`; **added** `START_REQUIRES_ASYNC_ORCHESTRATION` (wire value `StartRequiresAsyncOrchestration`, status **400** per §8).+- **`api_config.py`** — `ApiConfig.orchestration_mode: str` + `allow_request_orchestration_mode_override: bool`; **deleted** `_reject_fire_and_forget_default` (+ the now-unused `field_validator` and `PipelexExecutionMode` imports). `resolve_execution_mode → resolve_orchestration_mode(requested: str | None, *, config) -> str` (string compare; 403 via `ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN`; an *unregistered* token is NOT rejected here — it surfaces at dispatch). Module/class docstrings reworded to the open-token + endpoint-set-delivery model.+- **`api.toml`** — `orchestration_mode = "direct"` + `allow_request_orchestration_mode_override = false`; comments reworded (open token; delivery endpoint-set; flavor sets `orchestration_mode = "temporal"`).+- **`schemas/models.py`** — `_EXECUTION_MODE_DESCRIPTION → _ORCHESTRATION_MODE_DESCRIPTION` (open-token wording, delivery-not-requestable note); `PipelineApiExtras.orchestration_mode: str | None`; `PipelexApiExecuteRequest.orchestration_mode: str | None`; dropped the `PipelexExecutionMode` import.+- **`routes/pipelex/pipeline.py`** — **deleted `_async_start_mode`**. `_OrchestratorPipeRun` gained a `delivery: DeliveryMode` ctor field threaded into `orchestrator.run`. `/execute` drops the f&f-reject and dispatches `delivery=BLOCKING`. **`/start` restructured** so resolve → `get_optional` → `MissingOrchestratorError` if None → **capability check `if not orchestrator.supports_fire_and_forget: raise_bad_request(START_REQUIRES_ASYNC_ORCHESTRATION)`** all run **before `pipeline_run_setup`** (no library load on a doomed request); dispatch `delivery=FIRE_AND_FORGET`. `validate_verdict` param `requested_execution_mode → requested_orchestration_mode`; `_validate_extras` reads the `orchestration_mode` body key; both route handlers thread `requested_orchestration_mode=extras.orchestration_mode`; docstrings updated (the `/start` one: honest-400, not blocking-202). Import swap: drop `PipelexExecutionMode`, add `DeliveryMode`; `resolve_execution_mode → resolve_orchestration_mode`.+- **`routes/pipelex/validate.py`** — `ValidateRequest.execution_mode → orchestration_mode: str | None` (open-token description); route passes `requested_orchestration_mode=request_data.orchestration_mode`; dropped the `PipelexExecutionMode` import; "no verdict" docstring `execution_mode → orchestration_mode`.+- **OpenAPI** — `make openapi-export` regenerated `docs/openapi/pipelex-api.openapi.yaml` (now advertises `orchestration_mode`, no residual `execution_mode` / old enum values); `make openapi-check` clean.++**⚠️ Deviation reconciled — the pin was NOT editable as the cold-start note claimed.** The cold-start note (and §5) asserted "`pipelex-api` pins core editable." **In reality the branch sat on a git-SHA pin** (`[tool.uv.sources] pipelex = { git = …, rev = "51ff0941…" }`, from parent commit `c60be1f` "pin pipelex to a git SHA so CI can resolve it (was a local editable path)") — and that SHA predates Phase 1, so it has `execution_mode.py` and lacks `delivery_mode`/`orchestration_mode`. Phase 3 therefore did not compile against it. **Resolution:** flipped `[tool.uv.sources]` back to `pipelex = { path = "../_plugins", editable = true }` (matching `pipelex-temporal`'s convention) + `make li` to relock/reinstall, so the Phase 1 core split is live. **CARRY-FORWARD (the one open item):** this editable local-path pin is unresolvable in CI, so **before opening any `pipelex-api` PR, push the Phase 1 core and flip the pin to that pushed git+https SHA + relock** (the documented CI gotcha). `uv.lock` + `pyproject.toml` are part of `dd84a9a`.++**Tests (all updated to the two-axis model; §9 list).** `test_api_config.py` (dropped `TestApiConfigRejectsFireAndForgetDefault`; renamed to `TestResolveOrchestrationMode`; string tokens; 403 asserts `OrchestrationModeOverrideForbidden`), `test_execute_dispatch.py` (dropped the 400-f&f case; `_StubOrchestrator.run` gains `delivery` + `supports_fire_and_forget`; asserts `delivery is BLOCKING`), `test_start_dispatch.py` (deleted the `_async_start_mode` pure-mapping class; new `TestStartCapabilityGate` — ack-when-async-capable asserts `delivery is FIRE_AND_FORGET`, honest-400 asserts `StartRequiresAsyncOrchestration` + `stub.calls == []` proving `run` is never awaited), `test_validate_dispatch.py` (string `"temporal"` token; renamed forbidden-override test + asserts the error_type), `test_protocol_conformance.py` (`mode="direct"`; `fake_run` gains `delivery` + an `assert delivery is FIRE_AND_FORGET`; `fake_orchestrator.supports_fire_and_forget = True` so the new gate is exercised).++**Docs + CHANGELOG.** `docs/configuration.md` ("Execution mode" → "Orchestration mode", two-axis model, honest-400), `docs/pipe-run.md` (`/execute` + `/start` notes; the `workflow_id` field note — a 202 is now only ever an async backend's, direct returns 400), `docs/pipe-validate.md` (`orchestration_mode`-aware, no delivery axis), `CHANGELOG.md` `[Unreleased]` rewritten to the orchestration_mode/delivery model (the never-released flat `execution_mode` bullets reshaped — no backward-compat concern).++**Gates.** `pipelex-api` `make c` (ruff / pyright **0** / mypy **0**) + `make tp` (**319 passed**) + `make openapi-export` then `make openapi-check` (artifact up to date) — all green against the live Phase 1 core (editable `../_plugins`). Python 3.11.++**Code-review triage (Checkpoint 3).** One clean-context `pr-review-toolkit:code-reviewer` (no plan/rationale handed in; pointed only at `dd84a9a`, cross-checked against the live core contract in `../_plugins`). **Verdict: no must-fix issues — correct, complete, standards-compliant.** It independently confirmed: the capability-check ordering is right (resolve → lookup → missing-check → `supports_fire_and_forget` 400, all before `pipeline_run_setup`; the orchestrator is looked up **once** and reused at dispatch so the gate can't diverge); the **honest-400 actually fires on the real base** (core `DirectOrchestratorPlugin` in `plugins/builtins.py` registers `DirectOrchestrator` with `supports_fire_and_forget = False` under `"direct"`, so `/start` hits the capability 400, not the `MissingOrchestratorError` fallback — exactly as `test_start_honestly_400s…` asserts); no StrEnum/`==`-on-enum violation (`resolve_orchestration_mode`'s `==` is `str`/`str`; `DeliveryMode` only ever compared with `is`); Protocol signatures match; RFC 7807 helpers correct; the field-validator deletion + its test removal are justified (the invariant no longer exists), not a weakening; the test assertions strengthened (delivery-axis + never-awaited proofs). **One should-consider — the editable-pin CI break — is the carry-forward item already captured above** (flagged by the author in the commit body + inline comment; flip to a pushed git SHA before a PR). **No applied or deferred code items.**++**Cold-start note for Phase 4.** Phase 4 = the **bigger-than-first-estimated** Mistral plugin migration (real code, not a token swap — see the revised §4 Phase 4 for the verified `pipelex-mistralai-workflows` anchors: deleted-enum import, registration under the deleted member, its own `execution_mode` wire field + `is PipelexExecutionMode.DIRECT` compare, an orchestrator on the old `run` signature) **plus** bookkeeping: `_plugins/TODOS.md`, the two superseded sibling-plan markers, and `make check-spec-links` if any spec heading/test asserts mode values. Re-derive the §4 anchors against the live `pipelex-mistralai-workflows` tip before starting (they will have moved). The three landed repos (`_plugins` `e9e73e0eb`, `pipelex-temporal` `01d7dc3`, `pipelex-api` `dd84a9a`) all sit on `feature/orchestration-mode-delivery-split`, pinned core editable for local dev.++---++### Phase 4 — as-built (Mistral plugin migration + bookkeeping) — effort COMPLETE (2026-06-22)++**✅ This is the final phase. All four phases of the orchestration-mode/delivery split are landed.** The cold-start note just above is now historical — read this section for the as-built.++**⚠️ The deviation that shaped Phase 4 (the §4 anchors were re-derived and the pin assumption was wrong).** The plan (and §4/§5) assumed `pipelex-mistralai-workflows` pinned the Phase-1 core, so it would be "already red against Phase 1." **In reality the plugin pins core editable at `../_workflows`, NOT `../_plugins`** — and `_workflows` was on a *divergent* branch **`feature/mistralai-2x-bump`** that still carried the OLD `runtime_bridge` (had `execution_mode.py`, lacked `orchestration_mode`/`delivery_mode`). So the plugin was **green against its own core**, not red against the split — and could not be migrated in isolation. `feature/mistralai-2x-bump` and `feature/orchestration-mode-delivery-split` were divergent (neither an ancestor; common base `e460d072`); `mistralai-2x-bump` made **no** `runtime_bridge` changes relative to that base.++**Resolution (user-directed): merge the split into the plugin's core branch.** Merged `feature/orchestration-mode-delivery-split` → `feature/mistralai-2x-bump` in the `_workflows` worktree — **merge commit `a1448ad8f`, conflict-free** (the branches overlapped only in shared plugin-system work, which was identical). The merged core gives the plugin a runtime bridge carrying **both** the orchestration split and the mistralai-2x bump. Merged-core verification: `make tb` (boot/config) ✓, `make agent-check` (ruff/pyright 0/mypy 0/keyword-only) ✓, the merge-sensitive core suites (`runtime_bridge` + `plugins` + `test_plugin_config_loader` + `test_direct_bundle_validator`) all green.++**The Mistral migration — `pipelex-mistralai-workflows` commit `0eb3878` on `feature/Mistral-native`** (parent `e508e73`; nothing pushed). Same shape as the Phase 2 Temporal collapse — real code, not a token swap:++- **`mistral_orchestrator.py`** — defines `MISTRALAI_WORKFLOWS_ORCHESTRATION_MODE: Final[str] = "mistralai-workflows"` (the D-G token, **living with the orchestrator** exactly as `pipelex-temporal` puts `TEMPORAL_ORCHESTRATION_MODE` in `temporal_orchestrators.py`). `MistralNativeOrchestrator.run` adopts the new `OrchestratorProtocol.run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)` signature and declares **`supports_fire_and_forget = False`** — Mistral-native always runs to completion in-place, so it is blocking-only like core `DirectOrchestrator`. It accepts `delivery` and ignores it; the bridge's pre-dispatch capability gate (the §10 "Fix 1" Tier-3 guard, present in the merged core) rejects a `FIRE_AND_FORGET` request against it before `run` is ever called, so `run` provably never receives f&f. (No `# noqa: ARG002` needed — unlike core, this repo's ruff `select` omits `ARG`, so the unused-arg rule does not fire; the linter auto-removed the noqa as unused.)+- **`mistral_plugin.py`** — registers `add_orchestrator(mode=MISTRALAI_WORKFLOWS_ORCHESTRATION_MODE, …)`; dropped the deleted-enum import. **The plugin name / entry-point `MISTRAL_NATIVE_PLUGIN_NAME = "mistral_native"` is KEPT, deliberately distinct from the orchestration token** — the plugin-discovery identity vs the registry key, the same two-concept separation `pipelex-temporal` keeps between `TEMPORAL_PLUGIN_NAME` and `TEMPORAL_ORCHESTRATION_MODE`. (D-G renamed the *token*, not the plugin name; renaming the entry-point would force a reinstall for zero gain.)+- **`streaming.py`** — wire field `execution_mode → orchestration_mode` (on `PipelexPipeRunStreamingState` + the read from the input); the `is PipelexExecutionMode.DIRECT` enum compare → `== DIRECT_ORCHESTRATION_MODE` (both plain `str` now — `==` is correct, matches how core's `bridge.py` compares it); docstrings reworded off `DIRECT`/`TEMPORAL_BLOCKING`/`TEMPORAL_FIRE_AND_FORGET` onto the `direct`/non-direct orchestration-mode framing. The per-step-streaming branch still correctly keys on "is this the in-process `direct` mode".+- **`primitives/`** — doc/comment refs to the deleted enum; the internal scoped-library id prefix `mistral_native_` → `mistralai_workflows_` (`scoped_library.py`; purely internal, ephemeral per-call id — no wire contract).+- **Tests** — `test_plugin_discovery.py` looks up + asserts the orchestrator under `MISTRALAI_WORKFLOWS_ORCHESTRATION_MODE` (real registration check via full boot; renamed `test_mistralai_workflows_orchestrator_is_registered`); the three `test_activities_*` construct `PipelexPipeRunInput(orchestration_mode=DIRECT_ORCHESTRATION_MODE)`; `test_activities_streaming.py` asserts the renamed `orchestration_mode` wire field; `test_scoped_library.py` asserts the renamed id prefix; doc-only refs in `test_router_dispatch.py` / `test_native_recursion.py` / `test_data/bridge_funcs.py`. No assertion weakened — field-for-field.+- **Docs** — `CLAUDE.md` public-surface contract (the deleted `execution_mode` module + the two other now-deleted symbols `PipelexBridgeRuntimeError` / `MissingPipelexTemporalExtraError` — stale since the Temporal externalization — replaced with the real surface: `payloads`, `orchestration_mode`, `delivery_mode`, `PipelexBridgeDispatchError`); `README.md` quick-start + a recast "Orchestration mode and delivery" section (also dropped the dead `[temporal]` extra — the README now says install `pipelex-temporal` for the `temporal` mode); the `wip/boundary-violation-mistral-native.md` allowed-surface list; `CHANGELOG.md` (new `### Changed` reconciliation bullet + reshaped the never-released bullets that named the old enum); `pyproject.toml` entry-point comment.++**Gates (Mistral plugin, against the merged editable `../_workflows` core).** `make agent-check` (ruff / pyright **0** / mypy **0**) + `make agent-test` + the full migrated integration suite (`test_plugin_discovery` + `test_activities_{direct,offloaded,streaming}` + `test_native_recursion` + `test_scoped_library`) — **all green** (12 passed in the targeted verbose run). Python 3.13.++**Code-review triage (Phase 4).** One clean-context `pr-review-toolkit:code-reviewer` (no plan handed in; pointed only at `c2013fb` — the pre-amend SHA — cross-checked against the live merged core in `../_workflows` + the Temporal reference). **Verdict: the code migration is correct and complete.** It independently confirmed: the `run` signature conforms to the live `OrchestratorProtocol`; `supports_fire_and_forget=False` + ignoring `delivery` is safe given the bridge's pre-dispatch FF rejection; registration under `"mistralai-workflows"` with the name/token split is sound (not a bug); the `streaming.py` `==` string compare is correct (vs `is`); tests not weakened; no StrEnum/`==`-on-enum-member violation, no `except Exception`, keyword-only respected, no `__init__.py` re-exports; no new pyright/mypy regression (the `reportUnknown*` on the untyped Mistral SDK return are pre-existing, identical count on the parent). **One must-fix — APPLIED:** `README.md` still imported the deleted `PipelexExecutionMode` and documented it in a "## Execution modes" section (my source/test grep had been `.py`+config-scoped — the README slipped, the same `.md`-slips-a-`.py`-grep the Phase 1 review caught). Rewritten to `orchestration_mode`/`delivery` + the dead `[temporal]`-extra fix. **One nit — APPLIED:** the `wip/boundary-violation-mistral-native.md` allowed-surface list named the deleted `execution_mode` module → updated. Both fixes folded into the commit by amend (`c2013fb` → **`0eb3878`**). **No deferred code items.**++**Bookkeeping done.** Superseded markers added to both sibling plans (`execute-per-request-execution-mode.md`, `orchestrator-dispatched-validate.md`) pointing here; `_plugins/TODOS.md` follow-on entry flipped to "DONE (Phases 1–4)" + the merge deviation recorded; the `pipelex-api` `CHANGELOG.md` was already on the orchestration_mode/delivery model from Phase 3 (`dd84a9a`) — no Phase 4 change needed there; `make check-spec-links` **not** triggered (verified: zero `docs/specs/` or `conformance/` references to mode values).++**Carry-forward / open items (none block the effort; flagged for whoever pushes/PRs):**++1. **`pipelex-api` editable-pin → pushed git-SHA flip (from Phase 3, still open).** `pipelex-api` pins core editable `../_plugins`; before any `pipelex-api` PR, push the core split and flip to that pushed git+https SHA + relock (the documented CI gotcha).+2. **`pipelex-mistralai-workflows` core pin.** It pins `../_workflows` editable; for a real release the same flip applies (push the merged core, pin a published/pinned ref). The plugin's own CHANGELOG already notes the `../_workflows` co-dev override is stripped at v0.1.0.+3. **`_workflows` ↔ `_plugins` doc divergence.** The `a1448ad8f` merge copied a *pre-Phase-4 snapshot* of the orchestration tracker docs (this plan, `TODOS.md`, the sibling plans) into `_workflows`. The Phase 4 bookkeeping edits above landed on **`_plugins`** (`feature/orchestration-mode-delivery-split`, the tracker SoT), so the `_workflows` copies are momentarily stale. They reconcile on the next merge of `feature/orchestration-mode-delivery-split` → `feature/mistralai-2x-bump` (or whenever the two core branches are integrated). Not a code concern — `_workflows`'s *code* is the superset (it contains the split); only the in-flight tracker prose differs.+4. **Two divergent core branches now exist** (`feature/orchestration-mode-delivery-split` in `_plugins`, `feature/mistralai-2x-bump` in `_workflows`, the latter containing the former via `a1448ad8f`). Integrating/landing them is a branch-strategy decision for the user — out of scope for this effort.diff --git a/wip/plugins/orchestrator-agnostic-runner-and-flavors.md b/wip/plugins/orchestrator-agnostic-runner-and-flavors.mdnew file mode 100644index 000000000..ae78c9b8e--- /dev/null+++ b/wip/plugins/orchestrator-agnostic-runner-and-flavors.md@@ -0,0 +1,182 @@+# Orchestrator-agnostic runner & deployment flavors++**Status:** plan + agent handoff. Decisions locked (see §3). Not yet started in any consumer repo.+**Owner effort:** the plugin-system externalization (see [`TODOS.md`](../../TODOS.md) and the Phase 5 cut-over notes). This doc is the forward-looking *consumer* migration: it explains how the new plugin system changes the world for projects that consume `pipelex` and choose an orchestrator (Temporal, Mistral, or none), and gives the concrete, sequenced handoff to make `pipelex-api` an orchestrator-agnostic base plus N thin deployment flavors.++> **Why this exists.** Once Temporal left core (it is now the private `pipelex-temporal` plugin, discovered via the `pipelex.plugins` entry point), a consumer no longer gets Temporal "for free" by installing an extra. More importantly, the recon for this migration found that `pipelex-api` is **not** an orchestrator-agnostic runner today — it is wired directly to Temporal in production code. Making it the shared base for multiple deployment flavors (`pipelex-api-hosted` = Temporal, a future `pipelex-api-mistral` = Mistral) is a real decoupling refactor, not a dependency bump. This doc captures the target model, the locked design decisions, and the per-repo work.++---++## 1. The model shift (read this first)++The plugin system splits a previously-monolithic idea ("Temporal is on") into orthogonal concepts. Internalize these before touching any repo.++**Presence vs activation.**++- **Presence** = the plugin distribution is installed, so its `pipelex.plugins` entry point is discovered at boot and its `register()` runs. For an orchestrator plugin, `register()` always contributes its `execution_mode` arms to the orchestrator registry. Presence alone makes a mode *dispatchable*; it does not make it *used*.+- **Activation** = the runtime is actually told to use it. Activation decomposes into **two independent knobs** (this is the crux — see below).++**Two independent activation knobs.**++| Knob | Owns | Set by | Temporal | Mistral | Direct |+|---|---|---|---|---|---|+| `execution_mode` | which mode a *top-level* run dispatches as, through the bridge | runner config (default) + optional per-request override | `temporal_blocking` / `temporal_fire_and_forget` | `mistral_native` | `direct` |+| `plugins.boot_orchestrator` | whether to claim the process-global hub slots (content generator / pipe router / pipe run / task manager) at boot | deployment config TOML | `"temporal"` | *unset* | *unset* |++The two are **not derivable from each other** — and Mistral is the proof. A Temporal deployment needs *both* (the boot gate stands up the Temporal client/task-manager that the orchestrator uses to enqueue; the `execution_mode` tells the bridge to enqueue). A Mistral deployment needs **only** `execution_mode = mistral_native` — its plugin claims **no** hub slots (it is a self-contained per-call orchestrator), so there is no boot gate to set. A plain in-process deployment sets neither.++**Boot-global vs per-call orchestrators.** This is *why* the two knobs exist. A **boot-global** orchestrator (Temporal) replaces process-wide execution machinery — it must claim hub slots at boot, gated by `boot_orchestrator`. A **per-call** orchestrator (Mistral) only adds a dispatch arm — presence is enough; selection is per run via `execution_mode`. When you add a new orchestrator plugin, classify it first; the classification decides whether `boot_orchestrator` is even meaningful for it.++**Install channel follows license, not convenience.**++- `pipelex-temporal` is **private** → distributed by a pinned `git+ssh` ref (precedent: `pipelex-shared @ git+ssh://…/infra-python-tools.git@<ref>`). Only **private** images may pin it.+- `pipelex-mistralai-workflows` is **public** → plain PyPI install (`pipelex-mistralai-workflows==X`). (Currently pre-publish; gated on its own `mistralai-2.x` work — treat as PyPI once published.)++**Base-agnostic + flavor = base + one plugin.** The target: `pipelex/pipelex-api` (public, Docker Hub) names *no* orchestrator and ships only the in-process/direct path. Each deployment **flavor** is `base image + exactly one orchestrator plugin + that plugin's activation`. This is what lets a future `pipelex-api-mistral` exist at all — and it is the architecture the rest of this doc builds toward.++---++## 2. Where we are today (the gap)++`pipelex-api` is a Temporal-only runner. Concrete coupling sites (line numbers drift; locations hold):++- **Execution has no agnostic path.** `ApiRunner.start()` (`api/routes/pipelex/pipeline.py`) calls `make_temporal_pipe_run().start()` directly. There is **zero** use of the agnostic bridge entry `run_pipe_via_bridge` / `PipelexPipeRunInput` anywhere in `api/`.+- **Validate reads a deleted field.** `ApiRunner.validate()` branches on `get_config().temporal.is_enabled` — a config field **core has removed** — and on the "enabled" arm dispatches `dispatch_dry_validate(DryValidateArg(...))` from `pipelex.temporal.tprl_pipe`. The "disabled" arm already runs `validate_bundles_in_process(...)` in-process.+- **Exception handling imports `temporalio`.** `api/exception_handlers.py` imports `temporalio.exceptions.TemporalError` and registers a dedicated `handle_temporal_error`. It also references `WorkflowExecutionError` (which is a `PipelexError` via `TemporalFlowError`, so it is already covered generically by `handle_pipelex_error`).++Because core deleted both the `temporal` extra and the `temporal` config field, `pipelex-api` against the current `pipelex` worktree is broken on all three sites. The fix is the refactor below, not a pin nudge.++The plugin system inverted **core pipelex's** couplings (Phases 0–5). It did **not** provide seams for the **runner/HTTP layer** — exception-handler contribution and validate dispatch are `pipelex-api` concerns. One small new core seam is required (§3 F3).++---++## 3. Locked decisions++These were decided with the user. Do not re-litigate; if you believe one is wrong, raise it explicitly before deviating.++- **D1 — `boot_orchestrator` is set from the flavor's deployment TOML.** The Temporal flavor bakes `[plugins] boot_orchestrator = "temporal"` into its env-specific `pipelex_{env}.toml`. No change to `pipelex-api` for the gate. *Verified safe:* `Pipelex.setup()` only overrides `plugins.boot_orchestrator` when its param is non-`None` (`pipelex/pipelex.py` ~`:206`), so a TOML-provided value is **preserved**, not clobbered, and it is in place before `build_registrar` reads it. The core comment "set programmatically, not from pipelex.toml" describes core's *defaults*; a deployment override is a supported, distinct use.+- **D2 — plugin config files become env-aware, mirroring the main config.** Today `load_temporal_config()` resolves a *single* `temporal.toml` (project-then-global, no env layering), unlike the main config which layers `pipelex_{env}.toml` via `PIPELEX_ENV`. Add env-aware resolution so a plugin's config self-loads `name.toml` (packaged default) → `name_{env}.toml` → `name_override.toml`. Implement as a **reusable core helper** so every plugin inherits identical env semantics; `pipelex-temporal` uses it. One image bakes all envs; `PIPELEX_ENV` selects at runtime.+- **F1 — `execution_mode` = runner config default + optional per-request override, gated by a per-deployment policy.** `[api] execution_mode` sets the deployment default; the request may override it **only** when the deployment's policy allows it (so a locked-down Temporal runner can refuse a caller forcing `direct`). The runner builds `PipelexPipeRunInput(execution_mode=…)` and dispatches through `run_pipe_via_bridge`; the bridge output already carries `workflow_id`, so async-start semantics survive.+- **F2 — validate is always DIRECT in-process.** Drop the Temporal validate branch entirely; `ApiRunner.validate()` always calls `validate_bundles_in_process(...)`. Removes the `pipelex.temporal` imports (`dispatch_dry_validate` / `DryValidateArg`). **Behavior change to call out at handoff:** the API runner now loads the method library to validate, which the prior D10/D14 design deliberately pushed onto the worker. Acceptable per decision; note the resource/isolation implication on the hosted runner in that repo's docs.++ > **REVERSED / EXTENDED — orchestrator-dispatched `/validate`.** F2's "always DIRECT" is superseded: `/validate` is now `execution_mode`-aware through a new per-call `BundleValidatorRegistry` seam (mirroring the `OrchestratorRegistry`). `direct` keeps F2's in-process path (the agnostic base validates in-process); `temporal_*` dispatches the whole job to a worker, restoring what F2 removed — assembled back into the **same** canonical report API-side, so the verdict wire is byte-identical across backends. The library-load implication is now flavor-conditional, not absolute. Plan + as-built: [`orchestrator-dispatched-validate.md`](orchestrator-dispatched-validate.md).+- **F3 — orchestrator-specific HTTP error handling via a new plugin seam, contributed as a mapper (not a raw FastAPI handler).** Core's `PluginRegistrar` gains a framework-agnostic contribution: `add_http_error_mapper(*, exc_type: type[Exception], to_error_report: Callable[[Exception], ErrorReport])`. The orchestrator plugin registers its transport-fault mapping (e.g. `temporalio.TemporalError → ErrorReport[transient, RUNTIME]`). `pipelex-api` iterates the registrar's mappers at app construction and wraps each into a FastAPI handler using its existing RFC 7807 + `DisclosureMode` rendering. **FastAPI/Starlette stays only in `pipelex-api`; core and the plugin import neither.** This grows the plugin contract by one optional capability → bump `PLUGIN_API_VERSION` and document it in the orchestrator SPI.++---++## 4. Image architecture (handles public + private plugins uniformly)++A **two-layer** build per flavor. This isolates the plugin-install complexity (private SSH, version-locking) into one place and keeps the env-config layer as trivial as it is today.++```+pipelex/pipelex-api:<ver> (public base, orchestrator-agnostic, Docker Hub)+ │+ ├─ flavored base = base + ONE plugin (built once per flavor → private ECR)+ │ pipelex-api-temporal:<ver> = base + pipelex-temporal (git+ssh)+ │ pipelex-api-mistral:<ver> = base + pipelex-mistralai-workflows (PyPI)+ │+ └─ env-config child = flavored base + COPY env TOMLs (per env; stays config-only)+ pipelex-api-{dev,staging,prod} (Temporal)+ pipelex-api-mistral-{dev,staging,prod}+```++**One parameterized install recipe across flavors.** The flavored-base Dockerfile takes a build arg `PLUGIN_SPEC` and always exposes a BuildKit SSH mount:++- PyPI plugin (Mistral): `PLUGIN_SPEC="pipelex-mistralai-workflows==X"` → plain install, SSH mount unused.+- Private plugin (Temporal): `PLUGIN_SPEC="pipelex-temporal @ git+ssh://[email protected]/Pipelex/pipelex-temporal.git@<ref>"` → install uses `--mount=type=ssh`, with the deploy key wired into the CI builder (`docker build --ssh default`). Never bake a key into a layer.++Install mechanics that bite (encode these in the Dockerfile):++- The base image's Python lives in a **uv project venv at `/app/.venv`**, run via `uv run uvicorn …`. Install into that venv (`cd /app && uv pip install …`), do **not** let the resolver swap the pinned `pipelex` already baked in (pin / `--no-deps` as appropriate).+- The base image **purges `git`/`build-essential`** in a late layer. Re-add `git` (+`openssh-client` for the private case) in the flavored-base layer before installing.+- **Version-lock is load-bearing.** A plugin imports `pipelex` internals deeply; the plugin ref and the base `pipelex` version are a coupled pair that bump together. Track the pair wherever the flavor repo records versions (e.g. `api-config.toml`).++Whether the flavored base is a *separately-published* ECR image or just an earlier stage in a single multi-stage Dockerfile is a minor call: publish it only if more than one downstream consumes it; otherwise a multi-stage build keeps it to one artifact. Either way the env-config child stays a pure `COPY` of TOMLs, exactly like `pipelex-api-hosted/Dockerfile` today.++---++## 5. Phased plan++Order matters: enabling changes in `pipelex` and `pipelex-temporal` must land before `pipelex-api` can consume the new seam.++### Phase A — Core enabling changes (`pipelex`, the `_plugins` worktree)++1. **D2 env-aware plugin config loader.** Add a reusable helper on `config_manager` (e.g. `load_plugin_config(*, name, schema)`) that resolves `name.toml` (packaged default of the calling plugin) → `name_{environment}.toml` → `name_override.toml` with deep-merge, mirroring `load_config`'s env layering (`pipelex/system/configuration/config_loader.py`).+2. **F3 HTTP-error-mapper seam.** Add `add_http_error_mapper(*, exc_type, to_error_report)` to `PluginRegistrar` (`pipelex/plugins/registrar.py`) plus a read accessor for the collected mappers. Use `ErrorReport` (core type) in the signature — **no FastAPI import**. Bump `PLUGIN_API_VERSION` (`pipelex/plugins/contract.py`) and add the capability to the orchestrator SPI doc (`docs/under-the-hood/orchestrator-plugins.md`).+3. Tests: env-layered plugin-config round-trip; registrar collects mappers; import-light boot still green (the seam adds no SDK/HTTP import).++**Acceptance:** `make agent-check` clean · `make tb` green · `make agent-test` green. Core still names no orchestrator and imports no FastAPI.++### Phase B — `pipelex-temporal` (additive)++1. Switch `load_temporal_config()` to the D2 helper; ship `temporal_{env}.toml` resolution (the packaged default `temporal.toml` stays).+2. In `TemporalPlugin.register()`, register the F3 mapper: `temporalio.TemporalError → ErrorReport` classified transient / `RUNTIME` (port the classification currently living in `pipelex-api`'s `handle_temporal_error`). Keep `register` import-light — the mapper closure may import `temporalio` lazily when first invoked, matching the existing thunk discipline.++**Acceptance:** `pipelex-temporal` `make agent-check` + `make agent-test` green against the Phase A `pipelex`.++### Phase C — `pipelex-api` decoupling (the big one)++Make the base orchestrator-agnostic. No `pipelex.temporal.*`, no `temporalio`, anywhere in `api/`.++1. **Execution (F1).** Rewrite `ApiRunner.start()` / `execute` to build `PipelexPipeRunInput(…, execution_mode=…)` and call `run_pipe_via_bridge`; read `workflow_id` off the output payload. Resolve `execution_mode` from a new `[api] execution_mode` config default, with an optional per-request override gated by a per-deployment policy (e.g. `[api] allow_request_execution_mode_override` and/or an allowlist). Delete `make_temporal_pipe_run` usage + import.+2. **Validate (F2).** Collapse `ApiRunner.validate()` to the single in-process path (`validate_bundles_in_process`). Delete the `temporal.is_enabled` branch and the `dispatch_dry_validate` / `DryValidateArg` imports. Document the library-load-on-API behavior change in `pipelex-api/docs/`.+3. **Exception handlers (F3).** Remove the `temporalio` import and the hardcoded `handle_temporal_error` registration from `api/exception_handlers.py`. At app construction (`api/main.py` → `register_exception_handlers`), iterate `registrar` HTTP-error mappers and register a FastAPI handler per `exc_type` that runs the mapper then renders via the existing RFC 7807 + `DisclosureMode` path. Keep `handle_pipelex_error` (it already covers `WorkflowExecutionError`); drop the explicit `WorkflowExecutionError` import/catch in `validate.py` (now unreachable — validate is DIRECT and never raises it).+4. **Deps.** Drop the `temporal` extra from `pyproject.toml` (`pipelex[mistralai,anthropic,google,…]==<ver>`, no `temporal`). The base depends on **no** orchestrator plugin.+5. Tests: update `tests/unit/test_exception_handlers.py`, `test_validate_*` to the agnostic shape (a synthetic plugin contributing a mapper proves the seam without importing `temporalio`).++**Acceptance:** `pipelex-api` test suite green with **no** `pipelex.temporal` / `temporalio` import in `api/`; `grep -rn "temporal" api/` returns only incidental prose. A direct-mode run + validate works with no orchestrator plugin installed.++### Phase D — `pipelex-api-hosted` (Temporal flavor)++1. Flavored base (`pipelex-api-temporal`): `FROM pipelex/pipelex-api:<ver>` + install `pipelex-temporal @ git+ssh://…@<ref>` per §4 (re-add `git`+`openssh-client`, SSH mount, version-locked to `<ver>`). Wire the deploy key into CI (`--ssh default`).+2. Config migration in the baked `.pipelex/pipelex_{env}.toml`:+ - Replace `[temporal] is_enabled = true` → `[plugins] boot_orchestrator = "temporal"`.+ - Add `[api] execution_mode = "temporal_fire_and_forget"` (or `temporal_blocking` per route semantics) + the override policy you want.+ - **Move** the connection tree (`temporal_config`, `worker_config`, `queue_options`, `search_attributes`) **out** of `pipelex_{env}.toml` into baked `.pipelex/temporal_{env}.toml` files (the D2 plugin-config file). The main `pipelex_{env}.toml` no longer carries any `[temporal.*]` keys (core would reject them).+3. The env-config child Dockerfile stays a `COPY` — now copying `pipelex_{env}.toml` **and** `temporal_{env}.toml`.++**Acceptance:** dev image boots with Temporal active; a run enqueues to Temporal Cloud; `PIPELEX_ENV` selects the right `temporal_{env}.toml`.++### Phase E — `pipelex-api-mistral` (Mistral flavor, new repo)++Scaffold mirroring `pipelex-api-hosted` conventions.++1. Flavored base (`pipelex-api-mistral`): `FROM pipelex/pipelex-api:<ver>` + `uv pip install pipelex-mistralai-workflows==X` (PyPI — no SSH, no `git` re-add needed).+2. Config in baked `.pipelex/pipelex_{env}.toml`: `[api] execution_mode = "mistral_native"`. **No** `boot_orchestrator` (per-call orchestrator), **no** plugin config file (the Mistral plugin self-loads none today).+3. Gated on `pipelex-mistralai-workflows` being published and on its in-flight `mistralai-2.x` work — coordinate, don't disrupt.++**Acceptance:** image boots with `mistral_native` dispatchable; a run routes through the Mistral orchestrator; no Temporal anything present.++### Phase F — `pipelex-worker` pin flip (from the existing cut-list)++Per the Phase 5 cut-list in [`TODOS.md`](../../TODOS.md): `pyproject.toml` `pipelex[dynamodb,s3,temporal]==X` → `pipelex[dynamodb,s3]==X` + `pipelex-temporal @ git+ssh://…@<ref>`; `Dockerfile` CMD `["pipelex","worker",…]` → `["pipelex-temporal","worker",…]`; `Makefile` `pipelex worker` → `pipelex-temporal worker`; set `boot_orchestrator=temporal` for the worker process. Land in the **same** commit as the `pipelex` pin bump.++### Checkpoints++- **Checkpoint 1 — after Phase B:** core + `pipelex-temporal` green together; the seam exists and the plugin contributes through it. Natural session boundary before the `pipelex-api` refactor.+- **Checkpoint 2 — after Phase C:** `pipelex-api` is agnostic and green with no orchestrator installed. THE gate — flavors are meaningless until the base is clean. Fan out `/code-review` on the `api/` diff.+- **Checkpoint 3 — after Phase D:** first real flavor boots end-to-end against Temporal Cloud (dev). Validates the whole chain before replicating for Mistral.++---++## 6. Per-repo agent handoff summary++| Repo | Change | Channel |+|---|---|---|+| `pipelex` (`_plugins`) | D2 env-aware plugin-config helper; F3 `add_http_error_mapper` seam + `PLUGIN_API_VERSION` bump; SPI doc | normal release |+| `pipelex-temporal` | use D2 helper; register F3 `TemporalError → ErrorReport` mapper | private git+ssh |+| `pipelex-api` | drop `temporal` extra; route execution through bridge (F1); validate DIRECT (F2); consume F3 mappers; remove all `pipelex.temporal`/`temporalio` imports | public Docker Hub |+| `pipelex-api-hosted` | flavored-base install of `pipelex-temporal` (git+ssh); `boot_orchestrator=temporal` + `[api] execution_mode` in `pipelex_{env}.toml`; connection tree → `temporal_{env}.toml` | private ECR |+| `pipelex-api-mistral` (new) | scaffold; flavored-base install of `pipelex-mistralai-workflows` (PyPI); `[api] execution_mode=mistral_native`, no boot gate | private ECR |+| `pipelex-worker` | pin `pipelex-temporal` (git+ssh); CLI → `pipelex-temporal worker`; `boot_orchestrator=temporal` | private |++---++## 7. Open items / risks to verify during execution++- **F3 — confirm transport faults are wrapped, or rely on the mapper.** Verify whether the Temporal orchestrator already wraps bare `temporalio` transport errors into `PipelexError`s (`TemporalServerError` etc.) before they reach HTTP. If it does, the F3 mapper is belt-and-suspenders; if not, it is the only thing keeping such errors off the catch-all 500. Either way the mapper makes the base correct without naming Temporal.+- **F1 — define the override policy precisely.** Decide the exact shape (`allow_request_execution_mode_override` bool vs a per-deployment allowlist of modes) and the default (recommend: override **off** on hosted flavors). A caller must never be able to force `direct` on a runner whose whole point is distributed execution.+- **F2 — quantify the library-load cost on the hosted runner.** Validate now loads the method library API-side. Confirm the hosted runner image/resources are sized for it, and document it in `pipelex-api`/`pipelex-api-hosted` docs.+- **Version-lock bookkeeping.** Record the (base `pipelex-api` version ⇄ plugin ref) pair in each flavor repo's version file so a bump can't drift them apart.+- **Mistral publish gating.** `pipelex-mistralai-workflows` must be published to PyPI and its `mistralai-2.x` work settled before Phase E.+- **Docs deploy.** The Temporal error pages left in core `docs/errors/` on purpose (type_uri dereference targets) still need a home decision when `pipelex-temporal` docs exist — tracked in the Phase 5 cut-over notes, not here.diff --git a/wip/plugins/orchestrator-dispatched-validate-review-findings.md b/wip/plugins/orchestrator-dispatched-validate-review-findings.mdnew file mode 100644index 000000000..311850722--- /dev/null+++ b/wip/plugins/orchestrator-dispatched-validate-review-findings.md@@ -0,0 +1,47 @@+# Review findings — orchestrator-dispatched `/validate` (PR #998)++Captured from a pre-landing `/review` (gstack) of the `feature/Orchestrator-dispatched-validate` feature across all three repos (`pipelex` core = PR #998 vs `refactor/Plugins-4`; `pipelex-temporal` vs `refactor/own-temporal-config`; `pipelex-api` vs `refactor/orchestrator-agnostic-base`). Verdict: **landable.** No critical findings. Greptile P1 (verdict alias bound under `TYPE_CHECKING`) already fixed by `34a33633d`. The items below were surfaced by the testing + adversarial passes and are deferred (none is a clear, unambiguous win; the brief was clean-solid, no speculative guards). Recorded so they are not re-discovered cold.++## 1. MISTRAL_NATIVE `/validate` is unserviceable; its "missing" error names an already-installed package (deferred — out of scope by plan)++**Observation (adversarial, conf 8).** `pipelex-mistralai-workflows`'s plugin (`mistral_plugin.py:31`) registers only the MISTRAL_NATIVE *orchestrator*, never a bundle validator. So a deployment configured `execution_mode = mistral_native` (plugin installed, `/start` working) would fail every `/validate` with `MissingBundleValidatorError` → 5xx — and that error's MISTRAL_NATIVE branch (`pipelex/runtime_bridge/exceptions.py`) tells the operator to `pip install pipelex-mistralai-workflows`, a package that is already installed in that scenario. `/start` and `/validate` are asymmetric for the one orchestrator whose error message specifically promises serviceability.++**Why deferred, not fixed.** This is the **explicitly deferred Mistral arm**, not a regression this PR introduces:++- The plan (`orchestrator-dispatched-validate.md`) defers Mistral in three places: START-HERE ("Mistral is a gated follow-up — do **not** build it"), decision **C** ("Mistral deferred… gated on its own `mistralai-2.x` work"), and Phase **V-Mistral — DEFERRED** ("Add a `MistralBundleValidator` later… The generic seam means this lands without touching core or the API").+- `pipelex-mistralai-workflows` is itself gated from publish (HOLD on `mistralai-2.x` / instructor #2298), so `mistral_native` is not a shippable `/validate` config on any deployment that ships today. The dead path is unreachable in production.+- The error-message wording mirrors the parallel `MissingOrchestratorError` MISTRAL_NATIVE branch verbatim; once the Mistral validator lands, the install hint becomes directionally correct again. Touching only the message now (to soften the "install X" claim) is a judgment call, not a mechanical win, and would desync the two parallel error classes.++**When the Mistral validator lands (Phase V-Mistral):** register a `MistralBundleValidator` under MISTRAL_NATIVE (genuine dispatch, or an in-process passthrough — validation is backend-independent pure analysis). At that point `MissingBundleValidatorError`'s MISTRAL_NATIVE branch is reachable only as a true "plugin not installed" signal and its message is correct. If Mistral is *intentionally* never going to serve `/validate`, revisit the message then (e.g. point MISTRAL_NATIVE `/validate` at the DIRECT in-process validator as a fallback) — but that is a Phase-D/V-Mistral decision, not this PR's.++## 2. The `cast` at the verdict narrowing point launders a non-`PipelexValidationReport` valid arm (deferred — latent, loud-fail, future-plugin guard)++**Observation (adversarial, conf 6).** The seam's valid arm is typed at the protocol-level `ValidationReport` (a leaf type, to stay import-acyclic in core — see §9 divergence in the plan). `ApiRunner.validate_verdict` ends with `return cast("PipelexValidationReport | ErrorReport", verdict)`, and the route discriminates with `if not isinstance(verdict, PipelexValidationReport): return _invalid_report_response(verdict, ...)`. The protocol `ValidationReport` valid arm has neither `validation_errors` nor `message` (confirmed in `mthds/protocol/models.py`), and `_invalid_report_response` accesses `error_report.validation_errors` / `error_report.message`. So a *future* validator whose valid arm returned a bare `ValidationReport` (not the `PipelexValidationReport` subtype) would fail the `isinstance`, be routed to the invalid-arm handler, and crash on attribute access.++**Why not a bug today, why deferred.**++- Both shipped validators (`DirectBundleValidator`, `TemporalBundleValidator`) are annotated `-> PipelexValidationReport | ErrorReport` and in fact return the concrete envelope; the type checker enforces this for every in-tree validator. The `cast` is the single documented narrowing point the plan's §9 acceptance note already calls out.+- The failure mode for a hypothetical non-conforming external plugin is **loud (5xx crash), not silent data corruption** — the worst-case is an obvious server error, not a wrong verdict.+- Adding a runtime guard ("validator returned a non-Pipelex report → raise a clear fault") would broaden the seam's surface to defend a scenario no in-tree code can reach — exactly the speculative guarding the review brief excludes ("no guarding impossible scenarios, no broadening surface area").++**If revisited:** if external/third-party bundle validators ever become a supported extension point (not just core + the two first-party plugins), add a single explicit guard at the narrowing point in `ApiRunner.validate_verdict` — `if isinstance(verdict, ErrorReport) or isinstance(verdict, PipelexValidationReport): ...` else raise a clear "validator must produce a PipelexValidationReport" fault — so a misbehaving plugin gets a precise diagnostic instead of an `AttributeError` deep in the route. Until then the `cast` + the type-checked first-party validators are sufficient.++## 3. Test-coverage symmetry gaps (deferred — covered transitively / by convention)++Testing pass, all INFORMATIONAL, none a correctness gap:++- **Dispatched-*valid* arm has no ungated fast-suite test (conf 6).** `pipelex-api/tests/unit/test_validate_dispatch.py` exercises the non-direct *invalid* (`ErrorReport` → 200) and *fault* (→ 5xx) arms with a stub validator, but not the non-direct *valid* (`PipelexValidationReport` → 200 `is_valid: true`) arm. That valid dispatched path is covered by `pipelex-temporal/tests/integration/test_temporal_bundle_validator_in_memory.py`, which is `@pytest.mark.temporal`-gated (needs `temporalio` + an in-memory server) and excluded from the default fast `agent-test` run. The route mapping is backend-independent and the DIRECT valid arm is covered ungated by the existing `/validate` suite, so risk is low. This matches the plan's §7 V2 acceptance bar, which deliberately scoped the stub tests to invalid/fault/missing/403 and relied on V1 for the valid path. **If desired:** add a `TestValidateDispatch` case with a stub validator returning a real `PipelexValidationReport` for a temporal mode, asserting 200 + `is_valid: true` — proves the valid dispatched arm with no `temporalio` import. Nice-to-have symmetry, not a bug.+- **`DirectOrchestratorPlugin.register` validator line untested in core (conf 5)** and **`TemporalBundleValidator._require_temporal_extra` guard untested (conf 4).** Both are zero-coverage lines, but each is consistent with the pre-existing untested-by-convention sibling (the DIRECT *orchestrator* registration is likewise not directly asserted in core; the orchestrator's own `_require_temporal_extra` is likewise untested). Wiring is covered transitively by the cross-repo boot/envelope tests. Flagging only for the record; not worth a new test against the existing convention.++## Passes with no findings++- **Maintainability:** clean — no dead code/imports, F2 "always DIRECT" prose thoroughly reversed across docstrings + `docs/pipe-validate.md` + CHANGELOG + trackers, no DRY violation worth extracting (the `start`/`validate_verdict` duplication is the separately-noted deliberate mirror; the two `Missing*Error` classes are parallel-by-design), no module-boundary violation (the protocol-level typing is a *deliberate* import-cycle avoidance).+- **API contract:** verified byte-equivalent for the direct path; `execution_mode` field is backward-compatible (optional, default None → deployment default); the 200-always-for-verdict / non-2xx-for-no-verdict contract is preserved including the new forbidden-override 403; `isinstance(verdict, PipelexValidationReport)` is provably correct (disjoint MROs — neither `ErrorReport` nor `PipelexValidationReport` subclasses the other).+- **Enum completeness:** `MissingBundleValidatorError._build_message`'s `match` over `PipelexExecutionMode` is exhaustive (all four members, `-> str`, no `case _`); a 5th member would be caught by the linter. Mirrors the existing pattern in `execution_mode.py`.+- **Boot wiring:** `set_bundle_validator_registry` is called in the single boot block (`pipelex.py`) immediately after `set_orchestrator_registry`; identical lifetime guarantees to the already-production orchestrator registry — no partial-boot gap introduced.++## Already-captured (pre-existing) deferred notes — not re-opened++- `validate-dispatch-requires-boot-active.md` — dispatched `/validate` effectively requires `boot_orchestrator == "temporal"` at runtime (reused executor guard); identical to the run orchestrators, pre-existing, gated on Phase D's boot-model decision.+- `validate-verdict-subclass-parity-divergence.md` — the Temporal arm's `error_type != ValidateBundleError.__name__` string-equality vs the direct arm's `except ValidateBundleError` subclass-catch; latent (zero subclasses exist), forced by the wire boundary.+- `validate-start-mode-dispatch-duplication.md` — `start`/`validate_verdict` share a ~3-line resolve-then-dispatch shape; extraction deferred as added indirection, not removed complexity.diff --git a/wip/plugins/orchestrator-dispatched-validate.md b/wip/plugins/orchestrator-dispatched-validate.mdnew file mode 100644index 000000000..3539f4950--- /dev/null+++ b/wip/plugins/orchestrator-dispatched-validate.md@@ -0,0 +1,235 @@+# Plan — orchestrator-dispatched `/validate` (reverse/extend F2)++> **⚠️ SUPERSEDED (2026-06-22) by [`orchestration-mode-and-delivery-split.md`](orchestration-mode-and-delivery-split.md).** This plan was implemented (V0–V3, committed across all three repos) on top of the **flat** `PipelexExecutionMode` enum, and registered the bundle validator under **both** temporal modes (`temporal_blocking` + `temporal_fire_and_forget`) — a double-registration that existed only because the flat enum lacked a delivery axis. The orchestration-mode/delivery split (now DONE, incl. Phase 4 — the `pipelex-mistralai-workflows` migration) **removed that flatness**: delivery became a `DeliveryMode` parameter to `OrchestratorProtocol.run`, the two Temporal orchestrators collapsed into one registered **once** under `"temporal"`, and the `/validate` double-registration collapsed to a single `"temporal"` validator. The orchestrator-dispatched `/validate` mechanism this plan built still stands; only the keying changed (open `orchestration_mode` token, single registration). Read the split plan for the current surface; this doc is kept for history. (The "nothing coded yet" status below is itself stale — this plan was completed; see its sibling `execute-per-request-execution-mode.md` for the as-built cross-references.)++**Status:** ready-to-execute plan for a fresh session. Design is locked (decisions below); nothing coded yet. This is **Plan 1 of 2** — its sibling is `wip/plugins/execute-per-request-execution-mode.md` (run this one first). It **reverses/extends locked decision F2** of the orchestrator-agnostic-runner effort (tracker: `_plugins/TODOS.md`; parent plan: `_plugins/wip/plugins/orchestrator-agnostic-runner-and-flavors.md`).++> **START HERE (cold start).** Read this whole doc, then §"Current state" and §"Background". Begin on the branches named in §"Cross-repo state". The work spans `pipelex` (core seam) → `pipelex-temporal` (wire the surviving validate machinery into the seam) → `pipelex-api` (mode-aware dispatch + route). Mistral is a gated follow-up (do **not** build it). Same one-commit-per-checkpoint / clean-context-`/code-review` discipline as the rest of the effort.++---++## 1. The ask + locked decisions++**The ask (user, verbatim intent):** make `/validate` dispatchable through the active orchestrator, the way `/start` runs a pipe — instead of always validating in-process on the API runner.++1. **Temporal flavor** → `/validate` can dispatch the surviving `act_dry_validate` activity to a worker (restore the behavior F2 removed).+2. **Do both** → keep the in-process path for the agnostic base / `direct` mode AND add the dispatched path for orchestrator flavors. Validate becomes `execution_mode`-aware: `direct` → in-process; `temporal_*` → worker activity. The base validates in-process because it has no worker — expected.+3. **Mistral flavor** → the seam must be **generic across orchestrators**, not Temporal-specific, so a Mistral validate arm can be added later.++**Locked decisions (do not re-litigate):**++- **A — Verdict as value (return a union).** The seam **returns** the verdict, it does not raise it. Valid → the canonical report; invalid → the structured invalid report. Respect the existing `/validate` contracts in `docs/specs/` and `conformance/` and their spirit (200-always for any produced verdict; non-2xx = no verdict).+- **B — Seam returns the assembled report.** The worker returns the raw artifact bundle (`DryValidateResult`); the **orchestrator plugin** (running in-API) assembles the canonical report via core `build_validation_report`. The API route stays thin and uniform. (Revisit only if this turns out materially more complex than the in-route assembly alternative — see §"Open sub-decisions".)+- **C — Mistral deferred** as a follow-up (gated on its own `mistralai-2.x` work). Build the generic seam now so Mistral slots in later.+- **D — Reuse `PipelexExecutionMode`.** Validate is inherently blocking, so `temporal_blocking` and `temporal_fire_and_forget` collapse for validate: the Temporal plugin registers the **same** validator under **both** temporal modes. `/validate` reuses `resolve_execution_mode` + the override policy wholesale.+- **Per-call, presence-based, NOT boot-gated.** Validate dispatch joins the **per-call registry family** (like `/start`), not the boot-global hub-slot family (like `/execute` today). The validator is registered **unconditionally** by the plugin (exactly how `pipelex-temporal` already registers its run orchestrators); only hub-slot *claiming* is `boot_orchestrator`-gated, and validate claims no slot. This resolves the brief's old open question #5: `dispatch_dry_validate` needs only a reachable Temporal client + a worker with the validate workflow registered — **not** a runner booted-as-Temporal.++---++## 2. Current state (post-Phase C — the thing we are reversing)++Phase C made `pipelex-api` orchestrator-agnostic (`pipelex-api` tip `a39841e` on `refactor/orchestrator-agnostic-base`). As part of it, **F2** collapsed `/validate` to a single in-process path. F2 verbatim (from the parent plan, §3):++> **F2 — Validate always DIRECT in-process.** Drop the Temporal validate branch entirely; `ApiRunner.validate()` always calls `validate_bundles_in_process(...)`. Removes the `pipelex.temporal` imports (`dispatch_dry_validate` / `DryValidateArg`). Behavior change to call out at handoff: the API runner now loads the method library to validate, which the prior D10/D14 design deliberately pushed onto the worker. Acceptable per decision; note the resource/isolation implication on the hosted runner in that repo's docs.++So today:++- `pipelex-api/api/routes/pipelex/pipeline.py` `ApiRunner.validate` (lines `182-217`) calls only `validate_bundles_in_process(...)` — always in-process, loads the method library on the runner, **no `execution_mode` awareness**. Docstring at line `73` and the route comment at `190-193` assert "always DIRECT in-process (F2)" — **these are the sentences to edit.**+- The route `api/routes/pipelex/validate.py` (lines `141-201`) catches only `ValidateBundleError` (its old `WorkflowExecutionError` catch was removed).+- **The worker machinery survives intact and tested in `pipelex-temporal`, just orphaned** (no longer called by the API): `pipelex_temporal/tprl_pipe/{act_dry_validate,wf_dry_validate,dry_validate_dispatch}.py`, plus unit + in-memory-Temporal integration tests. **This is a re-wiring, not a rebuild.**+- **Phase C dropped `pipelex-api`'s dependency on `pipelex-temporal` entirely.** `pipelex-api/pyproject.toml` deps are now just `pipelex`, `mthds`, `fastapi`, `pyjwt`, `uvicorn`. The agnostic base must stay temporal-free **even in tests** (see §"Testing").++The old dual-path `ApiRunner.validate` (the exact code to mirror) is one commit back: `git -C pipelex-api show a39841e^:api/routes/pipelex/pipeline.py`. Its Temporal arm dispatched **first** (`dispatch_dry_validate(DryValidateArg(mthds_contents, mthds_sources, allow_signatures))`) so every failure surfaced through the worker's `validate_bundle` cascade with the same categorized `ValidateBundleError` identity the direct path raises; then it parsed blueprints **cheaply (no library)** via `PipelexInterpreter.make_pipelex_bundle_blueprint(...)` and assembled the canonical report via `build_validation_report(blueprints=…, pipe_io_contracts=result.pipe_io_contracts, dry_run_result=result.dry_run_outputs, pending_signatures=result.pending_signatures, graph_spec=result.graph_spec)`. The route's old arm caught `WorkflowExecutionError`, recovered the `ValidateBundleError` report (→ 200 invalid) or re-raised a genuine fault (→ 5xx). **Mirror this split**, but as a returned verdict (decision A), not an exception.++---++## 3. Background — the two existing dispatch seams (cold-start orientation)++The runner already has **two** orchestration mechanisms. Validate joins the first.++**(a) Per-call `OrchestratorRegistry`** (the model validate mirrors), all in core under `_plugins/pipelex/` (a worktree of `pipelex`):++- `pipelex/plugins/orchestrator_registry.py:21` — `OrchestratorProtocol.run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput` (single-method `@runtime_checkable` Protocol).+- `pipelex/plugins/orchestrator_registry.py:24-42` — `OrchestratorRegistry(orchestrators: dict[PipelexExecutionMode, OrchestratorProtocol])` with `get_optional(*, mode) -> OrchestratorProtocol | None` and `has(*, mode) -> bool`.+- `pipelex/plugins/registrar.py:163-173` — `PluginRegistrar.add_orchestrator(*, mode, orchestrator)` (uses an internal `_add` helper that fail-louds on duplicates).+- `pipelex/plugins/direct/direct_plugin.py:18-19` — core `DirectOrchestratorPlugin.register` calls `registrar.add_orchestrator(mode=DIRECT, orchestrator=DirectOrchestrator())`; wired unconditionally via `pipelex/plugins/builtins.py:27-28`.+- `pipelex/pipelex.py:391-395` — registries are built **once at boot** from the registrar and stored on the hub; `:206` shows `boot_orchestrator` is a separate boot-global string (claims hub slots) — orthogonal to per-call mode dispatch.+- `pipelex/runtime_bridge/bridge.py:92-95` — per-call dispatch: `get_orchestrator_registry().get_optional(mode=...)` → `MissingOrchestratorError` if absent (no silent fallback) → `orchestrator.run(...)`.+- `pipelex/runtime_bridge/execution_mode.py:4-27` — `PipelexExecutionMode`: `DIRECT`, `TEMPORAL_BLOCKING`, `TEMPORAL_FIRE_AND_FORGET`, `MISTRAL_NATIVE`.+- `pipelex/runtime_bridge/exceptions.py:9-40` — `MissingOrchestratorError` (carries a per-mode pip-install hint).+- `pipelex/runtime_bridge/direct_orchestrator.py` — `DirectOrchestrator.run` returns `serialize_completed_output(pipe_output=…, workflow_id=None)`; scopes an in-process router so nested sub-pipes stay in-process even inside a Temporal worker.+- `pipelex-temporal/pipelex_temporal/temporal_plugin.py:126-147` — `TemporalPlugin.register` adds the run orchestrators **unconditionally** under `TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`; `claim_*` hub slots only `if boot_orchestrator == "temporal"`. Entry point: `[project.entry-points."pipelex.plugins"] temporal = "pipelex_temporal.temporal_plugin:TemporalPlugin"`.++**(b) Boot-global hub slots** — `get_pipe_run()` / `get_pipe_router()` etc., claimed at boot when `boot_orchestrator == "temporal"`. This is what `/execute` uses today (sibling plan converts it to (a)). Validate does **not** touch this.++**Core validate building blocks** (reused as-is, all under `_plugins/pipelex/`):++- `pipelex/pipeline/validate_in_process.py:44-51` — `async validate_bundles_in_process(*, mthds_contents, mthds_sources=None, library_dirs=None, allow_signatures=False, log_context="validate") -> PipelexValidationReport`. Loads + tears down the method library internally (heavy, lifecycle-managed). Raises `ValidateBundleError` on an invalid bundle.+- `pipelex/pipeline/validation_report.py:63-94` — `build_validation_report(*, blueprints, pipe_io_contracts, dry_run_result, pending_signatures, graph_spec=None) -> PipelexValidationReport`. The single assembly point.+- `pipelex/pipeline/validation_report.py:29-60` — `PipelexValidationReport` (extends the neutral `ValidationReport`; `is_valid: Literal[True]` is the valid-arm discriminant; Pipelex-branded envelope).+- `pipelex/pipeline/exceptions.py:72-161` — `ValidateBundleError`; `.to_error_report()` (line `129`) → `ErrorReport` with `validation_errors` populated.+- `pipelex/base_exceptions.py:309-349` — `ErrorReport` (`validation_errors: list[ValidationErrorItem] | None`); `:269-307` `ValidationErrorItem`.+- `pipelex/core/interpreter/interpreter.py` — `PipelexInterpreter.make_pipelex_bundle_blueprint(*, mthds_content, mthds_source)` (cheap parse, no library).++**Surviving Temporal validate machinery** (in `pipelex-temporal`, reused as-is):++- `pipelex_temporal/tprl_pipe/act_dry_validate.py:62-74` — `DryValidateArg(mthds_contents, mthds_sources, allow_signatures, pipe_code)` (plain `BaseModel`, no `temporalio`); `:77-98` — `DryValidateResult(dry_run_outputs, graph_spec, pending_signatures, pipe_io_contracts)`. **These fields map 1:1 onto `build_validation_report`.**+- `pipelex_temporal/tprl_pipe/dry_validate_dispatch.py:22-66` — `async dispatch_dry_validate(arg, *, task_queue=None, should_auto_connect_temporal=True) -> DryValidateResult`. Needs a reachable Temporal client + a worker with `WfDryValidate`/`act_dry_validate` registered; **not** boot-as-Temporal. An invalid bundle comes back as `WorkflowExecutionError` carrying the recovered `ErrorReport` (`error_type == "ValidateBundleError"`); a genuine fault recovers no such report.+- Tests to mirror: `tests/unit/pipelex_temporal/test_dry_validate_dispatch.py`, `tests/integration/pipelex_temporal/test_dry_validate_activity_in_memory.py`, `tests/integration/pipelex_temporal/test_validate_sweep_stays_in_process.py`.++---++## 4. The design (locked)++A new **generic, per-call validate seam**, mirroring the orchestrator seam exactly. Core declares the contract (SDK-agnostic core types only); the plugin implements it; the API resolves the mode and dispatches. **The Temporal wire payloads (`DryValidateArg`/`DryValidateResult`) stay in `pipelex-temporal`** — the seam never exposes them, so no payload migration to core is needed.++### 4.1 Core seam (proposed names — finalize in implementation)++- **Verdict union** (core types only): `BundleValidationVerdict = PipelexValidationReport | ErrorReport`. Valid arm = `PipelexValidationReport`; invalid arm = `ErrorReport` (carrying `validation_errors`). This is exactly what the route already maps to the wire (`ValidReport` / `_invalid_report_response`). Discriminate in the route by `isinstance(verdict, PipelexValidationReport)`.+- **Protocol** `BundleValidatorProtocol` (e.g. `pipelex/plugins/bundle_validator_registry.py`):++ ```+ async def validate_bundles(+ self, *, mthds_contents, mthds_sources, allow_signatures, library_dirs,+ ) -> BundleValidationVerdict: ...+ ```++ Returns the verdict (valid report | invalid `ErrorReport`); **raises only** for no-verdict infra faults (→ global 5xx). Note: `library_dirs` is host context the **in-process** arm needs; the Temporal arm ignores it (the worker loads its own library).+- **Registry** `BundleValidatorRegistry(validators: dict[PipelexExecutionMode, BundleValidatorProtocol])` with `get_optional(*, mode)` / `has(*, mode)` — copy `OrchestratorRegistry`.+- **Registrar** `PluginRegistrar.add_bundle_validator(*, mode, validator)` — copy `add_orchestrator` (reuse the `_add` dup-guard helper).+- **Hub** `get_bundle_validator_registry()` + boot wiring in `pipelex/pipelex.py` next to the orchestrator registry (built once at boot from the registrar).+- **Missing-validator error** `MissingBundleValidatorError(mode=...)` — copy `MissingOrchestratorError`, with a per-mode install hint.+- **Core direct impl** `DirectBundleValidator` (registered by the core `direct` plugin, alongside `DirectOrchestrator`): calls `validate_bundles_in_process(mthds_contents=…, mthds_sources=…, library_dirs=…, allow_signatures=…, log_context="API validate")`; **catches `ValidateBundleError` → returns `exc.to_error_report()`**; returns the `PipelexValidationReport` on success. Any other exception propagates (→ 5xx). This gives the agnostic base in-process validate "for free" via the same seam.++### 4.2 Temporal impl (in `pipelex-temporal`)++- **`TemporalBundleValidator`** (implements `BundleValidatorProtocol`): builds `DryValidateArg(mthds_contents, mthds_sources, allow_signatures)` (ignores `library_dirs`) → `await dispatch_dry_validate(arg)`. On success (`DryValidateResult`): parse blueprints cheaply with `mthds_sources` threaded (so success-path `bundle_blueprint.source` matches the failure path), then `build_validation_report(blueprints=…, pipe_io_contracts=result.pipe_io_contracts, dry_run_result=result.dry_run_outputs, pending_signatures=result.pending_signatures, graph_spec=result.graph_spec)` → return the `PipelexValidationReport`. On `WorkflowExecutionError`: recover the `ErrorReport`; if it is a validation verdict (`error_type == ValidateBundleError.__name__`) → **return** the `ErrorReport` (invalid verdict); otherwise **re-raise** (genuine fault → 5xx).+- **Register it unconditionally** in `TemporalPlugin.register` under **both** `TEMPORAL_BLOCKING` and `TEMPORAL_FIRE_AND_FORGET` (decision D), right next to the existing `add_orchestrator` calls (which are already unconditional). Claims no hub slot.++### 4.3 API dispatch + route (in `pipelex-api`)++- The dispatch lives on `ApiRunner` (mirrors how `ApiRunner.start` resolves the mode and dispatches), but **returns the verdict union** so the route maps it as a value (decision A). Suggested shape: a method `ApiRunner.validate_verdict(*, mthds_contents, mthds_sources, allow_signatures, requested_execution_mode) -> BundleValidationVerdict` that does `resolve_execution_mode(...)` → `get_bundle_validator_registry().get_optional(mode=...)` (→ `MissingBundleValidatorError` if absent) → `validator.validate_bundles(mthds_contents=…, mthds_sources=…, allow_signatures=…, library_dirs=self.library_dirs)`. Keep `ApiRunner` as the holder of library config (unchanged `self.library_dirs` sourcing).+- **The protocol method** `ApiRunner.validate(...) -> PipelexValidationReport` (the `PipelexMTHDSProtocol` `@override`) stays raise-based for protocol conformance: it can delegate to `validate_verdict` and re-raise on the `ErrorReport` arm. The **route uses `validate_verdict`** for the verdict-as-value mapping. (If nothing besides the route calls the protocol `validate`, the cold session may simplify — confirm first.)+- **Route** `validate.py` (`141-201`): replace the `try/except ValidateBundleError` with: resolve render formats (unchanged) → `verdict = await ApiRunner(...).validate_verdict(..., requested_execution_mode=request_data.execution_mode_or_equivalent)` → `if isinstance(verdict, PipelexValidationReport): return <ValidReport built as today>` else `return _invalid_report_response(verdict, requested_formats=...)`. Genuine faults propagate to the global `problem+json` handler. **`ValidReport` / `InvalidReport` / `_invalid_report_response` are unchanged** (`validate.py:79-138, 204-228`) — only the control flow changes from catch-exception to match-on-returned-verdict.+- Wire the `execution_mode` extra into the validate request shape if not already present (the `/validate` request must carry the optional per-request mode the same way `/start` does, gated by `resolve_execution_mode`'s policy). Confirm `ValidateRequest` vs `PipelineApiExtras` and thread it consistently.++### 4.4 Why the wire contract is preserved (conformance-neutral)++The valid/invalid wire arms (`ValidReport`, `InvalidReport`), the 200-always verdict contract, and the structured `validation_errors[]` are **identical** regardless of backend. The dispatched arm assembles the **same** `PipelexValidationReport` via the **same** `build_validation_report`, and surfaces the **same** `ValidateBundleError`-derived `ErrorReport` on invalid. So the existing `/validate` conformance must pass **unchanged** — that byte-identity is the acceptance bar (§"Testing").++---++## 5. Phased implementation + checkpoints++Each phase lands in its own repo on its own branch, one commit per checkpoint. Cut new branches from the named tips.++### Phase V0 — core seam (`pipelex`, in `_plugins/`, branch off `refactor/Plugins-4`)++- Add the protocol, registry, registrar method, hub getter, boot wiring, `MissingBundleValidatorError`, the `BundleValidationVerdict` type, and `DirectBundleValidator` (registered by the `direct` plugin). Mirror the orchestrator seam file-for-file.+- Gates: `make agent-check`, `make tb` (boot/config sanity — the new registry must build at boot), `make agent-test`.++### Phase V1 — Temporal validator (`pipelex-temporal`, branch off `refactor/own-temporal-config` @ `6d9adff`)++- Implement `TemporalBundleValidator`; register it unconditionally under both temporal modes. Reuse `dispatch_dry_validate` / `DryValidateResult` as-is.+- Add an in-memory-Temporal integration test (mirror `test_dry_validate_activity_in_memory.py`): the validator returns an assembled `PipelexValidationReport` for a valid bundle and an `ErrorReport` (with `validation_errors`) for an invalid one; a genuine fault re-raises.+- Gates: `make agent-check`, `make agent-test`.++> **Checkpoint V-A** (MAJOR boundary, natural session break): core + `pipelex-temporal` green together; the seam exists and the Temporal plugin contributes through it, verified against the in-memory Temporal env. The base still has no idea Temporal exists. Fan out a clean-context `/code-review` on the core seam diff + the `pipelex-temporal` diff.++### Phase V2 — API dispatch + route + F2 reversal (`pipelex-api`, branch off `refactor/orchestrator-agnostic-base` @ `a39841e`)++- Add `ApiRunner.validate_verdict`, rewire the route to verdict-as-value, thread `execution_mode` into the validate request, keep `ValidReport`/`InvalidReport`/`_invalid_report_response` unchanged.+- Reverse the F2 prose: the docstring at `pipeline.py:73` and the route comment at `190-193`; restore the dual-backend description in `pipelex-api/docs/pipe-validate.md` (the runner-sizing/library-load note becomes flavor-conditional, not absolute).+- Gates: `make agent-check`, `make agent-test`. **Keep `pipelex-temporal` out of `pipelex-api`'s deps** — test the dispatch+route+verdict-mapping with a **stub** `BundleValidatorProtocol` registered for a test mode (no temporal import). The real Temporal arm is covered in V1.++### Phase V3 — bookkeeping + spec/conformance++- Update the parent plan's **F2** decision (mark it reversed/extended; point to this doc) in `_plugins/wip/plugins/orchestrator-agnostic-runner-and-flavors.md`, and the Phase C as-built note in `_plugins/TODOS.md`. Add a `CHANGELOG.md` entry (dual-backend `/validate`).+- `docs/specs/`: the verdict contract does **not** change. Optionally add one clarifying sentence that the verdict is backend-independent (in-process **or** orchestrator-dispatched) to `pipelex-mthds-protocol.md` "Validation status codes" (`130-151`) and/or `command-surface-map.md` row 6 — **only if** added with a `> Verified by:` link + a matching `pytest.mark.spec(...)`. If you touch any spec heading, run `make check-spec-links` (in `conformance/`).+- `conformance/`: the existing `/validate` tests (`tests/pipelex_api/test_validate_{valid,diagnostic,render}.py`, `tests/pipelex_agent/test_validate_{envelope,errors}.py`, the `validate-error-qa/` corpus) must pass **unchanged** against the in-process (`direct`) arm. A conformance test of the **dispatched** arm needs Temporal in the conformance env — **defer** it (note the gap explicitly, like the other deferred HTTP arms; the in-memory coverage in V1 is the safety net meanwhile).++> **Checkpoint V-B** (THE gate for validate): `pipelex-api` dispatches validate by mode, the route returns verdict-as-value, F2 prose reversed, all gates green in all three repos, existing `/validate` conformance green unchanged, `make check-spec-links` green if specs touched. Clean-context `/code-review` on the `pipelex-api` diff. Capture an as-built (final names, signatures, divergences, test evidence).++### Phase V-Mistral — DEFERRED (decision C)++- Add a `MistralBundleValidator` later (genuine dispatch mirroring Temporal **or** an in-process passthrough — decide then). Gated on `pipelex-mistralai-workflows`' `mistralai-2.x` work. The generic seam means this lands without touching core or the API.++---++## 6. Cross-repo state, pins, gates++**Branches / tips (verified 2026-06-21):**++- `_plugins` (pipelex core worktree): `refactor/Plugins-4` (this doc is the only dirty file — it overwrites the prior brief).+- `pipelex-api`: `refactor/orchestrator-agnostic-base` @ `a39841e` (clean).+- `pipelex-temporal`: `refactor/own-temporal-config` @ `6d9adff` (clean). **Private** (`LicenseRef-Proprietary` + `Private :: Do Not Upload`) — install via `git+ssh`, never PyPI.+- `pipelex-mistralai-workflows`: `feature/Mistral-native` (deferred — ignore).++**Editable pins (local dev):** `pipelex-api/pyproject.toml:84-85` and `pipelex-temporal/pyproject.toml` both pin `pipelex = { path = "../_plugins", editable = true }`. So a core (`_plugins`) edit is live in both immediately. **`pipelex-api` does not depend on `pipelex-temporal`** (Phase C dropped it) — for any *manual* end-to-end check of the Temporal arm in the API, install `pipelex-temporal` as an editable **dev-only** extra; never add it to the base runtime deps.++**Gates:** core `_plugins` → `make agent-check` / `make tb` / `make agent-test`. `pipelex-api` and `pipelex-temporal` → `make agent-check` / `make agent-test`. `conformance` → `make check-spec-links` (+ its pytest suite for the `/validate` arm).++---++## 7. Testing++- **Core (V0):** unit-test the registry + `DirectBundleValidator` (valid → report; invalid bundle → `ErrorReport` with `validation_errors`; missing mode → `MissingBundleValidatorError`). `make tb` must stay green (boot builds the new registry).+- **Temporal (V1):** in-memory-Temporal integration test of `TemporalBundleValidator` (valid → assembled report; invalid → `ErrorReport`; genuine fault → re-raise). Mirror the existing in-memory validate tests.+- **API (V2):** dispatch+route+verdict-mapping tested against a **stub** validator (no temporal). Assert: `direct` → in-process verdict; a stubbed temporal-mode validator's returned `ErrorReport` → 200 `InvalidReport`; a stubbed raised fault → 5xx; missing-mode → `MissingBundleValidatorError`.+- **Conformance (V3):** existing `/validate` suite green **unchanged** (byte-identical verdicts) = the acceptance bar. Dispatched-arm conformance deferred (note the gap).++---++## 8. Open sub-decisions for the implementing session++- **Final names/locations** for the seam (`BundleValidator*`, file homes). Proposed above; confirm against the orchestrator-seam layout.+- **Decision B sanity check:** if in-plugin assembly (Temporal arm calls `build_validation_report`) proves more tangled than expected, the fallback is in-route assembly (route gets `DryValidateResult` and assembles) — but that would re-expose worker payloads to the API, so prefer in-plugin. Flag if you hit friction.+- **Protocol `validate` fate:** keep as a raise-based conformance wrapper delegating to `validate_verdict`, or simplify if the route is its only caller. Confirm callers first.+- **`execution_mode` on the validate request:** confirm the request model/extras plumbing matches `/start`'s, and that the override policy (403 on forbidden override) applies identically.++---++## 9. Progress / as-built (live)++> Appended at each checkpoint. Final names, divergences, test evidence — enough to cold-resume.++### Phase V0 — core seam — DONE (uncommitted at Checkpoint V-A)++Mirrors the orchestrator seam file-for-file. Final names/locations:++- `pipelex/plugins/bundle_validator_registry.py` — `BundleValidatorProtocol` (`@runtime_checkable`, single async `validate_bundles(*, mthds_contents, mthds_sources, allow_signatures, library_dirs)`), `BundleValidatorRegistry` (`get_optional` / `has` / `modes`), and the `BundleValidationVerdict` type alias.+- `pipelex/pipeline/direct_bundle_validator.py` — `DirectBundleValidator` (calls `validate_bundles_in_process(..., log_context="API validate")`; catches `ValidateBundleError` → returns `exc.to_error_report()`; other exceptions propagate).+- `pipelex/plugins/exceptions.py` — `DuplicateBundleValidatorError`. `pipelex/plugins/registrar.py` — `bundle_validators` store + `_bundle_validator_sources` + `add_bundle_validator` (reuses the `_add` dup-guard). `pipelex/plugins/direct/direct_plugin.py` — registers `DirectBundleValidator` under DIRECT alongside `DirectOrchestrator`.+- `pipelex/runtime_bridge/exceptions.py` — `MissingBundleValidatorError(mode=...)` (per-mode install hint, mirrors `MissingOrchestratorError`). `pipelex/hub.py` — field + setter + getter + module `get_bundle_validator_registry()`. `pipelex/pipelex.py` — boot wiring `set_bundle_validator_registry(BundleValidatorRegistry(plugin_registrar.bundle_validators))` next to the orchestrator registry.++**Divergence from §4.1 (the only one, forced by `reportImportCycles = true`):** the verdict's valid arm is typed at the **MTHDS-protocol base `ValidationReport`** (a leaf type), NOT the concrete `PipelexValidationReport`. Naming the concrete envelope from the hub-reachable registry closes a real import cycle (`hub → bundle_validator_registry → validation_report → bundle_validator → pipe_run → hub`) — pyright follows `TYPE_CHECKING` edges, so the cycle gates. This mirrors how the orchestrator seam returns the leaf `PipelexPipeRunOutput`, and is brand-correct (the generic seam speaks the protocol report; the Pipelex envelope is recovered at the API edge). `DirectBundleValidator` still returns the precise `PipelexValidationReport | ErrorReport` (a covariant narrowing); the API route recovers the precise valid arm for its `isinstance(verdict, PipelexValidationReport)` narrowing. Acceptance note for V2: `ApiRunner.validate_verdict` will narrow/cast the protocol's `ValidationReport` valid arm back to `PipelexValidationReport` (single documented point).++Tests: `tests/unit/pipelex/plugins/test_bundle_validator_registry.py` (registrar+registry+dup), `tests/unit/pipelex/pipeline/test_direct_bundle_validator.py` (verdict-as-value branches, mocked sweep), `tests/unit/pipelex/runtime_bridge/test_missing_bundle_validator_error.py` (per-mode hints). Gates: `make agent-check` green, `make tb` green, `make agent-test` green.++### Phase V1 — Temporal validator — DONE (uncommitted at Checkpoint V-A)++- `pipelex-temporal/pipelex_temporal/temporal_bundle_validator.py` — `TemporalBundleValidator` (import-light; `temporalio` + dispatch/wire/exception types resolved lazily inside `validate_bundles`, mirroring the orchestrators). Dispatches `dispatch_dry_validate(DryValidateArg(...))`; on success parses blueprints cheaply (`PipelexInterpreter.make_pipelex_bundle_blueprint`, threading `mthds_sources`) → `build_validation_report(...)`; on `WorkflowExecutionError` recovers the report and **returns** it iff `error_type == ValidateBundleError.__name__` (invalid verdict), else **re-raises** (genuine fault → 5xx). `library_dirs` ignored (worker loads its own library). Copied verbatim from the old dual-path `ApiRunner.validate` Temporal arm, but as a returned verdict (decision A).+- `temporal_plugin.py` — registers the SAME instance under **both** `TEMPORAL_BLOCKING` and `TEMPORAL_FIRE_AND_FORGET` (decision D), unconditionally, claims no hub slot.++Tests: `tests/integration/pipelex_temporal/test_temporal_bundle_validator_in_memory.py` (real worker, in-memory server: valid → assembled `PipelexValidationReport`; invalid → `ErrorReport` verdict; patches `is_temporal_boot_active` + `get_temporal_client` to route the validator's own `dispatch_dry_validate` through the in-memory env), `tests/unit/pipelex_temporal/test_temporal_bundle_validator.py` (verdict-vs-fault branching, mocked dispatch). Gates: `make agent-check` green, `make agent-test` green.++**Deferred observation:** dispatched `/validate` effectively requires `boot_orchestrator == "temporal"` (the reused executor's `is_temporal_boot_active()` guard), slightly at odds with the plan's decision #5 "not a runner booted-as-Temporal" — but identical to the run orchestrators' behavior, pre-existing, out of scope. Captured in [`validate-dispatch-requires-boot-active.md`](validate-dispatch-requires-boot-active.md).++> **Checkpoint V-A closed.** Core green + `pipelex-temporal` green together; the seam exists and the Temporal plugin contributes through it (verified against the in-memory Temporal env). Clean-context `/code-review` on both diffs: **zero findings, no fixes** (core: 3 independent passes; temporal: each flagged path verified against ground truth). One latent observation deferred ([`validate-verdict-subclass-parity-divergence.md`](validate-verdict-subclass-parity-divergence.md) — the `error_type` string-equality predicate, copied verbatim from the old route, would diverge from the direct arm's `except` only if a `ValidateBundleError` subclass is ever introduced; none exists). Commits: core `de709eea4`, `pipelex-temporal` `459e04d` (branch `feature/Orchestrator-dispatched-validate` in each).++### Phase V2 — API dispatch + route + F2 reversal — DONE (`pipelex-api`)++- `api/routes/pipelex/pipeline.py` — **dropped the `ApiRunner.validate` `@override`** (its only caller was the route; the parity test uses the base `PipelexMTHDSProtocol.validate`) and added **`ApiRunner.validate_verdict(*, mthds_contents, mthds_sources, allow_signatures, requested_execution_mode) -> PipelexValidationReport | ErrorReport`**. Mirrors `start`: `resolve_execution_mode(...)` FIRST (forbidden override → 403 before any dispatch) → `get_bundle_validator_registry().get_optional(mode=…)` (→ `MissingBundleValidatorError` if absent) → `validator.validate_bundles(..., library_dirs=…)`. The seam's valid arm is typed at the protocol-level `ValidationReport`, so `validate_verdict` recovers the concrete `PipelexValidationReport` with a single documented `cast` — the only narrowing point.+- `api/routes/pipelex/validate.py` — route rewired to **verdict-as-value**: `verdict = await ApiRunner().validate_verdict(...)`; `if not isinstance(verdict, PipelexValidationReport): return _invalid_report_response(verdict, ...)` else build `ValidReport`. Removed the `try/except ValidateBundleError`. Added `execution_mode: PipelexExecutionMode | None` to `ValidateRequest` (same plumbing as `/start`). `ValidReport`/`InvalidReport`/`_invalid_report_response` unchanged.+- **F2 prose reversed:** the `ApiRunner` class docstring, the `validate_verdict` docstring, the route docstring, and `docs/pipe-validate.md` ("Where validation runs" + the resource note + the 403 in the no-verdict list) now describe `execution_mode`-aware dual-backend validation (flavor-conditional library load) instead of "always DIRECT in-process."++**Sub-decision resolved (Protocol `validate` fate):** dropped, not kept as a raise-based wrapper. The route is the only `ApiRunner.validate` caller and switches to `validate_verdict`; re-raising from the returned `ErrorReport` would fabricate an exception no consumer catches (the original `ValidateBundleError` was already converted to a report by the validator). `ApiRunner` inherits the base `PipelexMTHDSProtocol.validate` (in-process — a sensible agnostic-base fallback, unused in practice).++Tests: `tests/unit/test_validate_dispatch.py` (NEW — stub validator for a non-direct mode proves registry dispatch + verdict mapping; invalid `ErrorReport` → 200, raised fault → 5xx, `MissingBundleValidatorError`, forbidden override → 403; no `temporalio` import). Three tests in `test_validate_errors.py` that mocked the removed `ApiRunner.validate` were re-pointed at `validate_verdict` (verdict-as-value: invalid arms now **return** the `ErrorReport`, the fault arm still **raises**) — the wire output is byte-identical. All other `/validate` tests (envelope, render, parity, conformance, allow_signatures) pass **unchanged** against the real direct arm. Gates: `make agent-check` green, `make agent-test` green (full suite).++### Phase V3 — bookkeeping + spec/conformance — DONE++- Parent plan F2 marked **REVERSED / EXTENDED** in [`orchestrator-agnostic-runner-and-flavors.md`](orchestrator-agnostic-runner-and-flavors.md) and in `_plugins/TODOS.md` (locked-decisions F2 + Phase C as-built). `pipelex-api` `CHANGELOG.md` `[Unreleased]`: the F2 "always DIRECT" bullet was rewritten in place (F2 never shipped) to the dual-backend verdict-as-value `/validate`; the `execution_mode` config bullet now names `/v1/validate` alongside `/v1/start`.+- **Spec / conformance:** no spec heading touched — the verdict contract (200-always, `is_valid` discriminant, structured `validation_errors[]`) is **unchanged**, and no spec or conformance test asserts the backend, so `make check-spec-links` is not needed. The direct-arm verdict wire is proven byte-identical by the `pipelex-api` suite (`test_validate_*`, `test_protocol_parity`, `test_protocol_conformance`, all green). The separate `conformance/` repo's live-wire `/validate` suite (`test_validate_{valid,diagnostic,render}.py`) is therefore expected green unchanged; a conformance test of the **dispatched** arm needs Temporal in the conformance env and is **deferred** (the V1 in-memory coverage is the safety net), exactly as the other deferred HTTP arms.++> **Checkpoint V-B — THE gate for validate.** `pipelex-api` dispatches validate by mode, the route returns verdict-as-value, F2 prose reversed across code + docs + trackers, all gates green in all three repos, existing `/validate` conformance green unchanged. Clean-context `/code-review` on the `pipelex-api` diff pending at this checkpoint.diff --git a/wip/plugins/orchestrators-as-plugins.md b/wip/plugins/orchestrators-as-plugins.mdnew file mode 100644index 000000000..226abb881--- /dev/null+++ b/wip/plugins/orchestrators-as-plugins.md@@ -0,0 +1,106 @@+# Orchestrators as Pipelex plugins++**Status:** assessment / design notes (not started)+**Scope:** turn the *orchestrators* — the strategies that decide **where and how** a pipe runs (in-process, on a Temporal worker fleet, on Mistral Workflows) — into discoverable plugins, so core ships only the in-process orchestrator and every distributed one is an optional, separately-installable plugin wired through one seam.++> An **orchestrator** decides how a whole pipe job is scheduled and where it executes — distinct from a *pipe controller*, which sequences sub-pipes *within* a single run. (We deliberately avoid "backend" here: that word already names the inference SDK adapters in the [driver doc](inference-backends-as-plugins.md).) This is the **strategy** half of the plugin story: orchestrators *replace core orchestration behavior*. The **driver** half — inference SDK wrappers selected by a model's `sdk` handle — is in [`inference-backends-as-plugins.md`](inference-backends-as-plugins.md). Temporal's concrete seam-by-seam extraction lives in [`temporal-as-plugin.md`](temporal-as-plugin.md), which references this doc for the shared mechanism. Cross-cutting best practices are in [`README.md`](README.md).++## The category++An orchestrator answers *"where/how does this pipe run?"* — not *"which model/SDK?"* (that's a driver). Pipelex already models the choice as `PipelexExecutionMode` in `runtime_bridge/execution_mode.py`:++- **`DIRECT`** — in-process async. The trivial orchestrator. **Ships in core.**+- **`TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`** — distributed via Temporal. Lives in `pipelex/temporal/` today; destined for `pipelex-temporal`.+- **`MISTRAL_NATIVE`** — distributed via Mistral Workflows. Lives in the external `pipelex-mistralai-workflows` repo.++`DIRECT` is the only one that belongs in core. The other two are **peers**: optional, discovered, neither privileged. Today core fails that test — it names *both* external orchestrators by string (see the seam below). Mistral is the proof that "orchestrator" is a *category with ≥2 members*, so the seam must be generic and N-ary, not "Temporal plus a special case."++## Two worked instances++| | `DIRECT` (core) | `pipelex-temporal` | `pipelex-mistralai-workflows` |+|---|---|---|---|+| **Maps a pipe onto** | in-process async calls | Temporal workflows + activities | Mistral Workflows workflows + activities |+| **Pipelex contracts implemented** | core default | `PipeRouterProtocol`, `PipeRunProtocol`, `ContentGeneratorProtocol`, task manager | `PipeRunProtocol`, `PipeRouterProtocol` |+| **How core couples to it *today*** | n/a | lazy imports gated on `get_config().temporal.is_enabled` (hub injection in `pipelex.py`) + `bridge._run_temporal_*` + **hard** CLI imports in `_cli.py` | **hard-coded import** in `bridge._run_mistral_native` of `pipelex_mistralai_workflows.primitives.pipe_run` + a `requires_mistral_workflows_extra` gate |+| **Activation model** | default | **boot-global** (hub swap when enabled) **+ per-call** modes | **per-call** (`execution_mode`); installs its router *inside* the workflow at runtime |+| **Also a plugin for…** | — | its host: register workflows/activities with a Temporal worker | its host: register workflows/activities with a Mistral worker (`register_pipelex_primitives`) |++Two different couplings, same fix: invert them through one registry.++## The seam: an execution-mode → orchestrator registry++`bridge.py` currently does a `match input_payload.execution_mode:` whose arms hard-code both external orchestrators — the `TEMPORAL_*` arms lazily import `pipelex.temporal`, the `MISTRAL_NATIVE` arm lazily imports `pipelex_mistralai_workflows`. That is **two string couplings** to remove.++Replace the `match` with a registry keyed by execution mode. Core registers `DIRECT`; each orchestrator plugin registers its mode(s) through the shared `pipelex.plugins` entry point; the bridge collapses to a lookup:++```python+orchestrator = orchestrator_registry.get(input_payload.execution_mode)+if orchestrator is None:+ raise MissingOrchestratorError(+ mode=input_payload.execution_mode,+ hint="install pipelex-temporal / pipelex-mistralai-workflows",+ )+return await orchestrator.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)+```++**Keep the `PipelexExecutionMode` enum in core.** It is a small, stable *taxonomy of strategies*; core can know the *names* without the *implementations*. That is what lets the error above name the missing package, and keeps the `requires_pipelex_temporal` / `requires_mistral_workflows_extra` gates statically typed. Plugins supply *orchestrators* for existing enum values — they do not invent modes. (Third-party-defined modes would mean an open, string-keyed mode space; that's a larger change, defer it until something actually needs it.)++## Two activation models → the `Plugin` protocol is a menu of hooks++The orchestrators do not use the seam the same way, and the protocol must not be modelled on Temporal alone:++- **Per-call dispatch (primary).** The bridge picks an orchestrator by `execution_mode` for each run. Both Temporal modes *and* Mistral use this. Hook: **`register_orchestrators(registry)`**.+- **Boot-global swap (Temporal only).** When `temporal.is_enabled`, the default router / run / content-generator / task-manager are swapped at boot via hub injection — necessary because *inside a Temporal worker* the content generator must dispatch each operation as an activity. Hook: **`register_hub_implementations(hub, config)`** + lifecycle.+- **Mistral needs no boot swap.** It installs `MistralWorkflowsPipeRouter` *inside* the child workflow at runtime, per call — so it uses `register_orchestrators` and essentially nothing at boot. It is the counter-example that keeps the protocol honest: hooks are a **menu**, each plugin implements the subset it needs.+- **CLI is optional.** Temporal contributes `worker` + `setup-temporal-namespace`; Mistral contributes none (the worker is the user's Mistral worker). Hook: **`register_cli_commands(app)`**.++## The orchestrator SPI (published surface)++A clean plugin must depend only on a *designed* surface, never reach into internals. Where the two orchestrators sit today:++- **Temporal** depends on hub setters + the three protocols + its config field — mostly already protocol-shaped.+- **Mistral** reaches deeper, and says so: its own `wip/boundary-violation-mistral-native.md` documents that the native tier imports `pipelex.hub`, `pipelex.pipe_run.*`, `pipelex.core.*`, `pipelex.graph.*`, `pipelex.tracing.*` — well outside the declared `runtime_bridge.*`-only surface.++That is a **signal, not a sin**: running a pipe out-of-process genuinely needs library/crate access (`hub`), the `PipeJob` / `PipeOutput` types (`core`), the router/run protocols (`pipe_run`), and the trace/tracer hooks (`graph` / `tracing`). The fix is to **publish that set as an orchestrator SPI**, not to keep an honor-system "runtime_bridge only" rule that's already breached. Define the SPI as:++- the orchestrator registry + the `PipeRouterProtocol` / `PipeRunProtocol` / `ContentGeneratorProtocol`;+- `PipeJob` / `PipeOutput` and the boundary payload types;+- library-crate access (for per-call library hydration — Mistral already does this via `library_crate_dump`);+- the tracing/graph hooks an orchestrator must integrate with.++Anything an orchestrator imports *outside* the SPI is a design bug to resolve — promote it into the SPI, or remove the need. (This is the same theme as the OpenAI *substrate* becoming a deliberate contract in the [driver doc](inference-backends-as-plugins.md): once an out-of-tree consumer depends on you, the dependency surface must be designed, not accidental.)++## One repo, two hosts (the adapter shape)++`pipelex-mistralai-workflows` depends on **both** `pipelex` and `mistralai-workflows`. It is a bridge/adapter: a Pipelex orchestrator plugin on one side, a Mistral Workflows activity/workflow library on the other. The two "plugin" directions use **different mechanisms**, and conflating them causes confusion:++- **→ Pipelex: entry-point discovery.** Declare `[project.entry-points."pipelex.plugins"]`; installing the wheel makes `MISTRAL_NATIVE` available and core stops hard-coding the import.+- **→ Mistral: registration-by-import.** Mistral has no plugin registry; the user (or the package's `register_pipelex_primitives(workflows, activities)` helper) hands the workflow/activity classes to their worker. No entry point — and nothing to change. (Temporal is identical on *its* host: you always register workflows/activities with a worker explicitly.)++```toml+[project]+name = "pipelex-mistralai-workflows"+dependencies = ["pipelex>=X", "mistralai-workflows>=3.4.0"] # depends on BOTH hosts++# Pipelex side — discovery+[project.entry-points."pipelex.plugins"]+mistral-native = "pipelex_mistralai_workflows.plugin:MistralWorkflowsOrchestrator"++# Mistral side — NO entry point: users call register_pipelex_primitives(...) on their worker.+```++This is the natural shape for an adapter — **don't split it.** The host-agnostic decomposition plumbing (pipe classification) already lives in core under `runtime_bridge/primitives/`; everything left is inherently bi-lateral.++## Effort & risk++- **The per-call dispatch inversion is small** — one `match` → a registry lookup; both external orchestrators are already behind lazy/dynamic imports.+- **The SPI definition is the real design work** — settle what core publishes and resolve Mistral's boundary violation against it. This gates a *clean* extraction (the mechanical move is easy; the contract is the part worth getting right).+- **Temporal's boot-global hub-injection inversion is the heavier mechanical piece** — detailed seam-by-seam in [`temporal-as-plugin.md`](temporal-as-plugin.md).+- **Watch:** registration timing — `DIRECT` must always be registered, and a requested-but-absent orchestrator must produce the "install X" error, not a `None` deref. Keep the enum + `requires_*` gates as the typed source of truth for *which modes exist*.++## Suggested phasing++- **Phase 0 — define the seam.** `Plugin` protocol + entry-point discovery + the execution-mode → orchestrator registry + the published orchestrator SPI. Register `DIRECT` in core through it and collapse the bridge `match` to a registry lookup (external orchestrators still resolved as today, now *behind* the registry — no behaviour change). _Checkpoint: the bridge no longer matches on specific external modes; `DIRECT` flows through the registry; suite green._+- **Phase 1 (Temporal) — through the seam.** See [`temporal-as-plugin.md`](temporal-as-plugin.md) Phase 1 (register hub implementations + orchestrators + CLI; resolve the config cut line).+- **Phase 1′ (Mistral) — through the seam.** Replace `bridge._run_mistral_native`'s hard import with the orchestrator the `pipelex-mistralai-workflows` entry point registers; resolve its boundary violation against the published SPI. _Checkpoint: core has zero string references to `pipelex_mistralai_workflows`; `MISTRAL_NATIVE` works purely via discovery._+- **Phase 2 — repackage as needed.** Temporal: extract `pipelex/temporal/` → `pipelex-temporal` (its doc). Mistral: already external — just flip the dev path-pin to a published `pipelex` once the SPI lands.diff --git a/wip/plugins/phase-1-review-deferred.md b/wip/plugins/phase-1-review-deferred.mdnew file mode 100644index 000000000..acd779f45--- /dev/null+++ b/wip/plugins/phase-1-review-deferred.md@@ -0,0 +1,6 @@+# Phase 1 plugin-system review — deferred items++Items surfaced during the `/code-review` pass on the Phase 1 staged diff that are design tradeoffs (not bugs) and were intentionally NOT changed. Left here for later triage.++- `llm_worker_factory.make_llm_worker` computes `ModelHandle.make_for_inference_model(...)` to do the registry lookup by `model_handle.sdk`, and each vendor `_make_*_worker` closure recomputes it internally. The pre-migration `match` arms computed it once. The recompute is a trivial 3-field pydantic construction, off any hot loop (one per worker creation), so the cost is negligible — but if a later phase threads `model_handle` through the `MakeWorkerFn` signature for other reasons, this redundancy disappears for free. Not worth a seam-signature change on its own.+- `PluginDiscovery.targets_api` is recorded from `plugin.targets_api` on the disabled-plugin path (discovery.py) but from the literal `PLUGIN_API_VERSION` on the registered path (via `begin_plugin`). They are equal for any registered plugin (the version-mismatch check gates registration), so this is a stylistic asymmetry with no behavioral effect. Could be unified to always record `plugin.targets_api` if a future API allows a registered plugin to declare a value other than the supported one.diff --git a/wip/plugins/phase-2-review-deferred.md b/wip/plugins/phase-2-review-deferred.mdnew file mode 100644index 000000000..2c3f645a1--- /dev/null+++ b/wip/plugins/phase-2-review-deferred.md@@ -0,0 +1,29 @@+# Phase 2 plugin-seam review — deferred items++Items surfaced by the xhigh `/code-review` pass over the current PR (the inference-backend plugin seam + all four migrated families: LLM/ImgGen/Extract/Search). These are latent footguns reachable only by external entry-point plugins or not-yet-written callers, plus quality cleanups — none produces wrong output on a live path today, so they were intentionally NOT changed and are left here for later triage.++The three CONFIRMED pre-merge gaps from the same pass were fixed in this PR (import-light blocklist completeness, structured lookup-miss error, multi-spec `require_sdk` naming); only the items below remain.++## Latent footguns — decide at the Phase 3 staging boundary whether to gate or document++- **`build_registrar` silently drops three of the five registrar collections.** `pipelex.py` `build_registrar` accumulates `cli_commands` / `slot_claims` / `teardown_callbacks`, but boot only consumes `inference_backends` and `orchestrators`. An external `pipelex.plugins` entry-point plugin that calls `add_cli_command` / `claim_pipe_run` / `add_teardown` registers with no error and shows up in `pipelex plugins list`, yet its command never appears, its hub slot is never wired, and **its teardown never runs at `Pipelex.teardown()`** — any SDK client / external resource it opened leaks for the process lifetime. The discovery + iteration path is fully live, so this is reachable today by a third-party plugin; only the in-tree builtins (which use none of those three menus yet) keep it latent. At minimum, boot should warn on an unconsumed contribution rather than silently ignore it; better, wire the three collections as the orchestrator seam lands in Phase 3.++- **`make_search_worker` can silently drop usage reporting.** `cogt/search/search_worker_factory.py` moved `reporting_delegate` from an unconditional in-factory `get_report_delegate()` fetch to a parameter that **defaults to `None`**. The single production caller (`search_generate.py`) passes it, but any future caller (a Temporal search-activity path, a host runtime, a test) that omits it gets a search worker emitting no usage/cost report — silent zero where the old code always reported. The other three families route their delegate the same way, but search is the one whose factory previously self-fetched, so it is the regression-prone one. Consider making the param non-defaulting (force the caller to decide) or fall back to `get_report_delegate()` when `None`.++- **A callable plugin *instance* is mistaken for a factory.** `plugins/discovery.py` `_load_external_plugin` uses `loaded() if callable(loaded) else loaded`, which will **call** a plugin object whose class defines `__call__` instead of using it as the plugin. If that `__call__` returns `None` or raises, a valid installed plugin is rejected as `BrokenPluginError` with a confusing "failed to load entry point" message. Tighten the factory-vs-instance discrimination (e.g. accept only a class or a plain function as a factory, not any `callable`). Only reachable via an external entry point, so latent until the discovery path has real out-of-tree consumers.++- **The coverage test leans on an unenforced contract.** `tests/unit/pipelex/plugins/test_inference_backend_coverage.py` constructs `PluginRegistrar(config=cast('PipelexConfig', None))` and calls each builtin's `register()`, relying on the invariant that `register()` never reads `self.config`. If a future builtin reads `registrar.config` (a plausible mistake — `config` is an `__init__` param), the test crashes with `AttributeError: 'NoneType'` on a path unrelated to the change under test, masking the real registration logic behind a confusing CI failure instead of a clear contract violation. Either pass a minimal real/stub config, or make the contract explicit (assert/typed null-object) so a violation fails loud and on-point.++## Quality cleanups — fold into the Phase 3 seam threading++- **Redundant `ModelHandle` double-construction.** Already captured in [`phase-1-review-deferred.md`](phase-1-review-deferred.md) (item 1): each family factory builds a `ModelHandle` only to read `.sdk` for the lookup, then every `_make_*_worker` closure rebuilds the identical handle from the same `inference_model`. Disappears for free if a later phase threads `model_handle` through the `MakeWorkerFn` signature; not worth a seam-signature change on its own. Cross-referenced here because the xhigh pass re-surfaced it across all four families and ~two dozen closures.++- **Copy-pasted worker-builder preamble.** The `model_handle = ModelHandle.make_for_inference_model(...)` + `sdk_clients.get_or_create(handle=..., build=lambda: ...)` dance repeats across ~two dozen `_make_*_worker` closures (`google_plugin.py`, `bedrock_plugin.py`, the openai/gateway/portkey families, …), varying only in the build lambda and the worker constructed. A shared helper — `build_cached_client(inference_model, *, sdk_clients, build) -> (model_handle, sdk_instance)` — would shrink each closure to its build lambda plus the worker call, and localize any future change to client caching / handle derivation / teardown registration. (Composes naturally with threading `model_handle` through `MakeWorkerFn` above.)++- **Duplicate img-gen worker builders on the OpenAI-completions substrate.** `_make_openrouter_img_gen_worker` (`openrouter_plugin.py`), `_make_blackboxai_img_gen_worker` (`blackboxai_plugin.py`), and the gateway-completions img-gen variant are the same closure — `OpenAIClientFactory.make_openai_client` → `get_or_create` → `OpenAICompletionsImgGenWorker(...)` — differing only in the completions-factory subclass and the `is_http_url_enabled` flag. The same shape recurs across the gateway/portkey/openai `_make_*_completions_worker` LLM closures. Parameterize on the factory class so a substrate fix lands in one place.++- **Search rebuilds its worker on every call.** `cogt/content_generation/search_generate.py` (`search_gen_sourced_answer` / `search_gen_structured`) constructs a fresh `SearchWorker` per search call — re-resolving model + backend, re-running the registry lookup, and (for the linkup arm) re-probing `importlib.util.find_spec` — where LLM/img_gen/extract return a worker cached in `InferenceManager`. The SDK client itself is cached in `SdkClientRegistry`, but the worker rebuild + resolution + `find_spec` repeat. The per-call `find_spec` on the linkup search arm is **new in this PR** (the migration routed search through `require_sdk`). Cache the search worker per handle the way the other three families do.++- **Dead `self.model_handle` field on `AzureImgGenWorker`.** `plugins/azure_rest/azure_img_gen_worker.py` `__init__` stores `self.model_handle` and reads it exactly once on the next line to re-resolve the backend via `get_models_manager()` — then never references it again. Leftover per-instance state from the `Plugin` → `ModelHandle` rename; the factory already resolved the backend and could hand it over directly, and the test fixtures had to set `worker.model_handle = MagicMock()` purely to satisfy construction. Drop the field and the redundant hub re-resolution; have the factory pass the `InferenceBackend` it already fetched.++- **Missing `-> None` return annotations.** `plugins/sdk_client_registry.py` `teardown` (and `setup` / `teardown` in `sdk_client_manager.py`) omit the `-> None` return annotation, against the "every function return must be typed" standard. Pyright in `make agent-check` does not flag missing return annotations by default, so it passes CI silently. Carried verbatim from the deleted `plugin_sdk_registry.py` / `plugin_manager.py` during the rename — a pre-existing untyped shape, not newly authored. Low cost, no runtime consequence; annotate when these files are next touched.diff --git a/wip/plugins/phase-3-critical-cli-harvest-fragility.md b/wip/plugins/phase-3-critical-cli-harvest-fragility.mdnew file mode 100644index 000000000..95340cb13--- /dev/null+++ b/wip/plugins/phase-3-critical-cli-harvest-fragility.md@@ -0,0 +1,104 @@+# Phase 3 critical finding — the import-time CLI plugin-command harvest is an unguarded surface++> ✅ **RESOLVED BY REMOVAL (Option A).** This finding is moot: the CLI plugin-command harvest — and all four facets below — were **deleted**, not hardened. Temporal's `worker` / `setup-namespace` now ship as the standalone `pipelex-temporal` console script, so `pipelex --help` performs no config load, no entry-point scan, and no dynamic import; no plugin can shadow a core command or brick the recovery commands (`doctor` / `plugins` / `init`), because there is no longer any CLI-command contribution path. Decision + step-by-step: [`option-a-drop-cli-command-seam.md`](option-a-drop-cli-command-seam.md). Everything below is retained for historical context only.+>+> **Status:** ~~open, unsolved~~ **RESOLVED.** **Purpose (historical):** a cold-start problem statement. This doc explains *what was wrong and why it mattered* in enough depth to start a fresh session with no prior context. **It deliberately stopped short of proposing a fix** — the solution space was explored separately (and the chosen answer was to remove the surface). Surfaced by the xhigh `/code-review` over the Phase 3 checkpoint commit (`19e6ca66b` / `5f61db323`).+>+> Read the [TODOS cold-start primer](../../TODOS.md#cold-start-primer-read-this-first-if-youre-new-to-the-session) first for the plugin-seam vocabulary (registrar, `build_registrar`, `BUILTIN_PLUGINS`, slot-claims, D3) — this doc assumes it.++## TL;DR++Phase 3 (decision **D3**) harvests plugin-contributed CLI commands by running the full plugin-discovery pipeline (`build_registrar`) **at module-import time of `pipelex.cli._cli`**, on **every** `pipelex` invocation. That import path is now also doing config loading, external-plugin loading, and dynamic `import_module`/`getattr` — none of it guarded the way the boot path is, and one facet (command-name collision) has no guard *anywhere*. The net effect: a single broken/incompatible/colliding installed plugin, or an unreadable user config file, can make **every** `pipelex` command crash on import — including the very `--help` / `doctor` / `plugins` / `init` commands you would use to diagnose and disable the bad plugin. A fourth facet lets a plugin **silently shadow a core command** (`run`, `validate`, …) with no error or warning.++This is the single dominant correctness cluster of the Phase 3 review. It does not affect any in-tree-only install today (the built-ins contribute no CLI commands that collide, and a present-and-readable config parses), which is why the suite is green — but the surface is live for any external `pipelex.plugins` entry-point plugin, and that is exactly what Phase 5 ships.++## Where this lives++| Element | Location | What it does |+|---|---|---|+| The harvest entry point | `pipelex/cli/_cli.py` — `_register_discovered_cli_commands()` (~`:262`) and the module-level `_PLUGIN_COMMAND_NAMES = _register_discovered_cli_commands()` (~`:280`) | Runs at import of `_cli.py`. The `pipelex` console-script entry point is `pipelex.cli._cli:app`, so importing `app` runs this. |+| The config load for the harvest | `pipelex/cli/_cli.py` — `_config_for_cli_harvest()` (~`:246`) | Loads config without side effects; `except (TomlError, ValidationError)` → falls back to package defaults. |+| The discovery pipeline it calls | `pipelex/plugins/discovery.py` — `build_registrar()` (`:24`) | Iterates `BUILTIN_PLUGINS` then external `pipelex.plugins` entry points; **fail-loud** on every conflict (`BrokenPluginError`, `PluginApiVersionMismatchError`, `DuplicateInferenceBackendError`, `DuplicateOrchestratorError`, `HubSlotAlreadyClaimedError`, `CoreUnconditionalPluginDisabledError`). |+| The dynamic command resolve | `pipelex/cli/_cli.py` (~`:272-276`) | `module_path, _, attribute = import_path.partition(":")` → `getattr(importlib.import_module(module_path), attribute)` → `app.command(name=..., help=...)(command)`. |+| The registrar menu method with no guard | `pipelex/plugins/registrar.py` — `add_cli_command()` (`:135`) | Appends a `CliCommand` with **no** duplicate/collision check — unlike `add_orchestrator` (`:116`), `_claim` (`:145`), and the inference-backend add (`:110`), which all raise and name both contributors. |+| The command list that surfaces them | `pipelex/cli/_cli.py` — `PipelexCLI.list_commands` (~`:54`) returns `[*_CORE_COMMAND_ORDER, *_PLUGIN_COMMAND_NAMES]` | Plugin names are appended after core names. |++For contrast, the **boot** path runs the same `build_registrar` at `pipelex/pipelex.py` `setup()` (~`:390`) — but boot is allowed to fail loud; CLI startup is not.++## The contract this is supposed to honor++`_config_for_cli_harvest`'s own docstring states the intended invariant in plain words:++> "Runs on every `pipelex` invocation (including `--help` / `init`), so it must never create `~/.pipelex/` … and **must survive a broken user config** — a malformed override must not brick the very commands (`init`/`doctor`) that fix it. On any load or validation failure it falls back to the shipped package defaults, which always validate."++`build_registrar`'s docstring states the *other* intended invariant, which is in direct tension with the first:++> "is **fail-loud** on every conflict (duplicate backend/mode/slot, version mismatch, broken plugin)."++The Phase-1 boot-wiring note and the "Invariants that must survive every phase" section of TODOS both make **fail-loud** a deliberate, load-bearing property: *"installed but broken" must be loud, distinct from "not installed" (fine).* The problem is that Phase 3 moved that fail-loud pipeline onto the one path — CLI module import — where failing loud means failing *everything*, recovery commands included. The two invariants were never reconciled at this altitude.++## The problem, in four facets++All four are reachable through one surface: `_cli.py` import now performs config-load + plugin-discovery + dynamic import, none of it degraded for the "this must never brick `--help`" context.++### Facet 1 — `build_registrar()` is not wrapped; a broken external plugin bricks the whole CLI++`_register_discovered_cli_commands()` calls `build_registrar(config=...)` with no surrounding `try`/`except`. `build_registrar` is intentionally fail-loud: it raises on a broken entry point (`BrokenPluginError`), an API-version mismatch (`PluginApiVersionMismatchError`), and any duplicate `(family, sdk)` / duplicate mode / double-claimed slot across plugins. Pre-Phase-3, this pipeline ran **only inside `Pipelex.setup()`**, so a discovery failure could not touch `pipelex --help`.++**Failure scenario.** A user `pip install`s a broken / API-incompatible external pipelex plugin (or two plugins that collide on a mode/backend/slot). The exception now fires during `import pipelex.cli._cli`, before any command function runs. `pipelex --help`, `pipelex doctor`, `pipelex plugins`, and `pipelex init` all crash identically. The diagnostic commands (`doctor` / `plugins`) and the `plugins.disabled` denylist edit reachable via `init` — the documented recovery path for a broken installed plugin — are themselves unreachable. The CLI is fully bricked until the user manually `pip uninstall`s the offending dist by hand.++### Facet 2 — the config-load `except` is too narrow; an unreadable config file bricks the CLI++`_config_for_cli_harvest()` catches only `(TomlError, ValidationError)`. But the loader opens user TOML with `open(path, "rb")` (`pipelex/tools/misc/toml_utils.py:39`, `load_toml_from_path`), which catches only `tomli.TOMLDecodeError`. A file that is **present but unreadable** (wrong permissions, root-owned, on a flaky network mount) raises `PermissionError` / `OSError` from `open()` — neither is a `TomlError` nor a `ValidationError`, so it escapes the harvest's `except` uncaught and aborts the import.++**Failure scenario.** A user whose `~/.pipelex/pipelex.toml` or project `.pipelex/pipelex.toml` is present-but-unreadable gets every `pipelex` command — `--help`, `init`, `doctor` included — to abort at CLI module import with an uncaught `OSError`, instead of falling back to package defaults as the "bulletproof fallback" docstring promises. Again, the commands meant to fix config are unreachable.++> Note: the CHECKPOINT-3 review already added `test_harvest_config_falls_back_to_base_on_broken_user_config` (S1), but that test pins only the **malformed/parse** fallback (`TomlError`/`ValidationError`). The **unreadable/`OSError`** path is a distinct, still-uncovered gap — the existing test does not exercise it.++### Facet 3 — a malformed `import_path` produces an opaque crash, not a named one++`cli_command.import_path.partition(":")` on a path missing the `:attr` separator returns `("pkg.module", "", "")`, so the harvest calls `getattr(importlib.import_module("pkg.module"), "")`. There is no validation that `import_path` actually contains a `:`. `getattr(module, "")` raises an opaque `AttributeError` (or, on some objects, silently returns something unexpected) at `_cli.py` import time — again aborting the whole CLI, and without fail-loudly naming the offending plugin/command the way the rest of the registrar does.++**Failure scenario.** An external plugin registers `add_cli_command(import_path="pkg.module")` (forgetting the `:attr`). Every `pipelex` invocation crashes at import with an `AttributeError` that names neither the plugin nor the command, so the user cannot tell which installed plugin is at fault.++### Facet 4 — `add_cli_command` has no collision guard; a plugin can silently shadow a core command++This is the odd one out: it is not a crash, it is a **silent override**, and it is a correctness/trust problem rather than an availability one. Every *other* contribution type fail-loud on conflict and names both contributors: `add_orchestrator` (`:116`), `add_inference_backend` (`:110`), `_claim` (`:145`). `add_cli_command` (`:135`) performs **no** duplicate or core-name-collision detection at all. The harvest then calls `app.command(name="run", ...)(command)` at import time, registering a **second** `run` on the Typer app. Click resolves the last registration, so the plugin's callable silently wins; `list_commands` lists the name twice.++**Failure scenario.** A third-party (or buggy future in-tree) plugin advertises `add_cli_command(name="run", import_path="evil:run")`. On the next `pipelex run …`, the plugin's callable is dispatched **instead of** the core `run` command — no error, no warning — and the command appears twice in `pipelex --help`. Core execution is silently replaced by plugin code. The same applies to any core name (`validate`, `show`, `which`, `plugins`, …) and to two plugins colliding on the same custom name.++## Why this is the critical cluster (severity reasoning)++- **It defeats its own recovery path.** The defining property of all three crash facets is that the bricked commands *include the ones designed to recover from the brick* (`doctor`, `plugins`, `init` → `plugins.disabled`). A fail-loud error you cannot act on is worse than a fail-loud error you can.+- **It is live exactly where the project is heading.** It is latent for in-tree-only installs, but the entire point of the seam (Phase 5) is external `pipelex.plugins` dists. The surface becomes load-bearing precisely when externalization ships.+- **Facet 4 is a silent-override / trust hole**, which the "fail loud" invariant and the parallel guards on every sibling contribution type were specifically meant to prevent. CLI commands are the one unguarded menu, and the one that can shadow core behavior.+- **It is a regression in blast radius.** Pre-Phase-3, discovery failure was contained to boot (`Pipelex.setup()`); a user could still run `--help`/`doctor`. Phase 3 widened the blast radius to "every invocation."++## How to reproduce (write the failing tests first)++These sketch the minimal triggers; turn each into a red test before exploring fixes.++1. **Facet 1:** register a fake external `pipelex.plugins` entry point whose `register()` raises (or whose `targets_api` mismatches `PLUGIN_API_VERSION`), then invoke `pipelex --help`. Expect: today it raises at import; the desired contract is `--help` still works (core commands listed), with the breakage surfaced through `doctor`/`plugins`, not a hard crash.+2. **Facet 2:** create a `.pipelex/pipelex.toml`, `chmod 000` it (or simulate `open()` raising `PermissionError`), invoke `pipelex --help`. Expect: today it raises uncaught `PermissionError`; the contract wants a fallback to package defaults.+3. **Facet 3:** a fake plugin contributing `add_cli_command(name="x", help="…", import_path="some.module")` (no `:`). Invoke any `pipelex` command. Expect: today an opaque `AttributeError` at import; the contract wants a named, contextual error attributing the plugin/command (and ideally not bricking the rest of the CLI).+4. **Facet 4:** a fake plugin contributing `add_cli_command(name="run", help="…", import_path="…:something")`. Assert that `run` is registered twice / that core `run` is shadowed. Expect: today it silently overrides; the contract wants a fail-loud collision (parallel to `DuplicateOrchestratorError`).++The existing harvest tests to extend: `tests/.../test_plugin_cli_command_harvest.py` and `test_harvest_config_falls_back_to_base_on_broken_user_config` (the S1 test — covers Facet 2's *parse* path only).++## Solution space — open questions to explore (NOT answered here)++Listed only to map the decision space for the fresh session. Each is a genuine tension; none is pre-judged.++- **Altitude / timing.** Should the harvest run at import time at all, or be deferred to the point where the command list is actually needed (the `PipelexCLI` group already subclasses `TyperGroup` and overrides `list_commands`/`get_command`)? What does `--help` *need* vs. what does dispatching a plugin command *need*? Does shell tab-completion change the answer?+- **Reconciling the two invariants.** "Fail loud on a broken plugin" (boot) vs. "never brick the recovery commands" (CLI startup) are both deliberate. Where is the seam between them? Is the answer "fail loud, but only when the user actually invokes the broken plugin's command / a boot-requiring command, and degrade-with-a-visible-notice for `--help`/`doctor`/`init`"? What does "visible notice" mean for an LLM-facing vs. human-facing surface (see the workspace "format follows consumer" rule)?+- **Degrade vs. abort.** If discovery fails during the harvest, what is the right degraded state — core commands only, with the failure reported through `doctor`/`plugins`? Or is partial discovery (the built-ins succeeded, one external failed) meaningful?+- **Collision policy (Facet 4).** Should `add_cli_command` route through a `_claim`-style guard that refuses to shadow a core command name and refuses two plugins claiming the same name? Is "core command names are reserved" a hard rule? Is there *any* legitimate reason a plugin should override a core command? How does this interact with `_CORE_COMMAND_ORDER` being the source of truth for core names?+- **`import_path` validation.** Where should the `:`-presence (and attribute-exists) validation live — at `add_cli_command` registration time (fail-loud, names the plugin), or at harvest/dispatch time? Should `CliCommand` validate its own `import_path` shape as a pydantic model?+- **Single vs. double discovery.** Boot re-runs `build_registrar` (`pipelex.py:~390`) after the harvest already ran it, discarding the harvest result. Any fix that changes *when/whether* the harvest runs interacts with this redundancy (tracked separately in [phase-3-review-deferred.md](phase-3-review-deferred.md) under "double discovery / import-time cost"). Decide whether the two are solved together.++## Pointers++- Code: `pipelex/cli/_cli.py` (harvest), `pipelex/plugins/discovery.py` (`build_registrar`), `pipelex/plugins/registrar.py` (`add_cli_command` + the guarded siblings), `pipelex/tools/misc/toml_utils.py:39` (the `open()`).+- Decision: **D3** in [TODOS](../../TODOS.md) ("plugin CLI commands harvested by running the pure `build_registrar` at CLI-build; same fn runs again at boot").+- Invariants: TODOS "Invariants that must survive every phase" — *fail loud* on a broken plugin; "installed but broken" ≠ "not installed".+- Sibling deferred items (medium/low): [phase-3-review-deferred.md](phase-3-review-deferred.md).diff --git a/wip/plugins/phase-3-review-deferred.md b/wip/plugins/phase-3-review-deferred.mdnew file mode 100644index 000000000..16ef907ce--- /dev/null+++ b/wip/plugins/phase-3-review-deferred.md@@ -0,0 +1,31 @@+# Phase 3 plugin-seam review — deferred items++Items surfaced by the xhigh `/code-review` over the Phase 3 checkpoint commit (`19e6ca66b` / `5f61db323`) — orchestrators through the seam: the bridge `match` collapse into `OrchestratorRegistry`, the DIRECT/Temporal orchestrators, the slot-claim apply-points, the CLI-command harvest, and `MissingOrchestratorError`. These are the medium-to-low findings left for later triage; none produces wrong output on a live in-tree path today, so none was changed in the checkpoint.++> **The critical finding is split out.** The dominant correctness cluster — the import-time CLI plugin-command harvest is an unguarded surface that can brick every `pipelex` command (recovery commands included) and can silently shadow a core command — has its own cold-start problem statement in **[phase-3-critical-cli-harvest-fragility.md](phase-3-critical-cli-harvest-fragility.md)**. Read that first; it is the one item worth blocking on. This doc holds everything else.+>+> 19 of the 40 verified candidates were **refuted** by the verify pass (e.g. the `registrar.config.temporal.is_enabled` precondition — production callers always pass a full config; the exhaustive `MissingOrchestratorError._build_message` match — pyright-clean and correct per the no-default-case enum rule; several TASK_MANAGER-slot divergence claims that cause no functional bug today). Only the kept items are below.++## High — deferred by design, but the error message actively misleads++- **`MISTRAL_NATIVE` is currently a dead execution mode.** `runtime_bridge/bridge.py` deleted `_run_mistral_native` and flipped to registry discovery, but **no plugin contributes a `MISTRAL_NATIVE` orchestrator** — `pipelex-mistralai-workflows` has no `pipelex.plugins` entry point / `add_orchestrator` call (it exposes only `register_pipelex_primitives` + a `make_mistral_workflows_pipe_run` factory the *old* bridge imported directly). This is expected per the Phase-3 goal ("core names no integration — not by import, not by string"), and landing the MISTRAL_NATIVE plugin onto `dev` is tracked as its own sub-track ([../distributed-execution/mistral-workflows/](../distributed-execution/mistral-workflows/)). The deferred *defect* is the user-facing message: a consumer who **already has** `pipelex-mistralai-workflows` installed and submits `execution_mode=MISTRAL_NATIVE` now gets `MissingOrchestratorError` telling them to `pip install pipelex-mistralai-workflows` — a package they have. Until the plugin grows an entry point, MISTRAL_NATIVE is wholly unusable and the hint points at the wrong remedy. Secondary: the old arm returned `workflow_id=pipe_output.pipeline_run_id`; the seam provides no orchestrator preserving that non-null `workflow_id` contract.++## Medium — error-contract & teardown fragility at the new seam++- **The bridge no longer wraps orchestrator errors; the single-error-type contract now rests on each orchestrator's discipline.** `runtime_bridge/bridge.py` calls `orchestrator.run()` with no surrounding `try`/`except` — error-wrapping into `PipelexBridgeDispatchError` moved *into* each orchestrator. The in-tree DIRECT/TEMPORAL orchestrators kept their wrapping, but an **external** orchestrator (the future MISTRAL_NATIVE plugin) must now wrap its own pipe-execution errors; the old `_run_mistral_native` did that wrapping at the bridge. If an external orchestrator raises a raw `PipeRunError`/`PipeJobError`/`PipeExecutionError`, it propagates unwrapped through the bridge, and a host catching only the documented `PipelexBridgeDispatchError` fails to catch it. Decide whether the SPI documents "orchestrators MUST raise `PipelexBridgeDispatchError`" as a hard contract (with a conformance test), or whether the bridge keeps a thin outer wrap as a safety net.++- **`teardown()` runs plugin callbacks with no `try`/`finally`; one raising callback aborts all core teardown.** `pipelex/pipelex.py:teardown()` (~`:494`) iterates `reversed(self._plugin_registrar.teardown_callbacks)` and *then* runs core teardown (pipeline/telemetry/inference/reporting/sdk-client managers). If a plugin teardown callback raises — e.g. a Temporal worker's `task_manager.teardown()` or `temporal_hub.reset()` throwing during an unclean shutdown — the exception propagates out of the loop and **none** of the subsequent core teardown runs, leaking those resources/handles. The old code had this same shape for the single inlined Temporal block, so this is a pre-existing fragility now generalized across N callbacks — flagged per the "flag and fix pre-existing bugs" convention. Cleanup that must run on the error path belongs in `try`/`finally` (per the error-handling standard).++- **`_teardown_temporal` re-fetches the task manager from the global hub instead of the instance it built.** `pipelex/temporal/temporal_plugin.py` (~`:63`): `_setup_temporal_task_manager()` constructs a `TemporalTaskManager`, sets it on the module-global `temporal_hub`, and discards the local reference; `_teardown_temporal()` re-fetches via `temporal_hub.get_optional_task_manager()`. If any path between setup and teardown resets `temporal_hub` (a nested/aborted second boot, or a reentrant `TemporalPlugin` register that fails after claiming), the fetch returns `None`, teardown silently skips `task_manager.teardown()`, and `TemporalManager` session/sandbox state + the in-workflow content generator leak into the next worker boot. Holding the constructed instance and tearing it down directly (instead of round-tripping through the global) would forward the call unconditionally.++- ✅ **RESOLVED (Option A, step 14).** Folded into the orchestrator-plugins.md rewrite: the doc now says the Temporal plugin is discovered through `BUILTIN_PLUGINS` (pipelex declares **no** `pipelex.plugins` self-entry-point) and states the real Phase-5 delta (remove from `BUILTIN_PLUGINS` + add a `pipelex.plugins` entry point in the new dist). Original finding retained below. **The SPI doc is factually wrong about how the Temporal plugin is discovered.** `docs/under-the-hood/orchestrator-plugins.md` (~`:116`) states: "The Temporal plugin is in-tree today and discovered via an entry point pipelex declares on itself." False on two counts: (1) the Temporal plugin is in `BUILTIN_PLUGINS` (a hardcoded list in `pipelex/plugins/builtins.py`), and `pyproject.toml` declares **no** `[project.entry-points."pipelex.plugins"]` on pipelex itself; (2) it misstates the Phase-5 story — externalization is *not* "the same entry point shipped from its own dist," because there **is** no entry point today; it requires *adding* one and *removing* the builtin. An engineer doing Phase 5 (or debugging plugin-override pickup) would search `pyproject.toml` for a self-entry-point that does not exist. Fix the prose to describe the builtin-list mechanism and the real externalization delta. *(This is the same gap the Phase-3 as-built "Notes for Phase 4/5" already flags from the implementation side — keep the two in sync when fixing.)*++## Low — cleanup, non-blocking++- **Enum identity comparison instead of the property.** `runtime_bridge/bridge.py` `_validate_input` (~`:165`) singles out fire-and-forget via `input_payload.execution_mode is PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET` instead of the existing `PipelexExecutionMode.is_fire_and_forget` property added for exactly this case. Violates the python-standards rule "never test equality to an enum value; add an `@property` so the linter complains when a new value is added." A future fire-and-forget-like mode would silently miss this validation guard. Pre-existing (untouched by the diff but in the reshaped file) — flagged per "flag and fix pre-existing."++- ✅ **RESOLVED (Option A).** The import-time CLI harvest is gone, so the second discovery, the per-invocation config load (deep-merge + full `PipelexConfig` validation), and the `entry_points(group=…)` metadata scan no longer run on any `pipelex` invocation (including `--help` and shell tab-completion) — only boot discovers now. Original finding retained below. **Double discovery + import-time cost on every invocation.** On any boot-requiring command (`run`/`validate`/`build`), plugin discovery + config load runs **twice**: once in the import-time CLI harvest (`_cli.py:~280`), once again at boot (`pipelex.py:~390`, `build_registrar(config=get_config())`), with the harvest registrar discarded except for command names. Separately, the harvest adds a full config load (deep-merge of multiple TOML overrides + full `PipelexConfig` validation) **plus** an `importlib.metadata.entry_points(group=...)` metadata scan to **every** `pipelex` invocation — including `--help` and shell tab-completion, which did none of this pre-Phase-3. Both stem from the harvest running fully, at import, on every call. **Coupled to the critical finding:** any redesign that defers/memoizes the harvest (see [phase-3-critical-cli-harvest-fragility.md](phase-3-critical-cli-harvest-fragility.md) "Solution space → Single vs. double discovery") should resolve this for free — solve them together, don't apply a narrow perf patch here in isolation.++- **Dead `serialize_pipe_output` re-export.** `runtime_bridge/bridge.py` (~`:43`) keeps a `serialize_pipe_output` re-export commented "for host-runtime back-compat," but the real definition + every internal use moved to `serialization.py`, and grep across `pipelex/`, `tests/`, and the sibling runner repos (`pipelex-api` / `pipelex-worker` / `pipelex-mistralai-workflows`) finds **zero** importers of it from `bridge`. Under the workspace "No backward compatibility" policy this is unjustified duplicated surface that keeps `bridge.py` importing `serialization.py` purely to re-export, and silently rots. Drop it; in-tree callers and the orchestrator SPI already import from `serialization.py` directly.++- **Hardcoded count in a docstring.** `pipelex/temporal/temporal_plugin.py` (~`:11`) says the plugin "claims the **four** process-global hub slots," violating the workspace "never hardcode counts" rule. The next phase (Phase 4, the `model_lists` 5th seam) is precisely the kind of change that makes "four" go stale while the `claim_*` calls below it claim a different number. Write "the process-global hub slots" with no count.diff --git a/wip/plugins/phase-4-residual-core-vendor-couplings.md b/wip/plugins/phase-4-residual-core-vendor-couplings.mdnew file mode 100644index 000000000..248600333--- /dev/null+++ b/wip/plugins/phase-4-residual-core-vendor-couplings.md@@ -0,0 +1,36 @@+# Phase 4 — residual core→vendor couplings outside the enumerated seams++Phase 4 inverted the 5th and last *dispatch* seam (`model_lists.py`'s `match sdk:`), so every seam the plan enumerated now names no integration: the four inference worker factories, model listing, orchestrator dispatch, the boot/teardown hub swap, and the CLI (removed in Option A). With those done, "core names no integration" holds **for the dispatch surfaces the effort was scoped to**.++The CHECKPOINT-4 verification grep is broader than those surfaces, though, and it surfaces three **pre-existing** `pipelex.plugins.<vendor>` references in core that were never part of the Phase 0–4 seam list. They are recorded here so the "core names no integration" claim stays precise and a future session can decide their fate. None were introduced by Phase 4 — all three are unchanged from `main`.++## 1. `cogt/config_cogt.py` — vendor config models (by design, not a leak)++```+from pipelex.plugins.anthropic.anthropic_config import AnthropicConfig+from pipelex.plugins.google.google_config import GoogleConfig+from pipelex.plugins.mistral.mistral_config import MistralConfig+from pipelex.plugins.openai.openai_config import OpenAIConfig+```++This is the **typed-config-in-core** pattern, an explicit decision: the design's "Deferred / out of scope" list keeps per-plugin *typed config* in core ("Generic per-plugin typed-config namespace — Temporal config stays typed in core (design D7)"). Inverting it would mean a generic per-plugin config namespace, which the design deliberately declined. **Leave as-is** unless that decision is revisited.++## 2. `cogt/img_gen/img_gen_args_factory.py` — `OpenAIImgGenFactory` (genuine, unscoped)++`ImgGenArgsFactory` imports `OpenAIImgGenFactory` at module top to build provider-specific image-generation API arguments. This is a real core→vendor coupling, but it is an **argument-shaping** concern, not one of the worker-dispatch / listing / orchestrator / boot / config / CLI seams the plan enumerated. Inverting it would need its own capability on the contract (an "img-gen args" SPI), i.e. genuine new design — not a checkpoint cleanup. **Deferred; needs a design decision before it becomes a seam.**++## 3. `cogt/model_backends/backend_factory.py` — `VertexAIFactory` (genuine, unscoped)++```python+match name:+ case "vertexai":+ from pipelex.plugins.openai.vertexai_factory import VertexAIFactory # noqa: PLC0415+ endpoint, api_key = VertexAIFactory.make_endpoint_and_api_key(extra_config=extra_config)+```++A lazy, `match`-guarded import that resolves vertexai's special endpoint/api-key auth during backend construction. Like (2), a real coupling but a **backend-auth** concern outside the enumerated seams; inverting it would need a backend-auth capability on the contract. **Deferred; needs a design decision before it becomes a seam.**++## Disposition++- (1) is **intentional** per D7 — not a follow-up, just documented so the grep result reads honestly.+- (2) and (3) are **genuine additional seams** the original plan did not enumerate. They do not block CHECKPOINT 4 (whose primary assertion — `model_lists.py` names no integration — holds). If the goal is upgraded from "no integration in the enumerated seams" to a literal "no integration anywhere in core," they are the next two candidates, each requiring its own contract capability.diff --git a/wip/plugins/phase-4-review-followups.md b/wip/plugins/phase-4-review-followups.mdnew file mode 100644index 000000000..c3897ceca--- /dev/null+++ b/wip/plugins/phase-4-review-followups.md@@ -0,0 +1,42 @@+# Phase 4 — xhigh code-review follow-ups++Record of the xhigh `/code-review` run on the Phase 4 commit (`6a0796548` — "invert model listing, the 5th and last dispatch seam"). The review confirmed the inversion is **byte-equivalent to the old static `match`** for every shipped config — no crash or wrong-output bug. Adversarial verify kept a handful of findings; this note dispositions them.++Two cheap cleanups were **applied in the same pass** (see the bottom of this note). The four below are **design tradeoffs, not bugs** — deferred deliberately rather than reflexively "fixed," so the decision is on the record for a future session. Sibling to [`phase-4-residual-core-vendor-couplings.md`](phase-4-residual-core-vendor-couplings.md) (which covers core→vendor couplings *outside* the enumerated seams); this note covers review findings *on the model-listing seam itself*.++## 1. Model listing is now coupled to the plugin enable/disable switch — DEFERRED (arguably correct as-is)++`cogt/model_backends/model_lists.py`. The old static `match sdk:` listed `anthropic` / `mistral` / `bedrock` / `google` SDKs unconditionally. After the inversion, listing dispatches through the registry, and those four plugins are **not** in `CORE_UNCONDITIONAL_PLUGIN_NAMES` — so adding one to `config.plugins.disabled` removes its lister, and `pipelex show models <backend>` then reports that SDK as "we don't support for remote listing."++- **Real?** Yes — it is the one live behavior change in the commit.+- **Why deferred (and arguably *correct*):** a disabled plugin also registers no `make_worker`, so its models can't be *run* either. Listing models you can't use was the old anomaly; gating listing on the same enable switch as execution is the more consistent behavior. "Fixing" it (e.g. registering listers even for disabled plugins) would re-introduce the inconsistency.+- **Revisit if:** product decides `plugins.disabled` should mean "hide the driver but still enumerate its catalog" (a discovery-vs-activation split). That is a config-semantics decision, not a Phase 4 bug.++## 2. Bedrock lister key skew (`bedrock` / `bedrock_aioboto3` vs worker key `bedrock_boto3`) — DEFERRED (zero live impact)++`plugins/bedrock/bedrock_plugin.py`. The lister registers under `bedrock` (dead — no shipped model spec carries `sdk="bedrock"`) and `bedrock_aioboto3`, while the **inference backend** in the same `register()` body registers under `bedrock_boto3` / `bedrock_aioboto3`. So within one function the lister and worker key sets deliberately differ.++- **Real?** Byte-equivalent to the old `case "bedrock" | "bedrock_aioboto3"` arm — carried forward verbatim, **zero live impact**. The `bedrock` key is dead today.+- **Why deferred:** dropping the dead `bedrock` key is a tiny cleanup, but `tests/unit/pipelex/plugins/test_model_lister_coverage.py` pins the registered-key set as an *exact-match* contract (`EXPECTED_LISTER_SDKS` includes `bedrock`), so the cleanup is "drop the key **and** update the exact-set test + its docstring framing" — a coupled edit better done deliberately than as a checkpoint drive-by.+- **Footgun to watch:** a future bedrock variant whose model specs use `bedrock_boto3` would get "unsupported for listing" instead of a model list, because no lister is registered under that key. If/when bedrock keys are consolidated, align the lister keys with the worker keys in the same `register()` body.++## 3. `any_listed` leaks the listing loop's display state into the public SPI — DEFERRED (pre-existing, now on the public surface)++`plugins/*/*_plugin.py` lister closures + `ListModelsFn` + the authoring example in `docs/under-the-hood/inference-backend-plugins.md`. `any_listed` is CSV-header / separator suppression state owned by `ModelLister.list_models` — pure loop-presentation bookkeeping. The inversion re-published it as part of the *public* `ListModelsFn` contract, so every out-of-tree backend author must accept and correctly forward a flag whose meaning is internal to the loop; a lister that forwards it wrongly produces a duplicated/missing CSV header.++- **Real?** Pre-existing (the old per-arm calls already threaded it), but Phase 4 cemented it into the new SPI surface.+- **Why deferred:** the clean fix is to keep the lister contract to the data it needs (`sdk` / `backend_name` / `backend` / `flat`) and let the loop own the header decision (it already knows `any_listed` — pass an `is_first` hint or post-process). That is an SPI-shape change touching every lister + the doc + the display helpers — a focused refactor, not a checkpoint cleanup.+- **Best moment:** before any third-party backend ships against `ListModelsFn`, while the only implementers are in-tree.++## 4. `ListModelsFn = Callable[..., Awaitable[None]]` gives the loop's kwargs zero static checking — DEFERRED (follows Phase-1 precedent)++`plugins/model_lister_registry.py`. The whole byte-equivalence claim rests on every lister's kwargs matching the loop's `await` call, yet the `...` args mean pyright/mypy can't catch future drift: rename a lister param or change the loop's call kwargs and the mismatch surfaces only at runtime as a `TypeError` wrapped into a confusing `PipelexCLIError`.++- **Real?** Yes as a latent type-safety gap; not a live bug (the shipped kwargs match).+- **Why deferred:** a `Protocol` with the explicit keyword-only signature (the way `OrchestratorProtocol` already types the orchestrator seam) would catch drift statically at no runtime cost — but `MakeWorkerFn` uses the *same* loose `Callable[..., ...]` form, so tightening only `ListModelsFn` would split the convention. The right scope is "tighten both inference-family callables to Protocols together," a Phase-1-and-Phase-4 SPI-typing pass, not a model-listing-only edit.+- **Pairs with #3:** if `ListModelsFn` becomes a `Protocol`, drop `any_listed` from its signature at the same time.++## Applied in the same pass (not deferred)++- **Registrar `_add` helper** — `add_inference_backend` / `add_model_lister` / `add_orchestrator` each hand-rolled the identical dup-check → store → mirror-into-`_*_sources` → append-contribution shape. Generalized behind one `PluginRegistrar._add(...)` (each method supplies its keyed store, parallel sources dict, contribution label, and a factory that builds its distinctly-typed `Duplicate*Error`), mirroring the existing `_claim` helper on the slot menu. A future 6th registration seam now reuses `_add` instead of becoming a 4th copy.+- **Hardcoded counts in the Phase 4 as-built** — removed the item-count enumerations the workspace `CLAUDE.md` "Never hardcode counts" rule forbids (`9 expected sdk keys`, `4 soft-miss cases`, `2 new pages`, `274 unchanged`, `4 keys`, `2 keys`). Point-in-time test-run evidence (`715 passed`, pyright file count) was left as a verification snapshot.diff --git a/wip/plugins/phase-5-cutover-review-followups.md b/wip/plugins/phase-5-cutover-review-followups.mdnew file mode 100644index 000000000..5ca94312f--- /dev/null+++ b/wip/plugins/phase-5-cutover-review-followups.md@@ -0,0 +1,70 @@+# Phase 5 local cut-over — `/code-review` follow-ups++**Source:** xhigh `/code-review` on commit `a8382d1a1` (the `_plugins`/core side of the Temporal externalization — `git rm` of `pipelex/temporal/` + the `temporal` extra, pre-publish). This is the triage for that review (TODOS Checkpoint 5 → "Fan-out `/code-review` … Triage here").++**State that frames every decision below:** the cross-repo move is **already done locally** — `pipelex-temporal` is scaffolded at `../pipelex-temporal` (top-level `pipelex_temporal` package per D8), the temporal tree + tests + marker + `--temporal-server` conftest moved there, editable pins flipped, suites green on both sides. Nothing is published/merged/deployed. So "the following steps" = flip downstream consumer pins (`pipelex-worker`, `pipelex-api-hosted`), repoint `pipelex-mistralai-workflows`, run consumer suites, then publish/merge. None of the items below block those steps.++## Finalize-local pass (2026-06-20, session 2) — core dead-tooling removal + pipelex-temporal review triage++A second pre-publish local pass (still gated before publish). Goal: get **core clean of the moved Temporal dev-tooling** and review/triage the **pipelex-temporal side** of the cut-over (session 1 reviewed the core *deletion* side; this reviews the *additions* side). All changes local, uncommitted at time of writing, on top of the `a8382d1a1`/`27da150a5` cut-over commits (core) and `c39fbdc` (pipelex-temporal).++**Done:**++- **Both Temporal skills relocated to `pipelex-temporal` and removed from core.** Closes item 1 **and extends it** — alongside `temporal-e2e-validate` (the only one item 1 named), `temporal-test-crate` (`.claude/skills/temporal-test-crate/SKILL.md`, also dead in core: it pointed at the moved `tests/integration/pipelex/temporal/library_crate/`) was found and relocated too. Mechanical rewrite on the copies (`tests/{integration,unit}/pipelex/temporal/` → `tests/{integration,unit}/pipelex_temporal/`, `pipelex.temporal` → `pipelex_temporal`, and the one cross-repo `../pipelex/tests/` → `../pipelex-temporal/tests/`). Verified: residue grep clean (only the `.pipelex/temporal-payload-store/` runtime-dir false-positives remain — that dir name is config, not the module), the 3 scripts `py_compile`, and `submit_dry_validate.py`'s rewritten `pipelex_temporal.{exceptions,tprl_pipe.*}` imports resolve in pipelex-temporal's venv. Skills live in `.claude/` → **not** in the publishable wheel.+- **Core Makefile temporal targets removed** (the "or delete from core" half of item 2): help block, `.PHONY` entries, the `test-temporal`/`ttm` target + its `SRV`/`MODE`/`REG`/`TEMPORAL_*` vars, and `temporal-server`/`-stop`/`-worker`(+`-router`/`-runner`)/`-run`/`-run-dry`. Verified `make help`/`make -n` parse and every removed target reports "No rule to make target." The `agent-test-debug` target's temporal *comments* + the no-op `pkill temporal-sdk-python` were **left intentionally**, paired with the deferred `debugging-hanging-pytest-runs.md` (item 6) so the target and its doc de-temporalize together.+- **pipelex-temporal review (additions side): SHIP — clean.** `pr-review-toolkit:code-reviewer` built the wheel offline and verified empirically: `py.typed` packaged, `tests/` excluded, both entry points emitted, the `[tool.uv.sources] pipelex = {path=../_plugins}` pin does **not** leak into wheel METADATA (clean `Requires-Dist: pipelex>=0.35.0`), import-light proven at runtime (importing `temporal_plugin`/building the CLI leaves `temporalio` out of `sys.modules`), end-to-end entry-point discovery resolves, the namespace rewrite is logic-drift-free across the whole tree, suite collects standalone (0 errors). **No BLOCKER / SHOULD-FIX.** Two NITs:+ - **stray `.pipelex/traces/*.ndjson` (9 files) committed — FIXED.** Confirmed stray: tracing-guard tests write to `f"..._{uuid.uuid4().hex[:12]}"` dirs each run, so the committed fixed-uuid files are never read back. `git rm`'d + added `.pipelex/traces/` to `.gitignore`.+ - **shipped `test_extras/wf_test_*` back-import from the non-shipped `tests/` tree — DEFERRED.** `pipelex_temporal/test_extras/wf_test_content_generator_{child,pdf_page_views}.py` do `from tests.integration.pipelex_temporal.test_data import PipeTestCases`. Pre-existing (byte-faithful to `933b6c35c`) and dormant in production (gated behind `is_unit_testing` in `temporal_task_manager.run_worker`). Eventual fix: relocate `PipeTestCases` into `pipelex_temporal.test_extras` so nothing shipped depends on `tests/`. Not a cut-over blocker.++**Gates (this pass):** core `make tb` green + full `make agent-test` green (exit 0, HEAD `27da150a5`) + Makefile parses; pipelex-temporal `make agent-check` clean.++**Still open (added by this pass):**++- **pipelex-temporal Makefile lacks the Temporal dev targets** (`temporal-server`/`test-temporal`/`temporal-worker`/`temporal-run`) removed from core. Item 2's "move to pipelex-temporal's Makefile" half is **not** done — its Makefile delegates to `Makefiles/Makefile_basics.mk` and the targets can't be verified without a running Temporal server, so adding them was deferred rather than shipped unverified right before publish. Add when the repo's Temporal dev loop is next exercised.++## Fixed in session 1 (cut-over review, commit `a8382d1a1`)++Reachable breakages + cheap core-hygiene that stay in core and the cross-repo move would not otherwise touch:++- **kit `agent_rules/commands.md`** — removed the "Temporal Integration Test Options" section (pointed at the deleted `tests/integration/pipelex/temporal/`; shipped into client `.pipelex/`). Regenerated the derived `CLAUDE.md` via `pipelex-dev kit rules`; `check-rules` PASSED.+- **`uv.lock`** — regenerated; dropped the stale `temporal` extra (`temporalio`, `nexus-rpc`, `types-protobuf`). `uv lock --check` now clean.+- **`runtime_bridge/bridge.py`** docstring — "in-tree Temporal plugin" → external `pipelex-temporal`; dropped the "an in-tree orchestrator whose extra is absent" framing (no extra exists anymore).+- **`tests/unit/pipelex/system/test_config_temporal_optional_dep.py`** — docstring + assertion message no longer reference the deleted `temporal` extra (now: temporalio is not a core dep, ships only with `pipelex-temporal`).+- **5 plugin/cogt test stubs** (`test_plugin_discovery`, `test_inference_backend_coverage`, `test_model_lister_coverage`, `test_llm_worker_factory`, `test_img_gen_worker_factory`) — dropped the now-vestigial `temporal=SimpleNamespace(is_enabled=False)` stub field (no builtin's `register()` reads `config.temporal` post-cut-over — verified) and corrected the "TemporalPlugin is a builtin" comments.+- **Restored two core-coverage tests** for behavior that stayed in core but lost its test:+ - `test_trace_context_contract.py::test_non_direct_mode_nulls_host_trace_context` — pins the `trace_context if is_direct else None` guard in `bridge.py` (non-DIRECT modes must null the host trace_context; doubles as a check that no TEMPORAL orchestrator is registered in core).+ - `test_dispatch_options_no_temporalio.py` — `DispatchOptions` must be constructible without temporalio (complement to the AST scan, which cannot catch a regression of the `RetryPolicy = Any` placeholder to a bare forward ref → `PydanticUserError` at construction).++Gates after the fixes: targeted unit suites green (104 passed), `make agent-check` clean (pyright 0/0, mypy 0, ruff/plxt/keyword-only pass).++## Deferred — finalize with the cross-repo publish++These describe the Temporal surface that **moved** to `pipelex-temporal`. They don't bite core CI and don't block the publish steps; do them as part of finalizing the move (the package now exists locally to reference concretely).++1. **✅ DONE (session 2 — relocated, + `temporal-test-crate`; see *Finalize-local pass* above).** **Skill `temporal-e2e-validate`** (`.claude/skills/temporal-e2e-validate/`) — `scripts/submit_dry_validate.py` imports the deleted `pipelex.temporal.{exceptions,tprl_pipe.act_dry_validate,tprl_pipe.dry_validate_dispatch}` and its `DEFAULT_BUNDLE` points at the deleted `tests/integration/pipelex/temporal/...`. The whole skill exercises the Temporal arm, which now lives in `pipelex-temporal`. **Relocate the skill to `pipelex-temporal`** (and repoint imports to `pipelex_temporal.*`); it cannot run from the core venv (temporalio absent). Until then it is dead in core.++2. **✅ core-removed (session 2); ⏳ pipelex-temporal add still open (see *Finalize-local pass* above).** **Makefile temporal targets** (CI-isolated — `agent-test`/CI never invoke them; verified `agent-test:` deps don't reach them): `temporal-worker`/`tw` (runs deleted `pipelex.temporal.worker_cli`), `test-temporal`/`ttm` (`TEMPORAL_TESTS_DIR = tests/integration/pipelex/temporal/`, deleted), `temporal-run`/`temporal-run-dry`, `temporal-worker-router/-runner` (`twr`/`twn`), `temporal-server`/`ts`, and the debug-run guidance referencing the deleted temporal conftest. Move to `pipelex-temporal`'s Makefile (or delete from core). Also drop the temporal lines from the `make help` text and `.PHONY`.++3. **`docs/under-the-hood/orchestrator-plugins.md`** — rewrite Temporal as the canonical **external** orchestrator plugin (symmetric with `pipelex-mistralai-workflows`). Stale spots: frontmatter + line 10 ("in-tree Temporal plugin"), line 43/47 ("extra absent"; the install hint is rendered single-quoted `'pipelex-temporal'` while the code emits unquoted `pip install pipelex-temporal`), lines 82–89 (declares the `pipelex-temporal` console script in **core's** `pyproject.toml` — now in `pipelex-temporal`), lines 110–119 worked example ("in-tree today", "discovered through `BUILTIN_PLUGINS`", "Externalizing … (Phase 5)" as a *future* event — it has happened).++4. **`docs/under-the-hood/temporal-integration.md`** — the module map (~lines 106–276) enumerates concrete deleted `pipelex/temporal/**.py` files. Rewrite to point at `pipelex-temporal`, or reduce to an overview that links the external repo.++5. **`docs/errors/` temporal pages + `generate-error-pages`** — **do NOT run `generate-error-pages` in core until this is settled**: it deletes any page whose class is no longer importable from core, 404-ing the `type_uri` those errors emit. Per the cut-over (TODOS Phase-5 as-built), the split is:+ - **Stay in core** (classes in `system/configuration/{config_temporal,exceptions}.py`): `temporal-config-error.md` (`TemporalConfigError`), `worker-task-queue-unknown-error.md` (`WorkerTaskQueueUnknownError`). Pages valid; regen keeps them.+ - **Moved to `pipelex-temporal`**: `temporal-flow-error`, `temporal-server-error`, `unrecoverable-workflow-failure-error`, `workflow-execution-error`, `workflow-input-error`, `search-attribute-registration-error`, `worker-profile-config-error`, `worker-scope-config-error`, `content-generation-error`. Decide where these publish so `https://docs.pipelex.com/.../errors/<x>/` resolves (pipelex-temporal generates/publishes its own, or core retains authored stubs with `<!-- pipelex:authored -->`). Reconcile the links in `docs/errors/execution-and-runtime.md` accordingly.++6. **`docs/agents/debugging-hanging-pytest-runs.md`** (~lines 122, 161) — references the deleted `tests/integration/pipelex/temporal/` and its `boot_temporal` conftest teardown as "the known fixture-teardown hang point." That hang moved with the tests. Update or drop the temporal-specific example (the general methodology stays valid).++## Deferred — design decision (not a bug)++7. **Boot does not eagerly validate `temporal.is_enabled=true` when `pipelex-temporal` is absent.** With `TemporalPlugin` gone from `BUILTIN_PLUGINS`, nothing reads `is_enabled` or claims worker-runtime slots at boot; boot succeeds silently and the misconfiguration only surfaces at the first `TEMPORAL_*` dispatch (`MissingOrchestratorError`, which carries the install hint). When `pipelex-temporal` *is* installed, its entry-point plugin reads `is_enabled` and claims slots correctly. **Decide:** add an eager boot warning/validation for the enabled-but-not-installed case, or accept the lazy dispatch-time error. Note `Pipelex.setup` still accepts `temporal_enabled` and flips the config flag (`pipelex.py:204`) — keep or reconsider as part of the same decision.++## Deferred — cross-repo coupling guards (lower priority; review verifier marked "refuted" but the gap is real)++8. **`pipelex.toml` `non_retryable_error_types` pins externalized class-name string literals.** `worker_config.retry_policy_config.non_retryable_error_types` contains class names whose defining classes moved to `pipelex-temporal`. A rename there silently desyncs from the literal baked into core's `pipelex.toml`, re-making a deterministic failure retryable — the retry-forever family. `test_non_retryable_baseline_pins.py` only asserts the `DryRun*` entries, so this is unguarded. Add a cross-repo conformance pin (or move those literals to `pipelex-temporal`'s config) during the move.++9. **Error-report "parity" framing is now single-arm in core.** `tests/integration/pipelex/error_handling/test_data.py` (+ `test_error_report_local_full_chain.py`, `test_search_error_report_local_full_chain.py`) still frame the constants as driving "local / Temporal parity," but only the local arm runs in core (the Temporal arm moved). The shared constants are duplicated cross-repo with nothing enforcing they stay in sync. Reword the framing, or add a cross-repo guard.++## Deferred — harmless cleanup (do anytime / during the move)++10. **Dead crate fixture in core.** `tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening_inputs.json` + `CvBatchScreeningTemporalTestData.INPUTS_FILE` (`test_data.py:18`) have zero consumers in this repo — the direct-mode e2e builds inputs inline and reads only `BUNDLE_FILE`. **Not deleted now on purpose:** `test_data.py`'s header documents an *intentional* self-contained crate mirror shared with `pipelex-temporal`'s suite + the skill. Decide during the move: (a) delete the JSON + constant from core (crate copy becomes bundle-only), or (b) keep it a complete mirror. Cosmetic also: `CvBatchScreeningTemporalTestData` is now a direct-mode-only fixture in core — consider dropping "Temporal" from the name.diff --git a/wip/plugins/phase-c-http-error-mapper-consumer-path.md b/wip/plugins/phase-c-http-error-mapper-consumer-path.mdnew file mode 100644index 000000000..6251a6c5e--- /dev/null+++ b/wip/plugins/phase-c-http-error-mapper-consumer-path.md@@ -0,0 +1,33 @@+# Phase C decision — how `pipelex-api` reaches the F3 HTTP-error mappers++**Status:** deferred design decision, to settle when Phase C (`pipelex-api` decoupling) starts. Not a bug. Surfaced by the Phase A `/code-review` (finding #1).++## Context++Phase A added the F3 seam exactly as the plan scoped it: `PluginRegistrar.add_http_error_mapper(*, exc_type, to_error_report)` plus the read accessor `PluginRegistrar.get_http_error_mappers()`. It deliberately did **not** push the collected mappers onto the hub.++This differs from how the *other* plugin contributions are exposed: orchestrators / inference backends / model listers each get a hub registry (`set_orchestrator_registry(...)` etc.) that any consumer reads via a module-level `get_*_registry()` in `pipelex/hub.py`. The HTTP-error mappers have no such hub surface, and the boot registrar is held privately as `Pipelex._plugin_registrar` (no public getter).++## The question for Phase C++How does `pipelex-api` obtain the mappers at app construction (`api/main.py` → `register_exception_handlers`)? Two viable paths:++1. **Rebuild in the API (no further core change).** Call the documented pure, repeatable `build_registrar(config=get_config())` and iterate `registrar.get_http_error_mappers()`. Precedent: `pipelex/cli/commands/plugins_cmd.py` already calls `build_registrar` standalone. Cost: a third `build_registrar` pass (after CLI-build harvest is gone, this is boot + api-construction) — pure and import-light, so cheap, but it re-runs every plugin's `register()` and is *a* registrar, not *the* boot registrar (fine for a pure function whose output is deterministic for a given config).++2. **Expose the boot registrar from core.** Either (a) a public `get_plugin_registrar()` getter on `Pipelex`/hub returning the already-built registrar, or (b) a full `HttpErrorMapperRegistry` on the hub mirroring the orchestrator pattern. Either keeps a single registrar instance but is a **second core change**, which the per-repo handoff table frames as "Phase C = pipelex-api only".++## Recommendation (not yet ratified)++Default to **option 1** — it keeps Phase C a pure `pipelex-api` change (matching the per-repo table) and `build_registrar`'s contract explicitly blesses repeated calls. Choose option 2(b) only if a second consumer of the mappers appears, or if rebuilding in the API proves to interleave badly with boot ordering. Whichever is chosen, record it in the Phase C as-built.++## Why not just do it now++Per the deferral convention: this is a design tradeoff, not a silent bug, and the plan scoped Phase A to the registrar method + accessor only. Pre-building a hub registry now would be speculative surface for a consumer that doesn't exist yet in this repo. See [[feedback_defer_design_tradeoff_findings]].++## Phase B seam refinement — two consequences to note in the Phase C as-built++The Phase A seam was refined during Phase B (core commit `75196e2e3`): `add_http_error_mapper` now takes a lazy `exc_type_provider` thunk, resolved by `get_http_error_mappers()` at the host's read time, so a plugin's `register()` never imports the orchestrator SDK to name its exception class. Two downstream consequences for the Phase C consumer (surfaced by the Phase B `/code-review`, both deferred — fail-loud either way, no code change):++1. **Duplicate-mapper detection now fires at the host's app-construction read time, not at boot discovery.** `get_http_error_mappers()` is what runs the providers and raises `DuplicateHttpErrorMapperError`. For the locked one-orchestrator-per-image flavor model a collision can't occur, so this is theoretical — but a mis-packaged image with two mapper-contributing plugins fails when `pipelex-api` builds the app, not at boot. Phase C should know the raise lands at its `register_exception_handlers` call site.++2. **`pipelex plugins list` shows a generic `"http error mapper"` contribution line** (no `{exc_type.__qualname__}`), unlike the `family:sdk` / `mode` detail the other contributions carry — an unavoidable consequence of not resolving the type at register. If the per-type label matters, add an optional `display_label: str` to `add_http_error_mapper` (the plugin passes `"temporalio.TemporalError"` as a plain string). Speculative surface today; skip until something needs it.diff --git a/wip/plugins/plugin-api-version-gating-semver.md b/wip/plugins/plugin-api-version-gating-semver.mdnew file mode 100644index 000000000..c5338477c--- /dev/null+++ b/wip/plugins/plugin-api-version-gating-semver.md@@ -0,0 +1,71 @@+# Follow-up: the plugin-API version gate guards nothing — build real version gating (semver)++Surfaced while reconciling the orchestration-mode / delivery split branch against the merged `#1000` (`refactor: split execution_mode into orchestration_mode + DeliveryMode`). The question was whether `#1000`'s breaking change to the orchestrator contract warranted bumping `PLUGIN_API_VERSION` from 2 → 3. Investigating that turned up a deeper problem: **the version gate, as wired today, cannot reject any plugin — it is purely documentary.** This is the task to fix it.++This is the implementation gap behind synthesis point **#13** in [`README.md`](README.md) ("Version the core↔plugin contract … make the contract break *visible*"). The mechanism exists; it just doesn't do what #13 promises.++## The finding: the gate is toothless++The contract is a single coarse integer with an exact-match gate:++- `pipelex/plugins/contract.py:14` — `PLUGIN_API_VERSION: int = 2`. A plugin declares the version it targets via the `targets_api: int` field (`contract.py:35`).+- `pipelex/plugins/discovery.py:100` — the gate:++ ```python+ targets_api = getattr(plugin, "targets_api", None)+ if targets_api != PLUGIN_API_VERSION:+ raise PluginApiVersionMismatchError(...)+ registrar.begin_plugin(name=plugin.name, origin=origin, targets_api=PLUGIN_API_VERSION)+ ```++The mechanism can never fire, because **every plugin sets `targets_api = PLUGIN_API_VERSION` by importing the constant** — built-ins (`openai_plugin.py`, `anthropic_plugin.py`, `direct_plugin.py`, …) *and* both external orchestrator plugins:++- `pipelex-mistralai-workflows/pipelex_mistralai_workflows/mistral_plugin.py` — `from pipelex.plugins.contract import PLUGIN_API_VERSION` / `targets_api = PLUGIN_API_VERSION`.+- `pipelex-temporal/pipelex_temporal/temporal_plugin.py` — same pattern.++So a plugin's declared `targets_api` *always equals the core it is installed against*. `targets_api != PLUGIN_API_VERSION` is structurally unreachable for any cooperative plugin, no matter what the constant's value is. Nothing in the workspace pins a literal `targets_api` (a `grep -rE 'targets_api\s*=\s*[0-9]'` over every repo returns nothing). And note `discovery.py:105` even *records* the core's `PLUGIN_API_VERSION`, not the plugin's declared value — the system assumes the two are identical by construction.++## Worked proof that this already let a breaking change through silently++`#1000` reshaped the orchestrator SPI in ways that are each a hard break for a plugin built against the prior contract:++- `OrchestratorProtocol.run` gained a required `delivery: DeliveryMode` keyword (no default), and the bridge passes it **unconditionally** (`pipelex/runtime_bridge/bridge.py:102`) → a prior `run` raises `TypeError` at dispatch.+- A new required `supports_fire_and_forget: bool` attribute, read with **no getattr default** (`bridge.py:178`) → a prior orchestrator raises `AttributeError` at `/start`.+- `pipelex/runtime_bridge/execution_mode.py` (defining `PipelexExecutionMode`) was **deleted**; the registry is re-keyed by the open `OrchestrationMode` string token → a prior plugin importing `PipelexExecutionMode` fails at import.++That is unambiguously a breaking SPI change — yet `#1000` shipped with `PLUGIN_API_VERSION` still at **2**, and nothing flagged it. It couldn't: the two external orchestrator plugins were migrated to the new contract in lockstep and re-import the (still-`2`) constant, so the gate stayed satisfied. The version number now labels two incompatible contracts (pre- and post-`#1000`), which is exactly the silent-incompatibility footgun #13 says the gate exists to prevent.++## Why it's harmless *today* but must be fixed before externalization++No published, out-of-tree, third-party plugin exists, and no version of the plugin API has shipped — so there is no real artifact pinned to an old contract for the gate to protect. The bump-to-3 vs stay-at-2 question is therefore moot in practice (which is why the reconciliation just dropped the inert bump and matched origin).++It stops being moot the moment a third party publishes a plugin against a *released* pipelex and pins the version it built against. At that point the current design fails them twice: a plugin that imports the constant still can't detect skew, and a plugin that pins a literal hits an exact-match `!=` that also rejects *compatible* (additive) core versions.++## The proper design (semver)++Two distinct defects to fix — neither alone is sufficient:++1. **Plugins must pin the version they were built against as a literal, not `= PLUGIN_API_VERSION`.** Importing the constant is what makes the declared version auto-track core and the gate inert. The declared value has to be a fact about the plugin's *source* (the contract it was coded against), frozen at author time — not a value re-read from whatever core happens to be installed.++2. **Replace the exact-match gate with a semver compatibility rule.** Even with literals, `!=` is too brittle — it rejects a plugin built against an older *compatible* core.++Concretely:++- Core publishes the plugin-API version as `MAJOR.MINOR` (PATCH irrelevant to the contract). **MAJOR** bumps on a breaking change — a removed/renamed SPI symbol, a changed required signature, a changed registry key type (i.e. exactly `#1000`). **MINOR** bumps on a backward-compatible addition — e.g. v2's optional `add_http_error_mapper` was purely additive and would have been a MINOR bump that required no existing plugin to change.+- A plugin declares `built_against = (MAJOR, MINOR)` as a literal.+- Compatibility: `plugin.major == core.major AND plugin.minor <= core.minor`. Same major ⇒ no breaking change since the plugin was built; `minor <=` core ⇒ every capability the plugin relies on exists in the installed core. Reject on a major mismatch, **loud**, naming both versions and the remediation (rebuild against / upgrade to plugin API `N.x`) — that is the visible break #13 wants. (`PluginApiVersionMismatchError` already carries `targets_api` + `supported_api`; the message just needs to speak semver and name the fix.)+- Re-framed: `#1000` is a MAJOR bump `2.x → 3.0`; `pipelex-mistralai-workflows` and `pipelex-temporal` each declare `(3, 0)`; a stale third-party plugin pinned to `(2, x)` is cleanly rejected at boot with "rebuild against pipelex plugin API 3.x" instead of crashing at dispatch with a `TypeError`.++Optional refinement if needed later: let a plugin also declare a maximum supported major, so a plugin can advertise forward-compat across a range rather than a single built-against point. Start with built-against-min; add the range only if a real consumer needs it.++## Scope when this is picked up++- `pipelex/plugins/contract.py` — version constant → a `MAJOR.MINOR` pair (or a small `PluginApiVersion` value object) + the compatibility helper. Update the "bump this only on a breaking change" comment to define MAJOR vs MINOR.+- `pipelex/plugins/discovery.py:100` — gate calls the compatibility helper instead of `!=`; record the plugin's *declared* version, not core's.+- `pipelex/plugins/exceptions.py` — `PluginApiVersionMismatchError` message speaks semver and names the remediation.+- **Plugin authoring convention** — the real behavioral change: built-in and external plugins stop doing `targets_api = PLUGIN_API_VERSION` and pin a literal built-against version. Without this, no gate (semver or otherwise) has teeth. This is also a conformance-test opportunity: assert no plugin imports the constant for its declared version.+- Docs — `docs/under-the-hood/orchestrator-plugins.md` (the SPI/versioning section) and README #13.++## Status++Not started. Design task, deferred — no urgency while nothing is shipped, but it should land **before the first out-of-tree third-party plugin is published against a released pipelex**, since that is the first moment the gate is asked to do its job for real.diff --git a/wip/plugins/plugin-system-overview.html b/wip/plugins/plugin-system-overview.htmlnew file mode 100644index 000000000..6761ed0d4--- /dev/null+++ b/wip/plugins/plugin-system-overview.html@@ -0,0 +1,477 @@+<!DOCTYPE html>+<html lang="en">+<head>+<meta charset="utf-8">+<meta name="viewport" content="width=device-width, initial-scale=1">+<title>Pipelex Plugin System, Orchestrators, and Deployment Flavors — current state</title>+<style>+ :root{+ --bg:#0d1117; --ink:#e6edf3; --muted:#9aa7b4; --line:#283039;+ --brand:#7c9bff; --brand-soft:#16203a; --good:#3fb950; --good-soft:#102a17;+ --bad:#ff7b72; --bad-soft:#2d1416; --warn:#e3b341; --warn-soft:#2a2412;+ --code-bg:#010409; --code-ink:#e6edf3; --chip:#1c232c; --panel:#161b22;+ }+ *{box-sizing:border-box}+ body{margin:0;background:var(--bg);color:var(--ink);+ font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;+ -webkit-font-smoothing:antialiased}+ .wrap{max-width:980px;margin:0 auto;padding:40px 22px 90px}+ h1{font-size:30px;line-height:1.2;margin:0 0 6px}+ h2{font-size:22px;margin:46px 0 14px;padding-bottom:8px;border-bottom:2px solid var(--line)}+ h3{font-size:17px;margin:26px 0 8px}+ p{margin:12px 0}+ a{color:var(--brand);text-decoration:none}+ a:hover{text-decoration:underline}+ .sub{color:var(--muted);font-size:15px;margin:0 0 22px}+ code{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:13.5px;+ background:var(--chip);padding:1px 6px;border-radius:5px}+ pre{margin:0;overflow:auto}+ pre code{background:none;padding:0}+ .card{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:18px 20px;margin:16px 0;+ box-shadow:0 1px 2px rgba(0,0,0,.3)}+ .tldr{background:linear-gradient(180deg,#172244,#11161f);border:1px solid #2b3a5e;border-left:5px solid var(--brand);+ border-radius:12px;padding:20px 22px;margin:18px 0 8px}+ .tldr h2{border:0;margin:0 0 8px;font-size:19px;color:var(--brand)}+ .tldr ul{margin:8px 0 0;padding-left:20px}+ .tldr li{margin:5px 0}+ .callout{border-radius:10px;padding:12px 16px;margin:16px 0;font-size:14.5px}+ .callout.good{background:var(--good-soft);border:1px solid #1f6f33}+ .callout.warn{background:var(--warn-soft);border:1px solid #5a4d1a}+ .callout.info{background:var(--brand-soft);border:1px solid #2b3a5e}+ .chips{display:flex;flex-wrap:wrap;gap:8px;margin:14px 0}+ .chip{background:var(--chip);border:1px solid var(--line);border-radius:999px;padding:4px 12px;font-size:13px;color:var(--muted)}+ .chip b{color:var(--ink)}+ table{border-collapse:collapse;width:100%;margin:14px 0;font-size:14px}+ th,td{border:1px solid var(--line);padding:8px 11px;text-align:left;vertical-align:top}+ th{background:#1c232c;font-weight:600}+ td code{font-size:12.5px}+ .seam-table th:first-child,.seam-table td:first-child{width:38%;min-width:290px}+ .status-ok{color:var(--good);font-weight:600}+ .status-wait{color:var(--warn);font-weight:600}+ /* diff blocks */+ .code{background:var(--code-bg);color:var(--code-ink);border-radius:10px;padding:14px 16px;+ font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:13px;line-height:1.55;overflow:auto}+ .code .del{display:block;color:#ffb4ab;background:rgba(207,34,46,.16);margin:0 -16px;padding:0 16px}+ .code .add{display:block;color:#7ee2a8;background:rgba(26,127,55,.18);margin:0 -16px;padding:0 16px}+ .code .ctx{display:block;color:#9fb0c0}+ .code .cmt{color:#7d8c9c;font-style:italic}+ .code .kw{color:#ff9b72}+ .code .fn{color:#7cc7ff}+ /* tabs */+ .tabs{margin:16px 0}+ .tab-bar{display:flex;flex-wrap:wrap;gap:4px;border-bottom:1px solid var(--line);margin-bottom:0}+ .tab-btn{background:none;border:0;border-bottom:2px solid transparent;padding:9px 14px;font:inherit;font-size:14px;+ color:var(--muted);cursor:pointer}+ .tab-btn:hover{color:var(--ink)}+ .tab-btn.active{color:var(--brand);border-bottom-color:var(--brand);font-weight:600}+ .tab-panel{display:none;padding-top:14px}+ .tab-panel.active{display:block}+ details{background:var(--panel);border:1px solid var(--line);border-radius:10px;padding:2px 16px;margin:12px 0}+ details[open]{padding-bottom:14px}+ summary{cursor:pointer;font-weight:600;padding:12px 0;color:var(--ink)}+ summary::marker{color:var(--brand)}+ .mermaid{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:18px;margin:16px 0;text-align:center}+ figcaption{color:var(--muted);font-size:13px;text-align:center;margin-top:6px}+ .small{font-size:13px;color:var(--muted)}+ hr{border:0;border-top:1px solid var(--line);margin:40px 0}+ .legend{font-size:12.5px;color:var(--muted);margin-top:6px}+ .kbd{font-family:ui-monospace,monospace;font-size:12px;background:var(--chip);border:1px solid var(--line);+ border-bottom-width:2px;border-radius:5px;padding:1px 6px}+</style>+</head>+<body>+<div class="wrap">++<h1>Pipelex Plugin System, Orchestrators, and Deployment Flavors</h1>+<p class="sub">What we built, what changed since the first overview, and where it stands · branch <code>refactor/Plugins-4</code></p>++<div class="tldr">+ <h2>TL;DR</h2>+ <p><strong>Pipelex core no longer dispatches by vendor branches.</strong></p>+ <p>Backends, orchestrators, validators, model listers, boot-runtime claims, and HTTP error mappers now register themselves through one plugin contract. Core installs registries and looks up capabilities by data.</p>+ <ul>+ <li><strong>Mechanism:</strong> <code>PipelexPlugin</code> + <code>PluginRegistrar</code> + <code>build_registrar()</code>. Built-ins and <code>pipelex.plugins</code> entry points use the same path.</li>+ <li><strong>New since the first overview:</strong> plugin API v2, lazy HTTP error mappers, env-aware plugin configs, <code>plugins.boot_orchestrator</code>, <code>orchestration_mode</code> + <code>DeliveryMode</code>, and <code>BundleValidatorRegistry</code>.</li>+ <li><strong>Status:</strong> core is current on <code>refactor/Plugins-4</code>. Temporal is a private <code>pipelex-temporal</code> git+ssh dist. The public base is orchestrator-agnostic; deployment-flavor wiring and release pin flips remain.</li>+ </ul>+</div>++<div class="chips">+ <span class="chip"><b>core</b> pipelex @ <code>refactor/Plugins-4</code> · pushed ✓</span>+ <span class="chip"><b>plugin API</b> <code>2</code> · HTTP mappers + validators</span>+ <span class="chip">all gates <b class="status-ok">green</b></span>+ <span class="chip">deployment flavors <b class="status-wait">pending</b></span>+ <span class="chip"><b>pipelex-temporal</b> = private (git+ssh, not PyPI)</span>+</div>++<!-- ============================================================ -->+<h2>1 · The problem we set out to kill</h2>+<p>Pipelex core used to <em>hard-code</em> every optional integration behind a <code>match</code> statement. Adding a backend meant editing core; an optional SDK could be imported at the wrong time; and Temporal — a heavy, optional dependency — was woven through the core runtime. The coupling lived in a handful of <strong>seams</strong>:</p>++<figure>+<svg viewBox="0 0 900 330" width="100%" role="img" aria-label="Before and after architecture">+ <defs>+ <marker id="arr" markerWidth="9" markerHeight="9" refX="7" refY="3" orient="auto">+ <path d="M0,0 L7,3 L0,6 Z" fill="#8a98a8"/>+ </marker>+ <marker id="arrG" markerWidth="9" markerHeight="9" refX="7" refY="3" orient="auto">+ <path d="M0,0 L7,3 L0,6 Z" fill="#3fb950"/>+ </marker>+ </defs>+ <!-- BEFORE -->+ <text x="210" y="26" text-anchor="middle" font-size="14" font-weight="700" fill="#ff7b72">BEFORE — core hard-codes each integration</text>+ <rect x="150" y="48" width="120" height="46" rx="8" fill="#2d1416" stroke="#ff7b72" stroke-width="1.5"/>+ <text x="210" y="69" text-anchor="middle" font-size="13" font-weight="700" fill="#e6edf3">Pipelex core</text>+ <text x="210" y="85" text-anchor="middle" font-size="11" fill="#ff7b72">match sdk / mode</text>+ <g font-size="11.5" fill="#cdd6df">+ <rect x="40" y="150" width="90" height="34" rx="6" fill="#161b22" stroke="#5a4048"/><text x="85" y="171" text-anchor="middle">anthropic</text>+ <rect x="145" y="150" width="90" height="34" rx="6" fill="#161b22" stroke="#5a4048"/><text x="190" y="171" text-anchor="middle">openai</text>+ <rect x="250" y="150" width="90" height="34" rx="6" fill="#161b22" stroke="#5a4048"/><text x="295" y="171" text-anchor="middle">mistral</text>+ <rect x="120" y="210" width="110" height="34" rx="6" fill="#161b22" stroke="#5a4048"/><text x="175" y="231" text-anchor="middle">temporalio</text>+ </g>+ <g stroke="#8a98a8" stroke-width="1.3" marker-end="url(#arr)" fill="none">+ <path d="M180,94 L95,150"/><path d="M205,94 L195,150"/><path d="M235,94 L300,150"/><path d="M205,94 L185,210"/>+ </g>+ <!-- divider -->+ <line x1="450" y1="40" x2="450" y2="300" stroke="#283039" stroke-width="1.5" stroke-dasharray="4 4"/>+ <!-- AFTER -->+ <text x="690" y="26" text-anchor="middle" font-size="14" font-weight="700" fill="#3fb950">AFTER — plugins register; core names none</text>+ <rect x="630" y="48" width="120" height="46" rx="8" fill="#102a17" stroke="#3fb950" stroke-width="1.5"/>+ <text x="690" y="69" text-anchor="middle" font-size="13" font-weight="700" fill="#e6edf3">Pipelex core</text>+ <text x="690" y="85" text-anchor="middle" font-size="11" fill="#3fb950">lookup()</text>+ <rect x="600" y="135" width="180" height="40" rx="8" fill="#16203a" stroke="#7c9bff" stroke-width="1.5"/>+ <text x="690" y="160" text-anchor="middle" font-size="12.5" font-weight="700" fill="#7c9bff">Hub registries</text>+ <path d="M690,94 L690,135" stroke="#3fb950" stroke-width="1.4" marker-end="url(#arrG)" fill="none"/>+ <g font-size="11.5" fill="#cdd6df">+ <rect x="520" y="232" width="96" height="34" rx="6" fill="#161b22" stroke="#2f5a3a"/><text x="568" y="253" text-anchor="middle">anthropic</text>+ <rect x="626" y="232" width="96" height="34" rx="6" fill="#161b22" stroke="#2f5a3a"/><text x="674" y="253" text-anchor="middle">openai</text>+ <rect x="732" y="232" width="130" height="34" rx="6" fill="#161b22" stroke="#2f5a3a"/><text x="797" y="253" text-anchor="middle">pipelex-temporal</text>+ </g>+ <g stroke="#3fb950" stroke-width="1.3" marker-end="url(#arrG)" fill="none">+ <path d="M568,232 L660,175"/><path d="M674,232 L688,175"/><path d="M797,232 L720,175"/>+ </g>+ <text x="690" y="296" text-anchor="middle" font-size="11" fill="#9aa7b4">register(registrar) — at discovery</text>+</svg>+<figcaption>The inversion in one picture: the <code>match</code> fan-out becomes a registry that plugins populate.</figcaption>+</figure>++<table class="seam-table">+ <thead><tr><th>Seam (where core named an integration)</th><th>File(s)</th><th>How it's resolved now</th></tr></thead>+ <tbody>+ <tr><td>Inference dispatch — 4 families (LLM, image-gen, extract, search)</td><td><code>cogt/{llm,img_gen,extract,search}/*_worker_factory.py</code></td><td><code>InferenceBackendRegistry.lookup(family, sdk)</code></td></tr>+ <tr><td>Model listing (the <code>show models</code> path)</td><td><code>cogt/model_backends/model_lists.py</code></td><td><code>ModelListerRegistry.get_optional(sdk)</code></td></tr>+ <tr><td>Orchestrator dispatch (Direct / Temporal / Mistral)</td><td><code>runtime_bridge/bridge.py</code></td><td><code>OrchestratorRegistry.get_optional(mode)</code> + <code>DeliveryMode</code></td></tr>+ <tr><td>Bundle validation dispatch (<code>/validate</code>)</td><td><code>pipeline/*</code>, <code>runtime_bridge/*</code></td><td><code>BundleValidatorRegistry.get_optional(mode)</code></td></tr>+ <tr><td>HTTP transport-fault rendering</td><td>host runtimes such as <code>pipelex-api</code></td><td>plugin-contributed lazy <code>ErrorReport</code> mappers</td></tr>+ <tr><td>Boot / teardown hub-swap (Temporal runtime)</td><td><code>pipelex.py</code></td><td>hub-slot claims + LIFO teardown callbacks</td></tr>+ <tr><td>Config schema + plugin config files</td><td><code>configs.py</code>, <code>config_loader.py</code></td><td>Temporal schema relocated to core; plugin TOMLs load through env-aware layering</td></tr>+ <tr><td>Operational CLI commands</td><td>external plugin packages</td><td>ops commands ship as plugin-owned console scripts, not harvested onto <code>pipelex</code></td></tr>+ </tbody>+</table>+<p class="legend">Every row above used to contain a vendor or <code>temporalio</code> branch. After the refactor, dispatch and boot paths use registries; built-in vendors are named only by their plugin modules/list, typed config, and prose.</p>++<!-- ============================================================ -->+<h2>2 · The design — one contract, one registrar, one discovery pass</h2>+<p>A plugin is a tiny object with a <code>register</code> method. Discovery hands it a <strong>registrar</strong> (a menu of "I provide X" methods); the registrar accumulates contributions into registries that get installed on the hub at boot. That's the whole machinery.</p>++<h3>The contract</h3>+<div class="code"><pre><code><span class="cmt"># pipelex/plugins/contract.py</span>+PLUGIN_API_VERSION: int = 2++@runtime_checkable+<span class="kw">class</span> <span class="fn">PipelexPlugin</span>(Protocol):+ name: str+ targets_api: int+ <span class="kw">def</span> <span class="fn">register</span>(self, registrar: "PluginRegistrar") -> None: ...+ <span class="cmt"># INVARIANT: register is side-effect-free — it may ONLY call registrar</span>+ <span class="cmt"># menu methods. No hub access, no I/O, no SDK/client construction.</span>+ <span class="cmt"># Heavy work happens lazily inside worker closures, slot thunks,</span>+ <span class="cmt"># plugin config loaders, and HTTP-error type providers.</span></code></pre></div>++<h3>The discovery → boot flow</h3>+<pre class="mermaid">+sequenceDiagram+ autonumber+ participant Boot as Pipelex.setup()+ participant Disc as build_registrar(config)+ participant Built as BUILTIN_PLUGINS+ participant EP as entry_points("pipelex.plugins")+ participant Reg as PluginRegistrar+ participant Hub as Hub+ Boot->>Disc: build_registrar(config)+ Disc->>Built: iterate built-in plugins+ Built->>Reg: plugin.register(registrar)+ Disc->>EP: iterate installed dists (e.g. pipelex-temporal)+ EP->>Reg: plugin.register(registrar)+ Note over Reg: side-effect-free — menu calls only, imports no SDK+ Disc-->>Boot: registrar (registries + slot thunks + teardown cbs + mapper providers)+ Boot->>Hub: set inference / orchestrator / validator / model-lister registries+ Boot->>Hub: resolve hub slots (explicit param > slot thunk > core default)+</pre>+<figcaption>One pure pass builds the registrar; boot installs the result. Because <code>register</code> is side-effect-free, the pass is safely repeatable.</figcaption>++<h3>What the registrar offers</h3>+<table>+ <thead><tr><th>Menu method</th><th>Contribution</th></tr></thead>+ <tbody>+ <tr><td><code>add_inference_backend(family, sdk, make_worker)</code></td><td>a backend driver, keyed by <code>(family, sdk)</code></td></tr>+ <tr><td><code>add_orchestrator(mode, orchestrator)</code></td><td>an execution strategy, keyed by open <code>orchestration_mode</code></td></tr>+ <tr><td><code>add_bundle_validator(mode, validator)</code></td><td>a <code>/validate</code> strategy, keyed by <code>orchestration_mode</code></td></tr>+ <tr><td><code>add_model_lister(sdk, lister)</code></td><td>an optional model-listing capability</td></tr>+ <tr><td><code>add_http_error_mapper(exc_type_provider, to_error_report)</code></td><td>a lazy transport-exception → <code>ErrorReport</code> mapping for host runtimes</td></tr>+ <tr><td><code>claim_content_generator / pipe_router / pipe_run / task_manager(factory)</code></td><td>a hub-slot claim — a <em>deferred thunk</em>, run only if the slot is taken</td></tr>+ <tr><td><code>add_teardown(callback)</code></td><td>a teardown hook, run LIFO at shutdown</td></tr>+ </tbody>+</table>++<!-- ============================================================ -->+<h2>3 · The critical changes (minimal diffs)</h2>+<p>The heart of the work is a set of <code>match</code>-to-registry collapses. Pick a tab:</p>++<div class="tabs" data-group="diffs">+ <div class="tab-bar">+ <button class="tab-btn active" onclick="showTab('diffs',0)">Inference dispatch</button>+ <button class="tab-btn" onclick="showTab('diffs',1)">Orchestrator dispatch</button>+ <button class="tab-btn" onclick="showTab('diffs',2)">A driver plugin</button>+ <button class="tab-btn" onclick="showTab('diffs',3)">Temporal externalized</button>+ </div>++ <div class="tab-panel active">+ <p class="small"><code>cogt/llm/llm_worker_factory.py</code> — a ~230-line <code>match plugin.sdk</code> with ten vendor arms collapses to a registry lookup.</p>+ <div class="code"><pre><code><span class="ctx"> def make_llm_worker(inference_model, *, reporting_delegate=None):</span>+<span class="del">- match plugin.sdk:</span>+<span class="del">- case "gateway_completions": ... # ten vendor arms, each one:</span>+<span class="del">- case "anthropic" | "bedrock_anthropic": # require_sdk(...) + lazy import</span>+<span class="del">- client = AnthropicFactory.make_anthropic_client(...) # + build client</span>+<span class="del">- llm_worker = AnthropicLLMWorker(...) # + build worker</span>+<span class="del">- case "mistral": ... ; case "google": ...</span>+<span class="del">- case _:</span>+<span class="del">- raise NotImplementedError(f"{plugin.sdk} is not supported")</span>+<span class="del">- return llm_worker</span>+<span class="add">+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)</span>+<span class="add">+ backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)</span>+<span class="add">+ make_worker = get_inference_backend_registry().lookup(family=InferenceFamily.LLM, sdk=model_handle.sdk)</span>+<span class="add">+ worker = make_worker(inference_model=inference_model, backend=backend,</span>+<span class="add">+ sdk_clients=get_sdk_client_manager().sdk_client_registry,</span>+<span class="add">+ reporting_delegate=reporting_delegate)</span>+<span class="add">+ return cast("LLMWorkerInternalAbstract", worker)</span></code></pre></div>+ <p class="small">The same shape was applied to image-gen, extract, and search. A registry miss raises a structured <code>InferenceBackendNotFoundError</code> ("… is its plugin installed and enabled?"), never a <code>None</code> deref.</p>+ </div>++ <div class="tab-panel">+ <p class="small"><code>runtime_bridge/bridge.py</code> — the old <code>match execution_mode</code> (with hard <code>temporalio</code> / Mistral imports) became a registry lookup. Since then, the flat mode enum was split into an open <code>orchestration_mode</code> token plus a closed <code>DeliveryMode</code>.</p>+ <div class="code"><pre><code><span class="del">- match execution_mode:</span>+<span class="del">- case PipelexExecutionMode.DIRECT:</span>+<span class="del">- return await _run_direct(...)</span>+<span class="del">- case PipelexExecutionMode.TEMPORAL_BLOCKING:</span>+<span class="del">- return await _run_temporal_blocking(...) # imported temporalio</span>+<span class="del">- case PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:</span>+<span class="del">- return await _run_temporal_fire_and_forget(...)</span>+<span class="del">- case PipelexExecutionMode.MISTRAL_NATIVE:</span>+<span class="del">- return await _run_mistral_native(...) # imported mistral</span>+<span class="add">+ orchestrator = get_orchestrator_registry().get_optional(mode=input_payload.orchestration_mode)</span>+<span class="add">+ if orchestrator is None:</span>+<span class="add">+ raise MissingOrchestratorError(mode=input_payload.orchestration_mode)</span>+<span class="add">+ _validate_input(input_payload, orchestrator=orchestrator, delivery_assignment=delivery_assignment)</span>+<span class="add">+ return await orchestrator.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment,</span>+<span class="add">+ delivery=input_payload.delivery)</span></code></pre></div>+ <p class="small"><code>MissingOrchestratorError</code> is generic and plugin-decoupled. Fire-and-forget is checked against <code>supports_fire_and_forget</code> before dispatch, so blocking-only modes reject honestly instead of pretending to enqueue.</p>+ </div>++ <div class="tab-panel">+ <p class="small"><code>plugins/anthropic/anthropic_plugin.py</code> — a whole backend in one small file. <code>register</code> is trivial and import-light; the closure does the heavy lifting lazily.</p>+ <div class="code"><pre><code><span class="kw">def</span> <span class="fn">_make_anthropic_worker</span>(*, inference_model, backend, sdk_clients, reporting_delegate):+ require_sdk(spec="anthropic", extra="anthropic", msg=_ANTHROPIC_MISSING_MSG)+ <span class="kw">from</span> ...anthropic_factory <span class="kw">import</span> AnthropicFactory <span class="cmt"># lazy: no SDK at boot</span>+ <span class="kw">from</span> ...anthropic_llm_worker <span class="kw">import</span> AnthropicLLMWorker+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)+ sdk_instance = sdk_clients.get_or_create(handle=model_handle,+ build=<span class="kw">lambda</span>: AnthropicFactory.make_anthropic_client(model_handle=model_handle, backend=backend))+ <span class="kw">return</span> AnthropicLLMWorker(sdk_instance=sdk_instance, ...)++<span class="kw">class</span> <span class="fn">AnthropicPlugin</span>:+ name = "anthropic"+ targets_api = PLUGIN_API_VERSION+ <span class="kw">def</span> <span class="fn">register</span>(self, registrar):+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="anthropic", make_worker=_make_anthropic_worker)+ registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="bedrock_anthropic", make_worker=_make_anthropic_worker)+ registrar.add_model_lister(sdk="anthropic", lister=_list_anthropic_models)</code></pre></div>+ <p class="small">One plugin can serve several <code>(family, sdk)</code> keys and several families at once (Mistral registers into both LLM and Extract; Linkup into both Extract and Search).</p>+ </div>++ <div class="tab-panel">+ <p class="small">Once orchestrators went through the registry, Temporal no longer needed to live in core. It became its own private distribution, discovered by an entry point.</p>+ <div class="code"><pre><code><span class="cmt"># pipelex (core) — pyproject.toml: the temporal extra is gone</span>+<span class="ctx"> [project.optional-dependencies]</span>+<span class="del">-temporal = ["temporalio==1.24.0", "aiohttp>=3.14.0"]</span>+<span class="ctx"> [tool.pytest.ini_options] markers = [</span>+<span class="del">- "temporal: tests that require a Temporal server",</span>++<span class="cmt"># pipelex (core) — plugins/builtins.py: TemporalPlugin removed from the list</span>+<span class="ctx"> BUILTIN_PLUGINS = [ DirectOrchestratorPlugin(), OpenAIPlugin(), ... ]</span>+<span class="del">- TemporalPlugin(),</span>+<span class="add">+# Temporal is NOT here: it ships as the external pipelex-temporal distribution,</span>+<span class="add">+# discovered via its pipelex.plugins entry point. Zero config, no core temporalio import.</span></code></pre></div>+ <div class="code" style="margin-top:10px"><pre><code><span class="cmt"># pipelex-temporal (new dist) — pyproject.toml</span>+[project]+name = "pipelex-temporal"+dependencies = ["pipelex>=0.35.0", "temporalio==1.24.0", "aiohttp>=3.14.0"]++[project.scripts]+pipelex-temporal = "pipelex_temporal.temporal_cli:app" <span class="cmt"># worker / setup-namespace</span>++[project.entry-points."pipelex.plugins"]+temporal = "pipelex_temporal.temporal_plugin:TemporalPlugin" <span class="cmt"># <- discovery hook</span>++[tool.uv.sources]+pipelex = { path = "../_plugins", editable = true } <span class="cmt"># dev cross-pin → pipelex==Y at publish</span></code></pre></div>+ </div>+</div>++<!-- ============================================================ -->+<h2>4 · What ships in core today</h2>+<p>Core ships a list of built-in plugins. Importing this module imports <strong>no backend SDK</strong> — the SDKs load lazily inside the <code>make_worker</code> closures.</p>+<div class="code"><pre><code><span class="cmt"># pipelex/plugins/builtins.py</span>+BUILTIN_PLUGINS: list[PipelexPlugin] = [+ DirectOrchestratorPlugin(), <span class="cmt"># core orchestrator</span>+ OpenAIPlugin(), GatewayPlugin(), PortkeyPlugin(), AnthropicPlugin(),+ MistralPlugin(), BedrockPlugin(), GooglePlugin(), FalPlugin(),+ HuggingFacePlugin(), BlackboxaiPlugin(), OpenRouterPlugin(),+ AzureRestPlugin(), DoclingPlugin(), Pypdfium2Plugin(), LinkupPlugin(),+]+<span class="cmt"># Naming one of these in plugins.disabled is a config error, not a no-op:</span>+CORE_UNCONDITIONAL_PLUGIN_NAMES = frozenset({"direct", "openai"})</code></pre></div>+<p class="small">Between them these built-ins register every <code>(family, sdk)</code> inference backend across the four families. <code>direct</code> owns the in-process orchestration mode and bundle validator; <code>openai</code> is the always-on default driver. Observability: <code class="kbd">pipelex plugins list</code> shows what's discovered, what each registered, and the denylist state.</p>++<!-- ============================================================ -->+<h2>5 · Temporal as an external plugin</h2>+<p>Phase 5 lifted <code>pipelex/temporal/</code> into a sibling distribution, <strong>pipelex-temporal</strong> (top-level package <code>pipelex_temporal</code>). It depends <em>on</em> core; core depends on nothing Temporal. Installing the dist contributes the <code>"temporal"</code> orchestration mode, its async delivery capability, its bundle validator, and its optional boot-runtime slot claims — no core change.</p>++<pre class="mermaid">+graph TB+ subgraph core["pipelex (core, MIT) — refactor/Plugins-4"]+ bridge["runtime_bridge dispatch + validate"]+ orr["OrchestratorRegistry (on hub)"]+ bridge --> orr+ end+ subgraph pt["pipelex-temporal (PRIVATE dist) — main"]+ tp["TemporalPlugin (entry point: pipelex.plugins)"]+ cli["console script: pipelex-temporal worker / setup-namespace"]+ end+ tp -. "install via git+ssh: entry-point discovery" .-> orr+ pt -- "depends on" --> core+ subgraph priv["our private infra — installs pipelex-temporal via git+ssh"]+ w["pipelex-worker"]+ hosted["pipelex-api-hosted"]+ end+ subgraph pub["public packages (MIT) — drop the temporal coupling"]+ api["pipelex-api"]+ mw["pipelex-mistralai-workflows"]+ end+ w -. "git+ssh pin" .-> pt+ hosted -. "git+ssh pin (runner enqueues)" .-> pt+</pre>+<figcaption>Core never imports Temporal. The private dist plugs in through the <code>pipelex.plugins</code> entry point. Only <strong>our private infra</strong> installs it (via a <code>git+ssh</code> pin); the <strong>public MIT packages drop temporal</strong> — they can't depend on a private repo.</figcaption>++<details>+ <summary>The key prerequisite (Step 0b): why the config schema had to move first</summary>+ <p>Core config (<code>configs.py</code>) imported the Temporal config <em>schema</em>, which lived under <code>pipelex.temporal</code>. You cannot "keep config in core, move <code>temporal/</code> out" while core config imports from <code>temporal/</code>. So the schema (<code>config_temporal.py</code>) and the two exceptions it raises were relocated into core (<code>pipelex/system/configuration/{config_temporal,exceptions}.py</code>), re-based on <code>PipelexError</code>. After this, <strong>the last hard <code>pipelex.temporal</code> import in core was gone</strong> — and the move became pure packaging. The <code>RetryPolicy</code> field aliases to <code>Any</code> at runtime so the schema validates without importing <code>temporalio</code>.</p>+ <p class="small">A latent bug surfaced and was fixed along the way: a <code>(ValueError, PipelexError)</code> MRO meant <code>ValueError.__init__</code> won and <code>.message</code> was never set. Bases were reordered to <code>(PipelexError, ValueError)</code>, with a regression test sweeping every <code>PipelexError</code> subclass.</p>+</details>++<details>+ <summary>"Option A": why operational commands are a console script, not a plugin contribution</summary>+ <p>An earlier design harvested plugin-contributed CLI commands by running discovery while building the <code>pipelex</code> CLI. A review found this brittle: one broken or colliding external plugin (or an unreadable user config) could brick <em>every</em> command — including the <code>--help</code>/<code>doctor</code>/<code>plugins</code> recovery commands. <strong>We removed the surface instead of hardening it.</strong> Temporal's <code>worker</code> and <code>setup-namespace</code> now ship as a standalone <code>pipelex-temporal</code> console script (<code>[project.scripts]</code>), so <code>pipelex --help</code> no longer loads config or scans entry points, and a third party can't shadow a core command.</p>+</details>++<!-- ============================================================ -->+<h2>6 · The invariants that hold every phase</h2>+<div class="callout good"><strong>Import-light boot.</strong> Booting with built-ins registered imports no optional SDK (<code>anthropic</code>, <code>mistralai</code>, <code>boto3</code>, <code>fal_client</code>, …) and no <code>temporalio</code>. Enforced by a subprocess test with a <code>sys.meta_path</code> finder that <em>raises</em> on those modules — boot must not trip it.</div>+<div class="callout info"><strong>Side-effect-free <code>register</code>.</strong> A plugin's <code>register</code> may only call registrar menu methods — no hub access, no I/O, no client construction. This is what makes discovery a pure, repeatable function.</div>+<div class="callout warn"><strong>Fail loud.</strong> Duplicate <code>(family, sdk)</code> / duplicate mode / a double-claimed slot / an API-version mismatch / a broken entry point each raise a named, contextual error. A missing backend says "install X" — it never dereferences <code>None</code>.</div>++<h3>Presence vs activation (so you don't add a flag that shouldn't exist)</h3>+<table>+ <thead><tr><th>Question</th><th>Answer</th></tr></thead>+ <tbody>+ <tr><td>Is a plugin <em>present</em>?</td><td>It's in <code>BUILTIN_PLUGINS</code> or an installed dist exposes the <code>pipelex.plugins</code> entry point. Zero config.</td></tr>+ <tr><td>When does an inference backend <em>activate</em>?</td><td>By data — a model names its <code>sdk</code>; the registry resolves it.</td></tr>+ <tr><td>When does an orchestrator <em>activate</em>?</td><td>Per call — the request's open <code>orchestration_mode</code> selects the plugin; the endpoint-selected <code>DeliveryMode</code> says whether the caller waits.</td></tr>+ <tr><td>When does the Temporal <em>runtime</em> boot?</td><td>Only when <code>plugins.boot_orchestrator = "temporal"</code> — a per-process "boot as a Temporal worker" gate, not a generic on/off toggle.</td></tr>+ <tr><td>Can I turn a plugin off?</td><td>A narrow <code>plugins.disabled</code> denylist (can't disable a core-unconditional plugin). There is deliberately no redundant allowlist.</td></tr>+ </tbody>+</table>++<!-- ============================================================ -->+<h2>7 · Phase history</h2>+<p>The original plugin-system phases landed as checkpoint commits on <code>refactor/Plugins-3</code>. <code>refactor/Plugins-4</code> then added the consumer-facing seams needed for an orchestrator-agnostic API base and deployment flavors.</p>+<table>+ <thead><tr><th>Phase</th><th>What it did</th><th>Checkpoint commit</th></tr></thead>+ <tbody>+ <tr><td><b>0</b> · Rename</td><td>Freed the word "plugin": <code>Plugin→ModelHandle</code>, <code>PluginManager→SdkClientManager</code>, …</td><td><code>7213a90c0</code></td></tr>+ <tr><td><b>1</b> · Seam + LLM</td><td>Contract, registrar, discovery, registries; LLM family migrated (contract proven on a real family)</td><td><code>42e9f7797</code></td></tr>+ <tr><td><b>2</b> · Inference families</td><td>Image-gen / extract / search migrated; all four families go through the registry</td><td><code>d62628f0b</code>, <code>94fa908df</code></td></tr>+ <tr><td><b>3</b> · Orchestrators</td><td>Bridge <code>match</code> + four boot blocks + teardown → registry & hub-slot thunks</td><td><code>5f61db323</code></td></tr>+ <tr><td>· Option A</td><td>Dropped the CLI-harvest seam; ops commands → <code>pipelex-temporal</code> console script</td><td><code>989c9beed</code></td></tr>+ <tr><td><b>4</b> · Model listing</td><td>The 5th seam: <code>model_lists.py match</code> → <code>ModelListerRegistry</code></td><td><code>6a0796548</code>, <code>e4e021c2b</code></td></tr>+ <tr><td><b>5</b> · Step 0b</td><td>Relocated the Temporal config schema into core (the move's prerequisite)</td><td><code>7d64eb6ac</code></td></tr>+ <tr><td><b>5</b> · Cut-over</td><td>Externalized <code>pipelex/temporal/</code> → the <code>pipelex-temporal</code> dist</td><td><code>a8382d1a1</code>, <code>27da150a5</code></td></tr>+ <tr><td><b>5</b> · Finalize</td><td>Removed dead Temporal dev-tooling from core (skills + Makefile targets → relocated)</td><td><code>e410ae971</code></td></tr>+ <tr><td><b>Plugins-4 A</b></td><td>Added env-aware plugin-config loading and the HTTP error mapper seam; bumped <code>PLUGIN_API_VERSION</code> to 2</td><td><code>4b5449f88</code>, <code>75196e2e3</code></td></tr>+ <tr><td><b>Boot gate</b></td><td>Moved Temporal runtime activation from <code>temporal.is_enabled</code> to <code>plugins.boot_orchestrator</code></td><td><code>eaa671d10</code></td></tr>+ <tr><td><b>Mode split</b></td><td>Split flat <code>PipelexExecutionMode</code> into open <code>orchestration_mode</code> plus closed <code>DeliveryMode</code></td><td><code>bd4e52827</code></td></tr>+ <tr><td><b>Validate</b></td><td>Added per-call <code>BundleValidatorRegistry</code> dispatch for orchestrator-aware <code>/validate</code></td><td><code>bbe44c929</code></td></tr>+ </tbody>+</table>++<!-- ============================================================ -->+<h2>8 · Where we stand</h2>+<div class="callout good"><strong>Core foundation current.</strong> In this repo, built-in inference plugins, direct orchestration, model listing, HTTP mapper contributions, bundle validators, slot claims, and plugin discovery are wired through the registrar. The public core still imports no Temporal SDK at boot.</div>++<table>+ <thead><tr><th>Repo</th><th>Branch · tip</th><th>State</th></tr></thead>+ <tbody>+ <tr><td>pipelex (core)</td><td><code>refactor/Plugins-4</code> · <code>221a2c611</code></td><td class="status-ok">pushed — plugin API v2, mode/delivery split, bundle validator seam</td></tr>+ <tr><td>pipelex-temporal <span class="small">(private)</span></td><td><code>feature/Orchestrator-dispatched-validate</code></td><td class="status-ok">adopts the v2 seams: env-aware config, lazy Temporal error mapper, Temporal validator</td></tr>+ <tr><td>pipelex-api <span class="small">(public base)</span></td><td><code>refactor/orchestrator-agnostic-base</code> + follow-ons</td><td class="status-ok">Temporal dependency removed; consumes mapper and dispatch seams; follow-on execute/start work stacked</td></tr>+ <tr><td>pipelex-api-hosted <span class="small">(private flavor)</span></td><td>—</td><td class="status-wait">build Temporal flavor image: install <code>pipelex-temporal</code> via git+ssh and bake <code>pipelex_{env}.toml</code> + <code>temporal_{env}.toml</code></td></tr>+ <tr><td>pipelex-worker <span class="small">(private)</span></td><td>—</td><td class="status-wait">flip pin from <code>pipelex[temporal]</code> to <code>pipelex</code> + <code>pipelex-temporal</code>; CLI becomes <code>pipelex-temporal worker</code></td></tr>+ <tr><td>pipelex-mistralai-workflows <span class="small">(public)</span></td><td>feature branch</td><td class="status-ok">migrated to the open <code>orchestration_mode</code> / <code>DeliveryMode</code> surface in its plugin work</td></tr>+ </tbody>+</table>++<h3>What's left — deployment flavors and release pins</h3>+<p class="small"><code>pipelex-temporal</code> is <strong>private / not open-source</strong>, so it is <strong>not published to PyPI</strong>. It travels by a pinned <code>git+ssh</code> reference — exactly how this workspace already pins <code>pipelex-shared</code> (<code>… @ git+ssh://[email protected]/Pipelex/infra-python-tools.git@<ref></code>).</p>+<ol>+ <li>Build the private Temporal API flavor: base <code>pipelex-api</code> image + one <code>pipelex-temporal</code> git+ssh install + env config child images.</li>+ <li>Move Temporal connection settings into plugin-owned <code>temporal_{env}.toml</code>; the main <code>pipelex_{env}.toml</code> keeps only core/base settings such as <code>[plugins] boot_orchestrator = "temporal"</code>.</li>+ <li>Flip <code>pipelex-worker</code> to the external plugin in the same release window as the public core/API pins.</li>+ <li>Run consumer suites against the exact public version + private plugin ref pair before deploying hosted Temporal.</li>+</ol>+<p class="callout warn"><strong>Known boundary:</strong> the plugin system now supports both private and public orchestrator plugins, but the release/install channel is still per plugin. Private Temporal rides git+ssh; public Mistral rides PyPI once published. Do not add a central allowlist or a generic "enable plugin" flag to bridge that packaging difference.</p>++<!-- ============================================================ -->+<h2>9 · Links</h2>+<ul>+ <li><strong>pipelex (core):</strong> <a href="https://github.com/Pipelex/pipelex">github.com/Pipelex/pipelex</a> · branch <a href="https://github.com/Pipelex/pipelex/tree/refactor/Plugins-4">refactor/Plugins-4</a></li>+ <li><strong>pipelex-temporal:</strong> <a href="https://github.com/Pipelex/pipelex-temporal">github.com/Pipelex/pipelex-temporal</a> <span class="small">(private — git+ssh install)</span></li>+ <li><strong>Deployment consumers:</strong> <a href="https://github.com/Pipelex/pipelex-worker">pipelex-worker</a> <span class="small">(private)</span> · <a href="https://github.com/Pipelex/pipelex-api-hosted">pipelex-api-hosted</a> <span class="small">(private Temporal flavor)</span> · <a href="https://github.com/Pipelex/pipelex-api">pipelex-api</a> <span class="small">(public base)</span> · <a href="https://github.com/Pipelex/pipelex-mistralai-workflows">pipelex-mistralai-workflows</a> <span class="small">(public plugin)</span></li>+</ul>++<hr>+<p class="small">Updated for co-developer review · core <code>refactor/Plugins-4</code> @ <code>221a2c611</code>. Diagrams render with Mermaid (loaded from CDN when this file is opened in a browser).</p>++</div>++<script>+function showTab(group, idx){+ document.querySelectorAll('.tabs[data-group="'+group+'"] .tab-btn').forEach(function(b,i){b.classList.toggle('active', i===idx);});+ document.querySelectorAll('.tabs[data-group="'+group+'"] .tab-panel').forEach(function(p,i){p.classList.toggle('active', i===idx);});+}+</script>+<script type="module">+ import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs';+ mermaid.initialize({ startOnLoad: true, theme: 'dark', securityLevel: 'loose' });+</script>+</body>+</html>diff --git a/wip/plugins/resume-and-verify-state.md b/wip/plugins/resume-and-verify-state.mdnew file mode 100644index 000000000..73450d7e1--- /dev/null+++ b/wip/plugins/resume-and-verify-state.md@@ -0,0 +1,63 @@+# Resume & verify state — orchestrator-agnostic runner effort++Paste-ready orientation for picking up this multi-repo effort in a fresh session. It tells a new agent **what is true**, **how to verify it against the repos** (don't trust this doc over `git`), and **what's next**. Keep it current when the effort's state changes (it's a companion to `TODOS.md`, not a replacement).++The workspace is multi-repo under `/Users/lchoquel/repos/Pipelex/` (`_plugins` is the `pipelex` core worktree).++---++## 1. Read the trackers first (source of truth)++- [`TODOS.md`](../../TODOS.md) — the execution tracker (Phases A–F + a "Follow-on extensions" section). Read the **Status** block, **Follow-on extensions**, and the **Cold-start primer**.+- [`orchestrator-agnostic-runner-and-flavors.md`](orchestrator-agnostic-runner-and-flavors.md) — the decision-locked plan (the *why/how*; locked decisions D1/D2/F1/F2/F3 — don't re-litigate).+- [`execute-per-request-mode-deferred.md`](execute-per-request-mode-deferred.md) — deferred/flagged items from the two follow-ons. **Read before touching `/execute`, `/start`, or the OpenAPI generation.**++## 2. Expected state — verify each, flag any mismatch++```+for r in _plugins pipelex-api pipelex-temporal; do d=/Users/lchoquel/repos/Pipelex/$r; \+ echo "== $r =="; git -C "$d" branch --show-current; git -C "$d" log --oneline -1; \+ git -C "$d" status -sb | head -1; echo "unpushed: $(git -C "$d" log --oneline @{u}..HEAD | wc -l)"; done+```++Expected (the exact tips drift as docs land; the load-bearing checks are *branch*, *clean tree*, *0 unpushed*):++- **`_plugins`** (core, `Pipelex/pipelex`): branch `feature/Orchestrator-dispatched-validate`, clean, 0 unpushed (tip at/after `db4194a98`).+- **`pipelex-api`**: branch `feature/Execute-per-request-mode`, clean, 0 unpushed (tip `191c900`). Stack: `a39841e` (Phase C) → `72c0efc` (validate) → `191c900` (execute).+- **`pipelex-temporal`**: branch `feature/Orchestrator-dispatched-validate`, clean, 0 unpushed (tip `459e04d`).++PR #27 (the `/execute` follow-on) still merge-ready:++```+cd /Users/lchoquel/repos/Pipelex/pipelex-api && \+ gh pr view 27 --json state,mergeable,baseRefName,headRefName && \+ (gh pr checks 27 | grep -viE 'pass|skipping' || echo '(CI all green)')+```++Expected: `OPEN`, `MERGEABLE`, base `feature/Orchestrator-dispatched-validate`, CI all green, 0 open review threads.++Optional gate sanity (proves `pipelex-api` builds against the git-pinned core):++```+cd /Users/lchoquel/repos/Pipelex/pipelex-api && make agent-test # silent on success+```++## 3. What's DONE++- **Phases A + B + C** — `pipelex-api` is the orchestrator-agnostic base (MAJOR GATE 2 / "THE gate"). Committed + pushed.+- **Follow-on 1** — orchestrator-dispatched `/validate` (reverses/extends F2; new per-call `BundleValidatorRegistry` seam). Done across all 3 repos.+- **Follow-on 2** — per-request `execution_mode` on `/execute` (extends F1). PR #27, merge-ready, **NOT merged** (user merges).++## 4. What's NEXT — confirm go/no-go before starting++**Phase D** = `pipelex-api-hosted` Temporal flavor (MAJOR GATE 3): flavored base image (`FROM pipelex/pipelex-api` + `pipelex-temporal` via `git+ssh`), config migration (`[plugins] boot_orchestrator = "temporal"`, `[api] execution_mode`, connection tree → `temporal_{env}.toml`). See `TODOS.md` → "Phase D" + its checkpoint. **This is where outward-facing / deploy-breaking work begins — confirm with the user before any actual deploy.**++## 5. Open threads to keep in mind (not blocking)++- PR #27 and the validate branch are **stacked and unmerged to `dev`**; merging is the user's call.+- `pipelex-api`'s `pipelex` dep is pinned to a **core git SHA** (`51ff09417`) so CI resolves (the editable `../_plugins` path can't resolve on a CI runner). Both the SHA pin and the validate-branch stacking flip to PyPI `==` pins **at release**.+- A pre-existing **`/start` DIRECT-path resource leak** is flagged (deferred) in `execute-per-request-mode-deferred.md` — fix it on its own surface, not as part of Phase D.++---++**On resume:** run §2, then give a one-paragraph status confirmation (or a list of mismatches), and wait for the user to pick the next task.diff --git a/wip/plugins/temporal-as-plugin.md b/wip/plugins/temporal-as-plugin.mdnew file mode 100644index 000000000..9f7332cf6--- /dev/null+++ b/wip/plugins/temporal-as-plugin.md@@ -0,0 +1,90 @@+# Temporal as a Pipelex plugin++**Status:** assessment / proposed plan (not started)+**Scope:** the concrete, seam-by-seam extraction of the Temporal integration into a first-class plugin — wired through the shared orchestrator seam rather than named by string from core. An independent release cadence becomes a *possible consequence* of this, not the goal.++> This is the **Temporal-specific** detail of one orchestrator. The shared mechanism — the `Plugin` protocol, entry-point discovery, the execution-mode → orchestrator registry, the published orchestrator SPI, and the activation-model menu of hooks — lives in [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md), where Temporal and `pipelex-mistralai-workflows` are the two worked instances. The **driver** half (inference SDK wrappers) is in [`inference-backends-as-plugins.md`](inference-backends-as-plugins.md); cross-cutting best practices are in [`README.md`](README.md). Read the category doc first; this one assumes it.++## Verdict++Highly separable — the runtime is already ~90% decoupled. All Temporal code lives in `pipelex/temporal/` (plus its tests under `tests/{unit,integration}/pipelex/temporal/`). The dependency direction is one-way: `pipelex/temporal/` imports *from* core, and core almost never names `pipelex.temporal` back. The runtime already injects the Temporal implementations through the hub (`set_content_generator`, `set_pipe_router`, `set_pipe_run`, plus `temporal_hub.set_task_manager`) behind existing protocols, and nearly every reference is a **lazy** import gated on `if get_config().temporal.is_enabled:`. `temporalio` is already an opt-in extra, and `config_temporal.py` is deliberately importable *without* the SDK (a `TYPE_CHECKING` / `Any` placeholder stands in for `RetryPolicy`).++So this is **not** a refactor of the Temporal code. It is about (1) defining a plugin seam in core, and (2) inverting the handful of places where core still names `pipelex.temporal` so they go through that seam instead.++## How Temporal uses the orchestrator seam++Temporal is the **heaviest** instance of the orchestrator category — it exercises nearly every hook the shared `Plugin` protocol offers (Mistral, the other instance, uses only a subset; that contrast is what keeps the protocol generic — see the category doc's "activation models" section). What Temporal needs:++- **`register_hub_implementations(hub, config)`** — inject `ContentGeneratorInWorkflow`, `TemporalPipeRouter`, `TemporalPipeRun`, `TemporalTaskManager` when its config slice is enabled. This is the **boot-global** activation model; Temporal is the only current backend that needs it (inside a Temporal worker the content generator must dispatch each operation as an activity).+- **`register_orchestrators(registry)`** — own the `TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET` orchestrators the bridge dispatches to (the **per-call** activation model, shared with Mistral).+- **`register_cli_commands(app)`** — add the `worker` and `setup-temporal-namespace` Typer commands.+- **`register_config_section(...)`** — contribute (or validate) its `temporal` config slice. (See config decision below — this hook is only load-bearing if the schema moves out of core.)+- **lifecycle** — `setup()` / `teardown()` so the worker/task-manager teardown currently inlined in `pipelex.py` moves behind the plugin.++## What's already in our favour++- **One package, one direction.** Everything Temporal-specific is under `pipelex/temporal/` (`tprl_pipe/` router+run workflows, `tprl_content_generation/` activities, `codec/`, `temporal_data_converter.py`, `temporal_task_manager.py`, `config_temporal.py`, `exceptions.py`, `worker_cli.py`). Core → temporal references are the only thing to cut.+- **Hub injection seams exist.** `PipelexHub` already accepts swappable implementations behind protocols (`set_content_generator`, `set_pipe_router`, `set_pipe_run`); `temporal_hub.set_task_manager` covers the task manager. The Temporal variants (`ContentGeneratorInWorkflow`, `TemporalPipeRouter`, `TemporalPipeRun`, `TemporalTaskManager`) all implement those protocols.+- **Protocols exist.** `ContentGeneratorProtocol`, `PipeRouterProtocol`, `PipeRunProtocol` already abstract the swap points.+- **Opt-in extra.** `pyproject.toml`: `temporal = ["temporalio==1.24.0", "aiohttp>=3.14.0"]`. The SDK is never a hard dependency of base.+- **SDK-free config.** `config_temporal.py` imports no `temporalio` at runtime (`if TYPE_CHECKING: from temporalio.common import RetryPolicy` / `else: RetryPolicy = Any`), so the config schema is importable on installs that skipped the extra.+- **SDK-free capability detection.** `reporting/reporting_manager.py` detects "am I inside a Temporal activity?" via `sys.modules.get("temporalio.activity")` — no import, so it never pulls the SDK onto the boot hot path. This pattern survives extraction unchanged.++## The complete list of seams to cut++| # | Site | Current coupling | Hard or lazy |+|---|------|------------------|--------------|+| 1 | `system/configuration/configs.py` | `from pipelex.temporal.config_temporal import Temporal` (top import) + `temporal: Temporal` field on `PipelexConfig` | **HARD** (the only non-lazy import of the package) |+| 2 | `pipelex.py` boot | content generator (`ContentGeneratorInWorkflowFactory`), task manager (`Tasks` + `temporal_hub` + `TemporalTaskManager`), pipe router (`make_temporal_pipe_router`), pipe run (`make_temporal_pipe_run`) — all gated on `get_config().temporal.is_enabled`, injected via the hub | lazy |+| 3 | `pipelex.py` teardown | `temporal_hub` reset + `TemporalTaskManager.teardown()`, gated on a stored handle | lazy |+| 4 | `runtime_bridge/bridge.py` | `_run_temporal_blocking` / `_run_temporal_fire_and_forget` (lazy imports of `temporal.exceptions` + `make_temporal_pipe_run`) + `_require_pipelex_temporal_extra()` availability check | lazy |+| 5 | `cli/_cli.py` | top imports of `worker_cmd` / `setup_temporal_namespace_cmd` + their `app.command(...)` registration | **HARD** |+| 6 | `cli/commands/worker_cmd.py` | lazy `from pipelex.temporal.temporal_hub import get_task_manager` + `temporal_enabled=True` boot flag | lazy (rides with the command, moves wholesale) |+| 7 | `reporting/reporting_manager.py` | `sys.modules.get("temporalio.activity")` — introspection only, never imports | already decoupled — leave as-is |++Only **#1 and #5 are hard imports.** Everything else is already behind a lazy/gated seam and just needs to call through a registry instead of a literal module path.++## Recommended approach: invert Temporal's seams onto the shared registry++The generic plumbing — the `Plugin` protocol, entry-point discovery, the execution-mode → orchestrator registry, and the published orchestrator SPI — is defined once in [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md) (Phase 0 there). It does **not** belong to Temporal. This section assumes that seam exists and covers only what's Temporal-specific.++1. **Invert seams #2–#6 onto the seam.** Replace the four `if is_enabled: from pipelex.temporal...` boot blocks and the matching teardown with `register_hub_implementations` / lifecycle hooks. The bridge's run-mode dispatch (`TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`) becomes an orchestrator lookup the plugin populated, so `bridge.py` no longer names `pipelex.temporal` at all. CLI registration in `_cli.py` becomes plugin-driven `app.command(...)` instead of two hard imports.++2. **Decide the config-field cut line (#1) — the one real decision.** Two options:+ - **(Recommended) Keep the `Temporal` config schema in open `pipelex`** and move only the *implementation*. The schema is pure data with no `temporalio` runtime dependency; the genuine logic (workflows, activities, worker, converter, codec) is what moves. Lowest churn, keeps the root config statically typed, makes seam #1 a no-op. The `register_config_section` hook then stays dormant for Temporal.+ - **Move config out too** for a truly zero-`pipelex.temporal` open repo: replace the typed `temporal: Temporal` field with a pluggable section the plugin parses its own slice of. Cleaner boundary, but loses static typing on that subtree and adds validation plumbing — this is the only thing that makes `register_config_section` load-bearing.++3. **Repackage as a real plugin dist.** Move `pipelex/temporal/` and its tests into `pipelex-temporal`, depending on `pipelex` + `temporalio`, declaring the `pipelex.plugins` entry point. Downstream consumers (`pipelex-worker`, `pipelex-api-hosted`) pin `pipelex-temporal` instead of `pipelex[temporal]`. The base `pipelex` `temporal` extra is dropped (or reduced to just `temporalio` for anyone integrating against the protocols without our implementation). **This is the step that makes an independent release cadence trivial** — but it's optional: the plugin can equally well stay in-repo as long as it's wired through the seam.++## Open question to settle before coding++- **Does the config schema (`config_temporal.py`) stay open/in-core or move into the plugin?** This drives whether seam #1 is a no-op (schema stays, only implementation moves) or requires a pluggable-config mechanism. Recommendation above is to keep it in core.++## The data converter / codec++`temporal_data_converter.py` + `codec/` (incl. `codec_server_cli.py`, `storage_payload_codec.py`) is the most logic-heavy piece, but it is self-contained and Temporal-specific — it moves wholesale with the package. It registers through Temporal's own client/worker config, not through any core seam, so it adds no new coupling to cut.++## Distribution mechanics++- New package `pipelex-temporal`, depends on `pipelex` + `temporalio`, declares `[project.entry-points."pipelex.plugins"]`. Published wherever fits (PyPI or `git+ssh`) — the seam is identical either way.+- The `temporal` pytest marker and the `--temporal-server` CLI option plumbing (in `conftest.py`) relocate with the tests.+- Downstream pins flip from `pipelex[temporal]==X` to `pipelex-temporal==Y` (which itself pins a compatible `pipelex`).+- The base `pipelex` `temporal` extra is dropped or reduced to just `temporalio` for protocol-level integrators.++## Effort & risk++- **Low-to-moderate.** The hard work (protocol-based injection, lazy gating, SDK-free config, SDK-free activity detection) already landed with the runtime-bridge work. The remaining work is mostly mechanical: build the plugin/registry seam, rewire the small number of call sites, make CLI + run-mode registration dynamic, lift the package + tests.+- **Watch-items:**+ - The config-field decision above.+ - The data converter / codec is logic-heavy but self-contained.+ - The Temporal test suite (unit + integration) moves wholesale, with the marker and the `--temporal-server` option.+ - `reporting_manager.py`'s `sys.modules` check must keep working without importing `pipelex.temporal` (it already does).+ - Don't build the seam to Temporal's shape. Temporal exercises every hook (boot-global hub swap + per-call modes + CLI + config), which makes it tempting to over-fit. `pipelex-mistralai-workflows` uses only per-call mode registration and no boot swap — it's the counter-example that keeps the protocol a menu of hooks. See [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md).++## Suggested phasing++Assumes the shared seam from [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md) Phase 0 (Plugin protocol, entry-point discovery, execution-mode → orchestrator registry, SPI) already exists.++- **Phase 0 — decide config cut line.** Settle the open question. _Checkpoint: cut line agreed before any code moves._+- **Phase 1 — wire Temporal through the seam, in-tree.** Convert seams #2–#6: hub injection → `register_hub_implementations`; run-mode dispatch → registered handlers; CLI → `register_cli_commands`; teardown → lifecycle hook. No behaviour change (Temporal still in-repo, now registered through the seam instead of hard/lazy imports). _Checkpoint: full suite green with Temporal still in-repo but wired through the seam — proves it before extraction._+- **Phase 2 — extract.** Move `pipelex/temporal/` + tests into `pipelex-temporal`, wire its entry point, flip downstream pins, drop/reduce the base extra. _Checkpoint: base `pipelex` has zero `pipelex.temporal` references; `pipelex-temporal` test suite green against the published base._diff --git a/wip/plugins/temporal-config-out-of-core.md b/wip/plugins/temporal-config-out-of-core.mdnew file mode 100644index 000000000..ec32a7b69--- /dev/null+++ b/wip/plugins/temporal-config-out-of-core.md@@ -0,0 +1,137 @@+# Move Temporal config out of core (fix the core→`temporalio` type-check coupling)++**Status:** **Phases 1 & 2 DONE + verified green (Checkpoint 1 + Checkpoint 2) + COMMITTED (not pushed). Phase 3 (re-merge) deferred to a fresh session.** Core commit `eaa671d10` on `refactor/Plugins-3` (worktree `_plugins`); pipelex-temporal commit `187e846` on `refactor/own-temporal-config` (branched off `main`). **Priority:** ship-blocker for the plugin-system effort — it falsifies the effort's headline invariant ("core names no integration — not by import, not by string — anywhere").++> **As-built summary (see the "As-built" section at the bottom for detail).** The `temporalio` type-coupling is gone from core: `config_temporal.py` deleted, no `temporal` field on `PipelexConfig`, core `make agent-check` (pyright + mypy) clean, `make tb` / `make agent-test` green, `make gep` + `make check-config-sync` green. `pipelex-temporal` now owns the rich config (`config_temporal.py` + `temporal.toml` + `load_temporal_config` + `temporal_hub` cache) and gates on the generic core `plugins.boot_orchestrator` selector; its `make agent-check` + `make agent-test` are green against the editable new core. Both CLIs smoke clean (`--temporal/--no-temporal` → `--orchestrator`).++## The defect++`pipelex/system/configuration/config_temporal.py` lives in **core** but type-imports `from temporalio.common import RetryPolicy` (under `TYPE_CHECKING`, with a runtime `RetryPolicy = Any` placeholder). Core declares **no** `temporalio` dependency anywhere — not runtime, not an extra, not `dev`. So a clean type-check cannot resolve the import:++```+config_temporal.py:15 - error: Import "temporalio.common" could not be resolved (reportMissingImports)+… 13 pyright errors total, + the equivalent mypy import-not-found+```++It only ever passed because both the `_plugins` and `_workflows` venvs carried a **stale** `temporalio 1.24.0` left over from before the Phase 5 Temporal externalization. `make li` (lock + `uv sync --all-extras`) removes it and `make check` fails identically in `_plugins` itself. The merge of `refactor/Plugins-3` into `feature/mistralai-2x-bump` surfaced it because that merge correctly **drops the `temporal` extra** (public-repo rule), and the subsequent clean sync removed `temporalio`.++### Root cause++The plan's **D7** ("Temporal config stays typed in core") plus **Step 0b** (which *relocated* the Temporal config schema *into* core to break the `configs.py → pipelex.temporal` import cycle). That broke the cycle by importing Temporal's whole schema into core instead of inverting the dependency. Now that Temporal is an external plugin (`pipelex-temporal`), its config schema has no business in core.++### Rejected alternative++Adding `temporalio` to core's `dev` extra so the type-only import resolves: re-couples core's dev/CI environment to a SDK we just externalized and leaves the schema in core. Perpetuates exactly the coupling this effort exists to remove. Rejected.++## Decision (locked with the user)++**Move the Temporal config out of core entirely**, mirroring how `backends.toml` / `routing_profiles.toml` already work (separate `.pipelex/` TOML, resolved via `config_manager.resolve_config_file(...)`, parsed by their own subsystem at setup — *not* fields on `PipelexConfig`).++**Activation gate = generic core gate** (chosen over keeping a `--temporal` flag or dropping the flag): replace the temporal-specific `--temporal/--no-temporal` + `temporal_enabled` + `config.temporal.is_enabled` plumbing with a backend-agnostic selector. Core names **no** orchestrator; the plugin matches on its own name.++## Target architecture++The key property: the rich Temporal schema (the one with the `temporalio` import) **never loads in core or in `register`**. It loads only inside the boot thunks and `orchestrator.run` — all in `pipelex-temporal`, at runtime, where `temporalio` is a real dependency.++- **Generic gate in core config:** `PluginsConfig` gains `boot_orchestrator: str | None` (Optional → `= None` in the class def per repo rule; absent from `pipelex.toml`). It means "boot *this process* under the orchestrator plugin of this name" (`None`/any non-matching name = in-process default). This is plugin-system wiring, legitimately core's concern (core owns the orchestrator registry + boot slots) and carries no integration name.+- **`register` gates generically:** the Temporal plugin's `register` claims the boot slots iff `registrar.config.plugins.boot_orchestrator == self.name`. No temporal config read, no I/O, no `temporalio`. The TEMPORAL_* orchestrators are still added unconditionally (they need no config).+- **Rich config self-loads in the plugin:** `pipelex-temporal` ships its own default `temporal.toml`, reads user overrides from `.pipelex/temporal.toml` (layered project/global via `config_manager.resolve_config_file("temporal.toml")`), deep-merges, and parses into its `Temporal` schema — inside the boot thunks (apply-point, I/O fine) and `orchestrator.run` (runtime). The `temporalio` type-import travels with the schema into `pipelex-temporal`.+- **CLI:** `--temporal/--no-temporal` → `--orchestrator TEXT`; `temporal_enabled: bool | None` → `boot_orchestrator: str | None` through `cli_factory` → `Pipelex.setup()`. The override site (`pipelex.py:206-207`) sets `config.plugins.boot_orchestrator` instead of mutating `config.temporal.is_enabled`. `--orchestrator temporal` claims temporal's slots; `--orchestrator direct` (or anything ≠ a boot-orchestrator plugin name) → no claim → in-process. No sentinel needed: the gate is a name match.++## Core footprint (small)++- `configs.py:14` import of `Temporal` + `configs.py:257` `temporal: Temporal` field → removed.+- `pipelex.py:206-207` — the only core code reading `config.temporal` (the override) → rewired to `boot_orchestrator`.+- `--temporal/--no-temporal` flags threaded through: `cli/cli_factory.py`, `cli/commands/validate/{pipe,bundle,method}_cmd.py` + `_validate_core.py`, `cli/commands/run/bundle_cmd.py`, `pipelex.py` setup signatures.+- `[temporal]` TOML blocks in `pipelex/pipelex.toml`, `.pipelex/pipelex.toml`, `.pipelex/pipelex_override.toml`, **and** the kit templates under `pipelex/kit/configs/` (they seed `~/.pipelex`; `make check-config-sync` enforces `.pipelex ↔ kit` parity, so both sides move together). `tracing_config.temporal_dynamodb` is the separate tracing axis and **stays**.+- `config_temporal.py` (the 760-line schema) + the two exceptions it owns (`TemporalConfigError`, `WorkerTaskQueueUnknownError` in `pipelex/system/configuration/exceptions.py`) → moved to `pipelex-temporal`. `pipelex-temporal` already consumes the schema from core (its `temporal_connect`/`temporal_tasks`/`task_manager`/`temporal_task_manager`/`temporal_manager`/`namespace_check` import `pipelex.system.configuration.config_temporal`), so those imports become local.++## Plan (phased, cross-repo)++### Phase 1 — `pipelex-temporal` owns its config (additive, no core change yet)++- Add `config_temporal.py` to `pipelex_temporal/` (copy from core; the `temporalio` type-import now sits where `temporalio` is a real dep). Move `TemporalConfigError` + `WorkerTaskQueueUnknownError` into `pipelex_temporal`'s own `exceptions.py`.+- Add `pipelex_temporal/temporal.toml` = the default config (the relocated `[temporal]` block), shipped in the wheel.+- Add a config loader (`load_temporal_config() -> Temporal`) using `config_manager.resolve_config_file("temporal.toml")` + the packaged defaults, deep-merged.+- Repoint `pipelex-temporal`'s internal imports to its own `config_temporal` module.+- Keep the plugin reading core config for the gate *temporarily* (still `is_enabled`) so this phase stays green standalone.++🛑 **Checkpoint 1** — pipelex-temporal `make agent-check` + `make agent-test` green with its own schema (core still has its copy + field; nothing broken).++### Phase 2 — core sheds Temporal config + the generic gate (the breaking, atomic change)++- `PluginsConfig` += `boot_orchestrator: str | None = None`.+- Remove `temporal: Temporal` field + the `config_temporal` import from `configs.py`.+- Delete core `config_temporal.py` + the two exceptions from `pipelex/system/configuration/exceptions.py`.+- Remove every `[temporal]` block from core TOMLs (base + `.pipelex/*` + kit configs); keep `tracing_config.temporal_dynamodb`. Run `make check-config-sync`.+- Rewire CLI: `--temporal/--no-temporal` → `--orchestrator`; `temporal_enabled` → `boot_orchestrator` through `cli_factory` + `Pipelex.setup()`; the `pipelex.py` override site sets `config.plugins.boot_orchestrator`.+- Update the Temporal plugin `register` gate → `registrar.config.plugins.boot_orchestrator == self.name`; boot thunks + orchestrators self-load the rich config via the Phase-1 loader.+- Regenerate error pages (`make gep`) — the two relocated exceptions leave core's error catalog.++🛑 **Checkpoint 2** — **core `make check` pyright/mypy CLEAN (the `temporalio` import is gone from core), `make tb` green, `make agent-test` green**; `pipelex-temporal` `make agent-check` + `make agent-test` green (editable core). This is the gate that proves the defect fixed.++### Phase 3 — re-merge + re-sync++- Re-merge `refactor/Plugins-3` → `feature/mistralai-2x-bump` (worktree `_workflows`); the only expected conflict is `CHANGELOG.md` (union, `[Unreleased]` on top). `uv sync --all-extras` in `_workflows`. `make tb` + `make agent-check` + `make agent-test`.+- Then resume the original task (make `pipelex-mistralai-workflows` a discoverable `pipelex.plugins` entry-point plugin) against the corrected base.++## Risks / watch-items++- **`extra="forbid"` is unforgiving:** a single missed `[temporal]` key in any loaded TOML hard-fails boot. The TOML sweep (base + `.pipelex/*` + kit + tests) must be exhaustive; `make tb` is the fast catch.+- **Cross-repo ordering:** `pipelex-temporal` must have its own schema (Phase 1) before core deletes its copy (Phase 2), or the editable `pipelex-temporal` build breaks. Verify both green together under the editable pin before committing either.+- **Tests in core that assert `--temporal` / `config.temporal`:** update to `--orchestrator` / `boot_orchestrator`. Tests that reference the Temporal *schema* should already live in `pipelex-temporal`.++## Deferred (out of scope, follow-ups)++- **Tracing naming axis:** `TracingBackend.TEMPORAL_DYNAMODB`, `TemporalDynamoDBTracingConfig`, `tracing_config.temporal_dynamodb` are string/naming couplings (no `temporalio` import, don't break the build). Whether a fully-externalized Temporal should still have core's tracing system name a `temporal_dynamodb` backend is a separate question — leave for later.+- **Exact `--orchestrator` vocabulary** for force-in-process (`direct` vs `none`) — settled: the flag takes any plugin name; `--orchestrator temporal` boots Temporal, omitting it (or any non-boot-orchestrator name) runs in-process. No sentinel keyword is reserved.+- **Whether other (inference) plugin config should also leave `PipelexConfig`** — user explicitly scoped this pass to Temporal only.++## As-built (Phases 1 & 2 complete, uncommitted)++### What landed++**`pipelex-temporal` now owns the rich Temporal config** (new files):++- `pipelex_temporal/config_temporal.py` — the 760-line schema, relocated from core. The `temporalio.common.RetryPolicy` type-import now sits where `temporalio` is a real dependency. `is_enabled` dropped from the `Temporal` model (the gate moved to core).+- `pipelex_temporal/temporal.toml` — the packaged default (the old core `[temporal]` block, prefix-stripped to root tables, `is_enabled` removed). Shipped in the wheel; loaded as the base layer.+- `pipelex_temporal/temporal_config_loader.py` — `load_temporal_config()`: packaged default deep-merged with an optional `.pipelex/temporal.toml` override (project→global via `config_manager.resolve_config_file`). Mirrors `backends.toml` / `routing_profiles.toml`.+- `pipelex_temporal/temporal_activation.py` — `is_temporal_boot_active()` + `TEMPORAL_PLUGIN_NAME = "temporal"`. The runtime gate (`config.plugins.boot_orchestrator == "temporal"`), replacing the old `config.temporal.is_enabled` guard.+- `temporal_hub` caches the loaded `Temporal` (lazy-load + `set_temporal_config` for tests, cleared by `reset()`). All ~130 runtime `get_config().temporal.X` reads → `get_temporal_config().X`. Exceptions `TemporalConfigError` / `WorkerTaskQueueUnknownError` moved into `pipelex_temporal/exceptions.py`.+- Plugin gate flipped to `registrar.config.plugins.boot_orchestrator == self.name`; `worker_cli` / `worker_cmd` / `setup_namespace_cmd` / `codec_server_cli` boot with `Pipelex.make(boot_orchestrator=TEMPORAL_PLUGIN_NAME)`. The old `worker_cli` `is_enabled` force-on block is gone (redundant under the explicit boot gate).++**Core (`pipelex`) shed everything Temporal-config**:++- `PluginsConfig += boot_orchestrator: str | None = None` (generic gate; absent from `pipelex.toml`, set programmatically).+- Deleted `pipelex/system/configuration/config_temporal.py` **and** `pipelex/system/configuration/exceptions.py` (its only importer was `config_temporal`); removed `temporal: Temporal` from `PipelexConfig`.+- CLI: `--temporal/--no-temporal` (`bool|None`) → `--orchestrator` (`str|None`); `temporal_enabled` → `boot_orchestrator` through `cli_factory` → `Pipelex.make`/`setup`; the `pipelex.py` override sets `config.plugins.boot_orchestrator`.+- Removed every `[temporal]` block from `pipelex/pipelex.toml`, `.pipelex/pipelex.toml`, `.pipelex/pipelex_override.toml`, `pipelex/kit/configs/pipelex.toml`; `tracing_config.temporal_dynamodb` stays. `make check-config-sync` green.+- `make gep` regenerated: removed the 2 relocated core exceptions **and** 9 pre-existing stale `pipelex-temporal` flow-error pages (left over from the Phase-5 externalization) — core's error catalog now names no Temporal error.++### Deviations from the written plan (deliberate)++- **Phase 1 did more than "repoint imports."** Repointing only the schema *imports* to the local module while still reading `get_config().temporal` would create nominal-type conflicts (core `WorkerScope` vs local `WorkerScope`). So Phase 1 also rewired all rich reads to `get_temporal_config()` (local). The gate/guards stayed on core `is_enabled` through Phase 1 (faithful copy keeps `is_enabled` in the local schema); `is_enabled` was dropped in Phase 2 with the gate flip. Checkpoint 1 (pipelex-temporal green, core untouched) held.+- **Phase 2 = core cut-over (Step B) + pipelex-temporal gate flip (Step C), landed together.** Core green on its own; the editable pin means pipelex-temporal's gate flip had to land with it. Both verified green together at Checkpoint 2.++### Local-environment migration (one-off, done)++The config loader also merges the **global** `~/.pipelex/pipelex.toml`, which was seeded from an old kit and still carried `[temporal]` → `extra="forbid"` boot failure. Backed up to `~/.pipelex/pipelex.toml.pre-temporal-cut.bak` and removed the stale block. CI is unaffected (fresh global is kit-seeded from the now-clean kit). **Any developer/deployment upgrading past this change must drop `[temporal]` from their `~/.pipelex/pipelex.toml`** (breaking change, no back-compat per repo policy).++### Headline invariant — verified++Core has **no** `from temporalio... import` anywhere and **no** `temporal` config field. Remaining `temporal` mentions are all out of scope: the intentional `sys.modules.get("temporalio.activity")` *sniff* in `reporting_manager` (a string lookup that explicitly avoids importing — the crash-free-without-temporalio pattern), the execution-mode enum's `requires_pipelex_temporal` mode vocabulary, prose comments/docstrings, and the deferred `temporal_dynamodb` tracing axis. None is a type-check coupling.++### Gates (all green)++- core: `make agent-check` (pyright 0 / mypy 0 / keyword-only / plxt), `make tb`, `make agent-test` (exit 0), `make gep`, `make check-config-sync`; `pipelex run pipe --help` / `validate pipe --help` show `--orchestrator`.+- pipelex-temporal: `make agent-check` (pyright 0 / mypy 0), `make agent-test` (unit + integration time-skipping) against editable new core; `pipelex-temporal --help` smokes.++### Committed (not pushed)++- Core: `eaa671d10` on `refactor/Plugins-3` (`_plugins`) — the doc update lands in the immediate follow-up commit.+- pipelex-temporal: `187e846` on `refactor/own-temporal-config` (branched off `main`).++### Phase 3 — pickup for a fresh session (NOT started)++Re-merge `refactor/Plugins-3` → `feature/mistralai-2x-bump` (worktree `_workflows`); the only expected conflict is `CHANGELOG.md` (union, `[Unreleased]` on top). `uv sync --all-extras` in `_workflows`, then `make tb` + `make agent-check` + `make agent-test`. The base is now the corrected post-temporal-cut tree, so the clean sync that previously surfaced the defect (dropping `temporalio`) will pass. Then resume the original task: make `pipelex-mistralai-workflows` a discoverable `pipelex.plugins` entry-point plugin.+- **Docs follow-up (deferred).** Updated `docs/under-the-hood/orchestrator-plugins.md` (gate + module paths). Still stale and not touched this pass: `docs/under-the-hood/pipe-routing-and-execution.md` and `docs/distributed-execution/task-routing.md` still show `temporal.is_enabled` and `[temporal.worker_config.*]` / `[temporal.queue_options.*]` TOML examples. Those config tables now live at **root** in `pipelex-temporal`'s `temporal.toml` (no `[temporal.]` prefix), so the examples need rewriting — and the distributed-execution config docs arguably belong in the `pipelex-temporal` repo now. Left as a separate docs pass (involves a relocation decision).diff --git a/wip/plugins/validate-dispatch-requires-boot-active.md b/wip/plugins/validate-dispatch-requires-boot-active.mdnew file mode 100644index 000000000..09cf0fd28--- /dev/null+++ b/wip/plugins/validate-dispatch-requires-boot-active.md@@ -0,0 +1,34 @@+# Deferred: dispatched `/validate` effectively requires `boot_orchestrator == "temporal"`++**Status:** observation surfaced during Phase V1 of `orchestrator-dispatched-validate.md`. Not a bug introduced by this work — a pre-existing property of the reused dispatch machinery. Deferred, not fixed.++## What the plan says++The plan's "Per-call, presence-based, NOT boot-gated" decision (and its resolution of old open question #5) states:++> `dispatch_dry_validate` needs only a reachable Temporal client + a worker with the validate workflow registered — **not** a runner booted-as-Temporal.++## What the code actually does++`TemporalBundleValidator.validate_bundles` reuses `dispatch_dry_validate` as-is. That calls `WorkflowExecutorFactory().create_executor(...).execute_workflow(...)`, and `WorkflowExecutor.temporal_client()` (`pipelex_temporal/tprl/workflow_caller.py`) opens with:++```python+if not is_temporal_boot_active():+ raise AsyncExecutionNotEnabledError.with_default_message()+```++`is_temporal_boot_active()` is `config.plugins.boot_orchestrator == "temporal"`. So the executor refuses to connect unless **this process booted under Temporal** — i.e. dispatched `/validate` does in practice require `boot_orchestrator == "temporal"`, contrary to the plan's decision #5.++## Why this is consistent (and why it's only an observation)++The **registration** is correctly presence-based / unconditional — the validator is contributed under both temporal modes regardless of `boot_orchestrator`, exactly like the Temporal run orchestrators. The runtime boot-active requirement is **identical** for the run orchestrators: `TemporalBlockingOrchestrator.run` → `make_temporal_pipe_run().run` → the same `WorkflowExecutor.temporal_client()` boot-active guard. So validate dispatch is no more boot-coupled than pipe-run dispatch already is; the validator faithfully mirrors the established orchestrator behavior.++The plan's decision #5 was about **the seam not being boot-*gated* at registration** (true — no hub slot is claimed, no `boot_orchestrator` branch in `register`). The phrase "not a runner booted-as-Temporal" overreaches for the *runtime* connect path, which the reused `dispatch_dry_validate`/executor enforces.++## Why not fix it here++Relaxing the `is_temporal_boot_active()` guard so a non-Temporal-booted runner can still open a Temporal client for dispatch is a change to the **shared executor** that governs *all* Temporal dispatch (run + validate), not validate alone. It is out of scope for this feature, would broaden blast radius well beyond `/validate`, and is only meaningful once the hosted runner's boot story (does the agnostic runner boot as temporal, or dispatch per-request without booting?) is settled in the parent effort's Phase D. Until then this is a no-op for correctness: a Temporal-flavor runner boots as Temporal anyway.++## If revisited++Decide the hosted runner's boot model first (Phase D). If a single agnostic runner must dispatch *some* requests to Temporal without booting as Temporal, the fix belongs in `WorkflowExecutor.temporal_client()` (gate on client reachability, not `boot_orchestrator`) and must be validated against the run-dispatch path too — then update the plan's decision #5 wording to match.diff --git a/wip/plugins/validate-start-mode-dispatch-duplication.md b/wip/plugins/validate-start-mode-dispatch-duplication.mdnew file mode 100644index 000000000..f39b9ff98--- /dev/null+++ b/wip/plugins/validate-start-mode-dispatch-duplication.md@@ -0,0 +1,34 @@+# Deferred: `start` / `validate_verdict` mode-dispatch duplication (pipelex-api)++Surfaced during the `/code-review` of the `feature/Orchestrator-dispatched-validate` staged diff in `pipelex-api`. Deferred (not fixed) as a design tradeoff, not a clear win.++## The observation++`ApiRunner.start` and `ApiRunner.validate_verdict` (both in `api/routes/pipelex/pipeline.py`) now share the same three-line dispatch shape:++1. `execution_mode = resolve_execution_mode(requested_execution_mode, config=get_api_config())` — resolve the effective mode FIRST, so a forbidden per-request override is refused with a 403 before any library load / registry lookup.+2. `arm = get_<orchestrator|bundle_validator>_registry().get_optional(mode=execution_mode)`.+3. `if arm is None: raise Missing<Orchestrator|BundleValidator>Error(mode=execution_mode)`.++The "resolve mode FIRST … refused (403) here, before any … load" comment is also near-duplicated between the two methods.++The two sites differ only in the registry getter (`get_orchestrator_registry` vs `get_bundle_validator_registry`) and the error class (`MissingOrchestratorError` vs `MissingBundleValidatorError`).++## Why deferred (not fixed)++A helper such as `_resolve_arm(registry_getter, missing_error_cls)` would collapse the two sites to one, but:++- It is only ~4 lines × 2, and the two registries / error types are genuinely distinct seams.+- The helper would take a registry-getter callable and an exception class as parameters — that is *added* indirection and surface area, the opposite of removing complexity.+- The user's review brief for this change was explicit: clean, solid software, no speculative abstractions, and "in case of any doubt, do NOT fix." This is a judgment call, not an unambiguous win.++So the code is left as-is.++## When this becomes worth doing++Revisit the extraction only if a concrete pull arrives:++- A third surface acquires the same resolve-then-dispatch shape (the contract would then live in three places).+- The override-policy ordering guarantee changes (e.g. the "403 before any load" point moves), so a maintainer would otherwise have to remember to edit every site and keep the parallel comments in sync.++At that point a single small helper carries its weight; until then the duplication is the lower-churn, lower-surface choice.diff --git a/wip/plugins/validate-verdict-subclass-parity-divergence.md b/wip/plugins/validate-verdict-subclass-parity-divergence.mdnew file mode 100644index 000000000..3330f9c6d--- /dev/null+++ b/wip/plugins/validate-verdict-subclass-parity-divergence.md@@ -0,0 +1,50 @@+# `/validate` verdict/fault discrimination: latent subclass parity divergence between the two arms++Status: deferred observation from the `/code-review --fix` of the `TemporalBundleValidator` staged diff (`pipelex-temporal`). **Not a present bug — no code change applied.**++## The two arms discriminate "verdict vs fault" differently++`/validate` produces a verdict-as-value: an invalid bundle's `ValidateBundleError` is *returned* as a structured `ErrorReport` (the invalid arm), while a genuine infra fault *propagates* (a no-verdict 5xx). The two registered bundle validators implement that split with different predicates:++- **Direct (in-process) arm** — `pipelex/pipeline/direct_bundle_validator.py`:++ ```python+ except ValidateBundleError as exc:+ return exc.to_error_report()+ ```++ An `except ValidateBundleError` clause catches `ValidateBundleError` **and any subclass of it**.++- **Temporal (dispatched) arm** — `pipelex_temporal/temporal_bundle_validator.py`:++ ```python+ recovered_report = workflow_error.to_error_report()+ if recovered_report.error_type != ValidateBundleError.__name__:+ raise+ return recovered_report+ ```++ The verdict crosses the worker boundary as a structured `ErrorReport` whose `error_type` is the raised exception's `type(self).__name__` (see `pipelex/base_exceptions.py`, `to_error_report` sets `error_type=type(self).__name__`). The arm recognizes a verdict by **exact string equality** with `"ValidateBundleError"`. A subclass would carry the *subclass* name, so this predicate would **not** match it.++## Why it is not a bug today++The validation cascade in `pipelex/pipeline/validate_bundle.py` always raises the concrete `ValidateBundleError` class itself for every content verdict, and there are **no `ValidateBundleError` subclasses anywhere in the tree** (`grep -rn "class .*(ValidateBundleError)"` is empty). So for every reachable invalid-bundle verdict today, `error_type == "ValidateBundleError"` and both arms agree: return the verdict.++## The latent divergence++If someone later introduces a `ValidateBundleError` subclass and raises it from the validation cascade:++- the **direct arm** would still return it as a verdict (its `except` catches subclasses), but+- the **Temporal arm** would `raise` it as a no-verdict 5xx (its name-equality rejects the subclass name).++Same bundle, same intended verdict, two different HTTP outcomes depending on which orchestrator served the request. That is a cross-backend parity break, of exactly the kind the dispatched-`/validate` design is meant to avoid (both backends must answer identically for the same bundle).++## Why deferred, not fixed++- It is latent: zero subclasses exist, so there is no failing input today.+- The string-equality predicate is forced by the wire boundary — the Temporal arm only has the recovered `ErrorReport` (`error_type` string), not the live exception type, so it *cannot* use `isinstance`/`except` the way the in-process arm does without a richer recovery contract.+- The clean fix is not local to the reviewed file: it would mean teaching the recovery layer to carry a "is-this-a-ValidateBundleError-or-subclass" signal (e.g. an `is_validate_verdict` flag on the recovered report, or a verdict marker class hierarchy the recovery preserves), then having **both** arms branch on that signal — a seam change touching `pipelex` and `pipelex-temporal`, well outside the staged diff.++## If/when a `ValidateBundleError` subclass is introduced++Re-open this. The minimal correct fix is to make verdict-recognition subclass-aware on the wire side — e.g. have `convert_pipelex_errors` / `recover_error_report` stamp a stable "validate verdict" discriminator that does not depend on the exact leaf class name, and have both bundle validators branch on that discriminator instead of on `ValidateBundleError.__name__` (Temporal) / `except ValidateBundleError` (direct). That keeps the two arms provably in lockstep regardless of the concrete exception class.