Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feat(runtime-bridge): extract framework-agnostic Pipelex runtime bridge (#969)

Louis Choquel5fc2cbde9d0f
diff --git a/.claude/skills/temporal-e2e-validate/SKILL.md b/.claude/skills/temporal-e2e-validate/SKILL.md
index 002d98ec4..f42ef170b 100644
--- a/.claude/skills/temporal-e2e-validate/SKILL.md
+++ b/.claude/skills/temporal-e2e-validate/SKILL.md
@@ -305,7 +305,9 @@ loaded only for the work at hand. Read them as needed:
- "validate temporal" / "full temporal test" / a broad regression check → `mode-2-setup.md`, then `mode-2-tiers.md`; offer the two batteries as opt-in extras.
- "validate temporal error handling" / "error report" / "error propagation" → Mode 1 Step 2b above (the precise pytest assertions) **plus** `mode-2-setup.md` + `mode-2-tiers.md` Step 5f (Tiers 13–16, cross-process).
-- "cost report" / "distributed cost" / "mock inference" → `mode-2-setup.md` (split workers), then `mode-2-tiers.md` Step 5b' (Tier 8b) — `--mock-inference` makes it free and deterministic; no live spend needed.
+- "cost report" / "distributed cost" / "mock inference" → `mode-2-setup.md` (split workers), then `mode-2-tiers.md` Step 5b' (Tier 8b). This routing is **scope-aware**: once a request has landed here (via one of those three triggers), read the scope manifest at the top of Step 5b' and run the arms it selects:
+ - **default** (bare "cost report" / "distributed cost" / "mock inference") → the free, deterministic mock arms only (`--mock-inference`); no live spend.
+ - **explicit spend opt-in** — the canonical token `full` (aliases `thorough`, `every`, `with-spend`) qualifying the cost request (e.g. "cost reporting full", "cost report — every arm") → **every** cost arm, real spend authorized: mock primary + cross-child + CSV cross-check + the `--no-costs` negative gate + live LLM + live img-gen + live extract. Run them all and report PASS/FAIL per arm. Do **not** treat bare "live" or "all" as a spend opt-in — they're too easily incidental ("live" also collides with the default mock arm, which already runs in LIVE mode); if that's the only signal, confirm with the user before spending real money.
- "queue options" / "runtime profile" / "routing" → `mode-2-setup.md`, then `routing-battery.md` and/or `queue-options-battery.md`.
- Anything image-related → `mode-2-setup.md` + `mode-2-tiers.md` Tiers 4–5, run **live** — dry-run cannot surface payload-size bugs.
diff --git a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md b/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md
index fb00bdace..06906abc2 100644
--- a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md
+++ b/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md
@@ -407,6 +407,28 @@ listens on, and confirm both workers are running with the latest code
### Step 5b': Tier 8b — Cross-worker cost report assembly (`--mock-inference`, free + deterministic)
+#### Scope manifest — which arms to run
+
+Tier 8b has several arms. By default, run only the free, deterministic mock arms (the **default** scope — arms A–B). If the request carries an **explicit spend opt-in** — the canonical token `full` (aliases `thorough`, `every`, `with-spend`), shown as the **full** scope in the table below — run **every** arm, including the live ones that cost real money, and report PASS/FAIL for each. Do not stop after the cheap arms when a spend opt-in is present. Do **not** treat bare "live" or "all" as the opt-in (too easily incidental; "live" also collides with the default mock arm, which already runs in LIVE mode) — if that's the only signal, confirm before spending.
+
+Run each arm in listed order. After each, surface the asserted token totals (where the arm produces them) and PASS/FAIL, then continue. **Arm D is the exception** — it runs no assertion script and emits no `RESULT: PASS` or token totals; its pass is the *absence* of a cost table and usage events (see its sub-section), so do not treat the missing totals as a failure. For every arm that *does* run the assertion script, if it cannot reach `RESULT: PASS`, stop and report it rather than silently continuing.
+
+| # | Arm | Scope | Spend | Expected assertion |
+|---|---|---|---|---|
+| A | **Mock primary** — `--mock-inference` `native_text_sequence` | default + full | free | 2 events / 200 input / 100 output, `--expected-model-type llm --require-fallback` |
+| B | **Cross-child fan-out** — `--mock-inference` `temporal_parallel` | default + full | free | 3 events / 300 input / 150 output, `--expected-model-type llm --require-fallback` (the CLI assert checks the summed total, **not** workflow-span; the cross-child guarantee itself is enforced by the pytest counterpart `test_split_worker_cross_child_usage.py`) |
+| C | **CSV un-truncated cross-check** — flip `reporting_config.is_generate_cost_report_file_enabled=true`, rerun arm A, confirm CSV totals == NDJSON totals, restore to `false` | full | free | `csv tokens` line matches NDJSON totals |
+| D | **`--no-costs` negative gate** | full | free | no cost table, no `usage_report` events, `reactflow.html` still assembles |
+| E | **Live LLM arm** — drop `--mock-inference` on `native_text_sequence` | full | **real** | non-zero total, real `model_names` (not `mock_inference`), `--require-fallback --require-nonzero` |
+| F | **Live img-gen arm** — run a Tier 4/5 image bundle live with `--graph --costs` | full | **real** | `--expected-model-type img_gen --require-fallback --require-nonzero` |
+| G | **Live extract arm** — run the extract bundle `pdf_extract_page_views.mthds` (`--pipe pdf_extract_with_page_views`) live against the plain split workers with `--graph --costs` | full | **real** | `--expected-model-type extract --require-fallback --require-nonzero` |
+
+Arms F and G validate non-LLM usage, which `--mock-inference` cannot reach (the mock leaves raise `MockInferenceUnsupportedError`), so they are **live-only** — they are the sole way to prove img-gen / extract token usage crosses the runner fallback and aggregates into the cost report. Each arm's full command + assertion is detailed in the sub-sections below; arms A/C map to "Primary check", B to "Cross-child aggregation", D to "Negative check", E to "Live arm", F/G to "Non-LLM cross-worker cost". Arms F and G run their bundles on the **same plain router+runner split workers** as the other arms — they do **not** need the routing battery's multi-queue (`q_extract` / `q_image_gen`) setup. On plain split workers the img-gen / extract activity runs on the runner and emits via the `act_*` fallback, which is exactly what `--require-fallback` checks.
+
+Tier 8 (Step 5b above — the writer-id landing pytest) is the precursor to all of these and should pass first under any scope.
+
+---
+
Tier 8 proves a runner-side `UsageReportEvent` *lands* in the NDJSON partition.
Tier 8b proves the next link end-to-end: those cross-worker usage events
*assemble* onto `PipeOutput.tokens_usages` (via `act_assemble_tracing` with
@@ -448,25 +470,39 @@ Expect, at the end of the submitter output:
- A `reactflow.html` for the run (graph assembles from the *same* event read).
- Exit 0.
-Then confirm the usage came cross-worker, and that there is exactly **one**
-report (no double-count between the fast path and the runner fallback):
+Then **assert the numbers** (don't eyeball — the terminal table truncates wide
+columns to `0 … …`). The `assert_cross_worker_cost.py` helper sums input/output
+tokens straight from the NDJSON usage events, counts them, checks a runner `act_*`
+writer engaged, and (if a CSV report exists) cross-checks the un-truncated CSV
+totals. For mock-inference the per-call counts are fixed
+(`MOCK_INFERENCE_NB_TOKENS_BY_CATEGORY = {INPUT: 100, OUTPUT: 50}`), so a
+2-LLM-step sequence must total exactly 2 events / 200 input / 100 output:
```bash
RUN_ID=$(ls -t .pipelex/traces/ | head -1)
ls -la .pipelex/traces/$RUN_ID/
-grep -l '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson
-grep -hoE '"writer_id":"[^"]+"' .pipelex/traces/$RUN_ID/*.ndjson | sort -u
-grep -hc '"event_kind":"usage_report"' .pipelex/traces/$RUN_ID/*.ndjson | paste -sd+ - | bc
+.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \
+ --run-dir .pipelex/traces/$RUN_ID \
+ --expected-events 2 --expected-input 200 --expected-output 100 \
+ --expected-model-type llm --require-fallback
```
-Expect:
+Expect `RESULT: PASS`, with:
- At least one `wf_*__w_act_{pid}_{uuid}.ndjson` file (runner-side) carrying
- `usage_report` events with `writer_id` starting `act_`, alongside router-side
- `wf_*.ndjson` with `writer_id="primary"`.
-- The total `usage_report` event count equals the number of LLM steps (one per
- mocked `act_llm_gen_text`) — and the rendered table's row count / total match
- it. No duplicate usage rows.
+ `usage_report` events with `writer_id` starting `act_` (the `--require-fallback`
+ gate), alongside router-side `wf_*.ndjson` with `writer_id="primary"`.
+- The `usage events` count equal to the number of LLM steps (one per mocked
+ `act_llm_gen_text`), no double-count between the fast path and the fallback.
+- `total tokens : input=200 output=100`.
+
+**Un-truncated CSV cross-check (optional but recommended).** The Rich console
+table truncates; the CSV does not. Enable
+`reporting_config.is_generate_cost_report_file_enabled = true` in
+`.pipelex/pipelex.toml` before the run, then the run also writes
+`reports/cost_report*.csv`. The script auto-detects it and asserts the CSV token
+totals equal the NDJSON totals (`csv tokens` line in its output). Restore the
+flag to `false` afterwards.
**Cross-child aggregation (A+B+C → one report).** Run a fan-out bundle so usage
from multiple child workflows aggregates into a single submitter report:
@@ -480,7 +516,21 @@ echo "EXIT=$?"
```
Expect a single end-of-run cost table whose token totals sum the parent and both
-child-workflow branches — not one table per branch.
+child-workflow branches — not one table per branch. Assert it numerically: the
+parallel bundle has three LLM steps (`branch_tone`, `branch_length`,
+`summarize_results`), so 3 events / 300 input / 150 output, and usage must span
+**more than one workflow** (cross-child):
+
+```bash
+RUN_ID=$(ls -t .pipelex/traces/ | head -1)
+.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \
+ --run-dir .pipelex/traces/$RUN_ID \
+ --expected-events 3 --expected-input 300 --expected-output 150 \
+ --expected-model-type llm --require-fallback
+```
+
+The pytest counterpart for this cross-child aggregation (no spend) is
+`tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py`.
**Negative check — `--no-costs` gates costs only, graph unaffected:**
@@ -511,17 +561,51 @@ timeout 600 .venv/bin/pipelex run bundle \
--temporal --no-logo --graph --costs 2>&1 | tail -30
```
-Expect the same single end-of-run cost table, now with the real model handle and
-real token counts/cost. Use this only to validate the real-payload path; the mock
-arm above is the regression-friendly default.
+Real token counts are not predictable, so assert non-zero rather than exact —
+the run still must capture real provider tokens, cross the runner boundary, and
+aggregate to a non-zero total:
+
+```bash
+RUN_ID=$(ls -t .pipelex/traces/ | head -1)
+.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \
+ --run-dir .pipelex/traces/$RUN_ID \
+ --expected-model-type llm --require-fallback --require-nonzero
+```
+
+Expect `RESULT: PASS` with non-zero `total tokens` and a real `model_names` entry
+(not `mock_inference`). The pytest counterpart for this real-inference cross-worker
+path (gated, opt-in spend) is
+`tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py`
+(marked `inference`/`llm`; the no-spend counterpart is
+`tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.py`).
+
+**Non-LLM cross-worker cost (img-gen / extract — live only).** `--mock-inference`
+cannot cover image generation or extraction: their mock leaves raise
+`MockInferenceUnsupportedError`, so non-LLM usage only crosses the runner boundary
+on a real run. The img-gen / extract live tiers (Tiers 4 / 5 / 10c) already
+exercise those activities cross-process but never check the cost numbers. After
+running one of those live tiers with `--graph --costs` against split workers,
+assert its run dir surfaced non-zero non-LLM usage with the right model handle:
+
+```bash
+RUN_ID=$(ls -t .pipelex/traces/ | head -1)
+# --expected-model-type img_gen for an image-gen bundle, extract for an extract bundle
+.venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \
+ --run-dir .pipelex/traces/$RUN_ID \
+ --expected-model-type img_gen --require-fallback --require-nonzero
+```
+
+Expect `RESULT: PASS` proving image-gen/extract token usage (not just LLM) is
+captured, emitted via the runner fallback, and aggregated into the submitter's
+cost report. The no-spend unit counterparts (img-gen/extract usage through the
+fallback + aggregator) are
+`tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py` and
+`tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py`.
-After each run, tell the user: PASS/FAIL, whether the cost table rendered, the
-distinct `writer_id` set (must include an `act_*`), the usage-event count vs the
-rendered total, and the graph file path. The pytest counterpart with no spend is
-`tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.py`
-(LIVE `native_text_sequence` with `is_mock_inference=True` through `WfPipeRun`:
-the real `act_llm_gen_text` mocks the leaf and `tokens_usages` assembles back
-reportable under the `mock_inference` model).
+After each run, tell the user: the script's `RESULT: PASS/FAIL`, whether the cost
+table rendered, the distinct `writer_id` set (must include an `act_*`), the
+usage-event count and the summed input/output tokens (NDJSON, and CSV if
+enabled), and the graph file path.
### Step 5c: Tier 9 — Object generation through Temporal cross-process
@@ -991,7 +1075,7 @@ ls results/*/reactflow.html
| Tier 6: Codec transparency | Existing pipelines work unchanged with codec enabled | PASS/FAIL | path | — |
| Tier 7: Large payload | Multi-step pipeline with codec stress test | PASS/FAIL | path | — |
| Tier 8: Cross-worker usage | Runner-side `UsageReportEvent` lands in same NDJSON dir with `act_*` writer_id (live mode or integration test) | PASS/FAIL | — | — |
-| Tier 8b: Cross-worker cost report | `--mock-inference` (free): runner-side usage assembles onto `PipeOutput.tokens_usages` and the submitter renders a single non-suppressed cost report (model `mock_inference`); `--no-costs` renders none while `--graph` still assembles | PASS/FAIL | path | — |
+| Tier 8b: Cross-worker cost report | `--mock-inference` (free): runner-side usage assembles onto `PipeOutput.tokens_usages` and the submitter renders a single non-suppressed cost report (model `mock_inference`); `--no-costs` renders none while `--graph` still assembles. **Numeric assertion** via `scripts/assert_cross_worker_cost.py` (sums NDJSON usage tokens, checks count + `act_*` fallback + optional CSV cross-check) — mock: 2 events/200 input/100 output; parallel: 3/300/150; live arms `--require-nonzero`; img-gen/extract `--expected-model-type` (live only) | PASS/FAIL | path | — |
| Tier 9: Object gen cross-process | `act_llm_gen_object` / `act_llm_gen_object_list` survive the JSON round-trip with nested fields intact | PASS/FAIL | path | — |
| Tier 10a: Multi-activity routing | `activity_queues.default` routes both `act_llm_gen_text` and `act_img_gen_images` to their dedicated worker pools; default runner sees 0 hits for either | PASS/FAIL/SKIPPED | — | — |
| Tier 10b: Per-handle routing | `activity_queues.by_handle` overrides the activity default per model handle — two distinct handles in one workflow land on two distinct workers | PASS/FAIL/SKIPPED | path | — |
diff --git a/.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py b/.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py
new file mode 100755
index 000000000..a862ca7b6
--- /dev/null
+++ b/.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py
@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+"""Numeric cross-worker cost assertion for temporal-e2e-validate Tier 8b.
+
+Turns the old "eyeball that a cost table rendered" check into a real assertion:
+reads the NDJSON usage events a split-worker run produced, sums input/output
+tokens straight from the events, and (optionally) cross-checks them against the
+un-truncated CSV cost report and against expected per-call counts. The terminal
+Rich table truncates wide columns to ``0 … …``, so this is the only reliable way
+to confirm the real numbers cross the runner boundary and aggregate.
+
+Pure stdlib (no pipelex import) so it runs fast and never perturbs the run.
+
+Usage:
+ .venv/bin/python .claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py \
+ --run-dir .pipelex/traces/<RUN_ID> \
+ [--expected-events N] [--expected-input N] [--expected-output N] \
+ [--expected-model-type llm|img_gen|extract|search] \
+ [--reports-dir reports] [--require-fallback]
+
+Exit code 0 = all assertions passed, 1 = at least one failed.
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+_INPUT_KEYS = ("input",)
+_OUTPUT_KEYS = ("output",)
+
+
+def _load_usage_events(run_dir: Path) -> list[dict[str, Any]]:
+ events: list[dict[str, Any]] = []
+ for ndjson_path in sorted(run_dir.glob("*.ndjson")):
+ for line in ndjson_path.read_text(encoding="utf-8").splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ event = json.loads(line)
+ if event.get("event_kind") == "usage_report":
+ events.append(event)
+ return events
+
+
+def _sum_tokens(events: list[dict[str, Any]]) -> tuple[int, int]:
+ total_input = 0
+ total_output = 0
+ for event in events:
+ by_category = event.get("tokens_usage", {}).get("nb_tokens_by_category", {})
+ total_input += sum(int(by_category.get(key, 0)) for key in _INPUT_KEYS)
+ total_output += sum(int(by_category.get(key, 0)) for key in _OUTPUT_KEYS)
+ return total_input, total_output
+
+
+def _csv_token_totals(reports_dir: Path) -> tuple[int, int] | None:
+ """Sum input-joined + output token columns across the most recent cost_report CSV, if any."""
+ csv_paths = sorted(reports_dir.glob("cost_report*.csv"))
+ if not csv_paths:
+ return None
+ latest = csv_paths[-1]
+ total_input = 0
+ total_output = 0
+ with latest.open(encoding="utf-8") as csv_file:
+ for row in csv.DictReader(csv_file):
+ total_input += int(float(row.get("nb_tokens_input_joined") or 0))
+ total_output += int(float(row.get("nb_tokens_output") or 0))
+ return total_input, total_output
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--run-dir", required=True, type=Path, help="The .pipelex/traces/<RUN_ID> directory")
+ parser.add_argument("--expected-events", type=int, default=None, help="Expected usage-event count (== number of inference calls)")
+ parser.add_argument("--expected-input", type=int, default=None, help="Expected total input tokens")
+ parser.add_argument("--expected-output", type=int, default=None, help="Expected total output tokens")
+ parser.add_argument("--expected-model-type", default=None, help="Assert every usage record has this model_type")
+ parser.add_argument("--reports-dir", type=Path, default=Path("reports"), help="Dir holding the un-truncated cost_report*.csv")
+ parser.add_argument("--require-fallback", action="store_true", help="Require at least one act_* writer_id (runner fallback engaged)")
+ parser.add_argument(
+ "--require-nonzero", action="store_true", help="Require total input AND output tokens > 0 (real-spend arms, where exact counts are unknown)"
+ )
+ args = parser.parse_args()
+
+ run_dir: Path = args.run_dir
+ if not run_dir.is_dir():
+ print(f"FAIL: run dir not found: {run_dir}")
+ return 1
+
+ events = _load_usage_events(run_dir)
+ failures: list[str] = []
+
+ if not events:
+ print(f"FAIL: no usage_report events under {run_dir}")
+ return 1
+
+ writer_ids = sorted({event.get("writer_id", "") for event in events})
+ model_types = sorted({event.get("tokens_usage", {}).get("model_type", "?") for event in events})
+ model_names = sorted({event.get("tokens_usage", {}).get("inference_model_name", "?") for event in events})
+ total_input, total_output = _sum_tokens(events)
+
+ print(f"usage events : {len(events)}")
+ print(f"writer_ids : {writer_ids}")
+ print(f"model_types : {model_types}")
+ print(f"model_names : {model_names}")
+ print(f"total tokens : input={total_input} output={total_output} (joined+output={total_input + total_output})")
+
+ csv_totals = _csv_token_totals(args.reports_dir)
+ if csv_totals is not None:
+ csv_input, csv_output = csv_totals
+ print(f"csv tokens : input={csv_input} output={csv_output}")
+ if (csv_input, csv_output) != (total_input, total_output):
+ failures.append(f"CSV totals {csv_totals} != NDJSON totals {(total_input, total_output)}")
+ else:
+ print(f"csv tokens : (no cost_report*.csv under {args.reports_dir}; enable is_generate_cost_report_file_enabled to cross-check)")
+
+ if args.require_fallback and not any(writer_id.startswith("act_") for writer_id in writer_ids):
+ failures.append(f"no act_* writer_id (runner fallback did not engage); writer_ids={writer_ids}")
+ if args.require_nonzero and not (total_input > 0 and total_output > 0):
+ failures.append(f"expected non-zero input AND output tokens, got input={total_input} output={total_output}")
+ if args.expected_events is not None and len(events) != args.expected_events:
+ failures.append(f"event count {len(events)} != expected {args.expected_events}")
+ if args.expected_input is not None and total_input != args.expected_input:
+ failures.append(f"input tokens {total_input} != expected {args.expected_input}")
+ if args.expected_output is not None and total_output != args.expected_output:
+ failures.append(f"output tokens {total_output} != expected {args.expected_output}")
+ if args.expected_model_type is not None and model_types != [args.expected_model_type]:
+ failures.append(f"model_types {model_types} != [{args.expected_model_type!r}]")
+
+ if failures:
+ print("RESULT: FAIL")
+ for failure in failures:
+ print(f" - {failure}")
+ return 1
+ print("RESULT: PASS")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/.github/workflows/package-check.yml b/.github/workflows/package-check.yml
index b01217fda..5071f79a7 100644
--- a/.github/workflows/package-check.yml
+++ b/.github/workflows/package-check.yml
@@ -15,6 +15,16 @@ jobs:
with:
enable-cache: true
+ - name: Check requires-python floor stays at >=3.10
+ run: |
+ if ! grep -qE "^[[:space:]]*requires-python[[:space:]]*=[[:space:]]*['\"][^'\"]*>=3\.10([^0-9A-Za-z][^'\"]*)?['\"]" pyproject.toml; then
+ echo "❌ requires-python must start at >=3.10 — pipelex core supports Python 3.10+."
+ echo "The Mistral Workflows orchestration integration that needs Python 3.12 ships as the"
+ echo "separate pipelex-mistralai-workflows package; do not raise pipelex's own floor for it."
+ exit 1
+ fi
+ echo "✅ requires-python floor is >=3.10"
+
- name: Check if uv.lock is up to date
run: |
uv lock --locked
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0aa2182e3..59250d438 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,7 +2,7 @@
## [Unreleased]
-This cycle reworks **cost reporting** so it survives distributed execution and stops leaking. Cost now rides on the run result instead of a side buffer: a multi-worker Temporal run aggregates usage from every worker into a single end-of-run report, the success-path registry leak is gone by removal, and a new `--costs` switch decouples cost collection from `--graph`.
+This cycle reworks **cost reporting** so it survives distributed execution and stops leaking. Cost now rides on the run result instead of a side buffer: a multi-worker Temporal run aggregates usage from every worker into a single end-of-run report, the success-path registry leak is gone by removal, and a new `--costs` switch decouples cost collection from `--graph`. It also extracts the **framework-agnostic runtime bridge** so any host runtime — not just Mistral Workflows — can embed Pipelex through one boundary.
### Added
@@ -18,6 +18,8 @@ This cycle reworks **cost reporting** so it survives distributed execution and s
### Changed
+- **Mistral Workflows integration extracted into a dedicated package.** The optional `pipelex[mistralai-workflows]` extra and the `pipelex.plugins.mistralai_workflows.*` modules have been removed from `pipelex`. The Mistral Workflows integration now ships separately as its own package — coming soon; install and import details will follow once it publishes. The framework-agnostic runtime-bridge core (boundary types, `run_pipe_via_bridge`, `PipelexExecutionMode`, `ensure_pipelex_booted`) has been promoted from `pipelex.plugins.mistralai_workflows.*` to `pipelex.runtime_bridge.*` so any host runtime — not just Mistral Workflows — can embed Pipelex. No behavior changes; activities, boundary types, and execution modes are identical.
+
- **`--cost-report` removed, folded into `--costs`.** Breaking: `--cost-report/--no-cost-report` is gone from `run pipe|method|bundle`. Use `--costs` (default on) instead.
- **Cost reporting is event-sourced; the submitter-side `UsageRegistry` is removed.** The cost report is rendered from `PipeOutput.tokens_usages`, not from an in-process registry. `ReportingProtocol` (and `ReportingNoOp`) no longer expose `open_registry` / `close_registry` / `generate_report` / `inject_tokens_usages`, and the `UsageRegistry` model is gone. Embedders that rendered cost via `get_report_delegate().generate_report()` must render from the returned `pipe_output.tokens_usages` instead.
@@ -185,7 +187,7 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli
- **Pipeline run chain semantics shift (breaking, behavioral).** Because the Workflow ID is now derived from `pipeline_run_id`, callers that pass a stable `pipeline_run_id` to `PipelineFactory.make_pipeline(pipeline_run_id=...)` and re-execute now land on the same Temporal Workflow Execution Chain (with a fresh `run_id` per execution under the SDK-default `ALLOW_DUPLICATE` reuse policy). The previous behavior produced a fresh workflow_id per execution by accident, via the truncated session id and random shortuuid components — not by design. This is now documented behavior, not a bug.
-- **Child Workflow ID separator change (breaking).** Child workflow ids use `/` instead of `-` as the separator. Examples: the fixed-role `wf_pipe_router` child of `wf_pipe_run` is `{parent}/pipe-router`; a dynamic sub-pipe spawned by a router is `{parent}/{pipe_code}-{8-hex-chars}` (the 8 hex chars come from `workflow.uuid4()` for replay-safety). Operational tooling parsing the nested-id format must update.
+- **Child Workflow ID separator (breaking).** Child workflow ids use `_` as the separator (never `/`), so an id can be reused verbatim as an S3 key or file name without creating spurious path segments. Examples: the fixed-role `wf_pipe_router` child of `wf_pipe_run` is `{parent}_pipe-router`; a dynamic sub-pipe spawned by a router is `{parent}_{pipe_code}-{8-hex-chars}` (the 8 hex chars come from `workflow.uuid4()` for replay-safety). Operational tooling parsing the nested-id format must update.
- **Activity ID change (breaking).** Pipelex no longer customizes `activity_id`. The Temporal Python SDK auto-assigns deterministic sequential integers (`"1"`, `"2"`, …) per workflow run, which guarantees per-`(workflow_id, run_id)` uniqueness by construction and is replay-safe (assigned by history position). Per-call meaning that previously lived in `activity_id` now lives in the per-activity `summary=`. Anything that filtered or grouped Event History by the old semantic strings (`"craft-text"`, `"craft-object-direct"`, `"jinja2-text"`, `"extract-pages"`, etc.) must read the per-activity `summary` or the Activity Type instead.
diff --git a/README.md b/README.md
index f6f487a41..8e3075998 100644
--- a/README.md
+++ b/README.md
@@ -533,6 +533,10 @@ Install all extras:
uv pip install "pipelex[anthropic,google,google-genai,mistralai,bedrock,fal,linkup,docling]"
```
+> **Mistral Workflows orchestration** *(coming soon)* is not a pipelex extra. It runs pipes through the runtime bridge inside Mistral Workflows activities and requires **Python 3.12+**, so it will ship as a separate, dedicated package that bundles pipelex. pipelex itself supports **Python 3.10+**; only this optional orchestration path needs 3.12.
+>
+> This package is still in active development and not published yet — install and import instructions will land here once it ships.
+
---
**Privacy & Telemetry** — Pipelex Gateway collects only technical data (model names, token counts, latency) — never prompts or business data. If you want to avoid Gateway telemetry, disable `pipelex_gateway` and use your own provider keys or local AI instead. [Learn more](https://docs.pipelex.com/latest/setup/telemetry/)
diff --git a/docs/distributed-execution/index.md b/docs/distributed-execution/index.md
index 946481c84..52c6b1a1c 100644
--- a/docs/distributed-execution/index.md
+++ b/docs/distributed-execution/index.md
@@ -1,98 +1,28 @@
---
-title: "Distributed Execution with Temporal"
-description: "Run Pipelex pipelines as durable Temporal workflows across worker pools. Cluster setup, worker deployment, task-queue routing, and dashboard observability."
+title: "Distributed Execution"
+description: "Run Pipelex methods as durable workflows on Temporal with your own Pipelex Temporal workers. A second, managed backend (Mistral Workflows) is coming soon."
---
-# Distributed Execution with Temporal
+# Distributed Execution
-Run your Pipelex pipelines as durable Temporal workflows across one or more worker processes. This section walks through setting up the cluster, running workers, routing work to the right pools, and reading the Temporal Web UI.
+Pipelex methods run in-process by default — you call `pipelex run pipe ...` or invoke a pipe from Python and everything happens in one process. When you need durability, retries that survive failure, and horizontal scale, you run the same methods as **durable workflows** on Temporal.
-## What it is
+## Pipelex on Temporal
-Pipelex pipelines normally run in-process — you call `pipelex run pipe ...` or invoke a pipe from Python and everything happens in one Python process. With the optional `pipelex[temporal]` integration, the same pipelines run as **Temporal workflows**: each pipe execution becomes a workflow on a Temporal cluster, child pipes become child workflows, and every LLM call, image generation, or document extraction becomes a Temporal activity executed on a worker. Temporal handles durability, retries, scheduling, and visibility; Pipelex handles the AI work.
+**[Pipelex on Temporal](temporal/index.md)** runs Pipelex's own Temporal workers against a Temporal cluster you operate (self-hosted or Temporal Cloud) — you own the control plane. Python 3.10+, installed with `pipelex[temporal]`. Generally available.
-The whole layer is opt-in. Flip `[temporal] is_enabled = true` in `.pipelex/pipelex.toml`, install `pipelex[temporal]`, and the same `.mthds` methods run distributed without changing a line.
-
-## When you'd want it
-
-You don't need Temporal for local development — pipes run in-process by default. Reach for distributed execution when you need:
-
-- **Durability** — long-running pipelines survive process restarts and worker crashes. Temporal replays history to resume exactly where execution left off.
-- **Retries with budget control** — every activity (LLM call, extract, image gen) retries independently with per-activity timeouts and retry policies.
-- **Horizontal scale** — fan out work across multiple worker machines. Pipelex lets you route OpenAI calls to one pool, Anthropic to another, OCR to a third.
-- **Operational visibility** — every workflow shows up in the Temporal Web UI with the pipe code, pipeline run id, user, session, and a per-activity summary line.
-
-## The big picture
-
-```
-Your application
- │
- │ pipe_run(...) ──→ starts a Temporal workflow
- ▼
-┌────────────────────┐
-│ Temporal cluster │ durability, history, retries, search
-└─────────┬──────────┘
- │ tasks distributed by queue
- ▼
- ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
- │ worker │ │ worker │ │ worker │ poll task queues,
- │ (router) │ │ (LLM pool) │ │ (img-gen) │ execute workflows
- └─────────────┘ └─────────────┘ └─────────────┘ and activities
-```
-
-A submitter (your app or `pipelex run`) starts a workflow; one or more worker processes poll task queues for work; results flow back through the workflow's history. You can run a single worker that handles everything, or split workers by activity class for independent scaling and isolation.
-
-## What you'll set up
-
-Three things, in this order:
-
-- **[Cluster Setup](cluster-setup.md)** — one-time namespace prerequisites: register the custom search attributes Pipelex needs for the dashboard view. Run `pipelex setup-temporal-namespace` and you're done.
-- **[Worker Deployment](workers.md)** — how to run `pipelex worker`, what `--scope` and `--profile` do, and how to compose multiple workers into a deployment.
-- **[Task-Queue Routing](task-routing.md)** — when you outgrow a single queue: per-activity routing, per-queue timeouts and retry policies, per-handle overrides. Skip on day one; revisit when you need to isolate workloads.
-
-## What you'll see in the Temporal dashboard
-
-Every workflow Pipelex starts carries human-readable identity: a workflow id derived from your pipeline run id, a Markdown summary of the pipe and its inputs, and per-activity summary lines that explain what each LLM or extract call is doing. Filter by `PipeCode`, `PipelineRunId`, `UserId`, `SessionId`, or `DomainCode` from the dashboard's search attribute selector. See **[Workflow Observability](observability.md)** for the full reference.
-
-## Quick start
-
-Minimal configuration in `.pipelex/pipelex.toml`:
-
-```toml
-[temporal]
-is_enabled = true
-
-[temporal.worker_config]
-default_task_queue = "temporal_task_queue"
-
-[temporal.temporal_config]
-selected_server = "local"
-
-[temporal.temporal_config.temporal_server_configs.local]
-target_host = "localhost:7233"
-namespace = "default"
-api_key_method = "none"
-```
-
-Register the custom search attributes once (assumes Temporal is reachable):
-
-```bash
-pipelex setup-temporal-namespace
-```
-
-Boot a worker against the default task queue:
-
-```bash
-pipelex worker
-```
-
-Submit work from your application — the same `pipe_run(...)` you'd use locally now executes as a Temporal workflow because `[temporal] is_enabled = true`. The Pipelex Python API automatically dispatches through the Temporal hub.
+The same `.mthds` methods run distributed without rewriting. Pipelex's runtime bridge classifies controller pipes as child workflows and leaf operators (LLM calls, image generation, document extraction) as activities — so durability, retries, and observability attach at the right granularity.
## Where to go next
-- **[Cluster Setup](cluster-setup.md)** — search attributes, `pipelex setup-temporal-namespace`, Temporal Cloud permission model.
+- **[Overview](temporal/index.md)** — what it is, when you'd want it, the big picture, quick start.
+- **[Cluster Setup](cluster-setup.md)** — search attributes and `pipelex setup-temporal-namespace`.
- **[Worker Deployment](workers.md)** — `pipelex worker`, scopes, runtime profiles, multi-worker topologies.
-- **[Task-Queue Routing](task-routing.md)** — `activity_queues`, queue options, per-handle overrides, dispatch tracing.
-- **[Workflow Observability](observability.md)** — workflow ids, activity ids, summary fields, search-attribute filtering.
+- **[Task-Queue Routing](task-routing.md)** — per-activity routing, queue options, per-handle overrides.
+- **[Workflow Observability](observability.md)** — workflow ids, summary fields, search-attribute filtering.
+
+For the failure-handling model that motivates durable execution, see [Retries & Resilience](../reliability/retries-and-resilience.md).
+
+## Coming soon: a managed backend
-For the runtime mechanics that make distributed execution work under the hood — LibraryCrate propagation, deferred hydration, per-workflow class registry isolation — see [Temporal Integration](../under-the-hood/temporal-integration.md) and [Distributed Content Generation](../under-the-hood/distributed-content-generation.md).
+A second distributed-execution backend — running Pipelex pipes inside **Mistral Workflows**, Mistral's managed orchestration control plane (itself built on Temporal) — is in active development. It runs the same `.mthds` methods through the same runtime bridge, with no cluster for you to operate. Documentation and install instructions will land here once it ships.
diff --git a/docs/distributed-execution/mistral-workflows/choosing-a-backend.md b/docs/distributed-execution/mistral-workflows/choosing-a-backend.md
new file mode 100644
index 000000000..cb0ad59f7
--- /dev/null
+++ b/docs/distributed-execution/mistral-workflows/choosing-a-backend.md
@@ -0,0 +1,41 @@
+---
+title: "Choosing a Backend"
+description: "Pipelex on Temporal vs Pipelex on Mistral Workflows — both are durable execution on Temporal; pick based on who runs the control plane."
+---
+
+# Choosing a Backend
+
+Both distributed-execution backends are durable execution on Temporal underneath. They give you the same core guarantees: crash survival, activity-level retries, and deterministic replay (on recovery the workflow re-runs from its event history, reusing already-stored activity results rather than re-executing completed activities). The real choice is **which control plane you want to operate**: your own Temporal deployment, or Mistral's managed Workflows service.
+
+## Side by side
+
+| | Pipelex on Temporal | Pipelex on Mistral Workflows |
+|---|---|---|
+| **Control plane** | You run it (self-hosted Temporal or Temporal Cloud) | Mistral runs it (managed) |
+| **Install** | `pipelex[temporal]` | `pipelex-mistralai-workflows` |
+| **Python** | 3.10+ | 3.12+ |
+| **How you invoke** | Enable `[temporal] is_enabled` in `.pipelex/pipelex.toml` and call `pipe_run(...)` — it dispatches through the Temporal hub | Invoke a Pipelex pipe through the runtime bridge inside your Mistral Workflows activity |
+| **Best when** | You already operate Temporal, or want full control over the cluster | You're already building on Mistral's platform |
+| **Status** | Generally available | Preview |
+
+## When to pick Temporal
+
+- You already run a Temporal cluster or use Temporal Cloud.
+- You want to own worker topology, task-queue routing, and retry policy in detail.
+- You need the production operational surface available today: cluster setup, multi-worker scaling, per-activity routing, dashboard observability.
+
+## When to pick Mistral Workflows
+
+- You're already building workflows on Mistral's platform and want pipes to run alongside the rest of your orchestration.
+- You'd rather not operate a Temporal cluster yourself.
+- You're comfortable with a preview integration while the package stabilizes.
+
+## You can run the same methods on either
+
+Your `.mthds` methods don't change between backends. The choice is purely about where they run and who operates the control plane — so you can start in-process, move to one backend, and switch later without rewriting methods.
+
+## Related
+
+- **[Pipelex on Temporal](../temporal/index.md)** — the self-operated backend.
+- **[Pipelex on Mistral Workflows](index.md)** — the managed backend (preview).
+- **[Retries & Resilience](../../reliability/retries-and-resilience.md)** — the failure-handling model both backends build on.
diff --git a/docs/distributed-execution/mistral-workflows/execution-modes.md b/docs/distributed-execution/mistral-workflows/execution-modes.md
new file mode 100644
index 000000000..6b4479898
--- /dev/null
+++ b/docs/distributed-execution/mistral-workflows/execution-modes.md
@@ -0,0 +1,48 @@
+---
+title: "Execution Modes"
+description: "How a Pipelex pipe runs inside a Mistral Workflows activity — DIRECT, TEMPORAL_BLOCKING, TEMPORAL_FIRE_AND_FORGET, and MISTRAL_NATIVE — and what each requires."
+---
+
+# Execution Modes
+
+When you run a pipe through the bridge, the `execution_mode` field on the input selects *how* that pipe runs. The same activity code can run a pipe in-process, hand it off to your own Temporal workers, or decompose it into native Mistral Workflows primitives.
+
+## The runtime bridge
+
+Pipes are invoked through `run_pipe_via_bridge`, which takes a `PipelexPipeRunInput` and returns a `PipelexPipeRunOutput`. Both are plain JSON-safe models — the boundary never carries Pipelex internal types. The bridge boots Pipelex on first call, validates the input, optionally opens a per-call scoped library from a crate, then dispatches on `execution_mode`.
+
+## The modes
+
+| Mode | What it does | Requires |
+|------|--------------|----------|
+| `direct` | Runs the pipe in-process, inside the calling activity, blocking until it completes. Fastest feedback, simplest ops. The default. | Pipelex core |
+| `temporal_blocking` | Dispatches the pipe as a Pipelex Temporal workflow and awaits completion. The pipe runs durably on your own Temporal worker fleet. | `pipelex[temporal]` |
+| `temporal_fire_and_forget` | Dispatches the pipe as a Pipelex Temporal workflow and returns immediately with the `workflow_id`. Completion is signalled out-of-band via a delivery assignment (webhook / storage). | `pipelex[temporal]`, and `delivery_assignment_dump` |
+| `mistral_native` | Decomposes the pipe into native Mistral Workflows primitives — controllers as child workflows, leaf operators as activities — surfacing per-step retry, signals, and cancellation through the host runtime. | `pipelex-mistralai-workflows` |
+
+The `Mode` values above are the literal strings you set on the `execution_mode` field — lowercase, matching the `PipelexExecutionMode` enum values. (Mode names written in uppercase elsewhere in this guide refer to the same modes by name.)
+
+For local and most in-activity use, `direct` is the right default: the pipe simply runs where the activity runs, and Mistral Workflows provides the durability and retry around the activity as a whole.
+
+## Input fields that matter
+
+`PipelexPipeRunInput` carries everything the bridge needs:
+
+- `pipe_code` — the pipe to run.
+- `inputs` — the pipe's inputs, as a JSON dict.
+- `output_name` — optional override for the output variable name.
+- `pipeline_run_id`, `user_id` — identity carried into tracing and observability.
+- `execution_mode` — one of the modes above (defaults to `direct`).
+- `library_crate_dump` — an optional serialized library snapshot. Use it when the pipe isn't already loaded at boot: the bridge opens a per-call scoped library from the crate, runs the pipe, and tears it down afterward. This is the dump-based library transport that lets a worker run methods it didn't load at startup.
+- `delivery_assignment_dump` — required for `TEMPORAL_FIRE_AND_FORGET`; describes where to deliver the result (webhook / storage).
+
+## Missing-dependency behavior
+
+The bridge fails loudly and helpfully when a mode's dependency isn't installed:
+
+- Requesting a `TEMPORAL_*` mode without the extra raises `MissingPipelexTemporalExtraError` — *"Install with: pip install 'pipelex[temporal]'"*.
+- Requesting `MISTRAL_NATIVE` without the plugin raises `MissingMistralWorkflowsPluginError` — *"Install with: pip install pipelex-mistralai-workflows"*.
+- Requesting `TEMPORAL_FIRE_AND_FORGET` without a `delivery_assignment_dump` is rejected up front, so a completion is never silently dropped.
+
+!!! warning "MISTRAL_NATIVE is preview"
+ The dispatch hook for `MISTRAL_NATIVE` lives in Pipelex core, but the implementation that decomposes a pipe into native Workflows primitives ships in the preview `pipelex-mistralai-workflows` package. Treat end-to-end `MISTRAL_NATIVE` as preview and verify behavior against the installed package. For most integrations, running pipes in `DIRECT` mode inside your own activity already gives you durable, retryable execution from Mistral Workflows.
diff --git a/docs/distributed-execution/mistral-workflows/index.md b/docs/distributed-execution/mistral-workflows/index.md
new file mode 100644
index 000000000..259f98c79
--- /dev/null
+++ b/docs/distributed-execution/mistral-workflows/index.md
@@ -0,0 +1,38 @@
+---
+title: "Pipelex on Mistral Workflows"
+description: "Preview: run Pipelex pipes inside Mistral Workflows, Mistral's managed orchestration control plane built on Temporal, via the pipelex-mistralai-workflows package."
+---
+
+# Pipelex on Mistral Workflows
+
+!!! warning "Preview"
+ The `pipelex-mistralai-workflows` package is in preview and under active development. Install it with `pip install pipelex-mistralai-workflows` (Python 3.12+). APIs may change, and some integration symbols shown in this guide are illustrative — they are flagged where they appear. Pipelex itself supports Python 3.10+; only this orchestration path needs 3.12.
+
+> Mistral Workflows is an orchestration control plane designed to accelerate the development and reliable execution of complex, AI-driven workflows. Built on Temporal for fault-tolerant workflow execution, Workflows combines a user-friendly API with a rich Python framework optimized for Mistral's AI services.
+
+This guide shows how to run your Pipelex `.mthds` methods *inside* Mistral Workflows — so a single Mistral Workflows worker can orchestrate Pipelex pipes alongside the rest of your AI workflow, with durability, retries, and observability handled by the platform.
+
+## How it fits together
+
+You don't rewrite your methods. The same pipes that run in-process locally run unchanged here:
+
+- You write a Mistral Workflows worker (workflows and activities) the usual way.
+- Inside an activity, you invoke a Pipelex pipe through Pipelex's runtime bridge.
+- Mistral Workflows — on its Temporal foundation — gives durability, automatic retries, streaming progress, and observability; Pipelex runs the AI method.
+
+!!! info "Both backends are Temporal underneath"
+ Pipelex on Mistral Workflows and [Pipelex on Temporal](../temporal/index.md) are the same durability primitive packaged two ways. The difference is *who runs the control plane*: here it's Mistral's managed Workflows service; there it's a Temporal deployment you operate yourself. See [Choosing a Backend](choosing-a-backend.md).
+
+## What you get
+
+- **Durable execution** — pipe runs survive worker restarts and transient failures, resuming from history.
+- **Automatic retries** — what retries depends on the execution mode: in `direct` mode the whole pipe runs inside one host activity that retries as a unit; in `mistral_native` mode each leaf operator (LLM call, extraction, image generation) runs as its own host activity and retries independently. See [Execution Modes](execution-modes.md).
+- **Streaming progress** — surface live pipe progress through the Workflows Task event API. See [Streaming Progress](streaming.md).
+- **The same methods, unchanged** — your `.mthds` bundles don't change between local, Temporal, and Mistral Workflows execution.
+
+## Next steps
+
+- **[Installation & Preview Status](installation.md)** — install the preview package and its prerequisites.
+- **[Your First Pipelex Workflow](your-first-pipelex-workflow.md)** — register a Pipelex activity and run a pipe end to end.
+- **[Execution Modes](execution-modes.md)** — how a pipe runs inside an activity, and what each mode requires.
+- **[Choosing a Backend](choosing-a-backend.md)** — Mistral Workflows vs your own Temporal cluster.
diff --git a/docs/distributed-execution/mistral-workflows/installation.md b/docs/distributed-execution/mistral-workflows/installation.md
new file mode 100644
index 000000000..daa61aa6b
--- /dev/null
+++ b/docs/distributed-execution/mistral-workflows/installation.md
@@ -0,0 +1,50 @@
+---
+title: "Installation & Preview Status"
+description: "Install the preview pipelex-mistralai-workflows package (Python 3.12+) and understand what 'preview' means for the Mistral Workflows backend."
+---
+
+# Installation & Preview Status
+
+!!! warning "Preview"
+ `pipelex-mistralai-workflows` is not yet a stable release. Its public surface may change between versions, and there is no separate published documentation site for it yet — this guide is the reference for now.
+
+## Prerequisites
+
+Before installing, make sure you have:
+
+- **Python 3.12 or later.** Mistral Workflows requires 3.12+. Pipelex core supports 3.10+, but this orchestration path needs 3.12 because of the Workflows SDK.
+- **A Mistral Workflows setup** — a workspace and API key from the [Mistral Console](https://console.mistral.ai/), and the `mistralai-workflows` SDK. Follow Mistral's own Workflows installation and first-workflow guides for the worker, console, and `.env` setup; this guide does not duplicate them.
+- **A Pipelex method to run** — a `.mthds` bundle available on your `PIPELEXPATH`, or passed at call time as a library crate (see [Execution Modes](execution-modes.md)).
+
+## Install
+
+```bash
+pip install pipelex-mistralai-workflows
+```
+
+This pulls in `pipelex` automatically. If your project uses `uv`, the equivalent is `uv add pipelex-mistralai-workflows`.
+
+The package depends on the Mistral Workflows SDK (`mistralai-workflows`). If you are scaffolding a new Workflows project, install the SDK as Mistral documents it (`uv add mistralai-workflows`) and add `pipelex-mistralai-workflows` alongside it.
+
+## Verify
+
+Confirm both pieces import in your environment:
+
+```bash
+python -c "import mistralai.workflows; import pipelex_mistralai_workflows; print('ready')"
+```
+
+<!-- ILLUSTRATIVE: the importable top-level module name `pipelex_mistralai_workflows` matches the package's Python distribution name; confirm the exact public import path against the released package. -->
+
+!!! note "Preview verify"
+ The exact public import surface of `pipelex_mistralai_workflows` is still settling. If the import above fails after a successful install, check the package's release notes for the current module layout.
+
+## What "preview" means
+
+- The API may change between releases without a deprecation window.
+- Some examples in this guide are marked illustrative — confirm them against the installed package before relying on them in production.
+- Watch the changelog and release notes for breaking changes.
+
+## Next
+
+Continue to **[Your First Pipelex Workflow](your-first-pipelex-workflow.md)**.
diff --git a/docs/distributed-execution/mistral-workflows/streaming.md b/docs/distributed-execution/mistral-workflows/streaming.md
new file mode 100644
index 000000000..3cfee5cfa
--- /dev/null
+++ b/docs/distributed-execution/mistral-workflows/streaming.md
@@ -0,0 +1,41 @@
+---
+title: "Streaming Progress"
+description: "Surface live Pipelex pipe progress through Mistral Workflows' Task event API — one event per activity, or per-step updates in DIRECT mode."
+---
+
+# Streaming Progress
+
+!!! warning "Preview"
+ Streaming integration ships in the preview `pipelex-mistralai-workflows` package. The Mistral Workflows streaming primitives shown here are stable; the Pipelex-specific entry points are flagged inline.
+
+Mistral Workflows publishes real-time events from activities through its **Task** API, which powers live UIs and progress indicators. A Pipelex activity wraps a pipe run in a Task and emits progress as the pipe executes.
+
+## How streaming works
+
+Inside an activity you open a `Task` and push state updates as work proceeds:
+
+```python
+from mistralai.workflows.core.task import Task
+
+async with Task(type="pipelex_progress", state={"completed_steps": []}) as task:
+ # ... run the pipe, updating task.state as steps complete ...
+ await task.update_state({"completed_steps": task.state["completed_steps"] + ["answer_question"]})
+```
+
+Mistral Workflows wraps each update with workflow/activity context and publishes it through its event streaming system, which consumers subscribe to via the Workflows client's event-streaming APIs.
+
+## Two granularities
+
+The Pipelex integration surfaces progress at one of two levels:
+
+- **Per-activity** — a single Task per pipe run: started, in-progress, completed (or failed). This is the cheapest option and works regardless of execution mode.
+- **Per-step (`DIRECT` mode)** — driven by Pipelex's own execution-trace events, the activity emits a Task update for each pipe step as it starts and finishes, exposing started-steps, completed-steps, and the current pipe code. Because it rides on in-process trace events, per-step granularity applies to `DIRECT` execution; Temporal modes keep the single started/finished pair.
+
+<!-- ILLUSTRATIVE: the package's streaming entry-point names and the exact per-step task `type`/state schema are still settling in preview; the per-activity vs per-step distinction is the design-locked behavior. Confirm symbols against the installed package. -->
+
+## Consuming events
+
+Subscribe to the published events with the Mistral Workflows client's event-streaming APIs — see Mistral's "Consuming Streaming Events" guide. This guide doesn't duplicate the consumer side; the events a Pipelex activity emits are ordinary Workflows Task events.
+
+!!! note "Payload limit"
+ NATS messages are capped at **1 MB**. Stream small progress state — step names, counts, a current pipe code — not large pipe outputs. For large results, store them externally (or use the bridge's delivery assignment) and stream a reference.
diff --git a/docs/distributed-execution/mistral-workflows/your-first-pipelex-workflow.md b/docs/distributed-execution/mistral-workflows/your-first-pipelex-workflow.md
new file mode 100644
index 000000000..96226116e
--- /dev/null
+++ b/docs/distributed-execution/mistral-workflows/your-first-pipelex-workflow.md
@@ -0,0 +1,95 @@
+---
+title: "Your First Pipelex Workflow on Mistral Workflows"
+description: "Register a Pipelex activity on a Mistral Workflows worker, call it from a workflow, and run it — the preview end-to-end path."
+---
+
+# Your First Pipelex Workflow
+
+!!! warning "Preview"
+ This walkthrough targets the preview `pipelex-mistralai-workflows` package. The Pipelex bridge calls shown here (`run_pipe_via_bridge`, `PipelexPipeRunInput`) are part of Pipelex core and stable; anything specific to the plugin package is flagged inline.
+
+We'll register an activity that runs a Pipelex pipe, call it from a Mistral Workflow, and execute the whole thing on a worker.
+
+## Prerequisites
+
+- The preview package installed — see [Installation & Preview Status](installation.md).
+- A working Mistral Workflows setup (workspace, API key, `.env`) per Mistral's own first-workflow guide.
+- A Pipelex method available on your `PIPELEXPATH` (or passed as a library crate — see [Execution Modes](execution-modes.md)). The example below assumes a pipe named `answer_question`.
+
+## Step 1 — Define a worker that runs a Pipelex pipe
+
+Pipelex exposes a small, JSON-in / JSON-out entry point — `run_pipe_via_bridge(PipelexPipeRunInput(...))` — designed to be called from inside any host-runtime activity. The cleanest first integration is to write your own activity that calls it. Every symbol here is part of Pipelex core.
+
+Create `my_pipelex_worker.py`:
+
+```python
+import mistralai.workflows as workflows
+
+from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge
+
+
+async def run_pipelex_pipe(pipe_code: str, inputs: dict) -> dict:
+ """Run a Pipelex pipe inside a Mistral Workflows activity.
+
+ The bridge boots Pipelex on first call (idempotent), runs the pipe, and
+ returns a JSON-safe result. Only JSON crosses the activity boundary — no
+ Pipelex internal objects.
+ """
+ output = await run_pipe_via_bridge(
+ PipelexPipeRunInput(pipe_code=pipe_code, inputs=inputs)
+ )
+ return output.model_dump()
+
+
[email protected](name="pipelex_example_workflow")
+class PipelexExampleWorkflow:
+ @workflows.workflow.entrypoint
+ async def run(self, pipe_code: str, inputs: dict) -> dict:
+ return await run_pipelex_pipe(pipe_code, inputs)
+
+
+async def main() -> None:
+ await workflows.run_worker([PipelexExampleWorkflow])
+
+
+if __name__ == "__main__":
+ import asyncio
+
+ asyncio.run(main())
+```
+
+!!! note "Illustrative: ready-made activity"
+ The plugin package may also ship a ready-to-register Pipelex activity so you can add it to a worker without writing the wrapper yourself. Its exact name and registration signature are still settling in preview — confirm them against the installed package. Writing your own `@workflows.activity()` around `run_pipe_via_bridge`, as above, works today and is the safest starting point.
+
+## Step 2 — Run the worker
+
+```bash
+uv run python my_pipelex_worker.py
+```
+
+The worker connects to the Mistral API and registers `pipelex_example_workflow`. You should see it in the [Mistral Console](https://console.mistral.ai/build/workflows).
+
+## Step 3 — Trigger execution
+
+From the Console, select `pipelex_example_workflow`, click **Start Workflow**, and pass input such as:
+
+```json
+{ "pipe_code": "answer_question", "inputs": { "question": "What is MTHDS?" } }
+```
+
+You can also start it programmatically with the Mistral SDK via `client.workflows.*`.
+
+<!-- ILLUSTRATIVE: the exact client.workflows.* call shape is generic Mistral Workflows usage, not Pipelex-specific. Confirm against the Mistral SDK docs. -->
+
+## What just happened
+
+- The activity called `run_pipe_via_bridge`, which **booted Pipelex on first call** (idempotent — subsequent activities reuse the running instance).
+- With no `execution_mode` set, the pipe ran in **`direct`** mode: in-process inside the activity, blocking until it completed.
+- The bridge returned a `PipelexPipeRunOutput` — `output_dict`, `main_stuff_name`, `pipeline_run_id`, `is_completed`, and more — which the activity serialized with `model_dump()`.
+- The boundary is **JSON-only**: no Pipelex internal types (`PipeJob`, `WorkingMemory`, `PipeOutput`) cross the activity edge, which is what keeps the pipe replayable and the activity serializable.
+
+## Next
+
+- **[Execution Modes](execution-modes.md)** — run the pipe in-process, delegate to your own Temporal workers, or fire-and-forget.
+- **[Streaming Progress](streaming.md)** — surface live progress as the pipe runs.
diff --git a/docs/distributed-execution/observability.md b/docs/distributed-execution/observability.md
index e0ec95de4..ca3f13df6 100644
--- a/docs/distributed-execution/observability.md
+++ b/docs/distributed-execution/observability.md
@@ -43,16 +43,16 @@ Because the workflow id is derived from `pipeline_run_id`, a caller that control
### Child workflows
-Pipelex builds child workflow ids as slash-separated paths off the parent's workflow id:
+Pipelex builds child workflow ids off the parent's workflow id, joined with an underscore:
```
-{parent_workflow_id}/pipe-router # the fixed WfPipeRouter child of WfPipeRun
-{parent_workflow_id}/{pipe_code}-{8-hex} # a sub-pipe spawned by a router
+{parent_workflow_id}_pipe-router # the fixed WfPipeRouter child of WfPipeRun
+{parent_workflow_id}_{pipe_code}-{8-hex} # a sub-pipe spawned by a router
```
-The 8-hex suffix comes from `workflow.uuid4()` — Temporal's replay-safe UUID generator — so child workflow ids stay deterministic across worker restarts.
+The 8-hex suffix comes from `workflow.uuid4()` — Temporal's replay-safe UUID generator — so child workflow ids stay deterministic across worker restarts. The separator is an underscore, never a slash, so a workflow id can be reused verbatim as an S3 key or file name without creating spurious path segments.
-Operational tooling that parsed the previous nested-id format (which used `-` as the separator) must update to handle `/`.
+Operational tooling that parses the nested-id format must handle `_` as the separator.
---
diff --git a/docs/distributed-execution/temporal/index.md b/docs/distributed-execution/temporal/index.md
new file mode 100644
index 000000000..049a3ee4a
--- /dev/null
+++ b/docs/distributed-execution/temporal/index.md
@@ -0,0 +1,100 @@
+---
+title: "Pipelex on Temporal"
+description: "Run Pipelex pipelines as durable Temporal workflows on your own Temporal cluster or Temporal Cloud. Cluster setup, worker deployment, task-queue routing, and dashboard observability."
+---
+
+# Pipelex on Temporal
+
+Run your Pipelex pipelines as durable Temporal workflows across one or more worker processes. This section walks through setting up the cluster, running workers, routing work to the right pools, and reading the Temporal Web UI.
+
+This is the Temporal backend for distributed execution: you run **Pipelex's own Temporal workers** against a Temporal cluster you operate (self-hosted or Temporal Cloud) — you own the control plane. A second, managed alternative (Pipelex on Mistral Workflows) is coming soon. See the [Distributed Execution overview](../index.md) for the big picture.
+
+## What it is
+
+Pipelex pipelines normally run in-process — you call `pipelex run pipe ...` or invoke a pipe from Python and everything happens in one Python process. With the optional `pipelex[temporal]` integration, the same pipelines run as **Temporal workflows**: each pipe execution becomes a workflow on a Temporal cluster, child pipes become child workflows, and every LLM call, image generation, or document extraction becomes a Temporal activity executed on a worker. Temporal handles durability, retries, scheduling, and visibility; Pipelex handles the AI work.
+
+The whole layer is opt-in. Flip `[temporal] is_enabled = true` in `.pipelex/pipelex.toml`, install `pipelex[temporal]`, and the same `.mthds` methods run distributed without changing a line.
+
+## When you'd want it
+
+You don't need Temporal for local development — pipes run in-process by default. Reach for distributed execution when you need:
+
+- **Durability** — long-running pipelines survive process restarts and worker crashes. Temporal replays history to resume exactly where execution left off.
+- **Retries with budget control** — every activity (LLM call, extract, image gen) retries independently with per-activity timeouts and retry policies.
+- **Horizontal scale** — fan out work across multiple worker machines. Pipelex lets you route OpenAI calls to one pool, Anthropic to another, OCR to a third.
+- **Operational visibility** — every workflow shows up in the Temporal Web UI with the pipe code, pipeline run id, user, session, and a per-activity summary line.
+
+## The big picture
+
+```
+Your application
+ │
+ │ pipe_run(...) ──→ starts a Temporal workflow
+ ▼
+┌────────────────────┐
+│ Temporal cluster │ durability, history, retries, search
+└─────────┬──────────┘
+ │ tasks distributed by queue
+ ▼
+ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
+ │ worker │ │ worker │ │ worker │ poll task queues,
+ │ (router) │ │ (LLM pool) │ │ (img-gen) │ execute workflows
+ └─────────────┘ └─────────────┘ └─────────────┘ and activities
+```
+
+A submitter (your app or `pipelex run`) starts a workflow; one or more worker processes poll task queues for work; results flow back through the workflow's history. You can run a single worker that handles everything, or split workers by activity class for independent scaling and isolation.
+
+## What you'll set up
+
+Three things, in this order:
+
+- **[Cluster Setup](../cluster-setup.md)** — one-time namespace prerequisites: register the custom search attributes Pipelex needs for the dashboard view. Run `pipelex setup-temporal-namespace` and you're done.
+- **[Worker Deployment](../workers.md)** — how to run `pipelex worker`, what `--scope` and `--profile` do, and how to compose multiple workers into a deployment.
+- **[Task-Queue Routing](../task-routing.md)** — when you outgrow a single queue: per-activity routing, per-queue timeouts and retry policies, per-handle overrides. Skip on day one; revisit when you need to isolate workloads.
+
+## What you'll see in the Temporal dashboard
+
+Every workflow Pipelex starts carries human-readable identity: a workflow id derived from your pipeline run id, a Markdown summary of the pipe and its inputs, and per-activity summary lines that explain what each LLM or extract call is doing. Filter by `PipeCode`, `PipelineRunId`, `UserId`, `SessionId`, or `DomainCode` from the dashboard's search attribute selector. See **[Workflow Observability](../observability.md)** for the full reference.
+
+## Quick start
+
+Minimal configuration in `.pipelex/pipelex.toml`:
+
+```toml
+[temporal]
+is_enabled = true
+
+[temporal.worker_config]
+default_task_queue = "temporal_task_queue"
+
+[temporal.temporal_config]
+selected_server = "local"
+
+[temporal.temporal_config.temporal_server_configs.local]
+target_host = "localhost:7233"
+namespace = "default"
+api_key_method = "none"
+```
+
+Register the custom search attributes once (assumes Temporal is reachable):
+
+```bash
+pipelex setup-temporal-namespace
+```
+
+Boot a worker against the default task queue:
+
+```bash
+pipelex worker
+```
+
+Submit work from your application — the same `pipe_run(...)` you'd use locally now executes as a Temporal workflow because `[temporal] is_enabled = true`. The Pipelex Python API automatically dispatches through the Temporal hub.
+
+## Where to go next
+
+- **[Cluster Setup](../cluster-setup.md)** — search attributes, `pipelex setup-temporal-namespace`, Temporal Cloud permission model.
+- **[Worker Deployment](../workers.md)** — `pipelex worker`, scopes, runtime profiles, multi-worker topologies.
+- **[Task-Queue Routing](../task-routing.md)** — `activity_queues`, queue options, per-handle overrides, dispatch tracing.
+- **[Workflow Observability](../observability.md)** — workflow ids, activity ids, summary fields, search-attribute filtering.
+
+For the runtime mechanics that make distributed execution work under the hood — LibraryCrate propagation, deferred hydration, per-workflow class registry isolation — see [Temporal Integration](../../under-the-hood/temporal-integration.md) and [Distributed Content Generation](../../under-the-hood/distributed-content-generation.md).
diff --git a/docs/errors/event-log-error.md b/docs/errors/event-log-error.md
new file mode 100644
index 000000000..34aa9ed66
--- /dev/null
+++ b/docs/errors/event-log-error.md
@@ -0,0 +1,21 @@
+---
+title: "Event log"
+description: "Reference for the `EventLogError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Event log
+
+Base error for trace-event log backend failures.
+
+| Field | Value |
+|---|---|
+| `error_type` | `EventLogError` |
+| `title` | Event log |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/event-log-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.tracing.exceptions` |
+| Parent class | [`PipelexError`](pipelex-error.md) |
+
+[Back to Error Model overview](../under-the-hood/error-model.md)
diff --git a/docs/errors/event-log-read-error.md b/docs/errors/event-log-read-error.md
new file mode 100644
index 000000000..d918480ed
--- /dev/null
+++ b/docs/errors/event-log-read-error.md
@@ -0,0 +1,21 @@
+---
+title: "Event log read"
+description: "Reference for the `EventLogReadError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Event log read
+
+Raised when reading trace events from the configured backend fails.
+
+| Field | Value |
+|---|---|
+| `error_type` | `EventLogReadError` |
+| `title` | Event log read |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/event-log-read-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.tracing.exceptions` |
+| Parent class | [`EventLogError`](event-log-error.md) |
+
+[Back to Error Model overview](../under-the-hood/error-model.md)
diff --git a/docs/errors/event-log-setup-error.md b/docs/errors/event-log-setup-error.md
new file mode 100644
index 000000000..ec3b450d1
--- /dev/null
+++ b/docs/errors/event-log-setup-error.md
@@ -0,0 +1,21 @@
+---
+title: "Event log setup"
+description: "Reference for the `EventLogSetupError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Event log setup
+
+Raised when constructing the configured event-log backend fails.
+
+| Field | Value |
+|---|---|
+| `error_type` | `EventLogSetupError` |
+| `title` | Event log setup |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/event-log-setup-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.tracing.exceptions` |
+| Parent class | [`EventLogError`](event-log-error.md) |
+
+[Back to Error Model overview](../under-the-hood/error-model.md)
diff --git a/docs/errors/index.md b/docs/errors/index.md
index 1c4dd8471..5eeea1880 100644
--- a/docs/errors/index.md
+++ b/docs/errors/index.md
@@ -167,6 +167,12 @@ classification rules, and the cross-boundary Temporal bridge.
- [`DryRunError`](dry-run-error.md) — Dry run
+## EventLogError
+
+- [`EventLogError`](event-log-error.md) — Event log
+- [`EventLogReadError`](event-log-read-error.md) — Event log read
+- [`EventLogSetupError`](event-log-setup-error.md) — Event log setup
+
## GraphSpecError
- [`GraphSpecError`](graph-spec-error.md) — Graph spec
@@ -361,6 +367,13 @@ classification rules, and the cross-boundary Temporal bridge.
- [`BundleElaboratorError`](bundle-elaborator-error.md) — Bundle elaborator
- [`PipelexInterpreterError`](pipelex-interpreter-error.md) — Pipelex interpreter
+## PipelexRuntimeBridgeError
+
+- [`MissingMistralWorkflowsPluginError`](missing-mistral-workflows-plugin-error.md) — Missing mistral workflows plugin
+- [`MissingPipelexTemporalExtraError`](missing-pipelex-temporal-extra-error.md) — Missing pipelex temporal extra
+- [`PipelexBridgeDispatchError`](pipelex-bridge-dispatch-error.md) — Pipelex bridge dispatch
+- [`PipelexRuntimeBridgeError`](pipelex-runtime-bridge-error.md) — Pipelex runtime bridge
+
## PipelexServiceConfigValidationError
- [`PipelexServiceConfigValidationError`](pipelex-service-config-validation-error.md) — Pipelex service config validation
diff --git a/docs/errors/missing-mistral-workflows-plugin-error.md b/docs/errors/missing-mistral-workflows-plugin-error.md
new file mode 100644
index 000000000..e409e52df
--- /dev/null
+++ b/docs/errors/missing-mistral-workflows-plugin-error.md
@@ -0,0 +1,21 @@
+---
+title: "Missing mistral workflows plugin"
+description: "Reference for the `MissingMistralWorkflowsPluginError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Missing mistral workflows plugin
+
+Raised when MISTRAL_NATIVE mode is requested without ``pipelex-mistralai-workflows`` installed.
+
+| Field | Value |
+|---|---|
+| `error_type` | `MissingMistralWorkflowsPluginError` |
+| `title` | Missing mistral workflows plugin |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/missing-mistral-workflows-plugin-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.runtime_bridge.exceptions` |
+| Parent class | [`PipelexRuntimeBridgeError`](pipelex-runtime-bridge-error.md) |
+
+[Back to Error Model overview](../under-the-hood/error-model.md)
diff --git a/docs/errors/missing-pipelex-temporal-extra-error.md b/docs/errors/missing-pipelex-temporal-extra-error.md
new file mode 100644
index 000000000..b0a572e8b
--- /dev/null
+++ b/docs/errors/missing-pipelex-temporal-extra-error.md
@@ -0,0 +1,21 @@
+---
+title: "Missing pipelex temporal extra"
+description: "Reference for the `MissingPipelexTemporalExtraError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Missing pipelex temporal extra
+
+Raised when a TEMPORAL_* execution mode is requested without the pipelex[temporal] extra.
+
+| Field | Value |
+|---|---|
+| `error_type` | `MissingPipelexTemporalExtraError` |
+| `title` | Missing pipelex temporal extra |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/missing-pipelex-temporal-extra-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.runtime_bridge.exceptions` |
+| Parent class | [`PipelexRuntimeBridgeError`](pipelex-runtime-bridge-error.md) |
+
+[Back to Error Model overview](../under-the-hood/error-model.md)
diff --git a/docs/errors/pipelex-bridge-dispatch-error.md b/docs/errors/pipelex-bridge-dispatch-error.md
new file mode 100644
index 000000000..1e95f5b4b
--- /dev/null
+++ b/docs/errors/pipelex-bridge-dispatch-error.md
@@ -0,0 +1,21 @@
+---
+title: "Pipelex bridge dispatch"
+description: "Reference for the `PipelexBridgeDispatchError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Pipelex bridge dispatch
+
+Raised when a pipe dispatched through the bridge fails (validation or execution).
+
+| Field | Value |
+|---|---|
+| `error_type` | `PipelexBridgeDispatchError` |
+| `title` | Pipelex bridge dispatch |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/pipelex-bridge-dispatch-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.runtime_bridge.exceptions` |
+| Parent class | [`PipelexRuntimeBridgeError`](pipelex-runtime-bridge-error.md) |
+
+[Back to Error Model overview](../under-the-hood/error-model.md)
diff --git a/docs/errors/pipelex-runtime-bridge-error.md b/docs/errors/pipelex-runtime-bridge-error.md
new file mode 100644
index 000000000..3211280c6
--- /dev/null
+++ b/docs/errors/pipelex-runtime-bridge-error.md
@@ -0,0 +1,21 @@
+---
+title: "Pipelex runtime bridge"
+description: "Reference for the `PipelexRuntimeBridgeError` Pipelex error class."
+---
+
+<!-- pipelex:generated -->
+
+# Pipelex runtime bridge
+
+Base for errors raised by the Pipelex runtime-bridge surface.
+
+| Field | Value |
+|---|---|
+| `error_type` | `PipelexRuntimeBridgeError` |
+| `title` | Pipelex runtime bridge |
+| `type_uri` | `https://docs.pipelex.com/latest/errors/pipelex-runtime-bridge-error/` |
+| `error_domain` | _(inherited from parent)_ |
+| Defined in | `pipelex.runtime_bridge.exceptions` |
+| Parent class | [`PipelexError`](pipelex-error.md) |
+
+[Back to Error Model overview](../under-the-hood/error-model.md)
diff --git a/docs/features/distributed-execution.md b/docs/features/distributed-execution.md
index 0277bdd99..ea28b5bae 100644
--- a/docs/features/distributed-execution.md
+++ b/docs/features/distributed-execution.md
@@ -1,23 +1,34 @@
---
-title: "Distributed Execution with Temporal"
-description: "Run Pipelex pipelines as durable Temporal workflows across one or more worker processes. Per-activity routing, runtime profiles, and dashboard observability."
+title: "Distributed Execution"
+description: "Run Pipelex methods as durable workflows on either of two backends — your own Temporal cluster, or Mistral's managed Workflows control plane. Both built on Temporal."
---
-# Distributed Execution with Temporal
+# Distributed Execution
-Run your `.mthds` methods as durable Temporal workflows.
+Run your `.mthds` methods as durable workflows.
## Overview
-Pipelex pipelines normally run in-process. With the optional `pipelex[temporal]` integration, the same pipelines run as Temporal workflows: each pipe becomes a workflow, child pipes become child workflows, and every LLM call, image generation, or document extraction becomes an activity. Temporal handles durability, retries, scheduling, and visibility; Pipelex handles the AI work. Flip `[temporal] is_enabled = true` in `.pipelex/pipelex.toml` and the same methods run distributed without changing a line of method code.
+Pipelex pipelines normally run in-process. When you need durability, retries that survive failure, and horizontal scale, the same pipelines run as durable workflows — each pipe becomes a workflow, child pipes become child workflows, and every LLM call, image generation, or document extraction becomes an activity. The orchestration layer handles durability, retries, scheduling, and visibility; Pipelex handles the AI work, and the same methods run distributed without changing a line of method code.
-## Supported deployment patterns
+## Backends
+
+Distributed execution runs on Temporal. Today there is one backend, with a managed option coming soon:
+
+- **[Pipelex on Temporal](../distributed-execution/temporal/index.md)** — you run Pipelex's own Temporal workers against a Temporal cluster you operate (self-hosted or Temporal Cloud). Python 3.10+, `pipelex[temporal]`. Generally available.
+- **Pipelex on Mistral Workflows** *(coming soon)* — run pipes inside Mistral Workflows, Mistral's managed orchestration control plane (itself built on Temporal), with no cluster to operate. In active development; docs land once it ships.
+
+## Pipelex on Temporal
+
+Flip `[temporal] is_enabled = true` in `.pipelex/pipelex.toml`, install `pipelex[temporal]`, and the same methods run as Temporal workflows on workers you operate.
+
+### Supported deployment patterns
- **Single worker** — one process polls one task queue, runs everything. Right for most deployments.
- **Router + runners** — a dedicated workflow worker dispatches activities to one or more runner pools (LLM, image-gen, extract). Each runner pool scales independently and isolates failures.
- **Per-provider isolation** — separate worker pools for OpenAI, Anthropic, image generation, and OCR, each on its own task queue with its own retry policy and rate cap.
-## Configuration
+### Configuration
All knobs live under `[temporal.*]` in `.pipelex/pipelex.toml`:
@@ -31,4 +42,4 @@ All knobs live under `[temporal.*]` in `.pipelex/pipelex.toml`:
## Get started
-See the **[Distributed Execution with Temporal](../distributed-execution/index.md)** guide for the full walkthrough — overview, cluster setup, worker deployment, task-queue routing, and workflow observability.
+See the **[Distributed Execution](../distributed-execution/index.md)** guide for the full walkthrough — overview, cluster setup, worker deployment, task-queue routing, and workflow observability. A managed Mistral Workflows backend is coming soon.
diff --git a/docs/features/index.md b/docs/features/index.md
index 7fd3f06e3..0117dd853 100644
--- a/docs/features/index.md
+++ b/docs/features/index.md
@@ -95,9 +95,9 @@ This section covers all Pipelex capabilities, organized by area.
Pipeline validation without execution, dry run with mocked LLM responses, input validation, and allowed-to-fail pipes.
-- **[Distributed Execution with Temporal](distributed-execution.md)**
+- **[Distributed Execution](distributed-execution.md)**
- Run pipelines as durable Temporal workflows across worker pools. Per-activity routing, named runtime profiles, dashboard observability with custom search attributes.
+ Run methods as durable workflows on either backend — your own Temporal workers or Mistral's managed Workflows. Per-activity routing, named runtime profiles, and dashboard observability on the Temporal backend.
- **[Telemetry & Observability](telemetry.md)**
diff --git a/docs/reliability/automatic-retries.md b/docs/reliability/automatic-retries.md
new file mode 100644
index 000000000..55a2d5417
--- /dev/null
+++ b/docs/reliability/automatic-retries.md
@@ -0,0 +1,55 @@
+---
+title: "Automatic Retries"
+description: "The always-on mechanisms Pipelex applies on every execution path — transport-level retry, structured-output schema re-ask, and bounded fan-out for batches."
+---
+
+# Automatic Retries
+
+Whichever execution path you run, Pipelex applies a fixed set of automatic, bounded mechanisms to absorb transient failures and protect your providers. Transport retry is the one true retry here; the other two — schema re-ask and bounded fan-out — are closely related behaviors that shape output and control load. Together they are the always-on parts of the [retry model](retries-and-resilience.md#the-retry-model).
+
+## Tier 1 — transport retry
+
+Every inference SDK call retries transient transport failures before giving up. Pipelex makes this an explicit, uniform policy rather than inheriting each provider SDK's silent default.
+
+It is controlled by one top-level setting, which you override in your project's `.pipelex/pipelex.toml`:
+
+```toml
+[cogt]
+transport_max_retries = 2
+```
+
+`transport_max_retries` (default `2`) is the number of retries attempted **on top of** the initial request. A value of `2` allows up to 3 attempts total. What Pipelex sets uniformly is this *retry count*; the exact set of failures a retry fires on is the provider SDK's own transient-failure set — broadly connection errors, request timeouts (`408`), rate limits (`429`), and server errors (`5xx`), with minor per-SDK variation (Google's client, for instance, omits `409`). Pipelex's own SDK-less path (the raw-HTTP Azure image-generation client) pins this set explicitly to `408` / `409` / `429` / `5xx`. A `Retry-After` response header is honored where the SDK or that path supports it.
+
+This setting is wired into every inference SDK client that exposes a client-side retry budget — Anthropic, OpenAI / Azure OpenAI, the Pipelex Gateway LLM clients, Mistral, and Google — as well as the raw-HTTP Azure image-generation path. So the retry posture is one deliberate policy across those provider SDK clients. (The Pipelex Gateway's document-extraction and image-generation calls go through the Portkey SDK, which has no client-side retry budget; transport retries for those are owned by the gateway itself.)
+
+!!! info "Transport retry is not pipeline retry"
+ Tier 1 retries a *single* HTTP request to a provider. It does not re-run a pipe, re-run a step, or restart a pipeline. If a call still fails after its transport retries are exhausted, the error surfaces.
+
+## Structured output — schema re-ask
+
+When a pipe asks an LLM for a structured object, the model sometimes returns JSON that does not match the requested schema. Pipelex re-asks the model on that specific failure, via the `instructor` library.
+
+This is **output shaping, not resilience**. The re-ask happens only on a schema-validation failure — a transport error is *not* re-asked here; it propagates to Tier 1, which is the sole transport-retry layer. The re-ask count is configured separately:
+
+```toml
+[cogt.llm_config]
+schema_reask_max_attempts = 3 # instructor schema re-ask attempts — distinct from transport_max_retries
+```
+
+Keep the two settings distinct in your mind: `transport_max_retries` handles a flaky network; `schema_reask_max_attempts` handles a model that produced the wrong shape.
+
+## Bounded fan-out for batches
+
+When `PipeBatch` maps a pipe over a large list, it does not spawn every branch at once. Branches run in bounded concurrent chunks, capped by `max_concurrency` (default `8`):
+
+```toml
+[pipelex.pipeline_execution_config]
+max_concurrency = 8
+```
+
+This is **admission control, not retry** — it stops a batch over thousands of items from triggering a self-inflicted rate-limit storm against your provider. See [PipeBatch](../building-methods/pipes/pipe-controllers/PipeBatch.md#concurrency) for details.
+
+## Related
+
+- **[Failure Classification](failure-classification.md)** — how each failure is categorized, and which categories are worth retrying.
+- **[Durable Execution](durable-execution.md)** — Tier 2: activity-level retry and crash survival on a durable backend.
diff --git a/docs/reliability/durable-execution.md b/docs/reliability/durable-execution.md
new file mode 100644
index 000000000..330cd9361
--- /dev/null
+++ b/docs/reliability/durable-execution.md
@@ -0,0 +1,32 @@
+---
+title: "Durable Execution"
+description: "Tier 2 resilience — when to move from direct execution to a durable backend for crash survival, retry-under-failure, and horizontal scale."
+---
+
+# Durable Execution
+
+Direct execution makes one pipeline-level attempt and surfaces the error if it fails. **Durable execution** (Tier 2) adds activity-level retry and crash survival on top of the [automatic retries](automatic-retries.md) that run on every path. This is the resilient path described in the [retry model](retries-and-resilience.md#the-retry-model).
+
+## When to reach for it
+
+Stay on direct execution while you are developing, prototyping, or running short pipelines where a transient failure is acceptable to just re-run by hand.
+
+Move to a [durable backend](../distributed-execution/index.md) when you need:
+
+- **Crash survival** — a long pipeline resumes exactly where it left off after a worker restart.
+- **Retry under failure** — each LLM call, extraction, or image generation retries independently, with per-activity timeouts and a retry policy keyed off the [error category](failure-classification.md). (This per-leaf retry is the Temporal model, where each leaf maps to one host activity. On Mistral Workflows in `direct` mode the whole pipe runs inside a single host activity, so it retries as a unit; per-leaf retry there needs `mistral_native` mode.)
+- **Large durable batches** — running a pipe over thousands of items, durably and rate-limited.
+- **Horizontal scale** — fan work out across multiple worker machines.
+
+The same `.mthds` methods run on every path without changing a line — which durable backend runs them is a deployment choice, not a code change. The [Pipelex on Temporal](../distributed-execution/temporal/index.md) backend is switched on with `[temporal] is_enabled = true`, which dispatches the work through your Temporal cluster. A second, managed backend (Mistral Workflows) is coming soon: that flag will not apply to it — pipes run inside Workflows activities via the runtime bridge.
+
+!!! note "Durable backend"
+ Durable execution (Tier 2) runs on Temporal: your own Temporal cluster ([Pipelex on Temporal](../distributed-execution/temporal/index.md)). A managed option — Mistral's Workflows control plane, also built on Temporal — is coming soon; the resilience model described here will apply to it too, the difference being who operates the control plane.
+
+!!! tip "The error you see is the same on both paths"
+ A pipe that fails on a Temporal worker reaches your CLI or HTTP adapter with the *same* classification — category, retryable flag, model, provider, suggested action — as the identical failure run locally. Switching to a durable backend changes the resilience, not the error contract.
+
+## Related
+
+- **[Distributed Execution](../distributed-execution/index.md)** — set up a durable backend (Temporal or Mistral Workflows).
+- **[Failure Classification](failure-classification.md)** — the categories the Tier 2 retry policy keys off.
diff --git a/docs/reliability/failure-classification.md b/docs/reliability/failure-classification.md
new file mode 100644
index 000000000..0e2098e60
--- /dev/null
+++ b/docs/reliability/failure-classification.md
@@ -0,0 +1,21 @@
+---
+title: "Failure Classification"
+description: "How Pipelex classifies every inference failure into a category that drives retry decisions and reaches you in the error output."
+---
+
+# Failure Classification
+
+Whichever path runs, every inference failure is classified the moment it happens. The classification is what Tier 2 acts on, and it is what reaches you in the error output.
+
+| Category | Meaning | Retried by Temporal? |
+|----------|---------|----------------------|
+| `transient` | A brief, self-correcting failure (rate limit, 5xx, connection/timeout) | ✅ Yes |
+| `configuration` | The setup is wrong (bad API key, missing backend) | ❌ No |
+| `content` | The input or prompt is wrong (content-policy violation) | ❌ No |
+| `capacity` | Account quota or billing exhausted | ❌ No |
+| `ambiguous` | Outcome unknown — the call may have committed | ❌ No |
+| `unknown` | Could not classify | ❌ No |
+
+Only `transient` failures are worth retrying — re-running a call that failed because your API key is wrong just wastes time and money. On the [durable path](durable-execution.md), the activity retry policy uses exactly this signal: a `transient` failure retries, every other category fails fast.
+
+For the full mechanics of how errors are classified, carried, and reported, see [Error Model](../under-the-hood/error-model.md).
diff --git a/docs/reliability/retries-and-resilience.md b/docs/reliability/retries-and-resilience.md
index c56fb1537..e4a74b7d7 100644
--- a/docs/reliability/retries-and-resilience.md
+++ b/docs/reliability/retries-and-resilience.md
@@ -1,13 +1,13 @@
---
title: "Retries & Resilience"
-description: "How Pipelex handles transient failures — transport-level retry, the direct vs Temporal contract, and where retry legitimately lives."
+description: "How Pipelex handles transient failures — transport-level retry, the direct vs durable-execution contract, and where retry legitimately lives."
---
# Retries & Resilience
AI pipelines fail in transient ways: a provider rate-limits you, a connection drops, a model returns malformed JSON. Pipelex has a deliberate, honest model for handling those failures — and an equally deliberate line about what it does *not* do.
-The short version: **direct execution is for simplicity, Temporal execution is for resilience.** Both paths shrug off brief transport blips. Only the Temporal path survives a crash and retries under failure. There is no half-measure pretending otherwise.
+The short version: **direct execution is for simplicity, durable execution is for resilience.** Both paths shrug off brief transport blips. Only the durable path survives a crash and retries under failure. There is no half-measure pretending otherwise.
---
@@ -18,11 +18,11 @@ Pipelex runs your `.mthds` methods one of two ways, and they make different prom
| Path | What it promises | What it does on failure |
|------|------------------|-------------------------|
| **Direct execution** (default) | Simplicity, zero infrastructure | Retries transient *transport* failures via the provider SDK, then surfaces the error |
-| **Temporal execution** (opt-in) | Durability and resilience | The same transport retry, plus activity-level retry and crash survival |
+| **Durable execution** (opt-in) | Durability and resilience | The same transport retry, plus activity-level retry and crash survival |
Direct execution makes **one pipeline-level attempt**. If a pipe fails after the transport layer has done its retrying, the failure surfaces — the run does not restart, and it does not survive a process crash. That is the intended contract, not a gap. The value of the direct path is that it needs no cluster, no workers, no setup.
-When you need retry-under-failure and durability, you run the same methods on [Temporal](../distributed-execution/index.md) — see [When to reach for Temporal](#when-to-reach-for-temporal) below.
+When you need retry-under-failure and durability, you run the same methods on a durable backend — see [Durable Execution](durable-execution.md).
---
@@ -32,100 +32,21 @@ Retry in Pipelex lives in two well-defined tiers. Nothing retries outside them.
| Tier | What it does | Where it runs | Both paths? |
|------|--------------|---------------|-------------|
-| **Tier 1 — Transport retry** | Retries connection errors and HTTP `408` / `409` / `429` / `5xx`, honoring `Retry-After` | Inside the inference SDK / HTTP client | ✅ Yes |
-| **Tier 2 — Temporal durability** | Activity-level retry keyed off the error category, plus workflow durability and redelivery | The Temporal worker | ❌ Temporal only |
+| **[Tier 1 — Transport retry](automatic-retries.md)** | Retries connection errors and HTTP `408` / `409` / `429` / `5xx`, honoring `Retry-After` | Inside the inference SDK / HTTP client | ✅ Yes |
+| **[Tier 2 — Durable retry](durable-execution.md)** | Activity-level retry keyed off the error category, plus workflow durability and redelivery | The durable backend's worker | ❌ Durable only |
On the direct path there is **nothing between Tier 1 and Tier 2** — that gap *is* the difference between the two products.
```
Direct execution: [ Tier 1: transport retry ] → surface the error
-Temporal execution: [ Tier 1: transport retry ] → [ Tier 2: activity retry + durability ]
+Durable execution: [ Tier 1: transport retry ] → [ Tier 2: activity retry + durability ]
```
----
-
-## Tier 1 — transport retry
-
-Every inference SDK call retries transient transport failures before giving up. Pipelex makes this an explicit, uniform policy rather than inheriting each provider SDK's silent default.
-
-It is controlled by one top-level setting in `pipelex.toml`:
-
-```toml
-[cogt]
-transport_max_retries = 2
-```
-
-`transport_max_retries` (default `2`) is the number of retries attempted **on top of** the initial request. A value of `2` allows up to 3 attempts total. Retries fire on a connection error or an HTTP `408` / `409` / `429` / `5xx` response, and they honor a `Retry-After` response header when the provider sends one.
-
-This setting is wired uniformly into every inference SDK client — Anthropic, OpenAI / Azure OpenAI, the Pipelex Gateway clients, Mistral, and Google — as well as the raw-HTTP Azure image-generation path. So the retry posture is one deliberate policy across all providers.
-
-!!! info "Transport retry is not pipeline retry"
- Tier 1 retries a *single* HTTP request to a provider. It does not re-run a pipe, re-run a step, or restart a pipeline. If a call still fails after its transport retries are exhausted, the error surfaces.
-
----
-
-## Structured output — schema re-ask
-
-When a pipe asks an LLM for a structured object, the model sometimes returns JSON that does not match the requested schema. Pipelex re-asks the model on that specific failure, via the `instructor` library.
-
-This is **output shaping, not resilience**. The re-ask happens only on a schema-validation failure — a transport error is *not* re-asked here; it propagates to Tier 1, which is the sole transport-retry layer. The re-ask count is configured separately:
-
-```toml
-[cogt.llm_config]
-schema_reask_max_attempts = 3 # instructor schema re-ask attempts — distinct from transport_max_retries
-```
-
-Keep the two settings distinct in your mind: `transport_max_retries` handles a flaky network; `schema_reask_max_attempts` handles a model that produced the wrong shape.
-
----
-
-## Bounded fan-out for batches
-
-When `PipeBatch` maps a pipe over a large list, it does not spawn every branch at once. Branches run in bounded concurrent chunks, capped by `max_concurrency` (default `8`):
-
-```toml
-[pipelex.pipeline_execution_config]
-max_concurrency = 8
-```
-
-This is **admission control, not retry** — it stops a batch over thousands of items from triggering a self-inflicted rate-limit storm against your provider. See [PipeBatch](../building-methods/pipes/pipe-controllers/PipeBatch.md#concurrency) for details.
-
----
-
-## How a failure is classified
-
-Whichever path runs, every inference failure is classified the moment it happens. The classification is what Tier 2 acts on, and it is what reaches you in the error output.
-
-| Category | Meaning | Retried by Temporal? |
-|----------|---------|----------------------|
-| `transient` | A brief, self-correcting failure (rate limit, 5xx) | ✅ Yes |
-| `configuration` | The setup is wrong (bad API key, missing backend) | ❌ No |
-| `content` | The input or prompt is wrong (content-policy violation) | ❌ No |
-| `capacity` | Account quota or billing exhausted | ❌ No |
-| `ambiguous` | Outcome unknown — the call may have committed | ❌ No |
-| `unknown` | Could not classify | ❌ No |
-
-Only `transient` failures are worth retrying — re-running a call that failed because your API key is wrong just wastes time and money. On the Temporal path, the activity retry policy uses exactly this signal: a `transient` failure retries, every other category fails fast.
-
-For the full mechanics of how errors are classified, carried, and reported, see [Error Model](../under-the-hood/error-model.md).
-
----
-
-## When to reach for Temporal
-
-Stay on direct execution while you are developing, prototyping, or running short pipelines where a transient failure is acceptable to just re-run by hand.
-
-Move to [Temporal](../distributed-execution/index.md) when you need:
-
-- **Crash survival** — a long pipeline resumes exactly where it left off after a worker restart.
-- **Retry under failure** — each LLM call, extraction, or image generation retries independently, with per-activity timeouts and a retry policy keyed off the error category.
-- **Large durable batches** — running a pipe over thousands of items, durably and rate-limited.
-- **Horizontal scale** — fan work out across multiple worker machines.
-
-The same `.mthds` methods run on both paths without changing a line — flip `[temporal] is_enabled = true` and the work dispatches through Temporal.
+This section goes deeper on each piece:
-!!! tip "The error you see is the same on both paths"
- A pipe that fails on a Temporal worker reaches your CLI or HTTP adapter with the *same* classification — category, retryable flag, model, provider, suggested action — as the identical failure run locally. Switching to Temporal changes the resilience, not the error contract.
+- **[Automatic Retries](automatic-retries.md)** — Tier 1 transport retry, structured-output re-ask, and bounded fan-out — the always-on mechanisms.
+- **[Failure Classification](failure-classification.md)** — how each failure is categorized, and which categories are worth retrying.
+- **[Durable Execution](durable-execution.md)** — Tier 2: crash survival and retry-under-failure on a durable backend.
---
@@ -136,13 +57,15 @@ The same `.mthds` methods run on both paths without changing a line — flip `[t
| `transport_max_retries` | `[cogt]` | `2` | Tier 1 — transport retry attempts per request |
| `schema_reask_max_attempts` | `[cogt.llm_config]` | `3` | `instructor` schema re-ask attempts (structured output) |
| `max_concurrency` | `[pipelex.pipeline_execution_config]` | `8` | `PipeBatch` bounded fan-out (set `"unbounded"` to disable) |
-| `is_enabled` | `[temporal]` | `false` | Opt into Temporal execution (Tier 2) |
+| `is_enabled` | `[temporal]` | `false` | Opt into durable execution (Tier 2) |
---
## Next steps
-- [Distributed Execution with Temporal](../distributed-execution/index.md) — set up the resilient path
+- [Automatic Retries](automatic-retries.md) — the always-on transport retry, re-ask, and fan-out mechanisms
+- [Failure Classification](failure-classification.md) — how failures are categorized and which ones retry
+- [Durable Execution](durable-execution.md) — when to move to a durable backend, and what it adds
+- [Distributed Execution](../distributed-execution/index.md) — set up the durable backend (Temporal or Mistral Workflows)
- [Error Model](../under-the-hood/error-model.md) — how failures are classified and reported under the hood
- [Cogt Configuration](../configuration/config-technical/cogt-config.md) — the full `[cogt]` reference
-- [PipeBatch](../building-methods/pipes/pipe-controllers/PipeBatch.md) — mapping a pipe over a list, with bounded concurrency
diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.md
index 62100345f..bec00a2c0 100644
--- a/docs/under-the-hood/distributed-content-generation.md
+++ b/docs/under-the-hood/distributed-content-generation.md
@@ -5,7 +5,7 @@ description: "How Pipelex serializes dynamic Pydantic models and large binary co
# Distributed Content Generation
-This page is for contributors working on Pipelex internals. If you're deploying or operating Pipelex on Temporal, see the user-facing [Distributed Execution with Temporal](../distributed-execution/index.md) guide instead.
+This page is for contributors working on Pipelex internals. If you're deploying or operating Pipelex on Temporal, see the user-facing [Distributed Execution with Temporal](../distributed-execution/temporal/index.md) guide instead.
This page covers the serialization mechanisms that enable content generation (LLM structured output, image generation, PDF extraction) to run across Temporal worker processes. For the broader Temporal architecture — LibraryCrate propagation, deferred hydration, per-workflow scoping — see [Temporal Integration](./temporal-integration.md).
diff --git a/docs/under-the-hood/index.md b/docs/under-the-hood/index.md
index e44b67e84..ea5a59792 100644
--- a/docs/under-the-hood/index.md
+++ b/docs/under-the-hood/index.md
@@ -27,7 +27,7 @@ Welcome to the technical deep-dives of Pipelex. This section is for contributors
You don't need to read this section to use Pipelex effectively. The [Home](../index.md) section covers everything you need to build methods.
!!! tip "Looking for Temporal deployment and operations?"
- For running Pipelex on Temporal — cluster setup, workers, task-queue routing, and dashboard observability — see the user-facing [Distributed Execution with Temporal](../distributed-execution/index.md) guide. The pages below cover the runtime mechanics that make distributed execution work under the hood.
+ For running Pipelex on Temporal — cluster setup, workers, task-queue routing, and dashboard observability — see the user-facing [Distributed Execution with Temporal](../distributed-execution/temporal/index.md) guide. The pages below cover the runtime mechanics that make distributed execution work under the hood.
---
diff --git a/docs/under-the-hood/pipe-routing-and-execution.md b/docs/under-the-hood/pipe-routing-and-execution.md
index f25af8047..7697c7cc2 100644
--- a/docs/under-the-hood/pipe-routing-and-execution.md
+++ b/docs/under-the-hood/pipe-routing-and-execution.md
@@ -1,11 +1,11 @@
---
title: "Pipe Routing & Execution"
-description: "How PipeJobs are created, routed, and executed — covering both direct (single-process) and distributed (Temporal) execution modes."
+description: "How PipeJobs are created, routed, and executed — covering both direct (single-process) and distributed (host-runtime) execution."
---
# Pipe Routing & Execution
-Pipelex supports two execution modes for running pipes: **direct execution** (single-process) and **distributed execution** (via Temporal workers). Both modes share the same pipe definitions, library loading, and controller logic. The key difference is *where* the pipe runs and how the PipeJob travels to get there.
+Pipelex runs pipes either **direct** (single-process) or **distributed** on a host runtime. Both share the same pipe definitions, library loading, and controller logic. The key difference is *where* the pipe runs and how the PipeJob travels to get there. The distributed host runtime today is your own Temporal workers ([Pipelex on Temporal](../distributed-execution/temporal/index.md)); a managed Mistral Workflows backend (also built on Temporal) is coming soon. Both reach pipe execution through the same framework-agnostic runtime bridge.
---
@@ -14,7 +14,12 @@ Pipelex supports two execution modes for running pipes: **direct execution** (si
| Term | Meaning |
|------|---------|
| **Direct execution** | All pipes run in the same Python process. Library, class registry, and pipe resolution are shared in-memory. |
-| **Distributed execution** | PipeJob is serialized and sent to a remote Temporal worker. The worker is a separate process (potentially on a different machine). |
+| **Distributed execution** | PipeJob is serialized and sent to a remote worker — a Pipelex Temporal worker, or a Mistral Workflows activity. The worker is a separate process (potentially on a different machine). |
+
+---
+
+!!! note "The runtime bridge"
+ Both distributed paths go through the framework-agnostic runtime bridge (`pipelex/runtime_bridge/`). Its `primitives/` layer is shared by `pipelex.temporal` and the in-development Mistral Workflows host package: it classifies controller pipes as child workflows and leaf operators as activities (`pipe_classification.py`), hydrates working memory, flushes trace events, and delivers results in a host-neutral way. Graph and token-usage assembly is not a bridge primitive — it lives in `pipelex/pipe_run/tracing_assembly.py` and runs the same way for local and distributed runs.
---
diff --git a/docs/under-the-hood/temporal-integration.md b/docs/under-the-hood/temporal-integration.md
index a7896e46c..36bb881e4 100644
--- a/docs/under-the-hood/temporal-integration.md
+++ b/docs/under-the-hood/temporal-integration.md
@@ -5,7 +5,7 @@ description: "How Pipelex distributes pipe execution across Temporal workers —
# Temporal Integration
-This page is for contributors working on Pipelex internals. If you're deploying or operating Pipelex on Temporal, see the user-facing [Distributed Execution with Temporal](../distributed-execution/index.md) guide instead.
+This page is for contributors working on Pipelex internals. If you're deploying or operating Pipelex on Temporal, see the user-facing [Distributed Execution with Temporal](../distributed-execution/temporal/index.md) guide instead.
This page covers the Temporal-specific mechanisms that enable distributed pipe execution. For the general pipe routing architecture (shared between direct and distributed modes), see [Pipe Routing & Execution](./pipe-routing-and-execution.md).
diff --git a/mkdocs.yml b/mkdocs.yml
index bca7a271b..793a296f7 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -361,6 +361,16 @@ not_in_nav: |
!errors/index.md
agents/*.md
+# exclude_docs: forward-looking pages kept in the repo but NOT built or served on
+# the live site yet. The Mistral Workflows backend ships as a separate package
+# (pipelex-mistralai-workflows) that is not published yet, so its user docs are
+# held back to avoid advertising an unreleased package. Re-publish them — delete
+# this block and restore the "Mistral Workflows" nav entries under Distributed
+# Execution — once the package lands. See
+# wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md.
+exclude_docs: |
+ distributed-execution/mistral-workflows/
+
nav:
- Get Started:
- Build & Run AI Methods: index.md
@@ -397,13 +407,6 @@ nav:
- Viewpoint:
- The Know-How Graph: viewpoint/viewpoint.md
- Guide:
- - Retries & Resilience: reliability/retries-and-resilience.md
- - Distributed Execution with Temporal:
- - Overview: distributed-execution/index.md
- - Cluster Setup: distributed-execution/cluster-setup.md
- - Worker Deployment: distributed-execution/workers.md
- - Task-Queue Routing: distributed-execution/task-routing.md
- - Workflow Observability: distributed-execution/observability.md
- Features:
- Overview: features/index.md
- Declarative AI Methods:
@@ -424,7 +427,7 @@ nav:
- Execution Graph Visualization: features/execution-graph.md
- Production & Operations:
- Validation & Dry Run: features/validation-dry-run.md
- - Distributed Execution with Temporal: features/distributed-execution.md
+ - Distributed Execution: features/distributed-execution.md
- Telemetry & Observability: features/telemetry.md
- Cloud Storage: features/cloud-storage.md
- Cost Tracking & Reporting: features/cost-tracking.md
@@ -469,6 +472,19 @@ nav:
- Signature Pipes: building-methods/pipes/signature-pipes.md
- Optimize Cost & Quality: building-methods/configure-ai-llm-to-optimize-methods.md
- LLM Prompting Style: building-methods/adapt-to-llm-prompting-style-openai-anthropic-mistral.md
+ - Retries & Resilience:
+ - Overview: reliability/retries-and-resilience.md
+ - Automatic Retries: reliability/automatic-retries.md
+ - Failure Classification: reliability/failure-classification.md
+ - Durable Execution: reliability/durable-execution.md
+ - Distributed Execution:
+ - Overview: distributed-execution/index.md
+ - Temporal:
+ - Overview: distributed-execution/temporal/index.md
+ - Cluster Setup: distributed-execution/cluster-setup.md
+ - Worker Deployment: distributed-execution/workers.md
+ - Task-Queue Routing: distributed-execution/task-routing.md
+ - Workflow Observability: distributed-execution/observability.md
- Reference:
- CLI Reference:
- Overview: tools/cli/index.md
diff --git a/pipelex/cogt/extract/extract_report.py b/pipelex/cogt/extract/extract_report.py
index 7889a4d9a..0a4b2e6ec 100644
--- a/pipelex/cogt/extract/extract_report.py
+++ b/pipelex/cogt/extract/extract_report.py
@@ -12,7 +12,6 @@ class ExtractTokenCostReportField(StrEnum):
MODEL_TYPE = "model_type"
EXTRACT_NAME = "extract_name"
PLATFORM_EXTRACT_ID = "platform_extract_id"
- NB_TOKENS_INPUT = "nb_tokens_input"
NB_TOKENS_INPUT_CACHED = "nb_tokens_input_cached"
NB_TOKENS_INPUT_NON_CACHED = "nb_tokens_input_non_cached"
NB_TOKENS_INPUT_JOINED = "nb_tokens_input_joined"
diff --git a/pipelex/cogt/img_gen/img_gen_report.py b/pipelex/cogt/img_gen/img_gen_report.py
index 036b035a4..66d6351bb 100644
--- a/pipelex/cogt/img_gen/img_gen_report.py
+++ b/pipelex/cogt/img_gen/img_gen_report.py
@@ -12,7 +12,6 @@ class ImgGenTokenCostReportField(StrEnum):
MODEL_TYPE = "model_type"
IMG_GEN_NAME = "img_gen_name"
PLATFORM_IMG_GEN_ID = "platform_img_gen_id"
- NB_TOKENS_INPUT = "nb_tokens_input"
NB_TOKENS_INPUT_CACHED = "nb_tokens_input_cached"
NB_TOKENS_INPUT_NON_CACHED = "nb_tokens_input_non_cached"
NB_TOKENS_INPUT_JOINED = "nb_tokens_input_joined"
diff --git a/pipelex/cogt/llm/llm_report.py b/pipelex/cogt/llm/llm_report.py
index 0bc4cc3a0..17a8365d4 100644
--- a/pipelex/cogt/llm/llm_report.py
+++ b/pipelex/cogt/llm/llm_report.py
@@ -12,7 +12,6 @@ class LLMTokenCostReportField(StrEnum):
MODEL_TYPE = "model_type"
LLM_NAME = "llm_name"
PLATFORM_LLM_ID = "platform_llm_id"
- NB_TOKENS_INPUT = "nb_tokens_input"
NB_TOKENS_INPUT_CACHED = "nb_tokens_input_cached"
NB_TOKENS_INPUT_NON_CACHED = "nb_tokens_input_non_cached"
NB_TOKENS_INPUT_JOINED = "nb_tokens_input_joined" # joined = cached + non-cached
diff --git a/pipelex/cogt/search/search_report.py b/pipelex/cogt/search/search_report.py
index 4f7354092..ad958f1fd 100644
--- a/pipelex/cogt/search/search_report.py
+++ b/pipelex/cogt/search/search_report.py
@@ -12,7 +12,6 @@ class SearchTokenCostReportField(StrEnum):
MODEL_TYPE = "model_type"
SEARCH_NAME = "search_name"
PLATFORM_SEARCH_ID = "platform_search_id"
- NB_TOKENS_INPUT = "nb_tokens_input"
NB_TOKENS_INPUT_CACHED = "nb_tokens_input_cached"
NB_TOKENS_INPUT_NON_CACHED = "nb_tokens_input_non_cached"
NB_TOKENS_INPUT_JOINED = "nb_tokens_input_joined"
diff --git a/pipelex/hub.py b/pipelex/hub.py
index 2908a0a22..3331b8377 100644
--- a/pipelex/hub.py
+++ b/pipelex/hub.py
@@ -611,7 +611,55 @@ def get_required_concept(concept_ref: str) -> Concept:
return get_pipelex_hub().get_library().concept_library.get_required_concept(concept_ref=concept_ref)
+_current_pipe_router: ContextVar["PipeRouterProtocol | None"] = ContextVar("current_pipe_router", default=None)
+
+
+def set_pipe_router(pipe_router: "PipeRouterProtocol") -> None:
+ """Override the active pipe router for the current async context.
+
+ Used by host runtimes that want controllers to dispatch sub-pipes
+ through their own router (e.g. Mistral-native mode swaps in a router
+ that turns sub-pipe calls into child workflows / activities). The
+ override is contextvar-scoped, so concurrent runs on the same hub
+ don't leak into each other. Pass ``None`` via
+ ``teardown_current_pipe_router()`` to restore the hub default.
+ """
+ _current_pipe_router.set(pipe_router)
+
+
+def teardown_current_pipe_router() -> None:
+ """Clear any contextvar-scoped router override set by ``set_pipe_router``."""
+ _current_pipe_router.set(None)
+
+
+@contextmanager
+def scoped_pipe_router(pipe_router: "PipeRouterProtocol") -> Generator[None, None, None]:
+ """Set ``pipe_router`` as the active router for the scope, then restore the prior value on exit.
+
+ Captures the prior ``_current_pipe_router`` ContextVar value before setting
+ the new one. On exit — success or exception — restores the prior override
+ (or clears it if there wasn't one). Use this whenever a call needs its own
+ router for the *whole* run (root pipe + nested controller sub-pipes, which
+ resolve :func:`get_pipe_router`) without clobbering an outer caller's
+ override. Mirrors :func:`scoped_current_library`.
+
+ Prefer this over the raw ``set_pipe_router`` / ``teardown_current_pipe_router``
+ pair internally: the raw teardown unconditionally resets the override to
+ ``None`` and so does not restore an outer override. The raw pair is kept
+ because the external ``pipelex-mistralai-workflows`` plugin depends on it.
+ """
+ prev = _current_pipe_router.get()
+ _current_pipe_router.set(pipe_router)
+ try:
+ yield
+ finally:
+ _current_pipe_router.set(prev)
+
+
def get_pipe_router() -> "PipeRouterProtocol":
+ override = _current_pipe_router.get()
+ if override is not None:
+ return override
return get_pipelex_hub().get_required_pipe_router()
diff --git a/pipelex/pipe_run/delivery_assignment.py b/pipelex/pipe_run/delivery_assignment.py
index a4cfac876..44ebfd1b2 100644
--- a/pipelex/pipe_run/delivery_assignment.py
+++ b/pipelex/pipe_run/delivery_assignment.py
@@ -70,3 +70,13 @@ class DeliveryAssignment(BaseModel):
storage: StorageTarget | None = None
webhooks: list[WebhookTarget] = Field(default_factory=empty_list_factory_of(WebhookTarget))
+
+ @property
+ def has_delivery_target(self) -> bool:
+ """True when at least one real delivery target (storage or a webhook) is configured.
+
+ A ``DeliveryAssignment`` with no storage and no webhooks is a no-op: it would
+ persist nothing and notify no one, so a fire-and-forget completion would be
+ silently dropped.
+ """
+ return self.storage is not None or bool(self.webhooks)
diff --git a/pipelex/pipe_run/delivery_executor.py b/pipelex/pipe_run/delivery_executor.py
index 9957eab19..b8f5a7673 100644
--- a/pipelex/pipe_run/delivery_executor.py
+++ b/pipelex/pipe_run/delivery_executor.py
@@ -18,7 +18,7 @@ from pipelex.core.stuffs.stuff_viewer import render_stuff_viewer
from pipelex.graph.graph_factory import generate_graph_outputs
from pipelex.hub import get_class_registry, get_storage_provider
from pipelex.pipe_run.exceptions import PipeJobError, StorageDeliveryError, WebhookDeliveryError
-from pipelex.temporal.tprl_pipe.hydration import hydrate_content
+from pipelex.runtime_bridge.primitives.hydration import hydrate_content
from pipelex.tools.misc.json_utils import clean_json_dumps
from pipelex.tools.network.ssrf_guard import SsrfGuardedTransport
diff --git a/pipelex/pipe_run/tracing_assembly.py b/pipelex/pipe_run/tracing_assembly.py
index 5b6bef963..90adf1a37 100644
--- a/pipelex/pipe_run/tracing_assembly.py
+++ b/pipelex/pipe_run/tracing_assembly.py
@@ -11,7 +11,6 @@ that crosses the Temporal activity boundary (returned by ``act_assemble_tracing`
"""
import json
-from typing import cast
from pydantic import BaseModel, ConfigDict, ValidationError
@@ -23,20 +22,10 @@ from pipelex.graph.graphspec import GraphSpec, PipelineRef
from pipelex.reporting.reporting_types import AnyTokensUsage
from pipelex.system.exceptions import MissingDependencyError
from pipelex.tracing.event_log_factory import make_event_log
+from pipelex.tracing.exceptions import EventLogError
from pipelex.tracing.graphspec_assembler import GraphSpecAssembler
from pipelex.tracing.usage_aggregator import UsageAggregator
-# DynamoDB-backend read failures come in two sibling botocore base classes (neither subclasses the other):
-# ClientError (service-side throttle / auth) and BotoCoreError (transport / credential / timeout, e.g.
-# EndpointConnectionError, ReadTimeoutError, NoCredentialsError). Both are imported together (present or
-# absent together) and join the best-effort read catch when boto3 is installed — matching s3_storage_provider.
-try:
- from botocore.exceptions import BotoCoreError as _BotoCoreError # type: ignore[import-untyped]
- from botocore.exceptions import ClientError as _BotoClientError # type: ignore[import-untyped]
-except ImportError:
- _BotoCoreError = None # type: ignore[assignment, misc]
- _BotoClientError = None # type: ignore[assignment, misc]
-
class TracingAssembly(BaseModel):
"""Artifacts assembled from a single read of the trace-event stream.
@@ -71,10 +60,11 @@ def assemble_tracing(
Feeds the single event read into ``GraphSpecAssembler`` (when ``assemble_graph``)
and ``UsageAggregator`` (when ``assemble_usage``).
- Tracing is observability and treated as best-effort: runtime I/O issues
- (including the DynamoDB backend's botocore ``ClientError`` — service-side
- throttle / auth — and ``BotoCoreError`` — transport / credential / timeout —
- when boto3 is installed), malformed event data, and broken tracing
+ Tracing is observability and treated as best-effort: runtime I/O issues (the
+ DynamoDB backend converts its botocore throttle / auth / transport / timeout
+ failures into ``EventLogReadError`` on read and ``EventLogSetupError`` on client
+ construction — both subclasses of ``EventLogError``; NDJSON read failures surface
+ as ``OSError`` / ``JSONDecodeError``), malformed event data, and broken tracing
infrastructure (config errors, missing optional dependencies) are caught and
recorded in the ``*_error`` fields, not propagated. Programming bugs (KeyError,
AttributeError, etc.) propagate so they surface during development.
@@ -96,20 +86,18 @@ def assemble_tracing(
if not (assemble_graph or assemble_usage):
return result
- # Best-effort read: a backend failure degrades to an *_assembly_error note rather than failing the run.
- # Both botocore base classes (ClientError + BotoCoreError) join the catch together when boto3 is installed.
- read_errors: tuple[type[BaseException], ...] = (OSError, json.JSONDecodeError, ValidationError, PipelexConfigError, MissingDependencyError)
- if _BotoClientError is not None and _BotoCoreError is not None:
- # cast: botocore is untyped, so the imported classes are Unknown; we know they are BaseException types.
- read_errors = (*read_errors, cast("type[BaseException]", _BotoClientError), cast("type[BaseException]", _BotoCoreError))
-
+ # Best-effort: a backend failure degrades to an *_assembly_error note rather than failing the run.
+ # The DynamoDB backend converts its botocore failures (ClientError / BotoCoreError) into our
+ # EventLogError family — EventLogSetupError on client construction (make_event_log), EventLogReadError
+ # inside ``DynamoDBEventLog.read_events`` — so the assembly layer catches the EventLogError base and
+ # never imports boto3 itself; NDJSON read failures surface as OSError / JSONDecodeError.
try:
event_log = make_event_log(tracing_config)
try:
events = event_log.read_events(pipeline_run_id)
finally:
event_log.close()
- except read_errors as read_error:
+ except (OSError, json.JSONDecodeError, ValidationError, PipelexConfigError, MissingDependencyError, EventLogError) as read_error:
message = f"Tracing assembly failed to read events for pipeline_run_id={pipeline_run_id}: {read_error}"
log.warning(message)
if assemble_graph:
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index 84f681f55..6500f34db 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -101,6 +101,10 @@ class Pipelex(metaclass=MetaSingleton):
config_cls: type[ConfigRoot] | None = None,
config_overrides: dict[str, Any] | None = None,
) -> None:
+ # Readiness gate: flipped True only at the very end of make(), after setup() and the optional
+ # validate_model_deck() both succeed. Readers (ensure_pipelex_booted) must gate on this, NOT on
+ # mere registry presence -- MetaSingleton registers the instance before setup() configures the hub.
+ self.is_ready: bool = False
self.is_pipelex_service_enabled = False # Will be set during setup
self.config_dir_path = config_dir_path or config_manager.pipelex_config_dir
self.pipelex_hub = PipelexHub()
@@ -603,6 +607,10 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
if cls in MetaSingleton.instances:
del MetaSingleton.instances[cls]
raise
+ # Publish readiness only now: setup() AND the optional validate_model_deck() have both succeeded
+ # and the delete-on-failure handler above is behind us, so a reader can never adopt an instance
+ # that is about to be removed from the registry.
+ pipelex_instance.is_ready = True
log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} ready")
return pipelex_instance
@@ -611,6 +619,17 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
instance = MetaSingleton.instances.get(cls)
return cast("Self | None", instance)
+ @classmethod
+ def is_fully_booted(cls) -> bool:
+ """True only when a singleton exists AND has completed make() (setup + validation).
+
+ Distinct from ``get_optional_instance() is not None``: the metaclass registers the instance
+ before ``setup()`` configures the hub. Callers that must not touch a half-built instance
+ (``ensure_pipelex_booted``) gate on this.
+ """
+ instance = cls.get_optional_instance()
+ return instance is not None and instance.is_ready
+
@classmethod
def get_instance(cls) -> Self:
instance = MetaSingleton.instances.get(cls)
diff --git a/pipelex/plugins/gateway/gateway_completions_factory.py b/pipelex/plugins/gateway/gateway_completions_factory.py
index e28f59c73..7733178a3 100644
--- a/pipelex/plugins/gateway/gateway_completions_factory.py
+++ b/pipelex/plugins/gateway/gateway_completions_factory.py
@@ -15,6 +15,7 @@ from pipelex.cogt.extract.bounding_box import BoundingBox
from pipelex.cogt.extract.extract_output import ExtractedImageFromPage, ExtractOutput, Page
from pipelex.cogt.image.prompt_image import PromptImageDetail
from pipelex.cogt.image.prompt_image_utils import prep_prompt_images
+from pipelex.config import get_config
from pipelex.plugins.gateway.gateway_constants import GatewayOpenAISdkVariant
from pipelex.plugins.gateway.gateway_exceptions import GatewayExtractResponseError, GatewayFactoryError
from pipelex.plugins.gateway.gateway_factory import GatewayFactory
@@ -122,6 +123,9 @@ class GatewayCompletionsFactory(OpenAICompletionsFactory):
# not the OpenAI Authorization header. The SDK rejects an empty api_key since
# 2.34.0, so we pass a non-empty placeholder; the gateway ignores it.
api_key="unused-auth-via-portkey-headers",
+ # Tier 1 transport retry: set explicitly from config rather than inheriting the SDK default,
+ # so a transport_max_retries override applies to the gateway LLM path too (matches PortkeyCompletionsFactory).
+ max_retries=get_config().cogt.transport_max_retries,
default_headers=createHeaders(
api_key=api_key,
strict_open_ai_compliance=False,
diff --git a/pipelex/plugins/gateway/gateway_responses_factory.py b/pipelex/plugins/gateway/gateway_responses_factory.py
index f1f02f629..790074606 100644
--- a/pipelex/plugins/gateway/gateway_responses_factory.py
+++ b/pipelex/plugins/gateway/gateway_responses_factory.py
@@ -8,6 +8,7 @@ from portkey_ai import (
)
from typing_extensions import override
+from pipelex.config import get_config
from pipelex.plugins.gateway.gateway_constants import GatewayOpenAISdkVariant
from pipelex.plugins.gateway.gateway_exceptions import GatewayFactoryError
from pipelex.plugins.gateway.gateway_factory import GatewayFactory
@@ -40,6 +41,9 @@ class GatewayResponsesFactory(OpenAIResponsesFactory):
# not the OpenAI Authorization header. The SDK rejects an empty api_key since
# 2.34.0, so we pass a non-empty placeholder; the gateway ignores it.
api_key="unused-auth-via-portkey-headers",
+ # Tier 1 transport retry: set explicitly from config rather than inheriting the SDK default,
+ # so a transport_max_retries override applies to the gateway LLM path too (matches PortkeyResponsesFactory).
+ max_retries=get_config().cogt.transport_max_retries,
default_headers=createHeaders(
api_key=api_key,
debug=is_debug_enabled,
diff --git a/pipelex/runtime_bridge/__init__.py b/pipelex/runtime_bridge/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/runtime_bridge/bootstrap.py b/pipelex/runtime_bridge/bootstrap.py
new file mode 100644
index 000000000..e0aba0683
--- /dev/null
+++ b/pipelex/runtime_bridge/bootstrap.py
@@ -0,0 +1,51 @@
+"""Idempotent Pipelex boot helpers for use inside any host runtime that embeds Pipelex.
+
+Pipelex's own ``Pipelex.make()`` raises if a singleton already exists. The
+bridge boundary is a hot path that can be reached from many concurrent
+activities (Mistral Workflows, raw Temporal, future plugins), so we wrap the
+boot in an idempotent guard so callers don't have to think about it.
+"""
+
+import threading
+from typing import Any
+
+from pipelex.pipelex import Pipelex
+
+# Serializes the check-then-make below. Without it, two concurrent first-calls in
+# a fresh worker can both pass the readiness check and both call ``Pipelex.make``;
+# the loser then hits ``PipelexSetupError("Pipelex is already initialized")``.
+# ``MetaSingleton`` has no lock of its own. The race is across THREADS only — the
+# function is a sync ``def`` with no ``await`` between the check and the make, so
+# asyncio tasks on a single loop cannot interleave there; it bites Temporal's
+# sync-activity thread pool / multi-thread workers.
+_boot_lock = threading.Lock()
+
+
+def ensure_pipelex_booted(
+ config_overrides: dict[str, Any] | None = None,
+) -> None:
+ """Boot Pipelex on first call; no-op if already initialized.
+
+ Idempotent and thread-safe across its own callers: ``_boot_lock`` serializes
+ concurrent first-calls so ``Pipelex.make`` runs exactly once and late callers
+ re-check ``is_fully_booted()`` and adopt the result without touching a
+ half-built instance. If the host already booted Pipelex (e.g. at worker
+ startup) and that singleton is fully booted, this adopts it without
+ re-initializing — the intended pattern is to boot Pipelex before serving
+ bridge traffic.
+
+ Not supported: racing a *separate* external ``Pipelex.make()`` that is
+ mid-initialization on another thread. That external boot does not take
+ ``_boot_lock``, so within its setup window ``is_fully_booted()`` is False and
+ this helper would attempt its own ``make`` and hit
+ ``PipelexSetupError("Pipelex is already initialized")``. Standard hosts boot
+ before bringing up concurrent bridge callers, so this is not reached; the
+ residual race and a bounded-wait fix are written up in
+ ``wip/runtime-bridge/bridge-bootstrap-external-race.md``.
+ """
+ if Pipelex.is_fully_booted():
+ return
+ with _boot_lock:
+ # Re-check inside the lock: another thread may have finished booting while we waited.
+ if not Pipelex.is_fully_booted():
+ Pipelex.make(config_overrides=config_overrides)
diff --git a/pipelex/runtime_bridge/bridge.py b/pipelex/runtime_bridge/bridge.py
new file mode 100644
index 000000000..bc1f68339
--- /dev/null
+++ b/pipelex/runtime_bridge/bridge.py
@@ -0,0 +1,478 @@
+"""Framework-agnostic Pipelex runtime-bridge surface for host runtimes.
+
+This module contains the boundary types (``PipelexPipeRunInput`` /
+``PipelexPipeRunOutput``) and the dispatch entry-point
+(``run_pipe_via_bridge``) used by host runtimes (Mistral Workflows, raw
+Temporal, future plugins) to invoke Pipelex pipes from inside their own
+activities. It deliberately does NOT import any host-runtime-specific
+modules at module top-level so that callers can use the bridge directly
+(Tier 3 usage) and so that unit tests can exercise it without optional
+host-runtime deps installed.
+
+The Temporal extra is lazy-imported only inside the temporal-mode branches.
+"""
+
+from __future__ import annotations
+
+from contextlib import asynccontextmanager
+from typing import TYPE_CHECKING, Any, AsyncGenerator, Callable, cast
+from uuid import uuid4
+
+import shortuuid
+from kajson.class_registry import ClassRegistry
+from kajson.kajson_manager import KajsonManager
+from pydantic import BaseModel, ConfigDict, Field, ValidationError
+
+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.hub import (
+ get_library_manager,
+ get_required_pipe,
+ scoped_current_library,
+ scoped_pipe_router,
+)
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
+from pipelex.pipe_run.exceptions import PipeJobError, PipeRouterError, PipeRunError
+from pipelex.pipe_run.pipe_job_factory import PipeJobFactory
+from pipelex.pipe_run.pipe_router import PipeRouter
+from pipelex.pipe_run.pipe_run import PipeRun
+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.pipeline.exceptions import PipeExecutionError, PipelineExecutionError
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.runtime_bridge.bootstrap import ensure_pipelex_booted
+from pipelex.runtime_bridge.exceptions import (
+ MissingMistralWorkflowsPluginError,
+ MissingPipelexTemporalExtraError,
+ PipelexBridgeDispatchError,
+)
+from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode
+from pipelex.system.telemetry.otel_constants import OTelConstants
+
+if TYPE_CHECKING:
+ from pipelex.base_exceptions import PipelexError
+ from pipelex.core.memory.working_memory import WorkingMemory
+ from pipelex.core.pipes.pipe_output import PipeOutput
+ from pipelex.graph.trace_context import TraceContext
+ from pipelex.pipe_run.pipe_job import PipeJob
+ from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
+
+
+# Pipe-execution failures the bridge converts into PipelexBridgeDispatchError so a host can catch a
+# single error type regardless of execution mode. The Temporal branches additionally catch
+# WorkflowExecutionError (lazy-imported there to keep temporal off the module import path); wrapping
+# it with ``from exc`` loses no signal — the structured ErrorReport stays reachable via ``__cause__``
+# and is surfaced by PipelexBridgeDispatchError.to_error_report()'s cause-chain enrichment.
+_PIPE_DISPATCH_ERRORS: tuple[type[PipelexError], ...] = (
+ PipeRunError,
+ PipeJobError,
+ PipeRouterError,
+ PipeExecutionError,
+ PipelineExecutionError,
+)
+
+
+class PipelexPipeRunInput(BaseModel):
+ """JSON-safe input crossing the host-runtime / Temporal boundary."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ pipe_code: str
+ inputs: dict[str, Any] = Field(default_factory=dict)
+ output_name: str | None = None
+ pipeline_run_id: str | None = None
+ user_id: str | None = None
+ library_crate_dump: dict[str, Any] | None = None
+ execution_mode: PipelexExecutionMode = PipelexExecutionMode.DIRECT
+ delivery_assignment_dump: dict[str, Any] | None = None
+
+
+class PipelexPipeRunOutput(BaseModel):
+ """JSON-safe output crossing the host-runtime / Temporal boundary."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ output_dict: dict[str, Any]
+ main_stuff_name: str | None = None
+ pipeline_run_id: str
+ workflow_id: str | None = None
+ is_completed: bool
+ graph_spec_dump: dict[str, Any] | None = None
+ # graph_assembly_error / usage_assembly_error mirror the same fields on PipeOutput: a non-None
+ # value means assembly of the graph / token usage failed, which a host must be able to tell
+ # apart from "assembly was off" (a None graph_spec_dump / tokens_usages_dump). tokens_usages_dump
+ # is the JSON-safe dump of the AnyTokensUsage discriminated union so a host can render the
+ # end-of-run cost report: None when cost reporting was off, [] when on but no inference happened.
+ graph_assembly_error: str | None = None
+ tokens_usages_dump: list[dict[str, Any]] | None = None
+ usage_assembly_error: str | None = None
+
+
+async def run_pipe_via_bridge(
+ input_payload: PipelexPipeRunInput,
+ trace_context: TraceContext | None = None,
+) -> PipelexPipeRunOutput:
+ """Run a Pipelex pipe from inside a host-runtime activity.
+
+ Booting Pipelex on first call (no-op if already initialized); validating
+ the input; opening a per-call scoped library if a ``library_crate_dump``
+ is provided; then dispatching to the requested execution mode.
+
+ The optional ``trace_context`` is plumbed into ``JobMetadata`` so callers
+ (e.g. a streaming activity) that already opened a ``GraphTracerManager``
+ tracer for this pipeline run get per-step graph/usage trace events flowing
+ through the configured event log. ``trace_context`` is honored for ``DIRECT``
+ mode only — it is deliberately nulled for the Temporal modes, which have
+ their own event-log infrastructure via ``pipeline_run_setup``. Forwarding
+ a host ``trace_context`` to a Temporal mode would make Pipelex's Temporal
+ workflow open its tracer under the host's graph id and merge its trace
+ events into the host's graph, so the bridge does not thread it through.
+ """
+ ensure_pipelex_booted()
+
+ library_crate = _decode_library_crate(input_payload.library_crate_dump)
+ delivery_assignment = _decode_delivery_assignment(input_payload.delivery_assignment_dump)
+ _validate_input(input_payload, delivery_assignment=delivery_assignment)
+
+ async with _scoped_library_for_crate(library_crate):
+ # trace_context is honored for DIRECT only. The Temporal modes have their
+ # own event-log infrastructure (via pipeline_run_setup); forwarding a host
+ # trace_context there would make WfPipeRouter open its tracer under the
+ # host's graph_id and merge Pipelex's Temporal trace events into the host
+ # graph — exactly the cross-contamination the contract forbids. Null it
+ # for the non-DIRECT modes.
+ is_direct = input_payload.execution_mode is PipelexExecutionMode.DIRECT
+ pipe_job = build_pipe_job_from_input(
+ input_payload=input_payload,
+ library_crate=library_crate,
+ trace_context=trace_context if is_direct else None,
+ )
+
+ match input_payload.execution_mode:
+ case PipelexExecutionMode.DIRECT:
+ return await _run_direct(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+ case PipelexExecutionMode.TEMPORAL_BLOCKING:
+ _require_pipelex_temporal_extra()
+ return await _run_temporal_blocking(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+ case PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:
+ _require_pipelex_temporal_extra()
+ return await _run_temporal_fire_and_forget(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+ case PipelexExecutionMode.MISTRAL_NATIVE:
+ return await _run_mistral_native(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+
+
+def build_pipe_job_from_input(
+ input_payload: PipelexPipeRunInput,
+ library_crate: LibraryCrate | None,
+ trace_context: TraceContext | None = None,
+) -> PipeJob:
+ """Hydrate a PipeJob from JSON-safe input.
+
+ Looks up the pipe in the active library; the caller is responsible for
+ making sure the active library contains the pipe (by passing a
+ ``library_crate_dump`` or pre-loading the library at boot).
+
+ The optional ``trace_context`` is plumbed into ``JobMetadata`` so a
+ caller (e.g. a streaming activity) that has already opened a
+ ``GraphTracerManager`` tracer for this pipeline run can have per-step
+ ``PipeStartEvent`` / ``PipeEndSuccessEvent`` events flow through the
+ pipe execution. When ``None``, no tracing happens (current default).
+ """
+ pipe = get_required_pipe(pipe_code=input_payload.pipe_code)
+
+ # When the caller supplies a trace_context but no explicit pipeline_run_id, adopt the
+ # trace_context's lookup_key (= tracer_key or graph_id) as the run id. lookup_key is the key the
+ # caller's already-open tracer is registered under in GraphTracerManager AND the key PipeRun.run
+ # closes it under (close_tracer takes pipeline_run_id); minting a fresh id would leak the tracer.
+ #
+ # On every reachable path lookup_key == graph_id, so it is also the partition tracing_assembly
+ # reads under: only the Temporal child path (wf_pipe_router) opens a tracer with tracer_key !=
+ # graph_id, and it never routes through the bridge — non-DIRECT modes null trace_context, and it
+ # builds its PipeJob directly, not via this function. A host that hand-built a divergent keyed
+ # trace_context and forced it through DIRECT would split event emission (under graph_id) from
+ # assembly (under tracer_key); nothing constructs that, and it is documented as unsupported in
+ # wip/runtime-bridge/bridge-keyed-tracer-unsupported.md (graph_id "is typically the pipeline run id").
+ if input_payload.pipeline_run_id is not None:
+ pipeline_run_id = input_payload.pipeline_run_id
+ elif trace_context is not None:
+ pipeline_run_id = trace_context.lookup_key
+ else:
+ pipeline_run_id = shortuuid.uuid()
+
+ working_memory: WorkingMemory
+ if input_payload.inputs:
+ working_memory = WorkingMemoryFactory.make_from_pipeline_inputs(
+ pipeline_inputs=input_payload.inputs,
+ search_domain_codes=[pipe.domain_code],
+ )
+ else:
+ working_memory = WorkingMemoryFactory.make_empty()
+
+ job_metadata = JobMetadata(
+ user_id=input_payload.user_id or OTelConstants.DEFAULT_USER_ID,
+ pipeline_run_id=pipeline_run_id,
+ trace_context=trace_context,
+ )
+ pipe_run_params = PipeRunParamsFactory.make_run_params()
+
+ return PipeJobFactory.make_pipe_job(
+ pipe=pipe,
+ pipe_run_params=pipe_run_params,
+ job_metadata=job_metadata,
+ working_memory=working_memory,
+ output_name=input_payload.output_name,
+ library_crate=library_crate,
+ )
+
+
+def serialize_pipe_output(pipe_output: PipeOutput) -> dict[str, Any]:
+ """Dehydrate a PipeOutput's working memory to a JSON-safe dict.
+
+ Always uses ``WorkingMemory.dump_for_temporal()`` — the same format Pipelex
+ uses internally for Temporal transit. The shape is stable regardless of
+ whether a ``library_crate`` was attached:
+ ``{"root": {stuff_name: {"content": {...}, ...}}, "aliases": {...}}``.
+
+ Type metadata embedded by ``dump_for_temporal`` lets callers reconstruct a
+ typed ``WorkingMemory`` when they have the matching class registry in
+ scope (e.g. via ``hydrate_working_memory``).
+ """
+ return pipe_output.working_memory.dump_for_temporal()
+
+
+def _validate_input(input_payload: PipelexPipeRunInput, delivery_assignment: DeliveryAssignment | None) -> None:
+ if input_payload.execution_mode is PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:
+ if delivery_assignment is None or not delivery_assignment.has_delivery_target:
+ msg = (
+ "PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET requires a delivery_assignment_dump with at least one "
+ "delivery target (storage or a webhook); otherwise the pipe completion would be silently dropped."
+ )
+ raise PipelexBridgeDispatchError(msg)
+
+
+def _decode_library_crate(library_crate_dump: dict[str, Any] | None) -> LibraryCrate | None:
+ if library_crate_dump is None:
+ return None
+ try:
+ return LibraryCrate.model_validate(library_crate_dump)
+ except ValidationError as exc:
+ msg = f"Invalid library_crate_dump passed to the runtime bridge: {exc}"
+ raise PipelexBridgeDispatchError(msg) from exc
+
+
+def _decode_delivery_assignment(delivery_assignment_dump: dict[str, Any] | None) -> DeliveryAssignment | None:
+ if delivery_assignment_dump is None:
+ return None
+ try:
+ return DeliveryAssignment.model_validate(delivery_assignment_dump)
+ except ValidationError as exc:
+ msg = f"Invalid delivery_assignment_dump passed to the runtime bridge: {exc}"
+ raise PipelexBridgeDispatchError(msg) from exc
+
+
+@asynccontextmanager
+async def _scoped_library_for_crate(library_crate: LibraryCrate | None) -> AsyncGenerator[str | None, None]: # noqa: RUF029
+ """Open a per-call scoped library for the duration of a pipe run.
+
+ When ``library_crate`` is None, this is a no-op: callers fall back to the
+ library that was loaded into the active class registry at boot. When
+ provided, opens a fresh library, loads the crate into it, sets it as the
+ current library for the duration of the pipe execution, and tears it down
+ on the way out.
+ """
+ if library_crate is None:
+ yield None
+ return
+
+ library_manager = get_library_manager()
+ library_id = f"runtime_bridge_{uuid4().hex}"
+
+ # Pre-seed a per-call ClassRegistry from the global one so classes generated
+ # from the crate's inline structured concepts register into this scoped
+ # registry (discarded on teardown) rather than leaking into / colliding in
+ # the global Kajson registry. Mirrors the Temporal worker hydration path
+ # (see wf_pipe_router.py).
+ global_registry = KajsonManager.get_class_registry()
+ scoped_registry = ClassRegistry()
+ scoped_registry.register_classes_dict(global_registry.get_classes_dict())
+ # open_library + set_class_registry live inside the try so a throw between
+ # opening the library and reaching the yield still tears it down (the entry
+ # is registered in the manager the moment open_library returns). Mirrors the
+ # library_opened guard in submitter_hydration.rehydrate_pipe_output_with_crate.
+ library_opened = False
+ try:
+ _opened_library_id, library = library_manager.open_library(library_id=library_id)
+ library_opened = True
+ library.set_class_registry(scoped_registry)
+ # scoped_current_library captures and restores the prior current-library
+ # ContextVar, so a bridge call made from within an already-scoped library
+ # doesn't clobber the caller's context.
+ with scoped_current_library(library_id=library_id):
+ library_manager.load_from_crate(library_id=library_id, crate=library_crate)
+ yield library_id
+ finally:
+ if library_opened:
+ library_manager.teardown(library_id=library_id)
+
+
+async def _run_direct(
+ pipe_job: PipeJob,
+ delivery_assignment: DeliveryAssignment | None,
+) -> PipelexPipeRunOutput:
+ # DIRECT mode forces in-process execution even inside a Temporal-enabled
+ # worker. Scope the in-process router as the active router for the WHOLE
+ # run so nested controller sub-pipes — which dispatch through
+ # get_pipe_router() — resolve THIS router rather than falling back to the
+ # hub default. Without the scope, the hub default in a Temporal-enabled
+ # worker is the Temporal router, so a DIRECT-mode sequence/batch would leak
+ # its nested pipes to Temporal, defeating the point of DIRECT.
+ direct_router = PipeRouter()
+ with scoped_pipe_router(direct_router):
+ pipe_run = PipeRun(pipe_router=direct_router)
+ try:
+ pipe_output = await pipe_run.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+ except _PIPE_DISPATCH_ERRORS as exc:
+ msg = f"Pipe execution failed in DIRECT mode for pipe '{pipe_job.pipe.code}': {exc}"
+ raise PipelexBridgeDispatchError(msg) from exc
+
+ return _serialize_completed_output(
+ pipe_output=pipe_output,
+ workflow_id=None,
+ )
+
+
+async def _run_temporal_blocking(
+ pipe_job: PipeJob,
+ delivery_assignment: DeliveryAssignment | None,
+) -> PipelexPipeRunOutput:
+ from pipelex.temporal.exceptions import WorkflowExecutionError # noqa: PLC0415
+ from pipelex.temporal.tprl_pipe.temporal_pipe_run import make_temporal_pipe_run # noqa: PLC0415
+
+ # A pipe failure inside the Temporal workflow surfaces as WorkflowExecutionError (a
+ # TemporalFlowError, not in _PIPE_DISPATCH_ERRORS); catch it too so a Temporal-mode failure is
+ # wrapped just like DIRECT/mistral. `from exc` keeps its structured ErrorReport reachable.
+ dispatch_errors: tuple[type[PipelexError], ...] = (*_PIPE_DISPATCH_ERRORS, WorkflowExecutionError)
+ temporal_pipe_run = make_temporal_pipe_run()
+ try:
+ pipe_output = await temporal_pipe_run.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+ except dispatch_errors as exc:
+ msg = f"Pipe execution failed in TEMPORAL_BLOCKING mode for pipe '{pipe_job.pipe.code}': {exc}"
+ raise PipelexBridgeDispatchError(msg) from exc
+
+ # Report the actual Temporal workflow id, not the bare pipeline_run_id. run() started the
+ # workflow with make_workflow_id(pipeline_run_id), which prefixes the id in non-NORMAL run modes
+ # (ut-/ci-/cc-/cct- — see temporal_manager.make_top_workflow_id); the bare pipeline_run_id would
+ # not resolve there. Recomputing via the same make_workflow_id keeps a single source of truth and
+ # matches the id fire-and-forget returns from start(). In production (RunMode.NORMAL) the prefix
+ # is empty, so this equals pipeline_run_id.
+ workflow_id = temporal_pipe_run.make_workflow_id(pipeline_run_id=pipe_job.job_metadata.pipeline_run_id)
+
+ return _serialize_completed_output(
+ pipe_output=pipe_output,
+ workflow_id=workflow_id,
+ )
+
+
+async def _run_temporal_fire_and_forget(
+ pipe_job: PipeJob,
+ delivery_assignment: DeliveryAssignment | None,
+) -> PipelexPipeRunOutput:
+ from pipelex.temporal.exceptions import WorkflowExecutionError # noqa: PLC0415
+ from pipelex.temporal.tprl_pipe.temporal_pipe_run import make_temporal_pipe_run # noqa: PLC0415
+
+ # start() raises WorkflowExecutionError on a dispatch failure (a TemporalFlowError, not in
+ # _PIPE_DISPATCH_ERRORS); catch it too so fire-and-forget dispatch failures wrap uniformly.
+ dispatch_errors: tuple[type[PipelexError], ...] = (*_PIPE_DISPATCH_ERRORS, WorkflowExecutionError)
+ temporal_pipe_run = make_temporal_pipe_run()
+ try:
+ workflow_id, _handle = await temporal_pipe_run.start(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+ except dispatch_errors as exc:
+ msg = f"Pipe dispatch failed in TEMPORAL_FIRE_AND_FORGET mode for pipe '{pipe_job.pipe.code}': {exc}"
+ raise PipelexBridgeDispatchError(msg) from exc
+
+ return PipelexPipeRunOutput(
+ output_dict={},
+ main_stuff_name=None,
+ pipeline_run_id=pipe_job.job_metadata.pipeline_run_id,
+ workflow_id=workflow_id,
+ is_completed=False,
+ graph_spec_dump=None,
+ )
+
+
+def _serialize_completed_output(
+ pipe_output: PipeOutput,
+ workflow_id: str | None,
+) -> PipelexPipeRunOutput:
+ output_dict = serialize_pipe_output(pipe_output=pipe_output)
+
+ main_stuff_name = _resolve_main_stuff_root_key(pipe_output=pipe_output)
+
+ graph_spec_dump = pipe_output.graph_spec.model_dump(mode="json") if pipe_output.graph_spec is not None else None
+ tokens_usages_dump = [usage.model_dump(mode="json") for usage in pipe_output.tokens_usages] if pipe_output.tokens_usages is not None else None
+
+ return PipelexPipeRunOutput(
+ output_dict=output_dict,
+ main_stuff_name=main_stuff_name,
+ pipeline_run_id=pipe_output.pipeline_run_id,
+ workflow_id=workflow_id,
+ is_completed=True,
+ graph_spec_dump=graph_spec_dump,
+ graph_assembly_error=pipe_output.graph_assembly_error,
+ tokens_usages_dump=tokens_usages_dump,
+ usage_assembly_error=pipe_output.usage_assembly_error,
+ )
+
+
+def _resolve_main_stuff_root_key(pipe_output: PipeOutput) -> str | None:
+ """Return the actual ``root`` dict key under which the main stuff lives.
+
+ The main stuff can either sit directly at ``root[MAIN_STUFF_NAME]`` or be
+ referenced via ``aliases[MAIN_STUFF_NAME]`` pointing at its real name.
+ Callers indexing the output_dict need the actual root key, not the
+ stuff's display ``stuff_name``.
+ """
+ working_memory = pipe_output.working_memory
+ if MAIN_STUFF_NAME in working_memory.root:
+ return MAIN_STUFF_NAME
+ aliased_target = working_memory.aliases.get(MAIN_STUFF_NAME)
+ if aliased_target is not None and aliased_target in working_memory.root:
+ return aliased_target
+ return None
+
+
+def _require_pipelex_temporal_extra() -> None:
+ try:
+ import temporalio # noqa: F401, PLC0415
+ except ImportError as exc:
+ msg = "TEMPORAL_* execution modes require the pipelex[temporal] extra. Install with: pip install 'pipelex[temporal]'"
+ raise MissingPipelexTemporalExtraError(msg) from exc
+
+
+async def _run_mistral_native(
+ pipe_job: PipeJob,
+ delivery_assignment: DeliveryAssignment | None,
+) -> PipelexPipeRunOutput:
+ try:
+ from pipelex_mistralai_workflows.primitives.pipe_run import ( # type: ignore[import-not-found] # noqa: PLC0415 # pyright: ignore[reportMissingImports]
+ make_mistral_workflows_pipe_run as _make_pipe_run_untyped, # pyright: ignore[reportUnknownVariableType]
+ )
+ except ImportError as exc:
+ msg = (
+ "PipelexExecutionMode.MISTRAL_NATIVE requires the pipelex-mistralai-workflows "
+ "package. Install with: pip install pipelex-mistralai-workflows"
+ )
+ raise MissingMistralWorkflowsPluginError(msg) from exc
+
+ make_pipe_run = cast("Callable[[], PipeRunProtocol]", _make_pipe_run_untyped)
+ pipe_run = make_pipe_run()
+ try:
+ pipe_output = await pipe_run.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+ except (PipeRunError, PipeJobError, PipeRouterError, PipeExecutionError, PipelineExecutionError) as exc:
+ msg = f"Pipe execution failed in MISTRAL_NATIVE mode for pipe '{pipe_job.pipe.code}': {exc}"
+ raise PipelexBridgeDispatchError(msg) from exc
+
+ return _serialize_completed_output(
+ pipe_output=pipe_output,
+ workflow_id=pipe_output.pipeline_run_id,
+ )
diff --git a/pipelex/runtime_bridge/exceptions.py b/pipelex/runtime_bridge/exceptions.py
new file mode 100644
index 000000000..bf2bca3ca
--- /dev/null
+++ b/pipelex/runtime_bridge/exceptions.py
@@ -0,0 +1,23 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class PipelexRuntimeBridgeError(PipelexError):
+ """Base for errors raised by the Pipelex runtime-bridge surface."""
+
+
+class MissingPipelexTemporalExtraError(PipelexRuntimeBridgeError):
+ """Raised when a TEMPORAL_* execution mode is requested without the pipelex[temporal] extra."""
+
+ # The message carries the actionable pip-install hint; keep it under STRICT disclosure.
+ _authors_caller_facing_message = True
+
+
+class MissingMistralWorkflowsPluginError(PipelexRuntimeBridgeError):
+ """Raised when MISTRAL_NATIVE mode is requested without ``pipelex-mistralai-workflows`` installed."""
+
+ # The message carries the actionable pip-install hint; keep it under STRICT disclosure.
+ _authors_caller_facing_message = True
+
+
+class PipelexBridgeDispatchError(PipelexRuntimeBridgeError):
+ """Raised when a pipe dispatched through the bridge fails (validation or execution)."""
diff --git a/pipelex/runtime_bridge/execution_mode.py b/pipelex/runtime_bridge/execution_mode.py
new file mode 100644
index 000000000..4f2dd4396
--- /dev/null
+++ b/pipelex/runtime_bridge/execution_mode.py
@@ -0,0 +1,51 @@
+from pipelex.types import StrEnum
+
+
+class PipelexExecutionMode(StrEnum):
+ """How a Pipelex pipe runs inside a host runtime activity.
+
+ DIRECT: in-process; no Temporal involved on Pipelex's side; activity blocks
+ until the pipe completes. Fastest feedback, simplest ops.
+ TEMPORAL_BLOCKING: dispatch the pipe as a Pipelex Temporal workflow; the
+ activity awaits completion. Pipe runs durably on Pipelex's worker
+ fleet. Requires the pipelex[temporal] extra.
+ TEMPORAL_FIRE_AND_FORGET: dispatch the pipe as a Pipelex Temporal workflow
+ and return immediately with the workflow_id. Activity does NOT wait;
+ completion is signalled out-of-band via DeliveryAssignment (webhook /
+ storage). Same dep requirements as TEMPORAL_BLOCKING.
+ ``delivery_assignment_dump`` is required.
+ MISTRAL_NATIVE: decompose the pipe into native Mistral Workflows primitives
+ on the same worker — controllers as child workflows, leaves as
+ activities. Surfaces per-step retry / signals / cancellation through
+ the host-runtime APIs. Requires the
+ ``pipelex-mistralai-workflows`` package.
+ """
+
+ DIRECT = "direct"
+ TEMPORAL_BLOCKING = "temporal_blocking"
+ TEMPORAL_FIRE_AND_FORGET = "temporal_fire_and_forget"
+ MISTRAL_NATIVE = "mistral_native"
+
+ @property
+ def requires_pipelex_temporal(self) -> bool:
+ match self:
+ case PipelexExecutionMode.DIRECT | PipelexExecutionMode.MISTRAL_NATIVE:
+ return False
+ case PipelexExecutionMode.TEMPORAL_BLOCKING | PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:
+ return True
+
+ @property
+ def requires_mistral_workflows_extra(self) -> bool:
+ match self:
+ case PipelexExecutionMode.MISTRAL_NATIVE:
+ return True
+ case PipelexExecutionMode.DIRECT | PipelexExecutionMode.TEMPORAL_BLOCKING | PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:
+ return False
+
+ @property
+ def is_fire_and_forget(self) -> bool:
+ match self:
+ case PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:
+ return True
+ case PipelexExecutionMode.DIRECT | PipelexExecutionMode.TEMPORAL_BLOCKING | PipelexExecutionMode.MISTRAL_NATIVE:
+ return False
diff --git a/pipelex/runtime_bridge/primitives/__init__.py b/pipelex/runtime_bridge/primitives/__init__.py
new file mode 100644
index 000000000..31fa96856
--- /dev/null
+++ b/pipelex/runtime_bridge/primitives/__init__.py
@@ -0,0 +1,6 @@
+"""Framework-agnostic primitives shared across host-runtime integrations.
+
+These helpers are used by both ``pipelex.temporal`` (raw Temporal cluster
+mode) and ``pipelex_mistralai_workflows.primitives`` (Mistral-native mode).
+They contain no host-runtime-specific imports — only Pipelex core types.
+"""
diff --git a/pipelex/runtime_bridge/primitives/delivery.py b/pipelex/runtime_bridge/primitives/delivery.py
new file mode 100644
index 000000000..2d207254d
--- /dev/null
+++ b/pipelex/runtime_bridge/primitives/delivery.py
@@ -0,0 +1,37 @@
+"""Framework-agnostic delivery execution (storage + webhook).
+
+Wraps ``DeliveryExecutor.execute`` so host-runtime activities (Temporal /
+Mistral) can keep their decorators thin while sharing the same body.
+"""
+
+from pipelex.base_exceptions import ErrorReport
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus
+from pipelex.pipe_run.delivery_executor import DeliveryExecutor
+
+
+async def execute_delivery(
+ pipe_output: PipeOutput | None,
+ user_id: str,
+ pipeline_run_id: str,
+ delivery_assignment: DeliveryAssignment,
+ status: DeliveryStatus = DeliveryStatus.COMPLETED,
+ error_report: ErrorReport | None = None,
+ request_id: str | None = None,
+) -> None:
+ """Execute the full delivery (storage + webhooks).
+
+ ``error_report`` carries the worker-side classification on the failure path so the
+ webhook payload includes a structured ``error`` object. ``request_id`` is the
+ originating API request id, threaded into the delivery log lines for correlation.
+ """
+ executor = DeliveryExecutor()
+ await executor.execute(
+ pipe_output=pipe_output,
+ user_id=user_id,
+ pipeline_run_id=pipeline_run_id,
+ delivery_assignment=delivery_assignment,
+ status=status,
+ error_report=error_report,
+ request_id=request_id,
+ )
diff --git a/pipelex/temporal/tprl_pipe/hydration.py b/pipelex/runtime_bridge/primitives/hydration.py
similarity index 100%
rename from pipelex/temporal/tprl_pipe/hydration.py
rename to pipelex/runtime_bridge/primitives/hydration.py
diff --git a/pipelex/runtime_bridge/primitives/pipe_classification.py b/pipelex/runtime_bridge/primitives/pipe_classification.py
new file mode 100644
index 000000000..b39880b35
--- /dev/null
+++ b/pipelex/runtime_bridge/primitives/pipe_classification.py
@@ -0,0 +1,35 @@
+"""Pipe classification helpers for per-step-decomposition host runtimes.
+
+Framework-agnostic — answers the single question "should this pipe be
+dispatched as a child workflow (controller) or as an activity (leaf)?".
+This is bridge-side plumbing for the ``MISTRAL_NATIVE`` mode: the
+``pipelex_mistralai_workflows`` host package is the intended consumer of
+this controller/leaf split. There is no in-tree caller yet — the Temporal
+path makes the same decision elsewhere (``temporal_pipe_router``) — so the
+helpers live here, framework-agnostic, ready to be shared once the Mistral
+host integration lands.
+"""
+
+from pipelex.core.pipes.pipe_abstract import PipeAbstract
+from pipelex.pipe_controllers.pipe_controller import PipeController
+
+
+def is_controller_pipe(pipe: PipeAbstract) -> bool:
+ """Return True if ``pipe`` is a controller (PipeSequence, PipeBatch, …).
+
+ Controllers iterate over sub-pipes and are the right unit to dispatch as
+ a child workflow under per-step decomposition modes (e.g. MISTRAL_NATIVE).
+ Leaf / operator pipes (PipeLLM, PipeImg, PipePython, …) should run inside
+ activities, not workflows, because the controller boundary is exactly
+ where signals / timers / cancellation / per-step retry need to attach.
+ """
+ return isinstance(pipe, PipeController)
+
+
+def is_leaf_pipe(pipe: PipeAbstract) -> bool:
+ """Return True if ``pipe`` is a leaf operator (PipeLLM, PipeImg, …).
+
+ Convenience inverse of ``is_controller_pipe``. A leaf pipe maps to a
+ single I/O burst that fits inside one activity invocation.
+ """
+ return not is_controller_pipe(pipe)
diff --git a/pipelex/temporal/tprl_pipe/pipe_run_arg.py b/pipelex/runtime_bridge/primitives/pipe_run_arg.py
similarity index 100%
rename from pipelex/temporal/tprl_pipe/pipe_run_arg.py
rename to pipelex/runtime_bridge/primitives/pipe_run_arg.py
diff --git a/pipelex/temporal/tprl_pipe/submitter_hydration.py b/pipelex/runtime_bridge/primitives/submitter_hydration.py
similarity index 66%
rename from pipelex/temporal/tprl_pipe/submitter_hydration.py
rename to pipelex/runtime_bridge/primitives/submitter_hydration.py
index b8c7b3c6b..64c8d40e0 100644
--- a/pipelex/temporal/tprl_pipe/submitter_hydration.py
+++ b/pipelex/runtime_bridge/primitives/submitter_hydration.py
@@ -24,8 +24,8 @@ from kajson.class_registry import ClassRegistry
from kajson.kajson_manager import KajsonManager
from pipelex.core.pipes.pipe_output import PipeOutput
-from pipelex.hub import clear_current_library, get_current_library, get_library_manager, set_current_library
-from pipelex.temporal.tprl_pipe.hydration import hydrate_working_memory
+from pipelex.hub import get_library_manager, scoped_current_library
+from pipelex.runtime_bridge.primitives.hydration import hydrate_working_memory
if TYPE_CHECKING:
from pipelex.libraries.library_crate import LibraryCrate
@@ -55,12 +55,14 @@ def rehydrate_pipe_output_with_crate(
return pipe_output
library_manager = get_library_manager()
- rehydration_library_id = f"rehydrate_{uuid.uuid4().hex[:8]}"
- prev_library_id = _get_current_library_id_or_none()
+ rehydration_library_id = f"rehydrate_{uuid.uuid4().hex}"
+ # open_library + set_class_registry live inside the try so a throw before the yield
+ # still tears the library down. scoped_current_library captures and restores the
+ # prior current-library ContextVar, so this rehydration doesn't clobber the caller's
+ # context. Mirrors bridge._scoped_library_for_crate.
library_opened = False
- library_set_as_current = False
try:
- _lib_id, rehydration_library = library_manager.open_library(library_id=rehydration_library_id)
+ _opened_library_id, rehydration_library = library_manager.open_library(library_id=rehydration_library_id)
library_opened = True
global_registry = KajsonManager.get_class_registry()
@@ -68,28 +70,12 @@ def rehydrate_pipe_output_with_crate(
scoped_registry.register_classes_dict(global_registry.get_classes_dict())
rehydration_library.set_class_registry(scoped_registry)
- set_current_library(library_id=rehydration_library_id)
- library_set_as_current = True
- library_manager.load_from_crate(library_id=rehydration_library_id, crate=library_crate)
- pipe_output.working_memory = hydrate_working_memory(pipe_output.working_memory_raw)
- pipe_output.working_memory_raw = None
+ with scoped_current_library(library_id=rehydration_library_id):
+ library_manager.load_from_crate(library_id=rehydration_library_id, crate=library_crate)
+ pipe_output.working_memory = hydrate_working_memory(pipe_output.working_memory_raw)
+ pipe_output.working_memory_raw = None
finally:
- try:
- if library_opened:
- library_manager.teardown(library_id=rehydration_library_id)
- finally:
- if library_set_as_current:
- if prev_library_id is not None:
- set_current_library(library_id=prev_library_id)
- else:
- clear_current_library()
+ if library_opened:
+ library_manager.teardown(library_id=rehydration_library_id)
return pipe_output
-
-
-def _get_current_library_id_or_none() -> str | None:
- """Get the current library_id without raising if none is set."""
- try:
- return get_current_library()
- except RuntimeError:
- return None
diff --git a/pipelex/runtime_bridge/primitives/trace_flush.py b/pipelex/runtime_bridge/primitives/trace_flush.py
new file mode 100644
index 000000000..d287b18c4
--- /dev/null
+++ b/pipelex/runtime_bridge/primitives/trace_flush.py
@@ -0,0 +1,32 @@
+"""Framework-agnostic flush of buffered trace events to the configured backend.
+
+Called from a host-runtime activity (Temporal / Mistral) after a workflow's
+``BufferingEventLog`` has drained, since synchronous I/O (boto3, file
+writes) cannot run inside a workflow thread.
+"""
+
+from pipelex import log
+from pipelex.config import get_config
+from pipelex.tracing.event_log_factory import make_event_log
+from pipelex.tracing.trace_events import TraceEvent
+
+
+async def flush_trace_events_to_backend(events: list[TraceEvent]) -> None: # noqa: RUF029
+ """Persist buffered trace events to the configured backend.
+
+ No-op when the events list is empty or tracing is disabled.
+ """
+ if not events:
+ return
+
+ tracing_config = get_config().pipelex.tracing_config
+ if not tracing_config.is_enabled:
+ return
+
+ event_log = make_event_log(tracing_config)
+ try:
+ for event in events:
+ event_log.emit(event)
+ log.debug(f"Flushed {len(events)} trace events to backend={tracing_config.backend}")
+ finally:
+ event_log.close()
diff --git a/pipelex/temporal/tprl_pipe/act_deliver.py b/pipelex/temporal/tprl_pipe/act_deliver.py
index bb9c3d67b..688e85222 100644
--- a/pipelex/temporal/tprl_pipe/act_deliver.py
+++ b/pipelex/temporal/tprl_pipe/act_deliver.py
@@ -1,10 +1,16 @@
+"""Temporal activity wrapper for delivery (storage + webhook).
+
+Thin ``@activity.defn`` wrapper around the framework-agnostic core in
+``pipelex.runtime_bridge.primitives.delivery``.
+"""
+
from pydantic import BaseModel
from temporalio import activity
from pipelex.base_exceptions import ErrorReport
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus
-from pipelex.pipe_run.delivery_executor import DeliveryExecutor
+from pipelex.runtime_bridge.primitives.delivery import execute_delivery
from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors
@@ -28,9 +34,7 @@ class DeliveryActivityArg(BaseModel):
@activity.defn(name="act_deliver")
@convert_pipelex_errors
async def act_deliver(arg: DeliveryActivityArg) -> None:
- """Temporal activity that executes the full delivery (storage + webhooks)."""
- executor = DeliveryExecutor()
- await executor.execute(
+ await execute_delivery(
pipe_output=arg.pipe_output,
user_id=arg.user_id,
pipeline_run_id=arg.pipeline_run_id,
diff --git a/pipelex/temporal/tprl_pipe/act_flush_trace_events.py b/pipelex/temporal/tprl_pipe/act_flush_trace_events.py
index 5fd82ad61..5c09d647f 100644
--- a/pipelex/temporal/tprl_pipe/act_flush_trace_events.py
+++ b/pipelex/temporal/tprl_pipe/act_flush_trace_events.py
@@ -1,17 +1,15 @@
-"""Temporal activity for flushing buffered trace events to the configured backend.
+"""Temporal activity wrapper for flushing buffered trace events.
-Called from wf_pipe_router after pipe execution to persist trace events
-that were collected in-memory by BufferingEventLog during the workflow.
+Thin ``@activity.defn`` wrapper around the framework-agnostic core in
+``pipelex.runtime_bridge.primitives.trace_flush``.
"""
from pydantic import BaseModel, Field
from temporalio import activity
-from pipelex import log
-from pipelex.config import get_config
+from pipelex.runtime_bridge.primitives.trace_flush import flush_trace_events_to_backend
from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors
from pipelex.tools.typing.pydantic_utils import empty_list_factory_of
-from pipelex.tracing.event_log_factory import make_event_log
from pipelex.tracing.trace_events import TraceEvent
@@ -23,21 +21,5 @@ class FlushTraceEventsArg(BaseModel):
@activity.defn(name="act_flush_trace_events")
@convert_pipelex_errors
-async def act_flush_trace_events(arg: FlushTraceEventsArg) -> None: # noqa: RUF029
- """Write buffered trace events to the configured backend. Runs as a Temporal
- activity so synchronous I/O (boto3, file writes) doesn't block the workflow thread.
- """
- if not arg.events:
- return
-
- tracing_config = get_config().pipelex.tracing_config
- if not tracing_config.is_enabled:
- return
-
- event_log = make_event_log(tracing_config)
- try:
- for event in arg.events:
- event_log.emit(event)
- log.debug(f"Flushed {len(arg.events)} trace events to backend={tracing_config.backend}")
- finally:
- event_log.close()
+async def act_flush_trace_events(arg: FlushTraceEventsArg) -> None:
+ await flush_trace_events_to_backend(arg.events)
diff --git a/pipelex/temporal/tprl_pipe/temporal_pipe_router.py b/pipelex/temporal/tprl_pipe/temporal_pipe_router.py
index 213369dc6..d10a8c77c 100644
--- a/pipelex/temporal/tprl_pipe/temporal_pipe_router.py
+++ b/pipelex/temporal/tprl_pipe/temporal_pipe_router.py
@@ -11,6 +11,7 @@ from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.observer.observer_protocol import ObserverNoOp
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol
+from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate
from pipelex.temporal.exceptions import WorkflowExecutionError
from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment
from pipelex.temporal.temporal_workflow_utils import is_in_temporal_workflow
@@ -22,7 +23,6 @@ from pipelex.temporal.tprl.observability import (
stamp_submitter_session_id,
)
from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor, WorkflowExecutorFactory
-from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
@@ -61,12 +61,15 @@ class TemporalPipeRouter(WorkflowExecutor[PipeJob, PipeOutput], PipeRouterProtoc
if is_in_temporal_workflow():
# Child workflow dispatch (inside a Temporal workflow).
- # The child id is a slash-separated path off the parent's workflow id,
- # with a pipe-code prefix for readability and an 8-hex-char disambiguator
- # from ``workflow.uuid4()`` (replay-safe — Temporal's uuid4 is deterministic).
+ # The child id is built off the parent's workflow id with an underscore
+ # separator, a pipe-code segment for readability and an 8-hex-char
+ # disambiguator from ``workflow.uuid4()`` (replay-safe — Temporal's uuid4
+ # is deterministic). The separator is ``_``, never ``/``: workflow ids must
+ # stay free of path separators so they can be reused verbatim as S3 keys or
+ # file names without spawning spurious directory segments.
log.debug("TemporalPipeRouter: child workflow dispatch")
parent_workflow_id = workflow.info().workflow_id
- child_workflow_id = f"{parent_workflow_id}/{pipe_job.pipe.code}-{str(workflow.uuid4())[:8]}"
+ child_workflow_id = f"{parent_workflow_id}_{pipe_job.pipe.code}-{str(workflow.uuid4())[:8]}"
# Dispatch via ``workflow.execute_child_workflow`` directly. The
# ``WorkflowExecutorFactory`` is only safe at the submitter boundary
# because it reads ``get_config().temporal.worker_config`` to seed
diff --git a/pipelex/temporal/tprl_pipe/temporal_pipe_run.py b/pipelex/temporal/tprl_pipe/temporal_pipe_run.py
index d3b87162e..d526d78b2 100644
--- a/pipelex/temporal/tprl_pipe/temporal_pipe_run.py
+++ b/pipelex/temporal/tprl_pipe/temporal_pipe_run.py
@@ -10,6 +10,8 @@ from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_run_protocol import PipeRunProtocol
+from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg
+from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate
from pipelex.temporal.temporal_manager import TemporalWorkerEnvironment
from pipelex.temporal.tprl.conditional_worker import with_conditional_worker
from pipelex.temporal.tprl.observability import (
@@ -19,8 +21,6 @@ from pipelex.temporal.tprl.observability import (
stamp_submitter_session_id,
)
from pipelex.temporal.tprl.workflow_caller import WorkflowClass, WorkflowExecutor, WorkflowExecutorFactory
-from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg
-from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun
diff --git a/pipelex/temporal/tprl_pipe/wf_pipe_router.py b/pipelex/temporal/tprl_pipe/wf_pipe_router.py
index cfd2cebfd..2ec319777 100644
--- a/pipelex/temporal/tprl_pipe/wf_pipe_router.py
+++ b/pipelex/temporal/tprl_pipe/wf_pipe_router.py
@@ -14,11 +14,11 @@ with workflow.unsafe.imports_passed_through():
from pipelex.graph.graph_tracer_manager import GraphTracerManager
from pipelex.hub import clear_current_library, get_library_manager, get_report_delegate, set_current_library
from pipelex.pipe_run.pipe_job import PipeJob
+ from pipelex.runtime_bridge.primitives.hydration import hydrate_working_memory
from pipelex.temporal.log_temporal import WorkflowLog
from pipelex.temporal.tprl.temporal_error import TemporalError
from pipelex.temporal.tprl.workflow_caller import WorkflowClass
from pipelex.temporal.tprl_pipe.act_flush_trace_events import FlushTraceEventsArg, act_flush_trace_events
- from pipelex.temporal.tprl_pipe.hydration import hydrate_working_memory
from pipelex.tracing.buffering_event_log import BufferingEventLog
diff --git a/pipelex/temporal/tprl_pipe/wf_pipe_run.py b/pipelex/temporal/tprl_pipe/wf_pipe_run.py
index 55147a99a..02d3ad389 100644
--- a/pipelex/temporal/tprl_pipe/wf_pipe_run.py
+++ b/pipelex/temporal/tprl_pipe/wf_pipe_run.py
@@ -9,6 +9,7 @@ with workflow.unsafe.imports_passed_through():
from pipelex.base_exceptions import ErrorReport # noqa: TC001 # must traverse the workflow sandbox
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.delivery_assignment import DeliveryStatus
+ from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg
from pipelex.temporal.exceptions import WorkflowExecutionError
from pipelex.temporal.log_temporal import WorkflowLog
from pipelex.temporal.tprl.observability import build_search_attributes, build_static_summary
@@ -16,7 +17,6 @@ with workflow.unsafe.imports_passed_through():
from pipelex.temporal.tprl.workflow_caller import WorkflowClass
from pipelex.temporal.tprl_pipe.act_assemble_tracing import AssembleTracingArg, act_assemble_tracing
from pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg, act_deliver
- from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg
from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
@@ -59,15 +59,23 @@ class WfPipeRun(WorkflowClass[PipeRunArg, PipeOutput]):
pipe_output = await workflow.execute_child_workflow(
WfPipeRouter.run,
arg=pipe_job,
- id=f"{workflow.info().workflow_id}/pipe-router",
+ id=f"{workflow.info().workflow_id}_pipe-router",
search_attributes=build_search_attributes(pipe_job),
static_summary=build_static_summary(pipe_job.pipe),
)
workflow_log.debug("WfPipeRouter completed successfully")
except ChildWorkflowError as exc:
status = DeliveryStatus.FAILED
- # Hold the wrapped error for a deferred re-raise after delivery — raising here would short-circuit ``act_deliver``.
- # ``ChildWorkflowError`` exposes the underlying failure via ``exc.cause``, not ``__cause__``.
+ # Wrap the raw ``ChildWorkflowError`` as ``WorkflowExecutionError`` so the rest of
+ # this function and the outer ``execute_workflow`` caller continue to see the same
+ # Pipelex error type as before — the integration test ``test_wf_pipe_run_failure_path``
+ # pins the workflow_failure_exception_types contract on ``WorkflowExecutionError``.
+ # ``recover_error_report`` lifts the structured classification out of the child failure
+ # (``ChildWorkflowError`` exposes it via ``exc.cause``) so the FAILED webhook carries it.
+ #
+ # Manual ``__cause__`` wire (not ``raise X from exc``): we hold the wrapped error for a
+ # deferred re-raise in the post-delivery block below so ``act_deliver`` still fires on
+ # the failure path. Raising here would short-circuit delivery.
error_report = recover_error_report(exc.cause if exc.cause is not None else exc)
execution_error = WorkflowExecutionError("WfPipeRouter failed", error_report=error_report)
execution_error.__cause__ = exc
diff --git a/pipelex/tracing/dynamodb_event_log.py b/pipelex/tracing/dynamodb_event_log.py
index 368cd91f9..9a9fe2c13 100644
--- a/pipelex/tracing/dynamodb_event_log.py
+++ b/pipelex/tracing/dynamodb_event_log.py
@@ -21,6 +21,7 @@ from pipelex import log
from pipelex.system.exceptions import MissingDependencyError
from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
from pipelex.tracing.event_log_protocol import EventLogProtocol
+from pipelex.tracing.exceptions import EventLogReadError, EventLogSetupError
from pipelex.tracing.trace_events import AnyTraceEvent, TraceEvent
try:
@@ -58,8 +59,20 @@ class DynamoDBEventLog(EventLogProtocol):
self._writer_id = writer_id
self._sequence: int = 0
self._sequence_lock = threading.Lock()
- dynamodb: Any = boto3.resource("dynamodb", region_name=self._region) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
- self._table: Any = dynamodb.Table(self._table_name) # pyright: ignore[reportUnknownMemberType]
+ # Translate construction-time botocore failures (e.g. a misconfigured region) into our
+ # domain EventLogSetupError, mirroring the read_events boundary. Best-effort callers
+ # (tracing assembly) catch the EventLogError base and degrade rather than aborting the run.
+ from botocore.exceptions import ( # noqa: PLC0415 - optional dependency, lazy import
+ BotoCoreError,
+ ClientError,
+ )
+
+ try:
+ dynamodb: Any = boto3.resource("dynamodb", region_name=self._region) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType]
+ self._table: Any = dynamodb.Table(self._table_name) # pyright: ignore[reportUnknownMemberType]
+ except (ClientError, BotoCoreError) as exc:
+ msg = f"Failed to construct the DynamoDB event-log client for table={self._table_name} region={self._region}: {exc}"
+ raise EventLogSetupError(msg) from exc
@property
@override
@@ -106,23 +119,37 @@ class DynamoDBEventLog(EventLogProtocol):
@override
def read_events(self, pipeline_run_id: str) -> list[TraceEvent]:
- """Read all events for a pipeline run from DynamoDB."""
- items: list[dict[str, Any]] = []
+ """Read all events for a pipeline run from DynamoDB.
- response = self._table.query(
- KeyConditionExpression=self._key_condition(pipeline_run_id),
- ScanIndexForward=True,
+ Store-level failures (throttling, auth, network) are translated into
+ ``EventLogReadError`` so best-effort callers (graph assembly) can degrade
+ gracefully without catching raw botocore exceptions. Individual
+ unparseable items are skipped with a warning rather than failing the read.
+ """
+ from botocore.exceptions import ( # noqa: PLC0415 - optional dependency, lazy import
+ BotoCoreError,
+ ClientError,
)
- items.extend(response.get("Items", []))
- while "LastEvaluatedKey" in response:
+ items: list[dict[str, Any]] = []
+ try:
response = self._table.query(
KeyConditionExpression=self._key_condition(pipeline_run_id),
ScanIndexForward=True,
- ExclusiveStartKey=response["LastEvaluatedKey"],
)
items.extend(response.get("Items", []))
+ while "LastEvaluatedKey" in response:
+ response = self._table.query(
+ KeyConditionExpression=self._key_condition(pipeline_run_id),
+ ScanIndexForward=True,
+ ExclusiveStartKey=response["LastEvaluatedKey"],
+ )
+ items.extend(response.get("Items", []))
+ except (ClientError, BotoCoreError) as exc:
+ msg = f"Failed to read trace events from DynamoDB for pipeline_run_id={pipeline_run_id}: {type(exc).__name__}"
+ raise EventLogReadError(msg) from exc
+
events: list[TraceEvent] = []
for item in items:
payload = item.get("payload")
diff --git a/pipelex/tracing/exceptions.py b/pipelex/tracing/exceptions.py
new file mode 100644
index 000000000..393845614
--- /dev/null
+++ b/pipelex/tracing/exceptions.py
@@ -0,0 +1,30 @@
+from pipelex.base_exceptions import PipelexError
+
+
+class EventLogError(PipelexError):
+ """Base error for trace-event log backend failures."""
+
+
+class EventLogReadError(EventLogError):
+ """Raised when reading trace events from the configured backend fails.
+
+ Backends translate their own infrastructure exceptions (e.g. botocore
+ ``ClientError`` / ``BotoCoreError`` from DynamoDB on throttling, auth, or
+ network failures) into this domain error at the boundary. That lets
+ best-effort observability callers — graph assembly degrades to ``None``
+ rather than failing the pipeline run — catch a single Pipelex error instead
+ of re-widening their ``except`` clauses to raw third-party exception types.
+ """
+
+
+class EventLogSetupError(EventLogError):
+ """Raised when constructing the configured event-log backend fails.
+
+ The sibling of :class:`EventLogReadError` for the *construction* boundary:
+ backends translate the infrastructure exceptions their client setup can raise
+ (e.g. botocore ``ClientError`` / ``BotoCoreError`` from ``boto3.resource`` on a
+ misconfigured region or credential chain) into this domain error. Best-effort
+ observability callers catch the :class:`EventLogError` base so a backend that
+ cannot even be built degrades to an ``*_assembly_error`` note rather than
+ aborting the pipeline run.
+ """
diff --git a/pipelex/tracing/ndjson_event_log.py b/pipelex/tracing/ndjson_event_log.py
index 025005790..c830d5087 100644
--- a/pipelex/tracing/ndjson_event_log.py
+++ b/pipelex/tracing/ndjson_event_log.py
@@ -73,9 +73,10 @@ class NdjsonEventLog(EventLogProtocol):
def _file_name_for(workflow_id: str, writer_id: str) -> str:
"""File name for a (workflow_id, writer_id) pair.
- Child workflow IDs now use ``/`` as a path separator (e.g.
- ``ut-{uuid}/step_two-9a262f1f``); replace it with ``__`` so the
- derived file name stays flat inside the run directory.
+ Workflow ids use ``_`` as their separator and so should not contain
+ ``/`` (e.g. ``ut-{uuid}_step_two-9a262f1f``). The ``replace`` below is a
+ defensive guard that keeps the derived file name flat inside the run
+ directory even if a ``/`` ever slips into an id.
The legacy single-writer name is preserved when writer_id="primary"
so existing files continue to be written and read correctly.
diff --git a/pyproject.toml b/pyproject.toml
index a1e50ce8c..37b89f3f4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -91,7 +91,7 @@ linkup = ["linkup-sdk>=0.12.0"]
mistralai = ["mistralai>=1.12.0"]
dynamodb = ["boto3>=1.34.131"]
s3 = ["boto3>=1.34.131", "aioboto3>=13.4.0"]
-temporal = ["temporalio==1.23.0", "aiohttp>=3.14.0"]
+temporal = ["temporalio==1.24.0", "aiohttp>=3.14.0"]
docs = [
"mike>=2.1.3",
"mkdocs>=1.6.1",
@@ -159,6 +159,8 @@ module = [
]
[tool.pyright]
+venvPath = "."
+venv = ".venv"
pythonVersion = "3.11"
include = ["pipelex", "tests"]
exclude = [
diff --git a/tests/integration/pipelex/runtime_bridge/conftest.py b/tests/integration/pipelex/runtime_bridge/conftest.py
new file mode 100644
index 000000000..f37bf871e
--- /dev/null
+++ b/tests/integration/pipelex/runtime_bridge/conftest.py
@@ -0,0 +1,40 @@
+from collections.abc import Generator
+from pathlib import Path
+
+import pytest
+
+from pipelex.hub import get_func_registry, get_library_manager, set_current_library
+from tests.integration.pipelex.runtime_bridge.test_data.bridge_funcs import (
+ mistralai_workflows_bridge_echo,
+ mistralai_workflows_bridge_raise,
+)
+
+TEST_DATA_DIR = Path(__file__).parent / "test_data"
+
+
[email protected](scope="class")
+def bridge_test_library() -> Generator[str, None, None]:
+ """Open a class-scoped library populated with the bridge test pipe.
+
+ The pipe ``mistralai_workflows_bridge_test.bridge_func_pipe`` is registered
+ in the active library, and the matching Python function is registered in
+ the FuncRegistry. Both are torn down on exit.
+ """
+ func_registry = get_func_registry()
+ func_registry.register_function(mistralai_workflows_bridge_echo)
+ func_registry.register_function(mistralai_workflows_bridge_raise)
+
+ library_manager = get_library_manager()
+ library_id, _ = library_manager.open_library()
+ set_current_library(library_id=library_id)
+ library_manager.load_libraries(
+ library_id=library_id,
+ library_dirs=[TEST_DATA_DIR],
+ )
+ try:
+ yield library_id
+ finally:
+ library_manager.teardown(library_id=library_id)
+ for func_name in ("mistralai_workflows_bridge_echo", "mistralai_workflows_bridge_raise"):
+ if func_registry.has_function(func_name):
+ func_registry.unregister_function_by_name(func_name)
diff --git a/tests/integration/pipelex/runtime_bridge/test_bridge_direct.py b/tests/integration/pipelex/runtime_bridge/test_bridge_direct.py
new file mode 100644
index 000000000..628035411
--- /dev/null
+++ b/tests/integration/pipelex/runtime_bridge/test_bridge_direct.py
@@ -0,0 +1,272 @@
+"""Layer-1 integration tests for the runtime-bridge in DIRECT mode.
+
+These tests do NOT depend on any host-runtime optional package — they
+exercise only the framework-agnostic core (``run_pipe_via_bridge`` with a
+real loaded pipe). The Mistral Workflows activity wrapper is covered
+separately in the ``pipelex-mistralai-workflows`` package, which DOES
+require the optional dep.
+"""
+
+import json
+from typing import Any
+
+import pytest
+from kajson.kajson_manager import KajsonManager
+
+from pipelex.graph.graph_config import DataInclusionConfig
+from pipelex.graph.trace_context import TraceContext
+from pipelex.hub import get_current_library_id_or_none, get_library_manager
+from pipelex.runtime_bridge.bridge import (
+ PipelexPipeRunInput,
+ build_pipe_job_from_input,
+ run_pipe_via_bridge,
+)
+from pipelex.runtime_bridge.exceptions import PipelexBridgeDispatchError
+from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode
+
+PIPE_REF = "mistralai_workflows_bridge_test.bridge_func_pipe"
+ENVELOPE_PIPE_REF = "mistralai_workflows_bridge_test.bridge_envelope_pipe"
+RAISE_PIPE_REF = "mistralai_workflows_bridge_test.bridge_raise_pipe"
+
+
[email protected](loop_scope="class")
+class TestBridgeDirect:
+ async def test_direct_mode_with_globally_loaded_library(
+ self,
+ bridge_test_library: str, # noqa: ARG002
+ ) -> None:
+ """Bridge runs a pipe found in the active library when no crate is provided."""
+ result = await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code=PIPE_REF,
+ inputs={"input_text": "hello world"},
+ execution_mode=PipelexExecutionMode.DIRECT,
+ )
+ )
+
+ assert result.is_completed is True
+ assert result.workflow_id is None
+ assert result.main_stuff_name is not None
+ main_stuff_dump = result.output_dict["root"][result.main_stuff_name]
+ assert main_stuff_dump["content"]["text"] == "hello world"
+ # Usage fields exist on the boundary DTO (None here — this run does no inference
+ # and opens no tracer, so the cost-assembly path produces nothing).
+ assert result.tokens_usages_dump is None
+ assert result.usage_assembly_error is None
+
+ async def test_direct_mode_with_library_crate_dump(
+ self,
+ bridge_test_library: str,
+ ) -> None:
+ """Bridge round-trips through ``library_crate_dump`` end-to-end.
+
+ Captures a LibraryCrate from the loaded library, pipes it through the
+ bridge as a JSON-safe dict, and verifies the pipe still resolves and
+ runs against the per-call scoped library that the bridge opens.
+ """
+ crate = get_library_manager().get_crate(library_id=bridge_test_library)
+ assert crate is not None
+ crate_dump: dict[str, Any] = crate.model_dump(mode="json")
+
+ result = await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code=PIPE_REF,
+ inputs={"input_text": "via crate"},
+ library_crate_dump=crate_dump,
+ execution_mode=PipelexExecutionMode.DIRECT,
+ )
+ )
+
+ assert result.is_completed is True
+ assert result.main_stuff_name is not None
+ main_stuff_dump = result.output_dict["root"][result.main_stuff_name]
+ assert main_stuff_dump["content"]["text"] == "via crate"
+
+ async def test_direct_mode_uses_caller_pipeline_run_id(
+ self,
+ bridge_test_library: str, # noqa: ARG002
+ ) -> None:
+ """Caller-supplied ``pipeline_run_id`` propagates to the PipeJob."""
+ caller_run_id = "caller-supplied-run-id"
+ result = await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code=PIPE_REF,
+ inputs={"input_text": "trace me"},
+ pipeline_run_id=caller_run_id,
+ execution_mode=PipelexExecutionMode.DIRECT,
+ )
+ )
+
+ assert result.is_completed is True
+ assert result.pipeline_run_id == caller_run_id
+
+ async def test_direct_mode_dynamic_concept_round_trips_via_library_crate_dump(
+ self,
+ bridge_test_library: str,
+ ) -> None:
+ """A concept with an inline structure round-trips through ``library_crate_dump``.
+
+ ``EchoEnvelope`` is defined inline in the bridge_test bundle. The bridge
+ dehydrates the library to a JSON-safe crate dump, opens a per-call
+ scoped library on the receiving side, and re-hydrates the concept so
+ ``PipeCompose`` can construct a ``StructuredContent`` matching the
+ dynamic shape.
+ """
+ envelope_pipe_ref = "mistralai_workflows_bridge_test.bridge_envelope_pipe"
+ crate = get_library_manager().get_crate(library_id=bridge_test_library)
+ assert crate is not None
+ crate_dump: dict[str, Any] = crate.model_dump(mode="json")
+
+ result = await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code=envelope_pipe_ref,
+ inputs={"input_text": "wrapped"},
+ library_crate_dump=crate_dump,
+ execution_mode=PipelexExecutionMode.DIRECT,
+ )
+ )
+
+ assert result.is_completed is True
+ assert result.main_stuff_name is not None
+ main_stuff = result.output_dict["root"][result.main_stuff_name]
+ content = main_stuff["content"]
+ assert content["text"] == "wrapped"
+ assert content["origin"] == "mistralai_workflows_bridge"
+
+ async def test_crate_run_restores_prior_current_library(
+ self,
+ bridge_test_library: str,
+ ) -> None:
+ """A crate-scoped bridge run restores the caller's current library on exit.
+
+ The bridge opens a per-call scoped library for the crate. When it
+ returns, the previously active library (the fixture's) must be restored,
+ not cleared — otherwise a later no-crate bridge call in the same async
+ context could no longer resolve pipes.
+ """
+ assert get_current_library_id_or_none() == bridge_test_library
+
+ crate = get_library_manager().get_crate(library_id=bridge_test_library)
+ assert crate is not None
+ crate_dump: dict[str, Any] = crate.model_dump(mode="json")
+
+ await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code=PIPE_REF,
+ inputs={"input_text": "restore me"},
+ library_crate_dump=crate_dump,
+ execution_mode=PipelexExecutionMode.DIRECT,
+ )
+ )
+
+ assert get_current_library_id_or_none() == bridge_test_library
+
+ async def test_run_id_derives_from_trace_context_when_pipeline_run_id_omitted(
+ self,
+ bridge_test_library: str, # noqa: ARG002
+ ) -> None:
+ """With a trace_context but no pipeline_run_id, the run id adopts the lookup_key.
+
+ The caller's tracer is registered (and later closed) under trace_context.lookup_key
+ (tracer_key or graph_id), and tracing assembly reads events by job_metadata.pipeline_run_id.
+ Minting a fresh id would split the two — dropping the run's graph + cost data and leaking
+ the tracer — so the bridge adopts the lookup_key. On every reachable path lookup_key ==
+ graph_id (keyed tracers are a Temporal-child construct that never reaches the bridge — see
+ wip/runtime-bridge/bridge-keyed-tracer-unsupported.md). An explicit pipeline_run_id still
+ wins; absent both, a fresh id is minted.
+ """
+ data_inclusion = DataInclusionConfig(
+ stuff_json_content=False,
+ stuff_text_content=False,
+ stuff_html_content=False,
+ error_stack_traces=False,
+ pipe_and_concept_registry=False,
+ )
+ trace_context = TraceContext(graph_id="host-graph-id", data_inclusion=data_inclusion)
+
+ # No tracer_key: lookup_key == graph_id.
+ job_from_trace = build_pipe_job_from_input(
+ input_payload=PipelexPipeRunInput(pipe_code=PIPE_REF, inputs={"input_text": "x"}),
+ library_crate=None,
+ trace_context=trace_context,
+ )
+ assert job_from_trace.job_metadata.pipeline_run_id == "host-graph-id"
+
+ # tracer_key set: run id is the lookup_key (tracer_key), so tracer open/close stay aligned.
+ keyed_trace_context = TraceContext(graph_id="host-graph-id", tracer_key="host-tracer-key", data_inclusion=data_inclusion)
+ job_from_tracer_key = build_pipe_job_from_input(
+ input_payload=PipelexPipeRunInput(pipe_code=PIPE_REF, inputs={"input_text": "x"}),
+ library_crate=None,
+ trace_context=keyed_trace_context,
+ )
+ assert job_from_tracer_key.job_metadata.pipeline_run_id == "host-tracer-key"
+
+ job_explicit = build_pipe_job_from_input(
+ input_payload=PipelexPipeRunInput(pipe_code=PIPE_REF, inputs={"input_text": "x"}, pipeline_run_id="explicit-id"),
+ library_crate=None,
+ trace_context=trace_context,
+ )
+ assert job_explicit.job_metadata.pipeline_run_id == "explicit-id"
+
+ job_minted = build_pipe_job_from_input(
+ input_payload=PipelexPipeRunInput(pipe_code=PIPE_REF, inputs={"input_text": "x"}),
+ library_crate=None,
+ trace_context=None,
+ )
+ assert job_minted.job_metadata.pipeline_run_id
+ assert job_minted.job_metadata.pipeline_run_id != "host-graph-id"
+
+ async def test_direct_mode_wraps_user_code_exception_in_dispatch_error(
+ self,
+ bridge_test_library: str, # noqa: ARG002
+ ) -> None:
+ """A non-Pipelex exception from user pipe code surfaces as PipelexBridgeDispatchError.
+
+ The PipeFunc operator wraps the user function's ValueError into a
+ PipeRunError, which the bridge converts to its boundary error type — so a
+ raw ValueError never escapes the bridge across the host-runtime seam.
+ """
+ with pytest.raises(PipelexBridgeDispatchError):
+ await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code=RAISE_PIPE_REF,
+ inputs={"input_text": "boom"},
+ execution_mode=PipelexExecutionMode.DIRECT,
+ )
+ )
+
+ async def test_crate_inline_concept_does_not_leak_into_global_registry(
+ self,
+ bridge_test_library: str,
+ ) -> None:
+ """Classes generated from a crate's inline concepts stay out of the global registry.
+
+ The bridge loads the crate into a per-call scoped ``ClassRegistry`` so a
+ long-lived / multi-tenant host doesn't accumulate (or collide on) dynamic
+ concept classes across crate versions. We rename the bundle's inline
+ ``EchoEnvelope`` concept to a probe name that has never been loaded
+ globally, so any presence in the global registry afterwards is
+ unambiguously a leak from the bridge's crate path.
+ """
+ crate = get_library_manager().get_crate(library_id=bridge_test_library)
+ assert crate is not None
+ crate_dump: dict[str, Any] = crate.model_dump(mode="json")
+ probe_dump: dict[str, Any] = json.loads(json.dumps(crate_dump).replace("EchoEnvelope", "EchoEnvelopeProbe"))
+
+ global_registry = KajsonManager.get_class_registry()
+ before = set(global_registry.get_classes_dict().keys())
+ assert not any("EchoEnvelopeProbe" in name for name in before), "Probe class must not be globally loaded before the run"
+
+ result = await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code=ENVELOPE_PIPE_REF,
+ inputs={"input_text": "probe"},
+ library_crate_dump=probe_dump,
+ execution_mode=PipelexExecutionMode.DIRECT,
+ )
+ )
+
+ assert result.is_completed is True
+ after = set(global_registry.get_classes_dict().keys())
+ leaked = {name for name in (after - before) if "EchoEnvelopeProbe" in name}
+ assert not leaked, f"Crate-generated class(es) leaked into the global Kajson registry: {leaked}"
diff --git a/tests/integration/pipelex/runtime_bridge/test_data/bridge_funcs.py b/tests/integration/pipelex/runtime_bridge/test_data/bridge_funcs.py
new file mode 100644
index 000000000..492caebde
--- /dev/null
+++ b/tests/integration/pipelex/runtime_bridge/test_data/bridge_funcs.py
@@ -0,0 +1,25 @@
+"""Test functions registered for the runtime-bridge integration tests."""
+
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.stuffs.text_content import TextContent
+
+
+def mistralai_workflows_bridge_echo(working_memory: WorkingMemory) -> TextContent:
+ """Echo the ``input_text`` stuff back as a TextContent output.
+
+ Used by tests/integration/pipelex/runtime_bridge to validate end-to-end
+ pipe execution through the bridge without invoking inference.
+ """
+ input_text = working_memory.get_stuff_as_str("input_text")
+ return TextContent(text=input_text)
+
+
+def mistralai_workflows_bridge_raise(working_memory: WorkingMemory) -> TextContent: # noqa: ARG001
+ """Raise a plain ``ValueError`` to exercise the bridge's error boundary.
+
+ The PipeFunc operator wraps any exception from user code into a
+ ``PipeRunError``, which the bridge converts to ``PipelexBridgeDispatchError`` —
+ so a raw ``ValueError`` never escapes the bridge across the host-runtime seam.
+ """
+ msg = "intentional failure from bridge test func"
+ raise ValueError(msg)
diff --git a/tests/integration/pipelex/runtime_bridge/test_data/bridge_test.mthds b/tests/integration/pipelex/runtime_bridge/test_data/bridge_test.mthds
new file mode 100644
index 000000000..28d390eb1
--- /dev/null
+++ b/tests/integration/pipelex/runtime_bridge/test_data/bridge_test.mthds
@@ -0,0 +1,62 @@
+domain = "mistralai_workflows_bridge_test"
+description = "Test pipes for the mistralai_workflows plugin bridge"
+
+[concept.EchoEnvelope]
+description = "Custom dynamic concept used to exercise library_crate_dump round-trip for inline-structured concepts."
+
+[concept.EchoEnvelope.structure]
+text = { type = "text", required = true, description = "The echoed text" }
+origin = { type = "text", required = true, description = "Origin marker for the echo" }
+
+[pipe.bridge_func_pipe]
+type = "PipeFunc"
+description = "Echoes the input text back as output (DIRECT mode only — PipeFunc is not Temporal-compatible)"
+output = "Text"
+function_name = "mistralai_workflows_bridge_echo"
+
+[pipe.bridge_raise_pipe]
+type = "PipeFunc"
+description = "Raises a plain ValueError to exercise the bridge error boundary (DIRECT mode only)"
+output = "Text"
+function_name = "mistralai_workflows_bridge_raise"
+
+[pipe.bridge_compose_pipe]
+type = "PipeCompose"
+description = "Echoes the input_text via a Jinja2 template (Temporal-compatible)"
+inputs = { input_text = "Text" }
+output = "Text"
+template = "{{ input_text.text }}"
+
+[pipe.bridge_envelope_pipe]
+type = "PipeCompose"
+description = "Composes the input_text into a structured EchoEnvelope, exercising dynamic-concept round-trip via library_crate_dump"
+inputs = { input_text = "Text" }
+output = "EchoEnvelope"
+
+[pipe.bridge_envelope_pipe.construct]
+text = { from = "input_text.text" }
+origin = "mistralai_workflows_bridge"
+
+[pipe.bridge_seq_step_one]
+type = "PipeCompose"
+description = "First step of bridge_sequence_pipe — uppercases the input text"
+inputs = { input_text = "Text" }
+output = "Text"
+template = "{{ input_text.text | upper }}"
+
+[pipe.bridge_seq_step_two]
+type = "PipeCompose"
+description = "Second step of bridge_sequence_pipe — wraps the result with markers"
+inputs = { uppercased = "Text" }
+output = "Text"
+template = "[STEP2:{{ uppercased.text }}]"
+
+[pipe.bridge_sequence_pipe]
+type = "PipeSequence"
+description = "Two-step sequence used by Phase 2.1 streaming tests to assert per-step events"
+inputs = { input_text = "Text" }
+output = "Text"
+steps = [
+ { pipe = "bridge_seq_step_one", result = "uppercased" },
+ { pipe = "bridge_seq_step_two", result = "final_text" },
+]
diff --git a/tests/integration/pipelex/temporal/library_crate/helpers.py b/tests/integration/pipelex/temporal/library_crate/helpers.py
index cb4e698e4..5a7347b9a 100644
--- a/tests/integration/pipelex/temporal/library_crate/helpers.py
+++ b/tests/integration/pipelex/temporal/library_crate/helpers.py
@@ -9,7 +9,7 @@ from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.core.stuffs.text_content import TextContent
from pipelex.pipe_run.pipe_job import PipeJob
-from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
+from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate
from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
diff --git a/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.py b/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.py
index b92a37b72..16636579e 100644
--- a/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.py
+++ b/tests/integration/pipelex/temporal/library_crate/test_submitter_without_bundle.py
@@ -21,9 +21,9 @@ from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.hub import clear_current_library, get_current_library, set_current_library
from pipelex.pipe_run.exceptions import PipeJobError
from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.runtime_bridge.primitives.hydration import hydrate_working_memory
+from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate
from pipelex.temporal.temporal_hub import get_task_manager
-from pipelex.temporal.tprl_pipe.hydration import hydrate_working_memory
-from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
from tests.integration.pipelex.temporal.test_data import DeferredHydrationTestData
diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py b/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py
index 2b88b92c5..631480c4c 100644
--- a/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py
+++ b/tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py
@@ -28,9 +28,9 @@ from temporalio.worker import UnsandboxedWorkflowRunner, Worker
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus
from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg
from pipelex.temporal.exceptions import WorkflowExecutionError
from pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg
-from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg
from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun
from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
from tests.integration.pipelex.temporal.test_data import LibraryCrateTestData
diff --git a/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.py b/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.py
index 46f382068..d2d2b7a98 100644
--- a/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.py
+++ b/tests/integration/pipelex/temporal/test_wf_pipe_run_request_id_logging.py
@@ -28,9 +28,9 @@ from typing_extensions import override
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg
from pipelex.temporal.exceptions import WorkflowExecutionError
from pipelex.temporal.tprl_pipe.act_deliver import DeliveryActivityArg
-from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg
from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun
from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
from tests.integration.pipelex.temporal.test_data import LibraryCrateTestData
diff --git a/tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py b/tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py
index 5ddbc22f9..ad3f22bf3 100644
--- a/tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py
+++ b/tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py
@@ -30,10 +30,10 @@ from pipelex.config import get_config
from pipelex.pipe_run.delivery_assignment import DeliveryAssignment, DeliveryStatus, WebhookTarget
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg
from pipelex.temporal.exceptions import WorkflowExecutionError
from pipelex.temporal.temporal_hub import get_task_manager
from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor
-from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg
from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun
from tests.integration.pipelex.error_handling.test_data import ErrorReportParityTestData
diff --git a/tests/integration/pipelex/temporal/tracing/helpers.py b/tests/integration/pipelex/temporal/tracing/helpers.py
index 1061cec10..eb5dca3cd 100644
--- a/tests/integration/pipelex/temporal/tracing/helpers.py
+++ b/tests/integration/pipelex/temporal/tracing/helpers.py
@@ -2,7 +2,7 @@
import uuid
from collections import defaultdict
-from collections.abc import AsyncGenerator, Generator, Iterable
+from collections.abc import AsyncGenerator, Awaitable, Callable, Generator, Iterable
from contextlib import asynccontextmanager, contextmanager
from datetime import datetime
from pathlib import Path
@@ -251,6 +251,7 @@ async def make_split_workers(
temporal_client: TemporalClient,
q_router: str,
q_runner: str,
+ runner_act_llm_gen_text: Callable[[LLMAssignment], Awaitable[str]] = _runner_isolated_act_llm_gen_text,
) -> AsyncGenerator[None, None]:
"""Open two scoped workers on two task queues in the current process.
@@ -260,9 +261,12 @@ async def make_split_workers(
`task_queue` argument — the activity lands on the workflow's own queue
and would never be picked up if the router registered no activities.
- `q_runner`: activity-only (runner scope, `disable_all_workflows=True`),
- with `act_llm_gen_text` substituted by the isolation wrapper that clears
- the in-process `_event_log_contexts` cache so the runner cannot
- accidentally use the router's registered context.
+ with `act_llm_gen_text` substituted by ``runner_act_llm_gen_text``. The
+ default isolation wrapper clears the in-process `_event_log_contexts`
+ cache (so the runner cannot accidentally use the router's registered
+ context) and synthesizes usage instead of calling a real LLM. Pass a
+ different substitute — e.g. one that clears the cache and then runs real
+ inference — to exercise the fallback with real provider token counts.
Pair this with `worker_config.activity_queues[act_llm_gen_text.__name__] =
ActivityRouteConfig(default=q_runner, by_handle={})` so the workflow on
@@ -308,7 +312,7 @@ async def make_split_workers(
task_queue=q_runner,
is_not_sandboxed=True,
scope=runner_scope,
- substitute_activities={act_llm_gen_text: _runner_isolated_act_llm_gen_text},
+ substitute_activities={act_llm_gen_text: runner_act_llm_gen_text},
),
):
yield
diff --git a/tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.py b/tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.py
index 0fb69872c..558530492 100644
--- a/tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.py
+++ b/tests/integration/pipelex/temporal/tracing/test_mock_inference_temporal.py
@@ -26,8 +26,8 @@ from pipelex.cogt.content_generation.dry_mock import MOCK_INFERENCE_MODEL_NAME
from pipelex.cogt.usage.cost_registry import CostRegistry
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg
from pipelex.temporal.temporal_hub import get_task_manager
-from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg
from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun
from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
diff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py
new file mode 100644
index 000000000..26f4c8e03
--- /dev/null
+++ b/tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py
@@ -0,0 +1,155 @@
+"""Cross-child cost aggregation in split-worker mode.
+
+The fan-out join is by ``pipeline_run_id`` on a shared partition: each child
+workflow gets a distinct ``workflow_id`` but the same ``pipeline_run_id``, and
+only the top-level run reads the whole partition. No automated test asserts that
+usage emitted from *separate child workflows* sums into one parent total — the
+existing fan-out tracing tests are graph-only/dry-run, and the sequence
+split-worker tests have no child workflows.
+
+This runs the PipeParallel bundle (two branches dispatched as separate child
+workflows) in LIVE mode with the runner-side fallback, then asserts the usage
+events from multiple workflows aggregate into a single priced total.
+"""
+
+import uuid
+from collections.abc import Generator
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+import pytest
+from temporalio.client import Client as TemporalClient
+
+from pipelex.cogt.usage.cost_registry import CostRegistry
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.config import get_config
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.config_temporal import ActivityRouteConfig
+from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from pipelex.tracing.activity_event_log import ActivityEventLogCache
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from pipelex.tracing.trace_events import UsageReportEvent
+from pipelex.tracing.usage_aggregator import UsageAggregator
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.tracing.helpers import (
+ inject_trace_context,
+ make_split_workers,
+ ndjson_files_for_run,
+)
+from tests.integration.pipelex.temporal.tracing.test_data import ParallelTracingTestData
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+
+
[email protected](loop_scope="class")
+class TestSplitWorkerCrossChildUsage:
+ """Usage from fan-out child workflows aggregates into one parent cost total."""
+
+ @pytest.fixture
+ def split_queues(self) -> tuple[str, str]:
+ return (f"q_router_{uuid.uuid4().hex[:8]}", f"q_runner_{uuid.uuid4().hex[:8]}")
+
+ @pytest.fixture
+ def live_parallel_tracing_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ """The PipeParallel bundle in LIVE mode so branches dispatch ``act_llm_gen_text``.
+
+ LIVE forces the in-workflow activity hop (the runner-side fallback under test); the fake
+ ``act_llm_gen_text`` installed by ``make_split_workers`` returns a stub string and
+ synthesizes usage, so no real LLM call is made.
+ """
+ yield from pipe_job_from_bundle(
+ bundle_file=ParallelTracingTestData.BUNDLE_FILE,
+ pipe_code=ParallelTracingTestData.PIPE_CODE,
+ pipe_run_mode=PipeRunMode.LIVE,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+ @pytest.fixture
+ def route_llm_text_to_runner_queue(self, split_queues: tuple[str, str]) -> Generator[None, None, None]:
+ _q_router, q_runner = split_queues
+ worker_config = get_config().temporal.worker_config
+ activity_name = act_llm_gen_text.__name__
+ original_entry = worker_config.activity_queues.get(activity_name)
+ worker_config.activity_queues[activity_name] = ActivityRouteConfig(default=q_runner, by_handle={})
+ yield
+ if original_entry is None:
+ worker_config.activity_queues.pop(activity_name, None)
+ else:
+ worker_config.activity_queues[activity_name] = original_entry
+
+ @pytest.fixture(autouse=True)
+ def reset_activity_event_log(self) -> Generator[None, None, None]:
+ ActivityEventLogCache.reset_for_tests()
+ yield
+ ActivityEventLogCache.reset_for_tests()
+
+ async def _execute_split(
+ self,
+ live_parallel_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ split_queues: tuple[str, str],
+ ) -> str:
+ q_router, q_runner = split_queues
+ execution_run_id = f"split_child_{uuid.uuid4().hex[:12]}"
+ execution_job = inject_trace_context(live_parallel_tracing_job, execution_run_id)
+ workflow_id = f"wf_{uuid.uuid4().hex[:12]}"
+
+ async with make_split_workers(temporal_client, q_router=q_router, q_runner=q_runner):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=execution_job,
+ id=workflow_id,
+ task_queue=q_router,
+ )
+ rehydrate_pipe_output(pipe_output)
+ return execution_run_id
+
+ async def test_cross_child_usage_aggregates_to_single_parent_total(
+ self,
+ live_parallel_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ split_queues: tuple[str, str],
+ route_llm_text_to_runner_queue: None, # noqa: ARG002
+ ) -> None:
+ """Two branch child workflows + the parent's summarize step each emit usage via the runner
+ fallback into the shared ``pipeline_run_id`` partition. Reading the partition once aggregates
+ usage across all of them into a single priced total — proving the cross-child fan-out join.
+ """
+ run_id = await self._execute_split(live_parallel_tracing_job, temporal_client, split_queues)
+
+ # Multiple NDJSON files confirm the branches ran as separate child workflows (parent + children).
+ ndjson_files = ndjson_files_for_run(str(tracing_tmp_dir), run_id)
+ assert len(ndjson_files) >= 2, f"Expected parent + child-workflow NDJSON files, got {[path.name for path in ndjson_files]}"
+
+ reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))
+ try:
+ events = reader.read_events(run_id)
+ finally:
+ reader.close()
+
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert usage_events, "Expected usage events from the fan-out run"
+
+ # The branches are distinct nodes living in distinct child workflows; their usage must land in
+ # the shared partition under more than one workflow_id.
+ distinct_workflow_ids = {evt.workflow_id for evt in usage_events}
+ distinct_node_ids = {evt.node_id for evt in usage_events}
+ assert len(distinct_workflow_ids) >= 2, f"Expected usage across >=2 workflows (cross-child), got {distinct_workflow_ids}"
+ assert len(distinct_node_ids) >= 2, f"Expected usage from >=2 distinct nodes, got {distinct_node_ids}"
+
+ tokens_usages = UsageAggregator.aggregate(events)
+ expected_total = sum(
+ tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT, 0) + tokens_usage.nb_tokens_by_category.get(TokenCategory.OUTPUT, 0)
+ for tokens_usage in tokens_usages
+ )
+
+ aggregated = CostRegistry.aggregate_costs(tokens_usages=tokens_usages)
+ assert aggregated.total_nb_tokens == expected_total
+ assert aggregated.total_nb_tokens >= 2 * len(usage_events), "Each runner emit carries at least one input + one output token"
+ assert aggregated.has_reportable_usage is True
diff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py
new file mode 100644
index 000000000..fdf2d2b15
--- /dev/null
+++ b/tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py
@@ -0,0 +1,152 @@
+"""Gated real-inference cost test for split-worker (distributed) mode.
+
+Every other split-worker usage test substitutes ``act_llm_gen_text`` with a
+helper that hand-builds usage, so none of them prove that a *real* provider
+response is captured, emitted through the runner fallback, and aggregated into a
+non-zero cost total. Real distributed cost is otherwise validated only by manual
+eyeballing. This test closes that blind spot end-to-end.
+
+It is marked ``inference``/``llm`` so the default (no-spend) lanes skip it; run
+it explicitly when validating real-inference cost capture in distributed mode.
+The runner activity clears the in-process event-log context cache (cold-runner
+simulation) and then runs the genuine ``llm_gen_text`` — so the real LLM call
+happens on the runner queue and its usage report takes the fallback path, just
+as a physically separate runner process would.
+"""
+
+import uuid
+from collections.abc import Generator
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+import pytest
+from temporalio import activity
+from temporalio.client import Client as TemporalClient
+
+from pipelex.cogt.content_generation.assignment_models import LLMAssignment
+from pipelex.cogt.content_generation.llm_generate import llm_gen_text
+from pipelex.cogt.usage.cost_registry import CostRegistry
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.config import get_config
+from pipelex.hub import get_report_delegate
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.reporting.reporting_manager import ReportingManager
+from pipelex.temporal.config_temporal import ActivityRouteConfig
+from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_text
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from pipelex.tracing.activity_event_log import ActivityEventLogCache
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from pipelex.tracing.trace_events import UsageReportEvent
+from pipelex.tracing.usage_aggregator import UsageAggregator
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+from tests.integration.pipelex.temporal.tracing.helpers import inject_trace_context, make_split_workers
+from tests.integration.pipelex.temporal.tracing.test_data import SequenceTracingTestData
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+
+
[email protected](name="act_llm_gen_text")
+async def _real_runner_act_llm_gen_text(llm_assignment: LLMAssignment) -> str:
+ """Cold-runner real-inference substitute: clear contexts, then run the genuine LLM call.
+
+ Clearing ``_event_log_contexts`` makes the synchronous usage report take the runner-side
+ fallback (the router's registered context lives in the same process during a single-process
+ test); ``llm_gen_text`` performs the real provider call so the captured token counts are real.
+ """
+ delegate = get_report_delegate()
+ if isinstance(delegate, ReportingManager):
+ delegate._event_log_contexts.clear() # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+ return await llm_gen_text(llm_assignment=llm_assignment)
+
+
[email protected](loop_scope="class")
+class TestSplitWorkerRealInferenceCost:
+ """Real provider usage is captured, emitted via the runner fallback, and aggregated."""
+
+ @pytest.fixture
+ def split_queues(self) -> tuple[str, str]:
+ return (f"q_router_{uuid.uuid4().hex[:8]}", f"q_runner_{uuid.uuid4().hex[:8]}")
+
+ @pytest.fixture
+ def live_sequence_tracing_job(self, is_class_registry_isolated: bool) -> Generator[PipeJob, None, None]:
+ yield from pipe_job_from_bundle(
+ bundle_file=SequenceTracingTestData.BUNDLE_FILE,
+ pipe_code=SequenceTracingTestData.PIPE_CODE,
+ pipe_run_mode=PipeRunMode.LIVE,
+ isolated_registry=is_class_registry_isolated,
+ )
+
+ @pytest.fixture
+ def route_llm_text_to_runner_queue(self, split_queues: tuple[str, str]) -> Generator[None, None, None]:
+ _q_router, q_runner = split_queues
+ worker_config = get_config().temporal.worker_config
+ activity_name = act_llm_gen_text.__name__
+ original_entry = worker_config.activity_queues.get(activity_name)
+ worker_config.activity_queues[activity_name] = ActivityRouteConfig(default=q_runner, by_handle={})
+ yield
+ if original_entry is None:
+ worker_config.activity_queues.pop(activity_name, None)
+ else:
+ worker_config.activity_queues[activity_name] = original_entry
+
+ @pytest.fixture(autouse=True)
+ def reset_activity_event_log(self) -> Generator[None, None, None]:
+ ActivityEventLogCache.reset_for_tests()
+ yield
+ ActivityEventLogCache.reset_for_tests()
+
+ async def test_real_provider_usage_aggregates_to_nonzero_cost(
+ self,
+ live_sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ split_queues: tuple[str, str],
+ route_llm_text_to_runner_queue: None, # noqa: ARG002
+ ) -> None:
+ q_router, q_runner = split_queues
+ execution_run_id = f"split_real_{uuid.uuid4().hex[:12]}"
+ execution_job = inject_trace_context(live_sequence_tracing_job, execution_run_id)
+ workflow_id = f"wf_{uuid.uuid4().hex[:12]}"
+
+ async with make_split_workers(
+ temporal_client,
+ q_router=q_router,
+ q_runner=q_runner,
+ runner_act_llm_gen_text=_real_runner_act_llm_gen_text,
+ ):
+ pipe_output: PipeOutput = await temporal_client.execute_workflow( # pyright: ignore[reportUnknownMemberType]
+ workflow=WfPipeRouter.run,
+ arg=execution_job,
+ id=workflow_id,
+ task_queue=q_router,
+ )
+ rehydrate_pipe_output(pipe_output)
+
+ reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))
+ try:
+ events = reader.read_events(execution_run_id)
+ finally:
+ reader.close()
+
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ # The sequence bundle makes one real LLM call per step (step_one → step_two).
+ assert len(usage_events) == 2, f"Expected one usage event per inference call, got {len(usage_events)}"
+ assert all(evt.writer_id.startswith("act_") for evt in usage_events), "Runner fallback must have stamped act_* writer ids"
+
+ tokens_usages = UsageAggregator.aggregate(events)
+ assert all(
+ tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT, 0) > 0 and tokens_usage.nb_tokens_by_category.get(TokenCategory.OUTPUT, 0) > 0
+ for tokens_usage in tokens_usages
+ ), "Real provider responses must carry non-zero input and output token counts"
+ model_names = {tokens_usage.inference_model_name for tokens_usage in tokens_usages}
+ assert all(model_names), "Every usage record must name its real model handle"
+
+ aggregated = CostRegistry.aggregate_costs(tokens_usages=tokens_usages)
+ assert aggregated.total_nb_tokens > 0
+ assert aggregated.has_reportable_usage is True
diff --git a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py
index 826f8a8c2..e9fe34235 100644
--- a/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py
+++ b/tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py
@@ -24,6 +24,8 @@ from typing import TYPE_CHECKING
import pytest
from temporalio.client import Client as TemporalClient
+from pipelex.cogt.usage.cost_registry import CostRegistry
+from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.config import get_config
from pipelex.pipe_run.pipe_job import PipeJob
from pipelex.pipe_run.pipe_run_mode import PipeRunMode
@@ -230,3 +232,43 @@ class TestSplitWorkerUsageEmission:
assert tokens_usages, "Expected the aggregator to surface at least one token-usage record"
assert len(tokens_usages) == len(usage_events)
assert tokens_usages == [evt.tokens_usage for evt in usage_events]
+
+ async def test_runner_usage_sums_to_expected_cost_total(
+ self,
+ live_sequence_tracing_job: PipeJob,
+ temporal_client: TemporalClient,
+ tracing_tmp_dir: Path,
+ split_queues: tuple[str, str],
+ route_llm_text_to_runner_queue: None, # noqa: ARG002
+ ) -> None:
+ """The cross-worker usage events feed the production ``CostRegistry`` aggregation and price
+ into a single run total. This pins the *summed total* in split-worker mode — the previous
+ tests assert only landing / per-node de-dup / 1:1 passthrough, never that the numbers add up.
+
+ ``total_nb_tokens`` is computed independently from each runner-emitted event's raw token
+ counts and must equal what ``aggregate_costs`` reports. The fake runner model is priced at
+ zero, so ``total_cost`` is 0 while the run still counts as reportable (tokens > 0) — the same
+ free-model path a real zero-cost model would take.
+ """
+ run_id = await self._execute_split(live_sequence_tracing_job, temporal_client, split_queues)
+
+ reader = NdjsonEventLog(traces_dir=str(tracing_tmp_dir))
+ try:
+ events = reader.read_events(run_id)
+ finally:
+ reader.close()
+
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert usage_events, "Expected at least one UsageReportEvent across the run"
+
+ tokens_usages = UsageAggregator.aggregate(events)
+ expected_total = sum(
+ tokens_usage.nb_tokens_by_category.get(TokenCategory.INPUT, 0) + tokens_usage.nb_tokens_by_category.get(TokenCategory.OUTPUT, 0)
+ for tokens_usage in tokens_usages
+ )
+
+ aggregated = CostRegistry.aggregate_costs(tokens_usages=tokens_usages)
+ assert aggregated.total_nb_tokens == expected_total
+ assert expected_total >= len(usage_events), "Each runner emit carries at least one input + one output token"
+ assert aggregated.total_cost == 0.0
+ assert aggregated.has_reportable_usage is True
diff --git a/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.py b/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.py
index 6d040a779..50a966c09 100644
--- a/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.py
+++ b/tests/integration/pipelex/temporal/tracing/test_wf_pipe_run_tracing_assembly.py
@@ -28,8 +28,8 @@ from temporalio.client import Client as TemporalClient
from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.runtime_bridge.primitives.pipe_run_arg import PipeRunArg
from pipelex.temporal.temporal_hub import get_task_manager
-from pipelex.temporal.tprl_pipe.pipe_run_arg import PipeRunArg
from pipelex.temporal.tprl_pipe.wf_pipe_run import WfPipeRun
from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
from tests.integration.pipelex.temporal.tracing.helpers import inject_trace_context
diff --git a/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py b/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py
new file mode 100644
index 000000000..518a3a123
--- /dev/null
+++ b/tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py
@@ -0,0 +1,99 @@
+"""Capture-stage tests for the extract token-usage path.
+
+Extraction usage is either set by the provider inside ``_extract_pages`` or, when
+the provider reports none, filled by the page-count fallback in
+``ExtractWorkerAbstract.extract_pages`` (one "page" priced as 1,000,000 input +
+1,000,000 output tokens). Neither branch is asserted anywhere today — extract
+usage is uncovered in every mode. These tests pin both directions with a fake
+in-process worker, no provider SDK required.
+"""
+
+from __future__ import annotations
+
+import pytest
+from typing_extensions import override
+
+from pipelex.cogt.extract.extract_input import ExtractInput
+from pipelex.cogt.extract.extract_job import ExtractJob
+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams, ExtractJobReport
+from pipelex.cogt.extract.extract_output import ExtractOutput, Page
+from pipelex.cogt.extract.extract_worker_abstract import ExtractWorkerAbstract
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import NbTokensByCategoryDict, TokenCategory
+from pipelex.pipeline.job_metadata import JobMetadata
+
+_PER_PAGE_TOKENS = 1_000_000
+
+
+class _FakeExtractWorker(ExtractWorkerAbstract):
+ """Extract worker whose ``_extract_pages`` returns canned pages and optionally sets usage."""
+
+ def __init__(self, inference_model: InferenceModelSpec, nb_pages: int, usage_to_set: NbTokensByCategoryDict | None) -> None:
+ super().__init__(extra_config={}, inference_model=inference_model)
+ self._nb_pages = nb_pages
+ self._usage_to_set = usage_to_set
+
+ @override
+ async def _extract_pages(self, extract_job: ExtractJob) -> ExtractOutput:
+ if self._usage_to_set is not None and (usage := extract_job.job_report.extract_tokens_usage):
+ usage.nb_tokens_by_category = self._usage_to_set
+ return ExtractOutput(pages={index: Page(text="page text") for index in range(self._nb_pages)})
+
+
+def _make_extract_model() -> InferenceModelSpec:
+ return InferenceModelSpec(
+ backend_name="test",
+ name="test-extract",
+ sdk="test_extract",
+ model_type=ModelType.TEXT_EXTRACTOR,
+ model_id="test-extract-id",
+ inputs=["pdf"],
+ outputs=["text"],
+ costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 1.0},
+ thinking_mode=ThinkingMode.NONE,
+ max_tokens=None,
+ max_prompt_images=None,
+ )
+
+
+def _make_extract_job() -> ExtractJob:
+ return ExtractJob(
+ extract_input=ExtractInput(document_uri="/tmp/test.pdf"), # noqa: S108
+ job_params=ExtractJobParams.make_default_extract_job_params(),
+ job_config=ExtractJobConfig(),
+ job_report=ExtractJobReport(),
+ job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),
+ )
+
+
+class TestExtractWorkerUsageCapture:
+ """Extract usage is provider-set when reported, else filled by the page-count fallback."""
+
+ @pytest.mark.asyncio
+ async def test_page_count_fallback_fills_usage_when_provider_reports_none(self) -> None:
+ worker = _FakeExtractWorker(_make_extract_model(), nb_pages=3, usage_to_set=None)
+ job = _make_extract_job()
+
+ await worker.extract_pages(extract_job=job)
+
+ captured = job.job_report.extract_tokens_usage
+ assert captured is not None
+ assert captured.nb_tokens_by_category == {
+ TokenCategory.INPUT: 3 * _PER_PAGE_TOKENS,
+ TokenCategory.OUTPUT: 3 * _PER_PAGE_TOKENS,
+ }
+
+ @pytest.mark.asyncio
+ async def test_provider_reported_usage_is_not_overridden_by_fallback(self) -> None:
+ provider_usage: NbTokensByCategoryDict = {TokenCategory.INPUT: 42, TokenCategory.OUTPUT: 7}
+ worker = _FakeExtractWorker(_make_extract_model(), nb_pages=3, usage_to_set=provider_usage)
+ job = _make_extract_job()
+
+ await worker.extract_pages(extract_job=job)
+
+ captured = job.job_report.extract_tokens_usage
+ assert captured is not None
+ assert captured.nb_tokens_by_category == provider_usage
diff --git a/tests/unit/pipelex/pipe_run/test_scoped_pipe_router.py b/tests/unit/pipelex/pipe_run/test_scoped_pipe_router.py
new file mode 100644
index 000000000..792c2d59d
--- /dev/null
+++ b/tests/unit/pipelex/pipe_run/test_scoped_pipe_router.py
@@ -0,0 +1,51 @@
+import pytest
+
+from pipelex.hub import (
+ get_pipe_router,
+ scoped_pipe_router,
+ set_pipe_router,
+ teardown_current_pipe_router,
+)
+from pipelex.pipe_run.pipe_router import PipeRouter
+
+
+class TestScopedPipeRouter:
+ def test_sets_router_inside_block(self) -> None:
+ scoped = PipeRouter()
+ with scoped_pipe_router(scoped):
+ assert get_pipe_router() is scoped
+
+ def test_restores_prior_override_not_none_on_exit(self) -> None:
+ # The key improvement over the raw teardown_current_pipe_router(), which
+ # unconditionally resets the override to None: scoped_pipe_router must
+ # restore whatever override was active before the block.
+ base = PipeRouter()
+ set_pipe_router(base)
+ try:
+ scoped = PipeRouter()
+ with scoped_pipe_router(scoped):
+ assert get_pipe_router() is scoped
+ assert get_pipe_router() is base
+ finally:
+ teardown_current_pipe_router()
+
+ def test_nested_scope_restores_outer_override_on_inner_exit(self) -> None:
+ outer = PipeRouter()
+ inner = PipeRouter()
+ with scoped_pipe_router(outer):
+ assert get_pipe_router() is outer
+ with scoped_pipe_router(inner):
+ assert get_pipe_router() is inner
+ assert get_pipe_router() is outer
+
+ def test_restores_prior_override_even_when_block_raises(self) -> None:
+ base = PipeRouter()
+ set_pipe_router(base)
+ try:
+ scoped = PipeRouter()
+ msg = "boom"
+ with pytest.raises(RuntimeError, match="boom"), scoped_pipe_router(scoped):
+ raise RuntimeError(msg)
+ assert get_pipe_router() is base
+ finally:
+ teardown_current_pipe_router()
diff --git a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py b/tests/unit/pipelex/pipe_run/test_tracing_assembly.py
index 56ec40255..d009927b1 100644
--- a/tests/unit/pipelex/pipe_run/test_tracing_assembly.py
+++ b/tests/unit/pipelex/pipe_run/test_tracing_assembly.py
@@ -13,6 +13,7 @@ from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.pipe_run.tracing_assembly import TracingAssembly, assemble_tracing, assemble_tracing_on_output
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.system.exceptions import MissingDependencyError
+from pipelex.tracing.exceptions import EventLogReadError, EventLogSetupError
from pipelex.tracing.trace_events import UsageReportEvent
_MODULE = "pipelex.pipe_run.tracing_assembly"
@@ -143,30 +144,41 @@ class TestTracingAssembly:
assert result.usage_assembly_error is not None
event_log.close.assert_called_once()
- def test_read_botocore_errors_are_caught(self, mocker: MockerFixture) -> None:
- """DynamoDB backend failures degrade to an assembly error, never failing the run. Covers BOTH botocore
- base classes (siblings, neither a subclass of the other): ClientError (service-side throttle / auth)
- and BotoCoreError subclasses (transport / credential / timeout, e.g. EndpointConnectionError).
+ def test_read_event_log_read_error_is_caught(self, mocker: MockerFixture) -> None:
+ """A store-level read failure surfaces as our domain EventLogReadError and degrades to an assembly
+ error on every requested concern, never failing the run. The DynamoDB backend converts its botocore
+ ClientError / BotoCoreError into EventLogReadError at the store boundary (covered in
+ tests/unit/pipelex/tracing/test_dynamodb_event_log.py), so the assembly layer stays boto-agnostic.
"""
- botocore_exceptions = pytest.importorskip("botocore.exceptions")
- client_error = botocore_exceptions.ClientError(
- error_response={"Error": {"Code": "ProvisionedThroughputExceededException", "Message": "throttled"}},
- operation_name="Query",
- )
- transport_error = botocore_exceptions.EndpointConnectionError(endpoint_url="https://dynamodb.test")
- for backend_error in (client_error, transport_error):
- self._enable_tracing(mocker)
- event_log = mocker.MagicMock()
- event_log.read_events = mocker.MagicMock(side_effect=backend_error)
- mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)
-
- result = assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=True)
-
- assert result.graph_spec is None, backend_error
- assert result.tokens_usages is None, backend_error
- assert result.graph_assembly_error is not None, backend_error
- assert result.usage_assembly_error is not None, backend_error
- event_log.close.assert_called_once()
+ self._enable_tracing(mocker)
+ event_log = mocker.MagicMock()
+ event_log.read_events = mocker.MagicMock(side_effect=EventLogReadError("DynamoDB read failed"))
+ mocker.patch(f"{_MODULE}.make_event_log", return_value=event_log)
+
+ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=True)
+
+ assert result.graph_spec is None
+ assert result.tokens_usages is None
+ assert result.graph_assembly_error is not None
+ assert result.usage_assembly_error is not None
+ event_log.close.assert_called_once()
+
+ def test_make_event_log_setup_error_is_caught(self, mocker: MockerFixture) -> None:
+ """A construction-time backend failure degrades instead of aborting the run.
+
+ When the DynamoDB client cannot even be built (botocore raising inside the constructor, surfaced as
+ our domain EventLogSetupError from make_event_log), the assembly layer catches the EventLogError base
+ — covering both setup and read failures — and records an assembly error on every requested concern.
+ """
+ self._enable_tracing(mocker)
+ mocker.patch(f"{_MODULE}.make_event_log", side_effect=EventLogSetupError("DynamoDB client construction failed"))
+
+ result = assemble_tracing(pipeline_run_id="plr", assemble_graph=True, assemble_usage=True)
+
+ assert result.graph_spec is None
+ assert result.tokens_usages is None
+ assert result.graph_assembly_error is not None
+ assert result.usage_assembly_error is not None
def test_read_error_only_marks_requested_concern(self, mocker: MockerFixture) -> None:
"""A costs-only read failure sets usage_assembly_error but never graph_assembly_error."""
diff --git a/tests/unit/pipelex/plugins/openai/test_openai_completions_usage_capture.py b/tests/unit/pipelex/plugins/openai/test_openai_completions_usage_capture.py
new file mode 100644
index 000000000..c6fbca167
--- /dev/null
+++ b/tests/unit/pipelex/plugins/openai/test_openai_completions_usage_capture.py
@@ -0,0 +1,146 @@
+"""Capture-stage tests for the OpenAI Chat Completions token-usage path.
+
+Cost reporting starts by reading the provider response's token counts into
+``LLMJob.job_report.llm_tokens_usage.nb_tokens_by_category``. Mock inference
+short-circuits before a worker is ever built, so this branch — the
+``(usage := response.usage)`` read inside ``_gen_text`` and the factory mapping
+it delegates to — is asserted by nothing else. These tests feed canned provider
+responses through the real factory and worker to pin it.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.plugins.openai.openai_completions_factory import OpenAICompletionsFactory
+from pipelex.plugins.openai.openai_completions_llm_worker import OpenAICompletionsLLMWorker
+
+if TYPE_CHECKING:
+ from typing import Any
+
+ from pytest_mock import MockerFixture
+
+
+def _make_llm_job_with_empty_usage() -> LLMJob:
+ """Build an LLMJob whose tokens_usage is seeded empty, mirroring ``llm_job_before_start``."""
+ job_metadata = JobMetadata(user_id="test-user", pipeline_run_id="test-run")
+ tokens_usage = LLMTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 1.0},
+ nb_tokens_by_category={},
+ )
+ return LLMJob(
+ job_metadata=job_metadata,
+ llm_prompt=LLMPrompt(user_text="ping"),
+ job_params=LLMJobParams(temperature=0.5),
+ job_config=LLMJobConfig(schema_reask_max_attempts=1),
+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),
+ )
+
+
+def _make_worker(mocker: MockerFixture, response: Any) -> OpenAICompletionsLLMWorker:
+ """An OpenAICompletionsLLMWorker with the real factory and a canned-response client."""
+ worker = object.__new__(OpenAICompletionsLLMWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "gpt-4o"
+ mock_model.name = "gpt-4o"
+ mock_model.thinking_mode = None
+ mock_model.listed_constraints = []
+ mock_model.extra_headers = None
+ worker.inference_model = mock_model
+ worker.openai_completions_factory = OpenAICompletionsFactory(is_http_url_enabled=False)
+ client = mocker.MagicMock()
+ client.chat.completions.create = mocker.AsyncMock(return_value=response)
+ worker.openai_client_for_text = client
+ return worker
+
+
+def _make_text_response(mocker: MockerFixture, usage: Any, content: str = "pong") -> Any:
+ response = mocker.MagicMock()
+ choice = mocker.MagicMock(finish_reason="stop")
+ choice.message.content = content
+ response.choices = [choice]
+ response.usage = usage
+ return response
+
+
+class TestOpenAICompletionsUsageCapture:
+ """The Chat Completions capture branch reads provider token counts into the job's usage."""
+
+ def test_factory_maps_basic_usage(self, mocker: MockerFixture) -> None:
+ """``make_nb_tokens_by_category`` maps prompt/completion tokens to INPUT/OUTPUT."""
+ usage = mocker.MagicMock()
+ usage.prompt_tokens = 120
+ usage.completion_tokens = 47
+ usage.prompt_tokens_details = None
+ usage.completion_tokens_details = None
+
+ result = OpenAICompletionsFactory(is_http_url_enabled=False).make_nb_tokens_by_category(usage=usage)
+
+ assert result == {TokenCategory.INPUT: 120, TokenCategory.OUTPUT: 47}
+
+ def test_factory_maps_cached_audio_and_reasoning_details(self, mocker: MockerFixture) -> None:
+ """Cached/audio/reasoning/prediction detail fields map to their own token categories."""
+ usage = mocker.MagicMock()
+ usage.prompt_tokens = 200
+ usage.completion_tokens = 90
+ usage.prompt_tokens_details = mocker.MagicMock(audio_tokens=5, cached_tokens=64)
+ usage.completion_tokens_details = mocker.MagicMock(
+ audio_tokens=3,
+ reasoning_tokens=40,
+ accepted_prediction_tokens=2,
+ rejected_prediction_tokens=1,
+ )
+
+ result = OpenAICompletionsFactory(is_http_url_enabled=False).make_nb_tokens_by_category(usage=usage)
+
+ assert result[TokenCategory.INPUT] == 200
+ assert result[TokenCategory.OUTPUT] == 90
+ assert result[TokenCategory.INPUT_AUDIO] == 5
+ assert result[TokenCategory.INPUT_CACHED] == 64
+ assert result[TokenCategory.OUTPUT_AUDIO] == 3
+ assert result[TokenCategory.OUTPUT_REASONING] == 40
+ assert result[TokenCategory.OUTPUT_ACCEPTED_PREDICTION] == 2
+ assert result[TokenCategory.OUTPUT_REJECTED_PREDICTION] == 1
+
+ @pytest.mark.asyncio
+ async def test_worker_gen_text_captures_usage(self, mocker: MockerFixture) -> None:
+ """``_gen_text`` reads ``response.usage`` and populates the job's tokens_usage."""
+ usage = mocker.MagicMock(prompt_tokens=120, completion_tokens=47, prompt_tokens_details=None, completion_tokens_details=None)
+ worker = _make_worker(mocker, _make_text_response(mocker, usage=usage))
+ llm_job = _make_llm_job_with_empty_usage()
+
+ text = await worker._gen_text(llm_job=llm_job) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert text == "pong"
+ captured = llm_job.job_report.llm_tokens_usage
+ assert captured is not None
+ assert captured.nb_tokens_by_category == {TokenCategory.INPUT: 120, TokenCategory.OUTPUT: 47}
+
+ @pytest.mark.asyncio
+ async def test_worker_gen_text_no_usage_leaves_empty(self, mocker: MockerFixture) -> None:
+ """Pins today's behavior: a response with no ``usage`` leaves nb_tokens_by_category empty.
+
+ The job still carries a truthy tokens_usage, so a zero-token UsageReportEvent is later
+ emitted — the documented silent under-count. This test pins the capture side of it.
+ """
+ worker = _make_worker(mocker, _make_text_response(mocker, usage=None))
+ llm_job = _make_llm_job_with_empty_usage()
+
+ await worker._gen_text(llm_job=llm_job) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ captured = llm_job.job_report.llm_tokens_usage
+ assert captured is not None
+ assert captured.nb_tokens_by_category == {}
diff --git a/tests/unit/pipelex/plugins/openai/test_openai_img_gen_worker_usage_capture.py b/tests/unit/pipelex/plugins/openai/test_openai_img_gen_worker_usage_capture.py
new file mode 100644
index 000000000..8e8a651b8
--- /dev/null
+++ b/tests/unit/pipelex/plugins/openai/test_openai_img_gen_worker_usage_capture.py
@@ -0,0 +1,107 @@
+"""Capture-stage test for the OpenAI image-generation token-usage path.
+
+Image-gen usage (``ImgGenTokensUsage.nb_tokens_by_category``) is read from the
+``ImagesResponse.usage`` block inside ``_gen_image_list``. No test asserts this
+capture today — non-LLM usage is uncovered in every mode. This pins it by
+driving the worker with a canned ImagesResponse carrying a usage block.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import openai
+import pytest
+
+from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
+from pipelex.cogt.img_gen.img_gen_job_components import (
+ AspectRatio,
+ Background,
+ ImgGenJobConfig,
+ ImgGenJobParams,
+ ImgGenJobReport,
+)
+from pipelex.cogt.img_gen.img_gen_model_rules import ImgGenArgTopic, ImgGenModelRules
+from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
+from pipelex.cogt.llm.thinking_mode import ThinkingMode
+from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.plugins.openai.openai_img_gen_worker import OpenAIImgGenWorker
+from pipelex.tools.misc.image_utils import ImageFormat
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+
+def _minimal_text_to_image_rules() -> ImgGenModelRules:
+ """Rules without INPUT_IMAGES so the worker routes to ``images.generate`` (not ``images.edit``)."""
+ return {ImgGenArgTopic.PROMPT: "positive_only", ImgGenArgTopic.NUM_IMAGES: "fal"}
+
+
+def _make_img_gen_model() -> InferenceModelSpec:
+ return InferenceModelSpec(
+ backend_name="openai",
+ name="gpt-image-2",
+ sdk="openai_img_gen",
+ model_type=ModelType.IMG_GEN,
+ model_id="gpt-image-2",
+ inputs=["text"],
+ outputs=["image"],
+ costs={CostCategory.INPUT: 8, CostCategory.OUTPUT: 30},
+ thinking_mode=ThinkingMode.NONE,
+ max_tokens=None,
+ max_prompt_images=None,
+ rules=_minimal_text_to_image_rules(),
+ )
+
+
+def _make_img_gen_job() -> ImgGenJob:
+ return ImgGenJob(
+ img_gen_prompt=ImgGenPrompt(positive_text="A test prompt"),
+ job_params=ImgGenJobParams(
+ aspect_ratio=AspectRatio.SQUARE,
+ size=None,
+ background=Background.OPAQUE,
+ input_fidelity=None,
+ output_format=ImageFormat.PNG,
+ ),
+ job_config=ImgGenJobConfig(is_sync_mode=False),
+ job_report=ImgGenJobReport(),
+ job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),
+ )
+
+
+class TestOpenAIImgGenWorkerUsageCapture:
+ """``_gen_image_list`` reads ``ImagesResponse.usage`` into the job's tokens_usage."""
+
+ @pytest.mark.asyncio
+ async def test_worker_captures_img_gen_usage(self, mocker: MockerFixture) -> None:
+ openai_client = openai.AsyncOpenAI(api_key="sk-test")
+
+ class FakeImageData:
+ b64_json = "dGVzdA=="
+
+ class FakeUsage:
+ input_tokens = 11
+ output_tokens = 22
+
+ class FakeImagesResponse:
+ def __init__(self) -> None:
+ self.data = [FakeImageData()]
+ self.output_format = "png"
+ self.size = "1024x1024"
+ self.usage = FakeUsage()
+
+ mocker.patch.object(openai_client.images, "generate", mocker.AsyncMock(return_value=FakeImagesResponse()))
+
+ worker = OpenAIImgGenWorker(sdk_instance=openai_client, inference_model=_make_img_gen_model())
+ job = _make_img_gen_job()
+
+ await worker.gen_image_list(img_gen_job=job, nb_images=1)
+
+ captured = job.job_report.img_gen_tokens_usage
+ assert captured is not None
+ assert captured.nb_tokens_by_category == {TokenCategory.INPUT: 11, TokenCategory.OUTPUT: 22}
diff --git a/tests/unit/pipelex/plugins/openai/test_openai_responses_usage_capture.py b/tests/unit/pipelex/plugins/openai/test_openai_responses_usage_capture.py
new file mode 100644
index 000000000..03ddd5f22
--- /dev/null
+++ b/tests/unit/pipelex/plugins/openai/test_openai_responses_usage_capture.py
@@ -0,0 +1,126 @@
+"""Capture-stage tests for the OpenAI Responses API token-usage path.
+
+The Responses worker reads ``response.usage`` (input/output tokens plus cached
+and reasoning detail fields) into the job's tokens_usage. Like the Chat
+Completions path, this capture branch is exercised by nothing else because mock
+inference never builds a worker. These tests pin the factory mapping and the
+``_gen_text`` wiring with canned responses.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+
+from pipelex.cogt.llm.llm_job import LLMJob
+from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJobReport
+from pipelex.cogt.llm.llm_prompt import LLMPrompt
+from pipelex.cogt.llm.llm_report import LLMTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.plugins.openai.openai_responses_factory import OpenAIResponsesFactory
+from pipelex.plugins.openai.openai_responses_llm_worker import OpenAIResponsesLLMWorker
+
+if TYPE_CHECKING:
+ from typing import Any
+
+ from pytest_mock import MockerFixture
+
+
+def _make_llm_job_with_empty_usage() -> LLMJob:
+ job_metadata = JobMetadata(user_id="test-user", pipeline_run_id="test-run")
+ tokens_usage = LLMTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="test-model",
+ inference_model_id="test-model-id",
+ unit_costs={CostCategory.INPUT: 1.0, CostCategory.OUTPUT: 1.0},
+ nb_tokens_by_category={},
+ )
+ return LLMJob(
+ job_metadata=job_metadata,
+ llm_prompt=LLMPrompt(user_text="ping"),
+ job_params=LLMJobParams(temperature=0.5),
+ job_config=LLMJobConfig(schema_reask_max_attempts=1),
+ job_report=LLMJobReport(llm_tokens_usage=tokens_usage),
+ )
+
+
+def _make_worker(mocker: MockerFixture, response: Any) -> OpenAIResponsesLLMWorker:
+ worker = object.__new__(OpenAIResponsesLLMWorker)
+ mock_model = mocker.MagicMock()
+ mock_model.desc = "test-model-desc"
+ mock_model.model_id = "gpt-5"
+ mock_model.name = "gpt-5"
+ mock_model.thinking_mode = None
+ mock_model.extra_headers = None
+ worker.inference_model = mock_model
+ factory = OpenAIResponsesFactory(is_http_url_enabled=False)
+ # make_input_items prepares prompt images / documents — not the capture path under test;
+ # stub it so the test does not depend on image-prep plumbing. make_nb_tokens_by_category stays real.
+ mocker.patch.object(factory, "make_input_items", new_callable=mocker.AsyncMock, return_value=[])
+ worker.openai_responses_factory = factory
+ client = mocker.MagicMock()
+ client.responses.create = mocker.AsyncMock(return_value=response)
+ worker.openai_client_for_responses = client
+ return worker
+
+
+class TestOpenAIResponsesUsageCapture:
+ """The Responses capture branch reads provider token counts into the job's usage."""
+
+ def test_factory_maps_basic_usage(self, mocker: MockerFixture) -> None:
+ usage = mocker.MagicMock()
+ usage.input_tokens = 310
+ usage.output_tokens = 88
+ usage.input_tokens_details = None
+ usage.output_tokens_details = None
+
+ result = OpenAIResponsesFactory(is_http_url_enabled=False).make_nb_tokens_by_category(usage=usage)
+
+ assert result == {TokenCategory.INPUT: 310, TokenCategory.OUTPUT: 88}
+
+ def test_factory_maps_cached_and_reasoning_details(self, mocker: MockerFixture) -> None:
+ usage = mocker.MagicMock()
+ usage.input_tokens = 310
+ usage.output_tokens = 88
+ usage.input_tokens_details = mocker.MagicMock(cached_tokens=128)
+ usage.output_tokens_details = mocker.MagicMock(reasoning_tokens=55)
+
+ result = OpenAIResponsesFactory(is_http_url_enabled=False).make_nb_tokens_by_category(usage=usage)
+
+ assert result[TokenCategory.INPUT] == 310
+ assert result[TokenCategory.OUTPUT] == 88
+ assert result[TokenCategory.INPUT_CACHED] == 128
+ assert result[TokenCategory.OUTPUT_REASONING] == 55
+
+ @pytest.mark.asyncio
+ async def test_worker_gen_text_captures_usage(self, mocker: MockerFixture) -> None:
+ response = mocker.MagicMock()
+ response.output_text = "pong"
+ response.usage = mocker.MagicMock(input_tokens=310, output_tokens=88, input_tokens_details=None, output_tokens_details=None)
+ worker = _make_worker(mocker, response)
+ llm_job = _make_llm_job_with_empty_usage()
+
+ text = await worker._gen_text(llm_job=llm_job) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ assert text == "pong"
+ captured = llm_job.job_report.llm_tokens_usage
+ assert captured is not None
+ assert captured.nb_tokens_by_category == {TokenCategory.INPUT: 310, TokenCategory.OUTPUT: 88}
+
+ @pytest.mark.asyncio
+ async def test_worker_gen_text_no_usage_leaves_empty(self, mocker: MockerFixture) -> None:
+ """Pins today's behavior: a response with no ``usage`` leaves nb_tokens_by_category empty."""
+ response = mocker.MagicMock()
+ response.output_text = "pong"
+ response.usage = None
+ worker = _make_worker(mocker, response)
+ llm_job = _make_llm_job_with_empty_usage()
+
+ await worker._gen_text(llm_job=llm_job) # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+ captured = llm_job.job_report.llm_tokens_usage
+ assert captured is not None
+ assert captured.nb_tokens_by_category == {}
diff --git a/tests/unit/pipelex/plugins/test_transport_retry_wiring.py b/tests/unit/pipelex/plugins/test_transport_retry_wiring.py
index 50425f5b6..e5fdedcda 100644
--- a/tests/unit/pipelex/plugins/test_transport_retry_wiring.py
+++ b/tests/unit/pipelex/plugins/test_transport_retry_wiring.py
@@ -13,6 +13,8 @@ from typing import TYPE_CHECKING
from mistralai.utils import RetryConfig
from pipelex.plugins.anthropic.anthropic_factory import AnthropicFactory
+from pipelex.plugins.gateway.gateway_completions_factory import GatewayCompletionsFactory
+from pipelex.plugins.gateway.gateway_responses_factory import GatewayResponsesFactory
from pipelex.plugins.google.google_factory import GoogleFactory
from pipelex.plugins.mistral.mistral_factory import MistralFactory
from pipelex.plugins.openai.openai_client_factory import OpenAIClientFactory
@@ -132,3 +134,35 @@ class TestTransportRetryWiring:
mock_client.assert_called_once()
assert mock_client.call_args.kwargs["max_retries"] == 8
+
+ def test_gateway_completions_factory_passes_max_retries(self, mocker: MockerFixture) -> None:
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_completions_factory.get_config",
+ return_value=_config_with(mocker, 4),
+ )
+ mocker.patch("pipelex.plugins.gateway.gateway_completions_factory.GatewayFactory.is_debug_enabled", return_value=False)
+ mocker.patch("pipelex.plugins.gateway.gateway_completions_factory.GatewayFactory.get_endpoint", return_value="https://gateway.test")
+ mocker.patch("pipelex.plugins.gateway.gateway_completions_factory.GatewayFactory.get_api_key", return_value="gw-test")
+ mocker.patch("pipelex.plugins.gateway.gateway_completions_factory.GatewayOpenAISdkVariant.is_completions", return_value=True)
+ mock_client = mocker.patch("openai.AsyncOpenAI")
+
+ GatewayCompletionsFactory.make_portkey_openai_client_for_completions(plugin=mocker.MagicMock(), backend=mocker.MagicMock())
+
+ mock_client.assert_called_once()
+ assert mock_client.call_args.kwargs["max_retries"] == 4
+
+ def test_gateway_responses_factory_passes_max_retries(self, mocker: MockerFixture) -> None:
+ mocker.patch(
+ "pipelex.plugins.gateway.gateway_responses_factory.get_config",
+ return_value=_config_with(mocker, 9),
+ )
+ mocker.patch("pipelex.plugins.gateway.gateway_responses_factory.GatewayFactory.is_debug_enabled", return_value=False)
+ mocker.patch("pipelex.plugins.gateway.gateway_responses_factory.GatewayFactory.get_endpoint", return_value="https://gateway.test")
+ mocker.patch("pipelex.plugins.gateway.gateway_responses_factory.GatewayFactory.get_api_key", return_value="gw-test")
+ mocker.patch("pipelex.plugins.gateway.gateway_responses_factory.GatewayOpenAISdkVariant.is_responses", return_value=True)
+ mock_client = mocker.patch("openai.AsyncOpenAI")
+
+ GatewayResponsesFactory.make_portkey_openai_client_for_responses(plugin=mocker.MagicMock(), backend=mocker.MagicMock())
+
+ mock_client.assert_called_once()
+ assert mock_client.call_args.kwargs["max_retries"] == 9
diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
index 4da138ab2..0d4d5a600 100644
--- a/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback.py
@@ -22,6 +22,7 @@ from pipelex.cogt.llm.llm_job_components import LLMJobConfig, LLMJobParams, LLMJ
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_report import LLMTokensUsage
from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.cost_registry import CostRegistry
from pipelex.cogt.usage.token_category import TokenCategory
from pipelex.config import get_config
from pipelex.graph.graph_config import DataInclusionConfig
@@ -35,6 +36,7 @@ from pipelex.tracing.activity_event_log import ActivityEventLogCache
from pipelex.tracing.in_memory_event_log import InMemoryEventLog
from pipelex.tracing.ndjson_event_log import NdjsonEventLog
from pipelex.tracing.trace_events import UsageReportEvent
+from pipelex.tracing.usage_aggregator import UsageAggregator
DATA_INCLUSION_OFF = DataInclusionConfig(
pipe_and_concept_registry=False,
@@ -386,3 +388,43 @@ class TestEmitRunnerFallback:
events = [evt for evt in reader.read_events("run_abc") if isinstance(evt, UsageReportEvent)]
assert len(events) == 2
assert sorted(evt.sequence for evt in events) == [0, 1]
+
+ def test_retried_activity_double_counts_cost_total_documenting_r2(
+ self,
+ tmp_path: Path,
+ mocker: MockerFixture,
+ ) -> None:
+ """Pins R2 at the billing level: a retried inference is counted twice in the cost total.
+
+ The inference ran once (100 input + 50 output tokens), but the retry re-emits a second
+ usage event, so ``aggregate_costs`` sums both — doubling the reported tokens and cost. This
+ is the *accepted* current behavior (see docs tracing-cost-reporting R2); the test makes the
+ over-count regression-visible. If the idempotent-resequence fix lands, flip these to the
+ single-count values.
+ """
+ _enable_ndjson_tracing(mocker, tmp_path)
+
+ manager = ReportingManager()
+ manager.setup()
+ trace_context = _make_trace_context(graph_id="run_abc", tracer_key="wf_xyz")
+
+ single_input, single_output = 100, 50
+ manager.report_inference_job(
+ _make_llm_job("run_abc", trace_context=trace_context, nb_input_tokens=single_input, nb_output_tokens=single_output)
+ )
+ manager.report_inference_job(
+ _make_llm_job("run_abc", trace_context=trace_context, nb_input_tokens=single_input, nb_output_tokens=single_output)
+ )
+
+ reader = NdjsonEventLog(traces_dir=str(tmp_path))
+ events = reader.read_events("run_abc")
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert len(usage_events) == 2, "R2: the retried activity re-emits, yielding two usage events for one inference"
+
+ tokens_usages = UsageAggregator.aggregate(events)
+ aggregated = CostRegistry.aggregate_costs(tokens_usages=tokens_usages)
+
+ # Over-count: the single inference's tokens are counted twice in the run total.
+ assert aggregated.total_nb_tokens == 2 * (single_input + single_output)
+ # unit_costs are per-million: 100*1.0/1e6 + 50*2.0/1e6 = 2e-4 per emit, doubled by the retry.
+ assert abs(aggregated.total_cost - 2 * 2e-4) < 1e-12
diff --git a/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
new file mode 100644
index 000000000..1458d43ee
--- /dev/null
+++ b/tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py
@@ -0,0 +1,174 @@
+"""Runner-side fallback emission for NON-LLM inference jobs (img-gen / extract).
+
+``ReportingManager.report_inference_job`` dispatches by job type, and the
+img-gen / extract branches (``_report_img_gen_job`` / ``_report_extract_job``)
+are exercised by no test in any mode — every usage test drives an ``LLMJob``.
+This pins that a non-LLM job with no registered event-log context (the runner
+process) emits its ``UsageReportEvent`` through the per-process fallback, that
+the ``AnyTokensUsage`` discriminated union restores the right concrete type on
+read-back, and that it prices into the run total.
+"""
+
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.extract.extract_input import ExtractInput
+from pipelex.cogt.extract.extract_job import ExtractJob
+from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams, ExtractJobReport
+from pipelex.cogt.extract.extract_report import ExtractTokensUsage
+from pipelex.cogt.img_gen.img_gen_job import ImgGenJob
+from pipelex.cogt.img_gen.img_gen_job_components import (
+ AspectRatio,
+ Background,
+ ImgGenJobConfig,
+ ImgGenJobParams,
+ ImgGenJobReport,
+)
+from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
+from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.cost_registry import CostRegistry
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.config import get_config
+from pipelex.graph.graph_config import DataInclusionConfig
+from pipelex.graph.trace_context import TraceContext
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.reporting.reporting_manager import ReportingManager
+from pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend
+from pipelex.tools.misc.image_utils import ImageFormat
+from pipelex.tracing.activity_event_log import ActivityEventLogCache
+from pipelex.tracing.ndjson_event_log import NdjsonEventLog
+from pipelex.tracing.trace_events import UsageReportEvent
+from pipelex.tracing.usage_aggregator import UsageAggregator
+
+_DATA_INCLUSION_OFF = DataInclusionConfig(
+ pipe_and_concept_registry=False,
+ stuff_json_content=False,
+ stuff_text_content=False,
+ stuff_html_content=False,
+ error_stack_traces=False,
+)
+
+
[email protected](autouse=True)
+def _reset_activity_event_log_state() -> object: # pyright: ignore[reportUnusedFunction]
+ ActivityEventLogCache.reset_for_tests()
+ yield
+ ActivityEventLogCache.reset_for_tests()
+
+
+def _enable_ndjson_tracing(mocker: MockerFixture, traces_dir: Path) -> None:
+ cfg = get_config().pipelex.tracing_config
+ mocker.patch.object(cfg, "is_enabled", True)
+ mocker.patch.object(cfg, "backend", TracingBackend.NDJSON)
+ mocker.patch.object(cfg, "ndjson", NdjsonTracingConfig(traces_dir=str(traces_dir)))
+
+
+def _trace_context(run_id: str, tracer_key: str) -> TraceContext:
+ return TraceContext(
+ graph_id=run_id,
+ tracer_key=tracer_key,
+ parent_node_id="g:node_0",
+ node_sequence=0,
+ data_inclusion=_DATA_INCLUSION_OFF,
+ )
+
+
+def _job_metadata(run_id: str, tracer_key: str) -> JobMetadata:
+ now = datetime.now(timezone.utc)
+ return JobMetadata(
+ user_id="test-user",
+ pipeline_run_id=run_id,
+ trace_context=_trace_context(run_id, tracer_key),
+ started_at=now,
+ completed_at=now + timedelta(seconds=1),
+ )
+
+
+def _make_img_gen_job(run_id: str) -> ImgGenJob:
+ job_metadata = _job_metadata(run_id, "wf_img")
+ return ImgGenJob(
+ img_gen_prompt=ImgGenPrompt(positive_text="a prompt"),
+ job_params=ImgGenJobParams(
+ aspect_ratio=AspectRatio.SQUARE,
+ size=None,
+ background=Background.OPAQUE,
+ input_fidelity=None,
+ output_format=ImageFormat.PNG,
+ ),
+ job_config=ImgGenJobConfig(is_sync_mode=False),
+ job_report=ImgGenJobReport(
+ img_gen_tokens_usage=ImgGenTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="img-model",
+ inference_model_id="img-model-id",
+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 2000},
+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 200},
+ )
+ ),
+ job_metadata=job_metadata,
+ )
+
+
+def _make_extract_job(run_id: str) -> ExtractJob:
+ job_metadata = _job_metadata(run_id, "wf_extract")
+ return ExtractJob(
+ extract_input=ExtractInput(document_uri="/tmp/test.pdf"), # noqa: S108
+ job_params=ExtractJobParams.make_default_extract_job_params(),
+ job_config=ExtractJobConfig(),
+ job_report=ExtractJobReport(
+ extract_tokens_usage=ExtractTokensUsage(
+ job_metadata=job_metadata,
+ inference_model_name="extract-model",
+ inference_model_id="extract-model-id",
+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 1000},
+ nb_tokens_by_category={TokenCategory.INPUT: 300, TokenCategory.OUTPUT: 400},
+ )
+ ),
+ job_metadata=job_metadata,
+ )
+
+
+class TestEmitRunnerFallbackNonLLM:
+ """The runner fallback emits img-gen and extract usage, not just LLM usage."""
+
+ def test_img_gen_usage_emitted_via_runner_fallback(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ _enable_ndjson_tracing(mocker, tmp_path)
+ manager = ReportingManager()
+ manager.setup()
+
+ manager.report_inference_job(_make_img_gen_job("run_img"))
+
+ reader = NdjsonEventLog(traces_dir=str(tmp_path))
+ events = reader.read_events("run_img")
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert len(usage_events) == 1
+ assert usage_events[0].writer_id.startswith("act_")
+ assert isinstance(usage_events[0].tokens_usage, ImgGenTokensUsage)
+
+ aggregated = CostRegistry.aggregate_costs(tokens_usages=UsageAggregator.aggregate(events))
+ assert aggregated.total_nb_tokens == 300 # 100 input + 200 output
+ assert aggregated.model_types == {"img-model": "img_gen"}
+ assert aggregated.has_reportable_usage is True
+
+ def test_extract_usage_emitted_via_runner_fallback(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ _enable_ndjson_tracing(mocker, tmp_path)
+ manager = ReportingManager()
+ manager.setup()
+
+ manager.report_inference_job(_make_extract_job("run_extract"))
+
+ reader = NdjsonEventLog(traces_dir=str(tmp_path))
+ events = reader.read_events("run_extract")
+ usage_events = [evt for evt in events if isinstance(evt, UsageReportEvent)]
+ assert len(usage_events) == 1
+ assert usage_events[0].writer_id.startswith("act_")
+ assert isinstance(usage_events[0].tokens_usage, ExtractTokensUsage)
+
+ aggregated = CostRegistry.aggregate_costs(tokens_usages=UsageAggregator.aggregate(events))
+ assert aggregated.total_nb_tokens == 700 # 300 input + 400 output
+ assert aggregated.model_types == {"extract-model": "extract"}
+ assert aggregated.has_reportable_usage is True
diff --git a/tests/unit/pipelex/temporal/tprl_pipe/test_hydration.py b/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py
similarity index 98%
rename from tests/unit/pipelex/temporal/tprl_pipe/test_hydration.py
rename to tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py
index 81816719d..4955cb3df 100644
--- a/tests/unit/pipelex/temporal/tprl_pipe/test_hydration.py
+++ b/tests/unit/pipelex/runtime_bridge/primitives/test_hydration.py
@@ -11,7 +11,10 @@ from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.text_content import TextContent
from pipelex.hub import get_class_registry
from pipelex.pipe_run.exceptions import PipeJobError
-from pipelex.temporal.tprl_pipe.hydration import _hydrate_list_item, hydrate_working_memory # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+from pipelex.runtime_bridge.primitives.hydration import (
+ _hydrate_list_item, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+ hydrate_working_memory,
+)
def _make_text_concept() -> Concept:
diff --git a/tests/unit/pipelex/runtime_bridge/primitives/test_marker_isolation.py b/tests/unit/pipelex/runtime_bridge/primitives/test_marker_isolation.py
new file mode 100644
index 000000000..3dd6be18f
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/primitives/test_marker_isolation.py
@@ -0,0 +1,77 @@
+from typing import Any
+
+import pytest
+from kajson import kajson
+from kajson.exceptions import KajsonDecoderError
+from pydantic import BaseModel
+
+
+class _Envelope(BaseModel):
+ """Mimics a PipeOutput-shaped payload: a BaseModel carrying a plain-dict field.
+
+ The dehydrated working_memory_raw is exactly this shape: nested dicts that
+ travel through kajson.dumps / kajson.loads at the Temporal data-converter
+ boundary. The envelope itself is a BaseModel (so kajson encodes/decodes it),
+ but the payload field is a free-form dict whose nested keys are pipelex's
+ responsibility — kajson must leave it alone.
+ """
+
+ payload: dict[str, Any]
+
+
+class TestKajsonMarkerIsolation:
+ """Wire-format contract tests: kajson's universal decoder gates on __class__.
+
+ Pipelex's deferred-binding wire format uses __pipelex_class__/__pipelex_module__
+ deliberately so kajson's decoder ignores it. These tests pin that contract:
+ if anyone ever re-collides the namespace, both tests catch it.
+ """
+
+ def test_pipelex_markers_pass_through_unchanged(self) -> None:
+ """Nested dicts with __pipelex_class__ / __pipelex_module__ must round-trip untouched.
+
+ The class name points at a name that is NOT registered anywhere — if kajson
+ were to attempt resolution, it would raise. The fact that round-trip succeeds
+ proves kajson does not touch the pipelex namespace.
+ """
+ envelope = _Envelope(
+ payload={
+ "items": [
+ {
+ "text": "hello",
+ "__pipelex_class__": "DefinitelyNotRegistered__xyz",
+ "__pipelex_module__": "builtins",
+ }
+ ]
+ }
+ )
+
+ encoded = kajson.dumps(envelope)
+ decoded = kajson.loads(encoded)
+
+ assert isinstance(decoded, _Envelope)
+ assert decoded.payload == envelope.payload
+
+ def test_kajson_markers_raise_for_unknown_class(self) -> None:
+ """Inverse oracle: __class__ / __module__ for an unknown class MUST raise.
+
+ This is the original failure mode that triggered the rename. We keep this
+ test as a sanity check that kajson's decoder is still strict on its own
+ namespace — so any accidental re-collision shows up immediately.
+ """
+ envelope = _Envelope(
+ payload={
+ "items": [
+ {
+ "text": "hello",
+ "__class__": "DefinitelyNotRegistered__xyz",
+ "__module__": "builtins",
+ }
+ ]
+ }
+ )
+
+ encoded = kajson.dumps(envelope)
+
+ with pytest.raises(KajsonDecoderError):
+ kajson.loads(encoded)
diff --git a/tests/unit/pipelex/runtime_bridge/primitives/test_pipe_classification.py b/tests/unit/pipelex/runtime_bridge/primitives/test_pipe_classification.py
new file mode 100644
index 000000000..c391c0964
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/primitives/test_pipe_classification.py
@@ -0,0 +1,82 @@
+"""Unit tests for ``pipelex.runtime_bridge.primitives.pipe_classification``.
+
+The classifier answers a single question: "should this pipe be dispatched
+as a child workflow (controller) or as an activity (leaf)?". Both
+``pipelex.temporal`` and ``pipelex_mistralai_workflows.primitives`` rely
+on it to make that branching decision (see `wip/mistral-native-plan.md`
+§1.2 row classification), so the test pins down all four controllers and
+several leaf operators.
+
+We use ``MagicMock(spec=Cls)`` because ``isinstance(mock, AnyBase)``
+respects the spec's MRO — that's all the classifier checks. Constructing
+the real Pydantic models would require a populated library and
+domain/concept resolution that is irrelevant to this unit.
+"""
+
+from typing import TYPE_CHECKING
+from unittest.mock import MagicMock
+
+from pipelex.pipe_controllers.batch.pipe_batch import PipeBatch
+from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition
+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel
+from pipelex.pipe_controllers.sequence.pipe_sequence import PipeSequence
+from pipelex.pipe_operators.compose.pipe_compose import PipeCompose
+from pipelex.pipe_operators.func.pipe_func import PipeFunc
+from pipelex.pipe_operators.img_gen.pipe_img_gen import PipeImgGen
+from pipelex.pipe_operators.llm.pipe_llm import PipeLLM
+from pipelex.runtime_bridge.primitives.pipe_classification import (
+ is_controller_pipe,
+ is_leaf_pipe,
+)
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_abstract import PipeAbstract
+
+
+def _spec_pipe(cls: "type[PipeAbstract]") -> "PipeAbstract":
+ """Return a MagicMock that ``isinstance``-passes as ``cls``."""
+ return MagicMock(spec=cls)
+
+
+class TestIsControllerPipe:
+ def test_pipe_sequence_is_controller(self) -> None:
+ pipe = _spec_pipe(PipeSequence)
+ assert is_controller_pipe(pipe) is True
+ assert is_leaf_pipe(pipe) is False
+
+ def test_pipe_batch_is_controller(self) -> None:
+ pipe = _spec_pipe(PipeBatch)
+ assert is_controller_pipe(pipe) is True
+ assert is_leaf_pipe(pipe) is False
+
+ def test_pipe_condition_is_controller(self) -> None:
+ pipe = _spec_pipe(PipeCondition)
+ assert is_controller_pipe(pipe) is True
+ assert is_leaf_pipe(pipe) is False
+
+ def test_pipe_parallel_is_controller(self) -> None:
+ pipe = _spec_pipe(PipeParallel)
+ assert is_controller_pipe(pipe) is True
+ assert is_leaf_pipe(pipe) is False
+
+
+class TestIsLeafPipe:
+ def test_pipe_llm_is_leaf(self) -> None:
+ pipe = _spec_pipe(PipeLLM)
+ assert is_leaf_pipe(pipe) is True
+ assert is_controller_pipe(pipe) is False
+
+ def test_pipe_compose_is_leaf(self) -> None:
+ pipe = _spec_pipe(PipeCompose)
+ assert is_leaf_pipe(pipe) is True
+ assert is_controller_pipe(pipe) is False
+
+ def test_pipe_func_is_leaf(self) -> None:
+ pipe = _spec_pipe(PipeFunc)
+ assert is_leaf_pipe(pipe) is True
+ assert is_controller_pipe(pipe) is False
+
+ def test_pipe_img_gen_is_leaf(self) -> None:
+ pipe = _spec_pipe(PipeImgGen)
+ assert is_leaf_pipe(pipe) is True
+ assert is_controller_pipe(pipe) is False
diff --git a/tests/unit/pipelex/temporal/tprl_pipe/test_submitter_hydration.py b/tests/unit/pipelex/runtime_bridge/primitives/test_submitter_hydration.py
similarity index 98%
rename from tests/unit/pipelex/temporal/tprl_pipe/test_submitter_hydration.py
rename to tests/unit/pipelex/runtime_bridge/primitives/test_submitter_hydration.py
index 2af304a6c..c952ec481 100644
--- a/tests/unit/pipelex/temporal/tprl_pipe/test_submitter_hydration.py
+++ b/tests/unit/pipelex/runtime_bridge/primitives/test_submitter_hydration.py
@@ -9,7 +9,7 @@ from pipelex.core.stuffs.stuff import Stuff
from pipelex.core.stuffs.text_content import TextContent
from pipelex.hub import get_class_registry, get_library_manager
from pipelex.pipe_run.exceptions import PipeJobError
-from pipelex.temporal.tprl_pipe.submitter_hydration import rehydrate_pipe_output_with_crate
+from pipelex.runtime_bridge.primitives.submitter_hydration import rehydrate_pipe_output_with_crate
def _text_concept() -> Concept:
diff --git a/tests/unit/pipelex/temporal/tprl_pipe/test_validate_as_known_class.py b/tests/unit/pipelex/runtime_bridge/primitives/test_validate_as_known_class.py
similarity index 95%
rename from tests/unit/pipelex/temporal/tprl_pipe/test_validate_as_known_class.py
rename to tests/unit/pipelex/runtime_bridge/primitives/test_validate_as_known_class.py
index cf5713ea8..2b857f3a0 100644
--- a/tests/unit/pipelex/temporal/tprl_pipe/test_validate_as_known_class.py
+++ b/tests/unit/pipelex/runtime_bridge/primitives/test_validate_as_known_class.py
@@ -2,7 +2,7 @@ from pydantic import ConfigDict
from pipelex.core.stuffs.stuff_content import StuffContent
from pipelex.core.stuffs.text_content import TextContent
-from pipelex.temporal.tprl_pipe.hydration import _validate_as_known_class # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+from pipelex.runtime_bridge.primitives.hydration import _validate_as_known_class # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
class _StuffContentAllowExtra(StuffContent):
diff --git a/tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py b/tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py
new file mode 100644
index 000000000..6b209df85
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py
@@ -0,0 +1,201 @@
+import threading
+import time
+from types import SimpleNamespace
+
+from pytest_mock import MockerFixture
+
+from pipelex.base_exceptions import PipelexSetupError
+from pipelex.pipelex import Pipelex
+from pipelex.runtime_bridge.bootstrap import ensure_pipelex_booted
+
+
+class TestEnsurePipelexBootedConcurrency:
+ def test_concurrent_first_calls_boot_once_without_error(self, mocker: MockerFixture) -> None:
+ """Two threads hitting a fresh singleton must boot exactly once, no error.
+
+ Without the double-checked lock both threads pass the ``is None`` check
+ and both call ``Pipelex.make``; the loser then raises
+ ``PipelexSetupError("Pipelex is already initialized")``. The lock must
+ serialize them so ``make`` runs once and the sibling re-checks and skips.
+ """
+ make_calls = 0
+ instance_holder: list[object] = [] # empty == not booted, [obj] == booted
+
+ def fake_get_optional_instance() -> object | None:
+ return instance_holder[0] if instance_holder else None
+
+ def fake_make(**_kwargs: object) -> object:
+ nonlocal make_calls
+ make_calls += 1
+ if instance_holder:
+ # Faithfully reproduce the real make: it raises if a singleton already exists.
+ msg = "Pipelex is already initialized"
+ raise PipelexSetupError(msg)
+ booted = SimpleNamespace(is_ready=True)
+ instance_holder.append(booted)
+ # Hold "inside make" so the sibling thread is forced to wait on the
+ # lock and re-check, rather than racing the check-then-make.
+ time.sleep(0.05)
+ return booted
+
+ mocker.patch.object(Pipelex, "get_optional_instance", side_effect=fake_get_optional_instance)
+ mocker.patch.object(Pipelex, "make", side_effect=fake_make)
+
+ start_barrier = threading.Barrier(2)
+ boot_errors: list[PipelexSetupError] = []
+
+ def worker() -> None:
+ start_barrier.wait()
+ try:
+ ensure_pipelex_booted()
+ except PipelexSetupError as exc:
+ boot_errors.append(exc)
+
+ threads = [threading.Thread(target=worker) for _ in range(2)]
+ for thread in threads:
+ thread.start()
+ for thread in threads:
+ thread.join()
+
+ assert boot_errors == []
+ assert make_calls == 1
+ assert len(instance_holder) == 1
+
+ def test_reader_arriving_mid_setup_blocks_until_ready(self, mocker: MockerFixture) -> None:
+ """A reader arriving while the first boot is mid-setup must block, not adopt a half-built instance.
+
+ Thread A enters a patched ``make`` that registers an instance with ``is_ready=False`` -- the
+ half-built window the metaclass exposes -- then completes "setup" and flips ``is_ready=True`` only
+ just before returning. Threads B and C start after A has registered, so they hit the lock-free fast
+ path against a registered-but-not-ready instance. Gating on ``is_fully_booted()`` (not bare
+ presence) must force them onto the lock; they must not return until A finished. ``make`` runs once.
+ """
+ make_calls = 0
+ instance_holder: list[SimpleNamespace] = []
+ a_registered = threading.Event()
+ a_may_finish = threading.Event()
+
+ def fake_get_optional_instance() -> SimpleNamespace | None:
+ return instance_holder[0] if instance_holder else None
+
+ def fake_make(**_kwargs: object) -> SimpleNamespace:
+ nonlocal make_calls
+ make_calls += 1
+ if instance_holder:
+ # Faithful to the real make: it raises if a singleton already exists. Surfaces loudly if a
+ # reader ever slips past the lock into a second make during the window.
+ msg = "Pipelex is already initialized"
+ raise PipelexSetupError(msg)
+ half_built = SimpleNamespace(is_ready=False)
+ instance_holder.append(half_built) # registered, NOT ready (mirrors pipelex.py construct-then-setup)
+ a_registered.set()
+ assert a_may_finish.wait(timeout=5.0), "test failed to release thread A"
+ half_built.is_ready = True # mirrors the is_ready flip on make's success tail
+ return half_built
+
+ mocker.patch.object(Pipelex, "get_optional_instance", side_effect=fake_get_optional_instance)
+ mocker.patch.object(Pipelex, "make", side_effect=fake_make)
+
+ boot_errors: list[PipelexSetupError] = []
+ observed_not_ready: list[bool] = []
+
+ def reader() -> None:
+ try:
+ ensure_pipelex_booted()
+ except PipelexSetupError as exc:
+ boot_errors.append(exc)
+ return
+ current = instance_holder[0] if instance_holder else None
+ if current is not None and not current.is_ready:
+ observed_not_ready.append(True)
+
+ thread_a = threading.Thread(target=reader, daemon=True)
+ thread_a.start()
+ assert a_registered.wait(timeout=5.0), "thread A never registered the instance"
+
+ thread_b = threading.Thread(target=reader, daemon=True)
+ thread_c = threading.Thread(target=reader, daemon=True)
+ thread_b.start()
+ thread_c.start()
+ time.sleep(0.05) # let B and C reach and block on _boot_lock during A's window
+ assert thread_b.is_alive(), "thread B returned before A finished setup -- read a half-built instance"
+ assert thread_c.is_alive(), "thread C returned before A finished setup -- read a half-built instance"
+
+ a_may_finish.set()
+ for thread in (thread_a, thread_b, thread_c):
+ thread.join(timeout=5.0)
+ assert not thread.is_alive(), "a worker deadlocked on _boot_lock"
+
+ assert boot_errors == []
+ assert make_calls == 1
+ assert len(instance_holder) == 1
+ assert observed_not_ready == []
+
+ def test_setup_failure_lets_next_thread_reboot(self, mocker: MockerFixture) -> None:
+ """A setup failure that deletes the instance must let a thread waiting in the window re-boot.
+
+ Thread A's patched ``make`` registers a half-built instance, signals, waits, then mimics make's
+ delete-on-failure (clears the holder) and raises. Thread B arrives during A's window; gating on
+ ``is_fully_booted()`` (False throughout) keeps it on the lock, and once A fails and removes the
+ instance B re-checks, sees nothing booted, and re-boots to a ready instance -- it must NOT proceed
+ against the deleted instance.
+ """
+ make_calls = 0
+ instance_holder: list[SimpleNamespace] = []
+ a_registered = threading.Event()
+ a_may_finish = threading.Event()
+
+ def fake_get_optional_instance() -> SimpleNamespace | None:
+ return instance_holder[0] if instance_holder else None
+
+ def fake_make(**_kwargs: object) -> SimpleNamespace:
+ nonlocal make_calls
+ make_calls += 1
+ if instance_holder:
+ msg = "Pipelex is already initialized"
+ raise PipelexSetupError(msg)
+ if not a_registered.is_set():
+ # First call (thread A): register half-built, then fail + delete (mirrors make try/except).
+ half_built = SimpleNamespace(is_ready=False)
+ instance_holder.append(half_built)
+ a_registered.set()
+ assert a_may_finish.wait(timeout=5.0), "test failed to release thread A"
+ instance_holder.clear() # mirrors `del MetaSingleton.instances[cls]` on failure
+ msg = "boom during setup"
+ raise PipelexSetupError(msg)
+ # Second call (thread B): clean re-boot.
+ ready = SimpleNamespace(is_ready=True)
+ instance_holder.append(ready)
+ return ready
+
+ mocker.patch.object(Pipelex, "get_optional_instance", side_effect=fake_get_optional_instance)
+ mocker.patch.object(Pipelex, "make", side_effect=fake_make)
+
+ results: dict[str, PipelexSetupError | None] = {}
+
+ def reader(name: str) -> None:
+ try:
+ ensure_pipelex_booted()
+ results[name] = None
+ except PipelexSetupError as exc:
+ results[name] = exc
+
+ thread_a = threading.Thread(target=reader, args=("A",), daemon=True)
+ thread_a.start()
+ assert a_registered.wait(timeout=5.0), "thread A never registered the instance"
+
+ thread_b = threading.Thread(target=reader, args=("B",), daemon=True)
+ thread_b.start()
+ time.sleep(0.05) # let B reach and block on _boot_lock during A's window
+ assert thread_b.is_alive(), "thread B returned during A's setup window -- read a doomed instance"
+
+ a_may_finish.set()
+ for thread in (thread_a, thread_b):
+ thread.join(timeout=5.0)
+ assert not thread.is_alive(), "a worker deadlocked on _boot_lock"
+
+ assert isinstance(results["A"], PipelexSetupError)
+ assert results["B"] is None
+ assert make_calls == 2 # A failed, B succeeded
+ assert len(instance_holder) == 1
+ assert instance_holder[0].is_ready is True
diff --git a/tests/unit/pipelex/runtime_bridge/test_direct_router_scoping.py b/tests/unit/pipelex/runtime_bridge/test_direct_router_scoping.py
new file mode 100644
index 000000000..3833d401c
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_direct_router_scoping.py
@@ -0,0 +1,67 @@
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.hub import get_pipe_router
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_router import PipeRouter
+from pipelex.pipe_run.pipe_run import PipeRun
+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.runtime_bridge.bootstrap import ensure_pipelex_booted
+from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge
+from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode
+
+
+class TestDirectRouterScoping:
+ async def test_direct_mode_scopes_router_so_nested_dispatch_skips_hub_default(self, mocker: MockerFixture) -> None:
+ """A DIRECT bridge run installs its in-process router as the active router for the whole call.
+
+ Nested controller sub-pipes resolve their router via ``get_pipe_router()``.
+ Before the fix, ``_run_direct`` never set the contextvar override, so
+ nested dispatch fell back to the hub default — which is the Temporal
+ router in a Temporal-enabled worker, leaking nested pipes to Temporal.
+ After the fix, the run is wrapped in ``scoped_pipe_router`` so
+ ``get_pipe_router()`` returns the bridge's own in-process router for the
+ duration of the call, and the override is torn down on exit.
+ """
+ fake_pipe = mocker.MagicMock()
+ fake_pipe.code = "fake_pipe"
+ fake_pipe.domain_code = "fake_domain"
+ fake_job = PipeJob.model_construct(
+ pipe=fake_pipe,
+ working_memory=WorkingMemoryFactory.make_empty(),
+ pipe_run_params=PipeRunParamsFactory.make_run_params(),
+ job_metadata=JobMetadata(user_id="anonymous", pipeline_run_id="run-id"),
+ library_crate=None,
+ )
+ mocker.patch("pipelex.runtime_bridge.bridge.build_pipe_job_from_input", return_value=fake_job)
+
+ ensure_pipelex_booted()
+ # No override is active here, so this is the hub default — exactly what a
+ # nested sub-pipe would resolve if the DIRECT run failed to scope a router.
+ router_before = get_pipe_router()
+
+ captured: dict[str, object] = {}
+
+ async def capture_active_router(_self: PipeRun, *_args: object, **_kwargs: object) -> PipeOutput: # noqa: RUF029 — replaces an awaited method, must be a coroutine
+ # Stand-in for what a nested controller does mid-run: resolve the
+ # active router. It must see the scoped in-process router, NOT the
+ # hub default.
+ captured["router_during_run"] = get_pipe_router()
+ return PipeOutput(working_memory=WorkingMemoryFactory.make_empty(), pipeline_run_id="run-id")
+
+ mocker.patch.object(PipeRun, "run", new=capture_active_router)
+
+ await run_pipe_via_bridge(
+ PipelexPipeRunInput(pipe_code="fake_pipe", execution_mode=PipelexExecutionMode.DIRECT),
+ )
+
+ router_during_run = captured["router_during_run"]
+ assert isinstance(router_during_run, PipeRouter)
+ assert router_during_run is not router_before, "Nested dispatch must not resolve the hub default during a DIRECT run"
+ # The scope is torn down after the run: a later resolution falls back to
+ # the hub default again (override restored, not clobbered).
+ assert get_pipe_router() is router_before
diff --git a/tests/unit/pipelex/runtime_bridge/test_dispatch.py b/tests/unit/pipelex/runtime_bridge/test_dispatch.py
new file mode 100644
index 000000000..0a57a3ec9
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_dispatch.py
@@ -0,0 +1,206 @@
+import builtins
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.base_exceptions import ErrorDomain, ErrorReport
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run import PipeRun
+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge
+from pipelex.runtime_bridge.exceptions import MissingMistralWorkflowsPluginError, PipelexBridgeDispatchError
+from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode
+from pipelex.temporal.exceptions import WorkflowExecutionError
+
+
+def _make_fake_pipe_job(mocker: MockerFixture, pipe_code: str, pipeline_run_id: str) -> PipeJob:
+ """Build a PipeJob without triggering Pydantic's PipeAbstract validation.
+
+ Tests at the dispatch layer don't care about the concrete pipe — only
+ that the bridge routes the right pipe_job to the right PipeRun. Using
+ ``model_construct`` lets us pass a MagicMock as ``pipe`` without
+ constructing a full PipeAbstract subclass.
+ """
+ fake_pipe = mocker.MagicMock()
+ fake_pipe.code = pipe_code
+ fake_pipe.domain_code = "fake_domain"
+ return PipeJob.model_construct(
+ pipe=fake_pipe,
+ working_memory=WorkingMemoryFactory.make_empty(),
+ pipe_run_params=PipeRunParamsFactory.make_run_params(),
+ job_metadata=JobMetadata(user_id="anonymous", pipeline_run_id=pipeline_run_id),
+ library_crate=None,
+ )
+
+
+class TestDispatch:
+ async def test_direct_mode_calls_pipe_run_with_pipe_job(self, mocker: MockerFixture) -> None:
+ fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")
+ mocker.patch(
+ "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",
+ return_value=fake_job,
+ )
+
+ fake_output = PipeOutput(
+ working_memory=WorkingMemoryFactory.make_empty(),
+ pipeline_run_id="injected-run-id",
+ )
+ mock_run = mocker.patch.object(PipeRun, "run", new_callable=mocker.AsyncMock, return_value=fake_output)
+
+ result = await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code="fake_pipe",
+ execution_mode=PipelexExecutionMode.DIRECT,
+ pipeline_run_id="caller-run-id",
+ )
+ )
+
+ assert mock_run.await_count == 1
+ await_args = mock_run.await_args
+ assert await_args is not None
+ call_kwargs: dict[str, object] = dict(await_args.kwargs)
+ assert call_kwargs["delivery_assignment"] is None
+ assert call_kwargs["pipe_job"] is fake_job
+
+ assert result.is_completed is True
+ assert result.pipeline_run_id == "injected-run-id"
+ assert result.workflow_id is None
+ assert result.graph_spec_dump is None
+
+ async def test_temporal_blocking_dispatches_to_temporal_pipe_run(self, mocker: MockerFixture) -> None:
+ fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")
+ mocker.patch(
+ "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",
+ return_value=fake_job,
+ )
+
+ fake_output = PipeOutput(
+ working_memory=WorkingMemoryFactory.make_empty(),
+ pipeline_run_id="temporal-run-id",
+ )
+ fake_temporal_run = mocker.AsyncMock(return_value=fake_output)
+ fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")
+ fake_factory.return_value.run = fake_temporal_run
+ # The bridge reports the actual Temporal workflow id (make_workflow_id, which prefixes by run
+ # mode), not the bare pipeline_run_id. Stub it to a known value and assert the bridge surfaces it.
+ fake_factory.return_value.make_workflow_id.return_value = "ut-temporal-run-id"
+
+ result = await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code="fake_pipe",
+ execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING,
+ )
+ )
+
+ fake_factory.assert_called_once()
+ assert fake_temporal_run.await_count == 1
+ fake_factory.return_value.make_workflow_id.assert_called_once_with(pipeline_run_id="caller-run-id")
+ assert result.is_completed is True
+ assert result.workflow_id == "ut-temporal-run-id"
+
+ async def test_temporal_blocking_failure_wraps_in_dispatch_error_preserving_report(self, mocker: MockerFixture) -> None:
+ """A Temporal-mode pipe failure surfaces as WorkflowExecutionError, which the bridge wraps into
+ the uniform PipelexBridgeDispatchError (same contract as DIRECT/mistral). The structured
+ ErrorReport is not lost: it stays reachable via __cause__, and PipelexBridgeDispatchError's
+ to_error_report() surfaces the underlying classification via cause-chain enrichment.
+ """
+ fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")
+ mocker.patch(
+ "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",
+ return_value=fake_job,
+ )
+
+ report = ErrorReport(
+ error_type="CogtError",
+ message="rate limited on the worker",
+ title="AI inference failed",
+ type_uri="https://docs.pipelex.com/latest/errors/cogt-error/",
+ error_category="capacity",
+ error_domain=ErrorDomain.RUNTIME,
+ retryable=False,
+ model="gpt-5",
+ provider="openai",
+ )
+ workflow_failure = WorkflowExecutionError(report.message, error_report=report)
+ fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")
+ fake_factory.return_value.run = mocker.AsyncMock(side_effect=workflow_failure)
+
+ with pytest.raises(PipelexBridgeDispatchError) as exc_info:
+ await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code="fake_pipe",
+ execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING,
+ )
+ )
+
+ # The raw Temporal failure is preserved as the cause...
+ assert exc_info.value.__cause__ is workflow_failure
+ # ...and the structured classification is surfaced through the wrapper's report.
+ recovered = exc_info.value.to_error_report()
+ assert recovered.error_type == "PipelexBridgeDispatchError"
+ assert recovered.error_category == "capacity"
+ assert recovered.retryable is False
+ assert recovered.model == "gpt-5"
+ assert recovered.provider == "openai"
+
+ async def test_temporal_fire_and_forget_returns_workflow_id_without_completion(self, mocker: MockerFixture) -> None:
+ fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")
+ mocker.patch(
+ "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",
+ return_value=fake_job,
+ )
+
+ fake_handle = mocker.MagicMock()
+ fake_start = mocker.AsyncMock(return_value=("wf-id-42", fake_handle))
+ fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")
+ fake_factory.return_value.start = fake_start
+
+ result = await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code="fake_pipe",
+ execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,
+ delivery_assignment_dump={"webhooks": [{"url": "https://example.test/hook"}], "storage": None},
+ pipeline_run_id="caller-run-id",
+ )
+ )
+
+ fake_start.assert_awaited_once()
+ assert result.is_completed is False
+ assert result.workflow_id == "wf-id-42"
+ assert result.pipeline_run_id == "caller-run-id"
+ assert result.output_dict == {}
+
+ async def test_mistral_native_without_plugin_raises_with_install_hint(self, mocker: MockerFixture) -> None:
+ fake_job = _make_fake_pipe_job(mocker=mocker, pipe_code="fake_pipe", pipeline_run_id="caller-run-id")
+ mocker.patch(
+ "pipelex.runtime_bridge.bridge.build_pipe_job_from_input",
+ return_value=fake_job,
+ )
+
+ # Simulate the plugin not being installed by stubbing __import__ to fail on the
+ # plugin's primitives.pipe_run module. The bridge does the import lazily inside
+ # _run_mistral_native, so this only fires when MISTRAL_NATIVE is selected.
+ real_import = builtins.__import__
+
+ def fake_import(name: str, *args: object, **kwargs: object) -> object:
+ if name.startswith("pipelex_mistralai_workflows.primitives.pipe_run"):
+ msg = f"No module named {name!r}"
+ raise ImportError(msg)
+ return real_import(name, *args, **kwargs) # type: ignore[arg-type]
+
+ mocker.patch.object(builtins, "__import__", side_effect=fake_import)
+
+ with pytest.raises(MissingMistralWorkflowsPluginError) as exc_info:
+ await run_pipe_via_bridge(
+ PipelexPipeRunInput(
+ pipe_code="fake_pipe",
+ execution_mode=PipelexExecutionMode.MISTRAL_NATIVE,
+ )
+ )
+
+ assert "pipelex-mistralai-workflows" in str(exc_info.value)
+ assert "pip install" in str(exc_info.value)
diff --git a/tests/unit/pipelex/runtime_bridge/test_exceptions_disclosure.py b/tests/unit/pipelex/runtime_bridge/test_exceptions_disclosure.py
new file mode 100644
index 000000000..4e57e511c
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_exceptions_disclosure.py
@@ -0,0 +1,33 @@
+import pytest
+
+from pipelex.base_exceptions import INTERNAL_ERROR_PLACEHOLDER, DisclosureMode
+from pipelex.runtime_bridge.exceptions import (
+ MissingMistralWorkflowsPluginError,
+ MissingPipelexTemporalExtraError,
+ PipelexRuntimeBridgeError,
+)
+
+
+class TestRuntimeBridgeExceptionDisclosure:
+ @pytest.mark.parametrize(
+ ("error_class", "hint_fragment"),
+ [
+ (MissingPipelexTemporalExtraError, "pip install 'pipelex[temporal]'"),
+ (MissingMistralWorkflowsPluginError, "pip install pipelex-mistralai-workflows"),
+ ],
+ )
+ def test_install_hint_survives_strict_disclosure(
+ self,
+ error_class: type[PipelexRuntimeBridgeError],
+ hint_fragment: str,
+ ) -> None:
+ """These errors carry an actionable pip-install hint in their message.
+
+ ``_authors_caller_facing_message = True`` must keep that message intact
+ under STRICT disclosure; without the flag STRICT would replace it with
+ ``INTERNAL_ERROR_PLACEHOLDER`` and the hint would be lost.
+ """
+ message = f"mode unavailable. Install with: {hint_fragment}"
+ payload = error_class(message).to_error_report().to_dict(disclosure_mode=DisclosureMode.STRICT)
+ assert payload["message"] == message
+ assert payload["message"] != INTERNAL_ERROR_PLACEHOLDER
diff --git a/tests/unit/pipelex/runtime_bridge/test_execution_mode.py b/tests/unit/pipelex/runtime_bridge/test_execution_mode.py
new file mode 100644
index 000000000..4f0825c18
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_execution_mode.py
@@ -0,0 +1,27 @@
+from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode
+
+
+class TestPipelexExecutionMode:
+ def test_string_values_are_stable(self):
+ assert PipelexExecutionMode.DIRECT == "direct"
+ assert PipelexExecutionMode.TEMPORAL_BLOCKING == "temporal_blocking"
+ assert PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET == "temporal_fire_and_forget"
+ assert PipelexExecutionMode.MISTRAL_NATIVE == "mistral_native"
+
+ def test_requires_pipelex_temporal(self):
+ assert PipelexExecutionMode.DIRECT.requires_pipelex_temporal is False
+ assert PipelexExecutionMode.TEMPORAL_BLOCKING.requires_pipelex_temporal is True
+ assert PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET.requires_pipelex_temporal is True
+ assert PipelexExecutionMode.MISTRAL_NATIVE.requires_pipelex_temporal is False
+
+ def test_requires_mistral_workflows_extra(self):
+ assert PipelexExecutionMode.DIRECT.requires_mistral_workflows_extra is False
+ assert PipelexExecutionMode.TEMPORAL_BLOCKING.requires_mistral_workflows_extra is False
+ assert PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET.requires_mistral_workflows_extra is False
+ assert PipelexExecutionMode.MISTRAL_NATIVE.requires_mistral_workflows_extra is True
+
+ def test_is_fire_and_forget(self):
+ assert PipelexExecutionMode.DIRECT.is_fire_and_forget is False
+ assert PipelexExecutionMode.TEMPORAL_BLOCKING.is_fire_and_forget is False
+ assert PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET.is_fire_and_forget is True
+ assert PipelexExecutionMode.MISTRAL_NATIVE.is_fire_and_forget is False
diff --git a/tests/unit/pipelex/runtime_bridge/test_input_models.py b/tests/unit/pipelex/runtime_bridge/test_input_models.py
new file mode 100644
index 000000000..b710ba8d7
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_input_models.py
@@ -0,0 +1,69 @@
+import pytest
+from pydantic import ValidationError
+
+from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, PipelexPipeRunOutput
+from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode
+
+
+class TestInputOutputModels:
+ def test_input_defaults_match_design(self):
+ payload = PipelexPipeRunInput(pipe_code="some_pipe")
+ assert payload.pipe_code == "some_pipe"
+ assert payload.inputs == {}
+ assert payload.output_name is None
+ assert payload.pipeline_run_id is None
+ assert payload.user_id is None
+ assert payload.library_crate_dump is None
+ assert payload.execution_mode is PipelexExecutionMode.DIRECT
+ assert payload.delivery_assignment_dump is None
+
+ def test_input_forbids_extra_fields(self):
+ with pytest.raises(ValidationError):
+ PipelexPipeRunInput.model_validate(
+ {
+ "pipe_code": "some_pipe",
+ "unexpected": "field",
+ }
+ )
+
+ def test_input_requires_pipe_code(self):
+ with pytest.raises(ValidationError):
+ PipelexPipeRunInput.model_validate({})
+
+ def test_input_round_trip_via_json(self):
+ original = PipelexPipeRunInput(
+ pipe_code="some_pipe",
+ inputs={"foo": "bar"},
+ execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING,
+ pipeline_run_id="run-123",
+ user_id="alice",
+ )
+ round_tripped = PipelexPipeRunInput.model_validate(original.model_dump(mode="json"))
+ assert round_tripped == original
+
+ def test_output_required_fields(self):
+ with pytest.raises(ValidationError):
+ PipelexPipeRunOutput.model_validate({"output_dict": {}}) # missing pipeline_run_id and is_completed
+
+ def test_output_forbids_extra_fields(self):
+ with pytest.raises(ValidationError):
+ PipelexPipeRunOutput.model_validate(
+ {
+ "output_dict": {},
+ "pipeline_run_id": "run-1",
+ "is_completed": True,
+ "rogue_field": 42,
+ }
+ )
+
+ def test_output_round_trip_via_json(self):
+ original = PipelexPipeRunOutput(
+ output_dict={"foo": "bar"},
+ main_stuff_name="main",
+ pipeline_run_id="run-1",
+ workflow_id=None,
+ is_completed=True,
+ graph_spec_dump=None,
+ )
+ round_tripped = PipelexPipeRunOutput.model_validate(original.model_dump(mode="json"))
+ assert round_tripped == original
diff --git a/tests/unit/pipelex/runtime_bridge/test_output_serialization.py b/tests/unit/pipelex/runtime_bridge/test_output_serialization.py
new file mode 100644
index 000000000..18ca38d9f
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_output_serialization.py
@@ -0,0 +1,49 @@
+from pipelex.core.memory.working_memory import WorkingMemory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.runtime_bridge.bridge import (
+ _serialize_completed_output, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+)
+
+
+class TestSerializeCompletedOutput:
+ def test_assembly_results_and_errors_cross_the_bridge_boundary(self) -> None:
+ """tokens_usages + both assembly-error strings are propagated onto the boundary DTO.
+
+ Mirrors graph_spec: a run can aggregate usage / fail assembly and must not drop
+ that at the JSON boundary, or host runtimes can't render the end-of-run cost
+ report or tell an assembly failure apart from assembly being off.
+ """
+ pipe_output = PipeOutput(
+ working_memory=WorkingMemory(),
+ pipeline_run_id="run-with-usage",
+ tokens_usages=[],
+ usage_assembly_error="usage boom",
+ graph_assembly_error="graph boom",
+ )
+
+ dto = _serialize_completed_output(
+ pipe_output=pipe_output,
+ workflow_id=None,
+ )
+
+ assert dto.tokens_usages_dump == []
+ assert dto.usage_assembly_error == "usage boom"
+ assert dto.graph_assembly_error == "graph boom"
+ assert dto.pipeline_run_id == "run-with-usage"
+ assert dto.is_completed is True
+
+ def test_absent_assembly_serializes_to_none(self) -> None:
+ """When the run produced no usage/graph, the boundary fields stay None (not [])."""
+ pipe_output = PipeOutput(
+ working_memory=WorkingMemory(),
+ pipeline_run_id="run-without-usage",
+ )
+
+ dto = _serialize_completed_output(
+ pipe_output=pipe_output,
+ workflow_id=None,
+ )
+
+ assert dto.tokens_usages_dump is None
+ assert dto.usage_assembly_error is None
+ assert dto.graph_assembly_error is None
diff --git a/tests/unit/pipelex/runtime_bridge/test_temporal_blocking_workflow_id.py b/tests/unit/pipelex/runtime_bridge/test_temporal_blocking_workflow_id.py
new file mode 100644
index 000000000..9180523b3
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_temporal_blocking_workflow_id.py
@@ -0,0 +1,58 @@
+"""TEMPORAL_BLOCKING reports the actual Temporal workflow id, not the bare run id.
+
+``TemporalPipeRun.run()`` starts the workflow with ``make_workflow_id(pipeline_run_id)``,
+which prefixes the id in non-NORMAL run modes (``ut-``/``ci-``/``cc-``/``cct-`` — pinned
+per run mode by ``tests/unit/pipelex/temporal/test_workflow_id_construction.py``). The bridge
+must surface that same id so a caller can query/cancel the real workflow — matching what
+fire-and-forget already returns from ``start()``. This test pins the *wiring* (the bridge
+reports ``make_workflow_id``'s output, called with the job's pipeline_run_id), independent of
+the prefix table; the prefixing itself is covered by the temporal-manager test above.
+"""
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge
+from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode
+
+
+def _fake_pipe_job(mocker: MockerFixture) -> PipeJob:
+ fake_pipe = mocker.MagicMock()
+ fake_pipe.code = "fake_pipe"
+ fake_pipe.domain_code = "fake_domain"
+ return PipeJob.model_construct(
+ pipe=fake_pipe,
+ working_memory=WorkingMemoryFactory.make_empty(),
+ pipe_run_params=PipeRunParamsFactory.make_run_params(),
+ job_metadata=JobMetadata(user_id="anonymous", pipeline_run_id="bare-run-id"),
+ library_crate=None,
+ )
+
+
+class TestTemporalBlockingWorkflowId:
+ async def test_blocking_reports_make_workflow_id_not_bare_run_id(self, mocker: MockerFixture) -> None:
+ fake_job = _fake_pipe_job(mocker)
+ mocker.patch("pipelex.runtime_bridge.bridge.build_pipe_job_from_input", return_value=fake_job)
+
+ fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")
+ temporal_pipe_run = fake_factory.return_value
+ temporal_pipe_run.run = mocker.AsyncMock(
+ return_value=PipeOutput(working_memory=WorkingMemoryFactory.make_empty(), pipeline_run_id="bare-run-id"),
+ )
+ # Stand in for the real make_workflow_id, which prefixes by run mode. The bridge must report
+ # THIS value (the id Temporal actually started the workflow under), not pipe_output.pipeline_run_id.
+ temporal_pipe_run.make_workflow_id.return_value = "ut-bare-run-id"
+
+ result = await run_pipe_via_bridge(
+ PipelexPipeRunInput(pipe_code="fake_pipe", execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING),
+ )
+
+ temporal_pipe_run.make_workflow_id.assert_called_once_with(pipeline_run_id="bare-run-id")
+ assert result.workflow_id == "ut-bare-run-id"
+ assert result.workflow_id != result.pipeline_run_id
diff --git a/tests/unit/pipelex/runtime_bridge/test_trace_context_contract.py b/tests/unit/pipelex/runtime_bridge/test_trace_context_contract.py
new file mode 100644
index 000000000..7a3655eb3
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_trace_context_contract.py
@@ -0,0 +1,89 @@
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory
+from pipelex.core.pipes.pipe_output import PipeOutput
+from pipelex.graph.graph_config import DataInclusionConfig
+from pipelex.graph.trace_context import TraceContext
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run import PipeRun
+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.runtime_bridge.bridge import PipelexPipeRunInput, run_pipe_via_bridge
+from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode
+
+
+def _make_trace_context() -> TraceContext:
+ data_inclusion = DataInclusionConfig(
+ stuff_json_content=False,
+ stuff_text_content=False,
+ stuff_html_content=False,
+ error_stack_traces=False,
+ pipe_and_concept_registry=False,
+ )
+ return TraceContext(graph_id="host-graph-id", data_inclusion=data_inclusion)
+
+
+def _fake_pipe_job(mocker: MockerFixture) -> PipeJob:
+ fake_pipe = mocker.MagicMock()
+ fake_pipe.code = "fake_pipe"
+ fake_pipe.domain_code = "fake_domain"
+ return PipeJob.model_construct(
+ pipe=fake_pipe,
+ working_memory=WorkingMemoryFactory.make_empty(),
+ pipe_run_params=PipeRunParamsFactory.make_run_params(),
+ job_metadata=JobMetadata(user_id="anonymous", pipeline_run_id="run-id"),
+ library_crate=None,
+ )
+
+
+class TestTraceContextContract:
+ async def test_direct_mode_forwards_host_trace_context(self, mocker: MockerFixture) -> None:
+ trace_context = _make_trace_context()
+ captured: dict[str, object] = {}
+ fake_job = _fake_pipe_job(mocker)
+
+ def spy(**kwargs: object) -> PipeJob:
+ captured["trace_context"] = kwargs["trace_context"]
+ return fake_job
+
+ mocker.patch("pipelex.runtime_bridge.bridge.build_pipe_job_from_input", side_effect=spy)
+ mocker.patch.object(
+ PipeRun,
+ "run",
+ new_callable=mocker.AsyncMock,
+ return_value=PipeOutput(working_memory=WorkingMemoryFactory.make_empty(), pipeline_run_id="run-id"),
+ )
+
+ await run_pipe_via_bridge(
+ PipelexPipeRunInput(pipe_code="fake_pipe", execution_mode=PipelexExecutionMode.DIRECT),
+ trace_context=trace_context,
+ )
+
+ assert captured["trace_context"] is trace_context
+
+ async def test_temporal_mode_nulls_host_trace_context(self, mocker: MockerFixture) -> None:
+ trace_context = _make_trace_context()
+ captured: dict[str, object] = {}
+ fake_job = _fake_pipe_job(mocker)
+
+ def spy(**kwargs: object) -> PipeJob:
+ captured["trace_context"] = kwargs["trace_context"]
+ return fake_job
+
+ mocker.patch("pipelex.runtime_bridge.bridge.build_pipe_job_from_input", side_effect=spy)
+ fake_factory = mocker.patch("pipelex.temporal.tprl_pipe.temporal_pipe_run.make_temporal_pipe_run")
+ fake_factory.return_value.run = mocker.AsyncMock(
+ return_value=PipeOutput(working_memory=WorkingMemoryFactory.make_empty(), pipeline_run_id="temporal-run-id"),
+ )
+ # The blocking branch reports make_workflow_id(...) as the workflow id; stub it to a string
+ # so PipelexPipeRunOutput validation passes (this test only asserts trace_context nulling).
+ fake_factory.return_value.make_workflow_id.return_value = "ut-temporal-run-id"
+
+ await run_pipe_via_bridge(
+ PipelexPipeRunInput(pipe_code="fake_pipe", execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING),
+ trace_context=trace_context,
+ )
+
+ assert captured["trace_context"] is None
diff --git a/tests/unit/pipelex/runtime_bridge/test_validation.py b/tests/unit/pipelex/runtime_bridge/test_validation.py
new file mode 100644
index 000000000..e85ed8068
--- /dev/null
+++ b/tests/unit/pipelex/runtime_bridge/test_validation.py
@@ -0,0 +1,94 @@
+import pytest
+
+from pipelex.libraries.library_crate import LibraryCrate
+from pipelex.pipe_run.delivery_assignment import DeliveryAssignment
+from pipelex.runtime_bridge.bridge import (
+ PipelexPipeRunInput,
+ _decode_delivery_assignment, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+ _decode_library_crate, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+ _validate_input, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+ run_pipe_via_bridge,
+)
+from pipelex.runtime_bridge.exceptions import PipelexBridgeDispatchError
+from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode
+
+
+class TestBridgeValidationAndDecoding:
+ def test_validate_input_passes_for_direct_without_delivery(self):
+ payload = PipelexPipeRunInput(pipe_code="any", execution_mode=PipelexExecutionMode.DIRECT)
+ _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump)) # must not raise
+
+ def test_validate_input_passes_for_temporal_blocking_without_delivery(self):
+ payload = PipelexPipeRunInput(pipe_code="any", execution_mode=PipelexExecutionMode.TEMPORAL_BLOCKING)
+ _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump)) # must not raise
+
+ def test_validate_input_rejects_fire_and_forget_without_delivery(self):
+ payload = PipelexPipeRunInput(
+ pipe_code="any",
+ execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,
+ )
+ with pytest.raises(PipelexBridgeDispatchError, match="TEMPORAL_FIRE_AND_FORGET"):
+ _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump))
+
+ def test_validate_input_rejects_fire_and_forget_with_empty_delivery(self):
+ # A DeliveryAssignment with no storage and no webhooks is a no-op: completion would
+ # be silently dropped, so it must be rejected just like a missing dump.
+ payload = PipelexPipeRunInput(
+ pipe_code="any",
+ execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,
+ delivery_assignment_dump={"webhooks": [], "storage": None},
+ )
+ with pytest.raises(PipelexBridgeDispatchError, match="delivery target"):
+ _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump))
+
+ def test_validate_input_accepts_fire_and_forget_with_storage_target(self):
+ payload = PipelexPipeRunInput(
+ pipe_code="any",
+ execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,
+ delivery_assignment_dump={"webhooks": [], "storage": {"key_prefix": "runs/abc"}},
+ )
+ _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump)) # must not raise
+
+ def test_validate_input_accepts_fire_and_forget_with_webhook_target(self):
+ payload = PipelexPipeRunInput(
+ pipe_code="any",
+ execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,
+ delivery_assignment_dump={"webhooks": [{"url": "https://example.test/hook"}], "storage": None},
+ )
+ _validate_input(payload, delivery_assignment=_decode_delivery_assignment(payload.delivery_assignment_dump)) # must not raise
+
+ def test_decode_library_crate_returns_none_for_none(self):
+ assert _decode_library_crate(None) is None
+
+ def test_decode_library_crate_round_trips_empty(self):
+ empty = LibraryCrate()
+ decoded = _decode_library_crate(empty.model_dump(mode="json"))
+ assert decoded is not None
+ assert decoded.concepts == empty.concepts
+ assert decoded.pipes == empty.pipes
+
+ def test_decode_delivery_assignment_returns_none_for_none(self):
+ assert _decode_delivery_assignment(None) is None
+
+ def test_decode_delivery_assignment_round_trips(self):
+ assignment = DeliveryAssignment.model_validate(
+ {
+ "storage": {"key_prefix": "runs/abc"},
+ "webhooks": [{"url": "https://example.test/hook"}],
+ }
+ )
+ decoded = _decode_delivery_assignment(assignment.model_dump(mode="json"))
+ assert decoded is not None
+ assert decoded.storage is not None
+ assert decoded.storage.key_prefix == "runs/abc/" # storage validator appends trailing /
+ assert len(decoded.webhooks) == 1
+ assert decoded.webhooks[0].url == "https://example.test/hook"
+
+ @pytest.mark.asyncio
+ async def test_run_pipe_via_bridge_rejects_fire_and_forget_without_delivery(self):
+ payload = PipelexPipeRunInput(
+ pipe_code="any",
+ execution_mode=PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET,
+ )
+ with pytest.raises(PipelexBridgeDispatchError, match="TEMPORAL_FIRE_AND_FORGET"):
+ await run_pipe_via_bridge(payload)
diff --git a/tests/unit/pipelex/temporal/test_workflow_caller_passthrough.py b/tests/unit/pipelex/temporal/test_workflow_caller_passthrough.py
index 3c6d394bc..bce952630 100644
--- a/tests/unit/pipelex/temporal/test_workflow_caller_passthrough.py
+++ b/tests/unit/pipelex/temporal/test_workflow_caller_passthrough.py
@@ -96,7 +96,7 @@ class TestWorkflowCallerPassthrough:
await executor.execute_child_workflow(
workflow_class=_StubWorkflow,
workflow_arg={"foo": "bar"},
- workflow_id="ut-test-run/pipe-router",
+ workflow_id="ut-test-run_pipe-router",
search_attributes=_SEARCH_ATTRS,
static_summary=_SUMMARY,
static_details=_DETAILS,
@@ -117,7 +117,7 @@ class TestWorkflowCallerPassthrough:
await executor.start_child_workflow(
workflow_class=_StubWorkflow,
workflow_arg={"foo": "bar"},
- workflow_id="ut-test-run/pipe-router",
+ workflow_id="ut-test-run_pipe-router",
search_attributes=_SEARCH_ATTRS,
static_summary=_SUMMARY,
static_details=_DETAILS,
diff --git a/tests/unit/pipelex/temporal/test_workflow_id_construction.py b/tests/unit/pipelex/temporal/test_workflow_id_construction.py
index 446793f34..f15858164 100644
--- a/tests/unit/pipelex/temporal/test_workflow_id_construction.py
+++ b/tests/unit/pipelex/temporal/test_workflow_id_construction.py
@@ -4,8 +4,8 @@ Pins the shape documented in
``wip/temporal-primitives/id-and-naming-design.md`` §"Layer 1 — Identity":
- Top-level: ``{env_prefix}{pipeline_run_id}``.
-- Fixed-role child: ``{parent}/pipe-router``.
-- Dynamic child (sub-pipe spawned by a router): ``{parent}/{pipe_code}-{8-hex}``.
+- Fixed-role child: ``{parent}_pipe-router``.
+- Dynamic child (sub-pipe spawned by a router): ``{parent}_{pipe_code}-{8-hex}``.
The dynamic-child id uses ``workflow.uuid4()`` for replay-safety — Temporal's
``workflow.uuid4()`` is deterministic, unlike stdlib ``uuid.uuid4()`` (which
@@ -93,17 +93,19 @@ class TestChildWorkflowIdConstruction:
sites — the formula itself is the contract.
"""
- def test_fixed_role_child_uses_slash_pipe_router_suffix(self) -> None:
+ def test_fixed_role_child_uses_underscore_pipe_router_suffix(self) -> None:
parent_workflow_id = "ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c"
# Mirrors the construction in wf_pipe_run.py:
- child_id = f"{parent_workflow_id}/pipe-router"
+ child_id = f"{parent_workflow_id}_pipe-router"
- assert child_id == "ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c/pipe-router"
- # The separator is `/`, not `-`, to keep the path structure unambiguous.
- assert child_id.split("/") == [parent_workflow_id, "pipe-router"]
+ assert child_id == "ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c_pipe-router"
+ # The separator is `_`, never `/`: workflow ids must stay free of path
+ # separators so they can be reused verbatim as S3 keys / file names.
+ assert "/" not in child_id
+ assert child_id.endswith("_pipe-router")
- def test_dynamic_child_uses_slash_pipe_code_and_8_hex_disambiguator(self, mocker: MockerFixture) -> None:
+ def test_dynamic_child_uses_underscore_pipe_code_and_8_hex_disambiguator(self, mocker: MockerFixture) -> None:
"""The dynamic-child id must be fully determined by
(parent_workflow_id, pipe_code, workflow.uuid4() output) — i.e. the
implementation must use ``workflow.uuid4()`` (replay-safe), not
@@ -119,10 +121,11 @@ class TestChildWorkflowIdConstruction:
pipe_code = "translate_doc"
# Mirrors the construction in temporal_pipe_router.py (child branch):
- child_id = f"{parent_workflow_id}/{pipe_code}-{str(workflow.uuid4())[:8]}"
+ child_id = f"{parent_workflow_id}_{pipe_code}-{str(workflow.uuid4())[:8]}"
- assert child_id == "ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c/translate_doc-7c1e2f8a"
+ assert child_id == "ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c_translate_doc-7c1e2f8a"
+ assert "/" not in child_id
# Replay-determinism: re-running the same construction with the same
# mock yields the same id.
- child_id_again = f"{parent_workflow_id}/{pipe_code}-{str(workflow.uuid4())[:8]}"
+ child_id_again = f"{parent_workflow_id}_{pipe_code}-{str(workflow.uuid4())[:8]}"
assert child_id_again == child_id
diff --git a/tests/unit/pipelex/temporal/tprl_pipe/test_temporal_pipe_router_child_observability.py b/tests/unit/pipelex/temporal/tprl_pipe/test_temporal_pipe_router_child_observability.py
index aa5d0f4f1..385c899b3 100644
--- a/tests/unit/pipelex/temporal/tprl_pipe/test_temporal_pipe_router_child_observability.py
+++ b/tests/unit/pipelex/temporal/tprl_pipe/test_temporal_pipe_router_child_observability.py
@@ -114,5 +114,5 @@ class TestTemporalPipeRouterChildObservability:
# Existing kwargs must still be present on the child dispatch.
assert kwargs.get("static_summary") == expected_summary
assert kwargs.get("search_attributes") is not None
- # Child workflow id is built as `{parent}/{pipe_code}-{8 hex}`.
- assert kwargs.get("id") == "ut-parent-wf-id/translate_doc-deadbeef"
+ # Child workflow id is built as `{parent}_{pipe_code}-{8 hex}`.
+ assert kwargs.get("id") == "ut-parent-wf-id_translate_doc-deadbeef"
diff --git a/tests/unit/pipelex/tracing/test_dynamodb_event_log.py b/tests/unit/pipelex/tracing/test_dynamodb_event_log.py
new file mode 100644
index 000000000..f505f6212
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_dynamodb_event_log.py
@@ -0,0 +1,62 @@
+from typing import Any
+
+import pytest
+from botocore.exceptions import BotoCoreError, ClientError, EndpointConnectionError
+from pytest_mock import MockerFixture
+
+from pipelex.tracing.dynamodb_event_log import DynamoDBEventLog
+from pipelex.tracing.exceptions import EventLogReadError, EventLogSetupError
+
+
+class TestDynamoDBEventLogReadEvents:
+ def _make_event_log_with_table(self, mocker: MockerFixture) -> tuple[DynamoDBEventLog, Any]:
+ # boto3.resource(...) does not perform any network call at construction.
+ event_log = DynamoDBEventLog(table_name="trace-events-test", region="us-east-1")
+ mock_table = mocker.MagicMock()
+ mocker.patch.object(event_log, "_table", new=mock_table)
+ return event_log, mock_table
+
+ @pytest.mark.parametrize(
+ "backend_error",
+ [
+ ClientError(
+ error_response={"Error": {"Code": "ThrottlingException", "Message": "Rate exceeded"}},
+ operation_name="Query",
+ ),
+ EndpointConnectionError(endpoint_url="https://dynamodb.us-east-1.amazonaws.com"),
+ ],
+ )
+ def test_read_events_translates_backend_error(self, mocker: MockerFixture, backend_error: BotoCoreError) -> None:
+ # A store-level botocore failure (throttling, network) must surface as the domain
+ # EventLogReadError so best-effort callers can catch a single Pipelex error.
+ event_log, mock_table = self._make_event_log_with_table(mocker)
+ mock_table.query.side_effect = backend_error
+
+ with pytest.raises(EventLogReadError) as exc_info:
+ event_log.read_events("plr-throttled")
+
+ assert exc_info.value.__cause__ is backend_error
+
+ def test_read_events_translates_error_on_pagination_query(self, mocker: MockerFixture) -> None:
+ # The translation must also cover the pagination loop, not just the first query.
+ event_log, mock_table = self._make_event_log_with_table(mocker)
+ first_page: dict[str, Any] = {"Items": [], "LastEvaluatedKey": {"PK": "x", "SK": "y"}}
+ throttle = ClientError(
+ error_response={"Error": {"Code": "ThrottlingException", "Message": "Rate exceeded"}},
+ operation_name="Query",
+ )
+ mock_table.query.side_effect = [first_page, throttle]
+
+ with pytest.raises(EventLogReadError):
+ event_log.read_events("plr-paginated")
+
+ def test_init_translates_construction_error(self, mocker: MockerFixture) -> None:
+ # A botocore failure while building the boto3 client (e.g. a misconfigured region) must surface as
+ # the domain EventLogSetupError so best-effort callers (tracing assembly) degrade rather than abort.
+ construction_error = EndpointConnectionError(endpoint_url="https://dynamodb.bad-region.amazonaws.com")
+ mocker.patch("pipelex.tracing.dynamodb_event_log.boto3.resource", side_effect=construction_error)
+
+ with pytest.raises(EventLogSetupError) as exc_info:
+ DynamoDBEventLog(table_name="trace-events-test", region="bad-region")
+
+ assert exc_info.value.__cause__ is construction_error
diff --git a/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
new file mode 100644
index 000000000..7761d55d6
--- /dev/null
+++ b/tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py
@@ -0,0 +1,101 @@
+"""Aggregation + costing of NON-LLM token usage (img-gen / extract / search).
+
+Every usage test in the tree uses ``LLMTokensUsage`` only — img-gen, extract,
+and search usage is never routed through the aggregator + cost registry, and
+never round-tripped through the ``UsageReportEvent`` discriminated union that
+carries it across worker processes. This module pins all three:
+
+1. each non-LLM usage survives a ``UsageReportEvent`` JSON round-trip with its
+ ``model_type`` discriminator intact (the cross-process serialization concern),
+2. ``UsageAggregator.aggregate`` surfaces them, and
+3. ``CostRegistry.aggregate_costs`` prices them into the run totals.
+"""
+
+from datetime import datetime, timezone
+
+from pipelex.cogt.extract.extract_report import ExtractTokensUsage
+from pipelex.cogt.img_gen.img_gen_report import ImgGenTokensUsage
+from pipelex.cogt.search.search_report import SearchTokensUsage
+from pipelex.cogt.usage.cost_category import CostCategory
+from pipelex.cogt.usage.cost_registry import CostRegistry
+from pipelex.cogt.usage.token_category import TokenCategory
+from pipelex.pipeline.job_metadata import JobMetadata
+from pipelex.reporting.reporting_types import AnyTokensUsage
+from pipelex.tracing.trace_events import UsageReportEvent
+from pipelex.tracing.usage_aggregator import UsageAggregator
+
+
+def _job_metadata() -> JobMetadata:
+ return JobMetadata(user_id="test-user", pipeline_run_id="run-non-llm")
+
+
+def _usage_event(tokens_usage: AnyTokensUsage, sequence: int) -> UsageReportEvent:
+ return UsageReportEvent(
+ pipeline_run_id="run-non-llm",
+ workflow_id="wf-non-llm",
+ writer_id="act_test",
+ timestamp=datetime.now(timezone.utc),
+ sequence=sequence,
+ node_id=f"g:node_{sequence}",
+ tokens_usage=tokens_usage,
+ )
+
+
+class TestNonLLMUsageAggregation:
+ """Img-gen, extract, and search usage aggregate and cost like LLM usage does."""
+
+ def test_non_llm_usage_round_trips_aggregates_and_costs(self) -> None:
+ img_gen_usage = ImgGenTokensUsage(
+ job_metadata=_job_metadata(),
+ inference_model_name="img-gen-model",
+ inference_model_id="img-gen-model-id",
+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 2000},
+ nb_tokens_by_category={TokenCategory.INPUT: 100, TokenCategory.OUTPUT: 200},
+ )
+ extract_usage = ExtractTokensUsage(
+ job_metadata=_job_metadata(),
+ inference_model_name="extract-model",
+ inference_model_id="extract-model-id",
+ unit_costs={CostCategory.INPUT: 1000, CostCategory.OUTPUT: 1000},
+ nb_tokens_by_category={TokenCategory.INPUT: 300, TokenCategory.OUTPUT: 400},
+ )
+ search_usage = SearchTokensUsage(
+ job_metadata=_job_metadata(),
+ inference_model_name="search-model",
+ inference_model_id="search-model-id",
+ unit_costs={CostCategory.INPUT: 5000, CostCategory.OUTPUT: 5000},
+ nb_tokens_by_category={TokenCategory.INPUT: 10, TokenCategory.OUTPUT: 20},
+ )
+
+ raw_events = [
+ _usage_event(img_gen_usage, sequence=0),
+ _usage_event(extract_usage, sequence=1),
+ _usage_event(search_usage, sequence=2),
+ ]
+ # Round-trip each event through JSON, exactly as the runner fallback persists it and the
+ # assembler reads it back. This proves the AnyTokensUsage discriminated union restores the
+ # right concrete non-LLM type (a NamedTuple/dataclass would silently corrupt across the wire).
+ events = [UsageReportEvent.model_validate_json(event.model_dump_json()) for event in raw_events]
+
+ tokens_usages = UsageAggregator.aggregate(events)
+ assert [usage.model_type for usage in tokens_usages] == ["img_gen", "extract", "search"]
+ assert isinstance(tokens_usages[0], ImgGenTokensUsage)
+ assert isinstance(tokens_usages[1], ExtractTokensUsage)
+ assert isinstance(tokens_usages[2], SearchTokensUsage)
+
+ aggregated = CostRegistry.aggregate_costs(tokens_usages=tokens_usages)
+
+ # total_nb_tokens = sum(input_joined) + sum(output) = (100+300+10) + (200+400+20)
+ assert aggregated.total_nb_tokens == 1030
+ # per-usage cost (no cached tokens): input*unit_in/1e6 + output*unit_out/1e6
+ # img-gen: 100*0.001 + 200*0.002 = 0.5
+ # extract: 300*0.001 + 400*0.001 = 0.7
+ # search: 10*0.005 + 20*0.005 = 0.15
+ assert abs(aggregated.total_cost - 1.35) < 1e-9
+ assert aggregated.has_reportable_usage is True
+ assert set(aggregated.grouped_by_model.keys()) == {"img-gen-model", "extract-model", "search-model"}
+ assert aggregated.model_types == {
+ "img-gen-model": "img_gen",
+ "extract-model": "extract",
+ "search-model": "search",
+ }
diff --git a/uv.lock b/uv.lock
index 808a8e87b..d5b404b5f 100644
--- a/uv.lock
+++ b/uv.lock
@@ -2396,7 +2396,7 @@ wheels = [
[[package]]
name = "mistralai"
-version = "1.12.4"
+version = "1.12.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "eval-type-backport" },
@@ -2405,14 +2405,15 @@ dependencies = [
{ name = "opentelemetry-api" },
{ name = "opentelemetry-exporter-otlp-proto-http" },
{ name = "opentelemetry-sdk" },
+ { name = "opentelemetry-semantic-conventions" },
{ name = "pydantic" },
{ name = "python-dateutil" },
{ name = "pyyaml" },
{ name = "typing-inspection" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/aa/12/c3476c53e907255b5f485f085ba50dd9a84b40fe662e9a888d6ded26fa7b/mistralai-1.12.4.tar.gz", hash = "sha256:e52b53bab58025dcd208eeac13e3c3df5778d4112eeca1f08124096c7738929f", size = 243129, upload-time = "2026-02-20T17:55:13.73Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/35/7c/4c2404f49a9b0a3dcf7ce9e05b4749cf3d53f203e5724b430fa07bfebb48/mistralai-1.12.0.tar.gz", hash = "sha256:38149a598eab7ed4b876c92e6e1c89d31fe01935bbf19314549fd32b21a1135f", size = 242013, upload-time = "2026-02-04T14:50:04.166Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/c9/f9/98d825105c450b9c67c27026caa374112b7e466c18331601d02ca278a01b/mistralai-1.12.4-py3-none-any.whl", hash = "sha256:7b69fcbc306436491ad3377fbdead527c9f3a0ce145ec029bf04c6308ff2cca6", size = 509321, upload-time = "2026-02-20T17:55:15.27Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/c5/f188efddb2b6debf73f2b2fea70eb119d1935ee33a92947d0756e22476ba/mistralai-1.12.0-py3-none-any.whl", hash = "sha256:a5873d456b7920782f716d60593d4db32c9510b13fc98908081b9f36427e0e5a", size = 500154, upload-time = "2026-02-04T14:50:05.628Z" },
]
[[package]]
@@ -2936,14 +2937,14 @@ wheels = [
[[package]]
name = "nexus-rpc"
-version = "1.3.0"
+version = "1.4.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/2e/f2/d54f5c03d8f4672ccc0875787a385f53dcb61f98a8ae594b5620e85b9cb3/nexus_rpc-1.3.0.tar.gz", hash = "sha256:e56d3b57b60d707ce7a72f83f23f106b86eca1043aa658e44582ab5ff30ab9ad", size = 75650, upload-time = "2025-12-08T22:59:13.002Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/35/d5/cd1ffb202b76ebc1b33c1332a3416e55a39929006982adc2b1eb069aaa9b/nexus_rpc-1.4.0.tar.gz", hash = "sha256:3b8b373d4865671789cc43623e3dc0bcbf192562e40e13727e17f1c149050fba", size = 82367, upload-time = "2026-02-25T22:01:34.053Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/d6/74/0afd841de3199c148146c1d43b4bfb5605b2f1dc4c9a9087fe395091ea5a/nexus_rpc-1.3.0-py3-none-any.whl", hash = "sha256:aee0707b4861b22d8124ecb3f27d62dafbe8777dc50c66c91e49c006f971b92d", size = 28873, upload-time = "2025-12-08T22:59:12.024Z" },
+ { url = "https://files.pythonhosted.org/packages/11/52/6327a5f4fda01207205038a106a99848a41c83e933cd23ea2cab3d2ebc6c/nexus_rpc-1.4.0-py3-none-any.whl", hash = "sha256:14c953d3519113f8ccec533a9efdb6b10c28afef75d11cdd6d422640c40b3a49", size = 29645, upload-time = "2026-02-25T22:01:33.122Z" },
]
[[package]]
@@ -3341,32 +3342,32 @@ wheels = [
[[package]]
name = "opentelemetry-api"
-version = "1.41.1"
+version = "1.38.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "importlib-metadata" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/fa/fc/b7564cbef36601aef0d6c9bc01f7badb64be8e862c2e1c3c5c3b43b53e4f/opentelemetry_api-1.41.1.tar.gz", hash = "sha256:0ad1814d73b875f84494387dae86ce0b12c68556331ce6ce8fe789197c949621", size = 71416, upload-time = "2026-04-24T13:15:38.262Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/08/d8/0f354c375628e048bd0570645b310797299754730079853095bf000fba69/opentelemetry_api-1.38.0.tar.gz", hash = "sha256:f4c193b5e8acb0912b06ac5b16321908dd0843d75049c091487322284a3eea12", size = 65242, upload-time = "2025-10-16T08:35:50.25Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/29/59/3e7118ed140f76b0982ba4321bdaed1997a0473f9720de2d10788a577033/opentelemetry_api-1.41.1-py3-none-any.whl", hash = "sha256:a22df900e75c76dc08440710e51f52f1aa6b451b429298896023e60db5b3139f", size = 69007, upload-time = "2026-04-24T13:15:15.662Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/a2/d86e01c28300bd41bab8f18afd613676e2bd63515417b77636fc1add426f/opentelemetry_api-1.38.0-py3-none-any.whl", hash = "sha256:2891b0197f47124454ab9f0cf58f3be33faca394457ac3e09daba13ff50aa582", size = 65947, upload-time = "2025-10-16T08:35:30.23Z" },
]
[[package]]
name = "opentelemetry-exporter-otlp-proto-common"
-version = "1.41.1"
+version = "1.38.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "opentelemetry-proto" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/ae/fa/f9e3bd3c4d692b3ce9a2880a167d1f79681a1bea11f00d5bf76adc03e6ea/opentelemetry_exporter_otlp_proto_common-1.41.1.tar.gz", hash = "sha256:0e253156ea9c36b0bd3d2440c5c9ba7dd1f3fb64ba7a08fc85fbac536b56e1fb", size = 20409, upload-time = "2026-04-24T13:15:40.924Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/19/83/dd4660f2956ff88ed071e9e0e36e830df14b8c5dc06722dbde1841accbe8/opentelemetry_exporter_otlp_proto_common-1.38.0.tar.gz", hash = "sha256:e333278afab4695aa8114eeb7bf4e44e65c6607d54968271a249c180b2cb605c", size = 20431, upload-time = "2025-10-16T08:35:53.285Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/29/48/bce76d3ea772b609757e9bc844e02ab408a6446609bf74fb562062ba6b71/opentelemetry_exporter_otlp_proto_common-1.41.1-py3-none-any.whl", hash = "sha256:10da74dad6a49344b9b7b21b6182e3060373a235fde1528616d5f01f92e66aa9", size = 18366, upload-time = "2026-04-24T13:15:18.917Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/9e/55a41c9601191e8cd8eb626b54ee6827b9c9d4a46d736f32abc80d8039fc/opentelemetry_exporter_otlp_proto_common-1.38.0-py3-none-any.whl", hash = "sha256:03cb76ab213300fe4f4c62b7d8f17d97fcfd21b89f0b5ce38ea156327ddda74a", size = 18359, upload-time = "2025-10-16T08:35:34.099Z" },
]
[[package]]
name = "opentelemetry-exporter-otlp-proto-http"
-version = "1.41.1"
+version = "1.38.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "googleapis-common-protos" },
@@ -3377,48 +3378,48 @@ dependencies = [
{ name = "requests" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/33/5b/9d3c7f70cca10136ba82a81e738dee626c8e7fc61c6887ea9a58bf34c606/opentelemetry_exporter_otlp_proto_http-1.41.1.tar.gz", hash = "sha256:4747a9604c8550ab38c6fd6180e2fcb80de3267060bef2c306bad3cb443302bc", size = 24139, upload-time = "2026-04-24T13:15:42.977Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/81/0a/debcdfb029fbd1ccd1563f7c287b89a6f7bef3b2902ade56797bfd020854/opentelemetry_exporter_otlp_proto_http-1.38.0.tar.gz", hash = "sha256:f16bd44baf15cbe07633c5112ffc68229d0edbeac7b37610be0b2def4e21e90b", size = 17282, upload-time = "2025-10-16T08:35:54.422Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ba/4d/ef07ff2fc630849f2080ae0ae73a61f67257905b7ac79066640bfa0c5739/opentelemetry_exporter_otlp_proto_http-1.41.1-py3-none-any.whl", hash = "sha256:1a21e8f49c7a946d935551e90947d6c3eb39236723c6624401da0f33d68edcb4", size = 22673, upload-time = "2026-04-24T13:15:21.313Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/77/154004c99fb9f291f74aa0822a2f5bbf565a72d8126b3a1b63ed8e5f83c7/opentelemetry_exporter_otlp_proto_http-1.38.0-py3-none-any.whl", hash = "sha256:84b937305edfc563f08ec69b9cb2298be8188371217e867c1854d77198d0825b", size = 19579, upload-time = "2025-10-16T08:35:36.269Z" },
]
[[package]]
name = "opentelemetry-proto"
-version = "1.41.1"
+version = "1.38.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "protobuf" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/99/e8/633c6d8a9c8840338b105907e55c32d3da1983abab5e52f899f72a82c3d1/opentelemetry_proto-1.41.1.tar.gz", hash = "sha256:4b9d2eb631237ea43b80e16c073af438554e32bc7e9e3f8ca4a9582f900020e5", size = 45670, upload-time = "2026-04-24T13:15:49.768Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/51/14/f0c4f0f6371b9cb7f9fa9ee8918bfd59ac7040c7791f1e6da32a1839780d/opentelemetry_proto-1.38.0.tar.gz", hash = "sha256:88b161e89d9d372ce723da289b7da74c3a8354a8e5359992be813942969ed468", size = 46152, upload-time = "2025-10-16T08:36:01.612Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e4/1e/5cd77035e3e82070e2265a63a760f715aacd3cb16dddc7efee913f297fcc/opentelemetry_proto-1.41.1-py3-none-any.whl", hash = "sha256:0496713b804d127a4147e32849fbaf5683fac8ee98550e8e7679cd706c289720", size = 72076, upload-time = "2026-04-24T13:15:32.542Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/6a/82b68b14efca5150b2632f3692d627afa76b77378c4999f2648979409528/opentelemetry_proto-1.38.0-py3-none-any.whl", hash = "sha256:b6ebe54d3217c42e45462e2a1ae28c3e2bf2ec5a5645236a490f55f45f1a0a18", size = 72535, upload-time = "2025-10-16T08:35:45.749Z" },
]
[[package]]
name = "opentelemetry-sdk"
-version = "1.41.1"
+version = "1.38.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "opentelemetry-api" },
{ name = "opentelemetry-semantic-conventions" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/58/d0/54ee30dab82fb0acda23d144502771ff76ef8728459c83c3e89ef9fb1825/opentelemetry_sdk-1.41.1.tar.gz", hash = "sha256:724b615e1215b5aeacda0abb8a6a8922c9a1853068948bd0bd225a56d0c792e6", size = 230180, upload-time = "2026-04-24T13:15:50.991Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/85/cb/f0eee1445161faf4c9af3ba7b848cc22a50a3d3e2515051ad8628c35ff80/opentelemetry_sdk-1.38.0.tar.gz", hash = "sha256:93df5d4d871ed09cb4272305be4d996236eedb232253e3ab864c8620f051cebe", size = 171942, upload-time = "2025-10-16T08:36:02.257Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/b4/e7/a1420b698aad018e1cf60fdbaaccbe49021fb415e2a0d81c242f4c518f54/opentelemetry_sdk-1.41.1-py3-none-any.whl", hash = "sha256:edee379c126c1bce952b0c812b48fe8ff35b30df0eecf17e98afa4d598b7d85d", size = 180213, upload-time = "2026-04-24T13:15:33.767Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/2e/e93777a95d7d9c40d270a371392b6d6f1ff170c2a3cb32d6176741b5b723/opentelemetry_sdk-1.38.0-py3-none-any.whl", hash = "sha256:1c66af6564ecc1553d72d811a01df063ff097cdc82ce188da9951f93b8d10f6b", size = 132349, upload-time = "2025-10-16T08:35:46.995Z" },
]
[[package]]
name = "opentelemetry-semantic-conventions"
-version = "0.62b1"
+version = "0.59b0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "opentelemetry-api" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/9e/de/911ac9e309052aca1b20b2d5549d3db45d1011e1a610e552c6ccdd1b64f8/opentelemetry_semantic_conventions-0.62b1.tar.gz", hash = "sha256:c5cc6e04a7f8c7cdd30be2ed81499fa4e75bfbd52c9cb70d40af1f9cd3619802", size = 145750, upload-time = "2026-04-24T13:15:52.236Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/40/bc/8b9ad3802cd8ac6583a4eb7de7e5d7db004e89cb7efe7008f9c8a537ee75/opentelemetry_semantic_conventions-0.59b0.tar.gz", hash = "sha256:7a6db3f30d70202d5bf9fa4b69bc866ca6a30437287de6c510fb594878aed6b0", size = 129861, upload-time = "2025-10-16T08:36:03.346Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/eb/a6/83dc2ab6fa397ee66fba04fe2e74bdf7be3b3870005359ceb7689103c058/opentelemetry_semantic_conventions-0.62b1-py3-none-any.whl", hash = "sha256:cf506938103d331fbb78eded0d9788095f7fd59016f2bda813c3324e5a74a93c", size = 231620, upload-time = "2026-04-24T13:15:35.454Z" },
+ { url = "https://files.pythonhosted.org/packages/24/7d/c88d7b15ba8fe5c6b8f93be50fc11795e9fc05386c44afaf6b76fe191f9b/opentelemetry_semantic_conventions-0.59b0-py3-none-any.whl", hash = "sha256:35d3b8833ef97d614136e253c1da9342b4c3c083bbaf29ce31d572a1c3825eed", size = 207954, upload-time = "2025-10-16T08:35:48.054Z" },
]
[[package]]
@@ -3897,7 +3898,7 @@ requires-dist = [
{ name = "ruff", marker = "extra == 'dev'", specifier = "==0.14.13" },
{ name = "semantic-version", specifier = ">=2.10.0" },
{ name = "shortuuid", specifier = ">=1.0.13" },
- { name = "temporalio", marker = "extra == 'temporal'", specifier = "==1.23.0" },
+ { name = "temporalio", marker = "extra == 'temporal'", specifier = "==1.24.0" },
{ name = "tomli", specifier = ">=2.3.0" },
{ name = "tomlkit", specifier = ">=0.13.2" },
{ name = "typer", specifier = ">=0.16.0" },
@@ -5583,7 +5584,7 @@ wheels = [
[[package]]
name = "temporalio"
-version = "1.23.0"
+version = "1.24.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "nexus-rpc" },
@@ -5592,13 +5593,13 @@ dependencies = [
{ name = "types-protobuf" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/67/48/ba7413e2fab8dcd277b9df00bafa572da24e9ca32de2f38d428dc3a2825c/temporalio-1.23.0.tar.gz", hash = "sha256:72750494b00eb73ded9db76195e3a9b53ff548780f73d878ec3f807ee3191410", size = 1933051, upload-time = "2026-02-18T17:48:22.353Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/a7/b1/7d9b3104ab7994e7d49e765b92495aaff44810b1e066c874c284a93ebd55/temporalio-1.24.0.tar.gz", hash = "sha256:e534e2e71b4a721193ec4ff3dae521146d093554bd47a64f5605d4ca33e56718", size = 2040485, upload-time = "2026-03-23T15:33:33.638Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/6f/71/26c8f21dca9092201b3b9cb7aff42460b4864b5999aa4c6a4343ac66f1fd/temporalio-1.23.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:6b69ac8d75f2d90e66f4edce4316f6a33badc4a30b22efc50e9eddaa9acdc216", size = 12311037, upload-time = "2026-02-18T17:47:47.628Z" },
- { url = "https://files.pythonhosted.org/packages/ec/47/43102816139f2d346680cb7cc1e53da5f6968355ac65b4d35d4edbfca896/temporalio-1.23.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:1bbbb2f9c3cdd09451565163f6d741e51f109694c49435d475fdfa42b597219d", size = 11821906, upload-time = "2026-02-18T17:47:55.314Z" },
- { url = "https://files.pythonhosted.org/packages/00/b0/899ff28464a0e17adf17476bdfac8faf4ea41870358ff2d14737e43f9e66/temporalio-1.23.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cf6570e0ee696f99a38d855da4441a890c7187357c16505ed458ac9ef274ed70", size = 12063601, upload-time = "2026-02-18T17:48:03.994Z" },
- { url = "https://files.pythonhosted.org/packages/ed/17/b8c6d2ec3e113c6a788322513a5ff635bdd54b3791d092ed0e273467748a/temporalio-1.23.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b82d6cca54c9f376b50e941dd10d12f7fe5b692a314fb087be72cd2898646a79", size = 12394579, upload-time = "2026-02-18T17:48:11.65Z" },
- { url = "https://files.pythonhosted.org/packages/b4/b7/f9ef7fd5ee65aef7d59ab1e95cb1b45df2fe49c17e3aa4d650ae3322f015/temporalio-1.23.0-cp310-abi3-win_amd64.whl", hash = "sha256:43c3b99a46dd329761a256f3855710c4a5b322afc879785e468bdd0b94faace6", size = 12834494, upload-time = "2026-02-18T17:48:19.071Z" },
+ { url = "https://files.pythonhosted.org/packages/84/a9/30517c21d6155bce1c3dc0e420db48da0231230dbc683f40ab6d5fe22b37/temporalio-1.24.0-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:7f11e7b4f4d09bafba499b43188353e23dc128b1fe3f3160014476e3dce70760", size = 12223918, upload-time = "2026-03-23T15:33:05.045Z" },
+ { url = "https://files.pythonhosted.org/packages/73/d0/11aa103bde794524008c1850a84e06cde98698395ca1f8b12e1bd2390aa8/temporalio-1.24.0-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:5cff75a0ca922575b808a7fca1b0de38f6eea061f49e026664b8be9d5bb06ab8", size = 11708887, upload-time = "2026-03-23T15:33:11.67Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/f4/774b56100e6bb94e3757ec96fb5c2bc62d42defc7d6de0ee35a12273827a/temporalio-1.24.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ee7c13b6724dd0c304aa846aecf6da72a8550f4ade40a0a7f6dcc1c92ef35710", size = 12028303, upload-time = "2026-03-23T15:33:18.022Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/91/c05d0e9c2432fe8b1ea0d6fae321866ee49a320ad5e494e6ec9424ca5c28/temporalio-1.24.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aa71b9bfa42f951dd04ade97ce7f92ecedee8903047b4b41b122bb8cbd87a337", size = 12375155, upload-time = "2026-03-23T15:33:24.234Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/97/5c939e4609c164c8690a3b5a135eb828d531de8ef63ff447a2a439c0b0fb/temporalio-1.24.0-cp310-abi3-win_amd64.whl", hash = "sha256:52f6833647eceddbebcc376e2ea663a9f73b2b3a42675f503aeb27c98fd4daeb", size = 12720174, upload-time = "2026-03-23T15:33:30.826Z" },
]
[[package]]
diff --git a/wip/README.md b/wip/README.md
index 89b87c2dc..908bc4477 100644
--- a/wip/README.md
+++ b/wip/README.md
@@ -10,15 +10,15 @@ This folder holds the **active** planning, design, and current-state docs for th
## Tracks
-- **[distributed-execution/](distributed-execution/)** — running MTHDS methods as Temporal workflows across separate worker processes. Its [README](distributed-execution/README.md) is the priority-ordered plan; the folder also holds the tracing/cost-reporting as-built reference, the crate-packaging phases (6a/6b) and crate-first vision, the implemented Temporal IDs/naming design, and the deferred Temporal-exception and schema-reconstruction hardening proposals. **This is the active headline track.**
+- **[distributed-execution/](distributed-execution/)** — running MTHDS methods as Temporal workflows across separate worker processes. Its [README](distributed-execution/README.md) is the priority-ordered plan; the folder also holds the tracing/cost-reporting as-built reference, the crate-packaging phases (6a/6b) and crate-first vision, the implemented Temporal IDs/naming design, the deferred Temporal-exception and schema-reconstruction hardening proposals, the `/temporal-e2e-validate` skill validation status, and the [mistral-workflows/](distributed-execution/mistral-workflows/) sub-track (landing the MISTRAL_NATIVE plugin onto `dev`). **This is the active headline track.**
- **[error-handling/](error-handling/)** — the current-state reference for error handling across Pipelex, track-organized (metadata model, worker classification, extract/classify/render, retry, CLI delivery, Temporal integration, testing). Mostly landed; its [README](error-handling/README.md) lists what's still open.
- **[concurrency/](concurrency/)** — direct-mode batching / backpressure design. The retry-jitter quick win shipped; the remaining design work is split into fan-out scheduling, rate limiting, and batch partial-failure docs. Status: design scoping.
- **[csv-support/](csv-support/)** — reading `.csv` files into typed `ListContent` and writing flat list outputs back out via `--save-csv`. Its [README](csv-support/README.md) is the feature + reviewer reference (codec, core input hook, CLI output hook, the I/O contract, locked decisions); the folder also holds the design rationale and the deferred review/security follow-ups. Landed in #955.
+- **[runtime-bridge/](runtime-bridge/)** — review-triage and deferred-items record for the framework-agnostic `pipelex/runtime_bridge/` extraction (`feature/Runtime-bridge-extraction`, PR #969). Its [README](runtime-bridge/README.md) is the full triage of the SWE-agent review comments. The mechanical fixes and the design-fork resolutions (DIRECT-mode router scoping, the `graph_context`→`trace_context` contract, trace-flush false-positive) have all landed; what remains are the genuinely-deferred items, each with its own doc — notably the half-built-singleton boot race.
## Standalone docs
-- **[signature-based-validation.md](signature-based-validation.md)** — `PipeSignature` contract-only pipes plus strict/lenient (`--allow-signatures`) validation, on `feature/Validate-with-signatures-3`. Landed (Phases 1–7.3, tests green); current-state reference + reviewer verification map. The one open item is the Phase 7.4 cross-repo schema merge-gate (retire the branch-local MTHDS schema rule once `vscode-pipelex` ships the bundled schema).
-- **[temporal-e2e-validate-skill.md](temporal-e2e-validate-skill.md)** — validation status of the `/temporal-e2e-validate` skill and the Step 9 (queue-options) skill bugs patched in its `queue-options-battery.md`. Full dry + live run all green; open follow-ups are the vestigial `act_jinja2_gen_text` removal and a stale Mode 1 "Known xfails" note.
+- **[signature-based-validation.md](signature-based-validation/signature-based-validation.md)** — `PipeSignature` contract-only pipes plus strict/lenient (`--allow-signatures`) validation, on `feature/Validate-with-signatures-3`. Landed (Phases 1–7.3, tests green); current-state reference + reviewer verification map. The one open item is the Phase 7.4 cross-repo schema merge-gate (retire the branch-local MTHDS schema rule once `vscode-pipelex` ships the bundled schema).
- **[webhook-signing.md](webhook-signing.md)** — security track: HMAC body-signing for the completion-callback webhook, with a 3-step cross-repo rollout. The last open piece of the error-handling API-readiness companion work. Plan ready, not started.
- **[runtime-code-fixes.md](runtime-code-fixes.md)** — verified `pipelex/` runtime fixes to pick up.
- **[structured-logging.md](structured-logging.md)** — kickoff briefing for replacing string-interpolated log context with structured logging + contextvars. Kicked off, not started; sequenced after the error-handling stack merges.
@@ -35,4 +35,5 @@ A pickup order grounded in each doc's own status — adjust to current signals.
3. **[webhook-signing.md](webhook-signing.md)** — security-reviewed plan, ready to execute; closes the last open error-handling-companion item. Cross-repo, so coordinate the rollout.
4. **error-handling long tail** — the metadata-model followups and the `request_id` correlation on delivery error paths in [error-handling/](error-handling/). Mostly landed; these are the remainder.
5. **[concurrency/](concurrency/)** — pick up when batch-at-scale becomes a priority. Fan-out scheduling and batch partial-failure are coupled (take together); rate limiting is independent.
-6. **Deferred / larger / future** — [structured-logging.md](structured-logging.md) (after error-handling merges), [pathlib-refactor.workflow.js](pathlib-refactor.workflow.js) (run when ready), [stuffs-as-nodespec.md](stuffs-as-nodespec.md), the deferred Temporal-exception and schema-reconstruction proposals in [distributed-execution/](distributed-execution/), and [blueprint-elaboration-directives.md](blueprint-elaboration-directives.md). Take individually as needs and signals dictate.
+6. **[runtime-bridge/](runtime-bridge/)** — tracks PR #969 (in review). The mechanical fixes and design-fork resolutions have landed; the remaining deferred items (notably the half-built-singleton boot-race hardening) can wait for a human decision.
+7. **Deferred / larger / future** — [structured-logging.md](structured-logging.md) (after error-handling merges), [pathlib-refactor.workflow.js](pathlib-refactor.workflow.js) (run when ready), [stuffs-as-nodespec.md](stuffs-as-nodespec.md), the deferred Temporal-exception and schema-reconstruction proposals in [distributed-execution/](distributed-execution/), and [blueprint-elaboration-directives.md](blueprint-elaboration-directives.md). Take individually as needs and signals dictate.
diff --git a/wip/distributed-execution/README.md b/wip/distributed-execution/README.md
index a5941cadf..edaa948d2 100644
--- a/wip/distributed-execution/README.md
+++ b/wip/distributed-execution/README.md
@@ -14,6 +14,9 @@ This folder is the **distributed-execution track**: running MTHDS methods as Tem
- [temporal-exception-model-revamp.md](temporal-exception-model-revamp.md) — deferred proposal to reparent workflow exceptions onto `ApplicationError`.
- [schema-reconstruction-hardening.md](schema-reconstruction-hardening.md) — deferred hardening of cross-process Pydantic-schema reconstruction.
- [unit-testing-worker-sandbox-validation.md](unit-testing-worker-sandbox-validation.md) — deferred bug: a sandboxed worker can't boot under `--is-unit-testing` because the registered test workflows fail temporalio sandbox validation.
+- [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md) — **OPEN handoff:** how the `_bridge` dev-merge changed the runtime-bridge surface (param renames, deleted graph-assembly primitive, `graph_context→trace_context`, #967/#968) and what the downstream `pipelex-mistralai-workflows` + `_workflows` repos need reconciled. Greps/diagnosis not yet done.
+- [temporal-e2e-validate-skill.md](temporal-e2e-validate-skill.md) — validation status of the `/temporal-e2e-validate` skill and the Step 9 (queue-options) skill bugs patched in its `queue-options-battery.md`. Full dry + live run all green; open follow-ups are the vestigial `act_jinja2_gen_text` removal and a stale Mode 1 "Known xfails" note.
+- [mistral-workflows/](mistral-workflows/) — sub-track for landing the `pipelex-mistralai-workflows` plugin (the MISTRAL_NATIVE backend) onto current `dev`. Its [README](mistral-workflows/README.md) is the hub; verdict is split — land the bridge, hold the `mistralai` 1.x→2.x bump (blocked on instructor PR #2298). Carries the two ready-to-paste keep-alive prompts.
---
diff --git a/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md b/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md
new file mode 100644
index 000000000..c74e53805
--- /dev/null
+++ b/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md
@@ -0,0 +1,104 @@
+# Runtime-bridge changes → sibling-repo reconciliation
+
+**Status: DIAGNOSED — 2026-06-07.** Research + diagnosis are done (this session grepped both sibling repos, checked the dependency pins, and read the reference implementations). No sibling code has been changed yet — the reconciliation is sequenced behind #966 landing (see "Sequenced reconciliation plan"). This doc is now the authoritative breakage inventory; act on it, don't re-investigate.
+
+## TL;DR
+
+`_bridge` (this worktree, branch `feature/Runtime-bridge-extraction`, PR **#966**) is the clean extraction of the framework-agnostic runtime bridge, now reconciled against dev's distributed cost-reporting work (#967 + #968). That reconciliation renamed/removed parts of the bridge's public surface. Two downstream consumers depend on that surface:
+
+- **`pipelex-mistralai-workflows`** (`/Users/lchoquel/repos/Pipelex/pipelex-mistralai-workflows`) — the published package (PyPI `pipelex-mistralai-workflows`, import root `pipelex_mistralai_workflows`). The Mistral Workflows plugin that invokes Pipelex pipes from inside Mistral Workflows activities.
+- **`_workflows`** (`/Users/lchoquel/repos/Pipelex/_workflows`) — a pipelex worktree on branch `feature/Mistral-workflows-merge-4` (the bigger Mistral-Workflows-integration line, PR #954 family).
+
+**The break is deferred, not live.** Both consumers are pinned to / based on a *pre-rename* pipelex, so nothing is broken at this instant. The break materializes the moment either consumer is moved onto post-#966 pipelex. The fix is mechanical: the Mistral-native graph/usage path is a mirror of the Temporal path, and it needs the **exact same** `act_assemble_graph → act_assemble_tracing` rewiring the Temporal path already received in #967 — plus it gains cost reporting it never had.
+
+## Dependency reality (why it's deferred)
+
+- **`pipelex-mistralai-workflows/pyproject.toml`** pins `pipelex` via `[tool.uv.sources]` to a **git rev**: `pipelex = { git = "...pipelex.git", rev = "8f52d40130e3bf6043569b303b7984844272dd4d" }`. The editable `pipelex = { path = "../_workflows", editable = true }` override sits right above it, **commented out**. `[project].dependencies` separately declares `pipelex>=0.27.0`, but the `[tool.uv.sources]` git rev wins for resolution.
+ - Rev `8f52d40` = tip of `feature/Mistral-workflows-merge-3` (`backup/pre-dev-merge`), dated 2026-06-01. It is **pre-rename** (has `pipelex/graph/graph_context.py`, not `trace_context.py`) and is **not an ancestor of `origin/dev` nor of this `_bridge` branch**. So the package currently builds against old pipelex and is internally consistent — it does **not** break until that rev is bumped (or the editable override is uncommented against a reconciled `_workflows`).
+- **`_workflows` (merge-4)** is its own pipelex copy and is also **pre-rename / pre-#967**: it still has `pipelex/graph/graph_context.py` (no `trace_context.py`), has **no** `pipelex/pipe_run/tracing_assembly.py`, and still ships the old `pipelex/runtime_bridge/primitives/graph_assembly.py`. `origin/dev` is **not** an ancestor of merge-4; merge-4 carries ~100+ commits not in `_bridge`. It has the bridge extraction (`runtime_bridge/` with bootstrap/bridge/execution_mode/primitives) but built on a base that predates dev's #967/#968.
+
+Net: the editable override couples the two — when uncommented, the package's `pipelex` comes from `../_workflows`. So the package can only be **correctly fixed and validated** once `_workflows` itself carries the reconciled (post-#966/#967/#968) pipelex. Fixing the package in isolation against an un-merged `_workflows` would be wrong. This dictates the ordering below.
+
+## Breakage inventory — `pipelex-mistralai-workflows` package
+
+All paths under `pipelex_mistralai_workflows/`. Split into hard breaks (won't import/run against post-#966 pipelex) and a parity gap (compiles, but silently drops cost).
+
+### Hard breaks (renamed/removed pipelex surface)
+
+1. **`primitives/act_pipelex_assemble_graph.py`** — imports `from pipelex.runtime_bridge.primitives.graph_assembly import assemble_graph_for_pipeline_run`. **That module + function are deleted** (replaced by `pipelex/pipe_run/tracing_assembly.py::assemble_tracing`). This whole file must be rewritten as `act_pipelex_assemble_tracing.py` (see fix pattern). It defines `AssembleGraphArg` + `act_pipelex_assemble_graph`.
+2. **`primitives/wf_pipe_run.py`** (lines ~32-35, 112-123) — imports `AssembleGraphArg` / `act_pipelex_assemble_graph`, constructs `AssembleGraphArg(...)`, calls the activity, and copies **only** `graph_spec` + `graph_assembly_error` onto `pipe_output`. Must call the new tracing activity, gate on the run's emit flags, and copy all four fields.
+3. **`primitives/wf_pipe_router.py`** (lines ~134, 156-173) — reads `pipe_job.job_metadata.graph_context` and `model_copy(update={"graph_context": ...})`. **`JobMetadata.graph_context` → `trace_context`** (confirmed: `pipelex/pipeline/job_metadata.py:74 trace_context: TraceContext | None`). The per-step tracer-open attributes it reads (`graph_id`, `data_inclusion`, `parent_node_id`) **survive** on `TraceContext` unchanged, so only the container attribute/var name changes here.
+4. **`streaming.py`** (lines ~144, 167) — calls `run_pipe_via_bridge(input_payload, graph_context=graph_context)`. **The kwarg is renamed `graph_context` → `trace_context`** (confirmed: `bridge.py:88 run_pipe_via_bridge(input_payload, trace_context: TraceContext | None = None)`). The local var fed from `tracer_manager.open_tracer(...)` can be renamed for clarity but only the kwarg is load-bearing.
+5. **`primitives/__init__.py`** + **`primitives/registration.py`** — both re-export `act_pipelex_assemble_graph` (and `__init__` the `AssembleGraphArg` symbol via the module). Update to the new activity name when (1) is rewritten.
+
+### Parity gap (no compile break, but wrong post-#967)
+
+6. The Mistral-native path **never assembled usage/cost** — `wf_pipe_run.py` only ever set `graph_spec`. Post-#967 cost rides on `PipeOutput.tokens_usages` (+ `usage_assembly_error`). When you rewrite (1)+(2) to use `assemble_tracing`, you get usage assembly **for free** from the same single event read; copy `tokens_usages` + `usage_assembly_error` onto `pipe_output` too. This is the "Mistral-native gains cost reporting" win, not just a break-fix.
+
+### Confirmed NOT broken (don't touch — already verified to resolve against post-#966 pipelex)
+
+- `streaming_event_forwarder.py`, `streaming.py`, `wf_pipe_router.py`, `pipe_run.py`, `pipe_router.py`, `act_pipelex_leaf.py`, `act_pipelex_deliver.py`, `act_pipelex_flush_trace_events.py`, `scoped_library.py`, `_kajson_codec.py`, `dependency.py`, `activities.py` import these, all of which **survive**: `pipelex.graph.graph_config.DataInclusionConfig`, `pipelex.tracing.event_log_protocol.EventLogProtocol`, `pipelex.graph.graph_tracer_manager.GraphTracerManager`, `pipelex.runtime_bridge.bootstrap.ensure_pipelex_booted`, and the bridge primitives `trace_flush.flush_trace_events_to_backend` / `hydration.hydrate_working_memory` / `pipe_classification.is_controller_pipe` / `submitter_hydration.rehydrate_pipe_output_with_crate` / `delivery.execute_delivery`.
+- **`tests/integration/test_bridge_temporal_*.py`** import `from pipelex.temporal.tprl_pipe.temporal_pipe_router import make_temporal_pipe_router` — that path **still exists** in `_bridge` (only `pipe_run_arg` moved out of `temporal.tprl_pipe` into `runtime_bridge.primitives`). Not a break.
+- The package uses **none** of `is_generate_costs`, `UsageRegistry`, `open_registry`, `close_registry`, `generate_report`, `inject_tokens_usages`, `--cost-report` — so #968's cost-API removals don't hit it directly. Its cost story is purely the additive parity gain in (6).
+
+## The canonical fix pattern (mirror the Temporal path)
+
+The Temporal path already did this exact migration in #967. Use these `_bridge` files as the reference implementation:
+
+- **`pipelex/pipe_run/tracing_assembly.py`** — `assemble_tracing(pipeline_run_id, *, assemble_graph: bool, assemble_usage: bool, domain_code=None, main_pipe_code=None) -> TracingAssembly`. One event read → both artifacts. `TracingAssembly(extra="forbid")` carries `graph_spec` / `graph_assembly_error` / `tokens_usages` / `usage_assembly_error`. Best-effort read failures (incl. `EventLogReadError`) are caught inside and returned on the `*_error` fields; only programming bugs propagate.
+- **`pipelex/temporal/tprl_pipe/act_assemble_tracing.py`** — the activity wrapper. `AssembleTracingArg` adds `assemble_graph: bool` / `assemble_usage: bool` (mirror of the run's emit flags) on top of `pipeline_run_id` / `domain_code` / `main_pipe_code`, and returns `TracingAssembly`. Deliberately **not** wrapped in error-classification — observability, not a pipe step.
+- **`pipelex/temporal/tprl_pipe/wf_pipe_run.py`** (the dispatch, ~lines 87-122) — gates on `trace_context is not None and (trace_context.emit_graph_events or trace_context.emit_usage_events)`, passes `assemble_graph=trace_context.emit_graph_events` / `assemble_usage=trace_context.emit_usage_events`, then copies **all four** populated fields onto `pipe_output`. This is the precise shape `wf_pipe_run.py` in the Mistral package should take.
+
+Mistral-package edits, concretely:
+
+- Rename `act_pipelex_assemble_graph.py` → `act_pipelex_assemble_tracing.py`: `AssembleTracingArg(pipeline_run_id, domain_code, main_pipe_code, assemble_graph, assemble_usage)`; body `return assemble_tracing(...)` returning `TracingAssembly`; keep the Mistral `@activity(...)` decorator.
+- In `wf_pipe_run.py`: gate on the run's emit flags (the Mistral parent has the `PipeJob`, hence `job_metadata.trace_context`; if `trace_context` is None, default both to True to preserve today's always-assemble-graph behavior — or read the config gate the way DIRECT does, decide during impl), call the new activity, copy `graph_spec` / `graph_assembly_error` / `tokens_usages` / `usage_assembly_error`.
+- In `wf_pipe_router.py` + `streaming.py`: the `graph_context` → `trace_context` renames in the breakage inventory above.
+- Update `primitives/__init__.py` + `registration.py` exports.
+
+## `_workflows` (merge-4) reconciliation
+
+merge-4 is a divergent pipelex worktree that has the bridge extraction but predates dev's #967/#968. To carry the reconciled bridge it must absorb dev — the **same** reconciliation `_bridge` just performed, but it also has its own in-tree copies to migrate:
+
+- its `pipelex/runtime_bridge/primitives/graph_assembly.py` must be deleted in favor of `pipe_run/tracing_assembly.py` (the merge with dev surfaces this);
+- `pipelex/graph/graph_context.py` → `trace_context.py` + the `graph_context` → `trace_context` rename across `JobMetadata` and all callers;
+- `is_generate_costs` → `is_generate_usage`; `UsageRegistry` removal; `--cost-report` → `--costs` — all per #967/#968.
+
+Two viable paths; recommend (b):
+
+- **(a)** Merge `origin/dev` into `_workflows` directly and reconcile by hand (re-doing the tracing reconciliation a second time, in a much bigger tree).
+- **(b) Preferred:** land #966 onto dev first, *then* merge the post-#966 dev into `_workflows`. #966 is the already-reconciled bridge extraction, so this folds the bridge reconciliation in once and leaves only the Mistral-Workflows-specific integration code to fix up in merge-4. Confirmed prerequisite: merge-4 is downstream of the bridge work conceptually but **not** an ancestor/descendant of either dev or this branch today — it genuinely needs a merge, not a fast-forward.
+
+## Sequenced reconciliation plan (ordering matters)
+
+1. **Land #966** (this `_bridge` branch) onto `dev`. It carries the reconciled bridge surface (`trace_context`, unified `assemble_tracing`, `EventLogReadError` layering, deleted `graph_assembly` primitive).
+2. **Merge post-#966 `dev` into `_workflows` (merge-4)**, reconciling merge-4's in-tree pipelex copies (graph_assembly deletion, `graph_context→trace_context`, `is_generate_costs→is_generate_usage`, UsageRegistry removal, CLI flag). Path (b) above.
+3. **Fix the `pipelex-mistralai-workflows` package** against the reconciled API — the breakage inventory + canonical fix pattern above. Validate by **uncommenting** the editable `pipelex = { path = "../_workflows", editable = true }` override (now that `_workflows` carries the reconciled pipelex) and running the package's own pyright/mypy/tests + its `tests/integration/test_bridge_temporal_*` and Mistral-native suites.
+4. **Re-pin** the package: replace the editable override with a git rev (or a released `pipelex` version) that includes #966+#967+#968, then re-comment the editable override per the file's own instructions ("Strip this override before publishing... Add it back on the next dev cycle when the next breaking change to `pipelex.runtime_bridge` lands").
+
+Until step 1 happens, steps 2-4 are blocked and **no sibling code should change** — editing either consumer now would only desync it from its current (pre-rename) pipelex.
+
+**Ready-to-run per-repo handoffs** (self-contained, keyed to "#966 has hit dev"; this doc is the index, those are the executables):
+
+- `_workflows` → `/Users/lchoquel/repos/Pipelex/_workflows/wip/post-966-dev-reconciliation-handoff.md`. Key insight there: #966 *is* merge-4's bridge extraction, so once it's on dev merge-4's bridge work is redundant and its only remaining unique value is the mistralai-2.x HOLD group (blocked on `instructor` PR #2298). Recommends retiring merge-4 in favor of a minimal HOLD-group branch off the new dev, rather than a conflict-heavy merge.
+- `pipelex-mistralai-workflows` → `/Users/lchoquel/repos/Pipelex/pipelex-mistralai-workflows/wip/post-966-bridge-reconciliation-handoff.md`. The breakage inventory + canonical fix pattern below, written as concrete edits with code, plus the repin/validate/settle steps. Can pin straight to a post-#966 dev rev — does not require the `_workflows` consolidation.
+
+## The MISTRAL_NATIVE question — ANSWERED
+
+Original open question: did `_run_mistral_native` or the sibling package consume the now-deleted `assemble_graph_for_pipeline_run`?
+
+- **`_run_mistral_native`** (in `_bridge` `pipelex/runtime_bridge/bridge.py:394`) is **thin** — it imports `make_mistral_workflows_pipe_run`, calls `pipe_run.run(...)`, and serializes via `_serialize_completed_output`. It does **not** touch graph/usage assembly. No change needed there.
+- **The sibling package DOES** consume it — `primitives/act_pipelex_assemble_graph.py` imports `assemble_graph_for_pipeline_run` directly and `primitives/wf_pipe_run.py` orchestrates it. So the Mistral-native graph/usage handling needs the identical `act_assemble_graph → act_assemble_tracing` / unified-`assemble_tracing` rewiring the Temporal path got, and additionally **gains** usage/cost assembly it never performed (parity gap (6)).
+
+## Current state of `_bridge` (this worktree)
+
+- Branch `feature/Runtime-bridge-extraction`, worktree `/Users/lchoquel/repos/Pipelex/_bridge`, PR **#966** (#959 is closed).
+- 0 behind `dev`; unpushed commits ahead carry dev's #967 + the merge (tracing reconciliation: `EventLogReadError` layering, `graph_assembly` primitive deletion, `graph_context → trace_context`) + the TODOS retarget.
+- All green at last check: `make agent-check` (pyright clean, mypy clean), unit suite, DIRECT-tracing + `runtime_bridge` integration tests; temporal tracing tests collect clean.
+
+## Pointers / prior art
+
+- `wip/runtime-bridge/` — review-triage of #959/#966/#969 + open deferred follow-ups. README is the cold-start entry. (Superseded the retired `TODOS.md` PR reviewer's guide.)
+- `wip/distributed-execution/tracing-cost-reporting.md` — #967's as-built tracing/cost-reporting design.
+- Memory (auto-loaded): `project_runtime_bridge_pr959_review` (this PR's state), `project_registry_leak_fix` (#967), `project_temporal_not_shipped` (Temporal not in prod).
+- Reference impls in `_bridge` for the Mistral-package fix: `pipelex/pipe_run/tracing_assembly.py`, `pipelex/temporal/tprl_pipe/act_assemble_tracing.py`, `pipelex/temporal/tprl_pipe/wf_pipe_run.py`.
diff --git a/wip/distributed-execution/mistral-workflows/README.md b/wip/distributed-execution/mistral-workflows/README.md
new file mode 100644
index 000000000..0d28e4010
--- /dev/null
+++ b/wip/distributed-execution/mistral-workflows/README.md
@@ -0,0 +1,14 @@
+# Mistral Workflows merge — readiness & plan
+
+Sub-track of [distributed-execution](../README.md): bringing the `pipelex-mistralai-workflows` plugin (Mistral's native Workflows as a Pipelex execution backend) onto current `dev`. This is the MISTRAL_NATIVE execution mode's home outside the core runtime.
+
+## Docs
+
+- [mistral-workflows-merge-readiness.md](mistral-workflows-merge-readiness.md) — **the hub.** Cold-start assessment of whether `feature/Mistral-workflows-merge-4` is safe to PR into `dev`. Verdict: the `runtime_bridge` extraction is clean/additive/green and safe to land; the blocker is a `mistralai` 1.x→2.x bump that drags in a temporary `instructor` git-fork pin on a core dep (blocked on instructor PR #2298). Recommends splitting — land the bridge, hold the SDK bump.
+- [mistral-workflows-plugin-extract.md](mistral-workflows-plugin-extract.md) — design: extracting the plugin surface.
+- [mistral-workflows-sub-module.md](mistral-workflows-sub-module.md) — design: the sub-module layout option.
+- [mistral-workflows-merge-4-refresh-now-prompt.md](mistral-workflows-merge-4-refresh-now-prompt.md) — ready-to-paste prompt to keep `merge-4` alive while held: merge `dev` in now (only `CHANGELOG.md` conflicts; the mistral bump must survive).
+- [mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md](mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md) — ready-to-paste prompt for after the bridge squash-merges to `dev`: rebuild the branch off `dev` and re-apply only the mistral delta, to dodge the squash-merge tangle.
+- [docs-held-back-from-live-site.md](docs-held-back-from-live-site.md) — **re-publish checklist.** The Mistral Workflows user docs ship in the repo but are excluded from the published docs site (so it doesn't advertise the unreleased `pipelex-mistralai-workflows` package). What's held back and how to restore it when the package launches — claim the PyPI name first.
+
+Pairs with [bridge-changes-sibling-repo-reconciliation.md](../bridge-changes-sibling-repo-reconciliation.md) in the parent track, which tracks what the downstream `pipelex-mistralai-workflows` repo needs reconciled after the `_bridge` surface changes.
diff --git a/wip/distributed-execution/mistral-workflows/docs-held-back-from-live-site.md b/wip/distributed-execution/mistral-workflows/docs-held-back-from-live-site.md
new file mode 100644
index 000000000..35ccb3048
--- /dev/null
+++ b/wip/distributed-execution/mistral-workflows/docs-held-back-from-live-site.md
@@ -0,0 +1,42 @@
+# Mistral Workflows user docs — held back from the live site (re-publish on launch)
+
+**Status: HELD BACK on the bridge branch.** The Mistral Workflows user docs are written and live in the repo, but the bridge PR keeps them **out of the published docs site** (docs.pipelex.com) so the site doesn't advertise the `pipelex-mistralai-workflows` package before it exists on PyPI. Re-publish them when the package launches.
+
+## Why
+
+The PyPI distribution name `pipelex-mistralai-workflows` is not registered yet. Printing `pip install pipelex-mistralai-workflows` on a public page invites someone to squat the name. The package also isn't published, so any install instruction would just fail. So the bridge PR holds the user-facing docs back rather than shipping docs for an unreleased package.
+
+The **code** (the bridge entry point, primitives, execution modes) ships in this PR regardless — only the user-facing Mistral Workflows *docs* are held back.
+
+## What is held back
+
+The six pages under `docs/distributed-execution/mistral-workflows/` (`index`, `installation`, `your-first-pipelex-workflow`, `execution-modes`, `streaming`, `choosing-a-backend`). They still exist in the repo; they are excluded from the MkDocs build via `exclude_docs:` in `mkdocs.yml`, so they are neither served nor in the nav. The parent pages were softened to "coming soon" and de-linked from them (otherwise the `mkdocs build --strict` in `doc-check.yml` fails on links to excluded pages).
+
+## Prerequisite before re-publishing
+
+1. **Claim the name on PyPI first** — register `pipelex-mistralai-workflows` (a `0.0.0` placeholder release is enough) so re-publishing the docs no longer advertises an unclaimed name. This is the real squatting fix; the doc hold is only defense-in-depth (the name already appears across the public repo).
+2. The package must actually publish, so `pip install pipelex-mistralai-workflows` resolves.
+
+## Re-publish checklist (reverse of the hold)
+
+In `pipelex` (wherever the bridge has landed — `dev`, then merged into `_workflows`):
+
+- **`mkdocs.yml`** — delete the `exclude_docs:` block for `distributed-execution/mistral-workflows/`, and restore the `Mistral Workflows:` nav subsection (Overview / Installation & Preview Status / Your First Pipelex Workflow / Execution Modes / Streaming Progress / Choosing a Backend) under `Distributed Execution`.
+- **`docs/distributed-execution/index.md`** — restore the two-backend overview (Temporal + Mistral Workflows, with the links and the "what's shared" framing) instead of the Temporal-only + "coming soon" version.
+- **`docs/features/distributed-execution.md`** — restore the second backend bullet (link + name) and the "Choosing a Backend" link; restore the Mistral mention in the "Get started" line.
+- **`docs/distributed-execution/temporal/index.md`** — restore the "one of two backends … see Pipelex on Mistral Workflows / Choosing a Backend" cross-links.
+- **`docs/under-the-hood/pipe-routing-and-execution.md`** — restore the two-host-runtime framing and the `See [Execution Modes]` link.
+- **`docs/reliability/durable-execution.md`** — restore the two-backend wording and the Mistral / Choosing-a-Backend links.
+- **`README.md`** — restore the real install instruction (`pip install pipelex-mistralai-workflows`) in the "Mistral Workflows orchestration" note, replacing the "coming soon / install details follow" placeholder.
+- **`CHANGELOG.md`** — restore the install + import-path guidance (`pip install pipelex-mistralai-workflows`, import from `pipelex_mistralai_workflows.*`) in the "Mistral Workflows integration extracted" entry.
+
+After restoring, run `mkdocs build --strict` (matches `doc-check.yml`) and confirm the Mistral pages build and the nav renders.
+
+## Where the hold was done (to diff/revert against)
+
+On the bridge branch (`feature/Runtime-bridge-extraction`, PR #969):
+
+- de-advertising README/CHANGELOG (removed the `pip install` lines): commit `e8dfa0e2`.
+- unpublishing the docs (mkdocs `exclude_docs` + nav removal + parent-page softening): the commit that adds this file.
+
+`git show <sha>` on those two commits gives the exact reverse diff for the checklist above.
diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md
new file mode 100644
index 000000000..8361abb17
--- /dev/null
+++ b/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md
@@ -0,0 +1,27 @@
+# Prompt — rebuild `feature/Mistral-workflows-merge-4` after the runtime-bridge PR squash-merges to `dev` (use later)
+
+Paste the block below into a fresh Claude Code session in `/Users/lchoquel/repos/Pipelex/_workflows`. **Only use this once `feature/Runtime-bridge-extraction` (PR #959) has landed on `dev`.** Until then, keep the branch fresh with `wip/mistral-workflows-merge-4-refresh-now-prompt.md` instead.
+
+---
+
+Background: `feature/Mistral-workflows-merge-4` is a long-lived branch = **`dev` + a held-back `mistralai` 1.x→2.x SDK bump** (held on `instructor` PR #2298; see `wip/mistral-workflows-merge-readiness.md`). It carries its **own, original, unsquashed** copy of the runtime-bridge commits (`71de6fb1`, `1a0f5dfa`, the `runtime_bridge` extraction, etc.).
+
+The runtime-bridge work has now squash-merged into `dev` as a **single** commit — so `dev` and `merge-4` both contain the bridge, but as **different SHAs with diverged content** (dev has the review-followed-up version; merge-4 has the older one). A plain `git merge dev` would now conflict hard across every `pipelex/runtime_bridge/**` file — the classic "long-lived branch outlives the squash-merge of its own content" tangle. **Do not try to merge through that.**
+
+Instead, **rebuild the branch fresh off `dev` and re-apply only the mistral delta.** The branch's *unique* content is tiny — the mistral provider files plus the dependency bump; everything else (including the now-canonical, review-followed-up `runtime_bridge`) comes for free from `dev`.
+
+Please do this:
+
+1. `git fetch origin`. Confirm the bridge actually landed on `dev`: `git ls-tree origin/dev pipelex/runtime_bridge` should be non-empty. If it's empty, **stop** — the bridge hasn't landed; use the refresh prompt instead.
+2. Preserve the old branch: rename `feature/Mistral-workflows-merge-4` → `archive/Mistral-workflows-merge-4-presquash` (or just leave it and branch anew). Don't delete it until the rebuild is verified green.
+3. Create the new branch off `origin/dev` (call it `feature/Mistral-workflows-merge-5`, or reset `-4` to `origin/dev` if you prefer to keep the name).
+4. Re-apply **only the mistral delta**. The authoritative file list is the "Hold on the branch" section of `wip/mistral-workflows-merge-readiness.md` — read it, don't trust the list below blindly. As of writing it is:
+ - **source:** `pipelex/plugins/mistral/{mistral_config,mistral_extract_worker,mistral_factory,mistral_llm_worker,mistral_llms}.py` (includes the `mistralai.client.*` 2.x import reorg and the `message is None` retry guard in `mistral_llm_worker.py`).
+ - **tests:** `tests/unit/pipelex/plugins/{test_plugin_pipelex_storage_images,test_transport_retry_wiring}.py` and `tests/unit/pipelex/plugins/mistral/{test_mistral_worker_error_handling,test_extract_mistral_metadata,test_mistral_llm_worker_object_error_handling,test_mistral_extract_worker_semantic,test_mistral_reasoning}.py`.
+ - Bring these over with `git checkout <old-merge-4-branch> -- <files…>`. Reconcile by hand if `dev` changed any of them in the meantime.
+5. Re-apply the **dependency bump** to `pyproject.toml`: `mistralai>=2.4.4` and the `[tool.uv.sources] instructor = { git = "https://github.com/Ian321/instructor.git", rev = "…" }` fork pin (keep the `# Temporary: …` comment referencing instructor #2298). Then **regenerate the lockfile fresh** with `uv lock` — do **not** copy the old `uv.lock`; let it resolve against current `dev`.
+6. Run `make agent-check` and `make agent-test`; both must be green.
+7. Confirm the held-back state is correct: `uv.lock` resolves `mistralai` 2.x + `instructor` from the git rev, and the new branch's diff vs `dev` is *just* the mistral provider files + the two dependency files (no stale bridge churn). That small, crisp diff is exactly what becomes the PR once `instructor` ships mistralai-2.x support on PyPI.
+8. Do not push or open a PR unless I ask — but do report the final diff-vs-`dev` summary so I can sanity-check the delta.
+
+After this, update `wip/mistral-workflows-merge-readiness.md` and `wip/README.md` to reflect the new branch name and that the bridge is now upstream on `dev`.
diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-refresh-now-prompt.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-refresh-now-prompt.md
new file mode 100644
index 000000000..ff7ed8259
--- /dev/null
+++ b/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-refresh-now-prompt.md
@@ -0,0 +1,27 @@
+# Prompt — refresh `feature/Mistral-workflows-merge-4` from `dev` (use now)
+
+Paste the block below into a fresh Claude Code session in `/Users/lchoquel/repos/Pipelex/_workflows`.
+
+---
+
+We keep `feature/Mistral-workflows-merge-4` alive as a long-lived branch = **`dev` + a held-back `mistralai` 1.x→2.x SDK bump** (held because `instructor` has no PyPI release that supports mistralai 2.x — blocked on instructor PR #2298). The full rationale is in `wip/mistral-workflows-merge-readiness.md`; read its TL;DR before touching dependencies.
+
+I want to bring `feature/Mistral-workflows-merge-4` up to date with the latest `dev`. Please:
+
+1. `git fetch origin`, then check out `feature/Mistral-workflows-merge-4` (confirm the working tree is clean first).
+2. Merge `origin/dev` into it.
+
+**What to expect — heads up:**
+
+- The merge conflicts in **`CHANGELOG.md` only**. It's the routine "both sides added a `[Unreleased]` entry" conflict — resolve by keeping both sets of bullets (union), no content is lost. Nothing else should conflict.
+- **Do NOT merge `feature/Runtime-bridge-extraction` into this branch.** That branch's tip pins `mistralai 1.12.0` in the lockfile, which would conflict with — and risk silently reverting — this branch's whole reason to exist (the mistralai 2.4.4 bump). `dev` is the only correct merge source. (The runtime-bridge work isn't on `dev` yet anyway, and this branch already carries its own copy.)
+- **The mistral SDK bump must survive untouched.** After resolving, verify the held-back dependency state is still intact:
+ - `pyproject.toml` still has `mistralai>=2.4.4` and the `[tool.uv.sources] instructor = { git = "https://github.com/Ian321/instructor.git", rev = "…" }` fork pin.
+ - `uv.lock` still resolves `mistralai` 2.x (not 1.12.x) and `instructor` from the git rev.
+ - `pipelex/plugins/mistral/*.py` keep the `mistralai.client.*` 2.x import paths.
+ - `dev` brings no mistral dependency changes, so none of the above should have moved — but check, because this is the one thing we must not lose.
+
+3. Run `make agent-check` and `make agent-test`; both must be green before you finish.
+4. Do **not** push or open a PR — this is just keeping the branch fresh. Leave it on the merged commit and summarize what came in from `dev` (e.g. v0.31.0 release, #960, #962, #963).
+
+If anything other than `CHANGELOG.md` conflicts, stop and show me — that would mean the topology changed (e.g. the bridge already landed on `dev`), and the right move is then the rebuild path in `wip/mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md`, not a plain merge.
diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-readiness.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-readiness.md
new file mode 100644
index 000000000..27212e8a6
--- /dev/null
+++ b/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-readiness.md
@@ -0,0 +1,149 @@
+# Mistral Workflows — merge-to-`dev` readiness assessment
+
+**Branch:** `feature/Mistral-workflows-merge-4` · **Assessed:** 2026-06-03 · **HEAD:** `ff15bd58` · **dev tip / merge-base:** `fa15d15c`
+
+Cold-start reference for the decision: *is this branch safe to PR into `dev`?* Verified against the actual code. **This doc is the merge basis.** Companion design docs: [`mistral-workflows-plugin-extract.md`](mistral-workflows-plugin-extract.md), [`mistral-workflows-sub-module.md`](mistral-workflows-sub-module.md). Runtime-bridge review-triage + open follow-ups: [`../../runtime-bridge/README.md`](../../runtime-bridge/README.md).
+
+## TL;DR verdict
+
+The **`runtime_bridge` extraction is genuinely solid** — clean, additive, behavior-neutral, fully green — and safe to merge to `dev` on its own. The branch's problem is that it **couples that clean work to a Mistral SDK major bump (`mistralai` 1.x → 2.x) that drags in a temporary `instructor` git-fork pin on a *core* dependency**, blocked on an upstream PR that is still open with no release path. That dependency state is **not release-grade** and is the only reason not to merge the whole branch as-is.
+
+**Recommendation: split.** Land the bridge now; hold the `mistralai` 2.x bump + `instructor` fork pin until `instructor` ships PyPI support for mistralai 2.x. **The safe branch `feature/Runtime-bridge-extraction` was built off `dev` and is green on 2026-06-03** (`agent-check` + `agent-test` pass; mistralai held at 1.x, temporalio 1.24 included, the `message is None` guard left with the HOLD group). Details in [Recommendation](#recommendation).
+
+## Scope reality (read this first)
+
+The branch *looks* huge — `dev..HEAD` lists ~50 commits and `git diff --stat dev...HEAD` is 138 files / ~19k insertions — but most of that is **already in `dev`** or is docs:
+
+- The big **Temporal-primitives + text-then-object / PipeStructure** work already landed in `dev` as the squash-merge **#891** ("Temporal merge 3: distributed execution + PipeStructure + content generation collapse"). This branch carries the *original unsquashed commits* (different SHAs), so they show in `dev..HEAD` but **net out** in the diff.
+- Because the merge-base equals `dev`'s tip (`fa15d15c`), `git diff dev...HEAD` and `git diff dev HEAD` are identical — both show the true net add-on-top-of-dev.
+- **Net source diff vs `dev` is small: 26 files / +717 −95 in `pipelex/`.** The rest is additive docs (`docs/`, error pages), the vendored `.claude/skills/workflows/` reference set, and `wip/` planning docs.
+
+So this is NOT "merging the whole Temporal + TTO + Mistral epic." It is "merging the `runtime_bridge` extraction + a Mistral SDK bump + additive docs."
+
+## What's verified green ✅
+
+Run on HEAD `ff15bd58`, working tree clean before and after:
+
+- **`make agent-check`** → ruff + plxt (TOML/MTHDS) + **pyright 0 errors** + **mypy clean (2007 source files)**.
+- **`make agent-test`** → full non-inference suite, `-n auto`, markers `(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api` → **all passed**. This *includes* unit + every key-free integration test: the `runtime_bridge` DIRECT integration test (`tests/integration/pipelex/runtime_bridge/test_bridge_direct.py`) and the in-process Temporal tests.
+- `runtime_bridge` has **no `TODO`/`FIXME`/`NotImplementedError` stubs**; ~1,124 lines of new bridge tests (unit + integration).
+- **Not covered by `agent-test`** (key-gated, can't run locally): a **live Mistral inference + OCR round-trip**. The mistralai 2.x bump is verified at type/import level and by the non-inference suite, but not end-to-end against the real API. Run this with a real key before relying on it in production.
+
+## Quality of the net change
+
+Clean. Highlights:
+
+- **`pipelex/runtime_bridge/`** — framework-agnostic core so any host runtime (not just Mistral Workflows) can embed Pipelex. `bridge.py` (boundary types + `run_pipe_via_bridge` dispatch), `bootstrap.py` (`ensure_pipelex_booted`), `execution_mode.py` (`PipelexExecutionMode`), `exceptions.py`, and a `primitives/` subpackage (`delivery`, `graph_assembly`, `hydration`, `pipe_classification`, `submitter_hydration`, `trace_flush`, `pipe_run_arg`). Exhaustive `match/case`, enum `@property` predicates (respects the "never `==` an enum" rule), strong docstrings.
+- **Temporal activities became thin wrappers** delegating to `runtime_bridge.primitives` — e.g. `act_assemble_graph` is now a ~4-line `@activity.defn` over `assemble_graph_for_pipeline_run(...)`; the body moved verbatim into the primitive. **Behavior-neutral lift** (the CHANGELOG entry asserts the same). Touched: `temporal/tprl_pipe/{act_assemble_graph,act_deliver,act_flush_trace_events,temporal_pipe_router,temporal_pipe_run,wf_pipe_router,wf_pipe_run}.py` + `pipe_run/delivery_executor.py`.
+- **`MISTRAL_NATIVE` is the safe preview shape.** Inside `pipelex` it is an additive enum value whose dispatch branch (`bridge.py::_run_mistral_native`) does a *deferred import* of `pipelex_mistralai_workflows.primitives.pipe_run` and immediately raises **`MissingMistralWorkflowsPluginError`** (with a `pip install pipelex-mistralai-workflows` hint) if the external package is absent. No Mistral-specific logic lives in `pipelex`. **Merging it does not activate any preview behavior for existing users**; the `DIRECT` / `TEMPORAL_*` paths are untouched. The actual Mistral-native decomposition lives in the separate `pipelex-mistralai-workflows` repo.
+
+The CHANGELOG `[Unreleased]` "Changed" bullet frames this correctly as a pure extraction + promotion of `runtime_bridge` with **no behavior change**.
+
+## The blocker: dependency coupling ⚠️
+
+This is the "something bad" risk for `dev`. All in `pyproject.toml` / `uv.lock`:
+
+- `mistralai>=1.12.0` → **`mistralai>=2.4.4`** — a **major bump of the live Mistral LLM + OCR provider**. Forces the import reorg across `pipelex/plugins/mistral/*` (`mistralai.models` → `mistralai.client.models`, `mistralai.client.errors`, etc.). A genuine robustness fix rides along (a `message is None` guard in `mistral_llm_worker.py` that raises a transient/retryable error).
+- **`[tool.uv.sources] instructor = { git = "https://github.com/Ian321/instructor.git", rev = "4ea22d2396ca35514929f27faed866115e8b3583" }`** — marked *"Temporary."* `instructor` is a **core base dependency** (`pyproject.toml` line ~28, `instructor>=1.13`), so the fork pins the **entire lockfile**, not just the mistralai extra. `uv.lock` resolves `instructor 1.15.1` from that git rev.
+- `temporalio==1.23.0` → `temporalio==1.24.0` — minor pinned bump, low-risk, independent.
+
+Why the fork pin matters (checked 2026-06-03):
+
+- `instructor` PR **#2298 "support mistralai v2" is OPEN, not merged** (ready-for-review since 2026-05-06). There is **no PyPI `instructor` release that works with mistralai 2.x** — it's a known-broken combo (`instructor` issue **#2137**: mistralai 2.0.0 broke `from mistralai import Mistral`, which also breaks instructor for *other* providers when mistral is installed).
+- So the fork pin is **load-bearing with no near-term removal path** — it depends on an external maintainer merging and releasing.
+- **Published-package knock-on:** `[tool.uv.sources]` is a uv-workspace concept and is **not** baked into the published wheel's dependency metadata. It only saves *this repo's* lockfile/CI. A PyPI consumer running `pip install pipelex[mistralai]` would get `mistralai>=2.4.4` + PyPI-`instructor` (no 2.x support) → **broken structured-output combo**. So bumping the published `mistralai` floor couples pipelex's release health to instructor's, and should not reach a release until instructor publishes 2.x support.
+
+## Recommendation
+
+`runtime_bridge` has **zero `import mistralai`** — it is fully decoupled from the SDK bump. Two viable paths:
+
+### Option A — split (preferred)
+
+**Land on `dev` now** (additive, behavior-neutral, fully green):
+
+- `pipelex/runtime_bridge/**` (incl. `primitives/`)
+- Temporal rewiring: `pipelex/temporal/tprl_pipe/{act_assemble_graph,act_deliver,act_flush_trace_events,temporal_pipe_router,temporal_pipe_run,wf_pipe_router,wf_pipe_run}.py`, `pipelex/pipe_run/delivery_executor.py`
+- The `runtime_bridge` tests, the additive `docs/` (error pages + `docs/distributed-execution/mistral-workflows/`), the `.claude/skills/workflows/` reference set
+- The `MISTRAL_NATIVE` enum + deferred-dispatch + `MissingMistralWorkflowsPluginError` (harmless without the SDK bump — it just fails fast)
+- The CHANGELOG "Mistral Workflows extracted" bullet
+- The `temporalio==1.24.0` bump (verified independent of the SDK work; carried via `uv lock --upgrade-package temporalio` off dev's lock so mistralai stays pinned at 1.12.4)
+
+**Hold on the branch** until `instructor` ships PyPI mistralai-2.x support (track #2298):
+
+- `pyproject.toml` lines: `mistralai>=2.4.4` and the `[tool.uv.sources] instructor` fork pin. (`temporalio==1.24.0` is separable — it landed in the safe PR.)
+- the coupled `uv.lock` changes
+- **Mistral provider — source AND tests** (all carry the `mistralai.client.*` import reorg, which is required by 2.x):
+ - source: `pipelex/plugins/mistral/{mistral_config,mistral_extract_worker,mistral_factory,mistral_llm_worker,mistral_llms}.py`
+ - tests: `tests/unit/pipelex/plugins/{test_plugin_pipelex_storage_images,test_transport_retry_wiring}.py` and `tests/unit/pipelex/plugins/mistral/{test_mistral_worker_error_handling,test_extract_mistral_metadata,test_mistral_llm_worker_object_error_handling,test_mistral_extract_worker_semantic,test_mistral_reasoning}.py`
+- The **`message is None` guard** in `mistral_llm_worker.py` — see the correction below; it is 2.x-coupled and must travel with the bump.
+
+**Separability verified, and the safe branch built ✅ (2026-06-03).** Two passes. First, a throwaway check: reverted the complete HOLD group to dev (mistralai → 1.12.4, temporalio → 1.23.0, instructor → PyPI, provider source + seven tests), re-synced, `agent-check` + `agent-test` green. Then the real deliverable: branch **`feature/Runtime-bridge-extraction`** off `dev` (soft-reset so the net diff is one staged changeset), reverted the HOLD group, kept `temporalio==1.24.0`, regenerated the lock off dev's (mistralai held at 1.12.4), and re-ran both gates → **`agent-check` 0 errors / 2007 files, `agent-test` all passed.** The safe part — `runtime_bridge` (zero `import mistralai`) + Temporal rewiring + `MISTRAL_NATIVE` deferred dispatch + additive docs/tests — is provably independent of the SDK bump and the fork pin.
+
+**Two findings the build earned:**
+
+1. The HOLD group is wider than the provider *source* — the seven provider *test* files above were migrated to the 2.x `mistralai.client.*` paths and fail `pyright` (`reportMissingImports`) under mistralai 1.x, so they must move with the bump too.
+2. **The `message is None` guard is NOT SDK-independent** (my earlier claim was wrong). Under mistralai 1.x, `response.choices[0].message` is typed non-optional (`AssistantMessage`), so `pyright` rejects the guard as `reportUnnecessaryComparison`. It only becomes valid under 2.x, where `message` is `AssistantMessage | None`. So the guard **stays with the HOLD group** — do not cherry-pick it into the safe PR.
+
+### Option B — merge whole, with a gate
+
+Acceptable **only if `dev` is treated as a pure integration branch**: merge as-is, but add an explicit gate that **this must not reach `main` / a release while the `instructor` fork pin exists**, plus a tracking issue tied to instructor #2298. Given the pin is blocked on someone else's open PR with no ETA, prefer Option A.
+
+## Housekeeping
+
+- **`../../../TODOS.md` — refreshed 2026-06-03 (done).** Its "End state delivered" had predated the `runtime_bridge/primitives/` subpackage and the `MISTRAL_NATIVE` mode, and implied no mistral-missing exception existed (the old `MistralWorkflowsNotInstalledError` was deleted, but the new `MissingMistralWorkflowsPluginError` replaced it). The runtime_bridge end-state bullet + the Temporal-rewiring bullet were corrected, and a banner now points here as the merge basis. If `runtime_bridge`'s surface changes again, update both files.
+- **`mistralai-workflows` companion version** lives in the separate `pipelex-mistralai-workflows` repo (TODOS notes `mistralai-workflows==3.4.0` there) — out of scope for this pipelex-side merge decision.
+
+### Tracking issue for the `instructor` fork pin (to file)
+
+The fork pin must not silently rot. In-repo anchors today: this doc + the `# Temporary: …` comment above `[tool.uv.sources] instructor` in `pyproject.toml` (which already references #2298). The durable external tracker is a GitHub issue — none exists yet (`gh issue list --search instructor` was empty on 2026-06-03). Draft to file against `Pipelex/pipelex`:
+
+```
+Title: chore(deps): drop temporary `instructor` git-fork pin once mistralai 2.x lands on PyPI
+
+The `mistralai` 1.x→2.x bump (feature/Mistral-workflows-merge-4) needs `instructor` to
+support mistralai v2, which is NOT yet released on PyPI. Stopgap in pyproject.toml:
+
+ [tool.uv.sources]
+ instructor = { git = "https://github.com/Ian321/instructor.git",
+ rev = "4ea22d2396ca35514929f27faed866115e8b3583" }
+
+`instructor` is a CORE dependency, so this pins the entire lockfile (uv.lock resolves
+instructor 1.15.1 from that git rev). Published-package knock-on: `[tool.uv.sources]` is
+not in the wheel metadata, so `pip install pipelex[mistralai]` from PyPI gets mistralai 2.x
++ PyPI-instructor (no 2.x support) → broken structured-output combo. Must not reach a
+release while the pin exists.
+
+Blocked on:
+- instructor PR #2298 "support mistralai v2" — https://github.com/567-labs/instructor/pull/2298 (open as of 2026-06-03)
+- instructor issue #2137 — https://github.com/567-labs/instructor/issues/2137
+
+Removal steps (once a PyPI instructor release supports mistralai 2.x):
+1. Delete the `[tool.uv.sources] instructor = {...}` block in pyproject.toml.
+2. Raise the `instructor>=…` floor to the first release with mistralai-2.x support.
+3. `uv lock`; confirm instructor resolves from PyPI (not the git rev) in uv.lock.
+4. `make agent-check && make agent-test`; run a live Mistral inference + OCR round-trip.
+
+Context: wip/mistral-workflows-merge-readiness.md
+```
+
+File it by saving the body above to a file and running `gh issue create --repo Pipelex/pipelex --title '…' --body-file <path>`. Once filed, add the issue number to the `# Temporary: …` comment in `pyproject.toml` so the pin and the tracker cross-reference.
+
+## Cold-start: re-verify in a fresh session
+
+```bash
+cd /Users/lchoquel/repos/Pipelex/_workflows
+git rev-parse --short HEAD # expect ff15bd58 (or later)
+git merge-base HEAD dev | cut -c1-8 # dev tip; == merge-base means dev fully merged in
+git diff --stat dev...HEAD -- 'pipelex/**/*.py' # the real net source diff (~26 files)
+grep -n "instructor\|mistralai\|temporalio" pyproject.toml # the dep coupling
+make agent-check # ruff/plxt/pyright/mypy
+make agent-test # full non-inference suite
+```
+
+To confirm `MISTRAL_NATIVE` is still a no-op-without-the-package boundary: `pipelex/runtime_bridge/bridge.py::_run_mistral_native` should deferred-import `pipelex_mistralai_workflows` and raise `MissingMistralWorkflowsPluginError`.
+
+## Sources
+
+- instructor PR #2298 — <https://github.com/567-labs/instructor/pull/2298> (open, not merged as of 2026-06-03)
+- instructor issue #2137 — <https://github.com/567-labs/instructor/issues/2137>
+- mistralai on PyPI — <https://pypi.org/project/mistralai/>
diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-plugin-extract.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-plugin-extract.md
new file mode 100644
index 000000000..fed612968
--- /dev/null
+++ b/wip/distributed-execution/mistral-workflows/mistral-workflows-plugin-extract.md
@@ -0,0 +1,215 @@
+# Mistral Workflows ↔ Pipelex — Extract as a Real Mistral Plugin
+
+Self-contained plan. Sibling document `mistral-workflows-sub-module.md` (the
+former `TODOS.md`) records what was built in-tree under
+`pipelex/plugins/mistralai_workflows/` during Phases 1.x–2.1. That work is the
+*input* to this project, not background to redo.
+
+## 1. Why
+
+What we shipped is the **functional equivalent** of a Mistral Workflows plugin
+but packaged as an optional extra of `pipelex` (`pipelex[mistralai-workflows]`,
+imported from `pipelex.plugins.mistralai_workflows.*`). Mistral defines a
+plugin as a **standalone, pip-installable package** that depends on
+`mistralai-workflows>=2.0.0` and exposes activities / workflows / dependencies
+under its own top-level package (the `mistralai.workflows.plugins.*`
+namespace is reserved for Mistral-supported packages).
+
+To match that contract we need to:
+
+1. Stop shipping Mistral-specific code inside the `pipelex` distribution.
+2. Ship a separate distribution (PyPI: `pipelex-mistralai-workflows`,
+ Python package: `pipelex_mistralai_workflows`) that `pip install`s
+ `pipelex` and `mistralai-workflows>=3.3.0` and re-exports the same
+ activities, with a Mistral-style "component" / dependency wrapper.
+3. Keep the framework-agnostic embedding core (`bridge.py`,
+ `execution_mode.py`, `bootstrap.py`, `exceptions.py`) reachable from
+ *any* host, not just Mistral — so other durable runtimes (raw Temporal,
+ future plugins) can reuse it.
+
+## 2. Goals & non-goals
+
+**Goals**
+
+- Pure-Python `pipelex` distribution: no `mistralai-workflows` extra, no
+ `pipelex/plugins/mistralai_workflows/` directory.
+- New repo `pipelex-mistralai-workflows` scaffolded from
+ `pipelex-starter-python` and adapted for a library (not an app), with
+ identical lint/type/test toolchain to the rest of the workspace.
+- Public surface preserved: a user who today writes
+ `from pipelex.plugins.mistralai_workflows.activities import pipelex_run_pipe`
+ has a clear, mechanical migration to the new import path.
+- Mistral-idiomatic ergonomics: a `pipelex_dependency` (or equivalent
+ component) so workers wire Pipelex the same way they wire
+ `mistralai_chat_complete` today.
+- CI on the new repo runs the integration test layers that need the
+ optional dep (currently layer-2 / layer-3 in `mistral-workflows-sub-module.md` §3).
+- Docs split cleanly: embedding-core docs stay in pipelex; Mistral-specific
+ recipes move to the new repo's docs.
+
+**Non-goals**
+
+- Re-doing Phase 1.x / Phase 2.x design. The behavior, boundary types,
+ execution modes, streaming semantics, and gotchas are all locked in —
+ see `mistral-workflows-sub-module.md` §2 and §4. Treat them as spec.
+- Goal 1 from the original doc (porting Pipelex orchestration to run *on*
+ Mistral Workflows). Still out of scope.
+- Backwards-compatibility shims in `pipelex` for the old import path. Per
+ project rule (CLAUDE.md "No backward compatibility"), we just change it
+ and note the migration in the changelog.
+
+## 3. Decision points to resolve before coding
+
+A follow-up agent should not start until these are answered. Each has a
+proposed default; flag any that need a human call.
+
+1. **Where does the framework-agnostic core live?**
+ - Option A *(recommended default)*: keep `bridge.py`,
+ `execution_mode.py`, `bootstrap.py`, `exceptions.py` inside `pipelex`,
+ promoted out of `plugins/mistralai_workflows/` into something like
+ `pipelex/embedding/` (name TBD). The new plugin pkg imports from there.
+ Pros: any host (raw Temporal, future plugins) can reuse it; smaller
+ surface to maintain in the new repo.
+ - Option B: move *everything* into `pipelex-mistralai-workflows`. Pros:
+ `pipelex` stays leaner. Cons: any future host has to either depend on
+ the Mistral plugin pkg (wrong) or re-implement.
+ - This decision drives the rest of the file moves.
+
+2. **Package + distribution names.**
+ - PyPI: `pipelex-mistralai-workflows` (proposed). Confirm naming aligns
+ with other Pipelex packages on PyPI.
+ - Top-level Python package: `pipelex_mistralai_workflows`. The
+ `mistralai.workflows.plugins.*` namespace is reserved, so we cannot
+ squat there.
+ - GitHub repo: `pipelex-mistralai-workflows` under the existing org,
+ side-by-side with the other repos listed in the workspace `CLAUDE.md`.
+
+3. **Versioning & release cadence.**
+ - Pin a minimum `pipelex` version per release. Decide whether to track
+ `pipelex` major versions 1:1 or use independent SemVer.
+ - Decide whether the new repo follows the same release skill / version
+ conventions as `pipelex` (CHANGELOG.md format, `release/vX.Y.Z`
+ branches, etc.). Default: yes, identical.
+
+4. **Mistral component wrapper shape.**
+ - Mistral's existing plugins ship a "dependency" (e.g.
+ `mistralai_chat_complete`) that workers register. We should ship at
+ least one — likely a wrapper around `ensure_pipelex_booted()` plus a
+ `LibraryCrate` snapshot. Final shape needs a quick read of how
+ `mistralai.workflows.plugins.mistralai` exposes its dependency before
+ committing.
+
+5. **Cookbook example.**
+ - The deferred Phase 1.3 cookbook entry (`pipelex-cookbook/examples/c_advanced/mistral-workflows/`)
+ should land *after* the new package is on PyPI, importing from the new
+ path. Coordinate timing.
+
+## 4. Workstreams
+
+Three streams, parallelizable once §3 is resolved. Each has its own
+follow-up agent / PR.
+
+### Stream A — Refactor inside `pipelex`
+
+In this repo (`_mistral/`, eventually merged back). High-level only; the
+follow-up agent figures out the file moves once §3.1 is decided.
+
+- Lift the framework-agnostic core to its new home (Option A) or remove it
+ entirely (Option B).
+- Delete the Mistral-specific modules (`activities.py`, `streaming.py`,
+ `streaming_event_forwarder.py`) and their unit + integration tests.
+- Drop the `[mistralai-workflows]` extra from `pyproject.toml`. Drop the
+ `[[tool.mypy.overrides]]` block that exists only because Mistral's
+ source uses PEP 695 syntax (re-add it in the new repo).
+- Move docs: `docs/under-the-hood/mistralai-workflows-{plugin,recipes}.md`
+ go to the new repo's docs site. Leave a stub in pipelex docs that links
+ out.
+- CHANGELOG entry under `[Unreleased]` describing the move + migration.
+- Verify: `make agent-check` and `make agent-test` green; `git grep
+ mistralai_workflows` returns nothing in `pipelex/` after the move.
+
+### Stream B — Scaffold `pipelex-mistralai-workflows` from `pipelex-starter-python`
+
+New repo, side-by-side with the other workspace repos.
+
+- Copy `pipelex-starter-python/` as the starting point. Rename the package
+ dir, rewrite `pyproject.toml` (`name`, `description`, dependencies,
+ package list, classifiers).
+- Convert from "app starter" to "library":
+ - Drop the `my_project/hello_world.{mthds,py}` example.
+ - Add a real `LICENSE` (MIT, matching pipelex).
+ - Replace the README with one that explains: install, register the
+ activity on a worker, call it from a workflow, link to recipes.
+ - Wire `py.typed` (already present in starter — keep).
+- `pyproject.toml` deltas vs starter:
+ - `name = "pipelex-mistralai-workflows"`.
+ - `dependencies = ["pipelex>=X.Y", "mistralai-workflows>=3.3.0"]` (no
+ `[mistralai,anthropic,...]` extras — this is a library, not an app).
+ - Re-add the `[[tool.mypy.overrides]]` for `mistralai.workflows.*`
+ (PEP 695 source) — copy from pipelex `pyproject.toml`.
+ - Pytest markers: copy the relevant subset; drop `inference`/`llm`/etc.
+ if the test suite only does layer-2 / layer-3 worker tests.
+- Add a Makefile mirroring the `agent-check` / `agent-test` / `cleanderived`
+ targets used elsewhere in the workspace, so CLAUDE.md instructions in
+ this repo and the new repo overlap.
+- Add `CLAUDE.md` for the new repo. Short — point at workspace `CLAUDE.md`
+ and call out: must not depend on internal `pipelex` paths, only the
+ promoted public embedding core.
+- GitHub Actions: copy the matrix from `pipelex` if there is one, or wire
+ a minimal `uv sync && make agent-check && make agent-test` workflow.
+
+### Stream C — Move the plugin code into the new repo
+
+After A and B land (or in a coordinated PR pair).
+
+- Move `activities.py`, `streaming.py`, `streaming_event_forwarder.py`
+ into `pipelex_mistralai_workflows/`. Rewrite imports to point at the
+ promoted embedding core in `pipelex`.
+- Move the integration tests (`test_activities_direct.py`,
+ `test_activities_offloaded.py`, `test_activities_streaming.py`,
+ `test_bridge_temporal_blocking.py`, `test_bridge_temporal_fire_and_forget.py`)
+ and their fixtures (`conftest.py`, `test_data/bridge_test.mthds`,
+ `test_data/bridge_funcs.py`).
+- Layer-1 framework-agnostic tests (`test_bridge_direct.py`, the unit tests
+ under `tests/unit/pipelex/plugins/mistralai_workflows/`) follow the
+ embedding core — they stay in `pipelex` if we picked Option A, move if
+ Option B.
+- Add the Mistral component / dependency wrapper from §3.4.
+- First release to PyPI as `0.1.0`. Tag, write release notes, link from
+ the pipelex CHANGELOG migration entry.
+
+## 5. Migration story for users
+
+Single-paragraph note in pipelex CHANGELOG and in the new repo's README:
+
+> Mistral Workflows integration moved from
+> `pipelex[mistralai-workflows]` / `pipelex.plugins.mistralai_workflows.*`
+> into a dedicated package: `pip install pipelex-mistralai-workflows` and
+> import from `pipelex_mistralai_workflows.*`. No API changes; the
+> activities, boundary types, and execution modes are identical.
+
+Per project rule, no compat shim. The pipelex release that drops the
+extra and the first `pipelex-mistralai-workflows` release ship together.
+
+## 6. Open risks (track but don't block)
+
+- **Version coupling.** The plugin pkg depends on internal-but-public
+ embedding APIs of `pipelex`. Decide on a stable surface (probably what
+ `bridge.py` already exposes) and document it as such, or breakage will
+ cascade on every pipelex release.
+- **Test parity in CI.** Today layer-2 / layer-3 tests run on every PR in
+ pipelex CI because `uv sync --all-extras` pulls the optional dep. Once
+ extracted, those tests only run in the new repo. Make sure both repos'
+ CI matrices are healthy before flipping the switch.
+- **OffloadableField import path drift.** Already noted in
+ `mistral-workflows-sub-module.md` §6. Carries over verbatim.
+
+## 7. Resuming a session
+
+1. Read `mistral-workflows-sub-module.md` §2, §4, §5 — these are the
+ binding design decisions and gotchas you must respect.
+2. Read this file end-to-end.
+3. Resolve §3 with the user before writing code. Especially §3.1 (where
+ the framework-agnostic core lives) — every other choice depends on it.
+4. Pick a stream from §4. Streams A and B can run in parallel; Stream C
+ waits on both.
diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-sub-module.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-sub-module.md
new file mode 100644
index 000000000..3168bc18d
--- /dev/null
+++ b/wip/distributed-execution/mistral-workflows/mistral-workflows-sub-module.md
@@ -0,0 +1,301 @@
+# Mistral Workflows ↔ Pipelex Plugin — Plan & Progress
+
+Self-contained planning document. A fresh session can resume from this file
+alone; no need to read prior conversation history.
+
+For anything implemented (Phases 1.0–1.5), the code is authoritative. Read
+`pipelex/plugins/mistralai_workflows/` and the matching tests under
+`tests/{unit,integration}/pipelex/plugins/mistralai_workflows/`. The
+user-facing reference lives at
+`docs/under-the-hood/mistralai-workflows-{plugin,recipes}.md`.
+
+---
+
+## Status board
+
+| Phase | Scope | Status |
+| ----- | ------------------------------------------------------ | ------ |
+| 1.0 | Framework-agnostic core (`bridge.py`, modes, ...) | ✅ done |
+| 1.1 | Tier-1 `pipelex_run_pipe` activity wrapper | ✅ done |
+| 1.2 | `TEMPORAL_BLOCKING` + `TEMPORAL_FIRE_AND_FORGET` modes | ✅ done |
+| 1.3 | Docs + CHANGELOG (mkdocs nav wired) | ✅ done¹ |
+| 1.5 | Large-payload `pipelex_run_pipe_offloaded` variant | ✅ done |
+| 2.0 | Streaming v1 — one Mistral `task()` per activity | ✅ done |
+| 2.1 | Streaming v2 — per-step `task.update()` (DIRECT only) | ✅ done |
+
+¹ Cookbook example deferred to a follow-up PR in the sibling
+`pipelex-cookbook/` repo (suggested entry:
+`examples/c_advanced/mistral-workflows/` with a Tier-1 DIRECT-mode worker
+plus a Tier-2 typed activity that uses `library_crate_dump`).
+
+### Outstanding from Phase 1 (not blocking Phase 2)
+
+- [ ] `make agent-check` passes with `mistralai-workflows` NOT installed —
+ needs a fresh venv without the extra. The guard placement
+ (`activities.py` only; `bridge.py` / `execution_mode.py` /
+ `bootstrap.py` / `exceptions.py` are import-clean) should make this
+ pass; not yet verified end-to-end.
+- [ ] CI matrix for the optional dep — deferred. CI runs
+ `uv sync --all-extras`, so layer-2 and layer-3 tests already run on
+ every PR. Reconsider when we add an extra whose tests must NOT run
+ on the default lane.
+
+---
+
+## 1. Context
+
+Integrate Pipelex with Mistral Workflows (`mistralai-workflows>=3.3.0`),
+which wraps Temporal with a thicker DX layer. Two goals were considered:
+
+- **Goal 1** — Port Pipelex orchestration to run *on* Mistral Workflows as
+ an alternative durable runtime. Blocked on Mistral exposing extension
+ hooks for payload converter, codec, sandbox passthrough, and a
+ bare-Temporal `run_worker` mode. **Out of scope.** File issues with the
+ Mistral team if/when we want to revisit.
+- **Goal 2** — Let users invoke Pipelex pipes from inside their own
+ Mistral Workflows activities. **In scope.** Phase 1.x complete; Phase 2
+ pending.
+
+The `mistralai-workflows` dependency must remain strictly optional.
+
+---
+
+## 2. Locked-in design decisions (still binding for Phase 2)
+
+- **Library crate transport** is dump-based:
+ `PipelexPipeRunInput.library_crate_dump: dict[str, Any] | None`.
+- **Execution mode** is an exhaustive `StrEnum`, set per-call (not
+ per-worker): `DIRECT`, `TEMPORAL_BLOCKING`, `TEMPORAL_FIRE_AND_FORGET`.
+- **Streaming** is phased. Phase 2.0 ships a single Mistral `task()` per
+ activity (started / completed / failed). Phase 2.1 ships per-step
+ `task.update()` events driven by Pipelex's `report_delegate`,
+ conditional on demand.
+- **Plugin location**: `pipelex/plugins/mistralai_workflows/`.
+- **Optional-dep guard** lives in `activities.py` (and will live in
+ `streaming.py`). `__init__.py` is empty per Pipelex's "no re-exports"
+ rule. `bridge.py`, `execution_mode.py`, `bootstrap.py`, `exceptions.py`
+ are framework-agnostic and importable without `mistralai-workflows`.
+- **Boundary types are JSON-only.** No Pipelex internal types (`PipeJob`,
+ `PipeOutput`, `WorkingMemory`) cross the activity boundary.
+
+---
+
+## 3. Module layout
+
+```
+pipelex/plugins/mistralai_workflows/
+├── __init__.py # empty (no re-exports)
+├── exceptions.py # 4 exception classes
+├── execution_mode.py # PipelexExecutionMode StrEnum
+├── bridge.py # framework-agnostic core (NO mistralai imports)
+├── bootstrap.py # ensure_pipelex_booted() + DI helper
+├── activities.py # Tier-1 wrappers + offloaded variant
+└── streaming.py # Phase 2.0 — pipelex_run_pipe_streaming sibling activity
+
+tests/unit/pipelex/plugins/mistralai_workflows/
+├── test_input_models.py # boundary BaseModels
+├── test_execution_mode.py # StrEnum properties
+├── test_validation.py # _validate_input + decode helpers
+└── test_dispatch.py # mode dispatch with mocked PipeRun
+
+tests/integration/pipelex/plugins/mistralai_workflows/
+├── conftest.py # bridge_test_library fixture
+├── test_data/ # bridge_test.mthds + bridge_funcs.py
+├── test_bridge_direct.py # layer 1 (no optional dep)
+├── test_activities_direct.py # layer 2 (Mistral test worker)
+├── test_activities_offloaded.py # layer 2 — offloaded variant
+├── test_activities_streaming.py # layer 2 — Phase 2.0 streaming variant
+├── test_bridge_temporal_blocking.py # layer 3 (+ temporal extra)
+└── test_bridge_temporal_fire_and_forget.py # layer 3 (+ temporal extra)
+```
+
+Users import from full paths (no re-exports per Pipelex rule):
+
+```python
+from pipelex.plugins.mistralai_workflows.activities import (
+ pipelex_run_pipe,
+ pipelex_run_pipe_offloaded,
+ PipelexPipeRunInputOffloaded,
+ PipelexPipeRunOutputOffloaded,
+)
+from pipelex.plugins.mistralai_workflows.bridge import (
+ PipelexPipeRunInput, PipelexPipeRunOutput, run_pipe_via_bridge,
+)
+from pipelex.plugins.mistralai_workflows.execution_mode import PipelexExecutionMode
+from pipelex.plugins.mistralai_workflows.bootstrap import ensure_pipelex_booted
+from pipelex.plugins.mistralai_workflows.streaming import (
+ pipelex_run_pipe_streaming,
+ PipelexPipeRunStreamingState,
+ PIPELEX_PIPE_RUN_TASK_TYPE,
+)
+```
+
+---
+
+## 4. Gotchas from Phase 1 (read before Phase 2)
+
+These bit during Phase 1.1–1.2 and will likely bite again when adding
+`streaming.py` plus a new integration test module. Save the cycles.
+
+1. **Workflow sandbox rejects Pipelex imports** seen during workflow
+ class validation. Wrap them in
+ `mistralai_workflows.workflow.unsafe.imports_passed_through()` AND
+ pass `enforce_determinism=False` to `@workflow.define` for test
+ workflows. See `test_activities_direct.py:24-30` for the pattern.
+ Activities are unrestricted Python — this only matters inside
+ workflow class bodies.
+
+2. **`OtelTraceId` search attribute** must be pre-registered on the
+ test namespace. Mistral's `@workflow.define` upserts it on every
+ workflow run, and the dev server rejects activations against an
+ unknown attribute. Pass
+ `search_attributes=[SearchAttributeKey.for_keyword("OtelTraceId")]`
+ to `WorkflowEnvironment.start_local`.
+
+3. **Task queue mismatch.** Mistral's `@activity` wrapper schedules
+ activities on the *global* `mistralai_config.temporal.task_queue`
+ (default `"default"`), NOT the workflow's task queue. Spin a worker
+ on a custom test queue without overriding the global → activities
+ land on `"default"` and the workflow hangs forever. Fix: an autouse
+ module-scoped fixture pinning
+ `mistralai_config.temporal.task_queue = TEST_TASK_QUEUE` for the
+ duration of the test module. See `test_activities_direct.py:48-64`.
+
+4. **Result shape.** Mistral's `convert_result_to_temporal_format`
+ wraps non-BaseModel return values in `{"result": ...}`. Return a
+ BaseModel directly from the workflow entrypoint to skip the
+ wrapping (we return `PipelexPipeRunOutput`).
+
+5. **mypy + Mistral's PEP 695 type syntax.** `mistralai-workflows`'s
+ own source uses PEP 695 type parameters that mypy rejects under
+ `python_version=3.11`. The `[[tool.mypy.overrides]]` block for
+ `mistralai.workflows.*` (`follow_imports = "skip"`,
+ `ignore_errors = true`) in `pyproject.toml` is already in place.
+ Mypy will still flag `OffloadableField.get_value()` as returning
+ `Any` — assign through a typed intermediate variable.
+
+6. **`PipeFunc` is NOT Temporal-compatible.** `asyncio.to_thread`
+ inside `PipeFunc` raises `NotImplementedError` in the deterministic
+ workflow event loop. The bundle ships `bridge_func_pipe` (PipeFunc,
+ DIRECT only) for the no-Temporal layers and `bridge_compose_pipe`
+ (PipeCompose, Temporal-compatible) for the Temporal layers; the
+ `bridge_envelope_pipe` (PipeCompose with inline-structured concept)
+ exists to exercise dynamic-concept round-trip via
+ `library_crate_dump`. Pick the right one per execution mode.
+
+7. **`Pipelex.make()` is NOT idempotent** — it raises if a singleton
+ already exists. `bootstrap.ensure_pipelex_booted()` only calls it
+ when `Pipelex.get_optional_instance() is None`, so an externally-
+ booted singleton is adopted. Don't replace this guard with a
+ module-level boolean.
+
+---
+
+## 5. Phase 2 — Streaming
+
+Goal: surface live progress events from Pipelex pipes through Mistral's
+`task()` event API so users can subscribe via
+`create_capturing_mock_events_client` and friends.
+
+### Phase 2.0 — One task per activity (started / completed / failed) ✅
+
+A single Mistral task wraps the whole activity body — no per-step
+granularity. Cheapest path to "the user sees something happen."
+
+- [x] Created `pipelex/plugins/mistralai_workflows/streaming.py`. Reuses
+ the same optional-dep guard pattern as `activities.py` (top-level
+ `try`/`except ImportError` re-raising `MistralWorkflowsNotInstalledError`).
+- [x] **Decision: sibling activity** `pipelex_run_pipe_streaming`. Reasons:
+ - Keeps the silent path silent — Tier-1 users without observability
+ needs don't pay event publishing overhead.
+ - Mirrors the `pipelex_run_pipe_offloaded` sibling pattern.
+ - Workers register only the variant they actually use.
+- [x] Inside the activity: `async with Task[PipelexPipeRunStreamingState](...)`
+ then call `run_pipe_via_bridge`, then `update_state` to `phase="completed"`.
+ `Task.__aexit__` automatically emits `CustomTaskFailed` on exception
+ and the original exception propagates — no extra `try`/`except`
+ needed (per Pipelex "don't catch Exception speculatively" rule).
+- [x] Layer-2 integration test
+ `tests/integration/pipelex/plugins/mistralai_workflows/test_activities_streaming.py`
+ uses `create_test_worker_with_events` + `EventContext` + a
+ `create_capturing_mock_events_client` to assert that exactly one
+ `CustomTaskStarted`, ≥1 `CustomTaskInProgress`, and one
+ `CustomTaskCompleted` are emitted with the right `custom_task_type`
+ (`pipelex.pipe_run`) and payload shape.
+
+### Phase 2.1 — Per-step granularity (DIRECT mode only) ✅
+
+- [x] Subscribe to Pipelex's trace event channel from inside the
+ activity. Implemented as a queue-backed `EventLogProtocol`
+ (`QueueEventLog` in `streaming_event_forwarder.py`) injected into a
+ per-call `GraphTracerManager` tracer. Note: `ReportingProtocol`
+ itself has no observer methods — the right abstraction is the
+ event log, not the reporting delegate.
+- [x] Map Pipelex events to Mistral `task.update_state(...)` calls:
+ - `PipeStartEvent` → `in_progress` with `current_step_pipe_code`,
+ `current_step_node_id`, `last_event_kind="pipe_start"`,
+ `started_steps`.
+ - `PipeEndSuccessEvent` → `in_progress` with
+ `last_event_kind="pipe_end_success"`, `completed_steps`,
+ `last_output_stuff_name`.
+ - `PipeEndErrorEvent` → `in_progress` with
+ `last_event_kind="pipe_end_error"`. The activity's
+ `Task.__aexit__` then emits `CustomTaskFailed` with the
+ propagated exception.
+ - Other `TraceEventKind`s (edges, batch fan-out, controller
+ outputs, execution data, usage reports) intentionally suppressed
+ — too noisy for state updates, already captured by Pipelex's
+ own reporting / graph infrastructure.
+- [x] Forwarder side-task drains the event log and terminates cleanly
+ on both success and failure. Uses `try` / `finally` (no
+ `except Exception`); the forwarder is fully drained *before*
+ writing the final `phase="completed"` state so the snapshot
+ reflects the right phase.
+- [x] Multi-step test fixture: `bridge_sequence_pipe` (PipeSequence)
+ chaining `bridge_seq_step_one` and `bridge_seq_step_two`
+ (PipeCompose). Test asserts ≥3 pipe_start + ≥3 pipe_end_success
+ `CustomTaskInProgress` events with `started_steps` monotonic
+ `[1, 2, 3]` and per-step pipe codes in declaration order.
+
+Scope: DIRECT mode only. TEMPORAL_BLOCKING / TEMPORAL_FIRE_AND_FORGET
+keep Phase 2.0 single-pair semantics — per-step streaming across the
+Temporal worker boundary would need cross-process tee logic on top of
+the existing `pipeline_run_setup` event log infra, deferred until demand
+surfaces.
+
+---
+
+## 6. Open risks (track but don't block)
+
+- [ ] Concurrent activities sharing process-global Pipelex state — already
+ mitigated by per-call library scoping in
+ `bridge.py::_scoped_library_for_crate`. Verify no leakage under
+ concurrent load when we get there.
+- [ ] If Mistral upgrades change the `OffloadableField` import path
+ (currently `mistralai.workflows.core.encoding.fields_offloader`),
+ revisit the imports in `activities.py`.
+
+---
+
+## 7. Resuming a session
+
+1. Read this file end-to-end — it's short by design.
+2. **For anything in Phase 1.x: the code is authoritative.** Read
+ `pipelex/plugins/mistralai_workflows/*.py` and its tests; do not
+ re-derive from this doc.
+3. For Phase 2 work: start at §5, pick the first unchecked box, re-read
+ §4 (gotchas) before scaffolding the new test module.
+4. After each phase, run:
+ ```
+ make agent-check
+ .venv/bin/pytest tests/unit/pipelex/plugins/mistralai_workflows/ \
+ tests/integration/pipelex/plugins/mistralai_workflows/
+ ```
+ Before declaring Phase 2 done, run the broader sweep:
+ ```
+ .venv/bin/pytest -n auto \
+ -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" \
+ tests/unit/pipelex/plugins/ tests/integration/pipelex/plugins/ \
+ tests/unit/pipelex/builder/ tests/integration/pipelex/builder/
+ ```
+5. Update the status board and the §5 boxes as you go.
diff --git a/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md b/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md
new file mode 100644
index 000000000..5db3c55f1
--- /dev/null
+++ b/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md
@@ -0,0 +1,114 @@
+# `/temporal-e2e-validate` — prompt menu
+
+Operator cheat-sheet of prompts for the `/temporal-e2e-validate` skill. Sibling of [`temporal-e2e-validate-skill.md`](temporal-e2e-validate-skill.md) (what the skill validates + current-state record). The cost-reporting slice has its own deeper cheat-sheet: [`../runtime-bridge/cost-reporting-validation-prompts.md`](../runtime-bridge/cost-reporting-validation-prompts.md).
+
+Convention below: `$` marks prompts that spend real money (live inference / live image-gen / live extract). Everything else is free — dry-run or `--mock-inference`. A few of the live tiers (4, 5, 12) also *run* in dry mode for structural coverage but only catch real payload / cost bugs when run live; those are marked `$ (dry-capable)`.
+
+## Run it ALL in one go
+
+### Complete sweep, real spend authorized ($)
+
+```text
+/temporal-e2e-validate run the COMPLETE sweep with real spend authorized — do NOT skip the opt-in batteries or the live arms. Mode 1 pytest dry then live. Mode 2 every tier: 1/2/2b/3 core, 4 & 5 live (image payload), Step 4 graph tracing, Step 5 concurrent isolation, Tier 8 usage emission, Tier 8b cost reporting FULL (arms A–G), Tier 9 object-gen, Tier 11 extract, Tier 12 CV batch live, Tiers 13–16 error propagation, Tiers 6/7 codec. Then Step 8 routing battery (10a–10c) and Step 9 queue-options + worker-runtime-profile battery (Scenarios A–F). Finish with the master results table — PASS/FAIL per tier/scenario — and stop-and-report on any failure.
+```
+
+### Complete sweep, no spend (full structural pass)
+
+```text
+/temporal-e2e-validate run the full sweep DRY/mock only — no real spend. Mode 1 dry, Mode 2 all tiers in dry/mock (Tier 8b cost reporting via --mock-inference, arms A–B), Step 5 isolation, Tiers 6/7 codec, plus Scenario C (pytest) and Scenario F (CLI typo). Skip the live-only arms: image payload (4/5), CV batch live (12), routing battery (10a–10c), queue-options live scenarios (A/B/D/E), and the live cost arms (E–G). Master results table at the end.
+```
+
+Recommended order: run the **no-spend** sweep first to confirm the plumbing cheaply, then the **real-spend** sweep once it's green. The all-in-one is long and runs tiers sequentially (reporting after each); if a live backend isn't configured, that arm stops-and-reports rather than silently passing — see Prerequisites.
+
+## By area
+
+### Broad (skill's own recognized triggers)
+
+```text
+/temporal-e2e-validate # full regression; offers the two batteries as opt-in extras
+/temporal-e2e-validate full temporal test # same, explicit
+```
+
+### Mode 1 — fast pytest (real server, in-process worker)
+
+```text
+/temporal-e2e-validate run the Mode 1 pytest suite, dry # library_crate: crate, hydration, isolation, controllers
+/temporal-e2e-validate Mode 1 pytest, live LLM # $ same suite with real inference
+```
+
+### Core distributed execution (Mode 2, free)
+
+```text
+/temporal-e2e-validate validate core distributed execution — LibraryCrate, deferred hydration, parallel child workflows
+/temporal-e2e-validate validate concurrent isolation (concept / pipe / multi-concept) # Step 5
+/temporal-e2e-validate validate cross-process registry # Tier 2b
+```
+
+### Graph tracing (free)
+
+```text
+/temporal-e2e-validate validate cross-worker graph tracing / GraphSpec assembly # Step 4
+```
+
+### Images ($ — dry-capable, but run live to catch payload bugs)
+
+```text
+/temporal-e2e-validate image temporal # Tiers 4–5: image gen + image-to-LLM flow, payload storage
+```
+
+### Object generation / extract / batch
+
+```text
+/temporal-e2e-validate validate object generation cross-process # Tier 9, free
+/temporal-e2e-validate validate make_extract_pages two-activity cross-process # Tier 11, free
+/temporal-e2e-validate validate the CV batch deeply-nested controller stack # Tier 12, $ (dry-capable)
+```
+
+### Error propagation
+
+```text
+/temporal-e2e-validate validate error handling # Mode 1 Step 2b + Tiers 13–16
+/temporal-e2e-validate error report propagation across the activity → workflow → submitter boundary
+```
+
+Covers LLM, extract, image-gen, and fanned-out child-workflow (PipeBatch) failures carrying a structured `ErrorReport`. Free — failures are mocked or forced via a bad-credential worker (401), no successful inference billed.
+
+### Payload codec (free)
+
+```text
+/temporal-e2e-validate validate the StoragePayloadCodec — transparency + large-payload stress # Tiers 6–7
+```
+
+### Routing — v1 `activity_queues` ($)
+
+```text
+/temporal-e2e-validate routing # Tiers 10a–10c: multi-activity isolation, per-handle routing, two-activities-one-route
+```
+
+### Queue options + worker-runtime profiles — v2 (mixed)
+
+```text
+/temporal-e2e-validate queue options # Scenarios A–F
+/temporal-e2e-validate runtime profile # named worker-runtime profiles via --profile
+```
+
+Scenarios: A multi-class routing, B per-queue timeout, D queue rate-limit, E missing-worker negative (all live `$`); C per-handle override (pytest, free); F the `--task-queue` CLI typo check with "did you mean?" (CLI startup, free).
+
+### Cost reporting
+
+```text
+/temporal-e2e-validate cost reporting # free: mock arms A–B
+/temporal-e2e-validate cost reporting full # $ every arm A–G
+```
+
+Full detail (arm-by-arm assertions, aliases, ask-first behavior) in [`../runtime-bridge/cost-reporting-validation-prompts.md`](../runtime-bridge/cost-reporting-validation-prompts.md).
+
+## Recognized triggers vs descriptive prompts
+
+Some prompts hit **explicit routes** in the skill (`full temporal test`, `validate error handling`, `cost report`, `queue options` / `runtime profile` / `routing`, `image temporal`). The per-tier ones (Tier 9, 11, 12, codec, graph tracing…) are not separate routes — the agent reads `references/mode-2-tiers.md` and runs the named tier, so a descriptive prompt naming the concern is enough.
+
+## Prerequisites
+
+- `tmux` and the `temporal` CLI installed; the Temporal dev server reachable (the skill starts one if needed).
+- Mode 2 stands up split router+runner worker processes (see `references/mode-2-setup.md`).
+- Live arms need credentials: LLM inference (or the Pipelex Gateway), real image-gen, and extract = Azure Doc Intel via the gateway. Missing a backend makes that arm fail at the **run** step, not the assertion — the skill reports which arm and why.
diff --git a/wip/temporal-e2e-validate-skill.md b/wip/distributed-execution/temporal-e2e-validate-skill.md
similarity index 96%
rename from wip/temporal-e2e-validate-skill.md
rename to wip/distributed-execution/temporal-e2e-validate-skill.md
index 634a9620e..82a6b61ff 100644
--- a/wip/temporal-e2e-validate-skill.md
+++ b/wip/distributed-execution/temporal-e2e-validate-skill.md
@@ -2,6 +2,8 @@
Current-state record for the `/temporal-e2e-validate` skill (`.claude/skills/temporal-e2e-validate/`), captured 2026-06-02 on branch `feature/Validate-with-signatures-4-fix-dry-run`. It exists to cold-start a follow-up session: what the skill validates, what a full run found, the Step 9 skill bugs that were patched, and the open follow-ups.
+For ready-to-paste invocation prompts (per-area + run-it-all-in-one-go), see the sibling [`temporal-e2e-validate-prompt-menu.md`](temporal-e2e-validate-prompt-menu.md).
+
## What the skill is
`/temporal-e2e-validate` validates that Pipelex pipelines execute correctly across separate Temporal worker processes. Two modes:
@@ -36,7 +38,7 @@ Also fixed in the same file: Scenario D's measurement (it read a single workflow
- **Vestigial `act_jinja2_gen_text`.** Only reachable via the test fixture `pipelex/temporal/test_extras/wf_test_content_generator_child.py` (and the dry stub); still registered in `pipelex/temporal/tasks.py` and the `runner-jinja2` scope (`pipelex/pipelex.toml:665`). No pipe dispatches it (PipeCompose renders inline). Candidate for removal: the activity + its `act_jinja2_generate.py` + the `runner-jinja2` scope + the fixture path. User chose to defer; not removed.
- **Stale "Known xfails" note in `SKILL.md`** (Mode 1). The StuffArtefact-serialization xfails now xpass — the note should be refreshed or the xfail markers dropped. Out of scope for the Step 9 fixes; not patched.
-- **Worker can't boot under `--is-unit-testing` (product bug, not a skill issue).** Surfaced by the Scenario F sanity-check: a sandboxed worker started with `--is-unit-testing` fails temporalio's workflow-sandbox validation on the registered test workflows, so it never boots. Queue-name-independent and orthogonal to the queue-options surface — the task-queue validator (`worker_cli.py:75`) already passed before it fires. Tracked with full evidence and likely fix locus in [distributed-execution/unit-testing-worker-sandbox-validation.md](distributed-execution/unit-testing-worker-sandbox-validation.md). Not investigated or fixed here.
+- **Worker can't boot under `--is-unit-testing` (product bug, not a skill issue).** Surfaced by the Scenario F sanity-check: a sandboxed worker started with `--is-unit-testing` fails temporalio's workflow-sandbox validation on the registered test workflows, so it never boots. Queue-name-independent and orthogonal to the queue-options surface — the task-queue validator (`worker_cli.py:75`) already passed before it fires. Tracked with full evidence and likely fix locus in [unit-testing-worker-sandbox-validation.md](unit-testing-worker-sandbox-validation.md). Not investigated or fixed here.
## Cold-start: re-verify the patched skill
diff --git a/wip/observer-and-telemetry/identifier-visibility-and-redaction.md b/wip/observer-and-telemetry/identifier-visibility-and-redaction.md
new file mode 100644
index 000000000..cf9fca352
--- /dev/null
+++ b/wip/observer-and-telemetry/identifier-visibility-and-redaction.md
@@ -0,0 +1,77 @@
+# Identifier visibility vs. privacy — reflection kickoff
+
+Status: open question, no code touched. This note exists to start a fresh-session reflection. Keep the options open.
+
+## What triggered this
+
+A pipeline run through the API (Temporal path) returned a cost report whose `tokens_usages[]` records carried, per LLM call:
+
+```
+job_metadata.pipe_code = "analyze_cv_job_match"
+job_metadata.otel_context.trace_name = "analyze_cv_job_match_aced9b60"
+job_metadata.otel_context.trace_name_redacted = "aced9b60"
+job_metadata.trace_context.tracer_key = "<run_id>/pipe-router"
+job_metadata.trace_context.parent_node_id = "<run_id>:<run_id>/pipe-router:node_0"
+```
+
+The original question was narrow ("what is `/pipe-router`?"), but it opened a broader one: **which structural identifiers (pipe codes, run ids, user/session ids) are exposed on which surfaces, and is there a single policy that lets us hide them when we need to?**
+
+## ID semantics (so the cold reader isn't guessing)
+
+- **Temporal workflow id is a hierarchical path.** `WfPipeRun` (orchestrator) owns `{run_id}`; it spawns child `WfPipeRouter` with id `{run_id}/pipe-router` (hardcoded suffix, `wf_pipe_run.py:62`). When a controller dispatches a sub-pipe through `TemporalPipeRouter` *inside* a workflow, each sub-pipe becomes its own child workflow with id `{parent}/{pipe_code}-{uuid8}` (`temporal_pipe_router.py:69`).
+ - `{pipe_code}` = the literal sub-pipe code. `-{uuid8}` = `str(workflow.uuid4())[:8]`, a replay-deterministic collision-breaker for same-code siblings (e.g. batch). Not ordered, not meaningful.
+ - **Inconsistency:** the top pipe's workflow uses a generic `/pipe-router`; nested pipes use their pipe code. The top pipe also has a code that is currently thrown away here.
+- **`node_N` is unrelated to pipe codes.** It's `TraceContext.node_sequence` — a monotonic counter minted per workflow tracer (`graph_tracer.py:198` → `{graph_id}:{workflow_id}:node_{seq}`; direct mode `{graph_id}:node_{seq}`, `trace_context.py:64`). The counter **resets per workflow**, so the `{workflow_id}` component is load-bearing: it's the namespace that keeps `node_0` of one workflow distinct from `node_0` of another.
+
+## Core finding: the cost report is an ungoverned exposure surface
+
+`tokens_usages[]` embeds the **entire `JobMetadata`** (`llm_report.py:65-67`, dumped wholesale at `llm_report.py:44`). Nothing redacts it before it leaves the API. The pipe code alone appears up to three ways in one record: `pipe_code`, `otel_context.trace_name`, and (if the workflow id were renamed) `tracer_key`/`parent_node_id`. `user_id` / `session_id` ride along too (`job_metadata.py:41-74`).
+
+The irony: a **redacted twin already exists** — `OtelFactory.make_trace_names` (`otel_factory.py:131`) computes both `{pipe_code}_{hash}` and the pipe-code-free `{hash}`, and both are shipped side by side. The redaction concept the design intended is present and then ignored on this surface.
+
+Consequence for the `/pipe-router` → `/{pipe_code}` rename idea: it improves OSS observability/consistency but **adds a 4th pipe-code leak to the one surface with no redaction control** — wrong direction for hosted privacy. The "uninformative" `/pipe-router` is actually the *safe* default; the informative naming is what should be opt-in.
+
+## Existing knobs — and what each does NOT cover
+
+Several independent subsystems, each scoped to a different sink:
+
+| Subsystem | Config | Governs | Default | Covers cost-report output? |
+|---|---|---|---|---|
+| Telemetry export redaction | `[telemetry.custom_posthog.tracing.capture]` → `TelemetryRedactionConfig` (`telemetry_config.py:199`); `is_capture_pipe_codes_enabled()` (`telemetry_manager_abstract.py:44`) | pipe codes / content in PostHog/OTEL/Langfuse spans sent to external sinks | redact by default | No |
+| Temporal search attributes | `[temporal.search_attributes].enabled` + allowlist (`config_temporal.py:41`) | what's queryable in the Temporal UI | subset | No |
+| Temporal static summary/details | none (`observability.py:124-160`) | pipe code + description + user/session + inputs on every workflow | always on | No |
+| Graph data inclusion | `graph_config.data_inclusion` (`graph_config.py:6`) | stuff payload content (json/text/html), stack traces, registry in graph events | per-config | governs payloads, not the ids/codes |
+| Error disclosure | `disclosure_mode` STRICT/VERBOSE (`base_exceptions.py`) | error-message redaction in `ErrorReport` | — | separate |
+
+**Two surfaces fall through all of them:**
+
+1. The Temporal **workflow-id naming** (`/pipe-router` vs `/{pipe_code}-{uuid}`).
+2. The **`JobMetadata` serialized into `tokens_usages`** (and the runner fallback path that derives `workflow_id`/`node_id` from the same context, `reporting_manager.py:263-264`).
+
+## The tension to reconcile
+
+- OSS / self-host users want pipe codes visible — in the report, logs, Temporal UI, for debugging and querying.
+- Hosted runner needs the ability to hide structural detail for privacy.
+
+Both are legitimate; the answer is config-driven, not one-size.
+
+## Open decision (do not foreclose)
+
+The system already has the right primitive — a "may pipe codes be shown" boolean — but it only governs telemetry export. A reflection should weigh making structural-identifier visibility a first-class policy that *also* governs (1) workflow-id naming and (2) `JobMetadata`/cost-report serialization, defaulting visible for OSS and redacted for the hosted runner.
+
+Candidate homes for that policy — left open:
+
+- **(a)** Extend/reuse the telemetry `capture.pipe_codes` flag so one switch covers telemetry *and* run output.
+- **(b)** New knob alongside `data_inclusion` on `graph_config` / `tracing_config` (the cost report rides the same trace substrate).
+- **(c)** Derive from the existing `data_inclusion` family.
+
+Each has different blast radius and different "single source of truth" implications. Also unresolved: whether redaction should happen at serialization time (strip fields from `JobMetadata` before it reaches `tokens_usages`) or at id-construction time (never bake codes into workflow ids in the first place) — or both, since they protect different sinks.
+
+## Anchor points for a cold start
+
+- Workflow-id naming: `wf_pipe_run.py:62`, `temporal_pipe_router.py:69`
+- Node id minting: `graph_tracer.py:198`, `trace_context.py:48,64`
+- Cost-report exposure: `llm_report.py:44,65-67`, `job_metadata.py:41-74`
+- Redacted-twin generator: `otel_factory.py:131`
+- Telemetry redaction policy: `telemetry_config.py:199`, `telemetry_manager_abstract.py:44`
+- Other sinks: `config_temporal.py:41` (search attrs), `observability.py:124-160` (static summary/details, ungated), `graph_config.py:6` (data inclusion)
diff --git a/wip/observer-and-telemetry/observer-telemetry-posthog.md b/wip/observer-and-telemetry/observer-telemetry-posthog.md
new file mode 100644
index 000000000..76a809d45
--- /dev/null
+++ b/wip/observer-and-telemetry/observer-telemetry-posthog.md
@@ -0,0 +1,146 @@
+# How Observer, Telemetry, and PostHog actually fit together
+
+Written up after a wrong mental model surfaced during the runtime-bridge DIRECT-router review (`wip/runtime-bridge/direct-mode-nested-router-leak.md`). The belief under review was: *"the `multi_observer` has nothing to do with telemetry, and we don't use it at all in practice."* Both halves are wrong in instructive ways. This doc is the corrected map, traced against the code on `feature/Runtime-bridge-extraction`.
+
+## TL;DR — the corrected mental model
+
+1. **The Observer subsystem's *only* job is telemetry.** The default `multi_observer` is `{"noop": ObserverNoOp, "telemetry": ObserverTelemetry}`. The noop does nothing; `ObserverTelemetry` emits PostHog product-analytics events (`PIPE_RUN`, `PIPE_COMPLETE`). There is nothing else in it. "Observer" and "pipe-level PostHog telemetry" are the same thing.
+
+2. **It IS used in practice — but only on the Gateway/hosted path.** By default (BYO-key, direct Python SDK, pytest, CI) the telemetry manager resolves to `TelemetryManagerNoOp`, so the observer fires into a no-op and nothing leaves the process. The moment you run under **Pipelex Gateway**, a real `TelemetryManager` is built with `pipelex_telemetry_enabled=True` and those `PIPE_RUN`/`PIPE_COMPLETE` events flow to Pipelex's PostHog. So "we don't use it" is true for local/OSS, false for the hosted product.
+
+3. **The hub-level observer is dead code.** `PipelexHub.set_observer()` stores `self._observer` and **nothing ever reads it** — there is no `get_observer`. The only live observer is the instance attribute on the `PipeRouter` (`router.observer`), consumed inside `PipeRouterProtocol.run()`. The hub write is a red herring; it provides zero redundant coverage.
+
+## The Observer subsystem
+
+### The protocol
+
+`pipelex/observer/observer_protocol.py` — three async hooks plus a payload dict:
+
+```python
+class ObserverProtocol(Protocol):
+ async def observe_before_run(self, payload: PayloadType) -> None: ...
+ async def observe_after_successful_run(self, payload: PayloadType) -> None: ...
+ async def observe_after_failing_run(self, payload: PayloadType) -> None: ...
+```
+
+`PayloadType = dict[str, Any]`, keyed by `PayloadKey` (`PIPELINE_RUN_ID`, `PIPE_JOB`, `PIPE_OUTPUT`, `ERROR`).
+
+### Implementations that exist
+
+| Class | File | Does | Wired by default? |
+|---|---|---|---|
+| `ObserverNoOp` | `observer/observer_protocol.py:40` | nothing | yes (as `"noop"`) |
+| `MultiObserver` | `observer/multi_observer.py` | fans out to a named dict of sub-observers | yes (the container) |
+| `ObserverTelemetry` | `system/telemetry/observer_telemetry.py` | calls `telemetry_manager.track_event(PIPE_RUN / PIPE_COMPLETE)` | yes (as `"telemetry"`) |
+| `LocalObserver` | `observer/local_observer.py` | writes payloads to disk under `results/observer` | **no — defined but never instantiated in production wiring** |
+
+`LocalObserver` is reachable only if a caller passes a custom `observers=` dict into `Pipelex.make()`. Default boot never builds it. (Flag: dead-ish code, or a debugging aid nobody wired.)
+
+### Where the observer fires — the one live path
+
+`PipeRouterProtocol.run()` is the only caller of the observer hooks:
+
+- `pipelex/pipe_run/pipe_router_protocol.py:22` → `observe_before_run`
+- `:34` → `observe_after_successful_run`
+- `:46` → `observe_after_failing_run`
+
+It calls `self.observer`, i.e. the observer attached to **that router instance** at construction. `PipeRouter.__init__` defaults to `ObserverNoOp` when none is passed (`pipe_run/pipe_router.py:11`).
+
+### The wiring at boot (`pipelex/pipelex.py`)
+
+```python
+# pipelex.py ~417-452
+if not observers:
+ no_op_observer = ObserverNoOp()
+ observer_telemetry = ObserverTelemetry(telemetry_manager=self.telemetry_manager)
+ observers = {"noop": no_op_observer, "telemetry": observer_telemetry}
+multi_observer = MultiObserver(observers=observers)
+self.pipelex_hub.set_observer(observer=multi_observer) # <-- DEAD: never read back
+...
+# non-Temporal:
+self.pipelex_hub.set_pipe_router(PipeRouter(observer=multi_observer)) # <-- the ONLY live install
+# Temporal:
+self.pipelex_hub.set_pipe_router(make_temporal_pipe_router()) # observer = ObserverNoOp
+```
+
+So:
+
+- **Non-Temporal hub router** carries `multi_observer` → telemetry fires (into whatever the telemetry manager is — real or no-op).
+- **Temporal router** carries `ObserverNoOp` (`temporal/tprl_pipe/temporal_pipe_router.py:52`) → **no** pipe-level telemetry on Temporal at all. Precedent: "no router observer" is already a shipped state.
+
+### The dead hub write — proof
+
+```
+hub.py:89 self._observer: ObserverProtocol | None = None # declared
+hub.py:206 self._observer = observer # written by set_observer
+(no get_observer / get_required_observer anywhere in the repo)
+```
+
+Grep for `get_observer` / `_observer` readers returns only the declaration and the setter. The doc-in-review had hoped "the hub also sets the observer separately, so the router-level observer may be redundant." It isn't redundant — the hub copy is simply unused. Cleanup opportunity: either delete `_observer`/`set_observer`, or add a `get_observer` and actually route through it (decide intentionally).
+
+## The Telemetry layer — what the observer feeds into
+
+`ObserverTelemetry` is a thin adapter: it turns each pipe run into a `track_event` call (`observer_telemetry.py`). Everything past that is the **telemetry manager**, which has two independent sinks.
+
+### Sink A — custom PostHog (user's own analytics)
+
+- Config: `telemetry.toml` → `[custom_posthog] mode = ...`, model `TelemetryConfig` in `system/telemetry/telemetry_config.py`.
+- Default: `PostHogMode.OFF` (`telemetry_config.py:81`).
+- `track_event` (`telemetry_manager.py:245`) matches the mode: `OFF` → logs verbose and emits nothing; `ANONYMOUS`/`IDENTIFIED` → captures to the user's PostHog.
+
+### Sink B — Pipelex Gateway telemetry (our hosted analytics)
+
+- Flag: `pipelex_telemetry_enabled` on the `TelemetryManager`, gated by `is_pipelex_telemetry_enabled` in `TelemetryFactory.make_telemetry_manager` (`telemetry_factory.py`).
+- Turned on when running under **Pipelex Gateway** (needs the gateway API key as distinct_id; `GatewayApiKeyMissingError` if absent).
+- `track_event` always also fires Sink B when `_pipelex_telemetry_enabled` (`telemetry_manager.py:262`), **independent of `custom_posthog.mode`**. So even with custom PostHog `OFF`, Gateway users still emit `PIPE_RUN`/`PIPE_COMPLETE` to Pipelex's PostHog (`telemetry_factory.py:64-74`).
+
+### When the whole telemetry manager collapses to a no-op
+
+`TelemetryFactory.make_telemetry_manager` returns `TelemetryManagerNoOp` when:
+
+- `DO_NOT_TRACK` env var is truthy (always respected; Gateway+DNT raises `GatewayDoNotTrackConflictError` rather than tracking), OR
+- custom PostHog is `OFF` **and** Gateway telemetry is off, OR
+- the **integration mode** disallows custom telemetry and Gateway is off. `is_custom_telemetry_allowed_for_mode(integration_mode)` — e.g. `pytest`/`ci`/`python` default to not allowing custom telemetry (see `telemetry.toml` `[telemetry_allowed_modes]` and the model defaults).
+
+When it's `TelemetryManagerNoOp`, `ObserverTelemetry` still runs every pipe but `track_event` does nothing.
+
+### "Is it used in practice?" — the matrix
+
+| Context | custom PostHog | Gateway telemetry | Net effect of the observer |
+|---|---|---|---|
+| Local dev / direct Python SDK, no config | OFF (default) | off | **no-op** — fires into `TelemetryManagerNoOp` |
+| pytest / CI | disallowed by mode | off | **no-op** |
+| User explicitly sets `mode=anonymous/identified` | on | off | events → user's PostHog |
+| **Pipelex Gateway (hosted runner)** | usually OFF | **on** | **events → Pipelex PostHog** (live, real) |
+| Any of the above + `DO_NOT_TRACK=1` | — | forced off | no-op |
+| Temporal router (any context) | — | — | **no-op** (router uses `ObserverNoOp`) |
+
+So the honest statement is: *the observer is best-effort PostHog product analytics; it is genuinely inert in OSS/local/test, and genuinely live for Gateway/hosted non-Temporal runs.*
+
+## Don't confuse these three things — they are all "observability" but separate
+
+The word "tracing"/"observability" is overloaded across the codebase. They are independent subsystems:
+
+1. **Observer → PostHog events** (this doc). Coarse product analytics: a `PIPE_RUN` / `PIPE_COMPLETE` event per pipe, with `pipeline_run_id` + `pipe_type`. Driven by `router.observer`. Sinks: user PostHog and/or Pipelex PostHog.
+
+2. **OTel AI tracing** (`custom_posthog.tracing`, `pipelex_telemetry`). OpenTelemetry spans for LLM/inference calls, built in `TelemetryManager` (`telemetry_manager.py:83-106`, `otel_factory.py`). Separate enable flags, separate exporter. Not the observer.
+
+3. **Pipelex trace-event log** (`[pipelex.tracing_config]` in `pipelex.toml`, `flush_trace_events_to_backend`). The distributed run-trace log persisted to NDJSON / DynamoDB for cross-Temporal-worker run reconstruction. This is the subject of `wip/runtime-bridge/trace-flush-blocking-io.md`. Nothing to do with PostHog or the observer.
+
+If someone says "telemetry is off so tracing is off," check *which* of the three they mean — they don't share switches.
+
+## Implications for the bridge DIRECT-router decision
+
+(Full context in `wip/runtime-bridge/direct-mode-nested-router-leak.md`.) Because the router observer's sole effect is best-effort PostHog telemetry, and because:
+
+- the **root** pipe of a DIRECT run **already** carries no observer today (bare `PipeRouter()` → `ObserverNoOp`), so DIRECT runs already emit telemetry on nested pipes but not the root — an inconsistency, not a feature;
+- Temporal mode already emits nothing through this path;
+- threading the observer into the bridge (Option B) would require a `get_observer` on the hub that **doesn't exist**, i.e. re-coupling the framework-agnostic bridge to internals;
+
+→ **Option A (scope the bare router for the whole DIRECT run)** is safe: it makes root + nested consistently silent on PostHog pipe events, matching Temporal. The only loss is nested-pipe `PIPE_RUN`/`PIPE_COMPLETE` events **for Gateway users running bridge-DIRECT pipes** — a brand-new path no dashboard depends on yet. If complete pipe telemetry on this path is ever wanted, that's a separate "make telemetry whole" task (and would mean giving the root pipe an observer too, which today's code doesn't).
+
+## Flagged cleanups (separate from the bridge work)
+
+- `PipelexHub._observer` / `set_observer` is write-only dead code. Remove it, or wire `get_observer` and route the router through it intentionally.
+- `LocalObserver` is defined but never instantiated by default boot. Confirm whether it's a debugging aid worth keeping or removable.
+- `multi_observer` always contains a `"noop"` entry that does nothing on every pipe — harmless, but it's pure overhead in the fan-out loop. Likely a placeholder for "there's always at least one observer."
diff --git a/wip/runtime-bridge/README.md b/wip/runtime-bridge/README.md
new file mode 100644
index 000000000..1d57f57b5
--- /dev/null
+++ b/wip/runtime-bridge/README.md
@@ -0,0 +1,90 @@
+# runtime-bridge — review follow-ups
+
+Triage of the SWE-agent review comments on **PR #959** (`feat: extract framework-agnostic Pipelex runtime bridge`, branch `feature/Runtime-bridge-extraction`). The PR extracts `pipelex/runtime_bridge/` — a host-runtime-agnostic surface (`run_pipe_via_bridge`) that lets Mistral Workflows / raw Temporal / future plugins invoke Pipelex pipes from inside their own activities.
+
+Two review bots left 12 unresolved threads (**greptile-apps** + **cubic-dev-ai**). All 12 were verified against the code. **All are now resolved in the working tree** — the only thing left is replying to / resolving the GitHub threads (see "Remaining" at the bottom).
+
+## 1. Design forks — all resolved
+
+Each was a **confirmed** finding whose resolution was a genuine design choice. Each doc retains its full triage as the record of why.
+
+- **[direct-mode-nested-router-leak.md](direct-mode-nested-router-leak.md)** — ✅ **RESOLVED (Option A).** (greptile P1, + cubic P2 enabler) In `DIRECT` mode, nested controller sub-pipes resolved the *hub default* router and leaked to Temporal when a worker has `[temporal] is_enabled`. Fixed with a `scoped_pipe_router` helper in `hub.py` wrapping `_run_direct` (also resolves cubic's hub.py:615/teardown-clobber). The observer sub-question that gated the fork is written up in [`../observer-and-telemetry/observer-telemetry-posthog.md`](../observer-and-telemetry/observer-telemetry-posthog.md).
+- **[graph-context-temporal-contract.md](graph-context-temporal-contract.md)** — ✅ **RESOLVED (Option A).** (greptile P2) `graph_context` was threaded into the `PipeJob` for *all* modes despite the DIRECT-only docstring, and `WfPipeRouter` consumes it. Fixed: `run_pipe_via_bridge` now nulls `graph_context` for the Temporal modes (honoring the contract); docstring corrected. + regression test.
+- **[trace-flush-blocking-io.md](trace-flush-blocking-io.md)** — ✅ **RESOLVED (false positive / by-design).** (cubic P2) `flush_trace_events_to_backend` is `async` with blocking boto3/file I/O — but it runs in an *activity*, which is exactly where Temporal allows blocking I/O. No code change. The optional `asyncio.to_thread` throughput-offload note is kept on file for if/when Temporal ships and we profile worker contention.
+- **[bridge-error-name-collision.md](bridge-error-name-collision.md)** — ✅ **RESOLVED (renamed).** The near-mirror `PipelexBridgeRuntimeError` (leaf) vs `PipelexRuntimeBridgeError` (base) was renamed: the leaf is now **`PipelexBridgeDispatchError`**. Updated the class, `bridge.py` raises, `test_validation.py`, regenerated `docs/errors/` (old slug removed, new written; no mistral churn), and the `TODOS.md` prose.
+
+## 2. Mechanical fixes — all applied
+
+- **Boot race** — ✅ `pipelex/runtime_bridge/bootstrap.py`. Double-checked locking with a module-level `threading.Lock` (re-check inside the lock). Regression test: `tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py` (threaded barrier; asserts make runs once, no raise). **Follow-up (PR #966 review):** this closed the write-write race but not a lock-free read of a half-built singleton mid-`setup()` — **now also fixed** via Approach B (the bridge gates on `Pipelex.is_fully_booted()`; an instance-level `is_ready` is published only at the end of `make()`). See §3 / [`bootstrap-half-built-singleton-race.md`](bootstrap-half-built-singleton-race.md).
+- **Exceptions caller-facing flag** — ✅ `pipelex/runtime_bridge/exceptions.py`. Added `_authors_caller_facing_message = True` to `MissingPipelexTemporalExtraError` and `MissingMistralWorkflowsPluginError` so the pip-install hint survives STRICT disclosure. Regression test: `test_exceptions_disclosure.py`.
+- **streaming.mdx missing import** — ✅ added `import asyncio` to the example's import block.
+- **observability.mdx frontmatter id** — ✅ `id: observabilities` → `id: observability`.
+- **your-first-workflow.mdx grammar** — ✅ "It coordinate" → "It coordinates".
+- **your-first-workflow.mdx broken image** — ✅ removed the missing-PNG image, kept the instruction text.
+- **durable-execution.md overgeneralization** — ✅ rewrote the line to distinguish the two durable backends (the `[temporal] is_enabled` flag is the Pipelex-on-Temporal backend only; the Mistral Workflows path runs pipes via the runtime bridge inside Workflows activities).
+
+## 3. PR #966 pre-landing review
+
+A later `/review` pass on PR #966 (the bridge + the trace-event read hardening). One deferred item plus four cheap hardening fixes applied to the working tree.
+
+- **Bootstrap half-built-singleton race** — ✅ **applied (P1)**, see [`bootstrap-half-built-singleton-race.md`](bootstrap-half-built-singleton-race.md). The #959 lock closed the write-write race; this closes the remaining lock-free read of a mid-`setup()` singleton. Approach B (instance-level `is_ready`, gated via `Pipelex.is_fully_booted()`, published only at the end of `make()`) — chosen over a module-global flag so the existing `teardown_if_needed()` lifecycle auto-resets it. Regression tests added (mid-setup window + setup-failure re-boot).
+- **`library_id` widened** — ✅ `bridge.py` + `primitives/submitter_hydration.py`. Dropped the `uuid4().hex[:8]` truncation (32 bits) to full hex. `open_library` silently *reuses* a colliding id (`library_manager.py:148-149`), so a 32-bit collision between two overlapping calls would share and prematurely tear down one library; full hex matches the collision-safe Temporal path (full `workflow_id`).
+- **`act_assemble_graph` comment corrected** — ✅ the lifted primitive catches only a specific exception tuple, so the old "swallows every failure / no error ever crosses the boundary" comment was stale; reworded to say expected failures degrade to None while programming bugs deliberately propagate (pinned by `test_propagates_unexpected_keyerror`).
+- **Scoped-library open guarded** — ✅ `bridge.py::_scoped_library_for_crate` now opens the library inside the `try` (with a `library_opened` flag) so a throw between open and yield can't leak the manager entry — matches the sibling `rehydrate_pipe_output_with_crate`.
+- **Input decode wrapped** — ✅ `bridge.py` decodes `library_crate_dump` / `delivery_assignment_dump` once, translating raw `pydantic.ValidationError` into `PipelexBridgeDispatchError` so a malformed dump no longer escapes the entry point as a non-`PipelexError`; also removes the double-decode of the delivery dump. `_validate_input` now takes the decoded assignment (its unit tests updated).
+
+`make agent-check` (pyright 0/0/0, mypy clean) and `tests/unit/pipelex/runtime_bridge/` green after these.
+
+## Verification
+
+`make agent-check` (ruff, plxt, pyright 0/0/0, mypy) and `make agent-test` (full suite) both green.
+
+## 4. PR #969 review rounds (live PR)
+
+PR #969 is the live PR (#959 and #966 closed). Successive bot rounds (greptile / cubic / codex) each re-examined the prior round's fixes plus new surface. Round-4 dispositions:
+
+- **Keyed tracers through the bridge** (`bridge.py:179`, greptile P1 + cubic P2) — ✅ **RESOLVED (refuted as unreachable; no code change).** Full triage + the deferred structural option in [`bridge-keyed-tracer-unsupported.md`](bridge-keyed-tracer-unsupported.md). Correct abstract analysis but unreachable: keyed tracers (`tracer_key != graph_id`) are only opened by `wf_pipe_router`, which never calls the bridge; non-DIRECT modes null `trace_context`; on every reachable path `lookup_key == graph_id`. `lookup_key` is kept (round-3 fix; `graph_id` would re-break `close_tracer`) with a tightened comment documenting the unsupported keyed-tracer contract. A loud boundary guard was tried and **backed out** as over-engineering (defended a non-occurring config, was only partial, and stacked churn on the round-3 change defending the same case).
+- **Temporal blocking `workflow_id`** (`bridge.py:342`, codex P2 + cubic P1) — ✅ **RESOLVED (real fix).** Blocking reported the bare `pipeline_run_id`; the workflow is actually started with `make_workflow_id(...)`, which prefixes in non-NORMAL run modes (`ut-`/`ci-`/`cc-`/`cct-`). The bridge now reports `make_workflow_id(...)` — the same id fire-and-forget returns from `start()`, single source of truth. In `RunMode.NORMAL` (prod) the prefix is empty, so behavior is unchanged there. Wiring test: `tests/unit/pipelex/runtime_bridge/test_temporal_blocking_workflow_id.py`; prefix table covered by `tests/unit/pipelex/temporal/test_workflow_id_construction.py`.
+- **trace_flush pre-dedup** (`trace_flush.py:28`, cubic P2) — ➖ **False positive.** Dedup is a **read-side** contract: `EventLogProtocol.read_events` returns events deduplicated by `(workflow_id, writer_id, event_type, sequence)`, and the DynamoDB backend is write-idempotent via PK+SK ("natural deduplication for Temporal replay re-emissions"). NDJSON appends on retry but `read_events` collapses duplicates on assembly. The proposed pre-dedup key also **omits `pipeline_run_id`**, so it would wrongly collapse identical sequences across different runs. No code change. (Related async/blocking-IO note already on file: [`trace-flush-blocking-io.md`](trace-flush-blocking-io.md).)
+- **Mistral-workflows docs** (`index.md:29` cubic P2, `choosing-a-backend.md:8` cubic P3) — ✅ **RESOLVED (accuracy).** "Automatic activity retries — each leaf operator retries independently" overclaimed: in `direct` mode the whole pipe runs in one host activity. Reworded to be mode-aware (`direct` retries as a unit; `mistral_native` retries leaves independently). The "replay" guarantee line was reworded to state replay re-runs the workflow from history reusing stored activity results, not re-executing completed activities.
+
+Round-5 dispositions:
+
+- **Self-review back-out** — the round-4 keyed-tracer guard was removed (see the keyed-tracer bullet above; details in [`bridge-keyed-tracer-unsupported.md`](bridge-keyed-tracer-unsupported.md)).
+- **Gateway `transport_max_retries`** (`gateway_completions_factory.py` / `gateway_responses_factory.py`, cubic P2) — ✅ **RESOLVED (real fix).** The gateway LLM client factories built `openai.AsyncOpenAI` without `max_retries`, silently ignoring a `transport_max_retries` override on the primary inference path (the near-identical `Portkey*Factory` siblings pass it). Wired it into both. The gateway extract/img-gen path uses `AsyncPortkey` (no `max_retries` param — verified), so the `automatic-retries.md` doc was corrected to scope the claim precisely.
+
+Round-6 dispositions:
+
+- **Gateway wiring test missing** (`gateway_completions_factory.py:128`, cubic P2) — ✅ **RESOLVED (real gap, my round-5 omission).** `test_transport_retry_wiring.py` has a dedicated test per factory; the round-5 gateway fix shipped without the matching cases. Added `test_gateway_completions_factory_passes_max_retries` / `test_gateway_responses_factory_passes_max_retries`. (A wiring test is exactly what would have caught the round-5 gateway bug — so it earns its keep here, contra the general "don't test trivial lib-config swaps" rule.)
+- **Durable-execution retry claim** (`durable-execution.md:14`, cubic P2) — ✅ **RESOLVED (accuracy).** Same flavor as the round-4 docs fix: per-leaf retry is the Temporal model; Mistral `direct` mode retries the pipe as a unit. Qualified the bullet.
+- **choosing-a-backend table** (`choosing-a-backend.md:17`, cubic P3 ×2) — ✅ **RESOLVED.** Added the `.pipelex/pipelex.toml` config-path for `[temporal] is_enabled`, and reworded the Mistral cell to "invoke … through the runtime bridge inside your Mistral Workflows activity".
+- **`PipelexBridgeDispatchError` caller-facing flag** (`exceptions.py:22`, cubic P2) — ➖ **Declined.** The class is raised for BOTH caller-facing validation (malformed `library_crate_dump`/`delivery_assignment_dump`) AND execution wrapping that embeds downstream `{exc}` internals (sites 321/341/369/452). Marking the whole class `_authors_caller_facing_message=True` would expose execution internals under STRICT — the exact over-broad-disclosure leak greptile caught in round 3 for `PipelexSetupError`. STRICT is never the active render mode in prod (default VERBOSE — grep-confirmed no prod call site passes STRICT), so the validation detail already shows. If STRICT is ever adopted at a boundary AND validation detail is wanted there, the right fix is splitting the class, not flagging the mixed one — deferred.
+- **Bootstrap concurrent-boot race** (`bootstrap.py:39`, cubic P2 conf 8) — ✅ **RESOLVED (reachable cases) + deferred (residual).** Full triage in [`bridge-bootstrap-external-race.md`](bridge-bootstrap-external-race.md). The reachable case (two `ensure_pipelex_booted` threads) is lock-serialized and tested — `make()` runs inside `_boot_lock`, so no second `make()` races. The genuine residual is a *concurrent external* `Pipelex.make()` (not under `_boot_lock`) mid-setup; narrow/non-standard (hosts boot before serving). Tightened the docstring to state the real contract; the bounded-wait fix is recorded as deferred (poll loop + TOCTOU catch = over-engineering for a non-occurring setup).
+
+Round-7 disposition:
+
+- **Tracing assembly drops DynamoDB construction failures** (`tracing_assembly.py:98`, cubic P2 conf 8) — ✅ **RESOLVED (real regression from this PR's refactor).** Pre-PR `assemble_tracing` caught botocore directly, covering BOTH `make_event_log()` construction and `read_events()`. The refactor pushed the botocore→`EventLogReadError` conversion into `DynamoDBEventLog.read_events` only, but `__init__` eagerly builds the boto3 client — so a construction-time botocore failure (e.g. a misconfigured region) escaped the catch and would abort a completed run instead of degrading. Fixed: new `EventLogSetupError(EventLogError)`; `__init__` translates construction botocore → `EventLogSetupError`; `assemble_tracing` catches the `EventLogError` base (read + setup). Regression tests both sides; error docs regenerated.
+
+## 5. PR #969 `/review` final pass (gstack)
+
+The finalization `/review` pass (specialist army + adversarial Claude/Codex) on the full diff against `dev`. 0 critical. Findings triaged:
+
+- **WorkflowExecutionError escapes the dispatch contract** (`bridge.py:339,367`, red-team conf 8) — **decided with the user.** `_run_temporal_blocking` / `_run_temporal_fire_and_forget` catch `(PipeRunError, PipeJobError, PipeRouterError, PipeExecutionError, PipelineExecutionError)`, but `TemporalPipeRun.run()`/`.start()` actually raise `WorkflowExecutionError` (a `TemporalFlowError(PipelexError)`, not in the tuple) on a pipe failure (`workflow_caller.py:133/176`). So a Temporal-mode failure escapes `run_pipe_via_bridge` un-wrapped — asymmetric with DIRECT/mistral, and the catch tuple is dead for the dominant failure mode. The catch's presence shows uniform-wrapping was the intent. (Disposition recorded inline at finalization.)
+- **`pipe_classification` docstring falsely claimed `pipelex.temporal` uses it** — ✅ **RESOLVED (accuracy).** Zero in-tree callers; the helpers are forward-looking `MISTRAL_NATIVE` plumbing (sibling host package, not landed). Docstring corrected to say so; module kept (it is in-scope bridge-side plumbing per `TODOS.md`).
+- **`_serialize_completed_output(pipe_job=...)` unused param** (`bridge.py:383`, maintainability conf 7) — kept-for-future YAGNI; decided with the user at finalization.
+- **`LibraryManager._pipe_source_map` not per-call scoped + unlocked** (red-team conf 7/6) — ➖ **Deferred.** Pre-existing manager behaviour, observability-only impact, Temporal not in prod. Full mechanism + fix shape in [`library-manager-pipe-source-map-concurrency.md`](library-manager-pipe-source-map-concurrency.md).
+- **`output_dict` python-mode dump** (`bridge.py:80`, api-contract conf 8) — ➖ **Declined.** `dump_for_temporal()` uses `model_dump(serialize_as_any=True)`, but the model is JSON-safe via pydantic's own `model_dump(mode="json")`/`model_dump_json()` (and the kajson Temporal converter); only the naive `json.dumps(output.model_dump())` misuse fails, loudly. Usage caveat, not a defect. `extra="forbid"` skew, `trace_flush` async-IO, `delivery` pass-through, and the scoped-library 3× duplication were all re-confirmed as previously-considered/by-design.
+- **Test gaps** (testing conf 5–7) — `_decode_*` error path, `trace_flush` guards, `_run_mistral_native` success path are untested "lake" gaps; noted, added per the user's finalization call.
+
+## Open deferred follow-ups (not bugs)
+
+Carried forward, each recorded so review tooling doesn't re-raise them as new findings:
+
+- **[execution-mode-property-helpers-unused.md](execution-mode-property-helpers-unused.md)** — the three `PipelexExecutionMode` `@property` helpers are tested but not yet consumed by `bridge.py` (which inlines the equivalent checks). Wire them in or remove; no correctness impact.
+- **[library-manager-pipe-source-map-concurrency.md](library-manager-pipe-source-map-concurrency.md)** — pre-existing `LibraryManager._pipe_source_map` is manager-level, not per-call scoped. Observability-only, Temporal not in prod.
+- **[bridge-bootstrap-external-race.md](bridge-bootstrap-external-race.md)** — residual race against a *concurrent external* `Pipelex.make()` mid-setup; narrow/non-standard, bounded-wait fix deferred.
+- **[distributed-cost-reporting-test-coverage.md](distributed-cost-reporting-test-coverage.md)** — audit (from a full `/temporal-e2e-validate` Mode 2 run) of cost-reporting test coverage in split-worker mode. No automated test asserts real-inference distributed cost numbers; the real-provider *capture* branch is asserted nowhere; cross-child sum and img-gen/extract/search usage are uncovered; plus two real risks (R2 retry over-count — documented/accepted; zero-token silent emit). Code-grounded map + prioritized test additions by layer, written as the input for a follow-up planning session.
+
+## Remaining
+
+Code/docs are done. What's left is the GitHub side, per the `/review-pr-agents` skill: reply on each PR #969 thread (`✅ Fixed` / `➖ False positive`) and resolve them. No threads need to stay open — every fork was decided and applied.
diff --git a/wip/runtime-bridge/bootstrap-half-built-singleton-race.md b/wip/runtime-bridge/bootstrap-half-built-singleton-race.md
new file mode 100644
index 000000000..a5489526d
--- /dev/null
+++ b/wip/runtime-bridge/bootstrap-half-built-singleton-race.md
@@ -0,0 +1,92 @@
+# `ensure_pipelex_booted` can publish a half-built Pipelex singleton under concurrent first-boot
+
+**Status:** ✅ **APPLIED** — Approach B (instance-level `is_ready`). Real P1 race in new code, now fixed: `ensure_pipelex_booted` gates on `Pipelex.is_fully_booted()`, and the singleton is published as ready only at the very end of `make()` (after `setup()` + the optional `validate_model_deck()`). See [As applied](#as-applied) below.
+**Source:** PR #966 pre-landing review (`/review`), adversarial pass + source verification.
+**Severity:** real, but **scoped**: only bites when two threads race the *first* boot from inside activities; the recommended "boot once at the worker entry-point" path never hits it. Fails with a loud `RuntimeError`, not silent corruption.
+**Relation to the #959 "Boot race" fix:** this **extends** it. The double-checked `threading.Lock` added in #959 (`bootstrap.py`, see README §2) correctly serializes the two *writers* so the loser no longer raises `PipelexSetupError("already initialized")`. It does **not** close the second hole below: a *reader* taking the lock-free fast path while the winner is mid-`setup()`.
+
+## The finding
+
+`ensure_pipelex_booted` (`pipelex/runtime_bridge/bootstrap.py:34`) has a lock-free fast path:
+
+```python
+if Pipelex.get_optional_instance() is not None: # line 34 — no lock
+ return
+with _boot_lock:
+ if Pipelex.get_optional_instance() is None:
+ Pipelex.make(config_overrides=config_overrides)
+```
+
+The premise that makes this unsafe is that **the singleton becomes publicly visible before it is configured**:
+
+1. `MetaSingleton.__call__` registers the instance the moment `__init__` returns:
+ - `pipelex/system/registries/singleton.py:17` — `cls.instances[cls] = super().__call__(*args, **kwargs)`
+2. `Pipelex.make()` constructs **then** runs the slow `setup()`:
+ - `pipelex/pipelex.py:577` — `pipelex_instance = cls(config_overrides=config_overrides)` (registers now; `__init__:107` even swaps in a fresh **empty** hub via `set_pipelex_hub`)
+ - `pipelex/pipelex.py:579` — `pipelex_instance.setup(...)` (this is what assigns the library manager, class registry, pipe router, pipeline manager — slow: loads libraries, validates the model deck)
+3. `get_optional_instance()` reads exactly that registry:
+ - `pipelex/pipelex.py:611` — `return MetaSingleton.instances.get(cls)`
+
+**Chain:** Thread A holds `_boot_lock` and is inside `make()`, past line 577 (instance registered, empty hub installed) but still inside `setup()`. Thread B arrives fresh, hits the lock-free check at `bootstrap.py:34`, sees a non-`None` instance, returns immediately, and proceeds into `run_pipe_via_bridge` → `get_required_pipe()` / `get_library_manager()` against the still-empty hub → `RuntimeError` ("LibraryManager is not initialized" or similar partial-state failure).
+
+There is a second nasty interleaving: `make()` wraps `setup()` in `try/except BaseException` that **deletes** the instance on setup failure (`pipelex.py:601-605`). If A's `setup()` fails, B — which already returned thinking boot succeeded — is now running against an instance A just removed from the registry.
+
+## Why this is the multi-thread scenario the module targets
+
+`bootstrap.py:18-20` already calls out "Temporal's sync-activity thread pool / multi-thread workers" as the reason for the lock. The bridge is built for embedding (`pipelex-mistralai-workflows` calls `run_pipe_via_bridge` directly from inside an activity, with `ensure_pipelex_booted` as the documented in-activity first-boot). So concurrent first-boot across worker threads is a supported path, not a corner case — the lock proves the author intended it to be safe.
+
+The window is wide: `setup()` does real work (library load + model-deck validation), so it is realistically open for hundreds of ms on a cold worker.
+
+## Why the existing test does not catch it
+
+`tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py::test_concurrent_first_calls_boot_once_without_error` starts both worker threads at the same `threading.Barrier`, so both miss the fast path (no instance yet) and both queue on `_boot_lock`. The test asserts `make_calls == 1` and no raise — it verifies the *writers* are serialized. It never has a thread arrive **fresh during another thread's `setup()` window**, which is the only way to exercise the lock-free read of a half-built instance. The `fake_make` also conflates registration and setup (it appends to `instance_holder` then `time.sleep`), but the timing never lets a third party observe the in-between state.
+
+## Recommended fix
+
+> **Note:** the shape sketched here (a module-global `_booted`) is **not** what shipped — see [As applied](#as-applied). The applied fix is the `is_ready`-on-instance alternative called out in the caveat, chosen against the test lifecycle. This section is kept as the record of the original sketch and its trade-off.
+
+Gate the fast path on a "fully booted" flag flipped **only after** `make()` returns, and move the existence check inside the lock so the lock-free path can never read a half-built instance:
+
+```python
+_booted = False
+
+def ensure_pipelex_booted(config_overrides: dict[str, Any] | None = None) -> None:
+ global _booted
+ if _booted:
+ return
+ with _boot_lock:
+ if _booted:
+ return
+ if Pipelex.get_optional_instance() is None:
+ Pipelex.make(config_overrides=config_overrides)
+ _booted = True
+```
+
+This preserves the documented "adopt an externally-created singleton" contract: the first bridge call finds `_booted == False`, takes the lock, sees a fully-built external instance, skips `make()`, and sets `_booted = True`. The one requirement (already implicit today) is that any external boot must *complete* before the first bridge call — adopting a singleton another thread is mid-`make()`-ing externally is inherently racy and out of scope.
+
+### Caveat that makes this worth a deliberate decision, not a reflex fix
+
+`_booted` is module-global, so it survives across tests. The existing concurrency test (and any test that boots/tears down Pipelex) needs a reset hook (e.g. a fixture that sets `bootstrap._booted = False`), or the flag has to live somewhere resettable. A "check the instance is fully set up" alternative (an `is_ready` flag on `Pipelex` flipped at the end of `setup()`) avoids the module global but is more invasive (touches `pipelex.py`). Pick the shape against the boot/teardown lifecycle the test suite expects — hence deferred rather than auto-applied in the review pass.
+
+## As applied
+
+Shipped the `is_ready`-on-`Pipelex` alternative from the caveat, **not** the module-global `_booted` sketch above. The deciding factor is exactly the test-lifecycle point the caveat raised: the suite's module-scoped autouse fixture (`tests/conftest.py`) tears Pipelex down with `Pipelex.teardown_if_needed()`, which deletes the singleton from `MetaSingleton.instances`. An instance-level flag dies with the instance, so it auto-resets between test modules — no module-global to leak, no reset fixture to add, no second source of truth that can drift from the registry.
+
+Changes:
+
+- `pipelex/pipelex.py` — `__init__` initializes `self.is_ready = False` (set before the metaclass registers the instance, so it always exists before the instance is observable); `make()` flips `pipelex_instance.is_ready = True` on its success tail, **after** the `try/except`, so it is set only once `setup()` and the optional `validate_model_deck()` have both succeeded and the delete-on-failure handler is behind us; new classmethod `is_fully_booted()` returns `instance is not None and instance.is_ready`.
+- `pipelex/runtime_bridge/bootstrap.py` — `ensure_pipelex_booted` gates both the lock-free fast path and the in-lock re-check on `Pipelex.is_fully_booted()` instead of `get_optional_instance() is not None`. A registered-but-not-ready instance (mid-`setup()`, or about to be deleted on setup failure) is now treated as not-booted, so the reader blocks on the lock and re-checks. The "adopt an externally-created singleton" contract is preserved: a fully-booted external instance is `is_ready` → no-op.
+
+## Tests (done)
+
+`tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py` (all mock at the `Pipelex` classmethod boundary — never touching `MetaSingleton.instances`, so they coexist with the autouse session singleton):
+
+- `test_reader_arriving_mid_setup_blocks_until_ready` — three-thread test: A is held inside a patched `make()` that registers an `is_ready=False` instance, then flips it ready only just before returning; B and C arrive during the window and must block on the lock until A finishes (asserted via `is_alive()` mid-window). Fails against the pre-fix bare-presence check.
+- `test_setup_failure_lets_next_thread_reboot` — A's patched `make()` registers then clears + raises (mirroring delete-on-failure); B must fall through to the lock, find nothing booted, and re-boot rather than proceed against the deleted instance.
+- The original `test_concurrent_first_calls_boot_once_without_error` keeps the `make_calls == 1` assertion (no regression of the #959 write-write fix).
+
+## Related
+
+- README §2 "Boot race — ✅ applied" (the #959 double-checked lock this extends).
+- `pipelex/system/registries/singleton.py` (`MetaSingleton` — registration-at-construction).
+- `pipelex/pipelex.py:577-607` (`make` construct-then-setup, with delete-on-failure).
diff --git a/wip/runtime-bridge/bridge-bootstrap-external-race.md b/wip/runtime-bridge/bridge-bootstrap-external-race.md
new file mode 100644
index 000000000..89b2f840e
--- /dev/null
+++ b/wip/runtime-bridge/bridge-bootstrap-external-race.md
@@ -0,0 +1,50 @@
+# Bridge bootstrap vs a concurrent external Pipelex.make() — narrow, documented, deferred
+
+**Status:** ✅ RESOLVED for the reachable/standard cases (lock-serialized, tested). The residual race against a *concurrent external* boot is narrow/non-standard; the docstring now states the real contract, and the bounded-wait fix is recorded below as deferred (adding it now would be over-engineering for a config standard hosts don't produce).
+
+**Raised by:** PR #969 round-6 review — cubic-dev-ai **P2** (confidence 8), `pipelex/runtime_bridge/bootstrap.py:39`: "`ensure_pipelex_booted()` can raise `Pipelex is already initialized` during concurrent startup when another thread is already mid-boot, instead of safely adopting/waiting for that boot."
+
+## What's actually true
+
+`ensure_pipelex_booted()` is:
+```python
+if Pipelex.is_fully_booted():
+ return
+with _boot_lock:
+ if not Pipelex.is_fully_booted():
+ Pipelex.make(config_overrides=config_overrides)
+```
+`Pipelex.make()` raises `PipelexSetupError("Pipelex is already initialized")` if any instance is already registered (pipelex.py). `is_fully_booted()` is `instance is not None and instance.is_ready`; `is_ready` flips True only at the tail of `make()` (after setup + optional deck validation; a failed boot deletes the instance).
+
+**Reachable cases are safe** — verified against the three tests in `tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py`:
+
+- *Two `ensure_pipelex_booted` threads.* `make()` runs **inside** `with _boot_lock`, so thread A holds the lock for its entire boot. Thread B blocks on the lock and, when it finally enters, sees `is_fully_booted()==True` and skips. No second `make()`, no raise. (`test_concurrent_first_calls_boot_once_without_error`)
+- *Reader arriving mid-setup.* B/C gate on `is_fully_booted()` (not bare presence), so they block on the lock through A's half-built window and never read a not-ready instance. (`test_reader_arriving_mid_setup_blocks_until_ready`)
+- *Setup failure.* A registers half-built, fails, deletes the instance; B (held on the lock) then re-boots cleanly. (`test_setup_failure_lets_next_thread_reboot`)
+
+The cubic write-up's two-`ensure_pipelex_booted`-threads scenario is **not** a real race: it assumes B can run `make()` while A is mid-`make()`, but the lock makes that impossible.
+
+## The genuine residual (scenario B)
+
+The lock only serializes the bridge's *own* callers. A **separate external** `Pipelex.make()` (e.g. `worker_cli.py`, `codec_server_cli.py`, a CLI factory) does **not** take `_boot_lock`. If such an external boot is mid-setup (`is_ready=False`) on one thread while the bridge's `ensure_pipelex_booted()` runs on another, the bridge sees `is_fully_booted()==False`, acquires the (uncontended) lock, calls `make()`, and hits "already initialized".
+
+**Reachability: narrow.** Each external `make()` call site is a process entry point that boots Pipelex *before* serving work; activities (which reach the bridge) run after boot, when `is_fully_booted()` is already True and the bridge fast-paths. Hitting scenario B requires a non-standard host that boots Pipelex on one thread while *concurrently* serving bridge traffic on another. No code in the repo does this.
+
+## Decision: document the contract, defer the bounded-wait
+
+The docstring previously over-promised ("adopts that singleton without re-initializing") without qualifying the mid-boot window. It now states the real contract: the bridge serializes its own boot and adopts a *fully-booted* external singleton; it does **not** support racing a concurrent external `make()` that is mid-initialization — boot Pipelex before bringing up concurrent bridge callers. No code change: the failure mode is a loud raise (not a silent corruption) in a setup nothing standard produces.
+
+## Deferred: bounded-wait adoption
+
+If a real host ever needs to boot Pipelex concurrently with serving bridge traffic, make `ensure_pipelex_booted()` wait out a mid-flight external boot instead of colliding. Because a failed boot deletes the instance, the wait terminates:
+
+```python
+with _boot_lock:
+ while not Pipelex.is_fully_booted():
+ if Pipelex.get_optional_instance() is None:
+ Pipelex.make(config_overrides=config_overrides) # still needs to tolerate a TOCTOU raise vs external
+ else:
+ time.sleep(...) # external boot mid-setup: wait for is_ready, or for it to fail+disappear
+```
+
+This adds a poll loop, holds `_boot_lock` while waiting (acceptable — other bridge callers would block anyway), and must also catch the `make()` "already initialized" raise for the external-registers-between-check-and-make TOCTOU. That fragility for a non-occurring setup is why it's deferred, not done.
diff --git a/wip/runtime-bridge/bridge-error-name-collision.md b/wip/runtime-bridge/bridge-error-name-collision.md
new file mode 100644
index 000000000..fe245c2dd
--- /dev/null
+++ b/wip/runtime-bridge/bridge-error-name-collision.md
@@ -0,0 +1,42 @@
+# Bridge error name collision — `PipelexRuntimeBridgeError` vs `PipelexBridgeRuntimeError`
+
+**Status:** ✅ **RESOLVED — leaf renamed to `PipelexBridgeDispatchError`.** (Option A.) Applied: renamed the class in `exceptions.py`; updated the import + raise sites in `bridge.py` and the `pytest.raises` references in `test_validation.py`; regenerated `docs/errors/` (`pipelex-bridge-runtime-error.md` removed, `pipelex-bridge-dispatch-error.md` written, `index.md` updated — no spurious mistral-page churn); updated the `TODOS.md` prose. `make agent-check` + `make agent-test` green. Original triage retained below.
+
+**Spotted:** during the `dev` → `feature/Runtime-bridge-extraction` merge triage. Not a merge regression and not a stale duplicate — verified both classes are live and their docs are in sync (`pipelex-dev generate-error-pages` reports `Removed: 0`, no orphans).
+
+## The smell
+
+`pipelex/runtime_bridge/exceptions.py` declares two classes whose names differ **only by word order**:
+
+```python
+class PipelexRuntimeBridgeError(PipelexError): # :4 — base for the whole runtime-bridge surface
+class PipelexBridgeRuntimeError(PipelexRuntimeBridgeError): # :16 — concrete leaf: a bridge-dispatched pipe execution failed
+```
+
+"Runtime Bridge" vs "Bridge Runtime". They are genuinely distinct — the base also parents `MissingPipelexTemporalExtraError` and `MissingMistralWorkflowsPluginError` — but the near-mirror names are easy to misread, mistype, and mis-`except`. The leaf reads almost like its own base, which is exactly backwards. The two also generate two almost-identical doc-page slugs (`pipelex-runtime-bridge-error` / `pipelex-bridge-runtime-error`) that a reader can't tell apart at a glance.
+
+This is a readability/API-clarity issue only: the kebab slugs are distinct (no `type_uri` collision), and nothing is functionally wrong.
+
+## The fork
+
+- **(A) Rename the concrete leaf** to a self-evident name decoupled from the base's word-order — e.g. `BridgeDispatchError`, `BridgePipeExecutionError`, or `BridgeRunError`. The base `PipelexRuntimeBridgeError` (the namespace error) stays. This is the recommended direction: the leaf is the one at the call sites and the one whose meaning ("a pipe dispatched through the bridge blew up") should be obvious from its name.
+- **(B) Rename the base** instead (e.g. `BridgeError` as the family root) and keep the leaf — less compelling, since the base is rarely referenced directly and the family-prefix convention (`Pipelex…`) is intentional.
+- **(C) Leave it.** Zero churn; the confusion persists for every future reader and `except` site.
+
+## Recommendation
+
+Take **(A)**. Pick a leaf name that states what failed, not where it sits in the hierarchy. Per project policy there's no back-compat to preserve, so it's a clean rename.
+
+## Blast radius (small, contained)
+
+- `pipelex/runtime_bridge/exceptions.py` — the class definition.
+- `pipelex/runtime_bridge/bridge.py` — one import + the `raise` sites.
+- `tests/unit/pipelex/runtime_bridge/test_validation.py` — the `pytest.raises(...)` references.
+- `docs/errors/pipelex-bridge-runtime-error.md` — regenerated, old slug page removed (it's `<!-- pipelex:generated -->`, safe to delete).
+- `TODOS.md` mentions the name in prose — update if still relevant.
+
+## How to apply
+
+1. Rename the class in `exceptions.py`; update the import + raises in `bridge.py` and the references in `test_validation.py`.
+2. `pipelex-dev generate-error-pages` to refresh `docs/errors/` (writes the new slug page, removes the old one) and the `docs/errors/index.md` entry. ⚠️ This worktree runs `mistralai==2.4.4`; the generator's import-walk has tolerated it so far, but eyeball the diff for spurious mistral-page churn.
+3. `make agent-check` + `make agent-test`.
diff --git a/wip/runtime-bridge/bridge-keyed-tracer-unsupported.md b/wip/runtime-bridge/bridge-keyed-tracer-unsupported.md
new file mode 100644
index 000000000..a849d963d
--- /dev/null
+++ b/wip/runtime-bridge/bridge-keyed-tracer-unsupported.md
@@ -0,0 +1,36 @@
+# Bridge + keyed tracers (tracer_key ≠ graph_id) — unreachable, unsupported by contract
+
+**Status:** ✅ RESOLVED (refuted as unreachable; `lookup_key` kept with a precise comment; **no guard code** — that would be over-engineering for a configuration nothing produces). Bigger-hammer structural option recorded below for if keyed tracers ever need to flow through the bridge.
+
+**Raised by:** PR #969 round-4 review — greptile-apps **P1** (`bridge.py:179`, "Separate tracer keys") + cubic-dev-ai **P2** (same line, confidence 6).
+
+## The finding
+
+`build_pipe_job_from_input` derives the run id, when no explicit `pipeline_run_id` is given, as `trace_context.lookup_key` (= `tracer_key or graph_id`). The reviewers observed: when a host's `tracer_key != graph_id`, trace events are emitted under the tracer's *event* `pipeline_run_id` (which falls back to `graph_id`), **not** under `tracer_key`. But `PipeRun.run` assembles tracing by `job_metadata.pipeline_run_id` (now = `tracer_key`). So assembly would read the `tracer_key` partition, find no events, and the run would silently return an empty graph + no cost data — even though events were emitted.
+
+This is **correct analysis** of an abstract data-flow. It is **not a reachable bug**, and the convenient "fix" is wrong — so the resolution is a tightened comment + refutation, not new code.
+
+## Why it is not a reachable bug
+
+Verified against the code (`pipe_run.py`, `bridge.py`, `wf_pipe_router.py`, `trace_context.py`):
+
+- The in-process `PipeRun.run` keys **both** `close_tracer` (pipe_run.py:60) **and** `assemble_tracing_on_output` (pipe_run.py:78) off the **single** `job_metadata.pipeline_run_id` (pipe_run.py:35). One key, not two — and events are emitted under `pipeline_run_id or graph_id`.
+- The bridge **nulls `trace_context` for all non-DIRECT modes** (bridge.py:133), so only DIRECT runs ever receive one.
+- The **only** place that opens a keyed tracer (`tracer_key != graph_id`) is the Temporal child path `wf_pipe_router` — and it **never calls the bridge** (`grep`-confirmed: no `build_pipe_job_from_input` / `run_pipe_via_bridge` under `pipelex/temporal/`). It uses its own activity-based assembly, not `PipeRun.run`.
+- Every bridge-reachable caller opens its tracer the way `pipeline_run_setup` does — `graph_id == pipeline_run_id`, `tracer_key` unset — so `lookup_key == graph_id == event-partition` and all three keys coincide.
+
+So the divergence requires a *host integrator* to hand-build a Temporal-style keyed `TraceContext` and force it through the DIRECT bridge — which nothing does, and which the contract ("open a tracer for this pipeline run"; `graph_id` "typically the pipeline run id") does not anticipate. No internal Pipelex code reaches the bridge with a keyed trace_context.
+
+## Why `lookup_key` is the right value (and `graph_id` is not)
+
+The convenient counter-"fix" — switch back to `trace_context.graph_id` — is **wrong**: it re-breaks `close_tracer` for any keyed tracer (`close_tracer` needs the registration key, `lookup_key`). `lookup_key` and `graph_id` differ only for a divergent keyed tracer, which (per above) never reaches the bridge — so on every reachable path `lookup_key == graph_id` and `lookup_key` is correct for registration, close, event-emission, and assembly simultaneously.
+
+## Decision: comment, not guard
+
+An earlier round-4 attempt added a loud boundary guard that rejected a divergent `tracer_key` with `PipelexBridgeDispatchError`. It was **backed out** on review (this PR) as over-engineering: it defended a configuration no code path produces (only host misuse), it was only *partial* (it caught `tracer_key ≠ graph_id` but could not catch a host opening with `tracer_key=None, pipeline_run_id ≠ graph_id`, which the bridge cannot see from `graph_id` + `tracer_key` alone), and it sat on top of the round-3 `lookup_key` change that was itself defending the same unreachable case. The net of two rounds of churn over a non-occurring config was the smell.
+
+Resolution: keep `lookup_key`, document the reachability + the unsupported keyed-tracer contract in the code comment at the derivation site, and refute the review thread with the reachability proof. No behavior change, no new code.
+
+## Deferred: if keyed tracers ever need bridge support
+
+The structural fix, only worth it once a real caller needs `tracer_key != graph_id` through the bridge: give `TraceContext` an explicit event-partition field (the value the host passed to `open_tracer(pipeline_run_id=...)`) so the bridge stops *inferring* it from `graph_id`/`tracer_key`. Then `JobMetadata.pipeline_run_id` (assembly/close key) and the event-partition can be carried independently end-to-end. This touches `TraceContext` across `graph/`, `temporal/`, and the in-process tracing path, so it is over-engineering for a configuration nothing currently produces — hence deferred, not done.
diff --git a/wip/runtime-bridge/cost-reporting-test-cleanup-followups.md b/wip/runtime-bridge/cost-reporting-test-cleanup-followups.md
new file mode 100644
index 000000000..351ec5c0f
--- /dev/null
+++ b/wip/runtime-bridge/cost-reporting-test-cleanup-followups.md
@@ -0,0 +1,57 @@
+# Cost-reporting test additions — two cleanup follow-ups
+
+**Status:** 🧹 Optional cleanups, no correctness impact. Both items resolved. Surfaced by a code review of the distributed-cost-reporting test additions (see [`distributed-cost-reporting-test-coverage.md`](distributed-cost-reporting-test-coverage.md), "Resolution (as-built)"). The review found **no bugs**; these were cosmetic dead-code / weak-assertion items.
+
+---
+
+## 1. Dead CSV fallback + never-present NDJSON key in the skill helper ✅ DONE
+
+**Resolution:** Verified against `pipelex/cogt/usage/cost_registry.py` and applied the fix. `INPUT_JOINED` is written into an `nb_tokens_by_category` dict only inside `complete_cost_report` (which `pop`s `INPUT` first), and the CSV is written exclusively from completed reports — so the column is always `nb_tokens_input_joined`, never `nb_tokens_input`. Raw NDJSON `UsageReportEvent`s only ever carry `input`. Dropped the `nb_tokens_input` CSV fallback and set `_INPUT_KEYS = ("input",)`. (Note: a `NB_TOKENS_INPUT = "nb_tokens_input"` enum value does exist in `llm_report.py`, but it is never emitted into a record, so the conclusion holds.)
+
+**File:** `.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py`
+
+Two bits of defensive code reference a data shape that never actually occurs, which misleads a reader into thinking a second format exists:
+
+- **`_csv_token_totals` (~line 224):** `int(float(row.get("nb_tokens_input_joined") or row.get("nb_tokens_input") or 0))`. The cost-report CSV is written by `CostRegistry.save_to_csv(to_records(...))`, whose records only ever carry the column `nb_tokens_input_joined` — never `nb_tokens_input`. The `or row.get("nb_tokens_input")` branch is unreachable.
+- **`_sum_tokens` / `_INPUT_KEYS = ("input", "input_joined")`:** sums an `input_joined` key that is never present in a *raw* `UsageReportEvent`. `INPUT_JOINED` is synthesized later, inside `CostRegistry.complete_cost_report` (it pops `INPUT` and renames it on a copy used only for costing). The emitted event's `nb_tokens_by_category` always has `input`, never `input_joined`, so the extra key always contributes 0.
+
+**Why it's harmless today:** neither produces a wrong total — the CSV column genuinely matches the NDJSON `input` sum, and the missing keys add 0.
+
+**The real (latent) risk:** if the CSV schema ever *did* emit both `nb_tokens_input` and `nb_tokens_input_joined`, the union-sum logic would silently double-count input. Worth simplifying to just the column that exists.
+
+**Fix:** drop the `nb_tokens_input` fallback; set `_INPUT_KEYS = ("input",)`. Re-confirm the CSV column name in `pipelex/cogt/usage/cost_registry.py` (`save_to_csv` + `to_records`, and the `*TokenCostReportField` enums where `NB_TOKENS_INPUT_JOINED = "nb_tokens_input_joined"`) before editing, in case it has since changed.
+
+---
+
+## 2. Vacuous + duplicated sentinel guard in the real-inference test
+
+**File:** `tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py`
+
+The test declares a local `_FAKE_RUNNER_MODEL_NAME = "split_runner_fake"` and asserts (near the end of `test_real_provider_usage_aggregates_to_nonzero_cost`):
+
+```python
+assert _FAKE_RUNNER_MODEL_NAME not in model_names, "Real inference must report a real model handle, not the fake sentinel"
+```
+
+**Problem:** this test explicitly installs `runner_act_llm_gen_text=_real_runner_act_llm_gen_text` (real inference), so the *default* fake substitute — the only thing that ever emits `"split_runner_fake"` — is never wired in. The asserted string can never appear regardless of whether real inference actually ran, so the guard can never fail (false confidence).
+
+**Drift risk:** `"split_runner_fake"` is re-declared as a local literal instead of imported from the helper that owns it (`tests/integration/pipelex/temporal/tracing/helpers.py`, `_RUNNER_FAKE_INFERENCE_MODEL_NAME`). If the helper's sentinel is renamed, this assertion silently checks a stale string and keeps "passing" while guarding nothing.
+
+**Fix options (pick one):**
+
+- Drop the assertion — the adjacent `assert all(model_names)` (every usage record names a real handle) plus the non-zero-token assertions already prove real inference ran. This is the cleaner option; the sentinel-absence check adds nothing once the default substitute isn't in play.
+- Or, if a guard against accidentally falling back to the fake substitute is still wanted, import `_RUNNER_FAKE_INFERENCE_MODEL_NAME` from `helpers.py` rather than re-declaring it, so a rename can't rot it.
+
+---
+
+## Verification
+
+These files' tests are green as-is. After either change, re-run:
+
+```bash
+# helper (#1) — no test, just exercise it against a fresh run dir, or eyeball
+# real-inference test (#2) is marked inference/llm (opt-in spend); the no-spend lane:
+.venv/bin/pytest -p no:cacheprovider -q \
+ tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py \
+ tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py
+```
diff --git a/wip/runtime-bridge/cost-reporting-validation-prompts.md b/wip/runtime-bridge/cost-reporting-validation-prompts.md
new file mode 100644
index 000000000..417e8d74b
--- /dev/null
+++ b/wip/runtime-bridge/cost-reporting-validation-prompts.md
@@ -0,0 +1,50 @@
+# Cost-reporting validation — prompts for `/temporal-e2e-validate`
+
+Operator cheat-sheet for driving the distributed cost-reporting validation through the `/temporal-e2e-validate` skill. The skill's cost route is **scope-aware**: a bare cost request runs only the free deterministic arms; an explicit spend opt-in runs every arm including the live (real-money) ones. The arms (A–G) and their numeric assertions are defined in the skill's scope manifest at `.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md` (Step 5b', Tier 8b). The gap audit that motivated them is the sibling [`distributed-cost-reporting-test-coverage.md`](distributed-cost-reporting-test-coverage.md).
+
+## Prompt menu
+
+### Full thorough run — real spend (arms A–G)
+
+```text
+/temporal-e2e-validate cost reporting full
+```
+
+Routes to Tier 8b and runs every arm: mock primary + cross-child fan-out + CSV cross-check + `--no-costs` negative gate + live LLM + live img-gen + live extract — each asserted numerically with `assert_cross_worker_cost.py`, PASS/FAIL surfaced per arm. `thorough`, `every`, and `with-spend` are aliases of `full`:
+
+```text
+/temporal-e2e-validate distributed cost — every arm
+```
+
+### Free, deterministic pass — no spend (arms A–B)
+
+```text
+/temporal-e2e-validate cost reporting
+```
+
+Mock primary + cross-child fan-out only (`--mock-inference`). The quick regression to run before committing or before paying for the live arms.
+
+### Spend, but ask first
+
+```text
+/temporal-e2e-validate validate distributed cost on a live run
+```
+
+`live` / `all` are deliberately **not** treated as a spend opt-in (too easily incidental; `live` also collides with the default mock arm already running in LIVE mode). When that's the only signal, the agent runs the free arms and **confirms with you before** the real-money arms.
+
+### Full run, LLM cost path only
+
+```text
+/temporal-e2e-validate cost reporting full, LLM arms only (skip img-gen/extract)
+```
+
+Use when only LLM inference is wired up in your env (see caveat below).
+
+## Caveat — live non-LLM arms need backends
+
+Arms F (img-gen) and G (extract) are **live-only** — `--mock-inference` can't reach them (their mock leaves raise `MockInferenceUnsupportedError`), so they're the sole proof that non-LLM usage crosses the runner fallback and aggregates into the cost report. They need real img-gen and extract backends configured (extract = Azure Doc Intel via the Pipelex Gateway). If those aren't set up, the two arms fail at the **run** step, not the assertion — and the manifest's "stop and report" rule means you'll see exactly which arm and why.
+
+## Recommended sequence
+
+1. `/temporal-e2e-validate cost reporting` — free, fast, confirms the capture → emit → fallback → aggregate → render plumbing.
+2. `/temporal-e2e-validate cost reporting full` — spend on the live arms once the cheap ones are green.
diff --git a/wip/runtime-bridge/direct-mode-nested-router-leak.md b/wip/runtime-bridge/direct-mode-nested-router-leak.md
new file mode 100644
index 000000000..66268f133
--- /dev/null
+++ b/wip/runtime-bridge/direct-mode-nested-router-leak.md
@@ -0,0 +1,129 @@
+# DIRECT mode: nested controller sub-pipes leak to the hub default router
+
+**Status:** ✅ **RESOLVED — Option A applied.** (was: confirmed bug, deferred pending decision)
+**Source:** PR #959 review — greptile-apps (P1, `bridge.py:252-258`); the enabler is cubic-dev-ai (P2, `hub.py:615`, "prefer a `scoped_pipe_router` context manager").
+**Severity:** real but **latent** — only bites when a worker runs with `[temporal] is_enabled = true`, and Temporal hasn't shipped to prod. In the common (non-Temporal) worker it was silently harmless *except* for an observer inconsistency (see the fork, now resolved).
+
+## Resolution (Option A)
+
+Decision: **Option A** — scope the bridge's bare `PipeRouter()` for the whole DIRECT run. The observer sub-question was investigated and cleared the way for A: the router-level observer's *only* job is best-effort PostHog telemetry, it's inert by default and only live for Gateway users, the DIRECT root pipe already emitted none, and Temporal mode already uses `ObserverNoOp`. Threading the observer in (Option B) would have required a hub `get_observer` getter that doesn't exist — re-coupling the bridge to internals. Full analysis: [`../observer-and-telemetry/observer-telemetry-posthog.md`](../observer-and-telemetry/observer-telemetry-posthog.md).
+
+Applied:
+
+- Added `scoped_pipe_router(pipe_router)` to `pipelex/hub.py` (mirrors `scoped_current_library`: captures the prior `_current_pipe_router` override and restores it on exit — fixes cubic's hub.py:615/teardown-clobber point too). The raw `set_pipe_router` / `teardown_current_pipe_router` pair is kept for the external `pipelex-mistralai-workflows` plugin.
+- Wrapped `_run_direct` (`pipelex/runtime_bridge/bridge.py`) in `with scoped_pipe_router(direct_router):` so root + nested sub-pipes both resolve the in-process router. Kills the Temporal leak.
+- Tests: `tests/unit/pipelex/pipe_run/test_scoped_pipe_router.py` (set/restore/nested/raise semantics) and `tests/unit/pipelex/runtime_bridge/test_direct_router_scoping.py` (nested dispatch resolves the scoped router, not the hub default; override torn down after the run). `make agent-check` + `make agent-test` green.
+
+Everything below is the original triage write-up, retained as the record of why.
+
+---
+
+## The finding
+
+> `DIRECT` mode creates a fresh `PipeRouter`, so only the root pipe uses the bridge-local direct runner. Controller pipes dispatch their sub-pipes through `get_pipe_router()`, which resolves to the hub or context router. In a worker booted with Temporal enabled, a direct bridge run of a sequence or batch can unexpectedly send nested pipes to Temporal instead of running in-process.
+
+## Verified behavior (code trace)
+
+1. `_run_direct` builds a **bare** router and hands it only to `PipeRun`:
+ - `pipelex/runtime_bridge/bridge.py:256` — `pipe_run = PipeRun(pipe_router=PipeRouter())`
+2. `PipeRun.run` uses that router **only for the root pipe**, and never installs it into the routing ContextVar:
+ - `pipelex/pipe_run/pipe_run.py:42` — `pipe_output = await self._pipe_router.run(pipe_job)`
+3. Controllers dispatch every sub-pipe via `get_pipe_router()` — *not* the parent's router instance:
+ - `pipelex/pipe_controllers/sub_pipe.py:179` (batch), `:195` (condition), `:229` (normal) — all `await get_pipe_router().run(...)`
+4. `get_pipe_router()` returns the ContextVar override if set, else the **hub default**:
+ - `pipelex/hub.py:633-637`
+ ```python
+ def get_pipe_router() -> "PipeRouterProtocol":
+ override = _current_pipe_router.get()
+ if override is not None:
+ return override
+ return get_pipelex_hub().get_required_pipe_router()
+ ```
+5. The hub default is **Temporal-aware** when Temporal is enabled:
+ - `pipelex/pipelex.py:446-452`
+ ```python
+ if pipe_router:
+ self.pipelex_hub.set_pipe_router(pipe_router)
+ elif get_config().temporal.is_enabled:
+ self.pipelex_hub.set_pipe_router(make_temporal_pipe_router()) # TemporalPipeRouter
+ else:
+ self.pipelex_hub.set_pipe_router(PipeRouter(observer=multi_observer))
+ ```
+
+**Chain:** `_run_direct` never sets `_current_pipe_router` → nested `get_pipe_router()` sees no override → returns the hub default → which is `TemporalPipeRouter` in a Temporal-enabled worker → **nested sub-pipes of a DIRECT-mode sequence/batch get dispatched to Temporal**, contradicting the whole point of DIRECT (run in-process, here, now).
+
+This is not intentional: `set_pipe_router`'s own docstring (`pipelex/hub.py:618-619`) describes exactly this use case — *"Used by host runtimes that want controllers to dispatch sub-pipes through their own router."* DIRECT mode simply forgets to use it.
+
+The greptile comment also notes a Mistral-native variant of the same root cause: a `set_pipe_router` override installed by a host is honored by nested pipes (via `get_pipe_router()`) but bypassed by the root pipe (which runs on the bare `PipeRouter()`), so root vs nested dispatch is inconsistent.
+
+## Provenance — new in #959, not pre-existing in dev
+
+Checked against `dev` (`../pipelex`, HEAD `fa15d15c` — the commit this branch was cut from, no `runtime_bridge/`). **This bug is introduced by the bridge PR.** Both ingredients are new:
+
+- dev's `get_pipe_router()` (hub.py:612-613) has **no override layer** — it just `return get_pipelex_hub().get_required_pipe_router()`. The PR added `_current_pipe_router` + the module-level `set_pipe_router`/`teardown_current_pipe_router`.
+- dev never builds a fresh bare router for a root run: the hub's `PipeRun` uses the *hub's* router (`pipelex.py:461` — `PipeRun(pipe_router=self.pipelex_hub.get_required_pipe_router())`), and nested sub-pipes resolve the *same* hub router. So in dev root and nested always agree — non-Temporal → both `PipeRouter(observer=multi_observer)` (no observer split either); Temporal-enabled → both the Temporal router. There is no "in-process root while Temporal is enabled" concept in dev.
+
+The bridge introduces that concept (`DIRECT` = force in-process even in a Temporal worker) via the bare `PipeRouter()` in `_run_direct`, but doesn't install it into the new ContextVar — hence the leak. The **observer split** is likewise bridge-introduced.
+
+Pre-existing look-alike (out of scope, flagged for honesty, not #1): `pipelex/pipeline/bundle_validator.py:106` builds `PipeRun(pipe_router=PipeRouter(observer=ObserverNoOp()))` — same "fresh root router, nested falls to `get_pipe_router()`" shape, but it's the validation/dry-run path, predates the bridge, and is low-risk. Worth its own look someday, separately.
+
+## The mechanism for the fix (resolves cubic's hub.py:615 too)
+
+There's an established pattern in the same file — `scoped_current_library` (`pipelex/hub.py:517-532`) — a context manager that captures the prior ContextVar value, sets the new one, and restores on exit:
+
+```python
+@contextmanager
+def scoped_current_library(library_id: str) -> Generator[None, None, None]:
+ prev = _library_id.get()
+ _library_id.set(library_id)
+ try:
+ yield
+ finally:
+ _library_id.set(prev)
+```
+
+The router ContextVar has only the raw pair today, and the teardown clobbers any outer override:
+
+- `pipelex/hub.py:615` `set_pipe_router(router)` → `_current_pipe_router.set(router)`
+- `pipelex/hub.py:628-630` `teardown_current_pipe_router()` → `_current_pipe_router.set(None)` (unconditional — **does not restore a prior override**, cubic's P2)
+
+**Proposed shared fix:** add `scoped_pipe_router(router)` mirroring `scoped_current_library`, then wrap the DIRECT run in it so the in-process router is the active router for the *whole* call (root + nested). Keep the raw `set_pipe_router` / `teardown_current_pipe_router` — the external `pipelex-mistralai-workflows` plugin depends on them (separate repo, can't edit here) — but prefer the scoped helper internally.
+
+```python
+# bridge.py _run_direct
+direct_router = PipeRouter()
+with scoped_pipe_router(direct_router):
+ pipe_run = PipeRun(pipe_router=direct_router)
+ pipe_output = await pipe_run.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)
+```
+
+## The fork — what to do about the observer
+
+The hub default in-process router is `PipeRouter(observer=multi_observer)` (`pipelex.py:452`), but the bridge builds a bare `PipeRouter()` (`observer=None`, see `pipe_router.py:10`). So **today**, in a non-Temporal worker:
+
+- root pipe → bridge's bare router (no observer)
+- nested pipes → hub default router (**with** `multi_observer`)
+
+Scoping the bridge's bare router makes nested pipes consistent with the root — but they'd **lose** the router-level `multi_observer`. Open sub-question: does that matter? The hub *also* sets the observer separately (`pipelex.py:424` `set_observer(multi_observer)`), and the report delegate is reachable via the hub — so the router-level observer may be partly redundant. Needs a quick check of what `MultiObserver` does on a `PipeRouter` vs via the hub before choosing.
+
+**Option A — Fix now, bare router (recommended).** Scope the bridge's bare `PipeRouter()` for the whole DIRECT run.
+- Pros: smallest change; root + nested become consistent; matches what `_run_direct` already builds for the root; kills the Temporal-leak. Also delivers cubic's `scoped_pipe_router` request.
+- Cons: nested pipes lose the router-level `multi_observer` in non-Temporal workers (behavior change — possibly fine if the hub observer covers it, possibly a telemetry regression).
+
+**Option B — Fix now, preserve observer.** Scope an in-process router that carries `multi_observer`.
+- Pros: no observer behavior change; still kills the leak.
+- Cons: more plumbing — the bridge must source `multi_observer` (it lives in `Pipelex.make`, not trivially reachable from the framework-agnostic bridge); risks coupling the bridge back to internals it was extracted to avoid.
+
+**Option C — Defer.** Leave routing untouched; this doc is the record.
+- Pros: zero risk now; the bug can't fire until a Temporal-enabled worker ships.
+- Cons: leaves a known P1 latent in new code; the root-vs-nested observer split persists.
+
+## Test ideas (when fixed)
+
+- Unit: `scoped_pipe_router` — `get_pipe_router()` returns the scoped router inside the block; nested `scoped_pipe_router` restores the *outer* override (not `None`) on inner exit; restores prior on outer exit.
+- Bridge: install a sentinel/recording router as the hub default, run a sequence pipe in `DIRECT` mode via `run_pipe_via_bridge`, assert nested dispatch went through the scoped in-process router and **not** the hub default. (Optionally with `temporal_enabled=True` to prove no Temporal dispatch.)
+
+## Related
+
+- `graph-context-temporal-contract.md` — sibling bridge contract issue (#3).
+- The raw `set_pipe_router`/`teardown_current_pipe_router` consumer lives in `pipelex-mistralai-workflows` (other repo) — if `scoped_pipe_router` lands, that plugin can adopt it later to fix its own unconditional teardown.
diff --git a/wip/runtime-bridge/distributed-cost-reporting-test-coverage.md b/wip/runtime-bridge/distributed-cost-reporting-test-coverage.md
new file mode 100644
index 000000000..9f11e526b
--- /dev/null
+++ b/wip/runtime-bridge/distributed-cost-reporting-test-coverage.md
@@ -0,0 +1,214 @@
+# Cost reporting in distributed (split-worker) mode — test-coverage audit
+
+**Status:** ✅ **RESOLVED — coverage added.** The prioritized additions below were implemented and are green. See "Resolution (as-built)" for the exact files and the decisions taken. The audit body is retained as the code-grounded map and rationale.
+**Source:** Audit triggered during `/temporal-e2e-validate` (full Mode 2 run on `feature/Runtime-bridge-extraction`). Question asked: *do we have comprehensive testing of cost reporting in distributed mode with real inference — emission and aggregation — and what should we add?*
+**Scope:** the token-usage → cost-report path when inference runs on a **separate Temporal `runner` worker process** (split-worker topology), with **real** provider inference. Touches `pipelex/reporting/`, `pipelex/tracing/`, `pipelex/cogt/usage/`, `pipelex/temporal/tprl_pipe/`, and the test tree under `tests/`.
+
+---
+
+## Resolution (as-built)
+
+All four audit questions now have automated coverage. No production code changed — these are characterization/regression tests plus one test-helper extension and a skill upgrade. **R2 decision: keep accepting** the activity-retry over-count (matches the existing documented stance); it is now regression-visible at the cost-total level rather than fixed.
+
+**Capture (stage 1 — the biggest blind spot, real-provider branch previously asserted nowhere):**
+
+- `tests/unit/pipelex/plugins/openai/test_openai_completions_usage_capture.py` — factory mapping (incl. cached/audio/reasoning/prediction detail fields) + `_gen_text` worker wiring reads `response.usage` into `nb_tokens_by_category`; plus the no-`usage` case pinned (stays empty → the documented silent zero-token emit).
+- `tests/unit/pipelex/plugins/openai/test_openai_responses_usage_capture.py` — same for the Responses API.
+- `tests/unit/pipelex/plugins/openai/test_openai_img_gen_worker_usage_capture.py` — `_gen_image_list` reads `ImagesResponse.usage`.
+- `tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py` — extract page-count fallback, both directions (fills when provider reports none; not overridden when provider sets usage).
+
+**Aggregate + non-LLM (stage 3, the discriminated-union wire concern):**
+
+- `tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py` — img-gen/extract/search usage JSON-round-trip through `UsageReportEvent` (discriminator intact) → `UsageAggregator.aggregate` → `CostRegistry.aggregate_costs` with exact per-type totals.
+- `tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py` — the previously-untested `_report_img_gen_job` / `_report_extract_job` dispatch branches emit through the real runner fallback to NDJSON and read back as the right concrete type.
+
+**Distributed (split-worker) summed totals + cross-child + R2:**
+
+- `tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py` — added `test_runner_usage_sums_to_expected_cost_total` (summed total via `aggregate_costs`, computed independently from the cross-worker events — no longer just landing/dedup/passthrough).
+- `tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py` — PipeParallel fan-out: usage from ≥2 distinct child workflows aggregates into one parent total over the shared `pipeline_run_id` partition.
+- `tests/unit/pipelex/reporting/test_emit_runner_fallback.py` — added `test_retried_activity_double_counts_cost_total_documenting_r2` pinning R2 at the billing level (retry → doubled tokens/cost). Flip these assertions if the idempotent-resequence fix ever lands.
+
+**E2E (real inference, gated/opt-in — the thing that was invisible):**
+
+- `tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py` — marked `inference`/`llm` (default lanes deselect it; passes when run). Real `llm_gen_text` runs on the runner queue with the context cache cleared, so real provider tokens are captured → runner fallback → aggregated to a non-zero total with a real model handle. Enabled by a backward-compatible `runner_act_llm_gen_text` param on `make_split_workers` in `helpers.py`.
+
+**`/temporal-e2e-validate` skill — manual eyeball → numeric assertion:**
+
+- `.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py` — sums input/output tokens straight from the NDJSON usage events, checks event count + `act_*` fallback + optional un-truncated CSV cross-check, and asserts model-type / non-zero. Wired into Tier 8b (`references/mode-2-tiers.md`) for the mock, cross-child, live, and img-gen/extract-live arms.
+
+**Deferred / accepted (unchanged by design):**
+
+- **R2 over-count** — accepted; pinned, not fixed (decision above).
+- **Zero-token silent emit** — capture side pinned (no-`usage` → empty); the downstream zero-token `UsageReportEvent` emission remains accepted behavior.
+- **Best-effort drops** — unchanged; the existing `_emit_best_effort` / child-flush WARNING-and-drop behavior is by design and already has gating tests.
+- **Non-LLM full Temporal activity hop** — the runner-fallback emit path is type-agnostic (proven for LLM via the split-worker tests) and non-LLM usage is covered at the `ReportingManager`/aggregator level + the skill's live arm; a dedicated split-worker bundle with fake img-gen/extract activities was judged redundant.
+
+---
+
+## Context for a cold-start session
+
+Read this first if you're picking this up cold to write the detailed plan.
+
+**Why this matters now.** The runtime-bridge work makes Pipelex pipes runnable from inside a host's own Temporal activities, and the hosted platform runs Pipelex on Temporal with **split workers** — a `router` process runs the workflows, a `runner` process runs the inference activities. Cost reporting (token usage → priced cost table / `cost_report` JSON) is a billing-adjacent surface. In split-worker mode the usage numbers are produced on one process (the runner, where inference happens) and assembled on another (the router/submitter). That cross-process assembly is the thing under audit.
+
+**The cost path has three stages — keep them distinct when planning.**
+
+1. **Capture** — inside the inference activity on the runner, read the provider response's token counts into `LLMJob.llm_tokens_usage.nb_tokens_by_category` (and the img-gen / extract / search equivalents).
+2. **Emit** — turn that into a `UsageReportEvent` and persist it to the trace partition. In split-worker mode the runner has no registered event-log context, so emission goes through the **runner fallback** (`writer_id = act_{pid}_{uuid8}`, written to a `wf_*__w_act_*.ndjson` file or the DynamoDB partition).
+3. **Aggregate + render** — the parent `WfPipeRun` runs `act_assemble_tracing` (with `assemble_usage=True`), which reads *every* event in the `pipeline_run_id` partition (router + all runners + all child workflows), passes them through `UsageAggregator.aggregate` → `PipeOutput.tokens_usages`, and the submitter prices and renders the cost table.
+
+**The one-line finding.** Stages 2 and 3 are correct-by-construction and well unit-tested **in isolation / direct mode**. What is **not** tested anywhere by assertion is: (a) stage 1 with a *real* provider response, (b) the *summed total* end-to-end in distributed mode, (c) cross-child fan-out summation, and (d) any non-LLM usage (img-gen / extract / search). Real-inference distributed cost is exercised **only manually** via this skill's Tier 8b "live arm", which just eyeballs that a table rendered.
+
+**Two things that are real bugs, not just missing tests** — see "Correctness risks" below. R2 (activity-retry over-count) is already documented and test-pinned as *current accepted behavior*; the planning session should decide whether to fix it or keep accepting it.
+
+**How to use this doc.** The "How the path works" section is the map (with file:line anchors). "What is / isn't tested" is the gap inventory. "Prioritized additions" is a menu, not a committed plan — the planning session should sequence it and decide the R2 question. Nothing here has been implemented.
+
+---
+
+## TL;DR — the four questions, answered
+
+- **Comprehensive testing of distributed real-inference cost reporting?** No. Zero automated tests run real inference through split workers and assert the cost numbers. Only manual eyeballing (skill Tier 8b live arm).
+- **Do we correctly emit usage event logs?** For LLM: the emission plumbing is correct and well-tested, but the **real-capture branch is asserted by nothing** — mock-inference diverges from real *before* the worker is built, so a passing mock test does not prove real provider tokens are read. For img-gen / extract / search: emission is **untested in any mode**.
+- **Do we correctly aggregate them?** The aggregator + cost registry are correct and have exact-total unit tests — **but only in no-temporal/direct mode**. In distributed mode no test asserts a summed total (only "events land / no per-node dup / 1:1 passthrough"). Cross-child aggregation has **zero** automated assertions.
+- **What to add?** Real-provider capture unit tests; split-worker summed-total + cross-child + non-LLM integration tests; one gated real-inference e2e; and turn the skill's manual Tier 8b into a numeric assertion. Details below.
+
+---
+
+## How the path works (code-grounded map)
+
+### Capture (runner process, inside the activity)
+
+`LLMJob.llm_tokens_usage.nb_tokens_by_category` is seeded empty in `llm_job_before_start` (`pipelex/cogt/llm/llm_job.py:36-42`), then populated from the provider response inside the plugin worker:
+
+- OpenAI completions / gateway / Portkey (gateway subclasses completions — `gateway_completions_factory.py:39`): `openai_completions_llm_worker.py:201-202` (text), `:266-267` (object); mapping in `openai_completions_factory.py:101-111` (`usage.prompt_tokens`, `usage.completion_tokens`, cached/reasoning details).
+- OpenAI responses API: `openai_responses_llm_worker.py:179-180`; factory `openai_responses_factory.py:100-106`.
+- Google: `google_llm_worker.py:236-237, 313-325`; factory `google_factory.py:144-162`.
+- Bedrock: `bedrock_llm_worker.py:85-86`.
+- Img-gen: `openai_img_gen_worker.py:134, 149-150, 164`; reported at `img_gen_worker_abstract.py:69-70 / 108-110`.
+- Extract: provider sets `nb_tokens_by_category`, else a page-count fallback fills it (`extract_worker_abstract.py:96-101`); reported at `:105-106`.
+- Search: hardcoded (`linkup_search_worker.py:90, 154`); reported at `search_worker_abstract.py:49, 72`.
+
+All of this runs synchronously inside the activity on the runner process, so the populated `tokens_usage` is what the runner-side fallback emits.
+
+**Mock diverges before capture.** `llm_gen_text` short-circuits to the mock helper at the top (`pipelex/cogt/content_generation/llm_generate.py:14-15`; object/list at `:29-30`, `:50-51`) — it never builds an `LLMJob`, never gets a worker, never reads a provider response. Mock and real **re-converge only at `report_inference_job`** (the *emit* half). The mock helper hand-builds synthetic usage `MOCK_INFERENCE_NB_TOKENS_BY_CATEGORY = {INPUT: 100, OUTPUT: 50}` (`dry_mock.py:58-61, 94-108`). **Consequence: mock passing proves emit/aggregate, not capture.** And img-gen / extract leaves *raise* `MockInferenceUnsupportedError` under mock (`img_gen_generate.py:12-13`, `extract_generate.py:11-13`) — so mock cannot cover them at all.
+
+### Emit (runner fallback)
+
+`ReportingManager.report_inference_job` → `_emit_usage_event` (`pipelex/reporting/reporting_manager.py:149-185`):
+
+```python
+if not trace_context.emit_usage_events: return # :173 — the --costs gate, before context lookup
+context = self._event_log_contexts.get(trace_context.lookup_key) # :176
+if context is not None:
+ self._emit_via_registered_context(...) # FAST PATH, writer_id="primary"
+ return # :179 — hard return
+self._emit_usage_event_runner_fallback(...) # FALLBACK, writer_id="act_*"
+```
+
+On the runner, `_event_log_contexts` is empty (`set_event_log` is only ever called on the router — `wf_pipe_router.py:114-120`), so the fallback always wins. `writer_id` is generated once per process under a double-checked lock in `ActivityEventLogCache.get_or_create` as `f"act_{os.getpid()}_{uuid4().hex[:8]}"` (`activity_event_log.py:64-76`), and the one-shot WARNING "Runner-side usage event emission engaged" comes from `activity_event_log.py:93-99`. Event type: `UsageReportEvent` (`pipelex/tracing/trace_events.py:162-167`), carrying `tokens_usage: AnyTokensUsage` + a top-level `writer_id`.
+
+### Aggregate + render (parent / submitter)
+
+`act_assemble_tracing` (`pipelex/temporal/tprl_pipe/act_assemble_tracing.py:36-50`) → `assemble_tracing` (`pipelex/pipe_run/tracing_assembly.py:50-131`):
+
+- Reads events **once**, keyed by `pipeline_run_id` only (`:95-99`). The NDJSON read globs **every** `*.ndjson` in `{traces_dir}/{pipeline_run_id}/` (`ndjson_event_log.py:133`), so router `wf_*.ndjson` and all runner `wf_*__w_act_*.ndjson` files are picked up by the same glob. Read-side dedup key: `(workflow_id, writer_id, type(event).__name__, sequence)` (`ndjson_event_log.py:150`); DynamoDB equivalent `SK = EVENT#{workflow_id}#{writer_id}#{sequence:010d}` overwrites idempotently (`dynamodb_event_log.py:99-118`). This is what makes Temporal **replay** safe.
+- `if assemble_usage: result.tokens_usages = UsageAggregator.aggregate(events)` (`:112-113`). `UsageAggregator.aggregate` is a 1:1 passthrough — `[evt.tokens_usage for evt in events if isinstance(evt, UsageReportEvent)]` (`usage_aggregator.py:12-22`), **no dedup/grouping of its own** (dedup already happened in `read_events`).
+- `TracingAssembly` rides back; `wf_pipe_run.py:91-114` copies `tokens_usages` onto `PipeOutput.tokens_usages` (`pipe_output.py:32`).
+- Render: `cost_report_renderer.py:21-82` (called from `_run_core.py:320-324`); per-model grouping + pricing in `cost_registry.py:216-279`; price comes from `tokens_usage.unit_costs` captured at inference time, **not** re-priced from a catalog (`costs_per_token.py:4-31`). Suppression: `has_reportable_usage` = `total_nb_tokens > 0 or total_cost > 0` (`cost_registry.py:41-49`), so dry-run (zero) is suppressed; a real run on a free model still reports (tokens > 0, cost 0).
+
+### Cross-child fan-out
+
+Join is **by `pipeline_run_id` on a shared partition** — there is no per-workflow join. Each child gets a distinct `workflow_id` (`temporal_pipe_router.py:69`) but the **same `pipeline_run_id`** (`TraceContext.copy_for_child` preserves it — `trace_context.py:66-84`; the fallback stamps it from the job metadata — `reporting_manager.py:270`). Only the top-level `WfPipeRun` runs `act_assemble_tracing`, and its single `read_events(pipeline_run_id)` sweeps the whole partition. Load-bearing invariant: every child's `pipeline_run_id` must equal the parent's, and (NDJSON backend) `traces_dir` must be shared across all hosts — the hosted platform must use the DynamoDB backend for multi-host (`ndjson_event_log.py:38-40`).
+
+---
+
+## What is tested vs what only "runs without crashing"
+
+### Distributed (true split-worker) topology
+
+Only `tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py` runs a real two-queue split — and it **substitutes** `act_llm_gen_text` with a helper that hand-builds `LLMTokensUsage` (INPUT=1, OUTPUT=1) (`helpers.py:195-246`). So even here there is no real worker and no provider read. It asserts:
+
+- an `act_*` usage event **lands** in a runner file (`test_runner_usage_event_lands_in_same_ndjson_dir`),
+- exactly **one** event per `(node_id, workflow_id)` — fast-path/fallback de-dup (`test_no_double_emit_in_split_worker_pool`),
+- the aggregator returns one record per event, **none dropped, order preserved** (`test_runner_usage_aggregates_to_tokens_usages` → `tokens_usages == [evt.tokens_usage for evt in usage_events]`).
+
+None of them sums tokens or asserts `total == N×{input,output}`.
+
+### Numeric correctness — exists, but only no-temporal
+
+Exact-total assertions live entirely outside Temporal, on hand-built usage:
+
+- `tests/unit/pipelex/cogt/usage/test_cost_registry.py` — per-model groupby totals, `compute_total_cost==3.5`, `aggregate_costs` exact `total_cost==0.2` / `total_nb_tokens==150`, cached/non-cached splits. Richest numeric coverage.
+- `tests/unit/pipelex/reporting/test_cost_report_renderer.py` — free model `total_cost==0.0 / 150`, console `total_cost==0.2`.
+- `tests/integration/pipelex/pipeline/test_cost_report_rendering.py` — rendered string contains `"0.2000"` (DIRECT).
+- `tests/unit/pipelex/reporting/test_reporting_event_emission.py` — single emitted event has `nb_tokens_by_category[INPUT]==100` (fast path, in-memory log).
+
+### Mock-inference (synthetic tokens, emit-path only)
+
+- `test_mock_inference_temporal.py` — real `act_llm_gen_text` with `is_mock_inference=True`, but **single task queue** (not split-worker), so it doesn't even hit the runner fallback. Asserts `tokens_usages` reportable under the sentinel `mock_inference` model; no exact total.
+- `test_mock_inference_direct.py` — DIRECT; asserts non-None / `has_reportable_usage`; no exact total.
+
+### Graph-only / serialization-only (zero cost assertions despite real cross-child fan-out)
+
+- `test_wf_graph_tracing_batch.py`, `test_wf_graph_tracing_parallel.py` — real fan-out to child workflows, **dry-run**, assert only GraphSpec structure (node statuses, BATCH/CONTAINS/DATA edges, multiple NDJSON files). Never touch `tokens_usages`.
+- `test_split_worker_object_gen.py`, `test_split_worker_extract_pages.py` — in-process single worker; assert serialization / activity-id dispatch; **no usage/cost assertion**.
+
+### Precise answers
+
+- **A. Exact aggregated total in split-worker mode (even mock)?** No. The three split-worker tests assert landing / per-node de-dup / 1:1 passthrough, never a summed total.
+- **B. Real-inference cost correctness in distributed mode?** No automated test. Only manual via the skill's Tier 8b live arm ("expect a table with the real model handle and real token counts") — eyeball, no assertion.
+- **C. Cross-child sum → single parent total?** No automated assertion, mock or real. The fan-out tracing tests are graph-only/dry-run; the skill's "A+B+C → one report" block is manual.
+- **D. Img-gen / extract / search usage in distributed mode?** Zero coverage in **any** mode. Grep for `ImgGenTokensUsage` / `ExtractTokensUsage` / `SearchTokensUsage` across `tests/` returns nothing; every usage test uses `LLMTokensUsage` only.
+- **E. Fast-path vs fallback de-dup?** Covered by `test_no_double_emit_in_split_worker_pool` (buckets by `(node_id, workflow_id)`, asserts count==1 per key) plus unit gating tests. It does **not** assert token totals and does **not** cover the retry over-count (R2 below).
+
+---
+
+## Correctness risks (real bugs, not just missing tests)
+
+- **R2 — activity-retry over-count.** A Temporal-**retried** activity (not replay) re-emits the same inference's usage at sequence N+1 instead of overwriting N, defeating the `(workflow_id, writer_id, sequence)` dedup → counted twice. Documented at `docs/.../tracing-cost-reporting.md:51`, pinned as *current accepted behavior* by `tests/unit/pipelex/reporting/test_emit_runner_fallback.py:367-388` (asserts two events at sequences `[0,1]`). The `strict_mode` / idempotent-resequence fix is deferred. With M children doing retryable inference, the over-count probability scales with fan-out × retry rate. **The planning session must decide: fix or keep accepting.**
+- **Zero-token silent emit.** If a provider returns no `usage`, the worker skips the assignment (`and (usage := response.usage)` guards everywhere, e.g. `openai_completions_llm_worker.py:201`), but the job still has a truthy `LLMTokensUsage`, so `_report_llm_job` does not hit its `if not llm_tokens_usage` guard (`reporting_manager.py:116`) — a `UsageReportEvent` is emitted **with zero tokens**. Silent under-count, no flag.
+- **Best-effort drops.** Emit (`_emit_best_effort`, `reporting_manager.py:208-221`) and child flush (`wf_pipe_router.py:175-177`) swallow infra failures with a WARNING; an unwritable dir / DDB reject / unshared NDJSON `traces_dir` across hosts silently drops a slice. The run still succeeds; the report is quietly short. A whole-read failure is at least visible (`usage_assembly_error` → `tokens_usages=None` → no report), but per-line/per-file drops are not.
+
+The aggregator and renderer themselves are lossless — all risk lives in the capture/emit/persist/read layer.
+
+---
+
+## Prioritized additions (the original menu — now implemented; see "Resolution (as-built)" above for the landed files)
+
+### Unit — cheapest, closes the biggest blind spot (real capture)
+
+- **Real-provider capture tests.** Substitute the OpenAI/gateway client with a canned response carrying `usage.input_tokens=…` (and the cached/reasoning detail fields); assert `nb_tokens_by_category` is populated correctly. One each for completions, responses API, **img-gen, extract**. Deterministic, no spend — this is the single highest-value gap, since the real-capture branch is asserted nowhere today.
+- **No-`usage` provider case.** Decide intended behavior (today: silent zero-token emit) and pin it.
+- **Non-LLM through the aggregator.** A test routing `ImgGenTokensUsage` / `ExtractTokensUsage` / `SearchTokensUsage` through `UsageReportEvent → aggregate_costs` and asserting the per-type total (currently never exercised).
+
+### Integration — split-worker, fixture/mock (no spend)
+
+- **Summed-total assertion.** Extend `test_split_worker_usage.py`: inject known synthetic per-call tokens, assert `aggregate_costs(tokens_usages).total_nb_tokens == expected sum` (not just landing/dedup/passthrough).
+- **Cross-child sum.** PipeBatch/PipeParallel split-worker test where each child emits known usage; assert the parent's single total = sum across children. Highest-risk uncovered path (also the natural home for an R2 regression).
+- **Non-LLM cross-worker.** Split-worker img-gen + extract usage crossing the runner fallback via a fixture worker returning canned `ImgGenTokensUsage` / `ExtractTokensUsage`; assert aggregate + render. Closes D in distributed mode.
+- **R2 at the total level.** Force an activity retry and assert the cost total double-counts — make the known over-count regression-visible (and, if fixed, flip the assertion).
+
+### E2E — real inference, gated/opt-in lane
+
+- One marked real-inference split-worker test: assert `tokens_usages` non-empty, every entry non-zero, model handles match the real models, usage-record count == number of inference calls, total > 0. Even loose bounds catch the entire "real provider usage not captured/emitted/aggregated in distributed mode" class — the exact thing that's invisible today (and that I could not verify by eyeball this session because the live cost tables were column-truncated).
+
+### temporal-e2e-validate skill — make the manual check assert
+
+- Strengthen Tier 8b's live arm: parse the NDJSON usage events and assert **summed input/output across all `__w_act_*` files == rendered table total**, and usage-event count == inference-call count. Dump costs to CSV (un-truncated) so real non-zero numbers are actually visible — the current terminal table truncates to `0 … …`, making the eyeball check nearly worthless.
+- Add an img-gen / extract live cross-worker **cost** assertion (Tiers 4/5/10c run those live but never check the numbers).
+
+---
+
+## Key files
+
+- Capture: `pipelex/cogt/content_generation/llm_generate.py`, `dry_mock.py`, `img_gen_generate.py`, `extract_generate.py`; `pipelex/cogt/llm/llm_job.py`, `llm_worker_abstract.py`; `pipelex/plugins/openai/openai_completions_llm_worker.py` (+ `openai_completions_factory.py`, `openai_responses_factory.py`), `pipelex/plugins/openai/openai_img_gen_worker.py`; `pipelex/cogt/extract/extract_worker_abstract.py`.
+- Emit: `pipelex/reporting/reporting_manager.py`, `pipelex/tracing/activity_event_log.py`, `pipelex/tracing/trace_events.py`.
+- Aggregate + render: `pipelex/temporal/tprl_pipe/act_assemble_tracing.py`, `pipelex/pipe_run/tracing_assembly.py`, `pipelex/tracing/usage_aggregator.py`, `pipelex/tracing/ndjson_event_log.py`, `pipelex/tracing/dynamodb_event_log.py`, `pipelex/cogt/usage/cost_registry.py`, `pipelex/cogt/usage/costs_per_token.py`, `pipelex/reporting/cost_report_renderer.py`, `pipelex/core/pipes/pipe_output.py`.
+- Tests today: `tests/integration/pipelex/temporal/tracing/{test_split_worker_usage.py,test_mock_inference_temporal.py,test_split_worker_object_gen.py,test_split_worker_extract_pages.py,helpers.py}`; `tests/unit/pipelex/reporting/{test_emit_runner_fallback.py,test_emit_usage_event_gating.py,test_reporting_event_emission.py,test_cost_report_renderer.py}`; `tests/unit/pipelex/cogt/usage/test_cost_registry.py`; `tests/unit/pipelex/tracing/{test_usage_aggregator.py,test_writer_id_schema.py,test_trace_events.py}`; `tests/unit/pipelex/pipe_run/test_tracing_assembly.py`.
+
+## Related
+
+- [`cost-reporting-validation-prompts.md`](cost-reporting-validation-prompts.md) — operator cheat-sheet of `/temporal-e2e-validate` prompts that drive the Tier 8b arms this audit motivated.
+- `../observer-and-telemetry/` — sibling tracing/telemetry follow-ups.
+- `docs/.../tracing-cost-reporting.md` (R2 documentation) and the `wip/registry/cost-report-deferred-decisions.md` deferred items (e.g. `tracing_config.is_enabled=false` + `--costs` → silent no-report).
+- This audit was produced cross-checking three independent code sweeps (emission, aggregation, test inventory); the file:line anchors above are the load-bearing evidence — re-verify any that look stale before building on them.
diff --git a/wip/runtime-bridge/execution-mode-property-helpers-unused.md b/wip/runtime-bridge/execution-mode-property-helpers-unused.md
new file mode 100644
index 000000000..dfac43258
--- /dev/null
+++ b/wip/runtime-bridge/execution-mode-property-helpers-unused.md
@@ -0,0 +1,24 @@
+# `PipelexExecutionMode` property helpers — tested but not yet consumed
+
+**Status:** deferred follow-up (not a bug). Intentionally kept as a tested, intention-revealing surface; wiring them into `bridge.py` (or removing them) is a follow-up. Migrated here from the PR #969 reviewer's guide (`TODOS.md`) when that file was retired at merge — it was the only live item in that guide not already recorded under `wip/`.
+
+## What
+
+`PipelexExecutionMode` (`pipelex/runtime_bridge/execution_mode.py`) exposes three `@property` helpers:
+
+- `requires_pipelex_temporal` — True for `TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`.
+- `requires_mistral_workflows_extra` — True for `MISTRAL_NATIVE`.
+- `is_fire_and_forget` — True for `TEMPORAL_FIRE_AND_FORGET`.
+
+They are covered by `tests/unit/pipelex/runtime_bridge/test_execution_mode.py`, but **`bridge.py` does not consume them** — it inlines the equivalent `match` / identity checks at each dispatch and validation site instead.
+
+## Why it's on file, not fixed
+
+The helpers were written as the intention-revealing form of the mode predicates, ahead of `bridge.py` being refactored to call them. Keeping them tested-but-unused is a deliberate, low-risk surface (the `match` bodies are exhaustive, so the linter guards them if a mode is added), not a silent gap. There is no behavioural bug: the inlined checks in `bridge.py` and the helpers agree.
+
+## Follow-up (pick one when next touching the bridge)
+
+- **Wire them in** — replace the inlined predicates in `bridge.py` (`_validate_input`'s fire-and-forget guard, the per-mode extra-dependency guards) with the helpers, so the predicate lives in one place. Lowest-churn, removes the duplication the reviewer flagged.
+- **Or remove them** — if `bridge.py`'s `match execution_mode:` dispatch stays the single source of truth and the helpers earn no second caller, delete them with their tests.
+
+Not urgent either way; no correctness impact.
diff --git a/wip/runtime-bridge/graph-context-temporal-contract.md b/wip/runtime-bridge/graph-context-temporal-contract.md
new file mode 100644
index 000000000..899bbf041
--- /dev/null
+++ b/wip/runtime-bridge/graph-context-temporal-contract.md
@@ -0,0 +1,75 @@
+# `graph_context` threaded into Temporal modes despite a DIRECT-only contract
+
+**Status:** ✅ **RESOLVED — Option A applied.** (was: confirmed contract/docstring mismatch, deferred pending decision)
+**Source:** PR #959 review — greptile-apps (P2, `bridge.py:111-128`, "Temporal modes inherit tracing").
+**Severity:** real but **latent** — only mattered when a host passed a non-None `graph_context` *and* selected a Temporal mode.
+
+## Resolution (Option A)
+
+`run_pipe_via_bridge` now computes `is_direct = input_payload.execution_mode is PipelexExecutionMode.DIRECT` and passes `graph_context=graph_context if is_direct else None` to `build_pipe_job_from_input`, so a host `graph_context` is honored for DIRECT and **nulled for the Temporal modes** — honoring the documented contract and removing the cross-contamination foot-gun (`WfPipeRouter`'s `graph_context is not None` guard makes the None a clean no-op). The now-inaccurate "would be ignored anyway" docstring clause was rewritten to state the nulling plainly. Regression test: `tests/unit/pipelex/runtime_bridge/test_graph_context_contract.py` (DIRECT forwards the host context; TEMPORAL_BLOCKING nulls it). `make agent-check` + `make agent-test` green.
+
+The original triage below is retained as the record of why Option A over B/C.
+
+---
+
+## The finding
+
+> The docstring says `graph_context` is only honored for `DIRECT`, but this code builds the `PipeJob` with that context before dispatching every mode. For `TEMPORAL_BLOCKING` and `TEMPORAL_FIRE_AND_FORGET`, `WfPipeRouter` consumes the non-null context and opens a workflow tracer under the caller's graph id and parent node. A host activity that passes its own graph context can therefore merge or duplicate Pipelex Temporal trace events into the host activity graph, even though the bridge contract says Temporal modes ignore that context.
+
+## Verified behavior (code trace)
+
+**The documented contract (`pipelex/runtime_bridge/bridge.py:100-103`):**
+
+> The optional `graph_context` is only honored for `DIRECT` execution mode — TEMPORAL modes already have their own event-log infrastructure via `pipeline_run_setup` and a passed-in context **would be ignored anyway**.
+
+**What the code does** — `graph_context` is baked into the `PipeJob` *before* the mode `match`, so it flows into every branch:
+
+- `pipelex/runtime_bridge/bridge.py:112-116` — `build_pipe_job_from_input(..., graph_context=graph_context)` is called unconditionally, then `:118` `match input_payload.execution_mode:` dispatches DIRECT / TEMPORAL_BLOCKING / TEMPORAL_FIRE_AND_FORGET / MISTRAL_NATIVE.
+- `pipelex/runtime_bridge/bridge.py:161-165` — `graph_context` is stored into `JobMetadata`.
+
+**`WfPipeRouter` *does* consume it** (so "would be ignored anyway" is false):
+
+- `pipelex/temporal/tprl_pipe/wf_pipe_router.py:49` — `graph_context = workflow_arg.job_metadata.graph_context`
+- `:80` — `if tracing_config.is_enabled and graph_context is not None:`
+- `:87-94` — `open_tracer(graph_id=graph_context.graph_id, data_inclusion=graph_context.data_inclusion, ...)`
+- `:95-100` — preserves `graph_context.parent_node_id` so "CONTAINS edges link back to the parent workflow's controller node."
+
+So if a host passes its own `graph_context` with a Temporal mode, Pipelex's Temporal workflow opens its tracer under the **host's** `graph_id` and links edges to the **host's** `parent_node_id` — i.e. Pipelex Temporal trace events get merged into the host's graph. That is exactly the cross-contamination the docstring's contract was trying to forbid.
+
+Note the asymmetry that makes this a *contract* bug rather than a feature: for DIRECT the host's graph_context is intended (per-step events flow into the host's tracer — `bridge.py:97-99, 142-146`); for Temporal the bridge bypasses Pipelex's own `pipeline_run_setup` graph_context creation and instead forwards the host's, which `WfPipeRouter` then treats as if it were Pipelex's own.
+
+## The fork
+
+**Option A — Honor the contract (recommended).** Pass `graph_context` only for DIRECT; pass `None` for the Temporal modes. E.g. compute it per-mode:
+
+```python
+is_direct = input_payload.execution_mode is PipelexExecutionMode.DIRECT
+pipe_job = build_pipe_job_from_input(
+ input_payload=input_payload,
+ library_crate=library_crate,
+ graph_context=graph_context if is_direct else None,
+)
+```
+
+- Pros: makes the code match the documented intent; stops a host's graph_context from contaminating Pipelex's Temporal trace graph; tiny, low-risk; the `None` guard at `wf_pipe_router.py:80` already makes a None context a clean no-op. Aligns with the docstring author's clear intent.
+- Cons: if someone *wanted* host→Pipelex Temporal trace linking, this removes the only path to it. (No evidence anyone does, and `pipeline_run_setup` is the documented Temporal tracing path.)
+- Tidy-up if chosen: drop the now-inaccurate "would be ignored anyway" clause and state plainly that Temporal modes deliberately do not forward a host `graph_context`.
+
+**Option B — Fix the docstring, keep threading it.** Treat the current behavior as intended and document that Temporal modes consume `graph_context` (parent-linking via CONTAINS edges).
+- Pros: no behavior change; preserves any future host→Pipelex Temporal trace stitching.
+- Cons: keeps the cross-contamination foot-gun (a host that opens its own tracer and picks a Temporal mode silently merges Pipelex's Temporal events into its graph); the "their own event-log infrastructure via `pipeline_run_setup`" rationale is then misleading, since the bridge's Temporal path doesn't go through `pipeline_run_setup` for graph_context.
+
+**Option C — Defer.** This doc is the record; revisit when Temporal ships and the host-tracing story for Temporal modes is actually designed.
+
+## Recommendation
+
+Option A. It is the smaller change, removes a latent foot-gun, and makes the code obey its own stated contract. Revisit only if a concrete need for host→Pipelex Temporal trace stitching appears — at which point it should be designed explicitly (own graph id / explicit opt-in), not inherited by accident.
+
+## Test ideas (when fixed)
+
+- Bridge unit: call `run_pipe_via_bridge(input_payload=<TEMPORAL_*>, graph_context=<non-None>)` and assert the resulting `PipeJob.job_metadata.graph_context is None`; for `DIRECT` assert it is preserved. (Can test `build_pipe_job_from_input` selection logic without a live Temporal server.)
+
+## Related
+
+- `direct-mode-nested-router-leak.md` — sibling bridge issue (#1); both are about the bridge's contract for DIRECT vs Temporal dispatch.
+- `wf_pipe_router.py:80-103` is shared Temporal machinery — changing *it* would affect normal Pipelex Temporal runs; Option A keeps the change confined to the bridge boundary, which is correct.
diff --git a/wip/runtime-bridge/library-manager-pipe-source-map-concurrency.md b/wip/runtime-bridge/library-manager-pipe-source-map-concurrency.md
new file mode 100644
index 000000000..9805bd603
--- /dev/null
+++ b/wip/runtime-bridge/library-manager-pipe-source-map-concurrency.md
@@ -0,0 +1,40 @@
+# LibraryManager `_pipe_source_map` is not per-call scoped (deferred)
+
+**Status:** deferred. Pre-existing `LibraryManager` behaviour, surfaced more by the bridge's per-call scoped-library path. Observability-only impact, Temporal not in production. Not fixed in PR #969.
+
+## What
+
+`LibraryManager._pipe_source_map` (`pipelex/libraries/library_manager.py:98`) is a **manager-level** dict keyed by `pipe_ref` (`domain.pipe_code`):
+
+```python
+self._pipe_source_map: dict[str, Path] = {} # pipe_ref (domain.pipe_code) -> source .mthds file
+```
+
+Its siblings — `_blueprints`, `_crate_cache`, `_loaded_fingerprints` — are all keyed by `library_id` (one entry per open library). `_pipe_source_map` is the odd one out: it is shared across every open library.
+
+`load_from_crate` writes `self._pipe_source_map[pipe_ref] = Path(source)`, and the keyed `teardown(library_id=...)` iterates the torn-down library's pipes and `pop()`s those `pipe_ref`s out of the shared map (`library_manager.py:122`).
+
+## Why it can interfere across concurrent bridge calls
+
+`run_pipe_via_bridge` opens a fresh per-call scoped library (`bridge._scoped_library_for_crate`) with a unique `library_id`, loads the caller's crate into it, and tears it down on exit. Two **concurrent** bridge calls whose crates define the **same** `pipe_ref` will:
+
+- overwrite each other's source path (last writer wins), and
+- one call's teardown can `pop()` the `pipe_ref` out from under the other still-running call, so the survivor's `get_pipe_source()` returns `None` (or, after an overwrite, the other tenant's path).
+
+`get_pipe_source` feeds source attribution into the graph / observability layer only — it does **not** affect pipe resolution or run correctness. So the blast radius is a wrong-or-missing source path in a trace graph, not a wrong result.
+
+There is also a broader, lower-confidence concern: `LibraryManager` mutates these plain dicts (`open_library` / `teardown` / `load_from_crate`) with no lock, while the bridge advertises a concurrent-activity execution model. The no-arg `teardown()` branch reassigns the dicts wholesale; concurrent mutation during the keyed teardown's iterate-and-`pop` could in principle raise `dictionary changed size during iteration`. Each bridge call uses a unique `library_id` (isolating `_libraries`/`_blueprints`/etc.), which keeps the realistic blast radius confined to `_pipe_source_map`.
+
+## Why defer (not fix here)
+
+- **Pre-existing.** This is `LibraryManager` behaviour, not introduced by this PR. The pre-existing Temporal `wf_pipe_router` scoped-library path uses the same map. The bridge only makes the concurrent-scoped-library pattern more prominent.
+- **Observability-only.** Worst case is a wrong/missing source path in a trace graph. No run-correctness or data-loss impact.
+- **Not in production.** The concurrent-multi-tenant path that triggers it is the Temporal/Mistral worker model, which is not in production.
+- **Right fix touches `library_manager.py` beyond this PR's surface** and wants a deliberate design pass, not a reflexive patch on the way to landing the bridge.
+
+## Fix shape when picked up
+
+- Scope the source map per `library_id`: `dict[str, dict[str, Path]]` keyed by `library_id` (matching the sibling caches) so the keyed teardown pops only the calling library's entries and concurrent crates cannot clobber each other. Alternatively, store the source path on the `Library` object itself.
+- If `LibraryManager` is to formally support concurrent callers, add a lock around the structural mutations in `open_library` / `teardown` / `load_from_crate` (or document that mutation is not thread-safe and require callers to serialise). At minimum, guard the `_pipe_source_map` iterate-and-`pop` in the keyed teardown.
+
+Found by the `/review` adversarial (red-team) pass on PR #969.
diff --git a/wip/runtime-bridge/runtime-bridge.html b/wip/runtime-bridge/runtime-bridge.html
new file mode 100644
index 000000000..455159f37
--- /dev/null
+++ b/wip/runtime-bridge/runtime-bridge.html
@@ -0,0 +1,287 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8" />
+<meta name="viewport" content="width=device-width, initial-scale=1" />
+<title>Pipelex runtime bridge — extracting runtime-agnostic tools from Temporal</title>
+<style>
+ :root {
+ --bg: #0e1116;
+ --panel: #161b22;
+ --panel-2: #1c2330;
+ --ink: #e6edf3;
+ --ink-dim: #9aa6b2;
+ --accent: #79c0ff;
+ --accent-2: #d2a8ff;
+ --good: #7ee787;
+ --warn: #f5b352;
+ --bad: #ff7b72;
+ --line: #30363d;
+ --kbd: #21262d;
+ }
+ html { color-scheme: dark; }
+ body {
+ margin: 0; padding: 0;
+ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif;
+ background: var(--bg); color: var(--ink); line-height: 1.55;
+ }
+ .wrap { max-width: 1040px; margin: 0 auto; padding: 32px 28px 96px; }
+ header.title { border-bottom: 1px solid var(--line); margin-bottom: 22px; padding-bottom: 16px; }
+ header.title h1 { margin: 0 0 6px; font-size: 26px; line-height: 1.25; }
+ header.title p.lede { margin: 0; color: var(--ink-dim); font-size: 15px; }
+ .meta { display: flex; flex-wrap: wrap; gap: 6px 10px; margin-top: 14px; font-size: 13px; color: var(--ink-dim); }
+ .pill { display: inline-block; background: var(--panel); border: 1px solid var(--line); border-radius: 999px; padding: 2px 10px; color: var(--ink); }
+ .pill.added { color: var(--good); border-color: rgba(126,231,135,.4); }
+ .pill.changed { color: var(--accent); border-color: rgba(121,192,255,.4); }
+ .pill.moved { color: var(--accent-2); border-color: rgba(210,168,255,.4); }
+ .pill.future { color: var(--warn); border-color: rgba(245,179,82,.4); }
+ h2 { margin: 38px 0 12px; font-size: 21px; border-bottom: 1px solid var(--line); padding-bottom: 6px; }
+ h3 { margin: 24px 0 8px; font-size: 17px; color: var(--accent); }
+ h4 { margin: 16px 0 6px; font-size: 12px; color: var(--ink-dim); text-transform: uppercase; letter-spacing: .04em; }
+ p { margin: 8px 0 12px; }
+ ul, ol { margin: 8px 0 14px; padding-left: 22px; }
+ li { margin: 5px 0; }
+ a { color: var(--accent); }
+ code, pre, kbd { font-family: ui-monospace, "SF Mono", Menlo, Consolas, monospace; font-size: 13px; }
+ code { background: var(--kbd); border: 1px solid var(--line); padding: 1px 5px; border-radius: 4px; }
+ pre { background: var(--panel); border: 1px solid var(--line); border-radius: 8px; padding: 14px 16px; overflow: auto; margin: 10px 0 16px; line-height: 1.45; }
+ pre code { background: none; border: none; padding: 0; font-size: 12.5px; }
+ .tldr { background: linear-gradient(180deg, var(--panel-2), var(--panel)); border: 1px solid var(--line); border-radius: 12px; padding: 18px 22px; margin: 18px 0 8px; }
+ .tldr h2 { border: none; margin: 0 0 8px; padding: 0; font-size: 16px; letter-spacing: .04em; text-transform: uppercase; color: var(--accent-2); }
+ .tldr p:last-child { margin-bottom: 0; }
+ .tldr ul { margin: 10px 0 4px; }
+ .grid-2 { display: grid; grid-template-columns: 1fr 1fr; gap: 14px; }
+ @media (max-width: 820px) { .grid-2 { grid-template-columns: 1fr; } }
+ .card { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 14px 16px; }
+ .card h4 { margin-top: 0; color: var(--accent); }
+ .card p { margin: 6px 0 0; }
+ table { width: 100%; border-collapse: collapse; margin: 12px 0 18px; font-size: 13px; }
+ th, td { text-align: left; padding: 8px 10px; border-bottom: 1px solid var(--line); vertical-align: top; }
+ th { color: var(--ink-dim); font-weight: 600; font-size: 12px; text-transform: uppercase; letter-spacing: .03em; }
+ td code { white-space: nowrap; }
+ .figure { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 18px; margin: 16px 0; }
+ .figure .cap { color: var(--ink-dim); font-size: 12.5px; margin-top: 10px; text-align: center; }
+ .figure .mermaid { display: flex; justify-content: center; }
+ pre.diff { line-height: 1.5; }
+ pre.diff .add { color: var(--good); display: block; background: rgba(126,231,135,.08); }
+ pre.diff .del { color: var(--bad); display: block; background: rgba(255,123,114,.08); }
+ pre.diff .ctx { color: var(--ink-dim); display: block; }
+ pre.diff .hd { color: var(--accent-2); display: block; }
+ pre.diff .cmt { color: #6e7681; display: block; }
+ .note { border-left: 3px solid var(--accent); background: rgba(121,192,255,.06); padding: 10px 14px; border-radius: 0 8px 8px 0; margin: 14px 0; }
+ .note.good { border-left-color: var(--good); background: rgba(126,231,135,.06); }
+ .note.warn { border-left-color: var(--warn); background: rgba(245,179,82,.06); }
+ .note p:first-child { margin-top: 0; } .note p:last-child { margin-bottom: 0; }
+ .tag { font-size: 11px; padding: 1px 7px; border-radius: 999px; border: 1px solid var(--line); color: var(--ink-dim); }
+ .tag.ok { color: var(--good); border-color: rgba(126,231,135,.4); }
+ .tag.fix { color: var(--accent); border-color: rgba(121,192,255,.4); }
+ .tag.defer { color: var(--warn); border-color: rgba(245,179,82,.4); }
+ .tag.now { color: var(--good); border-color: rgba(126,231,135,.4); }
+ .tag.later { color: var(--warn); border-color: rgba(245,179,82,.4); }
+ details { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; margin: 10px 0; padding: 0 16px; }
+ details[open] { padding-bottom: 6px; }
+ summary { cursor: pointer; padding: 12px 0; font-weight: 600; color: var(--ink); list-style: none; display: flex; align-items: center; gap: 8px; }
+ summary::-webkit-details-marker { display: none; }
+ summary::before { content: "▸"; color: var(--accent); transition: transform .15s ease; }
+ details[open] summary::before { transform: rotate(90deg); }
+ .tabs { margin: 14px 0; }
+ .tab-bar { display: flex; gap: 4px; flex-wrap: wrap; border-bottom: 1px solid var(--line); }
+ .tab-btn { background: transparent; border: 1px solid transparent; border-bottom: none; color: var(--ink-dim); padding: 8px 14px; cursor: pointer; font-size: 13px; border-radius: 8px 8px 0 0; font-family: inherit; }
+ .tab-btn.active { color: var(--ink); background: var(--panel); border-color: var(--line); }
+ .tab-panel { display: none; border: 1px solid var(--line); border-top: none; border-radius: 0 0 8px 8px; padding: 4px 16px 8px; background: var(--panel); }
+ .tab-panel.active { display: block; }
+ .tab-panel pre { background: var(--bg); }
+ .small { font-size: 12.5px; color: var(--ink-dim); }
+ hr { border: none; border-top: 1px solid var(--line); margin: 28px 0; }
+</style>
+<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>
+<script>
+ mermaid.initialize({
+ startOnLoad: true,
+ theme: "dark",
+ securityLevel: "loose",
+ themeVariables: {
+ background: "#161b22", primaryColor: "#1c2330", primaryTextColor: "#e8eaf0",
+ primaryBorderColor: "#2c3142", lineColor: "#6ea8fe", fontFamily: "ui-monospace, monospace",
+ fontSize: "13px"
+ },
+ flowchart: { curve: "basis", htmlLabels: true }
+ });
+ function showTab(group, idx, btn) {
+ document.querySelectorAll('[data-tabgroup="' + group + '"] .tab-btn').forEach((b, i) => b.classList.toggle('active', i === idx));
+ document.querySelectorAll('[data-tabgroup="' + group + '"] .tab-panel').forEach((p, i) => p.classList.toggle('active', i === idx));
+ }
+</script>
+</head>
+<body>
+<div class="wrap">
+
+<header class="title">
+ <h1>Runtime bridge — extracting runtime-agnostic tools out of the Temporal subpackage</h1>
+ <p class="lede">A lot of what we built for Temporal isn't actually Temporal-specific. This PR pulls that logic into a neutral home so it can be shared across runtimes — used by Temporal today, by the Mistral Workflows plugin when it ships.</p>
+ <div class="meta">
+ <span class="pill">PR #969</span>
+ <span class="pill">branch <code>feature/Runtime-bridge-extraction</code></span>
+ <span class="pill">base <code>dev</code></span>
+ <span class="pill added">new package <code>pipelex/runtime_bridge/</code></span>
+ </div>
+</header>
+
+<div class="tldr">
+ <h2>TL;DR</h2>
+ <p>We're going to support more than one execution runtime. Today Pipelex runs pipes durably on <strong>Temporal</strong>; next we'll run them on <strong>Mistral Workflows</strong>. While building the Temporal integration, several pieces turned out <strong>not to be Temporal-specific at all</strong>: delivering results (storage + webhook), flushing trace events, hydrating working memory, rehydrating a library crate, classifying a pipe as controller-vs-leaf.</p>
+ <p>This PR <strong>lifts that runtime-agnostic logic out of the <code>pipelex.temporal</code> subpackage into a new neutral home — <code>pipelex/runtime_bridge/primitives/</code></strong> — and turns the Temporal activities into thin wrappers over it. Nothing about behaviour changes; only the <em>home</em> of the logic does.</p>
+ <ul>
+ <li><strong>Used by Temporal today</strong> <span class="tag now">now</span> — the <code>act_*.py</code> activities are now thin <code>@activity.defn</code> wrappers that delegate to the shared primitives; even core <code>delivery_executor</code> consumes one. One copy, instead of logic that would get re-implemented per runtime.</li>
+ <li><strong>Used by Mistral Workflows next</strong> <span class="tag later">later</span> — the same primitives are what the <code>pipelex-mistralai-workflows</code> plugin will consume when we ship it (currently preview).</li>
+ <li><strong>Also lands a forward-looking dispatch surface</strong> — <code>run_pipe_via_bridge()</code> + a <code>PipelexExecutionMode</code> enum + JSON-safe boundary types — designed and tested now as the door the Mistral plugin will enter through. <em>It has no in-tree caller yet</em>; Temporal keeps its existing entry points and just consumes the lifted primitives.</li>
+ </ul>
+ <p class="small" style="margin-top:10px">The diff looks large, but the code core is small and concentrated in <code>pipelex/runtime_bridge/</code>. Most of the line count is a vendored skill + triage notes that are <strong>not</strong> review material (see <em>What's not part of this work</em>).</p>
+</div>
+
+<div class="note">
+ <p style="margin:0 0 6px"><strong>A note on two words used throughout this doc.</strong> The relocation came in two flavours — the distinction matters when reading the diff:</p>
+ <ul style="margin:0">
+ <li><span class="pill moved">moved</span> — the module already stood on its own; it just changed directory, byte-for-byte. Git records it as a <em>rename</em> (high similarity), so the diff shows no real content change — only a new path. Example: <code>hydration.py</code>, <code>pipe_run_arg.py</code>, <code>submitter_hydration.py</code>.</li>
+ <li><span class="pill added">lifted</span> — the logic was <em>welded inside a Temporal activity's function body</em>. It was extracted into its own standalone, runtime-agnostic function, leaving a thin <code>@activity.defn</code> wrapper behind that just calls it. This is an <em>extraction / refactor</em>, so the diff shows code leaving one file and a new file appearing — not a clean rename. Example: <code>delivery.py</code>, <code>trace_flush.py</code>.</li>
+ </ul>
+ <p class="small" style="margin:8px 0 0">In short: <strong>moved = same code, new home</strong>; <strong>lifted = code pulled out of a Temporal wrapper into a reusable home, wrapper kept thin</strong>. Both are behaviour-neutral.</p>
+</div>
+
+<h2>The move at a glance</h2>
+
+<div class="figure">
+ <div class="mermaid">
+flowchart LR
+ subgraph temporal["pipelex.temporal — Temporal-specific glue"]
+ direction TB
+ ACT["act_*.py<br/><i>thin @activity.defn wrappers</i>"]
+ WF["wf_pipe_* · temporal_pipe_*"]
+ end
+ subgraph core["pipelex core"]
+ DEX["pipe_run.delivery_executor"]
+ end
+ subgraph bridge["pipelex.runtime_bridge — runtime-agnostic"]
+ direction TB
+ PRIM["primitives/<br/>delivery · trace_flush · hydration<br/>submitter_hydration · pipe_run_arg · pipe_classification"]
+ DISP["dispatch surface (forward-looking)<br/>run_pipe_via_bridge · execution_mode<br/>boundary types · bootstrap"]
+ end
+ MISTRAL["pipelex-mistralai-workflows<br/><i>ships later (preview)</i>"]
+ ACT -->|consumes today| PRIM
+ WF -->|consumes today| PRIM
+ DEX -->|consumes today| PRIM
+ MISTRAL -.->|will consume| PRIM
+ MISTRAL -.->|will enter through| DISP
+ classDef p fill:#16241a,stroke:#5fd38f,color:#cfeede;
+ classDef f fill:#2a2418,stroke:#e0b463,color:#f1e2c4;
+ class PRIM p;
+ class DISP f;
+ </div>
+ <div class="cap">Solid arrows = consumed in-tree today. Dotted = the Mistral plugin will consume it once shipped. The <span style="color:var(--good)">primitives</span> are the landed, in-use payload of this PR; the <span style="color:var(--warn)">dispatch surface</span> is built ahead of its consumer.</div>
+</div>
+
+<h2>What got generalized (the lift)</h2>
+
+<p>Each of these was a body that lived inside a Temporal activity or module. None of it needs Temporal. It moved to <code>runtime_bridge/primitives/</code> with <strong>no behaviour change</strong>, and the Temporal side now imports it back.</p>
+
+<table>
+ <thead><tr><th>Primitive</th><th>What it does</th><th>Consumed today by</th><th></th></tr></thead>
+ <tbody>
+ <tr><td><code>delivery.py</code></td><td>Execute the full delivery (storage + webhook) of a pipe result</td><td><code>act_deliver.py</code></td><td><span class="pill added">lifted</span></td></tr>
+ <tr><td><code>trace_flush.py</code></td><td>Flush a workflow's buffered trace events to the configured backend</td><td><code>act_flush_trace_events.py</code></td><td><span class="pill added">lifted</span></td></tr>
+ <tr><td><code>hydration.py</code></td><td>Rebuild typed content / working memory from a dump</td><td><code>wf_pipe_router.py</code> <strong>and core <code>delivery_executor.py</code></strong></td><td><span class="pill moved">moved</span></td></tr>
+ <tr><td><code>submitter_hydration.py</code></td><td>Rehydrate a <code>PipeOutput</code> with its <code>LibraryCrate</code></td><td><code>temporal_pipe_run.py</code> · <code>temporal_pipe_router.py</code></td><td><span class="pill moved">moved</span></td></tr>
+ <tr><td><code>pipe_run_arg.py</code></td><td>The <code>PipeRunArg</code> transit shape</td><td><code>temporal_pipe_run.py</code> · <code>wf_pipe_run.py</code></td><td><span class="pill moved">moved</span></td></tr>
+ <tr><td><code>pipe_classification.py</code></td><td>Controller-vs-leaf decision for per-step decomposition</td><td><span class="small">no in-tree caller yet — for Mistral</span></td><td><span class="pill future">new</span></td></tr>
+ </tbody>
+</table>
+
+<p class="small">That <code>hydration.py</code> is consumed by core <code>delivery_executor</code> — not just Temporal — is the tell that this logic was never Temporal-specific to begin with.</p>
+
+<h3>What a Temporal activity looks like now</h3>
+
+<p>The activity keeps only its Temporal-specific shell — the <code>@activity.defn</code> decorator and arg decode — and delegates the body to the shared primitive:</p>
+
+<pre class="diff"><span class="cmt"># act_deliver.py — Temporal shell only; the body now lives in the runtime-agnostic primitive.</span>
+<span class="ctx">from pipelex.runtime_bridge.primitives.delivery import execute_delivery</span>
+
+<span class="ctx">@activity.defn(name="act_deliver")</span>
+<span class="ctx">@convert_pipelex_errors</span>
+<span class="ctx">async def act_deliver(arg: DeliveryActivityArg) -> None:</span>
+<span class="ctx"> await execute_delivery(...) # the lifted, host-agnostic body</span></pre>
+
+<p class="small"><strong>Two robustness fixes also rode along on live code:</strong> tracing now degrades gracefully when the event-log <em>backend</em> can't even be constructed (new <code>EventLogSetupError</code>), not only when a read fails; and <code>hub.py</code> gains <code>scoped_pipe_router</code>, a contextvar router override that restores the prior value on exit.</p>
+
+<h2>The forward-looking dispatch surface</h2>
+
+<div class="note warn">
+ <p><strong>Designed, not yet wired in.</strong> <code>run_pipe_via_bridge()</code> + a <code>PipelexExecutionMode</code> enum + JSON-safe (<code>extra="forbid"</code>) boundary types are the door the Mistral plugin will enter through. They're built, unit-tested, and hardened — but have <strong>no in-tree caller yet</strong>. Temporal keeps its own entry points and just consumes the lifted primitives. Skim to know it's there; no need to review as live code.</p>
+</div>
+
+<details>
+ <summary>The four execution modes (reference)</summary>
+ <table>
+ <thead><tr><th>Mode</th><th>What it does</th><th>Needs</th></tr></thead>
+ <tbody>
+ <tr><td><code>DIRECT</code> <span class="small">(default)</span></td><td>In-process, no Temporal. Forces local execution even inside a Temporal-enabled worker.</td><td>nothing</td></tr>
+ <tr><td><code>TEMPORAL_BLOCKING</code></td><td>Dispatch as a Pipelex Temporal workflow and await it.</td><td><code>pipelex[temporal]</code></td></tr>
+ <tr><td><code>TEMPORAL_FIRE_AND_FORGET</code></td><td>Dispatch and return the <code>workflow_id</code>; completion arrives via the <code>DeliveryAssignment</code> (required here).</td><td><code>pipelex[temporal]</code> + delivery target</td></tr>
+ <tr><td><code>MISTRAL_NATIVE</code></td><td>Decompose into native Mistral Workflows primitives (controllers → child workflows, leaves → activities).</td><td><code>pipelex-mistralai-workflows</code> <span class="small">(preview)</span></td></tr>
+ </tbody>
+ </table>
+ <p class="small">Optional deps stay optional — <code>temporalio</code> / the Mistral plugin are lazy-imported only inside the matching branch; a missing extra raises a precise install-hint error. The surface is already covered by unit tests (dispatch, validation, idempotent boot-race, DIRECT router scoping, trace-context isolation), so it's ready the day the consumer lands.</p>
+</details>
+
+<h2>Why this is the right shape</h2>
+
+<div class="grid-2">
+ <div class="card">
+ <h4>The lift earns its keep now</h4>
+ <p class="small">It's not abstraction-for-later: the primitives are consumed in-tree today by Temporal (and core <code>delivery_executor</code>). Moving them just stops the logic from being duplicated when Mistral arrives.</p>
+ </div>
+ <div class="card">
+ <h4>Right home, no host coupling</h4>
+ <p class="small"><code>runtime_bridge/primitives/</code> imports only Pipelex core types — no <code>temporalio</code>, no <code>mistralai</code>. "Temporal SDK glue" stays in <code>pipelex.temporal</code>; the reusable bodies don't.</p>
+ </div>
+ <div class="card">
+ <h4>Not over-engineered</h4>
+ <p class="small">The dispatch surface is a flat enum, not a plugin registry; <code>__init__.py</code> is empty (no re-export indirection, per project convention). No speculative interfaces beyond the one real upcoming consumer.</p>
+ </div>
+</div>
+
+<h2>What's <em>not</em> part of this work</h2>
+
+<div class="note warn">
+ <p>The diff looks large, but a vendored <code>/workflows</code> skill under <code>.claude/skills/</code> accounts for most of the line count and is <strong>not part of this PR</strong> — it was adapted from Mistral's own Workflows skill and rode along on this branch for convenience. Internal triage notes under <code>wip/</code> are likewise not review material. A couple of unrelated review-driven fixes also rode along (e.g. honoring <code>transport_max_retries</code> on the gateway LLM client factories).</p>
+ <p style="margin-top:8px"><strong>Review the code core:</strong> <code>pipelex/runtime_bridge/</code>, <code>pipelex/temporal/tprl_pipe/act_*.py</code>, <code>pipelex/hub.py</code>, <code>pipelex/tracing/exceptions.py</code>, and <code>tests/unit/pipelex/runtime_bridge/</code>.</p>
+</div>
+
+<h2>File map</h2>
+
+<table>
+ <thead><tr><th>Path</th><th>Role</th><th></th></tr></thead>
+ <tbody>
+ <tr><td colspan="3" class="small" style="color:var(--good); text-transform:uppercase; letter-spacing:.04em;">Consumed today (the landed payload)</td></tr>
+ <tr><td><code>runtime_bridge/primitives/delivery.py · trace_flush.py</code></td><td>Activity bodies lifted out of Temporal; behaviour-neutral</td><td><span class="pill added">lifted</span></td></tr>
+ <tr><td><code>runtime_bridge/primitives/hydration.py · submitter_hydration.py · pipe_run_arg.py</code></td><td>Moved from <code>temporal/tprl_pipe/</code> (git rename)</td><td><span class="pill moved">moved</span></td></tr>
+ <tr><td><code>temporal/tprl_pipe/act_*.py</code></td><td>Now thin <code>@activity.defn</code> wrappers over the primitives</td><td><span class="pill changed">slimmed</span></td></tr>
+ <tr><td><code>hub.py</code></td><td><code>scoped_pipe_router</code> + contextvar override in <code>get_pipe_router</code></td><td><span class="pill changed">changed</span></td></tr>
+ <tr><td><code>tracing/exceptions.py</code> · <code>tracing_assembly.py</code></td><td><code>EventLogError</code> family + best-effort degradation on backend setup failure</td><td><span class="pill changed">new / changed</span></td></tr>
+ <tr><td colspan="3" class="small" style="color:var(--warn); text-transform:uppercase; letter-spacing:.04em;">Forward-looking (no in-tree caller yet — for Mistral)</td></tr>
+ <tr><td><code>runtime_bridge/bridge.py</code></td><td><code>run_pipe_via_bridge</code> + boundary types + the four mode runners</td><td><span class="pill future">ready</span></td></tr>
+ <tr><td><code>runtime_bridge/execution_mode.py · bootstrap.py · exceptions.py</code></td><td>Mode enum · idempotent boot · bridge error family</td><td><span class="pill future">ready</span></td></tr>
+ <tr><td><code>runtime_bridge/primitives/pipe_classification.py</code></td><td>Controller/leaf split for Mistral's per-step decomposition</td><td><span class="pill future">ready</span></td></tr>
+ </tbody>
+</table>
+
+<h2>Verification</h2>
+
+<div class="note good">
+ <p><code>make agent-check</code> (pyright 0/0/0, mypy) and <code>make agent-test</code> (full suite) both green. The lifted primitives are covered by the existing Temporal suite (behaviour unchanged); the forward-looking dispatcher ships with its own unit tests.</p>
+</div>
+
+<p class="small" style="margin-top:28px; color:#6e7681;">Co-developer overview of the runtime-bridge work (PR #969, branch <code>feature/Runtime-bridge-extraction</code>). Diagrams render via mermaid from CDN.</p>
+
+</div>
+</body>
+</html>
diff --git a/wip/runtime-bridge/temporalio-bump-1.24-provenance.md b/wip/runtime-bridge/temporalio-bump-1.24-provenance.md
new file mode 100644
index 000000000..8a5316de0
--- /dev/null
+++ b/wip/runtime-bridge/temporalio-bump-1.24-provenance.md
@@ -0,0 +1,15 @@
+# `temporalio==1.24.0` bump on the bridge branch
+
+**Status: RESOLVED — keep `1.24.0`** (maintainer decision, 2026-06-07). The bump lands with this PR; `dev` moves from `1.23.0` to `1.24.0` on merge. No code change needed; this note is kept as the provenance record.
+
+## What
+
+`pyproject.toml` on `feature/Runtime-bridge-extraction` pins `temporal = ["temporalio==1.24.0", ...]`, while `dev` is on `temporalio==1.23.0`. The bump was introduced by the branch's first commit (`204cd749`, the bridge extraction carved off the `_workflows` worktree), not by any later bridge work.
+
+## Why it's a question
+
+The runtime bridge does not depend on a specific Temporal SDK version — `temporalio` is lazy-imported only inside the `TEMPORAL_*` branches and the existing `pipelex.temporal` glue, none of which references a 1.24-only API. So this bump looks like it rode along from the upstream `_workflows` branch rather than being required by this PR.
+
+## Decision
+
+Keep `1.24.0` — confirmed wanted on `dev`. The bridge itself is version-agnostic (lazy import, no 1.24-only API), but staying current is fine and CI exercises it on the in-process test server.
diff --git a/wip/runtime-bridge/trace-flush-blocking-io.md b/wip/runtime-bridge/trace-flush-blocking-io.md
new file mode 100644
index 000000000..17c6ce7db
--- /dev/null
+++ b/wip/runtime-bridge/trace-flush-blocking-io.md
@@ -0,0 +1,54 @@
+# `flush_trace_events_to_backend` — async with blocking I/O
+
+**Status:** ✅ **RESOLVED — false positive / by-design.** No code change. The `asyncio.to_thread` offload below is kept on file for if/when Temporal ships and we profile activity-worker contention.
+**Source:** PR #959 review — cubic-dev-ai (P2, `pipelex/runtime_bridge/primitives/trace_flush.py:14`).
+
+## The finding
+
+> `async def` with `# noqa: RUF029` but contains synchronous blocking I/O — the emit loop calls `DynamoDBEventLog.emit()` (boto3 `put_item` — synchronous HTTP call) and `NdjsonEventLog.emit()` (synchronous file write+flush) without yielding the event loop. Because the caller `act_flush_trace_events` is an `async def` Temporal activity that awaits this coroutine, the blocking I/O holds Temporal's async activity worker event loop for the aggregate duration of all `emit()` calls.
+
+## Why this is acceptable (the activity vs workflow distinction)
+
+The facts are correct, but the conclusion ("holds the event loop") is the *intended and correct* place for blocking I/O in a Temporal app.
+
+- The function is **designed to run in an activity**, never a workflow. Its module docstring (`trace_flush.py:1-6`) says so: *"Called from a host-runtime activity (Temporal / Mistral) after a workflow's `BufferingEventLog` has drained, since synchronous I/O (boto3, file writes) cannot run inside a workflow thread."*
+- The emit loop is synchronous on purpose (`trace_flush.py:26-32`): `make_event_log(...)` then `for event in events: event_log.emit(event)`. The `# noqa: RUF029` (`trace_flush.py:14`) documents the deliberate async-with-no-await.
+- Workflows must stay deterministic and I/O-free, so they buffer in memory via `BufferingEventLog` and hand the drained events to this activity:
+ - `pipelex/temporal/tprl_pipe/wf_pipe_router.py:152-167` — drains `event_log` and `await workflow.execute_activity(act_flush_trace_events, ... start_to_close_timeout=30s, retry_policy=max 3)`.
+- The activity is a thin wrapper: `pipelex/temporal/tprl_pipe/act_flush_trace_events.py:22-25` — `@activity.defn(name="act_flush_trace_events")` / `async def act_flush_trace_events(arg)` → `await flush_trace_events_to_backend(arg.events)`.
+- `DynamoDBEventLog.emit` even warns against being called from a workflow (its docstring says the synchronous boto3 HTTP call would block the workflow thread and trip the deadlock detector) and points callers to exactly this `BufferingEventLog` + `act_flush_trace_events` pattern.
+
+Activities are the part of Temporal that is *allowed* to block — that's their whole purpose. So this is the textbook pattern, not a bug.
+
+## The kernel of truth (only if throughput ever matters)
+
+Temporal's **async** activities share the worker's event loop. A blocking call inside one async activity does stall *sibling* async activities on the same worker for its duration. Temporal's own guidance: an async activity doing blocking I/O should either be registered as a **sync** activity (runs in a thread-pool executor) or offload the blocking part via `asyncio.to_thread` / `run_in_executor`.
+
+For trace flushing this is negligible today: best-effort, bounded (30s `start_to_close_timeout`, ≤3 retries), batched per workflow, low volume, and Temporal isn't in prod. So **no change now.**
+
+If/when activity throughput becomes a real concern, the minimal mitigation is:
+
+```python
+# trace_flush.py — offload the blocking emit loop off the event loop
+import asyncio
+
+async def flush_trace_events_to_backend(events: list[TraceEvent]) -> None:
+ ...
+ def _emit_all() -> None:
+ try:
+ for event in events:
+ event_log.emit(event)
+ finally:
+ event_log.close()
+ await asyncio.to_thread(_emit_all)
+```
+
+(That also lets the `# noqa: RUF029` go away, since the function would then genuinely await.)
+
+## Recommendation
+
+Reply on the thread as a **false positive / by-design**, citing the activity-vs-workflow distinction, and resolve it. Keep the `asyncio.to_thread` mitigation above on file here for if/when Temporal goes to prod and we profile activity-worker contention.
+
+## Related
+
+- The other two PR #959 forks are real bugs: `direct-mode-nested-router-leak.md` (#1), `graph-context-temporal-contract.md` (#3). This one is the only "false positive" of the three.