Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

fix(validate): dry-run sweep no longer leaks nested controller sub-pipes to Temporal (#976)

Louis Choquel10f6ffdcc449
diff --git a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md b/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md
index 06906abc2..9fbc71832 100644
--- a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md
+++ b/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md
@@ -7,7 +7,7 @@
## Contents
-- **Step 3** — Sequential tests: Tier 1 (sequence), Tier 2 (hydration), Tier 2b (cross-process registry), Tier 3 (parallel), Tier 4 (image generation), Tier 5 (image flow)
+- **Step 3** — Sequential tests: Tier 1 (sequence), Tier 2 (hydration), Tier 2b (cross-process registry), Tier 2c (validate sweep stays in-process), Tier 3 (parallel), Tier 4 (image generation), Tier 5 (image flow)
- **Step 4** — Verify graph output
- **Step 5** — Concurrent isolation tests (concept / pipe / multi-concept)
- **Step 5b** — Tier 8: cross-worker usage emission; Tier 8b: cross-worker cost report assembly (`--mock-inference`, free)
@@ -100,6 +100,65 @@ After this completes, tell the user:
and quote the `KajsonDecoderError` and `ApplicationError: Failed decoding arguments`
lines verbatim, plus the activity name (`act_deliver`).
+**Tier 2c — Validate sweep stays in-process under a Temporal-enabled boot (submitter-side leak guard).**
+
+This is **not** a `run` — it's a `validate`. It guards the production bug where the `/validate`
+dry-run sweep leaked nested controller sub-pipes to Temporal: a standalone `PipeBatch` swept directly
+fans out over a mock list and dispatches each branch through `get_pipe_router()`, which under a
+Temporal-enabled hub is the `TemporalPipeRouter`. Those concurrent same-id top-level dispatches raised
+`WorkflowAlreadyStartedError` → the API returned **HTTP 422** with `Failed to execute workflow
+WfPipeRouter`. The fix scopes an in-process router for the whole sweep; the contract here is that
+validation **never dispatches to Temporal**, even when booted Temporal-enabled.
+
+The `--temporal` flag (parity with `pipelex run`) flips the boot's hub default to the Temporal router
+with no `pipelex_temporary_override.toml` juggling. With the fix, the sweep still stays in-process —
+which is exactly what this asserts. `temporal_batch.mthds` declares a standalone `type = "PipeBatch"`
+pipe (`batch_temporal_describe_topics`), so its sweep fans out — the exact shape that 422'd.
+
+```bash
+# Server + worker up (Mode 2). The boot connects Temporal-enabled; with the fix the sweep stays
+# in-process, so the worker must receive NO workflow for this validate run.
+# Capture pipelex's OWN exit code (load-bearing for GREEN/RED). Don't pipe the command into
+# `tail` and read `$?` — that's tail's exit; and `${PIPESTATUS[0]}` is bash-only (blank in zsh).
+# Redirect to a file, read `$?` with no pipe in between, then tail the log. Works in bash AND zsh.
+timeout 120 .venv/bin/pipelex validate bundle \
+ tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \
+ --temporal > /tmp/tier2c-validate.log 2>&1; echo "EXIT=$?"
+tail -20 /tmp/tier2c-validate.log
+```
+
+GREEN: `EXIT=0` and `Successfully validated bundle ...`. **Strong check** (the point of the scenario):
+the worker stayed idle — it received no top-level dispatch. Capture both worker sessions and confirm
+no new `WfPipeRouter` / `WfPipeRun` execution appeared for this run (split workers are the Step 2
+default; fall back to `temporal-worker` for the single-worker setup):
+
+```bash
+tmux capture-pane -t temporal-worker-router -p -S -200 | grep -i WfPipeRouter # expect: nothing new
+tmux capture-pane -t temporal-worker-runner -p -S -200 | grep -i WfPipeRun # expect: nothing new
+```
+
+RED (prove the scenario bites) — the fix is committed, so neutralize it in the working tree. Surgical:
+in `pipelex/pipeline/bundle_validator.py`, inside `validate_pipes`, drop the `with
+scoped_pipe_router(self._pipe_router):` wrapper so the sweep loop runs unscoped (de-indent the loop
+one level). Re-run the GREEN command — expect `EXIT=1`, `Dry run failed with 1 unexpected pipe
+failure(s): 'temporal_batch_test.batch_temporal_describe_topics': ... Failed to execute workflow
+WfPipeRouter`, and the worker session now shows `WfPipeRouter` activity (the leak). **Restore the fix
+immediately:** `git checkout -- pipelex/pipeline/bundle_validator.py`.
+
+After this completes, tell the user:
+- PASS (GREEN exits 0 **and** worker idle — no dispatch) / FAIL.
+- Caveats worth stating: only a *standalone* `PipeBatch`/`PipeParallel` swept directly turns the leak
+ fatal (concurrent same-id collision); a batch reached only as a sequence sub-pipe round-trips
+ Temporal but passes (a false pass), so "validate exited 0" alone is **not** sufficient — the
+ worker-idle check is the real assertion. Worker up vs down are **both** RED (collision
+ `WorkflowAlreadyStartedError` vs no-worker `RPCError`, same `except` branch); keep the worker up to
+ match production.
+- The cheaper CI-automated companion to this scenario is the Mode-1 pytest
+ `tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py` (real
+ `TemporalPipeRouter` as hub default, spies `WorkflowExecutor.execute_workflow`, asserts never
+ called). This Mode-2 scenario is the deployment-faithful demonstration across the real API↔worker
+ process boundary.
+
**Tier 3 — Do parallel branches execute as separate child workflows?**
This sends a PipeParallel controller that fans out into two branches (tone analysis and
@@ -1069,6 +1128,7 @@ ls results/*/reactflow.html
| Tier 1: Sequence | Worker can unpack crate and run a pipe sequence | PASS/FAIL | path | — |
| Tier 2: Hydration | Worker handles dynamic concepts it has never seen | PASS/FAIL | path | — |
| Tier 2b: Cross-process registry | `act_deliver` decodes hydrated `pipe_output` on runner (forced via `delivery_assignment`) | PASS/FAIL | — | — |
+| Tier 2c: Validate sweep stays in-process | `validate bundle --temporal` over a standalone `PipeBatch` exits 0 **and** the worker received no `WfPipeRouter` dispatch (the sweep never leaks to Temporal) | PASS/FAIL | — | — |
| Tier 3: Parallel | Branches execute as concurrent child workflows | PASS/FAIL | path | — |
| Tier 4: ImgGen | Image generation pipeline works through Temporal | PASS/FAIL | path | yes/no |
| Tier 5: Image flow | Generated image flows as input to next pipe step | PASS/FAIL | path | yes/no |
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f94cd234f..4953cda19 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,8 @@ This cycle reworks **cost reporting** so it survives distributed execution and s
### Added
+- **`--temporal` / `--no-temporal` on `pipelex validate`.** The `bundle`, `pipe`, and `method` validate subcommands (and `validate --all`) gain the same boot override `run` already had — it sets `temporal.is_enabled` for the process. Today the validation sweep always runs in-process regardless of backend (it scopes its own in-process router), so the flag does not change *what* validation does; it is the lever for booting validation under a Temporal-enabled hub — e.g. to verify the sweep stays in-process and never dispatches nested controller sub-pipes to Temporal. It is also forward-looking: once validation can run as a standalone Temporal activity, this flag becomes the switch that dispatches the sweep through that activity.
+
- **`--costs` / `--no-costs` (default on).** `pipelex run pipe|method|bundle` gains a dedicated cost switch that emits usage tracing events and renders the end-of-run cost report. It rides the shared trace-event transport independently of `--graph`, so `--no-graph --costs` reports cost without building a graph and `--graph --no-costs` builds a graph with no cost report. It replaces the removed `--cost-report` (see Changed).
- **Distributed cost reporting.** In Temporal mode, usage emitted from inference activities running on separate worker processes is now aggregated into a single cost report at the submitter. Usage is assembled from the trace-event stream onto `PipeOutput` in both direct and Temporal execution, so a cross-worker run reports its true total — previously cross-worker usage was emitted but never rendered into a report.
@@ -38,6 +40,10 @@ This cycle reworks **cost reporting** so it survives distributed execution and s
- **Temporal silent-hang fail-safe.** A pipelex domain error raised *inline* in workflow code — never dispatched through an activity — was neither an `ActivityError` nor an `ApplicationError`, so Temporal treated it as a workflow-task failure and retried it indefinitely: a silent, resource-burning hang that only surfaced (as a generic timeout) after the workflow execution timeout. `WfPipeRouter` and `WfPipeRun` now end their boundary handling with a `PipelexError` catch-all that converts a genuine inline error to a terminal, classified failure (the parent also still fires the FAILED webhook), and the worker registers `PipelexError` in `workflow_failure_exception_types` as a backstop so any uncaught domain error fails the workflow terminally instead of hanging. Scoped to domain errors only — transient Temporal/infra errors keep their (correct) task-retry behavior.
+- **`PipeSearch` on Temporal no longer hangs.** Web search was the only inference operator that ran its leaf inline instead of through the `ContentGenerator` seam, so it had no Temporal activity. Run through the Temporal path, a search step executed its real HTTP call inside the workflow event loop, and on failure raised a raw `SearchJobFailureError` (neither an `ActivityError` nor a `WorkflowExecutionError`) that Temporal treated as a workflow-task failure and retried forever — the submitter hung instead of getting a result. Even on success it was replay-unsafe (the search re-ran on every workflow replay). `PipeSearch` now dispatches its leaf through new `act_search_gen_sourced_answer` / `act_search_gen_structured` activities like every other operator: results are recorded in workflow history (replay-safe) and failures cross the workflow boundary as a terminal `WorkflowExecutionError` carrying the structured `ErrorReport` — identical classification to the local path. A `runner-search` worker scope and per-activity routing entries are available for split deployments.
+
+- **Bundle validation no longer leaks nested controller sub-pipes to Temporal.** With Temporal enabled, the in-process dry-run sweep (`BundleValidator`) ran the top-level pipe in-process but dispatched every *nested* controller sub-pipe through the hub-default router — the Temporal router — turning a no-cost dry run into real top-level workflow dispatches. A standalone `PipeBatch` / `PipeParallel` swept directly fanned out N concurrent same-id top-level dispatches and collided (`WorkflowAlreadyStartedError`), surfacing as **HTTP 422** on the runner's `/validate` route; other shapes silently round-tripped Temporal. The sweep now scopes its in-process router for the whole sweep (mirroring the DIRECT-mode runtime bridge), so nested controllers resolve the in-process router and validation stays fully in-process regardless of the configured backend.
+
## [v0.31.0] - 2026-06-04
This release hardens Pipelex at its edges. The headliners: a full **error-handling overhaul** that gives every error a stable, RFC 7807-shaped identity and carries it all the way out to webhooks and external surfaces; **lenient validation with pipe signatures**, so you can sketch and dry-run a whole pipeline top-down before a single pipe is implemented; and a first, **experimental cut of CSV tabular support** for reading and writing typed lists straight from `.csv`.
diff --git a/TODOS.md b/TODOS.md
new file mode 100644
index 000000000..8395ab551
--- /dev/null
+++ b/TODOS.md
@@ -0,0 +1,137 @@
+# Fix: `/validate` dry-run leaked nested controller sub-pipes to Temporal
+
+> **Status: DONE.** Bug fixed and all three layers of test coverage are in place. This doc is the PR-review guide — what the bug was, what changed, why it's correct, and how it's guarded.
+>
+> **Stacking:** this branch (`fix/Temporal-dry-run`) is stacked on `feature/PipeSearch-Temporal` (open PR [#974](https://github.com/Pipelex/pipelex/pull/974)). The `pipe_search.py` / `search_generate.py` / `content_generator_in_workflow.py` / `native_search.mthds` / `structured_search.mthds` / `test_workflow_search_*` files in the diff belong to **#974**, not to this change — review them there. The files owned by **this** change are listed under [Files in this change](#files-in-this-change) below.
+
+**Where the fix lives:** `pipelex` core (the runtime), **not** `pipelex-api`. All paths are relative to the `pipelex` repo (locally the editable `_search` worktree). No change was needed in `pipelex-api`.
+
+## Symptom
+
+With the API running Temporal-enabled (`.pipelex/pipelex_override.toml` → `[temporal] is_enabled = true`, task queue `pipelex_dev`) and a worker up:
+
+- `make bundle-validate BUNDLE=.../fashion_moodboard` → **HTTP 422**
+- `make bundle-validate BUNDLE=.../joke_judge` → **OK**
+- `make bundle-run BUNDLE=.../fashion_moodboard` (full run) → **OK**
+- `pipelex validate bundle .../fashion_moodboard` (CLI) → **OK**
+
+The 422 body:
+
+```
+ValidateBundleError (error_domain=input), 422
+detail: "Dry run failed with 1 unexpected pipe failure(s):
+ 'fashion_moodboard.create_moodboards': ... status=DryRunStatus.FAILURE
+ error_message=\"Dry run failed for pipe 'fashion_moodboard.create_moodboards':
+ Failed to execute workflow WfPipeRouter\""
+```
+
+The string **"Failed to execute workflow WfPipeRouter"** (no trailing detail) is produced only at `pipelex/temporal/tprl/workflow_caller.py:136`, the `(WorkflowAlreadyStartedError, RPCError)` branch of `WorkflowExecutor.execute_workflow` — i.e. a **top-level** Temporal workflow dispatch from the submitter side. A dry run should never reach that code.
+
+## Root cause
+
+The `/validate` path runs an **in-process** dry-run sweep, but nested controller sub-pipes leaked out to the Temporal router.
+
+Flow: `api/routes/pipelex/validate.py` → `validate_bundle()` (`pipelex/pipeline/validate_bundle.py`) → `BundleValidator.validate_pipes()` (`pipelex/pipeline/bundle_validator.py`).
+
+`BundleValidator` deliberately builds a **local, in-process** execution primitive so the sweep never touches Temporal — its own module docstring says so. **But that local router was only used for the _top-level_ pipe.** Every controller dispatches its nested sub-pipes through the **hub's** `get_pipe_router()`, not the injected router. This is broad — it is **not** specific to `PipeBatch`:
+
+```python
+# pipelex/pipe_controllers/batch/pipe_batch.py (_live_run_controller_pipe)
+return await get_pipe_router().run(pipe_job=PipeJobFactory.make_pipe_job(...))
+
+# pipelex/pipe_controllers/sub_pipe.py (SubPipe.run_pipe — used by EVERY PipeSequence step;
+# also the inline `batch_over` path and the PipeCondition path)
+pipe_output = await get_pipe_router().run(pipe_job=PipeJobFactory.make_pipe_job(...))
+```
+
+`get_pipe_router()` (`pipelex/hub.py`) returns a contextvar override **if one is set**, else the hub default. In the Temporal-enabled API process the hub default is the **`TemporalPipeRouter`** (wired in `pipelex/pipelex.py` when `get_config().temporal.is_enabled`). Called from the API process — i.e. **outside** any Temporal workflow (`is_in_temporal_workflow()` is False) — `TemporalPipeRouter._run_pipe_job` took the **top-level dispatch** branch → `executor.execute_workflow(WfPipeRouter, ...)` → failed → "Failed to execute workflow WfPipeRouter". The per-pipe dry run was marked `FAILURE`, `BundleValidator._aggregate` raised `DryRunError`, `validate_bundle` translated it to `ValidateBundleError`, and the global handler rendered 422.
+
+`BundleValidator` built the in-process router but **never installed it as the `scoped_pipe_router(...)` override**, so the nested controllers fell through to the hub default (Temporal).
+
+## FAIL vs. FALSE-PASS — the leak happened always; only some shapes 422'd
+
+The leak (nested dispatch → `TemporalPipeRouter`) happened for **every** bundle with a controller. Whether the dry run **failed** depended on a Temporal **workflow-id collision**:
+
+- `make_top_workflow_id` (`pipelex/temporal/temporal_manager.py`) is `f"{prefix}{pipeline_run_id}"` — derived from `pipeline_run_id` **alone**. The whole sweep shares **one** `dry_run_pipeline_id`, so **every** top-level Temporal dispatch in a sweep computed the **same** workflow id.
+- `WorkflowExecutor.execute_workflow` **waits for completion**, and no `id_reuse_policy` is set → Temporal's default `ALLOW_DUPLICATE` (reuse allowed once the prior run has *closed*). So **serial** dispatches reusing that id are fine; only **concurrent** same-id dispatches collide → `WorkflowAlreadyStartedError` → `WorkflowExecutionError("Failed to execute workflow WfPipeRouter")`. (`WorkflowExecutionError` is a `PipelexError`, so it records as a dry-run `FAILURE` → 422, not a 500.)
+
+The deciding structural factor is **standalone batch/parallel vs. batch-reached-as-a-sub-pipe**:
+
+- A **standalone top-level `PipeBatch`/`PipeParallel`** the sweep dry-runs *directly* runs its top level on the local in-process router, but its **fan-out loop executes in the API process** (`_live_run_controller_pipe` → `gather_bounded`) and fires **N concurrent top-level Temporal dispatches** with the same id → collision → **FAILURE**. A mocked "multiple" list yields `nb_stuffs = 3`, so N≥2.
+- A **batch reached as a sub-pipe** (inline `batch_over` step, or a batch nested under a sequence/parallel) is dispatched as a **single** top-level workflow — the fan-out into branches then happens **inside the worker** as child workflows (legal). Sequence steps run serially, so it's one dispatch at a time → **no concurrent collision** → it **passed** (still round-tripping Temporal, just not colliding — a false pass).
+
+So `fashion_moodboard`'s `create_moodboards` (a standalone `PipeBatch`) 422'd, while a sequence-nested batch silently round-tripped Temporal and passed. The fix removes the Temporal round-trip for the whole sweep, fixing both the real failure **and** the silent false-pass leak.
+
+## The fix as shipped
+
+### Part 1 — scope the in-process router for the whole sweep (commit `3377babb`)
+
+The codebase had already solved this exact leak for DIRECT mode in `pipelex/runtime_bridge/bridge.py::_run_direct`, with a comment describing the precise failure mode. `BundleValidator` now applies the same guard around its dry-run sweep:
+
+1. `__init__` keeps the router instance instead of discarding it inside `PipeRun(...)`:
+ ```python
+ self._pipe_router = PipeRouter(observer=ObserverNoOp())
+ self._pipe_run: PipeRunProtocol = PipeRun(pipe_router=self._pipe_router)
+ ```
+2. `validate_pipes` wraps the **per-pipe dry-run sweep loop** (step 4) in `scoped_pipe_router`:
+ ```python
+ with scoped_pipe_router(self._pipe_router):
+ for pipe in sweepable_pipes:
+ results[pipe.pipe_ref] = await self._classify_pipe(...)
+ ```
+ Wrapping `validate_pipes` covers all entry points (`acquire_and_validate`, `validate_current_library`, and the direct `validate_bundle` path all funnel through it). Steps 1–3 (wiring check, signature pre-pass, telemetry) don't run pipes, so they don't need the scope.
+
+**Why it's safe:**
+
+- `scoped_pipe_router` is **contextvar-scoped** and restores the prior value on exit, so concurrent `/validate` requests don't leak into each other.
+- `PipeBatch` fans branches out via `gather_bounded` → `asyncio.gather`, whose Tasks copy the current context at creation (and creation happens inside the `with` scope), so each branch task inherits the override and resolves the in-process router too.
+
+### Part 2 — `--temporal/--no-temporal` flag on `validate` (commit `b8eadeda`)
+
+A `--temporal/--no-temporal` flag was added to `pipelex validate bundle`, `pipe`, and `method` (and therefore `validate --all`, which routes to `validate pipe`), giving parity with `pipelex run`. It threads through `_validate_core.execute_validate(temporal=...)` → `make_pipelex_for_cli(temporal_enabled=...)`, overriding `temporal.is_enabled` **for the boot only**.
+
+**This flag is behavior-neutral for validation today.** The sweep always runs in-process regardless of the flag (Part 1 guarantees that). The flag does not change *what* validation does — it controls *how Pipelex boots*, which is the lever for exercising the "validation stays in-process even on a Temporal-enabled hub" contract without juggling a `pipelex_temporary_override.toml`. It is also forward-looking: once validation runs as a standalone Temporal activity ([`wip/dry-run-refactor/followup-temporal-validation-activity.md`](wip/dry-run-refactor/followup-temporal-validation-activity.md)), `--temporal` stops being a no-op and becomes the switch that dispatches the sweep through that activity.
+
+## Reviewer note — the graph step was deliberately NOT changed
+
+An earlier diagnosis claimed the best-effort **graph** step in `validate.py` — `dry_run_pipeline()` (`pipelex/pipe_run/dry_run_pipeline.py`) → `PipelexRunner(...).execute_pipeline(...)` — had "the same leak." **It does not**, and an attempt to "fix" it was **reverted**. The graph step does a **single** top-level dispatch of the main pipe — no concurrent same-id collision — so under Temporal it dispatches the one workflow to the worker, which (tracing enabled, `backend = "temporal_dynamodb"`) assembles the `GraphSpec` and returns it on `PipeOutput`. That is the intended distributed design: `pipelex-api` runs with tracing disabled (thin submitter); the **worker** owns tracing + graph assembly. Forcing it in-process broke graph generation (tracing off in the API → empty `GraphSpec` → graph dropped). **Do not "fix" the graph step to run in-process** — if a `/validate` graph is wanted without a worker, the lever is enabling tracing in the API, a separate deployment decision.
+
+## Test coverage — all three layers in place
+
+The contract — *the validation sweep never dispatches to Temporal* — is guarded at three complementary layers (none redundant):
+
+| Layer | Where it runs | CI-automated | What it adds | Status |
+|---|---|---|---|---|
+| **Sentinel regression test** (no Temporal) | `tests/integration/pipelex/pipeline/test_bundle_validator.py::TestBundleValidatorIntegration::test_standalone_batch_sweep_scopes_in_process_router` | ✅ | Installs a *raising* hub-default router; sweeps a standalone `PipeBatch`; asserts SUCCESS with the hub default never touched. Deterministic, zero Temporal infra. | exists |
+| **Mode-1 pytest** (real Temporal router) | `tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py::TestValidateSweepStaysInProcess` | ✅ | Resolves the **real** `TemporalPipeRouter` as the hub default (proving the config→hub-default *wiring* the sentinel bypasses); spies `WorkflowExecutor.execute_workflow` and asserts it's never called. | **new — this change** |
+| **Mode-2 scenario** (3 separate processes) | `temporal-e2e-validate` skill, Tier 2c (agent-run) | ✗ | Production-faithful API↔worker boundary: `validate bundle --temporal` over a standalone `PipeBatch` exits 0 **and** the worker stays idle (no `WfPipeRouter` dispatch). Documents the GREEN/RED procedure. | **new — this change** |
+
+The Mode-1 pytest is the cheaper automated catch (fast, deterministic, runs in CI); the Mode-2 scenario is the deployment-faithful demonstration and the only one that also exercises the genuinely cross-process surfaces (LibraryCrate propagation, serialization). Both were verified RED with the `scoped_pipe_router` scope removed and GREEN with it in place.
+
+The repro bundle reuses an existing fixture (no new fixture): `tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds` already declares a standalone `type = "PipeBatch"` pipe (`batch_temporal_describe_topics`) whose direct sweep fans out — the exact shape that 422'd.
+
+## Files in this change
+
+Owned by **this** change (the rest of the diff belongs to PR #974 — see [Stacking](#fix-validate-dry-run-leaked-nested-controller-sub-pipes-to-temporal)):
+
+- `pipelex/pipeline/bundle_validator.py` — the fix (`__init__` keeps `self._pipe_router`; `validate_pipes` wraps the sweep in `scoped_pipe_router`).
+- `pipelex/cli/commands/validate/{_validate_core,bundle_cmd,method_cmd,pipe_cmd}.py` — the `--temporal/--no-temporal` flag.
+- `tests/integration/pipelex/pipeline/test_bundle_validator.py` — sentinel regression test (+ siblings).
+- `tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py` — **new** Mode-1 pytest.
+- `tests/unit/pipelex/cli/test_validate_temporal_flag.py` — locks the CLI surface (each subcommand exposes `--temporal/--no-temporal`; `--help` short-circuits before boot).
+- `.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md` — **new** Tier 2c scenario (+ Contents + Step 7 row).
+- `CHANGELOG.md` — entry under `[Unreleased]`.
+- `TODOS.md`, `wip/distributed-execution/validate-sweep-temporal-leak-repro.md` — this guide + the repro brief.
+
+## How to verify (local setup)
+
+API on :8081 (`make run`), Temporal server + worker up (from `pipelex-worker`), both on the editable `pipelex` checkout.
+
+```bash
+# was failing → now 200 with validated bundle
+make bundle-validate BUNDLE=/Users/lchoquel/repos/Pipelex/pipelex-demos/mthds-wip/fashion_moodboard
+# regression guard → still 200
+make bundle-validate BUNDLE=/Users/lchoquel/repos/Pipelex/pipelex-demos/mthds-wip/joke_judge
+```
+
+CLI parity check (no override file needed): `pipelex validate bundle <bundle> --temporal` boots Temporal-enabled yet the sweep stays in-process — exits 0 with no workflow dispatched to the worker. See [`wip/distributed-execution/validate-sweep-temporal-leak-repro.md`](wip/distributed-execution/validate-sweep-temporal-leak-repro.md) for the full GREEN/RED procedure (now implemented as Tier 2c in the `temporal-e2e-validate` skill).
diff --git a/docs/distributed-execution/task-routing.md b/docs/distributed-execution/task-routing.md
index 56248a0a3..fd312c154 100644
--- a/docs/distributed-execution/task-routing.md
+++ b/docs/distributed-execution/task-routing.md
@@ -51,6 +51,8 @@ The **routing key** is the runtime handle the activity is about to call:
| `act_llm_gen_object_list` | LLM model handle | The `llm_handle` on the call |
| `act_img_gen_images` | `img_gen_handle` | The image generation backend |
| `act_extract_gen_extract_pages` | `extract_handle` | The OCR / extract backend |
+| `act_search_gen_sourced_answer` | `search_handle` | The web search backend |
+| `act_search_gen_structured` | `search_handle` | The web search backend |
| `act_jinja2_gen_text` | (none — pass `None`) | Template rendering has no handle |
| `act_render_page_views` | (none — pass `None`) | Page-view rendering has no handle |
diff --git a/docs/distributed-execution/workers.md b/docs/distributed-execution/workers.md
index 6fc61e707..f14f0a4c9 100644
--- a/docs/distributed-execution/workers.md
+++ b/docs/distributed-execution/workers.md
@@ -75,6 +75,7 @@ Built-in scopes:
| `runner-img-gen` | None | `act_img_gen_images`, `act_render_page_views` | One pool dedicated to image generation. |
| `runner-extract` | None | `act_extract_gen_extract_pages`, `act_render_page_views` | One pool dedicated to OCR / document extraction. |
| `runner-jinja2` | None | `act_jinja2_gen_text` | One pool dedicated to template rendering (rare; usually fine in-band). |
+| `runner-search` | None | `act_search_gen_sourced_answer`, `act_search_gen_structured` | One pool dedicated to web search calls. |
`act_render_page_views` is intentionally registered under **both** `runner-img-gen` and `runner-extract` so neither pool breaks when the activity is routed to either. The `runner-llm` scope splits all three LLM activity types (text, object, object-list) onto the same pool — they share the same provider connections so splitting further would only add operational overhead.
diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.md
index bec00a2c0..2b3005fb9 100644
--- a/docs/under-the-hood/distributed-content-generation.md
+++ b/docs/under-the-hood/distributed-content-generation.md
@@ -7,7 +7,10 @@ description: "How Pipelex serializes dynamic Pydantic models and large binary co
This page is for contributors working on Pipelex internals. If you're deploying or operating Pipelex on Temporal, see the user-facing [Distributed Execution with Temporal](../distributed-execution/temporal/index.md) guide instead.
-This page covers the serialization mechanisms that enable content generation (LLM structured output, image generation, PDF extraction) to run across Temporal worker processes. For the broader Temporal architecture — LibraryCrate propagation, deferred hydration, per-workflow scoping — see [Temporal Integration](./temporal-integration.md).
+This page covers the serialization mechanisms that enable content generation (LLM structured output, image generation, PDF extraction, web search) to run across Temporal worker processes. For the broader Temporal architecture — LibraryCrate propagation, deferred hydration, per-workflow scoping — see [Temporal Integration](./temporal-integration.md).
+
+!!! note "Every inference leaf goes through the same seam"
+ All inference operators dispatch their leaf call through the swappable `ContentGenerator` abstraction: direct inline (`ContentGenerator`), as a Temporal activity (`ContentGeneratorInWorkflow`), or as a dry mock (`ContentGeneratorDry`). Web search (`PipeSearch`) joined this seam last — its `act_search_gen_*` activities make search-on-Temporal replay-safe (the result is recorded in workflow history) and let search failures cross the workflow boundary as classified errors. An operator that runs its leaf inline instead would re-execute it on every replay and, on failure, hang the submitter with an unclassified workflow-task error.
---
@@ -137,6 +140,10 @@ sequenceDiagram
S->>S: model_validate() → original type
```
+### Structured web search reuses the same mechanism
+
+Structured web search (`PipeSearch` with a non-text output concept) faces the identical dynamic-class problem and solves it the same way. `SearchObjectAssignment` mirrors `ObjectAssignment` — it ships `output_class_name` + `output_class_schema` alongside the `SearchAssignment`, the activity reconstructs a throwaway class via `SchemaToModelFactory` to drive the provider call, and the activity returns the **raw result dict**. The submitter re-validates that dict against the original output class (`output_structure_class.model_validate(result_dict)`) — a pure, deterministic step that keeps the dynamic class on the submitter side and never ships it across the boundary. The sourced-answer path (`make_search_sourced_answer`) has no dynamic class at all: it returns a `SearchResultContent`, a native serializable model.
+
---
## Large Payload Management
@@ -179,6 +186,8 @@ async def act_img_gen_images(img_gen_assignment: ImgGenAssignment) -> list[Image
| Rendered page views | `act_render_page_views` | PDF page renders → S3 | `list[ImageContent]` with URIs |
| LLM text | `act_llm_gen_text` | Nothing stored (text is small) | Plain `str` |
| LLM object | `act_llm_gen_object` | Nothing stored (JSON is small) | `BaseModel` + `__kajson_class_source__` in metadata |
+| Search sourced answer | `act_search_gen_sourced_answer` | Nothing stored (answer + source refs are small) | `SearchResultContent` (answer + `DocumentContent` sources) |
+| Search structured | `act_search_gen_structured` | Nothing stored (JSON is small) | Raw `dict`, re-validated against the output class on the submitter |
---
@@ -215,13 +224,15 @@ Resolution order:
| Component | File |
|-----------|------|
| Assignment models (schema embedding) | `pipelex/cogt/content_generation/assignment_models.py` |
-| Schema-to-model reconstruction | `pipelex/cogt/content_generation/schema_to_model.py` |
+| Schema-to-model reconstruction | `pipelex/cogt/content_generation/schema_to_model_factory.py` |
| Content generator (type bridge) | `pipelex/cogt/content_generation/content_generator.py` |
| LLM generation functions | `pipelex/cogt/content_generation/llm_generate.py` |
+| Search generation functions | `pipelex/cogt/content_generation/search_generate.py` |
| Generated content factory (storage) | `pipelex/cogt/content_generation/generated_content_factory.py` |
| Temporal data converter | `pipelex/temporal/temporal_data_converter.py` |
| Image generation activity | `pipelex/temporal/tprl_content_generation/act_img_gen_generate.py` |
| Extract activity | `pipelex/temporal/tprl_content_generation/act_extract_generate.py` |
+| Search activity | `pipelex/temporal/tprl_content_generation/act_search_generate.py` |
| Page view rendering activity | `pipelex/temporal/tprl_content_generation/act_render_page_views.py` |
| Kajson serialization | `kajson` (external PyPI package) |
| ImageContent model | `pipelex/core/stuffs/image_content.py` |
diff --git a/pipelex/cli/commands/validate/_validate_core.py b/pipelex/cli/commands/validate/_validate_core.py
index c87832424..f73357992 100644
--- a/pipelex/cli/commands/validate/_validate_core.py
+++ b/pipelex/cli/commands/validate/_validate_core.py
@@ -163,9 +163,17 @@ def execute_validate(
library_dirs: list[Path] | None,
telemetry_command_label: str = COMMAND,
allow_signatures: bool = False,
+ temporal: bool | None = None,
) -> None:
- """Synchronous entry point wrapping the async validation with Pipelex setup/teardown."""
- make_pipelex_for_cli(context=ErrorContext.VALIDATION, needs_inference=False, needs_model_specs=True)
+ """Synchronous entry point wrapping the async validation with Pipelex setup/teardown.
+
+ ``temporal`` overrides ``temporal.is_enabled`` for the boot (mirroring ``run``). The
+ validation sweep itself always runs in-process — under a Temporal-enabled hub it scopes
+ its own in-process router so nested controller sub-pipes do not dispatch to Temporal — so
+ this flag does not change *what* validation does; it controls how Pipelex boots, which is
+ the lever for exercising the "validation stays in-process on a Temporal backend" contract.
+ """
+ make_pipelex_for_cli(context=ErrorContext.VALIDATION, needs_inference=False, needs_model_specs=True, temporal_enabled=temporal)
try:
with get_telemetry_manager().telemetry_context():
diff --git a/pipelex/cli/commands/validate/bundle_cmd.py b/pipelex/cli/commands/validate/bundle_cmd.py
index 5da47ad46..4e125f579 100644
--- a/pipelex/cli/commands/validate/bundle_cmd.py
+++ b/pipelex/cli/commands/validate/bundle_cmd.py
@@ -30,6 +30,14 @@ def validate_bundle_cmd(
help="Accept PipeSignature placeholders in the dependency graph (lenient mode).",
),
] = False,
+ temporal: Annotated[
+ bool | None,
+ typer.Option(
+ "--temporal/--no-temporal",
+ help="Override config temporal.is_enabled for the boot. The sweep stays in-process either way; "
+ "use it to verify validation does not dispatch to Temporal under a Temporal-enabled hub.",
+ ),
+ ] = None,
) -> None:
"""Validate a bundle file (.mthds) or pipeline directory.
@@ -96,4 +104,5 @@ def validate_bundle_cmd(
library_dirs=library_dirs_paths,
telemetry_command_label=f"{COMMAND} bundle",
allow_signatures=allow_signatures,
+ temporal=temporal,
)
diff --git a/pipelex/cli/commands/validate/method_cmd.py b/pipelex/cli/commands/validate/method_cmd.py
index 33372891b..cd8a46ec7 100644
--- a/pipelex/cli/commands/validate/method_cmd.py
+++ b/pipelex/cli/commands/validate/method_cmd.py
@@ -26,6 +26,14 @@ def validate_method_cmd(
help="Directory to search for pipe definitions (.mthds files). Can be specified multiple times.",
),
] = None,
+ temporal: Annotated[
+ bool | None,
+ typer.Option(
+ "--temporal/--no-temporal",
+ help="Override config temporal.is_enabled for the boot. The sweep stays in-process either way; "
+ "use it to verify validation does not dispatch to Temporal under a Temporal-enabled hub.",
+ ),
+ ] = None,
) -> None:
"""Validate an installed method by name.
@@ -51,4 +59,5 @@ def validate_method_cmd(
bundle_path=None,
library_dirs=library_dirs_paths,
telemetry_command_label=f"{COMMAND} method",
+ temporal=temporal,
)
diff --git a/pipelex/cli/commands/validate/pipe_cmd.py b/pipelex/cli/commands/validate/pipe_cmd.py
index aad45009b..dade273ec 100644
--- a/pipelex/cli/commands/validate/pipe_cmd.py
+++ b/pipelex/cli/commands/validate/pipe_cmd.py
@@ -41,6 +41,14 @@ def validate_pipe_cmd(
help="Accept PipeSignature placeholders in the dependency graph (lenient mode).",
),
] = False,
+ temporal: Annotated[
+ bool | None,
+ typer.Option(
+ "--temporal/--no-temporal",
+ help="Override config temporal.is_enabled for the boot. The sweep stays in-process either way; "
+ "use it to verify validation does not dispatch to Temporal under a Temporal-enabled hub.",
+ ),
+ ] = None,
) -> None:
"""Validate and dry run a pipe by code, or all pipes.
@@ -65,6 +73,7 @@ def validate_pipe_cmd(
library_dirs=library_dirs_paths,
needs_inference=False,
needs_model_specs=True,
+ temporal_enabled=temporal,
)
do_validate_all_libraries_and_dry_run(
library_dirs=library_dirs_paths,
@@ -118,4 +127,5 @@ def validate_pipe_cmd(
library_dirs=library_dirs_paths,
telemetry_command_label=f"{COMMAND} pipe",
allow_signatures=allow_signatures,
+ temporal=temporal,
)
diff --git a/pipelex/cogt/content_generation/assignment_models.py b/pipelex/cogt/content_generation/assignment_models.py
index 660318523..73380b4ad 100644
--- a/pipelex/cogt/content_generation/assignment_models.py
+++ b/pipelex/cogt/content_generation/assignment_models.py
@@ -10,6 +10,7 @@ from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPrompt
from pipelex.cogt.llm.llm_job_components import LLMJobParams
from pipelex.cogt.llm.llm_prompt import LLMPrompt
from pipelex.cogt.llm.llm_setting import LLMSetting
+from pipelex.cogt.search.search_setting import SearchSetting
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.cogt.templating.templating_style import TemplatingStyle
from pipelex.pipeline.job_metadata import JobMetadata
@@ -107,3 +108,49 @@ class RenderPageViewsAssignment(BaseModel):
job_metadata: JobMetadata
document_uri: str
page_views_dpi: int
+
+
+class SearchAssignment(BaseModel):
+ """Serializable unit for a single web-search leaf call.
+
+ Carries everything the framework-agnostic ``search_generate`` core needs to rebuild the
+ ``SearchJob`` on the other side of the Temporal boundary: the rendered ``query``, the fully
+ resolved ``search_setting`` (its ``model`` is the resolved provider handle, also the routing
+ key), and the per-call domain/date overrides. Mirrors ``LLMAssignment`` for the LLM leaf.
+ """
+
+ job_metadata: JobMetadata
+ query: str
+ search_setting: SearchSetting
+ include_domains: list[str] | None = None
+ exclude_domains: list[str] | None = None
+ from_date: str | None = None
+ to_date: str | None = None
+
+ @property
+ def search_handle(self) -> str:
+ return self.search_setting.model
+
+
+class SearchObjectAssignment(BaseModel):
+ """Structured-search counterpart of ``ObjectAssignment``.
+
+ Ships the output structure's JSON schema (not the live class) across the boundary so the
+ activity can reconstruct a throwaway class via ``SchemaToModelFactory`` for the provider call.
+ The submitter re-validates the returned dict against the original class.
+ """
+
+ output_class_name: str
+ output_class_schema: dict[str, Any]
+ search_assignment: SearchAssignment
+
+ @staticmethod
+ def make_for_class(
+ output_class: type[BaseModel],
+ search_assignment: SearchAssignment,
+ ) -> "SearchObjectAssignment":
+ return SearchObjectAssignment(
+ output_class_name=output_class.__name__,
+ output_class_schema=output_class.model_json_schema(),
+ search_assignment=search_assignment,
+ )
diff --git a/pipelex/cogt/content_generation/content_generator.py b/pipelex/cogt/content_generation/content_generator.py
index 6ad0de243..9265fbe19 100644
--- a/pipelex/cogt/content_generation/content_generator.py
+++ b/pipelex/cogt/content_generation/content_generator.py
@@ -9,6 +9,8 @@ from pipelex.cogt.content_generation.assignment_models import (
ImgGenAssignment,
LLMAssignment,
ObjectAssignment,
+ SearchAssignment,
+ SearchObjectAssignment,
TemplatingAssignment,
)
from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata
@@ -17,6 +19,7 @@ from pipelex.cogt.content_generation.extract_generate import extract_gen_pages
from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
from pipelex.cogt.content_generation.img_gen_generate import img_gen_image_list, img_gen_single_image
from pipelex.cogt.content_generation.llm_generate import llm_gen_object, llm_gen_object_list, llm_gen_text
+from pipelex.cogt.content_generation.search_generate import search_gen_sourced_answer, search_gen_structured
from pipelex.cogt.content_generation.templating_generate import templating_gen_text
from pipelex.cogt.extract.extract_input import ExtractInput
from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
@@ -31,6 +34,7 @@ from pipelex.cogt.templating.templating_style import TemplatingStyle
from pipelex.config import get_config
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.page_content import PageContent
+from pipelex.core.stuffs.search_result_content import SearchResultContent
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.misc.image_utils import ImageFormat
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
@@ -317,3 +321,24 @@ class ContentGenerator(ContentGeneratorProtocol):
page_content.page_view = page_view_contents.pop(0)
return page_contents
+
+ @override
+ async def make_search_sourced_answer(
+ self,
+ search_assignment: SearchAssignment,
+ ) -> SearchResultContent:
+ return await search_gen_sourced_answer(search_assignment=search_assignment)
+
+ @override
+ async def make_search_structured(
+ self,
+ output_structure_class: type[BaseModelTypeVar],
+ search_assignment: SearchAssignment,
+ ) -> BaseModelTypeVar:
+ result_dict = await search_gen_structured(
+ search_object_assignment=SearchObjectAssignment.make_for_class(
+ output_class=output_structure_class,
+ search_assignment=search_assignment,
+ ),
+ )
+ return output_structure_class.model_validate(result_dict)
diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.py
index eea614c22..3b0b78a30 100644
--- a/pipelex/cogt/content_generation/content_generator_dry.py
+++ b/pipelex/cogt/content_generation/content_generator_dry.py
@@ -3,8 +3,10 @@ from typing import Any
from typing_extensions import override
from pipelex import log
+from pipelex.cogt.content_generation.assignment_models import SearchAssignment
from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata
from pipelex.cogt.content_generation.dry_mock import build_mock_object, report_dry_llm_job
+from pipelex.cogt.content_generation.dry_run_factory import DryRunFactory
from pipelex.cogt.extract.extract_input import ExtractInput
from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
from pipelex.cogt.extract.extract_output import ExtractOutput, Page
@@ -17,8 +19,10 @@ from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.cogt.templating.templating_style import TemplatingStyle
from pipelex.config import get_config
+from pipelex.core.stuffs.document_content import DocumentContent
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.page_content import PageContent
+from pipelex.core.stuffs.search_result_content import SearchResultContent
from pipelex.core.stuffs.text_and_images_content import TextAndImagesContent
from pipelex.core.stuffs.text_content import TextContent
from pipelex.pipeline.job_metadata import JobMetadata
@@ -291,3 +295,25 @@ class ContentGeneratorDry(ContentGeneratorProtocol):
page_content.page_view = page_view_contents.pop(0)
return page_contents
+
+ @override
+ async def make_search_sourced_answer(
+ self,
+ search_assignment: SearchAssignment,
+ ) -> SearchResultContent:
+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.make_search_sourced_answer")
+ nb_sources = get_config().pipelex.dry_run_config.nb_list_items
+ doc_factory = DryRunFactory.make_dry_run_factory(DocumentContent)
+ mock_sources = [doc_factory.build() for _ in range(nb_sources)]
+ search_result_factory = DryRunFactory.make_dry_run_factory(SearchResultContent)
+ return search_result_factory.build(sources=mock_sources)
+
+ @override
+ async def make_search_structured(
+ self,
+ output_structure_class: type[BaseModelTypeVar],
+ search_assignment: SearchAssignment,
+ ) -> BaseModelTypeVar:
+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.make_search_structured")
+ structured_factory = DryRunFactory.make_dry_run_factory(output_structure_class)
+ return structured_factory.build()
diff --git a/pipelex/cogt/content_generation/content_generator_protocol.py b/pipelex/cogt/content_generation/content_generator_protocol.py
index a66b9e677..7a9f01e60 100644
--- a/pipelex/cogt/content_generation/content_generator_protocol.py
+++ b/pipelex/cogt/content_generation/content_generator_protocol.py
@@ -2,6 +2,7 @@ from collections.abc import Awaitable, Callable, Coroutine
from functools import wraps
from typing import Any, ParamSpec, Protocol, TypeVar
+from pipelex.cogt.content_generation.assignment_models import SearchAssignment
from pipelex.cogt.extract.extract_input import ExtractInput
from pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams
from pipelex.cogt.extract.extract_output import ExtractOutput
@@ -14,6 +15,7 @@ from pipelex.cogt.templating.template_category import TemplateCategory
from pipelex.cogt.templating.templating_style import TemplatingStyle
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.page_content import PageContent
+from pipelex.core.stuffs.search_result_content import SearchResultContent
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
@@ -125,3 +127,14 @@ class ContentGeneratorProtocol(Protocol):
extract_job_params: ExtractJobParams,
extract_job_config: ExtractJobConfig,
) -> Coroutine[Any, Any, list[PageContent]]: ...
+
+ def make_search_sourced_answer(
+ self,
+ search_assignment: SearchAssignment,
+ ) -> Coroutine[Any, Any, SearchResultContent]: ...
+
+ def make_search_structured(
+ self,
+ output_structure_class: type[BaseModelTypeVar],
+ search_assignment: SearchAssignment,
+ ) -> Coroutine[Any, Any, BaseModelTypeVar]: ...
diff --git a/pipelex/cogt/content_generation/search_generate.py b/pipelex/cogt/content_generation/search_generate.py
new file mode 100644
index 000000000..d61f88900
--- /dev/null
+++ b/pipelex/cogt/content_generation/search_generate.py
@@ -0,0 +1,76 @@
+"""Framework-agnostic web-search leaf, sibling of ``llm_generate``.
+
+These coroutines are the single place a ``PipeSearch`` step's provider spend happens. They take a
+serializable ``SearchAssignment`` / ``SearchObjectAssignment``, rebuild the ``SearchJob``, resolve the
+worker from the model handle, and run it. The direct ``ContentGenerator`` calls them inline; the Temporal
+``act_search_*`` activity calls them inside an activity so the result is recorded in workflow history and
+any failure is converted to a terminal ``ApplicationError`` (instead of running inline on the workflow
+loop, which left search failures hanging the submitter — see ``wip/`` brief).
+"""
+
+from typing import Any
+
+from pipelex.cogt.content_generation.assignment_models import SearchAssignment, SearchObjectAssignment
+from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedError
+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.search.search_job import SearchJob
+from pipelex.cogt.search.search_job_factory import SearchJobFactory
+from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract
+from pipelex.cogt.search.search_worker_factory import SearchWorkerFactory
+from pipelex.core.stuffs.search_result_content import SearchResultContent
+from pipelex.hub import get_model_deck
+
+
+def _guard_no_mock_inference(search_assignment: SearchAssignment) -> None:
+ # Web search has no leaf-level mock — under --mock-inference (run_mode stays LIVE) this live
+ # path runs, so fail loud rather than silently hit the real provider (use --dry-run instead).
+ if search_assignment.job_metadata.is_mock_inference:
+ error = MockInferenceUnsupportedError.for_operation("web search (PipeSearch)")
+ raise error
+
+
+def _make_search_worker(search_assignment: SearchAssignment) -> SearchWorkerAbstract:
+ model_deck = get_model_deck()
+ inference_model = model_deck.get_required_inference_model(
+ model_handle=search_assignment.search_setting.model,
+ model_type=ModelType.SEARCH,
+ )
+ return SearchWorkerFactory.make_search_worker(inference_model=inference_model)
+
+
+def _make_search_job(search_assignment: SearchAssignment) -> SearchJob:
+ return SearchJobFactory.make_search_job(
+ query=search_assignment.query,
+ search_setting=search_assignment.search_setting,
+ job_metadata=search_assignment.job_metadata,
+ include_domains=search_assignment.include_domains,
+ exclude_domains=search_assignment.exclude_domains,
+ from_date=search_assignment.from_date,
+ to_date=search_assignment.to_date,
+ )
+
+
+async def search_gen_sourced_answer(search_assignment: SearchAssignment) -> SearchResultContent:
+ _guard_no_mock_inference(search_assignment)
+ worker = _make_search_worker(search_assignment)
+ search_job = _make_search_job(search_assignment)
+ return await worker.search_sourced_answer(search_job=search_job)
+
+
+async def search_gen_structured(search_object_assignment: SearchObjectAssignment) -> dict[str, Any]:
+ """Run a structured search, returning the raw result dict.
+
+ The dict is re-validated against the original output structure class by the submitter
+ (``ContentGenerator`` / ``ContentGeneratorInWorkflow``) — that step is pure and deterministic,
+ which sidesteps shipping a dynamic output class across the Temporal boundary.
+ """
+ search_assignment = search_object_assignment.search_assignment
+ _guard_no_mock_inference(search_assignment)
+ worker = _make_search_worker(search_assignment)
+ search_job = _make_search_job(search_assignment)
+ output_class = SchemaToModelFactory.make_from_json_schema(
+ schema=search_object_assignment.output_class_schema,
+ class_name=search_object_assignment.output_class_name,
+ )
+ return await worker.search_structured(search_job=search_job, schema=output_class)
diff --git a/pipelex/pipe_operators/search/pipe_search.py b/pipelex/pipe_operators/search/pipe_search.py
index 7c79fba07..534726e32 100644
--- a/pipelex/pipe_operators/search/pipe_search.py
+++ b/pipelex/pipe_operators/search/pipe_search.py
@@ -3,23 +3,20 @@ from typing import TYPE_CHECKING, Any, Literal
from typing_extensions import override
from pipelex import log
-from pipelex.cogt.content_generation.dry_run_factory import DryRunFactory
-from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedError
+from pipelex.cogt.content_generation.assignment_models import SearchAssignment
+from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry
+from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol
from pipelex.cogt.exceptions import ModelChoiceNotFoundError
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_deck_check import check_search_choice_with_deck
-from pipelex.cogt.search.search_job_factory import SearchJobFactory
from pipelex.cogt.search.search_setting import SearchModelChoice, SearchSetting
-from pipelex.cogt.search.search_worker_factory import SearchWorkerFactory
from pipelex.cogt.templating.template_blueprint import TemplateBlueprint
from pipelex.cogt.templating.template_rendering import render_template
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecs
from pipelex.core.pipes.pipe_output import PipeOutput
-from pipelex.core.stuffs.document_content import DocumentContent
-from pipelex.core.stuffs.search_result_content import SearchResultContent
from pipelex.core.stuffs.stuff_factory import StuffFactory
-from pipelex.hub import get_model_deck
+from pipelex.hub import get_content_generator, get_model_deck
from pipelex.pipe_operators.pipe_operator import PipeOperator
from pipelex.pipe_run.pipe_run_params import PipeRunParams
from pipelex.pipeline.job_metadata import JobMetadata
@@ -82,13 +79,9 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):
working_memory: WorkingMemory,
pipe_run_params: PipeRunParams,
output_name: str | None = None,
+ content_generator: ContentGeneratorProtocol | None = None,
) -> PipeSearchOutput:
- # Web search has no leaf-level mock and (unlike LLM/img-gen/extract) no cogt content_generation
- # leaf — its provider spend happens right here. Under --mock-inference (run_mode stays LIVE) this
- # live path runs, so guard here: fail loud rather than silently hit the real search provider.
- if job_metadata.is_mock_inference:
- error = MockInferenceUnsupportedError.for_operation("web search (PipeSearch)")
- raise error
+ content_generator = content_generator or get_content_generator()
# 0. Log the search run
search_choice_desc = self.search_choice or "default"
@@ -106,7 +99,8 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):
search_choice: SearchModelChoice = self.search_choice or model_deck.search_choice_default
search_setting: SearchSetting = model_deck.get_search_setting(search_choice=search_choice)
- # 3. Resolve the model handle (waterfalls/aliases → actual provider/variant handle)
+ # 3. Resolve the model handle (waterfalls/aliases → actual provider/variant handle). Pin
+ # search_setting.model to the resolved handle so it doubles as the Temporal routing key.
inference_model = model_deck.get_required_inference_model(model_handle=search_setting.model, model_type=ModelType.SEARCH)
resolved_model_handle = inference_model.name
if resolved_model_handle != search_setting.model:
@@ -118,30 +112,31 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):
if self.max_results_override is not None:
search_setting = search_setting.model_copy(update={"max_results": self.max_results_override})
- # 5. Get search worker from factory
- worker = SearchWorkerFactory.make_search_worker(inference_model=inference_model)
-
- # 6. Create search job
- search_job = SearchJobFactory.make_search_job(
+ # 5. Build the serializable assignment and run the search behind the content-generation seam.
+ # The leaf goes through the same swappable generator as LLM/img-gen/extract: direct inline, a
+ # Temporal activity when in-workflow, or a dry mock. That makes search-on-Temporal replay-safe
+ # and lets its failures cross the workflow boundary as classified errors instead of hanging.
+ search_assignment = SearchAssignment(
+ job_metadata=job_metadata,
query=query_text,
search_setting=search_setting,
- job_metadata=job_metadata,
include_domains=self.include_domains,
exclude_domains=self.exclude_domains,
from_date=self.from_date,
to_date=self.to_date,
)
- # 7. Execute search based on output type
content: StuffContent
if not self.is_structured_output:
- content = await worker.search_sourced_answer(search_job=search_job)
+ content = await content_generator.make_search_sourced_answer(search_assignment=search_assignment)
else:
output_structure_class = self.output.concept.get_structure_class()
- result_dict = await worker.search_structured(search_job=search_job, schema=output_structure_class)
- content = output_structure_class.model_validate(result_dict)
+ content = await content_generator.make_search_structured(
+ output_structure_class=output_structure_class,
+ search_assignment=search_assignment,
+ )
- # 8. Create Stuff, set in working memory, and return output
+ # 6. Create Stuff, set in working memory, and return output
output_stuff = StuffFactory.make_stuff(
name=output_name,
concept=self.output.concept,
@@ -172,35 +167,14 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):
pipe_run_params: PipeRunParams,
output_name: str | None = None,
) -> PipeSearchOutput:
- content: StuffContent
- if not self.is_structured_output:
- doc_factory = DryRunFactory.make_dry_run_factory(DocumentContent)
- mock_sources = [doc_factory.build() for _ in range(3)]
- search_result_factory = DryRunFactory.make_dry_run_factory(SearchResultContent)
- content = search_result_factory.build(sources=mock_sources)
- else:
- output_structure_class = self.output.concept.get_structure_class()
- structured_factory = DryRunFactory.make_dry_run_factory(output_structure_class)
- content = structured_factory.build()
-
- output_stuff = StuffFactory.make_stuff(
- name=output_name,
- concept=self.output.concept,
- content=content,
- )
- working_memory.set_new_main_stuff(
- stuff=output_stuff,
- name=output_name,
- )
- execution_data_dict: dict[str, Any] = {
- "rendered_query": "mock",
- "resolved_model": "mock",
- "is_structured_output": self.is_structured_output,
- }
- self._register_execution_data(job_metadata, execution_data_dict)
- return PipeSearchOutput(
+ # Dry run reuses the live path with the dry content generator — identical to PipeLLM — so the
+ # search mock now lives behind the same seam (ContentGeneratorDry.make_search_*) as every leaf.
+ return await self._live_run_operator_pipe(
+ job_metadata=job_metadata,
working_memory=working_memory,
- pipeline_run_id=job_metadata.pipeline_run_id,
+ pipe_run_params=pipe_run_params,
+ output_name=output_name,
+ content_generator=ContentGeneratorDry(),
)
@override
diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.toml
index 5f416afff..0c26868d3 100644
--- a/pipelex/pipelex.toml
+++ b/pipelex/pipelex.toml
@@ -668,3 +668,15 @@ excluded_workflows = []
excluded_activities = []
disable_all_workflows = true
disable_all_activities = false
+
+[temporal.worker_scopes.scopes.runner-search]
+required_tasks_packs = []
+required_workflows = []
+required_activities = [
+ "act_search_gen_sourced_answer",
+ "act_search_gen_structured",
+]
+excluded_workflows = []
+excluded_activities = []
+disable_all_workflows = true
+disable_all_activities = false
diff --git a/pipelex/pipeline/bundle_validator.py b/pipelex/pipeline/bundle_validator.py
index 67a89302e..5211cb2b8 100644
--- a/pipelex/pipeline/bundle_validator.py
+++ b/pipelex/pipeline/bundle_validator.py
@@ -43,6 +43,7 @@ from pipelex.hub import (
get_library_manager,
get_pipe_library,
get_telemetry_manager,
+ scoped_pipe_router,
set_current_library,
)
from pipelex.libraries.pipe.exceptions import PipeNotFoundError
@@ -103,7 +104,12 @@ class BundleValidator:
def __init__(self) -> None:
# Locally-constructed, direct execution primitive — NOT get_pipe_run() (see module docstring).
# The ObserverNoOp keeps the sweep observation-silent; validation surfaces its own report.
- self._pipe_run: PipeRunProtocol = PipeRun(pipe_router=PipeRouter(observer=ObserverNoOp()))
+ # Keep the router instance (don't discard it inside PipeRun): the sweep installs it as the active
+ # router via scoped_pipe_router (see validate_pipes) so nested controller sub-pipes — which
+ # dispatch through get_pipe_router() — resolve THIS in-process router instead of the hub default.
+ # Mirrors runtime_bridge.bridge._run_direct.
+ self._pipe_router = PipeRouter(observer=ObserverNoOp())
+ self._pipe_run: PipeRunProtocol = PipeRun(pipe_router=self._pipe_router)
async def acquire_and_validate(
self,
@@ -235,10 +241,20 @@ class BundleValidator:
mock_inputs=True,
)
dry_run_pipeline_id = f"dry_run_{PipelineFactory.make_pipeline_run_id()}"
- for pipe in sweepable_pipes:
- results[pipe.pipe_ref] = await self._classify_pipe(
- pipe=pipe, library_id=library_id, execution_config=execution_config, dry_run_pipeline_id=dry_run_pipeline_id
- )
+ # Install the in-process router as the active router for the WHOLE sweep so nested controller
+ # sub-pipes — which dispatch through get_pipe_router() — resolve THIS router instead of falling
+ # back to the hub default. Under a Temporal-enabled hub the default is the Temporal router, so
+ # without the scope a controller (a PipeBatch/PipeParallel fan-out, or any PipeSequence step)
+ # would leak its nested pipes to Temporal — turning a no-cost in-process dry run into real
+ # top-level workflow dispatches (HTTP 422). Mirrors runtime_bridge.bridge._run_direct. The
+ # scope is contextvar-based, so concurrent /validate sweeps don't cross-contaminate, and the
+ # asyncio tasks a batch fan-out spawns copy the context at creation (inside this scope) so they
+ # inherit the override too.
+ with scoped_pipe_router(self._pipe_router):
+ for pipe in sweepable_pipes:
+ results[pipe.pipe_ref] = await self._classify_pipe(
+ pipe=pipe, library_id=library_id, execution_config=execution_config, dry_run_pipeline_id=dry_run_pipeline_id
+ )
# 5. Aggregate + report.
return self._aggregate(results=results, start_time=start_time)
diff --git a/pipelex/temporal/tasks.py b/pipelex/temporal/tasks.py
index 3aeabb6fa..795f71e03 100644
--- a/pipelex/temporal/tasks.py
+++ b/pipelex/temporal/tasks.py
@@ -6,6 +6,7 @@ from pipelex.temporal.tprl_content_generation.act_img_gen_generate import act_im
from pipelex.temporal.tprl_content_generation.act_jinja2_generate import act_jinja2_gen_text
from pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_object, act_llm_gen_object_list, act_llm_gen_text
from pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views
+from pipelex.temporal.tprl_content_generation.act_search_generate import act_search_gen_sourced_answer, act_search_gen_structured
from pipelex.temporal.tprl_pipe.act_assemble_tracing import act_assemble_tracing
from pipelex.temporal.tprl_pipe.act_deliver import act_deliver
from pipelex.temporal.tprl_pipe.act_flush_trace_events import act_flush_trace_events
@@ -31,6 +32,8 @@ class Tasks:
act_jinja2_gen_text,
act_extract_gen_extract_pages,
act_render_page_views,
+ act_search_gen_sourced_answer,
+ act_search_gen_structured,
],
),
PackName.PIPE: TaskPack(
diff --git a/pipelex/temporal/tprl_content_generation/act_search_generate.py b/pipelex/temporal/tprl_content_generation/act_search_generate.py
new file mode 100644
index 000000000..5df84925c
--- /dev/null
+++ b/pipelex/temporal/tprl_content_generation/act_search_generate.py
@@ -0,0 +1,23 @@
+from typing import Any
+
+from temporalio import activity
+
+from pipelex import log
+from pipelex.cogt.content_generation.assignment_models import SearchAssignment, SearchObjectAssignment
+from pipelex.cogt.content_generation.search_generate import search_gen_sourced_answer, search_gen_structured
+from pipelex.core.stuffs.search_result_content import SearchResultContent
+from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors
+
+
+@convert_pipelex_errors
+async def act_search_gen_sourced_answer(search_assignment: SearchAssignment) -> SearchResultContent:
+ log.dev("act_search_gen_sourced_answer")
+ return await search_gen_sourced_answer(search_assignment=search_assignment)
+
+
+@convert_pipelex_errors
+async def act_search_gen_structured(search_object_assignment: SearchObjectAssignment) -> dict[str, Any]:
+ log.dev("act_search_gen_structured")
+ return await search_gen_structured(search_object_assignment=search_object_assignment)
diff --git a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py b/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py
index 88b30529d..01283983f 100644
--- a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py
+++ b/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py
@@ -12,6 +12,8 @@ from pipelex.cogt.content_generation.assignment_models import (
LLMAssignment,
ObjectAssignment,
RenderPageViewsAssignment,
+ SearchAssignment,
+ SearchObjectAssignment,
TemplatingAssignment,
)
from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata
@@ -30,6 +32,7 @@ from pipelex.cogt.templating.templating_style import TemplatingStyle
from pipelex.config import get_config
from pipelex.core.stuffs.image_content import ImageContent
from pipelex.core.stuffs.page_content import PageContent
+from pipelex.core.stuffs.search_result_content import SearchResultContent
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.temporal.exceptions import ContentGenerationError
from pipelex.temporal.tprl.observability import build_activity_summary
@@ -43,6 +46,7 @@ from pipelex.temporal.tprl_content_generation.act_llm_generate import (
act_llm_gen_text,
)
from pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views
+from pipelex.temporal.tprl_content_generation.act_search_generate import act_search_gen_sourced_answer, act_search_gen_structured
from pipelex.tools.typing.pydantic_utils import BaseModelTypeVar
@@ -477,3 +481,77 @@ class ContentGeneratorInWorkflow(ContentGeneratorProtocol):
page_content.page_view = page_view_contents.pop(0)
return page_contents
+
+ @override
+ async def make_search_sourced_answer(
+ self,
+ search_assignment: SearchAssignment,
+ ) -> SearchResultContent:
+ worker_config = get_config().temporal.worker_config
+ dispatch_kwargs = worker_config.resolve_dispatch(
+ activity_name=act_search_gen_sourced_answer.__name__,
+ routing_key=search_assignment.search_handle,
+ queue_options_by_queue=get_config().temporal.queue_options,
+ is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,
+ ).to_execute_kwargs()
+ try:
+ search_result: SearchResultContent = await workflow.execute_activity(
+ act_search_gen_sourced_answer,
+ arg=search_assignment,
+ summary=build_activity_summary(
+ "Search sourced answer", search_assignment.job_metadata, extras={"model": search_assignment.search_handle}
+ ),
+ **dispatch_kwargs,
+ )
+ except ActivityError as exc:
+ log.error(f"ActivityError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ log.verbose(f"ContentGeneratorInWorkflow generated search result: {search_result}")
+ return search_result
+
+ @override
+ async def make_search_structured(
+ self,
+ output_structure_class: type[BaseModelTypeVar],
+ search_assignment: SearchAssignment,
+ ) -> BaseModelTypeVar:
+ worker_config = get_config().temporal.worker_config
+ search_object_assignment = SearchObjectAssignment.make_for_class(
+ output_class=output_structure_class,
+ search_assignment=search_assignment,
+ )
+ dispatch_kwargs = worker_config.resolve_dispatch(
+ activity_name=act_search_gen_structured.__name__,
+ routing_key=search_assignment.search_handle,
+ queue_options_by_queue=get_config().temporal.queue_options,
+ is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,
+ ).to_execute_kwargs()
+ try:
+ result_dict: dict[str, Any] = await workflow.execute_activity(
+ act_search_gen_structured,
+ arg=search_object_assignment,
+ summary=build_activity_summary(
+ "Search structured", search_assignment.job_metadata, extras={"class": output_structure_class.__name__}
+ ),
+ **dispatch_kwargs,
+ )
+ except ActivityError as exc:
+ log.error(f"ActivityError caused by: {exc.cause}")
+ if isinstance(exc.cause, ApplicationError):
+ raise TemporalError.from_app_error(exc=exc.cause) from exc
+ raise
+ # Re-validate on the submitter (workflow) side against the original class — pure and
+ # deterministic, so the dynamic output class never has to cross the activity boundary. The
+ # activity returns the provider's raw dict unvalidated (e.g. the gateway worker returns
+ # json.loads(...)), so a malformed structured response raises a bare ValidationError here in
+ # workflow code. Left raw it is neither WorkflowExecutionError nor PipelexError, so Temporal
+ # treats it as a workflow-task failure and retries forever, hanging the submitter — the exact
+ # failure mode this seam exists to prevent. Convert it to a terminal ContentGenerationError (a
+ # PipelexError) so the workflow fails and surfaces as an ErrorReport, matching the direct path.
+ try:
+ return output_structure_class.model_validate(result_dict)
+ except ValidationError as exc:
+ msg = f"Structured search result failed validation against {output_structure_class.__name__}: {exc}"
+ raise ContentGenerationError(msg) from exc
diff --git a/tests/integration/pipelex/error_handling/test_data.py b/tests/integration/pipelex/error_handling/test_data.py
index 486017135..c9e5f0421 100644
--- a/tests/integration/pipelex/error_handling/test_data.py
+++ b/tests/integration/pipelex/error_handling/test_data.py
@@ -17,7 +17,7 @@ drops a classification field, that arm fails.
from typing import ClassVar
-from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError
+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError, SearchJobFailureError
from pipelex.cogt.inference.error_classification import UserAction, UserActionKind
@@ -60,3 +60,48 @@ class ErrorReportParityTestData:
error.model_handle = cls.FAILURE_MODEL
error.backend_name = cls.FAILURE_PROVIDER
return error
+
+
+class SearchErrorReportParityTestData:
+ """Constants driving both arms of the local / Temporal ``ErrorReport`` parity test for web search.
+
+ The search counterpart of :class:`ErrorReportParityTestData`. A single deliberately-failing
+ ``PipeSearch`` leaf is the source of truth for both arms: the local arm runs the ``native_search``
+ pipe through the direct ``ContentGenerator`` with ``make_search_sourced_answer`` mocked to fail; the
+ Temporal arm runs the same pipe through ``WfPipeRouter`` with the activity-side ``search_gen_sourced_answer``
+ mocked to fail. Both assert the same ``EXPECTED_*`` classification, so parity holds by construction.
+ """
+
+ # A single-step PipeSearch bundle over the native SearchResult concept — no inputs, no dynamic classes.
+ BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate/native_search.mthds"
+ PIPE_CODE: ClassVar[str] = "native_search"
+
+ # The worker-side failure injected into the search call. CONFIGURATION is non-retryable, so the
+ # Temporal activity is not retried and the workflow fails on the first attempt — fast and deterministic.
+ FAILURE_MESSAGE: ClassVar[str] = "Search provider rejected the request: connection error"
+ FAILURE_MODEL: ClassVar[str] = "linkup/standard"
+ FAILURE_PROVIDER: ClassVar[str] = "linkup"
+ FAILURE_CATEGORY: ClassVar[InferenceErrorCategory] = InferenceErrorCategory.CONFIGURATION
+ FAILURE_USER_ACTION_DETAIL: ClassVar[str] = "Verify the API key configured for this search provider."
+
+ # The classification both arms must surface on the recovered ErrorReport.
+ EXPECTED_RETRYABLE: ClassVar[bool] = False
+ EXPECTED_USER_ACTION_KIND: ClassVar[UserActionKind] = UserActionKind.CHECK_CREDENTIALS
+
+ @classmethod
+ def make_failing_search_error(cls) -> SearchJobFailureError:
+ """Build a fresh classified ``SearchJobFailureError``.
+
+ Each arm builds its own instance — an exception carries traceback state, so a single shared
+ instance must not be reused across runs.
+ """
+ error = SearchJobFailureError(
+ cls.FAILURE_MESSAGE,
+ error_category=cls.FAILURE_CATEGORY,
+ user_action=UserAction(kind=cls.EXPECTED_USER_ACTION_KIND, detail=cls.FAILURE_USER_ACTION_DETAIL),
+ )
+ # model_handle / backend_name are declared on CogtError; a real search worker fills them at its
+ # public-method chokepoint. This test mocks above the worker, so set them directly.
+ error.model_handle = cls.FAILURE_MODEL
+ error.backend_name = cls.FAILURE_PROVIDER
+ return error
diff --git a/tests/integration/pipelex/error_handling/test_search_error_report_local_full_chain.py b/tests/integration/pipelex/error_handling/test_search_error_report_local_full_chain.py
new file mode 100644
index 000000000..b4994a67b
--- /dev/null
+++ b/tests/integration/pipelex/error_handling/test_search_error_report_local_full_chain.py
@@ -0,0 +1,63 @@
+"""Local arm of the local / Temporal ``ErrorReport`` parity pair for web search.
+
+Runs the ``native_search`` pipe through the local (non-Temporal) ``PipeRouter`` with the search call
+mocked to fail, and asserts the resulting ``ErrorReport`` carries the full classification —
+``error_category`` / ``retryable`` / ``model`` / ``provider`` / ``user_action``.
+
+This is the baseline the Temporal full-chain test
+(``tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.py``) must match.
+Both arms assert the same ``SearchErrorReportParityTestData`` constants, so local / Temporal parity holds
+by construction. The search operator does not wrap the leaf error (unlike PipeLLM), so a raw
+``SearchJobFailureError`` (a ``CogtError``) propagates and the ``PipeRouter`` re-raises it as-is.
+"""
+
+from collections.abc import Generator
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cogt.content_generation.content_generator import ContentGenerator
+from pipelex.cogt.exceptions import SearchJobFailureError
+from pipelex.hub import get_pipe_router
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from tests.integration.pipelex.error_handling.test_data import SearchErrorReportParityTestData
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+
+
[email protected](loop_scope="class")
+class TestSearchErrorReportLocalFullChain:
+ """A failing search pipe run locally yields a fully classified ``ErrorReport``."""
+
+ @pytest.fixture
+ def failing_search_pipe_job_local(self) -> Generator[PipeJob, None, None]:
+ """A PipeJob for the search pipe, in LIVE mode so the search call actually fires."""
+ yield from pipe_job_from_bundle(
+ bundle_file=SearchErrorReportParityTestData.BUNDLE_FILE,
+ pipe_code=SearchErrorReportParityTestData.PIPE_CODE,
+ pipe_run_mode=PipeRunMode.LIVE,
+ )
+
+ async def test_error_report_from_local_execution(
+ self,
+ mocker: MockerFixture,
+ failing_search_pipe_job_local: PipeJob,
+ ) -> None:
+ """Local execution surfaces the worker failure as a classified ``SearchJobFailureError``."""
+ mocker.patch.object(
+ ContentGenerator,
+ "make_search_sourced_answer",
+ side_effect=SearchErrorReportParityTestData.make_failing_search_error(),
+ )
+
+ with pytest.raises(SearchJobFailureError) as exc_info:
+ await get_pipe_router().run(pipe_job=failing_search_pipe_job_local)
+
+ # The classification fields — the parity target.
+ report = exc_info.value.to_error_report()
+ assert report.error_category == SearchErrorReportParityTestData.FAILURE_CATEGORY
+ assert report.retryable == SearchErrorReportParityTestData.EXPECTED_RETRYABLE
+ assert report.model == SearchErrorReportParityTestData.FAILURE_MODEL
+ assert report.provider == SearchErrorReportParityTestData.FAILURE_PROVIDER
+ assert report.user_action is not None
+ assert report.user_action.kind == SearchErrorReportParityTestData.EXPECTED_USER_ACTION_KIND
diff --git a/tests/integration/pipelex/pipeline/test_bundle_validator.py b/tests/integration/pipelex/pipeline/test_bundle_validator.py
index e8b8e4dd0..5b6ac6f45 100644
--- a/tests/integration/pipelex/pipeline/test_bundle_validator.py
+++ b/tests/integration/pipelex/pipeline/test_bundle_validator.py
@@ -14,18 +14,49 @@ from typing import Any
import pytest
from pytest_mock import MockerFixture
+from typing_extensions import override
+from pipelex.core.pipes.pipe_output import PipeOutput
from pipelex.hub import (
clear_current_library,
get_library_manager,
+ get_pipelex_hub,
get_required_pipe,
get_telemetry_manager,
)
+from pipelex.observer.observer_protocol import ObserverNoOp
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_router import PipeRouter
from pipelex.pipeline import bundle_validator
from pipelex.pipeline.bundle_validator import BundleValidator, DryRunStatus
from pipelex.pipeline.execution_seams import acquire_library
from pipelex.system.telemetry.events import EventName
+
+class _RaisingPipeRouter(PipeRouter):
+ """Hub-default stand-in for the Temporal router: trips if a sweep resolves the hub default.
+
+ A validation sweep is meant to run fully in-process. Nested controller sub-pipes dispatch through
+ ``get_pipe_router()``; if the sweep fails to scope its own in-process router, that resolves the hub
+ default — the Temporal router in a Temporal-enabled API process — and leaks the dry run to Temporal.
+ Installing this as the hub default makes that leak fail loudly instead of silently round-tripping.
+ """
+
+ def __init__(self) -> None:
+ super().__init__(observer=ObserverNoOp())
+ self.run_called = False
+
+ @override
+ async def _run_pipe_job(self, pipe_job: PipeJob) -> PipeOutput:
+ self.run_called = True
+ _ = pipe_job # required by the override signature; unused — the leak is detected by run_called
+ msg = (
+ "Hub-default pipe router was invoked during a validation sweep — a nested controller leaked "
+ "out of the scoped in-process router (the Temporal /validate leak regressed)."
+ )
+ raise AssertionError(msg)
+
+
_BV_DOMAIN = "bundle_validator_test"
_BV_MTHDS = f"""
domain = "{_BV_DOMAIN}"
@@ -78,6 +109,39 @@ branches = [
"""
+# A bundle whose top-level pipe is a STANDALONE PipeBatch — the shape that 422'd the Temporal /validate
+# route. Swept directly, PipeBatch fans out (mock_inputs mints a multi-item list) and each branch
+# dispatches through get_pipe_router(); without the sweep scoping its in-process router those branches
+# resolve the hub default (the Temporal router) and fire concurrent top-level workflow dispatches.
+_BV_BATCH_DOMAIN = "bundle_validator_batch"
+_BV_BATCH_MTHDS = f"""
+domain = "{_BV_BATCH_DOMAIN}"
+description = "Bundle with a standalone PipeBatch — router-scoping regression"
+
+[concept.Item]
+description = "An item to summarize"
+
+[concept.Summary]
+description = "A summary of an item"
+
+[pipe.summarize_item]
+type = "PipeLLM"
+description = "Summarize a single item"
+inputs = {{ item = "Item" }}
+output = "Summary"
+prompt = "Summarize $item"
+
+[pipe.summarize_items]
+type = "PipeBatch"
+description = "Standalone batch fanning out over items"
+inputs = {{ items = "Item[]" }}
+output = "Summary[]"
+branch_pipe_code = "summarize_item"
+input_list_name = "items"
+input_item_name = "item"
+"""
+
+
@pytest.mark.asyncio(loop_scope="class")
class TestBundleValidatorIntegration:
async def test_validate_pipes_success_real_path(self) -> None:
@@ -191,6 +255,32 @@ class TestBundleValidatorIntegration:
library_manager.teardown(library_id=library_id)
clear_current_library()
+ async def test_standalone_batch_sweep_scopes_in_process_router(self) -> None:
+ # Regression for the Temporal /validate leak: a standalone PipeBatch fans out its branches via
+ # get_pipe_router(). The sweep must scope its own in-process router for the whole sweep so those
+ # branches resolve it — NOT the hub default, which in a Temporal-enabled API process is the
+ # Temporal router (the leak that produced HTTP 422). We install a hub default that raises if its
+ # run is ever reached, then assert the batch (and its branch leaf) classify SUCCESS without the
+ # hub default ever being touched.
+ hub = get_pipelex_hub()
+ original_router = hub.get_required_pipe_router()
+ sentinel = _RaisingPipeRouter()
+ library_manager = get_library_manager()
+ library_id = "bv_batch_scope_lib"
+ acquire_library(library_id=library_id, mthds_contents=[_BV_BATCH_MTHDS])
+ hub.set_pipe_router(pipe_router=sentinel)
+ try:
+ batch_pipe = get_required_pipe(pipe_code=f"{_BV_BATCH_DOMAIN}.summarize_items")
+ leaf_pipe = get_required_pipe(pipe_code=f"{_BV_BATCH_DOMAIN}.summarize_item")
+ results = await BundleValidator().validate_pipes([batch_pipe, leaf_pipe], library_id=library_id)
+ assert results[f"{_BV_BATCH_DOMAIN}.summarize_items"].status.is_success
+ assert results[f"{_BV_BATCH_DOMAIN}.summarize_item"].status.is_success
+ assert sentinel.run_called is False, "Nested batch dispatch leaked to the hub default router"
+ finally:
+ hub.set_pipe_router(pipe_router=original_router)
+ library_manager.teardown(library_id=library_id)
+ clear_current_library()
+
async def test_one_pipe_dry_run_event_and_no_stray_pipeline_events(self, mocker: MockerFixture) -> None:
library_manager = get_library_manager()
library_id = "bv_telemetry_lib"
diff --git a/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.py b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.py
index dd5c750df..583895e68 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.py
@@ -1,12 +1,13 @@
-"""Integration test for the ``is_mock_inference`` hard guard in PipeSearch.
-
-Web search has no leaf-level mock, and unlike the LLM/img-gen/extract operators it has no
-``cogt/content_generation`` leaf — its provider spend happens directly in
-``PipeSearch._live_run_operator_pipe``. Under ``--mock-inference`` (``run_mode=LIVE`` +
-``JobMetadata.is_mock_inference``) the operator takes that live path, so the guard lives there and must
-fail loud before the search worker is built. ``MockInferenceUnsupportedError`` is a plain
-``PipelexError`` (not a model-availability error), so ``PipeOperator._live_run_pipe`` does not re-wrap
-it — it propagates unchanged through the router.
+"""Integration test for the ``is_mock_inference`` hard guard for web search.
+
+Web search has no leaf-level mock. It now goes through the same ``cogt/content_generation`` seam as
+LLM/img-gen/extract: ``PipeSearch`` calls the content generator, whose direct impl runs the
+framework-agnostic ``search_generate`` leaf. Under ``--mock-inference`` (``run_mode=LIVE`` +
+``JobMetadata.is_mock_inference``) the live path runs, so the guard lives at the top of that leaf
+(``search_gen_sourced_answer``) and must fail loud before the search worker is built.
+``MockInferenceUnsupportedError`` is a plain ``PipelexError`` (not a model-availability error and not a
+``CogtError``), so neither ``PipeOperator._live_run_pipe`` nor the router re-wraps it — it propagates
+unchanged.
"""
from typing import Callable
@@ -37,7 +38,7 @@ class TestPipeSearchMockInferenceGuard:
"""A LIVE PipeSearch run with is_mock_inference=True raises the guard before any provider call."""
load_empty_library()
- worker_factory_spy = mocker.patch("pipelex.pipe_operators.search.pipe_search.SearchWorkerFactory.make_search_worker")
+ worker_factory_spy = mocker.patch("pipelex.cogt.content_generation.search_generate.SearchWorkerFactory.make_search_worker")
pipe = PipeFactory[PipeSearch].make_from_blueprint(
domain_code="generic",
diff --git a/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_model_not_found_reroute.py b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_model_not_found_reroute.py
index d37d79ca5..46d2db530 100644
--- a/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_model_not_found_reroute.py
+++ b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_model_not_found_reroute.py
@@ -40,7 +40,7 @@ class TestPipeSearchModelNotFoundReroute:
fake_worker = mocker.MagicMock()
fake_worker.search_sourced_answer = mocker.AsyncMock(side_effect=not_found_error)
mocker.patch(
- "pipelex.pipe_operators.search.pipe_search.SearchWorkerFactory.make_search_worker",
+ "pipelex.cogt.content_generation.search_generate.SearchWorkerFactory.make_search_worker",
return_value=fake_worker,
)
diff --git a/tests/integration/pipelex/temporal/library_crate/native_search.mthds b/tests/integration/pipelex/temporal/library_crate/native_search.mthds
new file mode 100644
index 000000000..0d9f97bde
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/native_search.mthds
@@ -0,0 +1,8 @@
+domain = "native_search_test"
+description = "Single PipeSearch step over the native SearchResult concept (no dynamic classes)"
+
+[pipe.native_search]
+type = "PipeSearch"
+description = "Search the web for a fixed query"
+output = "SearchResult"
+prompt = "What are the latest developments in AI safety research?"
diff --git a/tests/integration/pipelex/temporal/library_crate/structured_search.mthds b/tests/integration/pipelex/temporal/library_crate/structured_search.mthds
new file mode 100644
index 000000000..c0243003f
--- /dev/null
+++ b/tests/integration/pipelex/temporal/library_crate/structured_search.mthds
@@ -0,0 +1,16 @@
+domain = "structured_search_test"
+description = "Single PipeSearch step over a custom structured TopicSummary concept (exercises act_search_gen_structured across the Temporal boundary)"
+
+[concept.TopicSummary]
+description = "A structured summary of web-search findings for a topic"
+
+[concept.TopicSummary.structure]
+title = { type = "text", description = "Topic title", required = true }
+summary = { type = "text", description = "Summary of the findings", required = true }
+key_points = { type = "list", item_type = "text", description = "Key points distilled from the search", required = true }
+
+[pipe.structured_search]
+type = "PipeSearch"
+description = "Search the web and return a structured TopicSummary"
+output = "TopicSummary"
+prompt = "Summarize the latest developments in AI safety research."
diff --git a/tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py b/tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py
new file mode 100644
index 000000000..5d71a2d75
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py
@@ -0,0 +1,69 @@
+"""Mode-1 Temporal guard: the BundleValidator dry-run sweep must NOT dispatch to Temporal.
+
+Deployment-faithful companion to the in-process sentinel regression test in
+``tests/integration/pipelex/pipeline/test_bundle_validator.py``
+(``test_standalone_batch_sweep_scopes_in_process_router``). Where the sentinel installs a *raising
+stand-in* as the hub default, this test resolves the **real** ``TemporalPipeRouter`` as the hub
+default — wired by this suite's ``boot_temporal`` fixture exactly as a Temporal-enabled API process
+wires it in ``pipelex.pipelex``. It therefore proves the config -> hub-default *wiring* the sentinel
+deliberately bypasses, on top of the same contract.
+
+The bug it guards: a standalone ``PipeBatch`` swept directly fans out over a mock list and
+dispatches each branch through ``get_pipe_router()``. Without the sweep scoping its own in-process
+router, those branches resolve the hub default — the ``TemporalPipeRouter`` — and fire concurrent
+top-level workflow dispatches (``WorkflowExecutor.execute_workflow``), which produced the production
+HTTP 422.
+
+The contract asserted: the whole sweep stays in-process, so the submitter-side top-level dispatch
+seam (``WorkflowExecutor.execute_workflow``) is never reached. Spying on that seam is the
+deterministic analogue of the Mode-2 "worker received no dispatch" check. No live Temporal server is
+needed precisely because GREEN means no dispatch ever happens; if the fix regressed, the spy fires
+and the test goes RED.
+"""
+
+from pathlib import Path
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.hub import clear_current_library, get_library_manager, get_pipelex_hub, get_required_pipe
+from pipelex.pipeline.bundle_validator import BundleValidator
+from pipelex.pipeline.execution_seams import acquire_library
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor
+from pipelex.temporal.tprl_pipe.temporal_pipe_router import TemporalPipeRouter
+from tests.integration.pipelex.temporal.test_data import PipeBatchTemporalTestData
+
+# The standalone ``type = "PipeBatch"`` pipe in temporal_batch.mthds — swept directly it fans out,
+# which is the exact shape that turned the leak fatal (concurrent same-id top-level dispatches).
+_STANDALONE_BATCH_PIPE_REF = f"{PipeBatchTemporalTestData.DOMAIN}.batch_temporal_describe_topics"
+
+
[email protected](loop_scope="class")
+class TestValidateSweepStaysInProcess:
+ async def test_standalone_batch_sweep_never_dispatches_to_temporal(self, mocker: MockerFixture) -> None:
+ # Precondition (the wiring this test exists to prove): under boot_temporal the hub default
+ # router IS the real TemporalPipeRouter — the same router a Temporal-enabled API process runs.
+ hub = get_pipelex_hub()
+ assert isinstance(hub.get_required_pipe_router(), TemporalPipeRouter), (
+ "Temporal suite precondition: the hub default router must be the real TemporalPipeRouter"
+ )
+
+ # Spy on the submitter-side top-level dispatch seam. A standalone-batch sweep that leaked to
+ # Temporal would reach this exact method (TemporalPipeRouter._run_pipe_job top-level branch ->
+ # WorkflowExecutorFactory.create_executor(...).execute_workflow(...)). With the fix the sweep
+ # stays in-process and never touches it.
+ execute_workflow_spy = mocker.spy(WorkflowExecutor, "execute_workflow")
+
+ library_manager = get_library_manager()
+ library_id = "validate_sweep_temporal_leak_lib"
+ mthds_content = Path(PipeBatchTemporalTestData.BUNDLE_FILE).read_text(encoding="utf-8")
+ acquire_library(library_id=library_id, mthds_contents=[mthds_content])
+ try:
+ batch_pipe = get_required_pipe(pipe_code=_STANDALONE_BATCH_PIPE_REF)
+ results = await BundleValidator().validate_pipes([batch_pipe], library_id=library_id)
+ assert results[_STANDALONE_BATCH_PIPE_REF].status.is_success
+ execute_workflow_spy.assert_not_called()
+ finally:
+ library_manager.teardown(library_id=library_id)
+ clear_current_library()
diff --git a/tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.py b/tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.py
new file mode 100644
index 000000000..0c3bd0416
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.py
@@ -0,0 +1,114 @@
+"""Temporal arm of the local / Temporal ``ErrorReport`` parity pair for web search.
+
+Runs the ``native_search`` pipe through a real ``WfPipeRouter`` workflow on the in-process Temporal
+server, submitted via ``WorkflowExecutor.execute_workflow``, with the activity-side search call mocked to
+fail. This is the regression test for the original bug: before ``PipeSearch`` routed its leaf through a
+Temporal activity, a search failure raised a raw ``SearchJobFailureError`` inside the workflow, which was
+neither an ``ActivityError`` nor a ``WorkflowExecutionError`` — so Temporal retried the workflow task
+forever and the submitter hung. Now the failure crosses the activity → workflow → submitter boundary as a
+terminal ``WorkflowExecutionError`` carrying the structured ``ErrorReport``.
+
+The local baseline is the separate module
+``tests/integration/pipelex/error_handling/test_search_error_report_local_full_chain.py``; both arms assert
+the same ``SearchErrorReportParityTestData`` constants, so local / Temporal parity holds by construction.
+"""
+
+import uuid
+from collections.abc import Generator
+from typing import TYPE_CHECKING
+
+import pytest
+from pytest_mock import MockerFixture
+from temporalio.client import Client as TemporalClient
+from temporalio.common import RetryPolicy
+
+from pipelex.config import get_config
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.exceptions import WorkflowExecutionError
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.error_handling.test_data import SearchErrorReportParityTestData
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+
+ACT_SEARCH_SOURCED_ANSWER_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_sourced_answer"
+
+
[email protected](loop_scope="class")
+class TestWorkflowSearchErrorReportFullChain:
+ """A failing search pipe's structured ``ErrorReport`` survives the Temporal workflow → submitter boundary."""
+
+ @pytest.fixture
+ def temporal_enabled(self) -> Generator[None, None, None]:
+ """Enable ``temporal.is_enabled`` for the test — ``WorkflowExecutor.temporal_client()`` raises when off."""
+ config = get_config()
+ previous = config.temporal.is_enabled
+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})
+ yield
+ config.temporal = config.temporal.model_copy(update={"is_enabled": previous})
+
+ @pytest.fixture
+ def failing_search_pipe_job(self) -> Generator[PipeJob, None, None]:
+ """A PipeJob for the search pipe, in LIVE mode so the workflow dispatches the search activity.
+
+ DRY mode short-circuits the ``act_search_*`` dispatch (the dry content generator reports inline
+ inside the workflow), so the activity — and the mock — never fire.
+ """
+ yield from pipe_job_from_bundle(
+ bundle_file=SearchErrorReportParityTestData.BUNDLE_FILE,
+ pipe_code=SearchErrorReportParityTestData.PIPE_CODE,
+ pipe_run_mode=PipeRunMode.LIVE,
+ )
+
+ async def test_search_error_report_survives_temporal_boundary(
+ self,
+ temporal_client: TemporalClient,
+ mocker: MockerFixture,
+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration
+ failing_search_pipe_job: PipeJob,
+ ) -> None:
+ """A failing search run through Temporal surfaces a fully classified ``WorkflowExecutionError`` (no hang)."""
+ # Mock the search core inside the real activity (in-process unsandboxed worker, same process).
+ # @convert_pipelex_errors packs the ErrorReport into ApplicationError.details.
+ mocker.patch(
+ ACT_SEARCH_SOURCED_ANSWER_TARGET,
+ new=mocker.AsyncMock(side_effect=SearchErrorReportParityTestData.make_failing_search_error()),
+ )
+
+ task_queue = f"q_search_err_{uuid.uuid4().hex[:8]}"
+ workflow_id = f"wf_search_err_{uuid.uuid4().hex[:8]}"
+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(
+ temporal_client=temporal_client,
+ task_queue=task_queue,
+ retry_policy=RetryPolicy(maximum_attempts=1),
+ )
+
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ with pytest.raises(WorkflowExecutionError) as exc_info:
+ await executor.execute_workflow(
+ workflow_class=WfPipeRouter,
+ workflow_arg=failing_search_pipe_job,
+ workflow_id=workflow_id,
+ )
+
+ error = exc_info.value
+ # The structured report was recovered across the workflow → submitter boundary.
+ assert error.error_report is not None
+ report = error.to_error_report()
+
+ # The real failure message survived — not the generic "Failed to execute workflow ...".
+ assert SearchErrorReportParityTestData.FAILURE_MESSAGE in report.message
+ assert "Failed to execute workflow" not in report.message
+
+ # The full classification survived the Temporal serialization round-trip.
+ assert report.error_category == SearchErrorReportParityTestData.FAILURE_CATEGORY
+ assert report.retryable == SearchErrorReportParityTestData.EXPECTED_RETRYABLE
+ assert report.model == SearchErrorReportParityTestData.FAILURE_MODEL
+ assert report.provider == SearchErrorReportParityTestData.FAILURE_PROVIDER
+ assert report.user_action is not None
+ assert report.user_action.kind == SearchErrorReportParityTestData.EXPECTED_USER_ACTION_KIND
diff --git a/tests/integration/pipelex/temporal/test_workflow_search_structured_roundtrip.py b/tests/integration/pipelex/temporal/test_workflow_search_structured_roundtrip.py
new file mode 100644
index 000000000..48590fc7b
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_workflow_search_structured_roundtrip.py
@@ -0,0 +1,114 @@
+"""Structured-output counterpart to the search success round-trip test.
+
+Runs a ``PipeSearch`` step whose output is a *custom structured concept* (``TopicSummary``) through a real
+``WfPipeRouter`` workflow on the in-process Temporal server, with the activity-side search call mocked to
+*succeed*. This exercises the ``act_search_gen_structured`` path — the riskier of the two search activities:
+the operator ships the output structure's JSON schema (not the live class) across the boundary via
+``SearchObjectAssignment``, the activity reconstructs a throwaway class to drive the provider and returns the
+**raw result dict**, and the submitter re-validates that dict against the original output class
+(``output_structure_class.model_validate(result_dict)``) — a pure, deterministic step that keeps the dynamic
+class on the submitter side. This test locks in that whole round-trip; the sourced-answer sibling
+(``test_workflow_search_success_roundtrip.py``) covers the non-structured path. The mock firing also proves the
+workflow dispatched ``act_search_gen_structured`` rather than running the leaf inline on the workflow loop.
+"""
+
+import uuid
+from collections.abc import Generator
+from typing import TYPE_CHECKING
+
+import pytest
+from pytest_mock import MockerFixture
+from temporalio.client import Client as TemporalClient
+from temporalio.common import RetryPolicy
+
+from pipelex.config import get_config
+from pipelex.core.stuffs.structured_content import StructuredContent
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+
+ACT_SEARCH_STRUCTURED_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_structured"
+
+BUNDLE_FILE = "tests/integration/pipelex/temporal/library_crate/structured_search.mthds"
+PIPE_CODE = "structured_search"
+
+# Flat, schema-shaped dict — matches what the real worker returns (LinkupSearchWorker._search_structured
+# returns response.model_dump()), which the submitter re-validates against the TopicSummary structure class.
+EXPECTED_TITLE = "AI safety in 2026"
+EXPECTED_SUMMARY = "Interpretability and scalable oversight lead the agenda."
+EXPECTED_KEY_POINTS = ["Mechanistic interpretability", "Scalable oversight"]
+
+
[email protected](loop_scope="class")
+class TestWorkflowSearchStructuredRoundtrip:
+ """A structured search result survives the Temporal activity → workflow → submitter round-trip."""
+
+ @pytest.fixture
+ def temporal_enabled(self) -> Generator[None, None, None]:
+ config = get_config()
+ previous = config.temporal.is_enabled
+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})
+ yield
+ config.temporal = config.temporal.model_copy(update={"is_enabled": previous})
+
+ @pytest.fixture
+ def structured_search_pipe_job(self) -> Generator[PipeJob, None, None]:
+ """A PipeJob for the structured search pipe, in LIVE mode so the workflow dispatches the search activity."""
+ yield from pipe_job_from_bundle(
+ bundle_file=BUNDLE_FILE,
+ pipe_code=PIPE_CODE,
+ pipe_run_mode=PipeRunMode.LIVE,
+ )
+
+ async def test_structured_search_result_survives_temporal_boundary(
+ self,
+ temporal_client: TemporalClient,
+ mocker: MockerFixture,
+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration
+ structured_search_pipe_job: PipeJob,
+ ) -> None:
+ """The activity's raw dict re-validates into the TopicSummary structure class on the submitter, intact."""
+ mock_result = {
+ "title": EXPECTED_TITLE,
+ "summary": EXPECTED_SUMMARY,
+ "key_points": EXPECTED_KEY_POINTS,
+ }
+ search_core_mock = mocker.AsyncMock(return_value=mock_result)
+ mocker.patch(ACT_SEARCH_STRUCTURED_TARGET, new=search_core_mock)
+
+ task_queue = f"q_search_struct_{uuid.uuid4().hex[:8]}"
+ workflow_id = f"wf_search_struct_{uuid.uuid4().hex[:8]}"
+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(
+ temporal_client=temporal_client,
+ task_queue=task_queue,
+ retry_policy=RetryPolicy(maximum_attempts=1),
+ )
+
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ pipe_output = await executor.execute_workflow(
+ workflow_class=WfPipeRouter,
+ workflow_arg=structured_search_pipe_job,
+ workflow_id=workflow_id,
+ )
+
+ # The structured activity dispatched (mock fired) — the leaf did not run inline in the workflow,
+ # and the SearchObjectAssignment (carrying output_class_schema) crossed the boundary.
+ search_core_mock.assert_awaited_once()
+
+ # The raw dict re-validated against the original output class on the submitter side and round-tripped
+ # intact (rehydrate the deferred working memory exactly as a production submitter does).
+ rehydrate_pipe_output(pipe_output, structured_search_pipe_job)
+ content = pipe_output.main_stuff.content
+ assert isinstance(content, StructuredContent)
+ data = content.model_dump()
+ assert data["title"] == EXPECTED_TITLE
+ assert data["summary"] == EXPECTED_SUMMARY
+ assert data["key_points"] == EXPECTED_KEY_POINTS
diff --git a/tests/integration/pipelex/temporal/test_workflow_search_structured_validation_failure.py b/tests/integration/pipelex/temporal/test_workflow_search_structured_validation_failure.py
new file mode 100644
index 000000000..48acf7b56
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_workflow_search_structured_validation_failure.py
@@ -0,0 +1,109 @@
+"""Regression test: a malformed structured-search result fails the Temporal workflow terminally, not a hang.
+
+The structured-search activity (``act_search_gen_structured``) returns the provider's *raw* result dict — the
+gateway worker returns ``json.loads(...)`` unvalidated against the schema — and the submitter re-validates it
+against the original output structure class with ``output_structure_class.model_validate(result_dict)`` in
+workflow code, *outside* the activity's ``convert_pipelex_errors`` boundary. Left to raise a bare
+``pydantic.ValidationError`` there, it would be neither a ``WorkflowExecutionError`` nor a ``PipelexError`` — the
+only two ``workflow_failure_exception_types`` — so Temporal would treat it as a *workflow-task* failure and retry
+indefinitely, hanging the submitter (the exact failure mode the whole search seam exists to prevent).
+``make_search_structured`` converts it to a terminal ``ContentGenerationError`` (a ``PipelexError``) instead; this
+test locks that in by feeding a malformed dict and asserting the submitter gets a terminal
+``WorkflowExecutionError`` rather than hanging. The success sibling is ``test_workflow_search_structured_roundtrip.py``.
+"""
+
+import uuid
+from collections.abc import Generator
+from typing import TYPE_CHECKING
+
+import pytest
+from pytest_mock import MockerFixture
+from temporalio.client import Client as TemporalClient
+from temporalio.common import RetryPolicy
+
+from pipelex.config import get_config
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.exceptions import WorkflowExecutionError
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+
+ACT_SEARCH_STRUCTURED_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_structured"
+
+BUNDLE_FILE = "tests/integration/pipelex/temporal/library_crate/structured_search.mthds"
+PIPE_CODE = "structured_search"
+
+# A dict the provider could plausibly return that does NOT satisfy the TopicSummary structure: the required
+# ``title`` field is missing, so re-validating it against the original output class on the submitter must fail.
+MALFORMED_RESULT = {
+ "summary": "Interpretability and scalable oversight lead the agenda.",
+ "key_points": ["Mechanistic interpretability", "Scalable oversight"],
+}
+
+
[email protected](loop_scope="class")
+class TestWorkflowSearchStructuredValidationFailure:
+ """A malformed structured-search result surfaces a terminal WorkflowExecutionError instead of hanging."""
+
+ @pytest.fixture
+ def temporal_enabled(self) -> Generator[None, None, None]:
+ config = get_config()
+ previous = config.temporal.is_enabled
+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})
+ yield
+ config.temporal = config.temporal.model_copy(update={"is_enabled": previous})
+
+ @pytest.fixture
+ def structured_search_pipe_job(self) -> Generator[PipeJob, None, None]:
+ """A PipeJob for the structured search pipe, in LIVE mode so the workflow dispatches the search activity."""
+ yield from pipe_job_from_bundle(
+ bundle_file=BUNDLE_FILE,
+ pipe_code=PIPE_CODE,
+ pipe_run_mode=PipeRunMode.LIVE,
+ )
+
+ @pytest.mark.timeout(60)
+ async def test_malformed_structured_result_fails_terminally(
+ self,
+ temporal_client: TemporalClient,
+ mocker: MockerFixture,
+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration
+ structured_search_pipe_job: PipeJob,
+ ) -> None:
+ """A structured-search dict that fails submitter-side re-validation terminates the workflow, not hangs."""
+ search_core_mock = mocker.AsyncMock(return_value=MALFORMED_RESULT)
+ mocker.patch(ACT_SEARCH_STRUCTURED_TARGET, new=search_core_mock)
+
+ task_queue = f"q_search_struct_bad_{uuid.uuid4().hex[:8]}"
+ workflow_id = f"wf_search_struct_bad_{uuid.uuid4().hex[:8]}"
+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(
+ temporal_client=temporal_client,
+ task_queue=task_queue,
+ retry_policy=RetryPolicy(maximum_attempts=1),
+ )
+
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ with pytest.raises(WorkflowExecutionError) as exc_info:
+ await executor.execute_workflow(
+ workflow_class=WfPipeRouter,
+ workflow_arg=structured_search_pipe_job,
+ workflow_id=workflow_id,
+ )
+
+ # The structured activity dispatched (mock fired) — the malformed dict came back through the boundary,
+ # and the failure happened on the submitter side, not inside the (successful) activity.
+ search_core_mock.assert_awaited_once()
+
+ # Terminal failure, not a hang: the submitter recovered a structured report whose message reflects the
+ # submitter-side structured-validation failure (not a generic "Failed to execute workflow ...").
+ error = exc_info.value
+ assert error.error_report is not None
+ report = error.to_error_report()
+ assert "validation" in report.message.lower()
+ assert "Failed to execute workflow" not in report.message
diff --git a/tests/integration/pipelex/temporal/test_workflow_search_success_roundtrip.py b/tests/integration/pipelex/temporal/test_workflow_search_success_roundtrip.py
new file mode 100644
index 000000000..bb85dcb07
--- /dev/null
+++ b/tests/integration/pipelex/temporal/test_workflow_search_success_roundtrip.py
@@ -0,0 +1,103 @@
+"""Success-path counterpart to the search error-report full-chain test.
+
+Runs the ``native_search`` pipe through a real ``WfPipeRouter`` workflow on the in-process Temporal server
+with the activity-side search call mocked to *succeed*, and asserts the ``SearchResultContent`` comes back
+intact across the activity → workflow → submitter boundary. This locks in the other half of the fix: search
+on Temporal now records its result via an activity (replay-safe) instead of running inline on the workflow
+loop, and the native ``SearchResultContent`` serializes cleanly back to the submitter. The mock firing also
+proves the workflow actually dispatched ``act_search_gen_sourced_answer`` rather than running the leaf inline.
+"""
+
+import uuid
+from collections.abc import Generator
+from typing import TYPE_CHECKING
+
+import pytest
+from pytest_mock import MockerFixture
+from temporalio.client import Client as TemporalClient
+from temporalio.common import RetryPolicy
+
+from pipelex.config import get_config
+from pipelex.core.stuffs.document_content import DocumentContent
+from pipelex.core.stuffs.search_result_content import SearchResultContent
+from pipelex.pipe_run.pipe_job import PipeJob
+from pipelex.pipe_run.pipe_run_mode import PipeRunMode
+from pipelex.temporal.temporal_hub import get_task_manager
+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor
+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter
+from tests.integration.pipelex.error_handling.test_data import SearchErrorReportParityTestData
+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle
+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output
+
+if TYPE_CHECKING:
+ from pipelex.core.pipes.pipe_output import PipeOutput
+
+ACT_SEARCH_SOURCED_ANSWER_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_sourced_answer"
+
+EXPECTED_ANSWER = "Recent AI safety work focuses on interpretability and scalable oversight."
+EXPECTED_SOURCE_TITLE = "AI safety overview"
+
+
[email protected](loop_scope="class")
+class TestWorkflowSearchSuccessRoundtrip:
+ """A successful search result survives the Temporal activity → workflow → submitter round-trip."""
+
+ @pytest.fixture
+ def temporal_enabled(self) -> Generator[None, None, None]:
+ config = get_config()
+ previous = config.temporal.is_enabled
+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})
+ yield
+ config.temporal = config.temporal.model_copy(update={"is_enabled": previous})
+
+ @pytest.fixture
+ def search_pipe_job(self) -> Generator[PipeJob, None, None]:
+ """A PipeJob for the search pipe, in LIVE mode so the workflow dispatches the search activity."""
+ yield from pipe_job_from_bundle(
+ bundle_file=SearchErrorReportParityTestData.BUNDLE_FILE,
+ pipe_code=SearchErrorReportParityTestData.PIPE_CODE,
+ pipe_run_mode=PipeRunMode.LIVE,
+ )
+
+ async def test_search_result_survives_temporal_boundary(
+ self,
+ temporal_client: TemporalClient,
+ mocker: MockerFixture,
+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration
+ search_pipe_job: PipeJob,
+ ) -> None:
+ """The activity's ``SearchResultContent`` comes back intact on the submitter's ``PipeOutput``."""
+ mock_result = SearchResultContent(
+ answer=EXPECTED_ANSWER,
+ sources=[DocumentContent(title=EXPECTED_SOURCE_TITLE, url="https://example.com/ai-safety", mime_type="text/html")],
+ )
+ search_core_mock = mocker.AsyncMock(return_value=mock_result)
+ mocker.patch(ACT_SEARCH_SOURCED_ANSWER_TARGET, new=search_core_mock)
+
+ task_queue = f"q_search_ok_{uuid.uuid4().hex[:8]}"
+ workflow_id = f"wf_search_ok_{uuid.uuid4().hex[:8]}"
+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(
+ temporal_client=temporal_client,
+ task_queue=task_queue,
+ retry_policy=RetryPolicy(maximum_attempts=1),
+ )
+
+ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):
+ pipe_output = await executor.execute_workflow(
+ workflow_class=WfPipeRouter,
+ workflow_arg=search_pipe_job,
+ workflow_id=workflow_id,
+ )
+
+ # The activity dispatched (mock fired) — the leaf did not run inline in the workflow.
+ search_core_mock.assert_awaited_once()
+
+ # The SearchResultContent round-tripped back to the submitter intact (rehydrate the deferred
+ # working memory exactly as a production submitter does).
+ rehydrate_pipe_output(pipe_output, search_pipe_job)
+ content = pipe_output.main_stuff.content
+ assert isinstance(content, SearchResultContent)
+ assert content.answer == EXPECTED_ANSWER
+ assert len(content.sources) == 1
+ assert content.sources[0].title == EXPECTED_SOURCE_TITLE
diff --git a/tests/unit/pipelex/cli/test_validate_temporal_flag.py b/tests/unit/pipelex/cli/test_validate_temporal_flag.py
new file mode 100644
index 000000000..745aee8d1
--- /dev/null
+++ b/tests/unit/pipelex/cli/test_validate_temporal_flag.py
@@ -0,0 +1,33 @@
+"""The ``--temporal`` flag surface on ``pipelex validate`` subcommands.
+
+Locks the public CLI contract that ``bundle`` / ``pipe`` / ``method`` each expose
+``--temporal/--no-temporal`` (parity with ``pipelex run``). The flag overrides
+``temporal.is_enabled`` for the boot; the validation sweep stays in-process regardless,
+so the *behavioural* guarantee (validation does not dispatch to Temporal under a
+Temporal-enabled hub) is exercised by the distributed e2e scenario, not here.
+
+Asserts on the resolved Click parameters rather than on rendered ``--help`` text: the
+help renderer line-wraps option names at narrow terminal widths (CI's non-tty width
+splits ``--temporal`` across lines), which is a presentation detail, not the contract.
+Introspecting the command's params is width-independent and short-circuits before any
+Pipelex boot, so this stays a true unit test.
+"""
+
+from __future__ import annotations
+
+import click
+import pytest
+from typer.main import get_command
+
+from pipelex.cli.commands.validate.app import validate_app
+
+
+class TestValidateTemporalFlag:
+ @pytest.mark.parametrize("subcommand", ["bundle", "pipe", "method"])
+ def test_temporal_flag_exposed(self, subcommand: str) -> None:
+ group = get_command(validate_app)
+ assert isinstance(group, click.Group)
+ command = group.commands[subcommand]
+ flag_opts: list[str] = [opt for param in command.params for opt in (*param.opts, *param.secondary_opts)]
+ assert "--temporal" in flag_opts
+ assert "--no-temporal" in flag_opts
diff --git a/tests/unit/pipelex/temporal/test_worker_scopes_specialized.py b/tests/unit/pipelex/temporal/test_worker_scopes_specialized.py
index d533da0e8..193956e4d 100644
--- a/tests/unit/pipelex/temporal/test_worker_scopes_specialized.py
+++ b/tests/unit/pipelex/temporal/test_worker_scopes_specialized.py
@@ -1,9 +1,9 @@
"""Unit tests for the specialized worker scopes (runner-llm, runner-img-gen,
-runner-extract, runner-jinja2) shipped in `pipelex/pipelex.toml`.
+runner-extract, runner-jinja2, runner-search) shipped in `pipelex/pipelex.toml`.
These scopes let deployment manifests spin up one worker pool per backend
class. The tests verify (a) each scope registers exactly the activities its
-name implies and (b) the union of all four specialized scopes covers the
+name implies and (b) the union of the specialized scopes covers the
full ``runner`` registration with no orphan activities.
"""
@@ -39,6 +39,7 @@ class TestSpecializedWorkerScopes:
("runner-img-gen", {"act_img_gen_images", "act_render_page_views"}),
("runner-extract", {"act_extract_gen_extract_pages", "act_render_page_views"}),
("runner-jinja2", {"act_jinja2_gen_text"}),
+ ("runner-search", {"act_search_gen_sourced_answer", "act_search_gen_structured"}),
],
)
def test_scope_registers_expected_activities(self, scope_name: str, expected_activity_names: set[str]) -> None:
@@ -61,7 +62,7 @@ class TestSpecializedWorkerScopes:
runner_names = {activity.__name__ for activity in runner_activities}
specialized_union: set[str] = set()
- for scope_name in ("runner-llm", "runner-img-gen", "runner-extract", "runner-jinja2"):
+ for scope_name in ("runner-llm", "runner-img-gen", "runner-extract", "runner-jinja2", "runner-search"):
scope = worker_scopes.scopes[scope_name]
_, scope_activities = tasks.workflows_and_activities(scope=scope)
specialized_union.update(activity.__name__ for activity in scope_activities)
diff --git a/wip/distributed-execution/README.md b/wip/distributed-execution/README.md
index edaa948d2..ef28506f0 100644
--- a/wip/distributed-execution/README.md
+++ b/wip/distributed-execution/README.md
@@ -15,6 +15,7 @@ This folder is the **distributed-execution track**: running MTHDS methods as Tem
- [schema-reconstruction-hardening.md](schema-reconstruction-hardening.md) — deferred hardening of cross-process Pydantic-schema reconstruction.
- [unit-testing-worker-sandbox-validation.md](unit-testing-worker-sandbox-validation.md) — deferred bug: a sandboxed worker can't boot under `--is-unit-testing` because the registered test workflows fail temporalio sandbox validation.
- [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md) — **OPEN handoff:** how the `_bridge` dev-merge changed the runtime-bridge surface (param renames, deleted graph-assembly primitive, `graph_context→trace_context`, #967/#968) and what the downstream `pipelex-mistralai-workflows` + `_workflows` repos need reconciled. Greps/diagnosis not yet done.
+- [validate-sweep-temporal-leak-repro.md](validate-sweep-temporal-leak-repro.md) — **bug brief + e2e repro (DONE):** the `/validate` dry-run sweep leaked nested controller sub-pipes to Temporal (HTTP 422 on a standalone `PipeBatch`). Fixed (commit `3377babb`); all three guard layers now in place — in-process sentinel test, Mode-1 pytest (`test_validate_sweep_stays_in_process.py`), and the deployment-faithful Mode-2 scenario (Tier 2c in the `temporal-e2e-validate` skill, using the `pipelex validate bundle --temporal` flag).
- [temporal-e2e-validate-skill.md](temporal-e2e-validate-skill.md) — validation status of the `/temporal-e2e-validate` skill and the Step 9 (queue-options) skill bugs patched in its `queue-options-battery.md`. Full dry + live run all green; open follow-ups are the vestigial `act_jinja2_gen_text` removal and a stale Mode 1 "Known xfails" note.
- [mistral-workflows/](mistral-workflows/) — sub-track for landing the `pipelex-mistralai-workflows` plugin (the MISTRAL_NATIVE backend) onto current `dev`. Its [README](mistral-workflows/README.md) is the hub; verdict is split — land the bridge, hold the `mistralai` 1.x→2.x bump (blocked on instructor PR #2298). Carries the two ready-to-paste keep-alive prompts.
diff --git a/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md b/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md
index 5db3c55f1..7b18c6635 100644
--- a/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md
+++ b/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md
@@ -44,6 +44,24 @@ Recommended order: run the **no-spend** sweep first to confirm the plumbing chea
/temporal-e2e-validate validate cross-process registry # Tier 2b
```
+### Validation sweep stays in-process — Temporal-leak guard (free)
+
+Guards the `/validate` dry-run leak (nested controller sub-pipes dispatching to Temporal → HTTP 422 on a standalone `PipeBatch`). Two layers, both free; the contract is "the validation sweep never dispatches to Temporal, even under a Temporal-enabled boot".
+
+```text
+/temporal-e2e-validate validate the dry-run sweep stays in-process under a Temporal-enabled boot # Mode 2 Tier 2c
+```
+
+- **Mode-1 pytest** (real `TemporalPipeRouter` as hub default, spies `WorkflowExecutor.execute_workflow`, asserts never called). It lives *outside* `library_crate/`, so the generic "Mode 1 pytest suite" prompt above does **not** pick it up — run it by path:
+
+ ```text
+ timeout 180 .venv/bin/pytest tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py -m temporal --temporal-server local --timeout=60
+ ```
+
+ (Self-contained — GREEN never dispatches, so no live server is actually required; `--temporal-server none` works too.)
+
+- **Mode-2 Tier 2c** (deployment-faithful: `validate bundle --temporal` over a standalone `PipeBatch` → exit 0 **and** worker idle / no `WfPipeRouter` dispatch). Reached via the prompt above; full GREEN/RED procedure in `references/mode-2-tiers.md` Step 3, Tier 2c.
+
### Graph tracing (free)
```text
diff --git a/wip/distributed-execution/validate-sweep-temporal-leak-repro.md b/wip/distributed-execution/validate-sweep-temporal-leak-repro.md
new file mode 100644
index 000000000..cd199d245
--- /dev/null
+++ b/wip/distributed-execution/validate-sweep-temporal-leak-repro.md
@@ -0,0 +1,150 @@
+# Bug brief + e2e repro — the `/validate` dry-run sweep leaked nested controller sub-pipes to Temporal
+
+> **Status:** bug FIXED (commit `3377babb` on `fix/Temporal-dry-run`). **All three guard layers are now implemented:** the in-process sentinel regression test, the **Mode-1 pytest** (`tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py`), and the **Mode-2 scenario** (Tier 2c in the `temporal-e2e-validate` skill). This doc is retained as the cold-start brief + reproduction record. See [`/TODOS.md`](../../TODOS.md) for the PR-review guide.
+>
+> **Track:** part of the [distributed-execution](./README.md) track — the class of bug that only the true 3-process topology surfaces.
+
+## 1. What the bug was
+
+Running the **Pipelex API** (`../pipelex-api`) Temporal-enabled (`.pipelex/pipelex_override.toml` → `[temporal] is_enabled = true`) with a separate **Pipelex worker** (`../pipelex-worker`) up, calling the runner's `/validate` endpoint on a bundle containing a controller pipe returned **HTTP 422**:
+
+```
+ValidateBundleError (error_domain=input), 422
+detail: "Dry run failed with 1 unexpected pipe failure(s):
+ '<domain>.<batch_pipe>': ... status=DryRunStatus.FAILURE
+ error_message=\"Dry run failed for pipe '<domain>.<batch_pipe>':
+ Failed to execute workflow WfPipeRouter\""
+```
+
+The string **"Failed to execute workflow WfPipeRouter"** is produced only at `pipelex/temporal/tprl/workflow_caller.py` (the `(WorkflowAlreadyStartedError, RPCError)` branch of `WorkflowExecutor.execute_workflow`) — a **top-level** Temporal workflow dispatch from the submitter side. **A dry run should never reach that code.**
+
+## 2. Root cause (one sentence)
+
+`BundleValidator` builds a local in-process `PipeRouter` for the sweep but **never installs it as the active `get_pipe_router()` override**, so every nested controller sub-pipe (`pipe_controllers/sub_pipe.py`, `pipe_controllers/batch/pipe_batch.py` — all dispatch via `get_pipe_router()`) falls through to the **hub default**, which under a Temporal-enabled hub is the `TemporalPipeRouter` (wired in `pipelex/pipelex.py` when `get_config().temporal.is_enabled`). The top-level pipe ran in-process; its nested pipes leaked to Temporal.
+
+**Why it 422s for a *standalone* `PipeBatch`/`PipeParallel` specifically:** the sweep dry-runs every pipe in the bundle individually. When it dry-runs a standalone batch pipe *directly*, the batch fans out over a mock list (`mock_inputs` mints ~3 items) and fires **N concurrent top-level Temporal dispatches with the same workflow id** (the id derives from the one shared `dry_run_pipeline_id`) → `WorkflowAlreadyStartedError` → `WorkflowExecutionError("Failed to execute workflow WfPipeRouter")` → the pipe is classified `FAILURE` → `DryRunError` → 422. A batch reached only *as a sub-pipe of a sequence* is a single serial dispatch — no collision — so it **silently round-trips Temporal and passes** (a false pass: still wrong, just not fatal).
+
+## 3. The fix (already committed: `3377babb`)
+
+Mirror the DIRECT-mode precedent in `pipelex/runtime_bridge/bridge.py::_run_direct`. In `pipelex/pipeline/bundle_validator.py`:
+
+- keep the in-process router as `self._pipe_router` (don't discard it inside `PipeRun(...)`);
+- wrap the per-pipe sweep loop in `validate_pipes` with `with scoped_pipe_router(self._pipe_router):`.
+
+Now nested controllers resolve the in-process router; the whole sweep stays in-process regardless of backend.
+
+**In-process regression guard (deterministic, no Temporal needed):**
+`tests/integration/pipelex/pipeline/test_bundle_validator.py::TestBundleValidatorIntegration::test_standalone_batch_sweep_scopes_in_process_router` — installs a hub-default router that raises if reached, sweeps a standalone `PipeBatch`, asserts SUCCESS with the hub default never touched. (Verified RED without the scope.)
+
+> ⚠️ A *separate* "secondary fix" to `dry_run_pipeline.py` (the graph step) was tried and **reverted** — it was misdiagnosed. See the correction block in [`/TODOS.md`](../../TODOS.md). Not relevant to this repro.
+
+## 4. Why our PyTest suite didn't catch it (answer to "can in-process pytest reproduce it?")
+
+**Partial confirm, with nuance — don't over-claim "impossible in-process":**
+
+- The **leak itself** is reproducible in a single process with zero Temporal infrastructure — that's exactly what the committed regression test does (sentinel hub-default router). The **collision symptom** is *also* reproducible in-process in principle: the in-process Temporal test server (`--temporal-server none`/`time-skipping`) enforces the same workflow-id-reuse semantics, so concurrent same-id dispatches would still raise `WorkflowAlreadyStartedError`.
+- So it escaped **not** because the topology can't show it, but because **no test crossed the seam "Temporal-enabled hub × the bundle-validation sweep":** the Temporal suite (`tests/integration/pipelex/temporal/`) exercises pipe *runs* via crate, never the validation *sweep*; the `BundleValidator` tests run under the default in-process hub, never the Temporal hub.
+- Two reasons the **distributed (Mode 2) setup is still the right home** for a scenario:
+ 1. The silent **false-pass** shapes are invisible to a "does validate pass?" assertion — only an explicit "no Temporal dispatch happened" check (the sentinel test) or the standalone-batch collision shape exposes them.
+ 2. The **production symptom** (collision → 422 across a real API↔worker process boundary) matches the deployment topology exactly here, where a human actually hit it.
+
+Net: the unit regression test guards the leak; the e2e scenario below guards the deployment-faithful symptom and documents the reproduction for the record.
+
+## 5. The new scenario to add to `temporal-e2e-validate`
+
+Lives under the skill at `.claude/skills/temporal-e2e-validate/`. Add this as a new scenario in `references/mode-2-tiers.md` (or a short new reference file).
+
+**Which home?** This bug is **submitter-side** — the colliding top-level dispatches fire at *dispatch time*, regardless of whether the worker runs in the same process. So the true 3-process topology is the *deployment-faithful* home, **not the only one**: the same failure also reproduces under the classic in-process pytest harness. Treat Mode 2 here as the production-faithful demonstration (and the only thing that also exercises the genuinely cross-process surfaces — LibraryCrate propagation, serialization), and add the cheaper automated catch as a pytest companion (next subsection).
+
+**Enabler that makes this clean:** `pipelex validate bundle` now has a `--temporal/--no-temporal` flag (added alongside the fix; parity with `pipelex run`). So the repro needs **no `pipelex_temporary_override.toml` juggling** — just pass `--temporal`. (The flag flips the boot's hub default to the Temporal router; with the fix, the sweep still stays in-process — which is the whole point being asserted.)
+
+> **Forward-looking note (why the flag exists despite being a no-op for the sweep today):** once validation runs as a standalone Temporal activity ([`../dry-run-refactor/followup-temporal-validation-activity.md`](../dry-run-refactor/followup-temporal-validation-activity.md)), `--temporal` on `validate` stops being a no-op and becomes the switch that dispatches the sweep through that activity. Until then it only controls the boot, which is exactly what this scenario needs.
+
+**Repro bundle — reuse an existing one, no new fixture needed:**
+`tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds` already declares a **standalone `type = "PipeBatch"` pipe** (`batch_temporal_describe_topics`). When the sweep dry-runs that pipe directly, it fans out → the exact shape that 422'd in production.
+
+### Prereqs
+Temporal server + worker up, per `references/mode-2-setup.md` (read it first). The worker config is irrelevant to the leak — the leak is entirely submitter-side — but keep a worker up to mirror production (and so the first dispatch actually *starts*, yielding the true `WorkflowAlreadyStartedError` rather than an `RPCError`; both land in the same `except` branch, so both are RED, but the former matches production).
+
+### GREEN — the fix in place (current `HEAD`)
+
+```bash
+# Server up (Mode 2). The boot connects to it; with the fix the sweep stays in-process,
+# so the worker should receive NO workflow for this validate run.
+timeout 120 .venv/bin/pipelex validate bundle \
+ tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \
+ --temporal 2>&1 | tail -20
+echo "EXIT=$?"
+```
+
+Expected: `EXIT=0`, `Successfully validated bundle ...`. Strong check: the worker session shows **no `WfPipeRouter` / `WfPipeRun` execution** for this run (it stayed idle) — `tmux capture-pane -t temporal-worker-router -p -S -200 | grep -i WfPipeRouter` returns nothing new.
+
+### RED — temporarily revert the fix to prove the scenario bites
+
+The fix is committed, so you must undo it in the working tree. **Surgical (preferred)** — neutralize just the scope in `pipelex/pipeline/bundle_validator.py`, inside `validate_pipes`:
+
+```python
+# change THIS:
+with scoped_pipe_router(self._pipe_router):
+ for pipe in sweepable_pipes:
+ results[pipe.pipe_ref] = await self._classify_pipe(...)
+
+# to THIS (drop the scope — the leak returns):
+for pipe in sweepable_pipes:
+ results[pipe.pipe_ref] = await self._classify_pipe(...)
+```
+
+(Alternative: `git revert --no-commit 3377babb` — but that also reverts the regression test + changelog; the surgical edit is cleaner.)
+
+Re-run the GREEN command. Expected RED:
+
+```
+EXIT=1
+Dry run failed with 1 unexpected pipe failure(s):
+ 'temporal_batch_test.batch_temporal_describe_topics': ... status=DryRunStatus.FAILURE
+ error_message="... Failed to execute workflow WfPipeRouter"
+```
+
+The worker session will now show `WfPipeRouter` activity (the leak — validation dispatched to Temporal). **Restore the fix immediately:**
+
+```bash
+git checkout -- pipelex/pipeline/bundle_validator.py # or: git revert --abort, if you used revert
+```
+
+### PASS criteria for the scenario
+- GREEN run exits 0 **and** the worker received no workflow dispatch (validation stayed in-process under a Temporal-enabled boot).
+- The documented RED procedure reproduces the `Failed to execute workflow WfPipeRouter` failure (sanity that the scenario actually exercises the leak path).
+
+### Caveats to put in the scenario text
+- **Shape matters.** Only a *standalone* `PipeBatch`/`PipeParallel` swept directly turns the leak fatal (concurrent same-id collision). A batch reached only as a sequence sub-pipe round-trips Temporal but passes — a false pass. `temporal_batch.mthds` has the standalone batch, so it's a valid RED trigger.
+- **Worker up vs down both RED** (collision → `WorkflowAlreadyStartedError` vs no-worker → `RPCError`; same `except` branch). Keep the worker up to match production.
+- **GREEN means "no dispatch happened."** Because the flag makes validation backend-agnostic, the strongest GREEN assertion is the worker-idle check, not just exit 0.
+
+### Automated companion — a Mode-1 pytest (recommended alongside the Mode 2 scenario)
+
+The Mode 2 scenario is agent-run, not CI-automated. Because this bug is submitter-side, the same catch can live as an ordinary pytest in the classic in-process Temporal harness (`tests/integration/pipelex/temporal/`, `--temporal-server none`/`local`, in-process worker) — fast, deterministic, runs in CI. That is the cheaper automated guard; Mode 2 remains the deployment-faithful demonstration. The three layers are complementary, not redundant:
+
+| Layer | Where it runs | CI-automated | Cross-process surfaces (crate / serialization) | Status |
+|---|---|---|---|---|
+| Sentinel regression test (no Temporal) | plain pytest (`tests/integration/pipelex/pipeline/test_bundle_validator.py`) | ✅ | ✗ (deterministic stand-in for the hub default) | **exists** (`test_standalone_batch_sweep_scopes_in_process_router`) |
+| **Mode-1 pytest (real Temporal, in-process server+worker)** | classic pytest temporal env | ✅ | ✗ (worker shares the process) | **proposed here** |
+| Mode 2 scenario (3 separate processes) | the skill, agent-run | ✗ | ✓ | proposed in §5 |
+
+Sketch of the Mode-1 pytest (author it in `tests/integration/pipelex/temporal/`, marked `temporal`, honoring `--temporal-server`):
+
+- Boot/scope a **Temporal-enabled hub** so `get_pipe_router()`'s default is the real `TemporalPipeRouter` (mirror the existing temporal-suite fixtures that stand up the in-process server + worker).
+- Run the real sweep — `BundleValidator().validate_pipes([...standalone PipeBatch...], ...)` — over `temporal_batch.mthds`'s `batch_temporal_describe_topics` (or an equivalent standalone `type = "PipeBatch"`).
+- **Assert GREEN, strongly:** the sweep succeeds **and** no top-level workflow was started — e.g. spy on `pipelex.temporal.tprl.workflow_caller.WorkflowExecutor.execute_workflow` and assert it was never called (the deterministic analogue of the Mode 2 "worker-idle" check). This is what makes the fix's contract ("the sweep never reaches the Temporal router") an automated invariant, not just "validate didn't 422."
+- Without the fix this test goes RED (the spy fires / a collision surfaces), exactly like the Mode 2 scenario.
+
+This differs from the existing sentinel test in one meaningful way: it resolves the **real** `TemporalPipeRouter` as the hub default instead of a raising stand-in, so it also proves the *wiring* (config → hub default router) that the sentinel test deliberately bypasses.
+
+## 6. Pointers
+
+- Fix commit: `3377babb` — `fix(bundle-validator): prevent nested controller sub-pipes from leaking to Temporal during dry runs`.
+- Fix site: `pipelex/pipeline/bundle_validator.py` (`__init__` keeps `self._pipe_router`; `validate_pipes` wraps the sweep in `scoped_pipe_router`). Precedent: `pipelex/runtime_bridge/bridge.py::_run_direct`.
+- Hub wiring that makes the hub default Temporal: `pipelex/pipelex.py` (`set_pipe_router(make_temporal_pipe_router())` when `temporal.is_enabled`); resolution in `pipelex/hub.py::get_pipe_router` (contextvar override → else hub default).
+- Leak sites: `pipelex/pipe_controllers/sub_pipe.py`, `pipelex/pipe_controllers/batch/pipe_batch.py` (`get_pipe_router().run(...)`).
+- New CLI flag: `--temporal/--no-temporal` on `pipelex validate {bundle,pipe,method}` + `validate --all` (`pipelex/cli/commands/validate/`), guarded by `tests/unit/pipelex/cli/test_validate_temporal_flag.py`.
+- In-process regression test: `tests/integration/pipelex/pipeline/test_bundle_validator.py::TestBundleValidatorIntegration::test_standalone_batch_sweep_scopes_in_process_router`.
+- Skill: `.claude/skills/temporal-e2e-validate/SKILL.md` + `references/mode-2-setup.md` + `references/mode-2-tiers.md`.