Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
fix(validate): dry-run sweep no longer leaks nested controller sub-pipes to Temporal (#976)
Louis Choquel10f6ffdcc449
diff --git a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md b/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.mdindex 06906abc2..9fbc71832 100644--- a/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md+++ b/.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md@@ -7,7 +7,7 @@## Contents-- **Step 3** — Sequential tests: Tier 1 (sequence), Tier 2 (hydration), Tier 2b (cross-process registry), Tier 3 (parallel), Tier 4 (image generation), Tier 5 (image flow)+- **Step 3** — Sequential tests: Tier 1 (sequence), Tier 2 (hydration), Tier 2b (cross-process registry), Tier 2c (validate sweep stays in-process), Tier 3 (parallel), Tier 4 (image generation), Tier 5 (image flow)- **Step 4** — Verify graph output- **Step 5** — Concurrent isolation tests (concept / pipe / multi-concept)- **Step 5b** — Tier 8: cross-worker usage emission; Tier 8b: cross-worker cost report assembly (`--mock-inference`, free)@@ -100,6 +100,65 @@ After this completes, tell the user:and quote the `KajsonDecoderError` and `ApplicationError: Failed decoding arguments`lines verbatim, plus the activity name (`act_deliver`).+**Tier 2c — Validate sweep stays in-process under a Temporal-enabled boot (submitter-side leak guard).**++This is **not** a `run` — it's a `validate`. It guards the production bug where the `/validate`+dry-run sweep leaked nested controller sub-pipes to Temporal: a standalone `PipeBatch` swept directly+fans out over a mock list and dispatches each branch through `get_pipe_router()`, which under a+Temporal-enabled hub is the `TemporalPipeRouter`. Those concurrent same-id top-level dispatches raised+`WorkflowAlreadyStartedError` → the API returned **HTTP 422** with `Failed to execute workflow+WfPipeRouter`. The fix scopes an in-process router for the whole sweep; the contract here is that+validation **never dispatches to Temporal**, even when booted Temporal-enabled.++The `--temporal` flag (parity with `pipelex run`) flips the boot's hub default to the Temporal router+with no `pipelex_temporary_override.toml` juggling. With the fix, the sweep still stays in-process —+which is exactly what this asserts. `temporal_batch.mthds` declares a standalone `type = "PipeBatch"`+pipe (`batch_temporal_describe_topics`), so its sweep fans out — the exact shape that 422'd.++```bash+# Server + worker up (Mode 2). The boot connects Temporal-enabled; with the fix the sweep stays+# in-process, so the worker must receive NO workflow for this validate run.+# Capture pipelex's OWN exit code (load-bearing for GREEN/RED). Don't pipe the command into+# `tail` and read `$?` — that's tail's exit; and `${PIPESTATUS[0]}` is bash-only (blank in zsh).+# Redirect to a file, read `$?` with no pipe in between, then tail the log. Works in bash AND zsh.+timeout 120 .venv/bin/pipelex validate bundle \+ tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \+ --temporal > /tmp/tier2c-validate.log 2>&1; echo "EXIT=$?"+tail -20 /tmp/tier2c-validate.log+```++GREEN: `EXIT=0` and `Successfully validated bundle ...`. **Strong check** (the point of the scenario):+the worker stayed idle — it received no top-level dispatch. Capture both worker sessions and confirm+no new `WfPipeRouter` / `WfPipeRun` execution appeared for this run (split workers are the Step 2+default; fall back to `temporal-worker` for the single-worker setup):++```bash+tmux capture-pane -t temporal-worker-router -p -S -200 | grep -i WfPipeRouter # expect: nothing new+tmux capture-pane -t temporal-worker-runner -p -S -200 | grep -i WfPipeRun # expect: nothing new+```++RED (prove the scenario bites) — the fix is committed, so neutralize it in the working tree. Surgical:+in `pipelex/pipeline/bundle_validator.py`, inside `validate_pipes`, drop the `with+scoped_pipe_router(self._pipe_router):` wrapper so the sweep loop runs unscoped (de-indent the loop+one level). Re-run the GREEN command — expect `EXIT=1`, `Dry run failed with 1 unexpected pipe+failure(s): 'temporal_batch_test.batch_temporal_describe_topics': ... Failed to execute workflow+WfPipeRouter`, and the worker session now shows `WfPipeRouter` activity (the leak). **Restore the fix+immediately:** `git checkout -- pipelex/pipeline/bundle_validator.py`.++After this completes, tell the user:+- PASS (GREEN exits 0 **and** worker idle — no dispatch) / FAIL.+- Caveats worth stating: only a *standalone* `PipeBatch`/`PipeParallel` swept directly turns the leak+ fatal (concurrent same-id collision); a batch reached only as a sequence sub-pipe round-trips+ Temporal but passes (a false pass), so "validate exited 0" alone is **not** sufficient — the+ worker-idle check is the real assertion. Worker up vs down are **both** RED (collision+ `WorkflowAlreadyStartedError` vs no-worker `RPCError`, same `except` branch); keep the worker up to+ match production.+- The cheaper CI-automated companion to this scenario is the Mode-1 pytest+ `tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py` (real+ `TemporalPipeRouter` as hub default, spies `WorkflowExecutor.execute_workflow`, asserts never+ called). This Mode-2 scenario is the deployment-faithful demonstration across the real API↔worker+ process boundary.+**Tier 3 — Do parallel branches execute as separate child workflows?**This sends a PipeParallel controller that fans out into two branches (tone analysis and@@ -1069,6 +1128,7 @@ ls results/*/reactflow.html| Tier 1: Sequence | Worker can unpack crate and run a pipe sequence | PASS/FAIL | path | — || Tier 2: Hydration | Worker handles dynamic concepts it has never seen | PASS/FAIL | path | — || Tier 2b: Cross-process registry | `act_deliver` decodes hydrated `pipe_output` on runner (forced via `delivery_assignment`) | PASS/FAIL | — | — |+| Tier 2c: Validate sweep stays in-process | `validate bundle --temporal` over a standalone `PipeBatch` exits 0 **and** the worker received no `WfPipeRouter` dispatch (the sweep never leaks to Temporal) | PASS/FAIL | — | — || Tier 3: Parallel | Branches execute as concurrent child workflows | PASS/FAIL | path | — || Tier 4: ImgGen | Image generation pipeline works through Temporal | PASS/FAIL | path | yes/no || Tier 5: Image flow | Generated image flows as input to next pipe step | PASS/FAIL | path | yes/no |diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f94cd234f..4953cda19 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,6 +6,8 @@ This cycle reworks **cost reporting** so it survives distributed execution and s### Added+- **`--temporal` / `--no-temporal` on `pipelex validate`.** The `bundle`, `pipe`, and `method` validate subcommands (and `validate --all`) gain the same boot override `run` already had — it sets `temporal.is_enabled` for the process. Today the validation sweep always runs in-process regardless of backend (it scopes its own in-process router), so the flag does not change *what* validation does; it is the lever for booting validation under a Temporal-enabled hub — e.g. to verify the sweep stays in-process and never dispatches nested controller sub-pipes to Temporal. It is also forward-looking: once validation can run as a standalone Temporal activity, this flag becomes the switch that dispatches the sweep through that activity.+- **`--costs` / `--no-costs` (default on).** `pipelex run pipe|method|bundle` gains a dedicated cost switch that emits usage tracing events and renders the end-of-run cost report. It rides the shared trace-event transport independently of `--graph`, so `--no-graph --costs` reports cost without building a graph and `--graph --no-costs` builds a graph with no cost report. It replaces the removed `--cost-report` (see Changed).- **Distributed cost reporting.** In Temporal mode, usage emitted from inference activities running on separate worker processes is now aggregated into a single cost report at the submitter. Usage is assembled from the trace-event stream onto `PipeOutput` in both direct and Temporal execution, so a cross-worker run reports its true total — previously cross-worker usage was emitted but never rendered into a report.@@ -38,6 +40,10 @@ This cycle reworks **cost reporting** so it survives distributed execution and s- **Temporal silent-hang fail-safe.** A pipelex domain error raised *inline* in workflow code — never dispatched through an activity — was neither an `ActivityError` nor an `ApplicationError`, so Temporal treated it as a workflow-task failure and retried it indefinitely: a silent, resource-burning hang that only surfaced (as a generic timeout) after the workflow execution timeout. `WfPipeRouter` and `WfPipeRun` now end their boundary handling with a `PipelexError` catch-all that converts a genuine inline error to a terminal, classified failure (the parent also still fires the FAILED webhook), and the worker registers `PipelexError` in `workflow_failure_exception_types` as a backstop so any uncaught domain error fails the workflow terminally instead of hanging. Scoped to domain errors only — transient Temporal/infra errors keep their (correct) task-retry behavior.+- **`PipeSearch` on Temporal no longer hangs.** Web search was the only inference operator that ran its leaf inline instead of through the `ContentGenerator` seam, so it had no Temporal activity. Run through the Temporal path, a search step executed its real HTTP call inside the workflow event loop, and on failure raised a raw `SearchJobFailureError` (neither an `ActivityError` nor a `WorkflowExecutionError`) that Temporal treated as a workflow-task failure and retried forever — the submitter hung instead of getting a result. Even on success it was replay-unsafe (the search re-ran on every workflow replay). `PipeSearch` now dispatches its leaf through new `act_search_gen_sourced_answer` / `act_search_gen_structured` activities like every other operator: results are recorded in workflow history (replay-safe) and failures cross the workflow boundary as a terminal `WorkflowExecutionError` carrying the structured `ErrorReport` — identical classification to the local path. A `runner-search` worker scope and per-activity routing entries are available for split deployments.++- **Bundle validation no longer leaks nested controller sub-pipes to Temporal.** With Temporal enabled, the in-process dry-run sweep (`BundleValidator`) ran the top-level pipe in-process but dispatched every *nested* controller sub-pipe through the hub-default router — the Temporal router — turning a no-cost dry run into real top-level workflow dispatches. A standalone `PipeBatch` / `PipeParallel` swept directly fanned out N concurrent same-id top-level dispatches and collided (`WorkflowAlreadyStartedError`), surfacing as **HTTP 422** on the runner's `/validate` route; other shapes silently round-tripped Temporal. The sweep now scopes its in-process router for the whole sweep (mirroring the DIRECT-mode runtime bridge), so nested controllers resolve the in-process router and validation stays fully in-process regardless of the configured backend.+## [v0.31.0] - 2026-06-04This release hardens Pipelex at its edges. The headliners: a full **error-handling overhaul** that gives every error a stable, RFC 7807-shaped identity and carries it all the way out to webhooks and external surfaces; **lenient validation with pipe signatures**, so you can sketch and dry-run a whole pipeline top-down before a single pipe is implemented; and a first, **experimental cut of CSV tabular support** for reading and writing typed lists straight from `.csv`.diff --git a/TODOS.md b/TODOS.mdnew file mode 100644index 000000000..8395ab551--- /dev/null+++ b/TODOS.md@@ -0,0 +1,137 @@+# Fix: `/validate` dry-run leaked nested controller sub-pipes to Temporal++> **Status: DONE.** Bug fixed and all three layers of test coverage are in place. This doc is the PR-review guide — what the bug was, what changed, why it's correct, and how it's guarded.+>+> **Stacking:** this branch (`fix/Temporal-dry-run`) is stacked on `feature/PipeSearch-Temporal` (open PR [#974](https://github.com/Pipelex/pipelex/pull/974)). The `pipe_search.py` / `search_generate.py` / `content_generator_in_workflow.py` / `native_search.mthds` / `structured_search.mthds` / `test_workflow_search_*` files in the diff belong to **#974**, not to this change — review them there. The files owned by **this** change are listed under [Files in this change](#files-in-this-change) below.++**Where the fix lives:** `pipelex` core (the runtime), **not** `pipelex-api`. All paths are relative to the `pipelex` repo (locally the editable `_search` worktree). No change was needed in `pipelex-api`.++## Symptom++With the API running Temporal-enabled (`.pipelex/pipelex_override.toml` → `[temporal] is_enabled = true`, task queue `pipelex_dev`) and a worker up:++- `make bundle-validate BUNDLE=.../fashion_moodboard` → **HTTP 422**+- `make bundle-validate BUNDLE=.../joke_judge` → **OK**+- `make bundle-run BUNDLE=.../fashion_moodboard` (full run) → **OK**+- `pipelex validate bundle .../fashion_moodboard` (CLI) → **OK**++The 422 body:++```+ValidateBundleError (error_domain=input), 422+detail: "Dry run failed with 1 unexpected pipe failure(s):+ 'fashion_moodboard.create_moodboards': ... status=DryRunStatus.FAILURE+ error_message=\"Dry run failed for pipe 'fashion_moodboard.create_moodboards':+ Failed to execute workflow WfPipeRouter\""+```++The string **"Failed to execute workflow WfPipeRouter"** (no trailing detail) is produced only at `pipelex/temporal/tprl/workflow_caller.py:136`, the `(WorkflowAlreadyStartedError, RPCError)` branch of `WorkflowExecutor.execute_workflow` — i.e. a **top-level** Temporal workflow dispatch from the submitter side. A dry run should never reach that code.++## Root cause++The `/validate` path runs an **in-process** dry-run sweep, but nested controller sub-pipes leaked out to the Temporal router.++Flow: `api/routes/pipelex/validate.py` → `validate_bundle()` (`pipelex/pipeline/validate_bundle.py`) → `BundleValidator.validate_pipes()` (`pipelex/pipeline/bundle_validator.py`).++`BundleValidator` deliberately builds a **local, in-process** execution primitive so the sweep never touches Temporal — its own module docstring says so. **But that local router was only used for the _top-level_ pipe.** Every controller dispatches its nested sub-pipes through the **hub's** `get_pipe_router()`, not the injected router. This is broad — it is **not** specific to `PipeBatch`:++```python+# pipelex/pipe_controllers/batch/pipe_batch.py (_live_run_controller_pipe)+return await get_pipe_router().run(pipe_job=PipeJobFactory.make_pipe_job(...))++# pipelex/pipe_controllers/sub_pipe.py (SubPipe.run_pipe — used by EVERY PipeSequence step;+# also the inline `batch_over` path and the PipeCondition path)+pipe_output = await get_pipe_router().run(pipe_job=PipeJobFactory.make_pipe_job(...))+```++`get_pipe_router()` (`pipelex/hub.py`) returns a contextvar override **if one is set**, else the hub default. In the Temporal-enabled API process the hub default is the **`TemporalPipeRouter`** (wired in `pipelex/pipelex.py` when `get_config().temporal.is_enabled`). Called from the API process — i.e. **outside** any Temporal workflow (`is_in_temporal_workflow()` is False) — `TemporalPipeRouter._run_pipe_job` took the **top-level dispatch** branch → `executor.execute_workflow(WfPipeRouter, ...)` → failed → "Failed to execute workflow WfPipeRouter". The per-pipe dry run was marked `FAILURE`, `BundleValidator._aggregate` raised `DryRunError`, `validate_bundle` translated it to `ValidateBundleError`, and the global handler rendered 422.++`BundleValidator` built the in-process router but **never installed it as the `scoped_pipe_router(...)` override**, so the nested controllers fell through to the hub default (Temporal).++## FAIL vs. FALSE-PASS — the leak happened always; only some shapes 422'd++The leak (nested dispatch → `TemporalPipeRouter`) happened for **every** bundle with a controller. Whether the dry run **failed** depended on a Temporal **workflow-id collision**:++- `make_top_workflow_id` (`pipelex/temporal/temporal_manager.py`) is `f"{prefix}{pipeline_run_id}"` — derived from `pipeline_run_id` **alone**. The whole sweep shares **one** `dry_run_pipeline_id`, so **every** top-level Temporal dispatch in a sweep computed the **same** workflow id.+- `WorkflowExecutor.execute_workflow` **waits for completion**, and no `id_reuse_policy` is set → Temporal's default `ALLOW_DUPLICATE` (reuse allowed once the prior run has *closed*). So **serial** dispatches reusing that id are fine; only **concurrent** same-id dispatches collide → `WorkflowAlreadyStartedError` → `WorkflowExecutionError("Failed to execute workflow WfPipeRouter")`. (`WorkflowExecutionError` is a `PipelexError`, so it records as a dry-run `FAILURE` → 422, not a 500.)++The deciding structural factor is **standalone batch/parallel vs. batch-reached-as-a-sub-pipe**:++- A **standalone top-level `PipeBatch`/`PipeParallel`** the sweep dry-runs *directly* runs its top level on the local in-process router, but its **fan-out loop executes in the API process** (`_live_run_controller_pipe` → `gather_bounded`) and fires **N concurrent top-level Temporal dispatches** with the same id → collision → **FAILURE**. A mocked "multiple" list yields `nb_stuffs = 3`, so N≥2.+- A **batch reached as a sub-pipe** (inline `batch_over` step, or a batch nested under a sequence/parallel) is dispatched as a **single** top-level workflow — the fan-out into branches then happens **inside the worker** as child workflows (legal). Sequence steps run serially, so it's one dispatch at a time → **no concurrent collision** → it **passed** (still round-tripping Temporal, just not colliding — a false pass).++So `fashion_moodboard`'s `create_moodboards` (a standalone `PipeBatch`) 422'd, while a sequence-nested batch silently round-tripped Temporal and passed. The fix removes the Temporal round-trip for the whole sweep, fixing both the real failure **and** the silent false-pass leak.++## The fix as shipped++### Part 1 — scope the in-process router for the whole sweep (commit `3377babb`)++The codebase had already solved this exact leak for DIRECT mode in `pipelex/runtime_bridge/bridge.py::_run_direct`, with a comment describing the precise failure mode. `BundleValidator` now applies the same guard around its dry-run sweep:++1. `__init__` keeps the router instance instead of discarding it inside `PipeRun(...)`:+ ```python+ self._pipe_router = PipeRouter(observer=ObserverNoOp())+ self._pipe_run: PipeRunProtocol = PipeRun(pipe_router=self._pipe_router)+ ```+2. `validate_pipes` wraps the **per-pipe dry-run sweep loop** (step 4) in `scoped_pipe_router`:+ ```python+ with scoped_pipe_router(self._pipe_router):+ for pipe in sweepable_pipes:+ results[pipe.pipe_ref] = await self._classify_pipe(...)+ ```+ Wrapping `validate_pipes` covers all entry points (`acquire_and_validate`, `validate_current_library`, and the direct `validate_bundle` path all funnel through it). Steps 1–3 (wiring check, signature pre-pass, telemetry) don't run pipes, so they don't need the scope.++**Why it's safe:**++- `scoped_pipe_router` is **contextvar-scoped** and restores the prior value on exit, so concurrent `/validate` requests don't leak into each other.+- `PipeBatch` fans branches out via `gather_bounded` → `asyncio.gather`, whose Tasks copy the current context at creation (and creation happens inside the `with` scope), so each branch task inherits the override and resolves the in-process router too.++### Part 2 — `--temporal/--no-temporal` flag on `validate` (commit `b8eadeda`)++A `--temporal/--no-temporal` flag was added to `pipelex validate bundle`, `pipe`, and `method` (and therefore `validate --all`, which routes to `validate pipe`), giving parity with `pipelex run`. It threads through `_validate_core.execute_validate(temporal=...)` → `make_pipelex_for_cli(temporal_enabled=...)`, overriding `temporal.is_enabled` **for the boot only**.++**This flag is behavior-neutral for validation today.** The sweep always runs in-process regardless of the flag (Part 1 guarantees that). The flag does not change *what* validation does — it controls *how Pipelex boots*, which is the lever for exercising the "validation stays in-process even on a Temporal-enabled hub" contract without juggling a `pipelex_temporary_override.toml`. It is also forward-looking: once validation runs as a standalone Temporal activity ([`wip/dry-run-refactor/followup-temporal-validation-activity.md`](wip/dry-run-refactor/followup-temporal-validation-activity.md)), `--temporal` stops being a no-op and becomes the switch that dispatches the sweep through that activity.++## Reviewer note — the graph step was deliberately NOT changed++An earlier diagnosis claimed the best-effort **graph** step in `validate.py` — `dry_run_pipeline()` (`pipelex/pipe_run/dry_run_pipeline.py`) → `PipelexRunner(...).execute_pipeline(...)` — had "the same leak." **It does not**, and an attempt to "fix" it was **reverted**. The graph step does a **single** top-level dispatch of the main pipe — no concurrent same-id collision — so under Temporal it dispatches the one workflow to the worker, which (tracing enabled, `backend = "temporal_dynamodb"`) assembles the `GraphSpec` and returns it on `PipeOutput`. That is the intended distributed design: `pipelex-api` runs with tracing disabled (thin submitter); the **worker** owns tracing + graph assembly. Forcing it in-process broke graph generation (tracing off in the API → empty `GraphSpec` → graph dropped). **Do not "fix" the graph step to run in-process** — if a `/validate` graph is wanted without a worker, the lever is enabling tracing in the API, a separate deployment decision.++## Test coverage — all three layers in place++The contract — *the validation sweep never dispatches to Temporal* — is guarded at three complementary layers (none redundant):++| Layer | Where it runs | CI-automated | What it adds | Status |+|---|---|---|---|---|+| **Sentinel regression test** (no Temporal) | `tests/integration/pipelex/pipeline/test_bundle_validator.py::TestBundleValidatorIntegration::test_standalone_batch_sweep_scopes_in_process_router` | ✅ | Installs a *raising* hub-default router; sweeps a standalone `PipeBatch`; asserts SUCCESS with the hub default never touched. Deterministic, zero Temporal infra. | exists |+| **Mode-1 pytest** (real Temporal router) | `tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py::TestValidateSweepStaysInProcess` | ✅ | Resolves the **real** `TemporalPipeRouter` as the hub default (proving the config→hub-default *wiring* the sentinel bypasses); spies `WorkflowExecutor.execute_workflow` and asserts it's never called. | **new — this change** |+| **Mode-2 scenario** (3 separate processes) | `temporal-e2e-validate` skill, Tier 2c (agent-run) | ✗ | Production-faithful API↔worker boundary: `validate bundle --temporal` over a standalone `PipeBatch` exits 0 **and** the worker stays idle (no `WfPipeRouter` dispatch). Documents the GREEN/RED procedure. | **new — this change** |++The Mode-1 pytest is the cheaper automated catch (fast, deterministic, runs in CI); the Mode-2 scenario is the deployment-faithful demonstration and the only one that also exercises the genuinely cross-process surfaces (LibraryCrate propagation, serialization). Both were verified RED with the `scoped_pipe_router` scope removed and GREEN with it in place.++The repro bundle reuses an existing fixture (no new fixture): `tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds` already declares a standalone `type = "PipeBatch"` pipe (`batch_temporal_describe_topics`) whose direct sweep fans out — the exact shape that 422'd.++## Files in this change++Owned by **this** change (the rest of the diff belongs to PR #974 — see [Stacking](#fix-validate-dry-run-leaked-nested-controller-sub-pipes-to-temporal)):++- `pipelex/pipeline/bundle_validator.py` — the fix (`__init__` keeps `self._pipe_router`; `validate_pipes` wraps the sweep in `scoped_pipe_router`).+- `pipelex/cli/commands/validate/{_validate_core,bundle_cmd,method_cmd,pipe_cmd}.py` — the `--temporal/--no-temporal` flag.+- `tests/integration/pipelex/pipeline/test_bundle_validator.py` — sentinel regression test (+ siblings).+- `tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py` — **new** Mode-1 pytest.+- `tests/unit/pipelex/cli/test_validate_temporal_flag.py` — locks the CLI surface (each subcommand exposes `--temporal/--no-temporal`; `--help` short-circuits before boot).+- `.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md` — **new** Tier 2c scenario (+ Contents + Step 7 row).+- `CHANGELOG.md` — entry under `[Unreleased]`.+- `TODOS.md`, `wip/distributed-execution/validate-sweep-temporal-leak-repro.md` — this guide + the repro brief.++## How to verify (local setup)++API on :8081 (`make run`), Temporal server + worker up (from `pipelex-worker`), both on the editable `pipelex` checkout.++```bash+# was failing → now 200 with validated bundle+make bundle-validate BUNDLE=/Users/lchoquel/repos/Pipelex/pipelex-demos/mthds-wip/fashion_moodboard+# regression guard → still 200+make bundle-validate BUNDLE=/Users/lchoquel/repos/Pipelex/pipelex-demos/mthds-wip/joke_judge+```++CLI parity check (no override file needed): `pipelex validate bundle <bundle> --temporal` boots Temporal-enabled yet the sweep stays in-process — exits 0 with no workflow dispatched to the worker. See [`wip/distributed-execution/validate-sweep-temporal-leak-repro.md`](wip/distributed-execution/validate-sweep-temporal-leak-repro.md) for the full GREEN/RED procedure (now implemented as Tier 2c in the `temporal-e2e-validate` skill).diff --git a/docs/distributed-execution/task-routing.md b/docs/distributed-execution/task-routing.mdindex 56248a0a3..fd312c154 100644--- a/docs/distributed-execution/task-routing.md+++ b/docs/distributed-execution/task-routing.md@@ -51,6 +51,8 @@ The **routing key** is the runtime handle the activity is about to call:| `act_llm_gen_object_list` | LLM model handle | The `llm_handle` on the call || `act_img_gen_images` | `img_gen_handle` | The image generation backend || `act_extract_gen_extract_pages` | `extract_handle` | The OCR / extract backend |+| `act_search_gen_sourced_answer` | `search_handle` | The web search backend |+| `act_search_gen_structured` | `search_handle` | The web search backend || `act_jinja2_gen_text` | (none — pass `None`) | Template rendering has no handle || `act_render_page_views` | (none — pass `None`) | Page-view rendering has no handle |diff --git a/docs/distributed-execution/workers.md b/docs/distributed-execution/workers.mdindex 6fc61e707..f14f0a4c9 100644--- a/docs/distributed-execution/workers.md+++ b/docs/distributed-execution/workers.md@@ -75,6 +75,7 @@ Built-in scopes:| `runner-img-gen` | None | `act_img_gen_images`, `act_render_page_views` | One pool dedicated to image generation. || `runner-extract` | None | `act_extract_gen_extract_pages`, `act_render_page_views` | One pool dedicated to OCR / document extraction. || `runner-jinja2` | None | `act_jinja2_gen_text` | One pool dedicated to template rendering (rare; usually fine in-band). |+| `runner-search` | None | `act_search_gen_sourced_answer`, `act_search_gen_structured` | One pool dedicated to web search calls. |`act_render_page_views` is intentionally registered under **both** `runner-img-gen` and `runner-extract` so neither pool breaks when the activity is routed to either. The `runner-llm` scope splits all three LLM activity types (text, object, object-list) onto the same pool — they share the same provider connections so splitting further would only add operational overhead.diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.mdindex bec00a2c0..2b3005fb9 100644--- a/docs/under-the-hood/distributed-content-generation.md+++ b/docs/under-the-hood/distributed-content-generation.md@@ -7,7 +7,10 @@ description: "How Pipelex serializes dynamic Pydantic models and large binary coThis page is for contributors working on Pipelex internals. If you're deploying or operating Pipelex on Temporal, see the user-facing [Distributed Execution with Temporal](../distributed-execution/temporal/index.md) guide instead.-This page covers the serialization mechanisms that enable content generation (LLM structured output, image generation, PDF extraction) to run across Temporal worker processes. For the broader Temporal architecture — LibraryCrate propagation, deferred hydration, per-workflow scoping — see [Temporal Integration](./temporal-integration.md).+This page covers the serialization mechanisms that enable content generation (LLM structured output, image generation, PDF extraction, web search) to run across Temporal worker processes. For the broader Temporal architecture — LibraryCrate propagation, deferred hydration, per-workflow scoping — see [Temporal Integration](./temporal-integration.md).++!!! note "Every inference leaf goes through the same seam"+ All inference operators dispatch their leaf call through the swappable `ContentGenerator` abstraction: direct inline (`ContentGenerator`), as a Temporal activity (`ContentGeneratorInWorkflow`), or as a dry mock (`ContentGeneratorDry`). Web search (`PipeSearch`) joined this seam last — its `act_search_gen_*` activities make search-on-Temporal replay-safe (the result is recorded in workflow history) and let search failures cross the workflow boundary as classified errors. An operator that runs its leaf inline instead would re-execute it on every replay and, on failure, hang the submitter with an unclassified workflow-task error.---@@ -137,6 +140,10 @@ sequenceDiagramS->>S: model_validate() → original type```+### Structured web search reuses the same mechanism++Structured web search (`PipeSearch` with a non-text output concept) faces the identical dynamic-class problem and solves it the same way. `SearchObjectAssignment` mirrors `ObjectAssignment` — it ships `output_class_name` + `output_class_schema` alongside the `SearchAssignment`, the activity reconstructs a throwaway class via `SchemaToModelFactory` to drive the provider call, and the activity returns the **raw result dict**. The submitter re-validates that dict against the original output class (`output_structure_class.model_validate(result_dict)`) — a pure, deterministic step that keeps the dynamic class on the submitter side and never ships it across the boundary. The sourced-answer path (`make_search_sourced_answer`) has no dynamic class at all: it returns a `SearchResultContent`, a native serializable model.+---## Large Payload Management@@ -179,6 +186,8 @@ async def act_img_gen_images(img_gen_assignment: ImgGenAssignment) -> list[Image| Rendered page views | `act_render_page_views` | PDF page renders → S3 | `list[ImageContent]` with URIs || LLM text | `act_llm_gen_text` | Nothing stored (text is small) | Plain `str` || LLM object | `act_llm_gen_object` | Nothing stored (JSON is small) | `BaseModel` + `__kajson_class_source__` in metadata |+| Search sourced answer | `act_search_gen_sourced_answer` | Nothing stored (answer + source refs are small) | `SearchResultContent` (answer + `DocumentContent` sources) |+| Search structured | `act_search_gen_structured` | Nothing stored (JSON is small) | Raw `dict`, re-validated against the output class on the submitter |---@@ -215,13 +224,15 @@ Resolution order:| Component | File ||-----------|------|| Assignment models (schema embedding) | `pipelex/cogt/content_generation/assignment_models.py` |-| Schema-to-model reconstruction | `pipelex/cogt/content_generation/schema_to_model.py` |+| Schema-to-model reconstruction | `pipelex/cogt/content_generation/schema_to_model_factory.py` || Content generator (type bridge) | `pipelex/cogt/content_generation/content_generator.py` || LLM generation functions | `pipelex/cogt/content_generation/llm_generate.py` |+| Search generation functions | `pipelex/cogt/content_generation/search_generate.py` || Generated content factory (storage) | `pipelex/cogt/content_generation/generated_content_factory.py` || Temporal data converter | `pipelex/temporal/temporal_data_converter.py` || Image generation activity | `pipelex/temporal/tprl_content_generation/act_img_gen_generate.py` || Extract activity | `pipelex/temporal/tprl_content_generation/act_extract_generate.py` |+| Search activity | `pipelex/temporal/tprl_content_generation/act_search_generate.py` || Page view rendering activity | `pipelex/temporal/tprl_content_generation/act_render_page_views.py` || Kajson serialization | `kajson` (external PyPI package) || ImageContent model | `pipelex/core/stuffs/image_content.py` |diff --git a/pipelex/cli/commands/validate/_validate_core.py b/pipelex/cli/commands/validate/_validate_core.pyindex c87832424..f73357992 100644--- a/pipelex/cli/commands/validate/_validate_core.py+++ b/pipelex/cli/commands/validate/_validate_core.py@@ -163,9 +163,17 @@ def execute_validate(library_dirs: list[Path] | None,telemetry_command_label: str = COMMAND,allow_signatures: bool = False,+ temporal: bool | None = None,) -> None:- """Synchronous entry point wrapping the async validation with Pipelex setup/teardown."""- make_pipelex_for_cli(context=ErrorContext.VALIDATION, needs_inference=False, needs_model_specs=True)+ """Synchronous entry point wrapping the async validation with Pipelex setup/teardown.++ ``temporal`` overrides ``temporal.is_enabled`` for the boot (mirroring ``run``). The+ validation sweep itself always runs in-process — under a Temporal-enabled hub it scopes+ its own in-process router so nested controller sub-pipes do not dispatch to Temporal — so+ this flag does not change *what* validation does; it controls how Pipelex boots, which is+ the lever for exercising the "validation stays in-process on a Temporal backend" contract.+ """+ make_pipelex_for_cli(context=ErrorContext.VALIDATION, needs_inference=False, needs_model_specs=True, temporal_enabled=temporal)try:with get_telemetry_manager().telemetry_context():diff --git a/pipelex/cli/commands/validate/bundle_cmd.py b/pipelex/cli/commands/validate/bundle_cmd.pyindex 5da47ad46..4e125f579 100644--- a/pipelex/cli/commands/validate/bundle_cmd.py+++ b/pipelex/cli/commands/validate/bundle_cmd.py@@ -30,6 +30,14 @@ def validate_bundle_cmd(help="Accept PipeSignature placeholders in the dependency graph (lenient mode).",),] = False,+ temporal: Annotated[+ bool | None,+ typer.Option(+ "--temporal/--no-temporal",+ help="Override config temporal.is_enabled for the boot. The sweep stays in-process either way; "+ "use it to verify validation does not dispatch to Temporal under a Temporal-enabled hub.",+ ),+ ] = None,) -> None:"""Validate a bundle file (.mthds) or pipeline directory.@@ -96,4 +104,5 @@ def validate_bundle_cmd(library_dirs=library_dirs_paths,telemetry_command_label=f"{COMMAND} bundle",allow_signatures=allow_signatures,+ temporal=temporal,)diff --git a/pipelex/cli/commands/validate/method_cmd.py b/pipelex/cli/commands/validate/method_cmd.pyindex 33372891b..cd8a46ec7 100644--- a/pipelex/cli/commands/validate/method_cmd.py+++ b/pipelex/cli/commands/validate/method_cmd.py@@ -26,6 +26,14 @@ def validate_method_cmd(help="Directory to search for pipe definitions (.mthds files). Can be specified multiple times.",),] = None,+ temporal: Annotated[+ bool | None,+ typer.Option(+ "--temporal/--no-temporal",+ help="Override config temporal.is_enabled for the boot. The sweep stays in-process either way; "+ "use it to verify validation does not dispatch to Temporal under a Temporal-enabled hub.",+ ),+ ] = None,) -> None:"""Validate an installed method by name.@@ -51,4 +59,5 @@ def validate_method_cmd(bundle_path=None,library_dirs=library_dirs_paths,telemetry_command_label=f"{COMMAND} method",+ temporal=temporal,)diff --git a/pipelex/cli/commands/validate/pipe_cmd.py b/pipelex/cli/commands/validate/pipe_cmd.pyindex aad45009b..dade273ec 100644--- a/pipelex/cli/commands/validate/pipe_cmd.py+++ b/pipelex/cli/commands/validate/pipe_cmd.py@@ -41,6 +41,14 @@ def validate_pipe_cmd(help="Accept PipeSignature placeholders in the dependency graph (lenient mode).",),] = False,+ temporal: Annotated[+ bool | None,+ typer.Option(+ "--temporal/--no-temporal",+ help="Override config temporal.is_enabled for the boot. The sweep stays in-process either way; "+ "use it to verify validation does not dispatch to Temporal under a Temporal-enabled hub.",+ ),+ ] = None,) -> None:"""Validate and dry run a pipe by code, or all pipes.@@ -65,6 +73,7 @@ def validate_pipe_cmd(library_dirs=library_dirs_paths,needs_inference=False,needs_model_specs=True,+ temporal_enabled=temporal,)do_validate_all_libraries_and_dry_run(library_dirs=library_dirs_paths,@@ -118,4 +127,5 @@ def validate_pipe_cmd(library_dirs=library_dirs_paths,telemetry_command_label=f"{COMMAND} pipe",allow_signatures=allow_signatures,+ temporal=temporal,)diff --git a/pipelex/cogt/content_generation/assignment_models.py b/pipelex/cogt/content_generation/assignment_models.pyindex 660318523..73380b4ad 100644--- a/pipelex/cogt/content_generation/assignment_models.py+++ b/pipelex/cogt/content_generation/assignment_models.py@@ -10,6 +10,7 @@ from pipelex.cogt.img_gen.img_gen_prompt import ImgGenPromptfrom pipelex.cogt.llm.llm_job_components import LLMJobParamsfrom pipelex.cogt.llm.llm_prompt import LLMPromptfrom pipelex.cogt.llm.llm_setting import LLMSetting+from pipelex.cogt.search.search_setting import SearchSettingfrom pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.templating_style import TemplatingStylefrom pipelex.pipeline.job_metadata import JobMetadata@@ -107,3 +108,49 @@ class RenderPageViewsAssignment(BaseModel):job_metadata: JobMetadatadocument_uri: strpage_views_dpi: int+++class SearchAssignment(BaseModel):+ """Serializable unit for a single web-search leaf call.++ Carries everything the framework-agnostic ``search_generate`` core needs to rebuild the+ ``SearchJob`` on the other side of the Temporal boundary: the rendered ``query``, the fully+ resolved ``search_setting`` (its ``model`` is the resolved provider handle, also the routing+ key), and the per-call domain/date overrides. Mirrors ``LLMAssignment`` for the LLM leaf.+ """++ job_metadata: JobMetadata+ query: str+ search_setting: SearchSetting+ include_domains: list[str] | None = None+ exclude_domains: list[str] | None = None+ from_date: str | None = None+ to_date: str | None = None++ @property+ def search_handle(self) -> str:+ return self.search_setting.model+++class SearchObjectAssignment(BaseModel):+ """Structured-search counterpart of ``ObjectAssignment``.++ Ships the output structure's JSON schema (not the live class) across the boundary so the+ activity can reconstruct a throwaway class via ``SchemaToModelFactory`` for the provider call.+ The submitter re-validates the returned dict against the original class.+ """++ output_class_name: str+ output_class_schema: dict[str, Any]+ search_assignment: SearchAssignment++ @staticmethod+ def make_for_class(+ output_class: type[BaseModel],+ search_assignment: SearchAssignment,+ ) -> "SearchObjectAssignment":+ return SearchObjectAssignment(+ output_class_name=output_class.__name__,+ output_class_schema=output_class.model_json_schema(),+ search_assignment=search_assignment,+ )diff --git a/pipelex/cogt/content_generation/content_generator.py b/pipelex/cogt/content_generation/content_generator.pyindex 6ad0de243..9265fbe19 100644--- a/pipelex/cogt/content_generation/content_generator.py+++ b/pipelex/cogt/content_generation/content_generator.py@@ -9,6 +9,8 @@ from pipelex.cogt.content_generation.assignment_models import (ImgGenAssignment,LLMAssignment,ObjectAssignment,+ SearchAssignment,+ SearchObjectAssignment,TemplatingAssignment,)from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata@@ -17,6 +19,7 @@ from pipelex.cogt.content_generation.extract_generate import extract_gen_pagesfrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactoryfrom pipelex.cogt.content_generation.img_gen_generate import img_gen_image_list, img_gen_single_imagefrom pipelex.cogt.content_generation.llm_generate import llm_gen_object, llm_gen_object_list, llm_gen_text+from pipelex.cogt.content_generation.search_generate import search_gen_sourced_answer, search_gen_structuredfrom pipelex.cogt.content_generation.templating_generate import templating_gen_textfrom pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParams@@ -31,6 +34,7 @@ from pipelex.cogt.templating.templating_style import TemplatingStylefrom pipelex.config import get_configfrom pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.page_content import PageContent+from pipelex.core.stuffs.search_result_content import SearchResultContentfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.tools.misc.image_utils import ImageFormatfrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVar@@ -317,3 +321,24 @@ class ContentGenerator(ContentGeneratorProtocol):page_content.page_view = page_view_contents.pop(0)return page_contents++ @override+ async def make_search_sourced_answer(+ self,+ search_assignment: SearchAssignment,+ ) -> SearchResultContent:+ return await search_gen_sourced_answer(search_assignment=search_assignment)++ @override+ async def make_search_structured(+ self,+ output_structure_class: type[BaseModelTypeVar],+ search_assignment: SearchAssignment,+ ) -> BaseModelTypeVar:+ result_dict = await search_gen_structured(+ search_object_assignment=SearchObjectAssignment.make_for_class(+ output_class=output_structure_class,+ search_assignment=search_assignment,+ ),+ )+ return output_structure_class.model_validate(result_dict)diff --git a/pipelex/cogt/content_generation/content_generator_dry.py b/pipelex/cogt/content_generation/content_generator_dry.pyindex eea614c22..3b0b78a30 100644--- a/pipelex/cogt/content_generation/content_generator_dry.py+++ b/pipelex/cogt/content_generation/content_generator_dry.py@@ -3,8 +3,10 @@ from typing import Anyfrom typing_extensions import overridefrom pipelex import log+from pipelex.cogt.content_generation.assignment_models import SearchAssignmentfrom pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadatafrom pipelex.cogt.content_generation.dry_mock import build_mock_object, report_dry_llm_job+from pipelex.cogt.content_generation.dry_run_factory import DryRunFactoryfrom pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParamsfrom pipelex.cogt.extract.extract_output import ExtractOutput, Page@@ -17,8 +19,10 @@ from pipelex.cogt.llm.llm_setting import LLMSettingfrom pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.templating_style import TemplatingStylefrom pipelex.config import get_config+from pipelex.core.stuffs.document_content import DocumentContentfrom pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.page_content import PageContent+from pipelex.core.stuffs.search_result_content import SearchResultContentfrom pipelex.core.stuffs.text_and_images_content import TextAndImagesContentfrom pipelex.core.stuffs.text_content import TextContentfrom pipelex.pipeline.job_metadata import JobMetadata@@ -291,3 +295,25 @@ class ContentGeneratorDry(ContentGeneratorProtocol):page_content.page_view = page_view_contents.pop(0)return page_contents++ @override+ async def make_search_sourced_answer(+ self,+ search_assignment: SearchAssignment,+ ) -> SearchResultContent:+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.make_search_sourced_answer")+ nb_sources = get_config().pipelex.dry_run_config.nb_list_items+ doc_factory = DryRunFactory.make_dry_run_factory(DocumentContent)+ mock_sources = [doc_factory.build() for _ in range(nb_sources)]+ search_result_factory = DryRunFactory.make_dry_run_factory(SearchResultContent)+ return search_result_factory.build(sources=mock_sources)++ @override+ async def make_search_structured(+ self,+ output_structure_class: type[BaseModelTypeVar],+ search_assignment: SearchAssignment,+ ) -> BaseModelTypeVar:+ log.verbose(f"🤡 DRY RUN: {self.__class__.__name__}.make_search_structured")+ structured_factory = DryRunFactory.make_dry_run_factory(output_structure_class)+ return structured_factory.build()diff --git a/pipelex/cogt/content_generation/content_generator_protocol.py b/pipelex/cogt/content_generation/content_generator_protocol.pyindex a66b9e677..7a9f01e60 100644--- a/pipelex/cogt/content_generation/content_generator_protocol.py+++ b/pipelex/cogt/content_generation/content_generator_protocol.py@@ -2,6 +2,7 @@ from collections.abc import Awaitable, Callable, Coroutinefrom functools import wrapsfrom typing import Any, ParamSpec, Protocol, TypeVar+from pipelex.cogt.content_generation.assignment_models import SearchAssignmentfrom pipelex.cogt.extract.extract_input import ExtractInputfrom pipelex.cogt.extract.extract_job_components import ExtractJobConfig, ExtractJobParamsfrom pipelex.cogt.extract.extract_output import ExtractOutput@@ -14,6 +15,7 @@ from pipelex.cogt.templating.template_category import TemplateCategoryfrom pipelex.cogt.templating.templating_style import TemplatingStylefrom pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.page_content import PageContent+from pipelex.core.stuffs.search_result_content import SearchResultContentfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVar@@ -125,3 +127,14 @@ class ContentGeneratorProtocol(Protocol):extract_job_params: ExtractJobParams,extract_job_config: ExtractJobConfig,) -> Coroutine[Any, Any, list[PageContent]]: ...++ def make_search_sourced_answer(+ self,+ search_assignment: SearchAssignment,+ ) -> Coroutine[Any, Any, SearchResultContent]: ...++ def make_search_structured(+ self,+ output_structure_class: type[BaseModelTypeVar],+ search_assignment: SearchAssignment,+ ) -> Coroutine[Any, Any, BaseModelTypeVar]: ...diff --git a/pipelex/cogt/content_generation/search_generate.py b/pipelex/cogt/content_generation/search_generate.pynew file mode 100644index 000000000..d61f88900--- /dev/null+++ b/pipelex/cogt/content_generation/search_generate.py@@ -0,0 +1,76 @@+"""Framework-agnostic web-search leaf, sibling of ``llm_generate``.++These coroutines are the single place a ``PipeSearch`` step's provider spend happens. They take a+serializable ``SearchAssignment`` / ``SearchObjectAssignment``, rebuild the ``SearchJob``, resolve the+worker from the model handle, and run it. The direct ``ContentGenerator`` calls them inline; the Temporal+``act_search_*`` activity calls them inside an activity so the result is recorded in workflow history and+any failure is converted to a terminal ``ApplicationError`` (instead of running inline on the workflow+loop, which left search failures hanging the submitter — see ``wip/`` brief).+"""++from typing import Any++from pipelex.cogt.content_generation.assignment_models import SearchAssignment, SearchObjectAssignment+from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedError+from pipelex.cogt.content_generation.schema_to_model_factory import SchemaToModelFactory+from pipelex.cogt.model_backends.model_type import ModelType+from pipelex.cogt.search.search_job import SearchJob+from pipelex.cogt.search.search_job_factory import SearchJobFactory+from pipelex.cogt.search.search_worker_abstract import SearchWorkerAbstract+from pipelex.cogt.search.search_worker_factory import SearchWorkerFactory+from pipelex.core.stuffs.search_result_content import SearchResultContent+from pipelex.hub import get_model_deck+++def _guard_no_mock_inference(search_assignment: SearchAssignment) -> None:+ # Web search has no leaf-level mock — under --mock-inference (run_mode stays LIVE) this live+ # path runs, so fail loud rather than silently hit the real provider (use --dry-run instead).+ if search_assignment.job_metadata.is_mock_inference:+ error = MockInferenceUnsupportedError.for_operation("web search (PipeSearch)")+ raise error+++def _make_search_worker(search_assignment: SearchAssignment) -> SearchWorkerAbstract:+ model_deck = get_model_deck()+ inference_model = model_deck.get_required_inference_model(+ model_handle=search_assignment.search_setting.model,+ model_type=ModelType.SEARCH,+ )+ return SearchWorkerFactory.make_search_worker(inference_model=inference_model)+++def _make_search_job(search_assignment: SearchAssignment) -> SearchJob:+ return SearchJobFactory.make_search_job(+ query=search_assignment.query,+ search_setting=search_assignment.search_setting,+ job_metadata=search_assignment.job_metadata,+ include_domains=search_assignment.include_domains,+ exclude_domains=search_assignment.exclude_domains,+ from_date=search_assignment.from_date,+ to_date=search_assignment.to_date,+ )+++async def search_gen_sourced_answer(search_assignment: SearchAssignment) -> SearchResultContent:+ _guard_no_mock_inference(search_assignment)+ worker = _make_search_worker(search_assignment)+ search_job = _make_search_job(search_assignment)+ return await worker.search_sourced_answer(search_job=search_job)+++async def search_gen_structured(search_object_assignment: SearchObjectAssignment) -> dict[str, Any]:+ """Run a structured search, returning the raw result dict.++ The dict is re-validated against the original output structure class by the submitter+ (``ContentGenerator`` / ``ContentGeneratorInWorkflow``) — that step is pure and deterministic,+ which sidesteps shipping a dynamic output class across the Temporal boundary.+ """+ search_assignment = search_object_assignment.search_assignment+ _guard_no_mock_inference(search_assignment)+ worker = _make_search_worker(search_assignment)+ search_job = _make_search_job(search_assignment)+ output_class = SchemaToModelFactory.make_from_json_schema(+ schema=search_object_assignment.output_class_schema,+ class_name=search_object_assignment.output_class_name,+ )+ return await worker.search_structured(search_job=search_job, schema=output_class)diff --git a/pipelex/pipe_operators/search/pipe_search.py b/pipelex/pipe_operators/search/pipe_search.pyindex 7c79fba07..534726e32 100644--- a/pipelex/pipe_operators/search/pipe_search.py+++ b/pipelex/pipe_operators/search/pipe_search.py@@ -3,23 +3,20 @@ from typing import TYPE_CHECKING, Any, Literalfrom typing_extensions import overridefrom pipelex import log-from pipelex.cogt.content_generation.dry_run_factory import DryRunFactory-from pipelex.cogt.content_generation.exceptions import MockInferenceUnsupportedError+from pipelex.cogt.content_generation.assignment_models import SearchAssignment+from pipelex.cogt.content_generation.content_generator_dry import ContentGeneratorDry+from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocolfrom pipelex.cogt.exceptions import ModelChoiceNotFoundErrorfrom pipelex.cogt.model_backends.model_type import ModelTypefrom pipelex.cogt.models.model_deck_check import check_search_choice_with_deck-from pipelex.cogt.search.search_job_factory import SearchJobFactoryfrom pipelex.cogt.search.search_setting import SearchModelChoice, SearchSetting-from pipelex.cogt.search.search_worker_factory import SearchWorkerFactoryfrom pipelex.cogt.templating.template_blueprint import TemplateBlueprintfrom pipelex.cogt.templating.template_rendering import render_templatefrom pipelex.core.memory.working_memory import WorkingMemoryfrom pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecsfrom pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.core.stuffs.document_content import DocumentContent-from pipelex.core.stuffs.search_result_content import SearchResultContentfrom pipelex.core.stuffs.stuff_factory import StuffFactory-from pipelex.hub import get_model_deck+from pipelex.hub import get_content_generator, get_model_deckfrom pipelex.pipe_operators.pipe_operator import PipeOperatorfrom pipelex.pipe_run.pipe_run_params import PipeRunParamsfrom pipelex.pipeline.job_metadata import JobMetadata@@ -82,13 +79,9 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):working_memory: WorkingMemory,pipe_run_params: PipeRunParams,output_name: str | None = None,+ content_generator: ContentGeneratorProtocol | None = None,) -> PipeSearchOutput:- # Web search has no leaf-level mock and (unlike LLM/img-gen/extract) no cogt content_generation- # leaf — its provider spend happens right here. Under --mock-inference (run_mode stays LIVE) this- # live path runs, so guard here: fail loud rather than silently hit the real search provider.- if job_metadata.is_mock_inference:- error = MockInferenceUnsupportedError.for_operation("web search (PipeSearch)")- raise error+ content_generator = content_generator or get_content_generator()# 0. Log the search runsearch_choice_desc = self.search_choice or "default"@@ -106,7 +99,8 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):search_choice: SearchModelChoice = self.search_choice or model_deck.search_choice_defaultsearch_setting: SearchSetting = model_deck.get_search_setting(search_choice=search_choice)- # 3. Resolve the model handle (waterfalls/aliases → actual provider/variant handle)+ # 3. Resolve the model handle (waterfalls/aliases → actual provider/variant handle). Pin+ # search_setting.model to the resolved handle so it doubles as the Temporal routing key.inference_model = model_deck.get_required_inference_model(model_handle=search_setting.model, model_type=ModelType.SEARCH)resolved_model_handle = inference_model.nameif resolved_model_handle != search_setting.model:@@ -118,30 +112,31 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):if self.max_results_override is not None:search_setting = search_setting.model_copy(update={"max_results": self.max_results_override})- # 5. Get search worker from factory- worker = SearchWorkerFactory.make_search_worker(inference_model=inference_model)-- # 6. Create search job- search_job = SearchJobFactory.make_search_job(+ # 5. Build the serializable assignment and run the search behind the content-generation seam.+ # The leaf goes through the same swappable generator as LLM/img-gen/extract: direct inline, a+ # Temporal activity when in-workflow, or a dry mock. That makes search-on-Temporal replay-safe+ # and lets its failures cross the workflow boundary as classified errors instead of hanging.+ search_assignment = SearchAssignment(+ job_metadata=job_metadata,query=query_text,search_setting=search_setting,- job_metadata=job_metadata,include_domains=self.include_domains,exclude_domains=self.exclude_domains,from_date=self.from_date,to_date=self.to_date,)- # 7. Execute search based on output typecontent: StuffContentif not self.is_structured_output:- content = await worker.search_sourced_answer(search_job=search_job)+ content = await content_generator.make_search_sourced_answer(search_assignment=search_assignment)else:output_structure_class = self.output.concept.get_structure_class()- result_dict = await worker.search_structured(search_job=search_job, schema=output_structure_class)- content = output_structure_class.model_validate(result_dict)+ content = await content_generator.make_search_structured(+ output_structure_class=output_structure_class,+ search_assignment=search_assignment,+ )- # 8. Create Stuff, set in working memory, and return output+ # 6. Create Stuff, set in working memory, and return outputoutput_stuff = StuffFactory.make_stuff(name=output_name,concept=self.output.concept,@@ -172,35 +167,14 @@ class PipeSearch(PipeOperator[PipeSearchOutput]):pipe_run_params: PipeRunParams,output_name: str | None = None,) -> PipeSearchOutput:- content: StuffContent- if not self.is_structured_output:- doc_factory = DryRunFactory.make_dry_run_factory(DocumentContent)- mock_sources = [doc_factory.build() for _ in range(3)]- search_result_factory = DryRunFactory.make_dry_run_factory(SearchResultContent)- content = search_result_factory.build(sources=mock_sources)- else:- output_structure_class = self.output.concept.get_structure_class()- structured_factory = DryRunFactory.make_dry_run_factory(output_structure_class)- content = structured_factory.build()-- output_stuff = StuffFactory.make_stuff(- name=output_name,- concept=self.output.concept,- content=content,- )- working_memory.set_new_main_stuff(- stuff=output_stuff,- name=output_name,- )- execution_data_dict: dict[str, Any] = {- "rendered_query": "mock",- "resolved_model": "mock",- "is_structured_output": self.is_structured_output,- }- self._register_execution_data(job_metadata, execution_data_dict)- return PipeSearchOutput(+ # Dry run reuses the live path with the dry content generator — identical to PipeLLM — so the+ # search mock now lives behind the same seam (ContentGeneratorDry.make_search_*) as every leaf.+ return await self._live_run_operator_pipe(+ job_metadata=job_metadata,working_memory=working_memory,- pipeline_run_id=job_metadata.pipeline_run_id,+ pipe_run_params=pipe_run_params,+ output_name=output_name,+ content_generator=ContentGeneratorDry(),)@overridediff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex 5f416afff..0c26868d3 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -668,3 +668,15 @@ excluded_workflows = []excluded_activities = []disable_all_workflows = truedisable_all_activities = false++[temporal.worker_scopes.scopes.runner-search]+required_tasks_packs = []+required_workflows = []+required_activities = [+ "act_search_gen_sourced_answer",+ "act_search_gen_structured",+]+excluded_workflows = []+excluded_activities = []+disable_all_workflows = true+disable_all_activities = falsediff --git a/pipelex/pipeline/bundle_validator.py b/pipelex/pipeline/bundle_validator.pyindex 67a89302e..5211cb2b8 100644--- a/pipelex/pipeline/bundle_validator.py+++ b/pipelex/pipeline/bundle_validator.py@@ -43,6 +43,7 @@ from pipelex.hub import (get_library_manager,get_pipe_library,get_telemetry_manager,+ scoped_pipe_router,set_current_library,)from pipelex.libraries.pipe.exceptions import PipeNotFoundError@@ -103,7 +104,12 @@ class BundleValidator:def __init__(self) -> None:# Locally-constructed, direct execution primitive — NOT get_pipe_run() (see module docstring).# The ObserverNoOp keeps the sweep observation-silent; validation surfaces its own report.- self._pipe_run: PipeRunProtocol = PipeRun(pipe_router=PipeRouter(observer=ObserverNoOp()))+ # Keep the router instance (don't discard it inside PipeRun): the sweep installs it as the active+ # router via scoped_pipe_router (see validate_pipes) so nested controller sub-pipes — which+ # dispatch through get_pipe_router() — resolve THIS in-process router instead of the hub default.+ # Mirrors runtime_bridge.bridge._run_direct.+ self._pipe_router = PipeRouter(observer=ObserverNoOp())+ self._pipe_run: PipeRunProtocol = PipeRun(pipe_router=self._pipe_router)async def acquire_and_validate(self,@@ -235,10 +241,20 @@ class BundleValidator:mock_inputs=True,)dry_run_pipeline_id = f"dry_run_{PipelineFactory.make_pipeline_run_id()}"- for pipe in sweepable_pipes:- results[pipe.pipe_ref] = await self._classify_pipe(- pipe=pipe, library_id=library_id, execution_config=execution_config, dry_run_pipeline_id=dry_run_pipeline_id- )+ # Install the in-process router as the active router for the WHOLE sweep so nested controller+ # sub-pipes — which dispatch through get_pipe_router() — resolve THIS router instead of falling+ # back to the hub default. Under a Temporal-enabled hub the default is the Temporal router, so+ # without the scope a controller (a PipeBatch/PipeParallel fan-out, or any PipeSequence step)+ # would leak its nested pipes to Temporal — turning a no-cost in-process dry run into real+ # top-level workflow dispatches (HTTP 422). Mirrors runtime_bridge.bridge._run_direct. The+ # scope is contextvar-based, so concurrent /validate sweeps don't cross-contaminate, and the+ # asyncio tasks a batch fan-out spawns copy the context at creation (inside this scope) so they+ # inherit the override too.+ with scoped_pipe_router(self._pipe_router):+ for pipe in sweepable_pipes:+ results[pipe.pipe_ref] = await self._classify_pipe(+ pipe=pipe, library_id=library_id, execution_config=execution_config, dry_run_pipeline_id=dry_run_pipeline_id+ )# 5. Aggregate + report.return self._aggregate(results=results, start_time=start_time)diff --git a/pipelex/temporal/tasks.py b/pipelex/temporal/tasks.pyindex 3aeabb6fa..795f71e03 100644--- a/pipelex/temporal/tasks.py+++ b/pipelex/temporal/tasks.py@@ -6,6 +6,7 @@ from pipelex.temporal.tprl_content_generation.act_img_gen_generate import act_imfrom pipelex.temporal.tprl_content_generation.act_jinja2_generate import act_jinja2_gen_textfrom pipelex.temporal.tprl_content_generation.act_llm_generate import act_llm_gen_object, act_llm_gen_object_list, act_llm_gen_textfrom pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views+from pipelex.temporal.tprl_content_generation.act_search_generate import act_search_gen_sourced_answer, act_search_gen_structuredfrom pipelex.temporal.tprl_pipe.act_assemble_tracing import act_assemble_tracingfrom pipelex.temporal.tprl_pipe.act_deliver import act_deliverfrom pipelex.temporal.tprl_pipe.act_flush_trace_events import act_flush_trace_events@@ -31,6 +32,8 @@ class Tasks:act_jinja2_gen_text,act_extract_gen_extract_pages,act_render_page_views,+ act_search_gen_sourced_answer,+ act_search_gen_structured,],),PackName.PIPE: TaskPack(diff --git a/pipelex/temporal/tprl_content_generation/act_search_generate.py b/pipelex/temporal/tprl_content_generation/act_search_generate.pynew file mode 100644index 000000000..5df84925c--- /dev/null+++ b/pipelex/temporal/tprl_content_generation/act_search_generate.py@@ -0,0 +1,23 @@+from typing import Any++from temporalio import activity++from pipelex import log+from pipelex.cogt.content_generation.assignment_models import SearchAssignment, SearchObjectAssignment+from pipelex.cogt.content_generation.search_generate import search_gen_sourced_answer, search_gen_structured+from pipelex.core.stuffs.search_result_content import SearchResultContent+from pipelex.temporal.tprl.activity_error_boundary import convert_pipelex_errors+++@convert_pipelex_errors+async def act_search_gen_sourced_answer(search_assignment: SearchAssignment) -> SearchResultContent:+ log.dev("act_search_gen_sourced_answer")+ return await search_gen_sourced_answer(search_assignment=search_assignment)+++@convert_pipelex_errors+async def act_search_gen_structured(search_object_assignment: SearchObjectAssignment) -> dict[str, Any]:+ log.dev("act_search_gen_structured")+ return await search_gen_structured(search_object_assignment=search_object_assignment)diff --git a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py b/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.pyindex 88b30529d..01283983f 100644--- a/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py+++ b/pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py@@ -12,6 +12,8 @@ from pipelex.cogt.content_generation.assignment_models import (LLMAssignment,ObjectAssignment,RenderPageViewsAssignment,+ SearchAssignment,+ SearchObjectAssignment,TemplatingAssignment,)from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol, update_job_metadata@@ -30,6 +32,7 @@ from pipelex.cogt.templating.templating_style import TemplatingStylefrom pipelex.config import get_configfrom pipelex.core.stuffs.image_content import ImageContentfrom pipelex.core.stuffs.page_content import PageContent+from pipelex.core.stuffs.search_result_content import SearchResultContentfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.temporal.exceptions import ContentGenerationErrorfrom pipelex.temporal.tprl.observability import build_activity_summary@@ -43,6 +46,7 @@ from pipelex.temporal.tprl_content_generation.act_llm_generate import (act_llm_gen_text,)from pipelex.temporal.tprl_content_generation.act_render_page_views import act_render_page_views+from pipelex.temporal.tprl_content_generation.act_search_generate import act_search_gen_sourced_answer, act_search_gen_structuredfrom pipelex.tools.typing.pydantic_utils import BaseModelTypeVar@@ -477,3 +481,77 @@ class ContentGeneratorInWorkflow(ContentGeneratorProtocol):page_content.page_view = page_view_contents.pop(0)return page_contents++ @override+ async def make_search_sourced_answer(+ self,+ search_assignment: SearchAssignment,+ ) -> SearchResultContent:+ worker_config = get_config().temporal.worker_config+ dispatch_kwargs = worker_config.resolve_dispatch(+ activity_name=act_search_gen_sourced_answer.__name__,+ routing_key=search_assignment.search_handle,+ queue_options_by_queue=get_config().temporal.queue_options,+ is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,+ ).to_execute_kwargs()+ try:+ search_result: SearchResultContent = await workflow.execute_activity(+ act_search_gen_sourced_answer,+ arg=search_assignment,+ summary=build_activity_summary(+ "Search sourced answer", search_assignment.job_metadata, extras={"model": search_assignment.search_handle}+ ),+ **dispatch_kwargs,+ )+ except ActivityError as exc:+ log.error(f"ActivityError caused by: {exc.cause}")+ if isinstance(exc.cause, ApplicationError):+ raise TemporalError.from_app_error(exc=exc.cause) from exc+ raise+ log.verbose(f"ContentGeneratorInWorkflow generated search result: {search_result}")+ return search_result++ @override+ async def make_search_structured(+ self,+ output_structure_class: type[BaseModelTypeVar],+ search_assignment: SearchAssignment,+ ) -> BaseModelTypeVar:+ worker_config = get_config().temporal.worker_config+ search_object_assignment = SearchObjectAssignment.make_for_class(+ output_class=output_structure_class,+ search_assignment=search_assignment,+ )+ dispatch_kwargs = worker_config.resolve_dispatch(+ activity_name=act_search_gen_structured.__name__,+ routing_key=search_assignment.search_handle,+ queue_options_by_queue=get_config().temporal.queue_options,+ is_traced=get_config().temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced,+ ).to_execute_kwargs()+ try:+ result_dict: dict[str, Any] = await workflow.execute_activity(+ act_search_gen_structured,+ arg=search_object_assignment,+ summary=build_activity_summary(+ "Search structured", search_assignment.job_metadata, extras={"class": output_structure_class.__name__}+ ),+ **dispatch_kwargs,+ )+ except ActivityError as exc:+ log.error(f"ActivityError caused by: {exc.cause}")+ if isinstance(exc.cause, ApplicationError):+ raise TemporalError.from_app_error(exc=exc.cause) from exc+ raise+ # Re-validate on the submitter (workflow) side against the original class — pure and+ # deterministic, so the dynamic output class never has to cross the activity boundary. The+ # activity returns the provider's raw dict unvalidated (e.g. the gateway worker returns+ # json.loads(...)), so a malformed structured response raises a bare ValidationError here in+ # workflow code. Left raw it is neither WorkflowExecutionError nor PipelexError, so Temporal+ # treats it as a workflow-task failure and retries forever, hanging the submitter — the exact+ # failure mode this seam exists to prevent. Convert it to a terminal ContentGenerationError (a+ # PipelexError) so the workflow fails and surfaces as an ErrorReport, matching the direct path.+ try:+ return output_structure_class.model_validate(result_dict)+ except ValidationError as exc:+ msg = f"Structured search result failed validation against {output_structure_class.__name__}: {exc}"+ raise ContentGenerationError(msg) from excdiff --git a/tests/integration/pipelex/error_handling/test_data.py b/tests/integration/pipelex/error_handling/test_data.pyindex 486017135..c9e5f0421 100644--- a/tests/integration/pipelex/error_handling/test_data.py+++ b/tests/integration/pipelex/error_handling/test_data.py@@ -17,7 +17,7 @@ drops a classification field, that arm fails.from typing import ClassVar-from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError+from pipelex.cogt.exceptions import InferenceErrorCategory, LLMCompletionError, SearchJobFailureErrorfrom pipelex.cogt.inference.error_classification import UserAction, UserActionKind@@ -60,3 +60,48 @@ class ErrorReportParityTestData:error.model_handle = cls.FAILURE_MODELerror.backend_name = cls.FAILURE_PROVIDERreturn error+++class SearchErrorReportParityTestData:+ """Constants driving both arms of the local / Temporal ``ErrorReport`` parity test for web search.++ The search counterpart of :class:`ErrorReportParityTestData`. A single deliberately-failing+ ``PipeSearch`` leaf is the source of truth for both arms: the local arm runs the ``native_search``+ pipe through the direct ``ContentGenerator`` with ``make_search_sourced_answer`` mocked to fail; the+ Temporal arm runs the same pipe through ``WfPipeRouter`` with the activity-side ``search_gen_sourced_answer``+ mocked to fail. Both assert the same ``EXPECTED_*`` classification, so parity holds by construction.+ """++ # A single-step PipeSearch bundle over the native SearchResult concept — no inputs, no dynamic classes.+ BUNDLE_FILE: ClassVar[str] = "tests/integration/pipelex/temporal/library_crate/native_search.mthds"+ PIPE_CODE: ClassVar[str] = "native_search"++ # The worker-side failure injected into the search call. CONFIGURATION is non-retryable, so the+ # Temporal activity is not retried and the workflow fails on the first attempt — fast and deterministic.+ FAILURE_MESSAGE: ClassVar[str] = "Search provider rejected the request: connection error"+ FAILURE_MODEL: ClassVar[str] = "linkup/standard"+ FAILURE_PROVIDER: ClassVar[str] = "linkup"+ FAILURE_CATEGORY: ClassVar[InferenceErrorCategory] = InferenceErrorCategory.CONFIGURATION+ FAILURE_USER_ACTION_DETAIL: ClassVar[str] = "Verify the API key configured for this search provider."++ # The classification both arms must surface on the recovered ErrorReport.+ EXPECTED_RETRYABLE: ClassVar[bool] = False+ EXPECTED_USER_ACTION_KIND: ClassVar[UserActionKind] = UserActionKind.CHECK_CREDENTIALS++ @classmethod+ def make_failing_search_error(cls) -> SearchJobFailureError:+ """Build a fresh classified ``SearchJobFailureError``.++ Each arm builds its own instance — an exception carries traceback state, so a single shared+ instance must not be reused across runs.+ """+ error = SearchJobFailureError(+ cls.FAILURE_MESSAGE,+ error_category=cls.FAILURE_CATEGORY,+ user_action=UserAction(kind=cls.EXPECTED_USER_ACTION_KIND, detail=cls.FAILURE_USER_ACTION_DETAIL),+ )+ # model_handle / backend_name are declared on CogtError; a real search worker fills them at its+ # public-method chokepoint. This test mocks above the worker, so set them directly.+ error.model_handle = cls.FAILURE_MODEL+ error.backend_name = cls.FAILURE_PROVIDER+ return errordiff --git a/tests/integration/pipelex/error_handling/test_search_error_report_local_full_chain.py b/tests/integration/pipelex/error_handling/test_search_error_report_local_full_chain.pynew file mode 100644index 000000000..b4994a67b--- /dev/null+++ b/tests/integration/pipelex/error_handling/test_search_error_report_local_full_chain.py@@ -0,0 +1,63 @@+"""Local arm of the local / Temporal ``ErrorReport`` parity pair for web search.++Runs the ``native_search`` pipe through the local (non-Temporal) ``PipeRouter`` with the search call+mocked to fail, and asserts the resulting ``ErrorReport`` carries the full classification —+``error_category`` / ``retryable`` / ``model`` / ``provider`` / ``user_action``.++This is the baseline the Temporal full-chain test+(``tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.py``) must match.+Both arms assert the same ``SearchErrorReportParityTestData`` constants, so local / Temporal parity holds+by construction. The search operator does not wrap the leaf error (unlike PipeLLM), so a raw+``SearchJobFailureError`` (a ``CogtError``) propagates and the ``PipeRouter`` re-raises it as-is.+"""++from collections.abc import Generator++import pytest+from pytest_mock import MockerFixture++from pipelex.cogt.content_generation.content_generator import ContentGenerator+from pipelex.cogt.exceptions import SearchJobFailureError+from pipelex.hub import get_pipe_router+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from tests.integration.pipelex.error_handling.test_data import SearchErrorReportParityTestData+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle++[email protected](loop_scope="class")+class TestSearchErrorReportLocalFullChain:+ """A failing search pipe run locally yields a fully classified ``ErrorReport``."""++ @pytest.fixture+ def failing_search_pipe_job_local(self) -> Generator[PipeJob, None, None]:+ """A PipeJob for the search pipe, in LIVE mode so the search call actually fires."""+ yield from pipe_job_from_bundle(+ bundle_file=SearchErrorReportParityTestData.BUNDLE_FILE,+ pipe_code=SearchErrorReportParityTestData.PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ )++ async def test_error_report_from_local_execution(+ self,+ mocker: MockerFixture,+ failing_search_pipe_job_local: PipeJob,+ ) -> None:+ """Local execution surfaces the worker failure as a classified ``SearchJobFailureError``."""+ mocker.patch.object(+ ContentGenerator,+ "make_search_sourced_answer",+ side_effect=SearchErrorReportParityTestData.make_failing_search_error(),+ )++ with pytest.raises(SearchJobFailureError) as exc_info:+ await get_pipe_router().run(pipe_job=failing_search_pipe_job_local)++ # The classification fields — the parity target.+ report = exc_info.value.to_error_report()+ assert report.error_category == SearchErrorReportParityTestData.FAILURE_CATEGORY+ assert report.retryable == SearchErrorReportParityTestData.EXPECTED_RETRYABLE+ assert report.model == SearchErrorReportParityTestData.FAILURE_MODEL+ assert report.provider == SearchErrorReportParityTestData.FAILURE_PROVIDER+ assert report.user_action is not None+ assert report.user_action.kind == SearchErrorReportParityTestData.EXPECTED_USER_ACTION_KINDdiff --git a/tests/integration/pipelex/pipeline/test_bundle_validator.py b/tests/integration/pipelex/pipeline/test_bundle_validator.pyindex e8b8e4dd0..5b6ac6f45 100644--- a/tests/integration/pipelex/pipeline/test_bundle_validator.py+++ b/tests/integration/pipelex/pipeline/test_bundle_validator.py@@ -14,18 +14,49 @@ from typing import Anyimport pytestfrom pytest_mock import MockerFixture+from typing_extensions import override+from pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.hub import (clear_current_library,get_library_manager,+ get_pipelex_hub,get_required_pipe,get_telemetry_manager,)+from pipelex.observer.observer_protocol import ObserverNoOp+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_router import PipeRouterfrom pipelex.pipeline import bundle_validatorfrom pipelex.pipeline.bundle_validator import BundleValidator, DryRunStatusfrom pipelex.pipeline.execution_seams import acquire_libraryfrom pipelex.system.telemetry.events import EventName++class _RaisingPipeRouter(PipeRouter):+ """Hub-default stand-in for the Temporal router: trips if a sweep resolves the hub default.++ A validation sweep is meant to run fully in-process. Nested controller sub-pipes dispatch through+ ``get_pipe_router()``; if the sweep fails to scope its own in-process router, that resolves the hub+ default — the Temporal router in a Temporal-enabled API process — and leaks the dry run to Temporal.+ Installing this as the hub default makes that leak fail loudly instead of silently round-tripping.+ """++ def __init__(self) -> None:+ super().__init__(observer=ObserverNoOp())+ self.run_called = False++ @override+ async def _run_pipe_job(self, pipe_job: PipeJob) -> PipeOutput:+ self.run_called = True+ _ = pipe_job # required by the override signature; unused — the leak is detected by run_called+ msg = (+ "Hub-default pipe router was invoked during a validation sweep — a nested controller leaked "+ "out of the scoped in-process router (the Temporal /validate leak regressed)."+ )+ raise AssertionError(msg)++_BV_DOMAIN = "bundle_validator_test"_BV_MTHDS = f"""domain = "{_BV_DOMAIN}"@@ -78,6 +109,39 @@ branches = ["""+# A bundle whose top-level pipe is a STANDALONE PipeBatch — the shape that 422'd the Temporal /validate+# route. Swept directly, PipeBatch fans out (mock_inputs mints a multi-item list) and each branch+# dispatches through get_pipe_router(); without the sweep scoping its in-process router those branches+# resolve the hub default (the Temporal router) and fire concurrent top-level workflow dispatches.+_BV_BATCH_DOMAIN = "bundle_validator_batch"+_BV_BATCH_MTHDS = f"""+domain = "{_BV_BATCH_DOMAIN}"+description = "Bundle with a standalone PipeBatch — router-scoping regression"++[concept.Item]+description = "An item to summarize"++[concept.Summary]+description = "A summary of an item"++[pipe.summarize_item]+type = "PipeLLM"+description = "Summarize a single item"+inputs = {{ item = "Item" }}+output = "Summary"+prompt = "Summarize $item"++[pipe.summarize_items]+type = "PipeBatch"+description = "Standalone batch fanning out over items"+inputs = {{ items = "Item[]" }}+output = "Summary[]"+branch_pipe_code = "summarize_item"+input_list_name = "items"+input_item_name = "item"+"""++@pytest.mark.asyncio(loop_scope="class")class TestBundleValidatorIntegration:async def test_validate_pipes_success_real_path(self) -> None:@@ -191,6 +255,32 @@ class TestBundleValidatorIntegration:library_manager.teardown(library_id=library_id)clear_current_library()+ async def test_standalone_batch_sweep_scopes_in_process_router(self) -> None:+ # Regression for the Temporal /validate leak: a standalone PipeBatch fans out its branches via+ # get_pipe_router(). The sweep must scope its own in-process router for the whole sweep so those+ # branches resolve it — NOT the hub default, which in a Temporal-enabled API process is the+ # Temporal router (the leak that produced HTTP 422). We install a hub default that raises if its+ # run is ever reached, then assert the batch (and its branch leaf) classify SUCCESS without the+ # hub default ever being touched.+ hub = get_pipelex_hub()+ original_router = hub.get_required_pipe_router()+ sentinel = _RaisingPipeRouter()+ library_manager = get_library_manager()+ library_id = "bv_batch_scope_lib"+ acquire_library(library_id=library_id, mthds_contents=[_BV_BATCH_MTHDS])+ hub.set_pipe_router(pipe_router=sentinel)+ try:+ batch_pipe = get_required_pipe(pipe_code=f"{_BV_BATCH_DOMAIN}.summarize_items")+ leaf_pipe = get_required_pipe(pipe_code=f"{_BV_BATCH_DOMAIN}.summarize_item")+ results = await BundleValidator().validate_pipes([batch_pipe, leaf_pipe], library_id=library_id)+ assert results[f"{_BV_BATCH_DOMAIN}.summarize_items"].status.is_success+ assert results[f"{_BV_BATCH_DOMAIN}.summarize_item"].status.is_success+ assert sentinel.run_called is False, "Nested batch dispatch leaked to the hub default router"+ finally:+ hub.set_pipe_router(pipe_router=original_router)+ library_manager.teardown(library_id=library_id)+ clear_current_library()+async def test_one_pipe_dry_run_event_and_no_stray_pipeline_events(self, mocker: MockerFixture) -> None:library_manager = get_library_manager()library_id = "bv_telemetry_lib"diff --git a/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.py b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.pyindex dd5c750df..583895e68 100644--- a/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.py+++ b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_mock_inference_guard.py@@ -1,12 +1,13 @@-"""Integration test for the ``is_mock_inference`` hard guard in PipeSearch.--Web search has no leaf-level mock, and unlike the LLM/img-gen/extract operators it has no-``cogt/content_generation`` leaf — its provider spend happens directly in-``PipeSearch._live_run_operator_pipe``. Under ``--mock-inference`` (``run_mode=LIVE`` +-``JobMetadata.is_mock_inference``) the operator takes that live path, so the guard lives there and must-fail loud before the search worker is built. ``MockInferenceUnsupportedError`` is a plain-``PipelexError`` (not a model-availability error), so ``PipeOperator._live_run_pipe`` does not re-wrap-it — it propagates unchanged through the router.+"""Integration test for the ``is_mock_inference`` hard guard for web search.++Web search has no leaf-level mock. It now goes through the same ``cogt/content_generation`` seam as+LLM/img-gen/extract: ``PipeSearch`` calls the content generator, whose direct impl runs the+framework-agnostic ``search_generate`` leaf. Under ``--mock-inference`` (``run_mode=LIVE`` ++``JobMetadata.is_mock_inference``) the live path runs, so the guard lives at the top of that leaf+(``search_gen_sourced_answer``) and must fail loud before the search worker is built.+``MockInferenceUnsupportedError`` is a plain ``PipelexError`` (not a model-availability error and not a+``CogtError``), so neither ``PipeOperator._live_run_pipe`` nor the router re-wraps it — it propagates+unchanged."""from typing import Callable@@ -37,7 +38,7 @@ class TestPipeSearchMockInferenceGuard:"""A LIVE PipeSearch run with is_mock_inference=True raises the guard before any provider call."""load_empty_library()- worker_factory_spy = mocker.patch("pipelex.pipe_operators.search.pipe_search.SearchWorkerFactory.make_search_worker")+ worker_factory_spy = mocker.patch("pipelex.cogt.content_generation.search_generate.SearchWorkerFactory.make_search_worker")pipe = PipeFactory[PipeSearch].make_from_blueprint(domain_code="generic",diff --git a/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_model_not_found_reroute.py b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_model_not_found_reroute.pyindex d37d79ca5..46d2db530 100644--- a/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_model_not_found_reroute.py+++ b/tests/integration/pipelex/pipes/operator/pipe_search/test_pipe_search_model_not_found_reroute.py@@ -40,7 +40,7 @@ class TestPipeSearchModelNotFoundReroute:fake_worker = mocker.MagicMock()fake_worker.search_sourced_answer = mocker.AsyncMock(side_effect=not_found_error)mocker.patch(- "pipelex.pipe_operators.search.pipe_search.SearchWorkerFactory.make_search_worker",+ "pipelex.cogt.content_generation.search_generate.SearchWorkerFactory.make_search_worker",return_value=fake_worker,)diff --git a/tests/integration/pipelex/temporal/library_crate/native_search.mthds b/tests/integration/pipelex/temporal/library_crate/native_search.mthdsnew file mode 100644index 000000000..0d9f97bde--- /dev/null+++ b/tests/integration/pipelex/temporal/library_crate/native_search.mthds@@ -0,0 +1,8 @@+domain = "native_search_test"+description = "Single PipeSearch step over the native SearchResult concept (no dynamic classes)"++[pipe.native_search]+type = "PipeSearch"+description = "Search the web for a fixed query"+output = "SearchResult"+prompt = "What are the latest developments in AI safety research?"diff --git a/tests/integration/pipelex/temporal/library_crate/structured_search.mthds b/tests/integration/pipelex/temporal/library_crate/structured_search.mthdsnew file mode 100644index 000000000..c0243003f--- /dev/null+++ b/tests/integration/pipelex/temporal/library_crate/structured_search.mthds@@ -0,0 +1,16 @@+domain = "structured_search_test"+description = "Single PipeSearch step over a custom structured TopicSummary concept (exercises act_search_gen_structured across the Temporal boundary)"++[concept.TopicSummary]+description = "A structured summary of web-search findings for a topic"++[concept.TopicSummary.structure]+title = { type = "text", description = "Topic title", required = true }+summary = { type = "text", description = "Summary of the findings", required = true }+key_points = { type = "list", item_type = "text", description = "Key points distilled from the search", required = true }++[pipe.structured_search]+type = "PipeSearch"+description = "Search the web and return a structured TopicSummary"+output = "TopicSummary"+prompt = "Summarize the latest developments in AI safety research."diff --git a/tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py b/tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.pynew file mode 100644index 000000000..5d71a2d75--- /dev/null+++ b/tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py@@ -0,0 +1,69 @@+"""Mode-1 Temporal guard: the BundleValidator dry-run sweep must NOT dispatch to Temporal.++Deployment-faithful companion to the in-process sentinel regression test in+``tests/integration/pipelex/pipeline/test_bundle_validator.py``+(``test_standalone_batch_sweep_scopes_in_process_router``). Where the sentinel installs a *raising+stand-in* as the hub default, this test resolves the **real** ``TemporalPipeRouter`` as the hub+default — wired by this suite's ``boot_temporal`` fixture exactly as a Temporal-enabled API process+wires it in ``pipelex.pipelex``. It therefore proves the config -> hub-default *wiring* the sentinel+deliberately bypasses, on top of the same contract.++The bug it guards: a standalone ``PipeBatch`` swept directly fans out over a mock list and+dispatches each branch through ``get_pipe_router()``. Without the sweep scoping its own in-process+router, those branches resolve the hub default — the ``TemporalPipeRouter`` — and fire concurrent+top-level workflow dispatches (``WorkflowExecutor.execute_workflow``), which produced the production+HTTP 422.++The contract asserted: the whole sweep stays in-process, so the submitter-side top-level dispatch+seam (``WorkflowExecutor.execute_workflow``) is never reached. Spying on that seam is the+deterministic analogue of the Mode-2 "worker received no dispatch" check. No live Temporal server is+needed precisely because GREEN means no dispatch ever happens; if the fix regressed, the spy fires+and the test goes RED.+"""++from pathlib import Path++import pytest+from pytest_mock import MockerFixture++from pipelex.hub import clear_current_library, get_library_manager, get_pipelex_hub, get_required_pipe+from pipelex.pipeline.bundle_validator import BundleValidator+from pipelex.pipeline.execution_seams import acquire_library+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor+from pipelex.temporal.tprl_pipe.temporal_pipe_router import TemporalPipeRouter+from tests.integration.pipelex.temporal.test_data import PipeBatchTemporalTestData++# The standalone ``type = "PipeBatch"`` pipe in temporal_batch.mthds — swept directly it fans out,+# which is the exact shape that turned the leak fatal (concurrent same-id top-level dispatches).+_STANDALONE_BATCH_PIPE_REF = f"{PipeBatchTemporalTestData.DOMAIN}.batch_temporal_describe_topics"++[email protected](loop_scope="class")+class TestValidateSweepStaysInProcess:+ async def test_standalone_batch_sweep_never_dispatches_to_temporal(self, mocker: MockerFixture) -> None:+ # Precondition (the wiring this test exists to prove): under boot_temporal the hub default+ # router IS the real TemporalPipeRouter — the same router a Temporal-enabled API process runs.+ hub = get_pipelex_hub()+ assert isinstance(hub.get_required_pipe_router(), TemporalPipeRouter), (+ "Temporal suite precondition: the hub default router must be the real TemporalPipeRouter"+ )++ # Spy on the submitter-side top-level dispatch seam. A standalone-batch sweep that leaked to+ # Temporal would reach this exact method (TemporalPipeRouter._run_pipe_job top-level branch ->+ # WorkflowExecutorFactory.create_executor(...).execute_workflow(...)). With the fix the sweep+ # stays in-process and never touches it.+ execute_workflow_spy = mocker.spy(WorkflowExecutor, "execute_workflow")++ library_manager = get_library_manager()+ library_id = "validate_sweep_temporal_leak_lib"+ mthds_content = Path(PipeBatchTemporalTestData.BUNDLE_FILE).read_text(encoding="utf-8")+ acquire_library(library_id=library_id, mthds_contents=[mthds_content])+ try:+ batch_pipe = get_required_pipe(pipe_code=_STANDALONE_BATCH_PIPE_REF)+ results = await BundleValidator().validate_pipes([batch_pipe], library_id=library_id)+ assert results[_STANDALONE_BATCH_PIPE_REF].status.is_success+ execute_workflow_spy.assert_not_called()+ finally:+ library_manager.teardown(library_id=library_id)+ clear_current_library()diff --git a/tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.py b/tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.pynew file mode 100644index 000000000..0c3bd0416--- /dev/null+++ b/tests/integration/pipelex/temporal/test_workflow_search_error_report_full_chain.py@@ -0,0 +1,114 @@+"""Temporal arm of the local / Temporal ``ErrorReport`` parity pair for web search.++Runs the ``native_search`` pipe through a real ``WfPipeRouter`` workflow on the in-process Temporal+server, submitted via ``WorkflowExecutor.execute_workflow``, with the activity-side search call mocked to+fail. This is the regression test for the original bug: before ``PipeSearch`` routed its leaf through a+Temporal activity, a search failure raised a raw ``SearchJobFailureError`` inside the workflow, which was+neither an ``ActivityError`` nor a ``WorkflowExecutionError`` — so Temporal retried the workflow task+forever and the submitter hung. Now the failure crosses the activity → workflow → submitter boundary as a+terminal ``WorkflowExecutionError`` carrying the structured ``ErrorReport``.++The local baseline is the separate module+``tests/integration/pipelex/error_handling/test_search_error_report_local_full_chain.py``; both arms assert+the same ``SearchErrorReportParityTestData`` constants, so local / Temporal parity holds by construction.+"""++import uuid+from collections.abc import Generator+from typing import TYPE_CHECKING++import pytest+from pytest_mock import MockerFixture+from temporalio.client import Client as TemporalClient+from temporalio.common import RetryPolicy++from pipelex.config import get_config+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.temporal.exceptions import WorkflowExecutionError+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.error_handling.test_data import SearchErrorReportParityTestData+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle++if TYPE_CHECKING:+ from pipelex.core.pipes.pipe_output import PipeOutput++ACT_SEARCH_SOURCED_ANSWER_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_sourced_answer"++[email protected](loop_scope="class")+class TestWorkflowSearchErrorReportFullChain:+ """A failing search pipe's structured ``ErrorReport`` survives the Temporal workflow → submitter boundary."""++ @pytest.fixture+ def temporal_enabled(self) -> Generator[None, None, None]:+ """Enable ``temporal.is_enabled`` for the test — ``WorkflowExecutor.temporal_client()`` raises when off."""+ config = get_config()+ previous = config.temporal.is_enabled+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})+ yield+ config.temporal = config.temporal.model_copy(update={"is_enabled": previous})++ @pytest.fixture+ def failing_search_pipe_job(self) -> Generator[PipeJob, None, None]:+ """A PipeJob for the search pipe, in LIVE mode so the workflow dispatches the search activity.++ DRY mode short-circuits the ``act_search_*`` dispatch (the dry content generator reports inline+ inside the workflow), so the activity — and the mock — never fire.+ """+ yield from pipe_job_from_bundle(+ bundle_file=SearchErrorReportParityTestData.BUNDLE_FILE,+ pipe_code=SearchErrorReportParityTestData.PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ )++ async def test_search_error_report_survives_temporal_boundary(+ self,+ temporal_client: TemporalClient,+ mocker: MockerFixture,+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration+ failing_search_pipe_job: PipeJob,+ ) -> None:+ """A failing search run through Temporal surfaces a fully classified ``WorkflowExecutionError`` (no hang)."""+ # Mock the search core inside the real activity (in-process unsandboxed worker, same process).+ # @convert_pipelex_errors packs the ErrorReport into ApplicationError.details.+ mocker.patch(+ ACT_SEARCH_SOURCED_ANSWER_TARGET,+ new=mocker.AsyncMock(side_effect=SearchErrorReportParityTestData.make_failing_search_error()),+ )++ task_queue = f"q_search_err_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_search_err_{uuid.uuid4().hex[:8]}"+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(+ temporal_client=temporal_client,+ task_queue=task_queue,+ retry_policy=RetryPolicy(maximum_attempts=1),+ )++ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):+ with pytest.raises(WorkflowExecutionError) as exc_info:+ await executor.execute_workflow(+ workflow_class=WfPipeRouter,+ workflow_arg=failing_search_pipe_job,+ workflow_id=workflow_id,+ )++ error = exc_info.value+ # The structured report was recovered across the workflow → submitter boundary.+ assert error.error_report is not None+ report = error.to_error_report()++ # The real failure message survived — not the generic "Failed to execute workflow ...".+ assert SearchErrorReportParityTestData.FAILURE_MESSAGE in report.message+ assert "Failed to execute workflow" not in report.message++ # The full classification survived the Temporal serialization round-trip.+ assert report.error_category == SearchErrorReportParityTestData.FAILURE_CATEGORY+ assert report.retryable == SearchErrorReportParityTestData.EXPECTED_RETRYABLE+ assert report.model == SearchErrorReportParityTestData.FAILURE_MODEL+ assert report.provider == SearchErrorReportParityTestData.FAILURE_PROVIDER+ assert report.user_action is not None+ assert report.user_action.kind == SearchErrorReportParityTestData.EXPECTED_USER_ACTION_KINDdiff --git a/tests/integration/pipelex/temporal/test_workflow_search_structured_roundtrip.py b/tests/integration/pipelex/temporal/test_workflow_search_structured_roundtrip.pynew file mode 100644index 000000000..48590fc7b--- /dev/null+++ b/tests/integration/pipelex/temporal/test_workflow_search_structured_roundtrip.py@@ -0,0 +1,114 @@+"""Structured-output counterpart to the search success round-trip test.++Runs a ``PipeSearch`` step whose output is a *custom structured concept* (``TopicSummary``) through a real+``WfPipeRouter`` workflow on the in-process Temporal server, with the activity-side search call mocked to+*succeed*. This exercises the ``act_search_gen_structured`` path — the riskier of the two search activities:+the operator ships the output structure's JSON schema (not the live class) across the boundary via+``SearchObjectAssignment``, the activity reconstructs a throwaway class to drive the provider and returns the+**raw result dict**, and the submitter re-validates that dict against the original output class+(``output_structure_class.model_validate(result_dict)``) — a pure, deterministic step that keeps the dynamic+class on the submitter side. This test locks in that whole round-trip; the sourced-answer sibling+(``test_workflow_search_success_roundtrip.py``) covers the non-structured path. The mock firing also proves the+workflow dispatched ``act_search_gen_structured`` rather than running the leaf inline on the workflow loop.+"""++import uuid+from collections.abc import Generator+from typing import TYPE_CHECKING++import pytest+from pytest_mock import MockerFixture+from temporalio.client import Client as TemporalClient+from temporalio.common import RetryPolicy++from pipelex.config import get_config+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output++if TYPE_CHECKING:+ from pipelex.core.pipes.pipe_output import PipeOutput++ACT_SEARCH_STRUCTURED_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_structured"++BUNDLE_FILE = "tests/integration/pipelex/temporal/library_crate/structured_search.mthds"+PIPE_CODE = "structured_search"++# Flat, schema-shaped dict — matches what the real worker returns (LinkupSearchWorker._search_structured+# returns response.model_dump()), which the submitter re-validates against the TopicSummary structure class.+EXPECTED_TITLE = "AI safety in 2026"+EXPECTED_SUMMARY = "Interpretability and scalable oversight lead the agenda."+EXPECTED_KEY_POINTS = ["Mechanistic interpretability", "Scalable oversight"]++[email protected](loop_scope="class")+class TestWorkflowSearchStructuredRoundtrip:+ """A structured search result survives the Temporal activity → workflow → submitter round-trip."""++ @pytest.fixture+ def temporal_enabled(self) -> Generator[None, None, None]:+ config = get_config()+ previous = config.temporal.is_enabled+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})+ yield+ config.temporal = config.temporal.model_copy(update={"is_enabled": previous})++ @pytest.fixture+ def structured_search_pipe_job(self) -> Generator[PipeJob, None, None]:+ """A PipeJob for the structured search pipe, in LIVE mode so the workflow dispatches the search activity."""+ yield from pipe_job_from_bundle(+ bundle_file=BUNDLE_FILE,+ pipe_code=PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ )++ async def test_structured_search_result_survives_temporal_boundary(+ self,+ temporal_client: TemporalClient,+ mocker: MockerFixture,+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration+ structured_search_pipe_job: PipeJob,+ ) -> None:+ """The activity's raw dict re-validates into the TopicSummary structure class on the submitter, intact."""+ mock_result = {+ "title": EXPECTED_TITLE,+ "summary": EXPECTED_SUMMARY,+ "key_points": EXPECTED_KEY_POINTS,+ }+ search_core_mock = mocker.AsyncMock(return_value=mock_result)+ mocker.patch(ACT_SEARCH_STRUCTURED_TARGET, new=search_core_mock)++ task_queue = f"q_search_struct_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_search_struct_{uuid.uuid4().hex[:8]}"+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(+ temporal_client=temporal_client,+ task_queue=task_queue,+ retry_policy=RetryPolicy(maximum_attempts=1),+ )++ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):+ pipe_output = await executor.execute_workflow(+ workflow_class=WfPipeRouter,+ workflow_arg=structured_search_pipe_job,+ workflow_id=workflow_id,+ )++ # The structured activity dispatched (mock fired) — the leaf did not run inline in the workflow,+ # and the SearchObjectAssignment (carrying output_class_schema) crossed the boundary.+ search_core_mock.assert_awaited_once()++ # The raw dict re-validated against the original output class on the submitter side and round-tripped+ # intact (rehydrate the deferred working memory exactly as a production submitter does).+ rehydrate_pipe_output(pipe_output, structured_search_pipe_job)+ content = pipe_output.main_stuff.content+ assert isinstance(content, StructuredContent)+ data = content.model_dump()+ assert data["title"] == EXPECTED_TITLE+ assert data["summary"] == EXPECTED_SUMMARY+ assert data["key_points"] == EXPECTED_KEY_POINTSdiff --git a/tests/integration/pipelex/temporal/test_workflow_search_structured_validation_failure.py b/tests/integration/pipelex/temporal/test_workflow_search_structured_validation_failure.pynew file mode 100644index 000000000..48acf7b56--- /dev/null+++ b/tests/integration/pipelex/temporal/test_workflow_search_structured_validation_failure.py@@ -0,0 +1,109 @@+"""Regression test: a malformed structured-search result fails the Temporal workflow terminally, not a hang.++The structured-search activity (``act_search_gen_structured``) returns the provider's *raw* result dict — the+gateway worker returns ``json.loads(...)`` unvalidated against the schema — and the submitter re-validates it+against the original output structure class with ``output_structure_class.model_validate(result_dict)`` in+workflow code, *outside* the activity's ``convert_pipelex_errors`` boundary. Left to raise a bare+``pydantic.ValidationError`` there, it would be neither a ``WorkflowExecutionError`` nor a ``PipelexError`` — the+only two ``workflow_failure_exception_types`` — so Temporal would treat it as a *workflow-task* failure and retry+indefinitely, hanging the submitter (the exact failure mode the whole search seam exists to prevent).+``make_search_structured`` converts it to a terminal ``ContentGenerationError`` (a ``PipelexError``) instead; this+test locks that in by feeding a malformed dict and asserting the submitter gets a terminal+``WorkflowExecutionError`` rather than hanging. The success sibling is ``test_workflow_search_structured_roundtrip.py``.+"""++import uuid+from collections.abc import Generator+from typing import TYPE_CHECKING++import pytest+from pytest_mock import MockerFixture+from temporalio.client import Client as TemporalClient+from temporalio.common import RetryPolicy++from pipelex.config import get_config+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.temporal.exceptions import WorkflowExecutionError+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle++if TYPE_CHECKING:+ from pipelex.core.pipes.pipe_output import PipeOutput++ACT_SEARCH_STRUCTURED_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_structured"++BUNDLE_FILE = "tests/integration/pipelex/temporal/library_crate/structured_search.mthds"+PIPE_CODE = "structured_search"++# A dict the provider could plausibly return that does NOT satisfy the TopicSummary structure: the required+# ``title`` field is missing, so re-validating it against the original output class on the submitter must fail.+MALFORMED_RESULT = {+ "summary": "Interpretability and scalable oversight lead the agenda.",+ "key_points": ["Mechanistic interpretability", "Scalable oversight"],+}++[email protected](loop_scope="class")+class TestWorkflowSearchStructuredValidationFailure:+ """A malformed structured-search result surfaces a terminal WorkflowExecutionError instead of hanging."""++ @pytest.fixture+ def temporal_enabled(self) -> Generator[None, None, None]:+ config = get_config()+ previous = config.temporal.is_enabled+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})+ yield+ config.temporal = config.temporal.model_copy(update={"is_enabled": previous})++ @pytest.fixture+ def structured_search_pipe_job(self) -> Generator[PipeJob, None, None]:+ """A PipeJob for the structured search pipe, in LIVE mode so the workflow dispatches the search activity."""+ yield from pipe_job_from_bundle(+ bundle_file=BUNDLE_FILE,+ pipe_code=PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ )++ @pytest.mark.timeout(60)+ async def test_malformed_structured_result_fails_terminally(+ self,+ temporal_client: TemporalClient,+ mocker: MockerFixture,+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration+ structured_search_pipe_job: PipeJob,+ ) -> None:+ """A structured-search dict that fails submitter-side re-validation terminates the workflow, not hangs."""+ search_core_mock = mocker.AsyncMock(return_value=MALFORMED_RESULT)+ mocker.patch(ACT_SEARCH_STRUCTURED_TARGET, new=search_core_mock)++ task_queue = f"q_search_struct_bad_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_search_struct_bad_{uuid.uuid4().hex[:8]}"+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(+ temporal_client=temporal_client,+ task_queue=task_queue,+ retry_policy=RetryPolicy(maximum_attempts=1),+ )++ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):+ with pytest.raises(WorkflowExecutionError) as exc_info:+ await executor.execute_workflow(+ workflow_class=WfPipeRouter,+ workflow_arg=structured_search_pipe_job,+ workflow_id=workflow_id,+ )++ # The structured activity dispatched (mock fired) — the malformed dict came back through the boundary,+ # and the failure happened on the submitter side, not inside the (successful) activity.+ search_core_mock.assert_awaited_once()++ # Terminal failure, not a hang: the submitter recovered a structured report whose message reflects the+ # submitter-side structured-validation failure (not a generic "Failed to execute workflow ...").+ error = exc_info.value+ assert error.error_report is not None+ report = error.to_error_report()+ assert "validation" in report.message.lower()+ assert "Failed to execute workflow" not in report.messagediff --git a/tests/integration/pipelex/temporal/test_workflow_search_success_roundtrip.py b/tests/integration/pipelex/temporal/test_workflow_search_success_roundtrip.pynew file mode 100644index 000000000..bb85dcb07--- /dev/null+++ b/tests/integration/pipelex/temporal/test_workflow_search_success_roundtrip.py@@ -0,0 +1,103 @@+"""Success-path counterpart to the search error-report full-chain test.++Runs the ``native_search`` pipe through a real ``WfPipeRouter`` workflow on the in-process Temporal server+with the activity-side search call mocked to *succeed*, and asserts the ``SearchResultContent`` comes back+intact across the activity → workflow → submitter boundary. This locks in the other half of the fix: search+on Temporal now records its result via an activity (replay-safe) instead of running inline on the workflow+loop, and the native ``SearchResultContent`` serializes cleanly back to the submitter. The mock firing also+proves the workflow actually dispatched ``act_search_gen_sourced_answer`` rather than running the leaf inline.+"""++import uuid+from collections.abc import Generator+from typing import TYPE_CHECKING++import pytest+from pytest_mock import MockerFixture+from temporalio.client import Client as TemporalClient+from temporalio.common import RetryPolicy++from pipelex.config import get_config+from pipelex.core.stuffs.document_content import DocumentContent+from pipelex.core.stuffs.search_result_content import SearchResultContent+from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.temporal.temporal_hub import get_task_manager+from pipelex.temporal.tprl.workflow_caller import WorkflowExecutor+from pipelex.temporal.tprl_pipe.wf_pipe_router import WfPipeRouter+from tests.integration.pipelex.error_handling.test_data import SearchErrorReportParityTestData+from tests.integration.pipelex.fixtures.pipe_job_helpers import pipe_job_from_bundle+from tests.integration.pipelex.temporal.library_crate.helpers import rehydrate_pipe_output++if TYPE_CHECKING:+ from pipelex.core.pipes.pipe_output import PipeOutput++ACT_SEARCH_SOURCED_ANSWER_TARGET = "pipelex.temporal.tprl_content_generation.act_search_generate.search_gen_sourced_answer"++EXPECTED_ANSWER = "Recent AI safety work focuses on interpretability and scalable oversight."+EXPECTED_SOURCE_TITLE = "AI safety overview"++[email protected](loop_scope="class")+class TestWorkflowSearchSuccessRoundtrip:+ """A successful search result survives the Temporal activity → workflow → submitter round-trip."""++ @pytest.fixture+ def temporal_enabled(self) -> Generator[None, None, None]:+ config = get_config()+ previous = config.temporal.is_enabled+ config.temporal = config.temporal.model_copy(update={"is_enabled": True})+ yield+ config.temporal = config.temporal.model_copy(update={"is_enabled": previous})++ @pytest.fixture+ def search_pipe_job(self) -> Generator[PipeJob, None, None]:+ """A PipeJob for the search pipe, in LIVE mode so the workflow dispatches the search activity."""+ yield from pipe_job_from_bundle(+ bundle_file=SearchErrorReportParityTestData.BUNDLE_FILE,+ pipe_code=SearchErrorReportParityTestData.PIPE_CODE,+ pipe_run_mode=PipeRunMode.LIVE,+ )++ async def test_search_result_survives_temporal_boundary(+ self,+ temporal_client: TemporalClient,+ mocker: MockerFixture,+ temporal_enabled: None, # noqa: ARG002 - enables temporal.is_enabled for the duration+ search_pipe_job: PipeJob,+ ) -> None:+ """The activity's ``SearchResultContent`` comes back intact on the submitter's ``PipeOutput``."""+ mock_result = SearchResultContent(+ answer=EXPECTED_ANSWER,+ sources=[DocumentContent(title=EXPECTED_SOURCE_TITLE, url="https://example.com/ai-safety", mime_type="text/html")],+ )+ search_core_mock = mocker.AsyncMock(return_value=mock_result)+ mocker.patch(ACT_SEARCH_SOURCED_ANSWER_TARGET, new=search_core_mock)++ task_queue = f"q_search_ok_{uuid.uuid4().hex[:8]}"+ workflow_id = f"wf_search_ok_{uuid.uuid4().hex[:8]}"+ executor: WorkflowExecutor[PipeJob, PipeOutput] = WorkflowExecutor(+ temporal_client=temporal_client,+ task_queue=task_queue,+ retry_policy=RetryPolicy(maximum_attempts=1),+ )++ async with get_task_manager().make_worker(temporal_client, task_queue=task_queue, is_not_sandboxed=True):+ pipe_output = await executor.execute_workflow(+ workflow_class=WfPipeRouter,+ workflow_arg=search_pipe_job,+ workflow_id=workflow_id,+ )++ # The activity dispatched (mock fired) — the leaf did not run inline in the workflow.+ search_core_mock.assert_awaited_once()++ # The SearchResultContent round-tripped back to the submitter intact (rehydrate the deferred+ # working memory exactly as a production submitter does).+ rehydrate_pipe_output(pipe_output, search_pipe_job)+ content = pipe_output.main_stuff.content+ assert isinstance(content, SearchResultContent)+ assert content.answer == EXPECTED_ANSWER+ assert len(content.sources) == 1+ assert content.sources[0].title == EXPECTED_SOURCE_TITLEdiff --git a/tests/unit/pipelex/cli/test_validate_temporal_flag.py b/tests/unit/pipelex/cli/test_validate_temporal_flag.pynew file mode 100644index 000000000..745aee8d1--- /dev/null+++ b/tests/unit/pipelex/cli/test_validate_temporal_flag.py@@ -0,0 +1,33 @@+"""The ``--temporal`` flag surface on ``pipelex validate`` subcommands.++Locks the public CLI contract that ``bundle`` / ``pipe`` / ``method`` each expose+``--temporal/--no-temporal`` (parity with ``pipelex run``). The flag overrides+``temporal.is_enabled`` for the boot; the validation sweep stays in-process regardless,+so the *behavioural* guarantee (validation does not dispatch to Temporal under a+Temporal-enabled hub) is exercised by the distributed e2e scenario, not here.++Asserts on the resolved Click parameters rather than on rendered ``--help`` text: the+help renderer line-wraps option names at narrow terminal widths (CI's non-tty width+splits ``--temporal`` across lines), which is a presentation detail, not the contract.+Introspecting the command's params is width-independent and short-circuits before any+Pipelex boot, so this stays a true unit test.+"""++from __future__ import annotations++import click+import pytest+from typer.main import get_command++from pipelex.cli.commands.validate.app import validate_app+++class TestValidateTemporalFlag:+ @pytest.mark.parametrize("subcommand", ["bundle", "pipe", "method"])+ def test_temporal_flag_exposed(self, subcommand: str) -> None:+ group = get_command(validate_app)+ assert isinstance(group, click.Group)+ command = group.commands[subcommand]+ flag_opts: list[str] = [opt for param in command.params for opt in (*param.opts, *param.secondary_opts)]+ assert "--temporal" in flag_opts+ assert "--no-temporal" in flag_optsdiff --git a/tests/unit/pipelex/temporal/test_worker_scopes_specialized.py b/tests/unit/pipelex/temporal/test_worker_scopes_specialized.pyindex d533da0e8..193956e4d 100644--- a/tests/unit/pipelex/temporal/test_worker_scopes_specialized.py+++ b/tests/unit/pipelex/temporal/test_worker_scopes_specialized.py@@ -1,9 +1,9 @@"""Unit tests for the specialized worker scopes (runner-llm, runner-img-gen,-runner-extract, runner-jinja2) shipped in `pipelex/pipelex.toml`.+runner-extract, runner-jinja2, runner-search) shipped in `pipelex/pipelex.toml`.These scopes let deployment manifests spin up one worker pool per backendclass. The tests verify (a) each scope registers exactly the activities its-name implies and (b) the union of all four specialized scopes covers the+name implies and (b) the union of the specialized scopes covers thefull ``runner`` registration with no orphan activities."""@@ -39,6 +39,7 @@ class TestSpecializedWorkerScopes:("runner-img-gen", {"act_img_gen_images", "act_render_page_views"}),("runner-extract", {"act_extract_gen_extract_pages", "act_render_page_views"}),("runner-jinja2", {"act_jinja2_gen_text"}),+ ("runner-search", {"act_search_gen_sourced_answer", "act_search_gen_structured"}),],)def test_scope_registers_expected_activities(self, scope_name: str, expected_activity_names: set[str]) -> None:@@ -61,7 +62,7 @@ class TestSpecializedWorkerScopes:runner_names = {activity.__name__ for activity in runner_activities}specialized_union: set[str] = set()- for scope_name in ("runner-llm", "runner-img-gen", "runner-extract", "runner-jinja2"):+ for scope_name in ("runner-llm", "runner-img-gen", "runner-extract", "runner-jinja2", "runner-search"):scope = worker_scopes.scopes[scope_name]_, scope_activities = tasks.workflows_and_activities(scope=scope)specialized_union.update(activity.__name__ for activity in scope_activities)diff --git a/wip/distributed-execution/README.md b/wip/distributed-execution/README.mdindex edaa948d2..ef28506f0 100644--- a/wip/distributed-execution/README.md+++ b/wip/distributed-execution/README.md@@ -15,6 +15,7 @@ This folder is the **distributed-execution track**: running MTHDS methods as Tem- [schema-reconstruction-hardening.md](schema-reconstruction-hardening.md) — deferred hardening of cross-process Pydantic-schema reconstruction.- [unit-testing-worker-sandbox-validation.md](unit-testing-worker-sandbox-validation.md) — deferred bug: a sandboxed worker can't boot under `--is-unit-testing` because the registered test workflows fail temporalio sandbox validation.- [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md) — **OPEN handoff:** how the `_bridge` dev-merge changed the runtime-bridge surface (param renames, deleted graph-assembly primitive, `graph_context→trace_context`, #967/#968) and what the downstream `pipelex-mistralai-workflows` + `_workflows` repos need reconciled. Greps/diagnosis not yet done.+- [validate-sweep-temporal-leak-repro.md](validate-sweep-temporal-leak-repro.md) — **bug brief + e2e repro (DONE):** the `/validate` dry-run sweep leaked nested controller sub-pipes to Temporal (HTTP 422 on a standalone `PipeBatch`). Fixed (commit `3377babb`); all three guard layers now in place — in-process sentinel test, Mode-1 pytest (`test_validate_sweep_stays_in_process.py`), and the deployment-faithful Mode-2 scenario (Tier 2c in the `temporal-e2e-validate` skill, using the `pipelex validate bundle --temporal` flag).- [temporal-e2e-validate-skill.md](temporal-e2e-validate-skill.md) — validation status of the `/temporal-e2e-validate` skill and the Step 9 (queue-options) skill bugs patched in its `queue-options-battery.md`. Full dry + live run all green; open follow-ups are the vestigial `act_jinja2_gen_text` removal and a stale Mode 1 "Known xfails" note.- [mistral-workflows/](mistral-workflows/) — sub-track for landing the `pipelex-mistralai-workflows` plugin (the MISTRAL_NATIVE backend) onto current `dev`. Its [README](mistral-workflows/README.md) is the hub; verdict is split — land the bridge, hold the `mistralai` 1.x→2.x bump (blocked on instructor PR #2298). Carries the two ready-to-paste keep-alive prompts.diff --git a/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md b/wip/distributed-execution/temporal-e2e-validate-prompt-menu.mdindex 5db3c55f1..7b18c6635 100644--- a/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md+++ b/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md@@ -44,6 +44,24 @@ Recommended order: run the **no-spend** sweep first to confirm the plumbing chea/temporal-e2e-validate validate cross-process registry # Tier 2b```+### Validation sweep stays in-process — Temporal-leak guard (free)++Guards the `/validate` dry-run leak (nested controller sub-pipes dispatching to Temporal → HTTP 422 on a standalone `PipeBatch`). Two layers, both free; the contract is "the validation sweep never dispatches to Temporal, even under a Temporal-enabled boot".++```text+/temporal-e2e-validate validate the dry-run sweep stays in-process under a Temporal-enabled boot # Mode 2 Tier 2c+```++- **Mode-1 pytest** (real `TemporalPipeRouter` as hub default, spies `WorkflowExecutor.execute_workflow`, asserts never called). It lives *outside* `library_crate/`, so the generic "Mode 1 pytest suite" prompt above does **not** pick it up — run it by path:++ ```text+ timeout 180 .venv/bin/pytest tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py -m temporal --temporal-server local --timeout=60+ ```++ (Self-contained — GREEN never dispatches, so no live server is actually required; `--temporal-server none` works too.)++- **Mode-2 Tier 2c** (deployment-faithful: `validate bundle --temporal` over a standalone `PipeBatch` → exit 0 **and** worker idle / no `WfPipeRouter` dispatch). Reached via the prompt above; full GREEN/RED procedure in `references/mode-2-tiers.md` Step 3, Tier 2c.+### Graph tracing (free)```textdiff --git a/wip/distributed-execution/validate-sweep-temporal-leak-repro.md b/wip/distributed-execution/validate-sweep-temporal-leak-repro.mdnew file mode 100644index 000000000..cd199d245--- /dev/null+++ b/wip/distributed-execution/validate-sweep-temporal-leak-repro.md@@ -0,0 +1,150 @@+# Bug brief + e2e repro — the `/validate` dry-run sweep leaked nested controller sub-pipes to Temporal++> **Status:** bug FIXED (commit `3377babb` on `fix/Temporal-dry-run`). **All three guard layers are now implemented:** the in-process sentinel regression test, the **Mode-1 pytest** (`tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py`), and the **Mode-2 scenario** (Tier 2c in the `temporal-e2e-validate` skill). This doc is retained as the cold-start brief + reproduction record. See [`/TODOS.md`](../../TODOS.md) for the PR-review guide.+>+> **Track:** part of the [distributed-execution](./README.md) track — the class of bug that only the true 3-process topology surfaces.++## 1. What the bug was++Running the **Pipelex API** (`../pipelex-api`) Temporal-enabled (`.pipelex/pipelex_override.toml` → `[temporal] is_enabled = true`) with a separate **Pipelex worker** (`../pipelex-worker`) up, calling the runner's `/validate` endpoint on a bundle containing a controller pipe returned **HTTP 422**:++```+ValidateBundleError (error_domain=input), 422+detail: "Dry run failed with 1 unexpected pipe failure(s):+ '<domain>.<batch_pipe>': ... status=DryRunStatus.FAILURE+ error_message=\"Dry run failed for pipe '<domain>.<batch_pipe>':+ Failed to execute workflow WfPipeRouter\""+```++The string **"Failed to execute workflow WfPipeRouter"** is produced only at `pipelex/temporal/tprl/workflow_caller.py` (the `(WorkflowAlreadyStartedError, RPCError)` branch of `WorkflowExecutor.execute_workflow`) — a **top-level** Temporal workflow dispatch from the submitter side. **A dry run should never reach that code.**++## 2. Root cause (one sentence)++`BundleValidator` builds a local in-process `PipeRouter` for the sweep but **never installs it as the active `get_pipe_router()` override**, so every nested controller sub-pipe (`pipe_controllers/sub_pipe.py`, `pipe_controllers/batch/pipe_batch.py` — all dispatch via `get_pipe_router()`) falls through to the **hub default**, which under a Temporal-enabled hub is the `TemporalPipeRouter` (wired in `pipelex/pipelex.py` when `get_config().temporal.is_enabled`). The top-level pipe ran in-process; its nested pipes leaked to Temporal.++**Why it 422s for a *standalone* `PipeBatch`/`PipeParallel` specifically:** the sweep dry-runs every pipe in the bundle individually. When it dry-runs a standalone batch pipe *directly*, the batch fans out over a mock list (`mock_inputs` mints ~3 items) and fires **N concurrent top-level Temporal dispatches with the same workflow id** (the id derives from the one shared `dry_run_pipeline_id`) → `WorkflowAlreadyStartedError` → `WorkflowExecutionError("Failed to execute workflow WfPipeRouter")` → the pipe is classified `FAILURE` → `DryRunError` → 422. A batch reached only *as a sub-pipe of a sequence* is a single serial dispatch — no collision — so it **silently round-trips Temporal and passes** (a false pass: still wrong, just not fatal).++## 3. The fix (already committed: `3377babb`)++Mirror the DIRECT-mode precedent in `pipelex/runtime_bridge/bridge.py::_run_direct`. In `pipelex/pipeline/bundle_validator.py`:++- keep the in-process router as `self._pipe_router` (don't discard it inside `PipeRun(...)`);+- wrap the per-pipe sweep loop in `validate_pipes` with `with scoped_pipe_router(self._pipe_router):`.++Now nested controllers resolve the in-process router; the whole sweep stays in-process regardless of backend.++**In-process regression guard (deterministic, no Temporal needed):**+`tests/integration/pipelex/pipeline/test_bundle_validator.py::TestBundleValidatorIntegration::test_standalone_batch_sweep_scopes_in_process_router` — installs a hub-default router that raises if reached, sweeps a standalone `PipeBatch`, asserts SUCCESS with the hub default never touched. (Verified RED without the scope.)++> ⚠️ A *separate* "secondary fix" to `dry_run_pipeline.py` (the graph step) was tried and **reverted** — it was misdiagnosed. See the correction block in [`/TODOS.md`](../../TODOS.md). Not relevant to this repro.++## 4. Why our PyTest suite didn't catch it (answer to "can in-process pytest reproduce it?")++**Partial confirm, with nuance — don't over-claim "impossible in-process":**++- The **leak itself** is reproducible in a single process with zero Temporal infrastructure — that's exactly what the committed regression test does (sentinel hub-default router). The **collision symptom** is *also* reproducible in-process in principle: the in-process Temporal test server (`--temporal-server none`/`time-skipping`) enforces the same workflow-id-reuse semantics, so concurrent same-id dispatches would still raise `WorkflowAlreadyStartedError`.+- So it escaped **not** because the topology can't show it, but because **no test crossed the seam "Temporal-enabled hub × the bundle-validation sweep":** the Temporal suite (`tests/integration/pipelex/temporal/`) exercises pipe *runs* via crate, never the validation *sweep*; the `BundleValidator` tests run under the default in-process hub, never the Temporal hub.+- Two reasons the **distributed (Mode 2) setup is still the right home** for a scenario:+ 1. The silent **false-pass** shapes are invisible to a "does validate pass?" assertion — only an explicit "no Temporal dispatch happened" check (the sentinel test) or the standalone-batch collision shape exposes them.+ 2. The **production symptom** (collision → 422 across a real API↔worker process boundary) matches the deployment topology exactly here, where a human actually hit it.++Net: the unit regression test guards the leak; the e2e scenario below guards the deployment-faithful symptom and documents the reproduction for the record.++## 5. The new scenario to add to `temporal-e2e-validate`++Lives under the skill at `.claude/skills/temporal-e2e-validate/`. Add this as a new scenario in `references/mode-2-tiers.md` (or a short new reference file).++**Which home?** This bug is **submitter-side** — the colliding top-level dispatches fire at *dispatch time*, regardless of whether the worker runs in the same process. So the true 3-process topology is the *deployment-faithful* home, **not the only one**: the same failure also reproduces under the classic in-process pytest harness. Treat Mode 2 here as the production-faithful demonstration (and the only thing that also exercises the genuinely cross-process surfaces — LibraryCrate propagation, serialization), and add the cheaper automated catch as a pytest companion (next subsection).++**Enabler that makes this clean:** `pipelex validate bundle` now has a `--temporal/--no-temporal` flag (added alongside the fix; parity with `pipelex run`). So the repro needs **no `pipelex_temporary_override.toml` juggling** — just pass `--temporal`. (The flag flips the boot's hub default to the Temporal router; with the fix, the sweep still stays in-process — which is the whole point being asserted.)++> **Forward-looking note (why the flag exists despite being a no-op for the sweep today):** once validation runs as a standalone Temporal activity ([`../dry-run-refactor/followup-temporal-validation-activity.md`](../dry-run-refactor/followup-temporal-validation-activity.md)), `--temporal` on `validate` stops being a no-op and becomes the switch that dispatches the sweep through that activity. Until then it only controls the boot, which is exactly what this scenario needs.++**Repro bundle — reuse an existing one, no new fixture needed:**+`tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds` already declares a **standalone `type = "PipeBatch"` pipe** (`batch_temporal_describe_topics`). When the sweep dry-runs that pipe directly, it fans out → the exact shape that 422'd in production.++### Prereqs+Temporal server + worker up, per `references/mode-2-setup.md` (read it first). The worker config is irrelevant to the leak — the leak is entirely submitter-side — but keep a worker up to mirror production (and so the first dispatch actually *starts*, yielding the true `WorkflowAlreadyStartedError` rather than an `RPCError`; both land in the same `except` branch, so both are RED, but the former matches production).++### GREEN — the fix in place (current `HEAD`)++```bash+# Server up (Mode 2). The boot connects to it; with the fix the sweep stays in-process,+# so the worker should receive NO workflow for this validate run.+timeout 120 .venv/bin/pipelex validate bundle \+ tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \+ --temporal 2>&1 | tail -20+echo "EXIT=$?"+```++Expected: `EXIT=0`, `Successfully validated bundle ...`. Strong check: the worker session shows **no `WfPipeRouter` / `WfPipeRun` execution** for this run (it stayed idle) — `tmux capture-pane -t temporal-worker-router -p -S -200 | grep -i WfPipeRouter` returns nothing new.++### RED — temporarily revert the fix to prove the scenario bites++The fix is committed, so you must undo it in the working tree. **Surgical (preferred)** — neutralize just the scope in `pipelex/pipeline/bundle_validator.py`, inside `validate_pipes`:++```python+# change THIS:+with scoped_pipe_router(self._pipe_router):+ for pipe in sweepable_pipes:+ results[pipe.pipe_ref] = await self._classify_pipe(...)++# to THIS (drop the scope — the leak returns):+for pipe in sweepable_pipes:+ results[pipe.pipe_ref] = await self._classify_pipe(...)+```++(Alternative: `git revert --no-commit 3377babb` — but that also reverts the regression test + changelog; the surgical edit is cleaner.)++Re-run the GREEN command. Expected RED:++```+EXIT=1+Dry run failed with 1 unexpected pipe failure(s):+ 'temporal_batch_test.batch_temporal_describe_topics': ... status=DryRunStatus.FAILURE+ error_message="... Failed to execute workflow WfPipeRouter"+```++The worker session will now show `WfPipeRouter` activity (the leak — validation dispatched to Temporal). **Restore the fix immediately:**++```bash+git checkout -- pipelex/pipeline/bundle_validator.py # or: git revert --abort, if you used revert+```++### PASS criteria for the scenario+- GREEN run exits 0 **and** the worker received no workflow dispatch (validation stayed in-process under a Temporal-enabled boot).+- The documented RED procedure reproduces the `Failed to execute workflow WfPipeRouter` failure (sanity that the scenario actually exercises the leak path).++### Caveats to put in the scenario text+- **Shape matters.** Only a *standalone* `PipeBatch`/`PipeParallel` swept directly turns the leak fatal (concurrent same-id collision). A batch reached only as a sequence sub-pipe round-trips Temporal but passes — a false pass. `temporal_batch.mthds` has the standalone batch, so it's a valid RED trigger.+- **Worker up vs down both RED** (collision → `WorkflowAlreadyStartedError` vs no-worker → `RPCError`; same `except` branch). Keep the worker up to match production.+- **GREEN means "no dispatch happened."** Because the flag makes validation backend-agnostic, the strongest GREEN assertion is the worker-idle check, not just exit 0.++### Automated companion — a Mode-1 pytest (recommended alongside the Mode 2 scenario)++The Mode 2 scenario is agent-run, not CI-automated. Because this bug is submitter-side, the same catch can live as an ordinary pytest in the classic in-process Temporal harness (`tests/integration/pipelex/temporal/`, `--temporal-server none`/`local`, in-process worker) — fast, deterministic, runs in CI. That is the cheaper automated guard; Mode 2 remains the deployment-faithful demonstration. The three layers are complementary, not redundant:++| Layer | Where it runs | CI-automated | Cross-process surfaces (crate / serialization) | Status |+|---|---|---|---|---|+| Sentinel regression test (no Temporal) | plain pytest (`tests/integration/pipelex/pipeline/test_bundle_validator.py`) | ✅ | ✗ (deterministic stand-in for the hub default) | **exists** (`test_standalone_batch_sweep_scopes_in_process_router`) |+| **Mode-1 pytest (real Temporal, in-process server+worker)** | classic pytest temporal env | ✅ | ✗ (worker shares the process) | **proposed here** |+| Mode 2 scenario (3 separate processes) | the skill, agent-run | ✗ | ✓ | proposed in §5 |++Sketch of the Mode-1 pytest (author it in `tests/integration/pipelex/temporal/`, marked `temporal`, honoring `--temporal-server`):++- Boot/scope a **Temporal-enabled hub** so `get_pipe_router()`'s default is the real `TemporalPipeRouter` (mirror the existing temporal-suite fixtures that stand up the in-process server + worker).+- Run the real sweep — `BundleValidator().validate_pipes([...standalone PipeBatch...], ...)` — over `temporal_batch.mthds`'s `batch_temporal_describe_topics` (or an equivalent standalone `type = "PipeBatch"`).+- **Assert GREEN, strongly:** the sweep succeeds **and** no top-level workflow was started — e.g. spy on `pipelex.temporal.tprl.workflow_caller.WorkflowExecutor.execute_workflow` and assert it was never called (the deterministic analogue of the Mode 2 "worker-idle" check). This is what makes the fix's contract ("the sweep never reaches the Temporal router") an automated invariant, not just "validate didn't 422."+- Without the fix this test goes RED (the spy fires / a collision surfaces), exactly like the Mode 2 scenario.++This differs from the existing sentinel test in one meaningful way: it resolves the **real** `TemporalPipeRouter` as the hub default instead of a raising stand-in, so it also proves the *wiring* (config → hub default router) that the sentinel test deliberately bypasses.++## 6. Pointers++- Fix commit: `3377babb` — `fix(bundle-validator): prevent nested controller sub-pipes from leaking to Temporal during dry runs`.+- Fix site: `pipelex/pipeline/bundle_validator.py` (`__init__` keeps `self._pipe_router`; `validate_pipes` wraps the sweep in `scoped_pipe_router`). Precedent: `pipelex/runtime_bridge/bridge.py::_run_direct`.+- Hub wiring that makes the hub default Temporal: `pipelex/pipelex.py` (`set_pipe_router(make_temporal_pipe_router())` when `temporal.is_enabled`); resolution in `pipelex/hub.py::get_pipe_router` (contextvar override → else hub default).+- Leak sites: `pipelex/pipe_controllers/sub_pipe.py`, `pipelex/pipe_controllers/batch/pipe_batch.py` (`get_pipe_router().run(...)`).+- New CLI flag: `--temporal/--no-temporal` on `pipelex validate {bundle,pipe,method}` + `validate --all` (`pipelex/cli/commands/validate/`), guarded by `tests/unit/pipelex/cli/test_validate_temporal_flag.py`.+- In-process regression test: `tests/integration/pipelex/pipeline/test_bundle_validator.py::TestBundleValidatorIntegration::test_standalone_batch_sweep_scopes_in_process_router`.+- Skill: `.claude/skills/temporal-e2e-validate/SKILL.md` + `references/mode-2-setup.md` + `references/mode-2-tiers.md`.